kafka0.9升级纪实_cdh kafka升级

Kafka0.9升级到1.1纪实

本文结合官网和实际项目详细介绍kafka升级过程。

• 项目简介和升级需求
• Kafka升级介绍
• 升级实测

项目简介和升级需求

先贴图，如下是框架结构以及涉及的kafka版本

可以看到，整个技术栈都是用的是kafka0.9，CDH对应的kafka版本可以通过http://archive.cloudera.com/kafka/进行查询。
目前需要升级两个kafka集群到1.1版本，所以需要确定升级后对业务的影响。其实，一个优秀的组件，升级是不会对原有的上下游产生影响的，但BOSS不放心我也没辙，只得去测。

Kafka升级介绍

这里直接粘贴官网，需要注意的是需要滚动重启。

kafka集群：

Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, or 1.0.x to 1.1.x

Kafka 1.1.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below, you guarantee no downtime during the upgrade. However, please review the notable changes in 1.1.0 before upgrading.

For a rolling upgrade:

1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.

   • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0).

   • log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.)

   If you are upgrading from 0.11.0.x or 1.0.x and you have not overridden the message format, then you only need to override the inter-broker protocol format.

   • inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0 or 1.0)

2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it.

3. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 1.1.

4. Restart the brokers one by one for the new protocol version to take effect.

5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 1.1 on each broker and restart them one by one. Note that the older Scala consumer does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to take advantage of exactly once semantics), the newer Java consumer must be used.

CDH KAFKA集群

具体参见https://www.cloudera.com/documentation/kafka/latest/topics/kafka_installing.html

升级测试

测试环境：在本地部署单机Zookeeper和Kafka 1.1（网上资料很多），使用之前工程直接测试，发现收发数据都没有问题，符合之前的预期，即kafka集群升级没有影响上下游工程数据的接发。