卡夫卡消息服务器,单台服务器部署 Zookeeper 及 kafka 消息中间件服务

在开发环境中，为减少服务器成本和提高开发效率，通常在一台测试服务器部署各种服务，使用不同的本地 IP 和不同端口号将涉及到集群的服务往往跑在一台机器上模拟集群，在这里我们称做伪集群。

最近在研究一个矿池的源码，在搭建服务的过程中消息中间件 kafka (是用于构建实时数据管道和流应用程序) 需要用到 Zookeeper 集群。这篇文章记录在一台开发机器部署伪集群 Zookeeper 服务。

安装 Zookeeper

cd /tmp

// 下载最新版本的 zookeeper

wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz

// 解压缩安装包到指定目录, 在这里我们放在 /usr/local/ 目录下

tar -C /usr/local/ -zxf zookeeper-3.4.11.tar.gz

在 /usr/local/zookeeper-3.4.11 目录下可看到解压缩之后的文件夹，包含了 zookeeper 服务的配置文件和可执行文件。

修改配置文件

在 /usr/local/zookeeper-3.4.11/conf 目录下，有一个 zoo_sample.cfg 文件，首先我们复制三份，分别命名为 zoo_1.cfg, zoo_2.cfg, zoo_3.cfg

sudo cp zoo_sample.cfg zoo_1.cfg

sudo cp zoo_sample.cfg zoo_2.cfg

sudo cp zoo_sample.cfg zoo_3.cfg

然后修改这三个配置文件的内容，分别修改 dataDir, dataLogDir, clientPort 和 server.x 选项。

// zoo_1

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/home/hww/source_code/zookeeper/zk1

dataLogDir=/home/hww/source_code/zookeeper/zk1/log

clientPort=2181

server.1=127.0.0.1:2888:3888

// zoo_2

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/home/hww/source_code/zookeeper/zk2

dataLogDir=/home/hww/source_code/zookeeper/zk2/log

clientPort=2182

server.2=127.0.0.2:2888:3888

...

接着在每个节点的文件目录下增加 myid 文件和 version-2 文件夹。

// zoo_1 对应的数据文件夹

cd /home/hww/source_code/zookeeper/zk1

touch myid

echo 1 > myid

mkdir version-2

cd /home/hww/source_code/zookeeper/zk2

touch myid

echo 2 > myid

mkdir version-2

...

启动伪集群服务

根据不同的配置文件启动伪集群服务:

sudo /usr/local/zookeeper-3.4.11/bin/zkServer.sh start conf/zoo_x.cfg

sudo /usr/local/zookeeper-3.4.11/bin/zkServer.sh start conf/zoo_1.cfg

sudo /usr/local/zookeeper-3.4.11/bin/zkServer.sh start conf/zoo_2.cfg

sudo /usr/local/zookeeper-3.4.11/bin/zkServer.sh start conf/zoo_3.cfg

ps -ef | grep zoo_

root 845 1 0 16:30 pts/3 00:00:03 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/local/zookeeper-3.4.11/bin/../build/classes:/usr/local/zookeeper-3.4.11/bin/../build/lib/*.jar:/usr/local/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/usr/local/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/usr/local/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/usr/local/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/usr/local/zookeeper-3.4.11/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain conf/zoo_1.cfg

root 1380 1 0 16:37 pts/3 00:00:03 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/local/zookeeper-3.4.11/bin/../build/classes:/usr/local/zookeeper-3.4.11/bin/../build/lib/*.jar:/usr/local/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/usr/local/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/usr/local/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/usr/local/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/usr/local/zookeeper-3.4.11/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain conf/zoo_2.cfg

root 2855 1 0 16:46 pts/3 00:00:02 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/local/zookeeper-3.4.11/bin/../build/classes:/usr/local/zookeeper-3.4.11/bin/../build/lib/*.jar:/usr/local/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/usr/local/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/usr/local/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/usr/local/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/usr/local/zookeeper-3.4.11/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain conf/zoo_3.cfg

三个节点的集群服务已正常启动。

kafka 消息中间件

kafka 安装

sudo apt-get install -y default-jre

cd /tmp

wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/0.11.0.2/kafka_2.11-0.11.0.2.tgz

sudo tar -C /usr/local/kafka -zxf kafka_2.11-0.11.0.2.tgz --strip 1

kafka 配置

cd /usr/local/kafka

sudo vi config/server.properties

// 修改 kafka 配置文件

...

// 连接集群节点

zookeeper.connect=127.0.0.1:2181,127.0.0.2:2182,127.0.0.3:2183

...

启动 kafka 服务

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties

操作 kafka

在 zk1 新建 topic

/usr/local/kafka/bin/kafka-topics.sh --create --topic topic-test --zookeeper 1270.0.1:2181 --partitions 1 --replication-factor 1

列出 zk1 可用的 topic

/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181

通过 kafka-console-producer.sh 脚本生成消息

echo "hello world" | /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic topic-test

通过 kafka-console-consumer.sh 脚本消费消息

/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic topic-test --from-beginning

参考链接