Docker-compose搭建zookeeper-zookeeper集群

前言

最近在研究kafka与rabbitMQ对比，而因为自己电脑性能不是那么好，要搭建一个kafka集群和zookeeper集群最少也要开三台虚拟机，而且也挺麻烦，所以毋庸置疑选择了docker-compose来搭建一个伪分布式的集群。参考了很多资料，很多乱七八糟都是转发的，真正去研究的很少，所以在这里记录一下自己的搭建过程

软件版本

• docker： Docker version 1.13.1, build b2f74b2/1.13.1
• docke-compose：docker-compose version 1.23.2, build 1110ad0
• docker_ zookeeper_image : zookeeper:3.5.7
• docker_ kafka_iamge: wurstmeister/kafka:latest
• docker_ kafka-manager_image: sheepkiller/kafka-manager:latest

Compose file format compatibility matrix

怎么安装的在这里就不多赘述了，相信必应、百度都有很多

准备

如果你对防火墙不感冒，那么在测试的时候建议你关闭防火墙；反之请开放如下端口

宿主机防火墙开放：9000/tcp 、2181/tcp、2182/tcp、2183/tcp、9092/tcp、9093/tcp、9094/tcp端口

Kafka是依赖zookeeper来存放一些数据的，所以kafka集群的启动必须依赖zookeeper集群

docker-compose.yml

version: '2'

services:
  zookeeper1:
    image: zookeeper:3.5.7
    hostname: zookeeper1
    container_name: zookeeper1
    privileged: true
    ports:
      - "2181:2181"
    volumes:
      - ./zookeeper1/conf:/conf
      - ./zookeeper1/data:/data
      - ./zookeeper1/datalog:/datalog
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zookeeper2:2888:3888;2181 server.3=zookeeper3:2888:3888;2181
    networks:
      net_zoo_kafka:
        ipv4_address: 172.23.0.11

  zookeeper2:
    image: zookeeper:3.5.7
    hostname: zookeeper2
    container_name: zookeeper2
    privileged: true
    ports:
      - "2182:2181"
    volumes:
      - ./zookeeper2/conf:/conf
      - ./zookeeper2/data:/data
      - ./zookeeper2/datalog:/datalog
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zookeeper1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zookeeper3:2888:3888;2181
    networks:
      net_zoo_kafka:
        ipv4_address: 172.23.0.12

  zookeeper3:
    image: zookeeper:3.5.7
    hostname: zookeeper3
    container_name: zookeeper3
    privileged: true
    ports:
      - "2183:2181"
    volumes:
      - ./zookeeper3/conf:/conf
      - ./zookeeper3/data:/data
      - ./zookeeper3/datalog:/datalog
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zookeeper1:2888:3888;2181 server.2=zookeeper2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
    networks:
      net_zoo_kafka:
        ipv4_address: 172.23.0.13

  kafka1:
    image: docker.io/wurstmeister/kafka
    hostname: kafka1
    container_name: kafka1
    privileged: true
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.94.133:9092
      KAFKA_ADVERTISED_HOST_NAME: 192.168.94.133
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./kafka1:/kafka
    links:
      - zookeeper1
      - zookeeper2
      - zookeeper3
    networks:
      net_zoo_kafka:
        ipv4_address: 172.23.0.14

  kafka2:
    image: docker.io/wurstmeister/kafka
    hostname: kafka2
    container_name: kafka2
    privileged: true
    ports:
      - "9093:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.94.133:9093
      KAFKA_ADVERTISED_HOST_NAME: 192.168.94.133
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./kafka2:/kafka
    links:
      - zookeeper1
      - zookeeper2
      - zookeeper3
    networks:
      net_zoo_kafka:
        ipv4_address: 172.23.0.15

  kafka3:
    image: docker.io/wurstmeister/kafka
    hostname: kafka3
    container_name: kafka3
    privileged: true
    ports:
      - "9094:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.94.133:9094
      KAFKA_ADVERTISED_HOST_NAME: 192.168.94.133
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./kafka3:/kafka
    links:
      - zookeeper1
      - zookeeper2
      - zookeeper3
    networks:
      net_zoo_kafka:
        ipv4_address: 172.23.0.16

  kafka-manager:
    image: docker.io/sheepkiller/kafka-manager
    container_name: kafka-manager
    hostname: kafka-manager
    privileged: true
    ports:
      - "9000:9000"
    links:            # 连接本compose文件创建的container
      - zookeeper1
      - zookeeper2
      - zookeeper3
      - kafka1
      - kafka2
      - kafka3
    # external_links:   # 连接compose以外的container
    environment:
      ZK_HOSTS: zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka
      KAFKA_BROKERS: kafka1:9092,kafka2:9093,kafka3:9094
      APPLICATION_SECRET: letmein
      KM_ARGS: -Djava.net.preferIPv4Stack=true
    networks:
      net_zoo_kafka:
        ipv4_address: 172.23.0.17

networks:
  net_zoo_kafka:
    driver: bridge
    ipam:
      config:
        - subnet: 172.23.0.0/25
          gateway: 172.23.0.1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157

详解

1. version 选择 2 版本，因为3版本不支持gateway, 我这里为了方便，将网络的创建放在了compose文件中，你也可以在外部用docker network create --driver bridge --subnet 172.23.0.0/25 --gateway 172.23.0.1 zookeeper_network的方式创建一个bridge的网桥，compose文件中用external指定外部网络
2. 所有容器都在创建的网桥的子网内，避免访问不到的问题
3. zookeeper集群只需要暴露每个容器的2181端口出去，而2888/3888端口，因为容器之间在一个子网，可以互通，所以不需要暴露
4. kafka 集群的 KAFKA_ADVERTISED_LISTENERS / KAFKA_ADVERTISED_HOST_NAME 配置成你宿主机的IP ! 记得修改

验证zookeeper集群

启动后用 docker ps命令查看容器启动没有问题，应该成功启动了3个zookeeper容器、3个kafka容器和1个kafka-manager容器

进入zookeeper集群中任意一个容器

输入 zkServer.sh status命令查看zookeeper集群状态，此时你看到该节点 Node: follower 或者 Node: leader,表示zookeeper集群启动成功

验证kafka集群

Ctrl + p + q 退出容器

进入任意一台kafka容器 docker exec -it kafka1 /bin/bash ，进入 /opt/kafka/bin 目录

键入命令创建一个topic ./kafka-topics.sh --zookeeper zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka --create --replication-factor 2 --partitions 1 --topic topic1
键入命令查看所有topic ./kafka-topics.sh --zookeeper zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka --list
键入命令删除topic ./kafka-topics.sh --zookeeper zookeeper1:2181/kafka,zookeeper2:2182/kafka,zookeeper3:2183/kafka --delete --topic topic1

注：kafka的topic信息都是保存在zookeeper中的，所以要连接zookeeper集群，并且在上面的compose文件中我们明确的指定了kafka在zookeeper中的namespace, 即/kafka, 所以在操作topic时，务必指定namespace, 否则，会报错提示broker 为0

验证使用 Kafka-manager

再次确认宿主机 9000/tcp 端口是否开放，或者防火墙是否关闭

浏览器访问 宿主机IP:9000

添加一个Cluster

标红的地方注意一下，其他的地方根据你的需求修改；输入完成点击save

OK, 至此，整个zookeeper集群、kafka集群、kafka-manger都部署成功

结语

坑总是一个又一个，踩坑了，睡一觉，但别忘了爬出来。
如果大家有什么疑问，欢迎给我留言