devbox
在线问答5
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

docker部署kafka zookeeper模式集群_docker kafka集群

作者：在线问答5 | 2024-07-29 07:40:58

赞

踩

docker kafka集群

单机模式链接：https://blog.csdn.net/wsdhla/article/details/133032238

kraft集群模式链接：部署Kafka_kafka 部署-CSDN博客

zookeeper选举机制举例：

目前有5台服务器，每台服务器均没有数据，它们的编号分别是1,2,3,4,5,按编号依次启动，它们的选择举过程如下：

  • 服务器1启动，给自己投票，然后发投票信息，由于其它机器还没有启动所以它收不到反馈信息，服务器1的状态一直属于Looking(选举状态)。
  • 服务器2启动，给自己投票，同时与之前启动的服务器1交换结果，由于服务器2的编号大所以服务器2胜出，但此时投票数没有大于半数，所以两个服务器的状态依然是LOOKING。
  • 服务器3启动，给自己投票，同时与之前启动的服务器1,2交换信息，由于服务器3的编号最大所以服务器3胜出，此时投票数正好大于半数，所以服务器3成为领导者，服务器1,2成为小弟。
  • 服务器4启动，给自己投票，同时与之前启动的服务器1,2,3交换信息，尽管服务器4的编号大，但之前服务器3已经胜出，所以服务器4只能成为小弟。
  • 服务器5启动，后面的逻辑同服务器4成为小弟。

Docker 自定义静态网络

Docker默认提供了3种网络模式，生成容器时不指定网络模式下默认使用bridge桥接模式。每次容器启动时ip会发生变更，无法达到我们集群的效果。

创建自定义网络模式，指定网段。

docker network create --subnet=172.18.0.0/16 net-kafka

若报错：Error response from daemon: Pool overlaps with other one on this address space

说明指定的网络地址段已存在

1、Docker部署zookeeper集群

zk使用官方镜像：

1.1、主机创建目录

  1. [zk01]
  2. D:/mysoft/dev/docker-zk-cluster/zk01/conf
  3. D:/mysoft/dev/docker-zk-cluster/zk01/data
  4. D:/mysoft/dev/docker-zk-cluster/zk01/datalog
  5.  
  6.  
  7. [zk02]
  8. D:/mysoft/dev/docker-zk-cluster/zk02/conf
  9. D:/mysoft/dev/docker-zk-cluster/zk02/data
  10. D:/mysoft/dev/docker-zk-cluster/zk02/datalog
  11.  
  12.  
  13. [zk03]
  14. D:/mysoft/dev/docker-zk-cluster/zk03/conf
  15. D:/mysoft/dev/docker-zk-cluster/zk03/data
  16. D:/mysoft/dev/docker-zk-cluster/zk03/datalog

1.2、修改配置文件

分别拷贝到zk01、zk02、zk03

  1. [zoo.cfg]
  2.  
  3. dataDir=/data
  4. dataLogDir=/datalog
  5. tickTime=2000
  6. initLimit=5
  7. syncLimit=2
  8. autopurge.snapRetainCount=3
  9. autopurge.purgeInterval=0
  10. maxClientCnxns=60
  11. standaloneEnabled=true
  12. admin.enableServer=true
  13. server.1=172.18.0.11:2888:3888;2181
  14. server.2=172.18.0.12:2888:3888;2181
  15. server.3=172.18.0.13:2888:3888;2181

1.3、指定ip创建启动容器

  1. docker run -d --name zk01 -p 2181:2181 --network net-kafka --ip 172.18.0.11 -v D:/mysoft/dev/docker-zk-cluster/zk01/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk01/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk01/datalog:/datalog zookeeper
  2.  
  3. docker run -d --name zk02 -p 2182:2181 --network net-kafka --ip 172.18.0.12 -v D:/mysoft/dev/docker-zk-cluster/zk02/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk02/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk02/datalog:/datalog zookeeper
  4.  
  5. docker run -d --name zk03 -p 2183:2181 --network net-kafka --ip 172.18.0.13 -v D:/mysoft/dev/docker-zk-cluster/zk03/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk03/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk03/datalog:/datalog zookeeper

zk01可以正常启动，zk02和zk03启动失败

警告：这个正常，是因为节点启动时选举机制的原因

zk02：

WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 2 at election address /172.18.0.12:3888

zk03：

WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888

异常：

zk02：

ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Exception while listening to address /172.18.0.11:3888

ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Leaving listener thread for address 172.18.0.11:3888 after 3 errors. Use zookeeper.electionPortBindRetry property to increase retry count.

ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - As I'm leaving the listener thread, I won't be able to participate in leader election any longer: 172.18.0.11:3888

zk03：

ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Exception while listening to address /172.18.0.11:3888

ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Leaving listener thread for address 172.18.0.11:3888 after 3 errors. Use zookeeper.electionPortBindRetry property to increase retry count.

ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - As I'm leaving the listener thread, I won't be able to participate in leader election any longer: 172.18.0.11:3888

原因是没有配置quorumListenOnAllIPs，默认为false，需要设置为true

  1. [zoo.cfg]
  2.  
  3. dataDir=/data
  4. dataLogDir=/datalog
  5. tickTime=2000
  6. initLimit=5
  7. syncLimit=2
  8. autopurge.snapRetainCount=3
  9. autopurge.purgeInterval=0
  10. maxClientCnxns=60
  11. standaloneEnabled=true
  12. admin.enableServer=true
  13. quorumListenOnAllIPs=true
  14. server.1=172.18.0.11:2888:3888;2181
  15. server.2=172.18.0.12:2888:3888;2181
  16. server.3=172.18.0.13:2888:3888;2181

quorumListenOnAllIPs：

当设置为true时，ZooKeeper服务器将在所有可用IP地址上侦听来自其对等方的连接，而不仅是在配置文件的服务器列表中配置的地址。它会影响处理ZAB协议和快速领导者选举协议的连接。默认值为false。

有一些WARN:

  1. ### 1、只启动zk01：
  2. # zk01：
  3. 01:21:17.746 [QuorumConnectionThread-[myid=1]-3] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 2 at election address /172.18.0.12:3888
  4. java.net.NoRouteToHostException: No route to host (Host unreachable)
  5.  
  6. 01:21:20.904 [QuorumConnectionThread-[myid=1]-1] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
  7. java.net.NoRouteToHostException: No route to host (Host unreachable)
  8.  
  9.  
  10. ### 2、 只启动zk01、zk02：
  11. # zk01：
  12. 01:21:46.504 [QuorumConnectionThread-[myid=1]-3] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
  13. java.net.NoRouteToHostException: No route to host (Host unreachable)
  14.  
  15. # zk02：
  16. 01:21:43.350 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
  17.  
  18. 01:21:48.824 [QuorumConnectionThread-[myid=1]-2] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
  19. java.net.NoRouteToHostException: No route to host (Host unreachable)
  20.  
  21.  
  22. ### 2、 启动zk01、zk02、zk03：
  23. # zk01正常：
  24. 01:31:34.579 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
  25. 01:31:34.579 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
  26. 01:31:34.580 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
  27. 01:31:34.584 [QuorumConnectionThread-[myid=1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
  28. 01:31:34.584 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
  29. 01:31:34.585 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
  30. 01:31:34.585 [QuorumConnectionThread-[myid=1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
  31. 01:31:34.585 [QuorumConnectionThread-[myid=1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
  32. 01:31:34.585 [QuorumConnectionThread-[myid=1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:2)
  33. 01:31:34.586 [QuorumConnectionThread-[myid=1]-21] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2)
  34. 01:31:34.585 [QuorumConnectionThread-[myid=1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
  35. 01:31:34.586 [QuorumConnectionThread-[myid=1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:3)
  36. 01:31:34.586 [QuorumConnectionThread-[myid=1]-22] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3)
  37. 01:31:52.173 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - node0 scavenging sessions
  38. 01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.SessionHandler873610597==dftMaxIdleSec=-1 scavenging sessions
  39. 01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.SessionHandler873610597==dftMaxIdleSec=-1 scavenging session ids []
  40. 01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.DefaultSessionCache@57d7f8ca[evict=-1,removeUnloadable=false,saveOnCreate=false,saveOnInactiveEvict=false] checking expiration on []
  41.  
  42. # zk02：
  43. 01:35:15.331 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
  44. 01:35:15.332 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
  45. 01:35:15.333 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
  46. 01:35:15.334 [QuorumConnectionThread-[myid=1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
  47. 01:35:15.334 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
  48. 01:35:15.335 [QuorumConnectionThread-[myid=1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
  49. 01:35:15.335 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
  50. 01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.12:50246
  51. 01:35:15.335 [QuorumConnectionThread-[myid=1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:2)
  52. 01:35:15.335 [QuorumConnectionThread-[myid=1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
  53. 01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.12:50246
  54. 01:35:15.335 [QuorumConnectionThread-[myid=1]-27] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2)
  55. 01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
  56. 01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
  57. 01:35:15.335 [QuorumConnectionThread-[myid=1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
  58. 01:35:15.335 [QuorumConnectionThread-[myid=1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:3)
  59. 01:35:15.336 [QuorumConnectionThread-[myid=1]-28] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3)
  60. 01:35:34.600 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.11:49750
  61. 01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.11:49750
  62. 01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
  63. 01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
  64. 01:35:59.199 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.13:56956
  65. 01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.13:56956
  66. 01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
  67. 01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
  68.  
  69. # zk03：
  70. 01:36:59.201 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
  71. 01:36:59.201 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
  72. 01:36:59.202 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
  73. 01:36:59.202 [QuorumConnectionThread-[myid=1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
  74. 01:36:59.203 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
  75. 01:36:59.203 [QuorumConnectionThread-[myid=1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
  76. 01:36:59.203 [QuorumConnectionThread-[myid=1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:2)
  77. 01:36:59.203 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
  78. 01:36:59.203 [QuorumConnectionThread-[myid=1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
  79. 01:36:59.203 [QuorumConnectionThread-[myid=1]-15] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2)
  80. 01:36:59.204 [QuorumConnectionThread-[myid=1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
  81. 01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.13:35192
  82. 01:36:59.204 [QuorumConnectionThread-[myid=1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:3)
  83. 01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.13:35192
  84. 01:36:59.204 [QuorumConnectionThread-[myid=1]-16] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3)
  85. 01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
  86. 01:36:59.205 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
  87. 01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.12:35956
  88. 01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.12:35956
  89. 01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
  90. 01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
  91. 01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.11:46420
  92. 01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.11:46420
  93. 01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
  94. 01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.

找到原因了，因为data/myid没有修改，目前3个节点的配置全部为1，分别改为1、2、3就正常了。

启动zkui

docker run -d --name zkui -p 9090:9090  --network net-kafka -e ZKLIST=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 maauso/zkui

访问localhost:9090，登录之后报错：KeeperErrorCode = ConnectionLoss for /

删除data\version-2\snapshot.0依然不行

stat is not executed because it is not in the whitelist.
envi is not executed because it is not in the whitelist.

4lw.commands.whitelist=*

  1. 16:49:58.466 [NIOWorkerThread-10] DEBUG org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /172.18.0.2:42978 (no session established for client)
  2. 16:49:58.499 [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181] DEBUG org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /172.18.0.2:42980
  3. 16:49:58.500 [NIOWorkerThread-11] WARN org.apache.zookeeper.server.NIOServerCnxn - Close of session 0x0
  4. java.io.IOException: ZooKeeperServer not running
  5.         at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:569)
  6.         at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:349)
  7.         at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
  8.         at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:153)
  9.         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  10.         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  11.         at java.base/java.lang.Thread.run(Unknown Source)

好像改为zk的data/myid配置，上面的问题就没有了。

2、Docker部署kafka集群 

kafka没有官方镜像，使用wurstmeister/kafka镜像

2.1、主机创建目录

  1. [kafka01]
  2. D:/mysoft/dev/docker-kafka-cluster/kafka01/config
  3. D:/mysoft/dev/docker-kafka-cluster/kafka01/data
  4. D:/mysoft/dev/docker-kafka-cluster/kafka01/logs
  5. D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts
  6.  
  7.  
  8. [kafka02]
  9. D:/mysoft/dev/docker-kafka-cluster/kafka02/config
  10. D:/mysoft/dev/docker-kafka-cluster/kafka02/data
  11. D:/mysoft/dev/docker-kafka-cluster/kafka02/logs
  12. D:/mysoft/dev/docker-kafka-cluster/kafka02/hosts
  13.  
  14.  
  15. [kafka03]
  16. D:/mysoft/dev/docker-kafka-cluster/kafka03/config
  17. D:/mysoft/dev/docker-kafka-cluster/kafka03/data
  18. D:/mysoft/dev/docker-kafka-cluster/kafka03/logs
  19. D:/mysoft/dev/docker-kafka-cluster/kafka03/hosts

2.2、修改配置文件

分别拷贝到kafka01、kafka02、kafka03

  1. [server.properties]
  2.  
  3. broker.id=1
  4. advertised.listeners=PLAINTEXT://172.18.0.21:9092
  5. # port=9092
  6. # log.dirs=/kafka/kafka-logs-1ddafafe54d0
  7. listeners=PLAINTEXT://0.0.0.0:9092
  8. zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181

2.3、指定ip创建启动容器

  1. # docker run -d --name kafka01 --restart=always --net=host -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc/hosts -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_HOST_NAME=192.168.180.42 -e JMX_PORT=9999 -e HOST_IP=127.0.0.1 -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_ZOOKEEPER_CONNECT=127.0.0.1:2181 -e KAFKA_BROKER_ID=1 -e KAFKA_HEAP_OPTS="-Xmx6G -Xms6G" wurstmeister/kafka
  2. # docker run -d --name kafka01 --restart=always --network net-kafka --ip 172.18.0.21 -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_HOST_NAME=172.18.0.21 -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.21:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=1 -e KAFKA_LOG_DIRS=/opt/kafka/logs wurstmeister/kafka
  3.  
  4.  
  5. docker run -d --name kafka01 -p 9192:9092 --restart=always --network net-kafka --ip 172.18.0.21 -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.21:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=1 wurstmeister/kafka
  6.  
  7.  
  8. docker run -d --name kafka02 -p 9292:9092 --restart=always --network net-kafka --ip 172.18.0.22 -v D:/mysoft/dev/docker-kafka-cluster/kafka02/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka02/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka02/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka02/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.22:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=2 wurstmeister/kafka
  9.  
  10.  
  11. docker run -d --name kafka03 -p 9392:9092 --restart=always --network net-kafka --ip 172.18.0.23 -v D:/mysoft/dev/docker-kafka-cluster/kafka03/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka03/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka03/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka03/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.23:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=3 wurstmeister/kafka

kafka容器异常，一直自动反复重启，UP状态维持1分钟左右

  1. [Configuring] 'advertised.listeners' in '/opt/kafka/config/server.properties'
  2. [Configuring] 'port' in '/opt/kafka/config/server.properties'
  3. Excluding KAFKA_HOME from broker config
  4. [Configuring] 'log.dirs' in '/opt/kafka/config/server.properties'
  5. [Configuring] 'listeners' in '/opt/kafka/config/server.properties'
  6. Excluding KAFKA_VERSION from broker config
  7. [Configuring] 'zookeeper.connect' in '/opt/kafka/config/server.properties'
  8. [Configuring] 'broker.id' in '/opt/kafka/config/server.properties'
  9. log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/bin/../config/log4j.properties].
  10. java.io.FileNotFoundException: /opt/kafka/bin/../config/log4j.properties (No such file or directory)
  11.         at java.base/java.io.FileInputStream.open0(Native Method)
  12.         at java.base/java.io.FileInputStream.open(Unknown Source)
  13.         at java.base/java.io.FileInputStream.<init>(Unknown Source)
  14.         at java.base/java.io.FileInputStream.<init>(Unknown Source)
  15.         at java.base/sun.net.www.protocol.file.FileURLConnection.connect(Unknown Source)
  16.         at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(Unknown Source)
  17.         at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
  18.         at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
  19.         at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
  20.         at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
  21.         at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
  22.         at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
  23.         at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
  24.         at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
  25.         at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
  26.         at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
  27.         at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
  28.         at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala:25)
  29.         at kafka.Kafka$.<clinit>(Kafka.scala:30)
  30.         at kafka.Kafka.main(Kafka.scala)
  31. log4j:ERROR Ignoring configuration file [file:/opt/kafka/bin/../config/log4j.properties].
  32. log4j:WARN No appenders could be found for logger (kafka.utils.Log4jControllerRegistration$).
  33. log4j:WARN Please initialize the log4j system properly.
  34. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

指定-e KAFKA_LOG_DIRS也依然不可以，手动拷贝一份log4j.properties到映射的目录D:/mysoft/dev/docker-kafka-cluster/kafka01/logs下问题就解决了。

但是出现另外一个异常：

  1. [2023-12-23 15:20:50,136] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  2. [2023-12-23 15:20:50,204] INFO Socket connection established, initiating session, client: /172.18.0.21:39814, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
  3. [2023-12-23 15:20:50,273] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  4. [2023-12-23 15:20:50,749] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  5. [2023-12-23 15:20:50,811] INFO Socket connection established, initiating session, client: /172.18.0.21:49980, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn)
  6. [2023-12-23 15:20:50,914] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  7. [2023-12-23 15:20:52,268] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  8. [2023-12-23 15:20:52,356] INFO Socket connection established, initiating session, client: /172.18.0.21:40344, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn)
  9. [2023-12-23 15:20:52,446] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  10. [2023-12-23 15:20:52,617] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
  11. [2023-12-23 15:20:52,817] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  12. [2023-12-23 15:20:52,855] INFO Socket connection established, initiating session, client: /172.18.0.21:39828, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
  13. [2023-12-23 15:20:53,036] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
  14. [2023-12-23 15:20:53,037] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
  15. [2023-12-23 15:20:53,076] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
  16. [2023-12-23 15:20:53,222] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
  17. kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
  18.         at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:271)
  19.         at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)
  20.         at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)
  21.         at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)
  22.         at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)
  23.         at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
  24.         at kafka.Kafka$.main(Kafka.scala:109)
  25.         at kafka.Kafka.main(Kafka.scala)
  26. [2023-12-23 15:20:53,290] INFO shutting down (kafka.server.KafkaServer)
  27. [2023-12-23 15:20:53,375] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
  28. [2023-12-23 15:20:53,440] INFO shut down completed (kafka.server.KafkaServer)
  29. [2023-12-23 15:20:53,515] ERROR Exiting Kafka. (kafka.Kafka$)
  30. [2023-12-23 15:20:53,590] INFO shutting down (kafka.server.KafkaServer)

回头看zk的控制台，发现报错java.io.IOException: ZooKeeperServer not running，跟用zkui客户端连zk集群一样的错，通过修改zk的data/myid配置，之后，3个kafka服务都能正常启动了，而且也不会反复自动重启了。

  1. # kafka01
  2. [2023-12-25 02:08:01,176] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)
  3.  
  4. # kafka02
  5. [2023-12-25 02:09:56,391] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)
  6.  
  7. # kafka03
  8. [2023-12-25 02:10:14,479] INFO [KafkaServer id=3] started (kafka.server.KafkaServer)

虽然kafka服务在启动的初始阶段，仍然有一下异常，但是目前没有发现有什么影响：

  1. [2023-12-25 00:52:15,882] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  2. [2023-12-25 00:52:15,932] INFO Socket connection established, initiating session, client: /172.18.0.21:49568, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
  3. [2023-12-25 00:52:15,984] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  4. [2023-12-25 00:52:17,040] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  5. [2023-12-25 00:52:17,094] INFO Socket connection established, initiating session, client: /172.18.0.21:59734, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn)
  6. [2023-12-25 00:52:17,148] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  7. [2023-12-25 00:52:17,979] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  8. [2023-12-25 00:52:18,033] INFO Socket connection established, initiating session, client: /172.18.0.21:50098, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn)
  9. [2023-12-25 00:52:18,088] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  10. [2023-12-25 00:52:19,308] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  11. [2023-12-25 00:52:19,357] INFO Socket connection established, initiating session, client: /172.18.0.21:49580, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
  12. [2023-12-25 00:52:19,403] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  13. [2023-12-25 00:52:19,974] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  14. [2023-12-25 00:52:20,027] INFO Socket connection established, initiating session, client: /172.18.0.21:59746, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn)
  15. [2023-12-25 00:52:20,075] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
  16. [2023-12-25 00:52:20,549] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
  17. [2023-12-25 00:52:20,812] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
  18. [2023-12-25 00:52:20,860] INFO Socket connection established, initiating session, client: /172.18.0.21:50110, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn)
  19. [2023-12-25 00:52:21,016] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
  20. [2023-12-25 00:52:21,017] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
  21. [2023-12-25 00:52:21,062] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
  22. [2023-12-25 00:52:21,164] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
  23. kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
  24.         at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:271)
  25.         at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)
  26.         at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)
  27.         at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)
  28.         at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)
  29.         at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
  30.         at kafka.Kafka$.main(Kafka.scala:109)
  31.         at kafka.Kafka.main(Kafka.scala)
  32. [2023-12-25 00:52:21,213] INFO shutting down (kafka.server.KafkaServer)
  33. [2023-12-25 00:52:21,274] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
  34. [2023-12-25 00:52:21,319] INFO shut down completed (kafka.server.KafkaServer)
  35. [2023-12-25 00:52:21,371] ERROR Exiting Kafka. (kafka.Kafka$)
  36. [2023-12-25 00:52:21,425] INFO shutting down (kafka.server.KafkaServer)

启动sheepkiller/kafka-manager

  1. # docker run -d --name kafka-manager -p 9000:9000 --network net-kafka --ip 172.18.0.31 -e ZK_HOSTS=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 sheepkiller/kafka-manager
  2.  
  3. docker run -d --name kafka-manager -p 9000:9000 --network net-kafka -e ZK_HOSTS=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 sheepkiller/kafka-manager

或者使用provectuslabs/kafka-ui:latest

  1. # docker run -d --name kafka-ui -p 8080:8080 --network net-kafka --ip 172.18.0.32 -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-ui
  2.  
  3. docker run -d --name kafka-ui -p 8080:8080 --network net-kafka -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-ui

3、完整配置

3.1 zk完整配置

zk01/conf/zoo.cfg

  1. dataDir=/data
  2. dataLogDir=/datalog
  3. tickTime=2000
  4. initLimit=5
  5. syncLimit=2
  6. autopurge.snapRetainCount=3
  7. autopurge.purgeInterval=0
  8. maxClientCnxns=60
  9. standaloneEnabled=true
  10. admin.enableServer=true
  11. quorumListenOnAllIPs=true
  12. server.1=172.18.0.11:2888:3888;2181
  13. server.2=172.18.0.12:2888:3888;2181
  14. server.3=172.18.0.13:2888:3888;2181
  15. 4lw.commands.whitelist=*

zk01/data/myid

1

zk02/conf/zoo.cfg

  1. dataDir=/data
  2. dataLogDir=/datalog
  3. tickTime=2000
  4. initLimit=5
  5. syncLimit=2
  6. autopurge.snapRetainCount=3
  7. autopurge.purgeInterval=0
  8. maxClientCnxns=60
  9. standaloneEnabled=true
  10. admin.enableServer=true
  11. quorumListenOnAllIPs=true
  12. server.1=172.18.0.11:2888:3888;2181
  13. server.2=172.18.0.12:2888:3888;2181
  14. server.3=172.18.0.13:2888:3888;2181
  15. 4lw.commands.whitelist=*

 zk02/data/myid

2

zk03/conf/zoo.cfg

  1. dataDir=/data
  2. dataLogDir=/datalog
  3. tickTime=2000
  4. initLimit=5
  5. syncLimit=2
  6. autopurge.snapRetainCount=3
  7. autopurge.purgeInterval=0
  8. maxClientCnxns=60
  9. standaloneEnabled=true
  10. admin.enableServer=true
  11. quorumListenOnAllIPs=true
  12. server.1=172.18.0.11:2888:3888;2181
  13. server.2=172.18.0.12:2888:3888;2181
  14. server.3=172.18.0.13:2888:3888;2181
  15. 4lw.commands.whitelist=*

 zk03/data/myid

3

3.2 kafka完整配置

kafka01/config/server.properties

  1. broker.id=1
  2. advertised.listeners=PLAINTEXT://172.18.0.21:9092
  3. port=9092
  4. log.dirs=/kafka/kafka-logs-7725cecb8a4f
  5. listeners=PLAINTEXT://0.0.0.0:9092
  6. zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181

kafka02/config/server.properties

  1. broker.id=2
  2. advertised.listeners=PLAINTEXT://172.18.0.22:9092
  3. port=9092
  4. log.dirs=/kafka/kafka-logs-2b3d2a64f445
  5. listeners=PLAINTEXT://0.0.0.0:9092
  6. zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181

kafka03/config/server.properties

  1. broker.id=3
  2. advertised.listeners=PLAINTEXT://172.18.0.23:9092
  3. port=9092
  4. log.dirs=/kafka/kafka-logs-d9101a867f99
  5. listeners=PLAINTEXT://0.0.0.0:9092
  6. zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181

4、问题

SpringBoot配置：

  1. spring:
  2.   kafka:
  3.     bootstrap-servers: localhost:9192,localhost:9292,localhost:9392

报错，与单机版错误一致

  1. 2023-12-26 15:27:29.296  INFO 11904 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.8.1
  2. 2023-12-26 15:27:29.298  INFO 11904 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 839b886f9b732b15
  3. 2023-12-26 15:27:29.298  INFO 11904 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1703575649293
  4. 2023-12-26 15:27:29.302  INFO 11904 --- [           main] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Subscribed to topic(s): test-topic03
  5.  
  6.  
  7. 2023-12-26 15:27:30.191  INFO 11904 --- [ntainer#0-0-C-1] org.apache.kafka.clients.Metadata        : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Cluster ID: o6TS0_kER-KgmyB7NWjjEw
  8. 2023-12-26 15:27:30.194  INFO 11904 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Discovered group coordinator 172.18.0.23:9092 (id: 2147483644 rack: null)
  9. 2023-12-26 15:27:30.200  INFO 11904 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] (Re-)joining group
  10. 2023-12-26 15:27:38.804  INFO 11904 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Group coordinator 172.18.0.23:9092 (id: 2147483644 rack: null) is unavailable or invalid due to cause: null.isDisconnected: true. Rediscovery will be attempted.

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/在线问答5/article/detail/897783
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号