热门标签
热门文章
- 11.Tomcat和JavaEE入门_tomcatee
- 2Unity性能优化之 - 图集_unity 动态加载的icon需要打图集吗
- 392岁高龄的创始人张忠谋谈台积电发展史
- 4《Git常用命令》详细讲解·第5篇（git log和git blame）(1)_git blame grep
- 5QUARTUS 下载文件到flash中_quartus 程序下载到flash
- 6ARM 处理器 MIPS/DMIPS/MFLOPS_嵌入式arm mflops测试
- 7启明云端分享| 盘点 ESP32-S3到底有哪些功能特性_esp32 s3 awtk
- 8qmake手册(Qt5.9.3)_some exotic name (tested names were: qmake qmake-q
- 9Python计算机视觉编程_03_章毓晋计算机视觉教程python代码实现
- 10接口测试工具有哪些，哪些比较火
当前位置: article > 正文
使用docker 和 kubnernetes 部署单节点/多节点 kafka 环境_kubernetes部署单节点kafka集群
作者：Monodyee | 2024-05-31 10:56:33
赞
踩
kubernetes部署单节点kafka集群
参考资料
- https://kafka.apachecn.org/documentation.html#configuration
kafka的broker有三个核心配置
- broker.id
- log.dirs
- zookeeper.connect
docker启动单节点kafka环境
可配置的环境变量，https://gallery.ecr.aws/bitnami/zookeeper
$ docker run -it --rm --name my-zookeeper -p 2181:2181 -e ALLOW_ANONYMOUS_LOGIN=yes zookeeper:3.7
# 查看节点状态
$ /apache-zookeeper-3.7.0-bin/bin/zkServer.sh status
- 1
- 2
- 3
- 4
启动kafka
可配置的环境变量，https://gallery.ecr.aws/bitnami/kafka
$ docker run -it --rm --name my-kafka -p 9092:9092 --link my-zookeeper \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ZOOKEEPER_CONNECT=my-zookeeper:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e ALLOW_PLAINTEXT_LISTENER=yes public.ecr.aws/bitnami/kafka:3.2.3
- 1
- 2
- 3
- 4
- 5
- 6
查看和创建topic
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic test --create --partitions 1 --replication-factor 1
- 1
- 2
测试本地环境
$ kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --from-beginning
$ kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
- 1
- 2
- 3
docker-compose启动单节点kafka
使用docker-composer部署
version: '3.8' services: dk-zookeeper: image: zookeeper:3.7.0 ports: - 2182:2181 environment: - ALLOW_ANONYMOUS_LOGIN=yes dk-kafka-01: image: public.ecr.aws/bitnami/kafka:3.2.3 ports: - 9092:9092 environment: KAFKA_BROKER_ID: 0 KAFKA_CFG_LISTENERS: PLAINTEXT://:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092 # KAFKA_CREATE_TOPICS: "test:1:0" //创建test的topic，具备1分区0副本 KAFKA_ZOOKEEPER_CONNECT: dk-zookeeper:2181 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 ALLOW_PLAINTEXT_LISTENER: yes depends_on: - dk-zookeeper
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
docker-compose启动多节点kafka集群
https://blog.csdn.net/qq_39526294/article/details/124293954
创建自定义网络
$ docker network create zookeeper_network
- 1
部署zookeeper
创建zookeeper服务
version: '3.8' networks: default: external: name: zookeeper_network services: zoo1: image: zookeeper:3.7.0 hostname: zoo1 ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 zoo2: image: zookeeper:3.7.0 hostname: zoo2 ports: - 2182:2181 environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181 zoo3: image: zookeeper:3.7.0 hostname: zoo3 ports: - 2183:2181 environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
启动zookeeper
$ docker-compose up
- 1
部署kafka
创建kafka服务
version: '3.8' networks: default: external: name: zookeeper_network services: kafka1: image: public.ecr.aws/bitnami/kafka:3.2.3 restart: unless-stopped container_name: kafka1 hostname: kafka1 ports: - "9092:9092" external_links: - zoo1 - zoo2 - zoo3 environment: KAFKA_BROKER_ID: 1 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.31.18.4:9092 KAFKA_ADVERTISED_HOST_NAME: kafka1 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181" ALLOW_PLAINTEXT_LISTENER: yes kafka2: image: public.ecr.aws/bitnami/kafka:3.2.3 restart: unless-stopped container_name: kafka2 hostname: kafka2 ports: - "9093:9092" external_links: - zoo1 - zoo2 - zoo3 environment: KAFKA_BROKER_ID: 2 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.225:9093 ## 宿主机IP KAFKA_ADVERTISED_HOST_NAME: kafka2 KAFKA_ADVERTISED_PORT: 9093 KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181" ALLOW_PLAINTEXT_LISTENER: yes kafka3: image: public.ecr.aws/bitnami/kafka:3.2.3 restart: unless-stopped container_name: kafka3 hostname: kafka3 ports: - "9094:9092" external_links: - zoo1 - zoo2 - zoo3 environment: KAFKA_BROKER_ID: 3 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.225:9094 ## 宿主机IP KAFKA_ADVERTISED_HOST_NAME: kafka3 KAFKA_ADVERTISED_PORT: 9094 KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181" ALLOW_PLAINTEXT_LISTENER: yes kafka-manager: image: sheepkiller/kafka-manager:latest restart: unless-stopped container_name: kafka-manager hostname: kafka-manager ports: - "9000:9000" links: - kafka1 - kafka2 - kafka3 external_links: - zoo1 - zoo2 - zoo3 environment: ZK_HOSTS: zoo1:2181,zoo2:2181,zoo3:2181 KAFKA_BROKERS: kafka1:9092,kafka2:9093,kafka3:9094
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
启动kafka
$ docker-compose up
- 1
查看kafka ui界面
kubernetes启动kafka集群
注意，这里目的仅仅是临时测试，因而回收策略设置为Delete
手动部署
单节点部署
apiVersion: v1 kind: Service metadata: labels: app: zookeeper-service name: zookeeper-service spec: ports: - name: zookeeper-port port: 2181 targetPort: 2181 selector: app: zookeeper --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: zookeeper name: zookeeper spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: #- image: wurstmeister/zookeeper - image: xxxxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/zookeeper:3.7.0 imagePullPolicy: IfNotPresent name: zookeeper ports: - containerPort: 2181 --- apiVersion: v1 kind: Service metadata: name: kafka-service labels: app: kafka spec: type: NodePort ports: - port: 9092 name: kafka-port targetPort: 9092 protocol: TCP selector: app: kafka --- kind: Deployment apiVersion: apps/v1 metadata: name: kafka-deployment spec: replicas: 1 selector: matchLabels: app: kafka template: metadata: labels: app: kafka spec: containers: - name: kafka image: public.ecr.aws/bitnami/kafka:3.2.3 imagePullPolicy: IfNotPresent ports: - containerPort: 9092 env: - name: KAFKA_ADVERTISED_PORT value: "9092" - name: KAFKA_ADVERTISED_HOST_NAME value: "kafka-host" - name: KAFKA_ZOOKEEPER_CONNECT value: "zookeeper-service:2181" - name: KAFKA_BROKER_ID value: "1" - name: ALLOW_PLAINTEXT_LISTENER value: "yes"
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
多节点部署
https://blog.51cto.com/binghe001/5245687
相对来说比较复杂，网络上较多demo
使用operator部署
https://github.com/strimzi/strimzi-kafka-operator
https://github.com/strimzi/strimzi-kafka-operator/blob/0.34.0/examples/kafka/kafka-ephemeral.yaml
一键部署operator
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
- 1
创建单节点的zk-kafka环境
kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.34.0/examples/kafka/kafka-ephemeral-single.yaml -n kafka
- 1
创建多节点临时存储kafka集群
kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.34.0/examples/kafka/kafka-ephemeral.yaml -n kafka
- 1
yaml文件如下
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: version: 3.4.0 replicas: 3 listeners: - name: plain port: 9092 type: internal tls: false - name: tls port: 9093 type: internal tls: true config: offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log.min.isr: 2 default.replication.factor: 3 min.insync.replicas: 2 inter.broker.protocol.version: "3.4" storage: type: ephemeral zookeeper: replicas: 3 storage: type: ephemeral entityOperator: topicOperator: {} userOperator: {}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
operator的方式还是方便的，删除环境只需要删除crd即可
$ k get pod -n kafka
NAME READY STATUS RESTARTS AGE
my-cluster-entity-operator-578d74f8f9-p9w5c 1/3 Running 0 16s
my-cluster-kafka-0 1/1 Running 0 60s
my-cluster-kafka-1 1/1 Running 0 60s
my-cluster-kafka-2 1/1 Running 0 60s
my-cluster-zookeeper-0 1/1 Running 0 114s
my-cluster-zookeeper-1 1/1 Running 0 114s
my-cluster-zookeeper-2 1/1 Running 0 114s
strimzi-cluster-operator-58967f75c7-8gp4w 1/1 Running 0 7m2s
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
测试集群
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic test --create --partitions 1 --replication-factor 1
$ kafka-topics --zookeeper zookeeper-service:2181 --list
$ kafka-console-producer --topic test-topic --broker-list localhost:9092
$ kafka-console-consumer --topic test-topic --from-beginning --bootstrap-server localhost:9092
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/Monodyee/article/detail/651937
推荐阅读
相关标签
