devbox
从前慢现在也慢
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

3节点ubuntu22.04服务器docker-compose方式部署高可用elk+kafka日志系统并接入nginx日志_ubuntu三台机器中安装docker

作者：从前慢现在也慢 | 2024-05-14 18:04:41

赞

踩

ubuntu三台机器中安装docker

一：系统版本:

二：部署环境：

节点名称

IP

部署组件及版本

配置文件路径

机器CPU

机器内存

机器存储

Log-001

10.10.100.1

zookeeper:3.4.13

kafka:2.8.1

elasticsearch:7.7.0

logstash:7.7.0

kibana:7.7.0

zookeeper:/data/zookeeper

kafka:/data/kafka

elasticsearch:/data/es

logstash:/data/logstash

kibana:/data/kibana

2*1c/16cores

62g

50g 系统

800g 数据盘

Log-002

10.10.100.2

zookeeper:3.4.13

kafka:2.8.1

elasticsearch:7.7.0

logstash:7.7.0

kibana:7.7.0

zookeeper:/data/zookeeper

kafka:/data/kafka

elasticsearch:/data/es

logstash:/data/logstash

kibana:/data/kibana

2*1c/16cores

62g

50g 系统

800g 数据盘

Log-003

10.10.100.3

zookeeper:3.4.13

kafka:2.8.1

elasticsearch:7.7.0

logstash:7.7.0

kibana:7.7.0

zookeeper:/data/zookeeper

kafka:/data/kafka

elasticsearch:/data/es

logstash:/data/logstash

kibana:/data/kibana

2*1c/16cores

62g

50g 系统

800g 数据盘

三：部署流程:

（1）安装docker和docker-compose

  1. apt-get install -y docker 
  2. wget https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64 mv docker-compose-Linux-x86_64 /usr/bin/docker-compose

（2）提前拉取需要用到的镜像

  1. docker pull zookeeper:3.4.13 
  2. docker pull wurstmeister/kafka 
  3. docker pull elasticsearch:7.7.0 
  4. docker pull daocloud.io/library/kibana:7.7.0 
  5. docker pull daocloud.io/library/logstash:7.7.0 
  6. docker tag wurstmeister/kafka:latest kafka:2.12-2.5.0 
  7. docker tag docker.io/zookeeper:3.4.13 docker.io/zookeeper:3.4.13 
  8. docker tag daocloud.io/library/kibana:7.7.0 kibana:7.7.0 
  9. docker tag daocloud.io/library/logstash:7.7.0 logstash:7.7.0

（3）准备应用的配置文件

  1. mkdir -p /data/zookeeper 
  2. mkdir -p /data/kafka 
  3. mkdir -p /data/logstash/conf 
  4. mkdir -p /data/es/conf 
  5. mkdir -p /data/es/data 
  6. chmod 777 /data/es/data 
  7. mkdir -p /data/kibana

（4）编辑各组件配置文件

  1. ## es配置文件
  2. ~]# cat /data/es/conf/elasticsearch.yml 
  3.  
  4. cluster.name: es-cluster
  5. network.host: 0.0.0.0
  6. node.name: master1      ## 每台节点需要更改此node.name,e.g master2,master3
  7. http.cors.enabled: true
  8. http.cors.allow-origin: "*" ## 防止跨域问题
  9. node.master: true
  10. node.data: true
  11. network.publish_host: 10.10.100.1      ## 每台节点需要更改为本机IP地址
  12. discovery.zen.minimum_master_nodes: 1
  13. discovery.zen.ping.unicast.hosts: ["10.10.100.1","10.10.100.2","10.10.100.3"]
  14. cluster.initial_master_nodes: ["10.10.100.1","10.10.100.2","10.10.100.3"]
  15.  
  16. ## elasticsearch启动过程会有报错，提前做以下操作
  17. ~]# vim /etc/sysctl.conf
  18. vm.max_map_count=655350
  19. ~]# sysctl -p
  20. ~]# cat /etc/security/limits.conf
  21. *        -        nofile        100000
  22. *        -        fsize        unlimited
  23. *        -        nproc        100000           ## unlimited nproc for *
  24.  
  25. ## logstash配置文件
  26. ~]# cat /data/logstash/conf/logstash.conf 
  27.  
  28. input{
  29.   kafka{
  30.     topics => ["system-log"]   ## 必须和前后配置的topic统一
  31.       bootstrap_servers => ["10.10.100.1:9092,10.10.100.2:9092,10.10.100.3:9092"]
  32.   }
  33. }
  34. filter{
  35.   grok{
  36.     match =>{ 
  37.       "message" => "%{SYSLOGTIMESTAMP:timestamp} %{IP:ip} %{DATA:syslog_program} %{GREEDYDATA:message}"
  38.     }
  39.     overwrite => ["message"]
  40.   }
  41.   date {
  42.     match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  43.   }
  44. }
  45.  
  46. output{
  47.   elasticsearch{
  48.     hosts => ["10.10.100.1:9200","10.10.100.2:9200","10.10.100.3:9200"]
  49.     index => "system-log-%{+YYYY.MM.dd}"
  50.   }
  51.   stdout{
  52.     codec => rubydebug
  53.   }
  54. }
  55.  
  56. ~]# cat /data/logstash/conf/logstash.yml
  57.  
  58. http.host: "0.0.0.0"
  59. xpack.monitoring.elasticsearch.hosts: [ "http://10.10.100.1:9200","http://10.10.100.2:9200","http://10.10.100.3:9200" ]
  60.  
  61. ## kibana配置文件
  62. ~]# cat /data/kibana/conf/kibana.yml 
  63. #
  64. # ** THIS IS AN AUTO-GENERATED FILE **
  65. #
  66.  
  67. # Default Kibana configuration for docker target
  68. server.name: kibana
  69. server.host: "0.0.0.0"
  70. elasticsearch.hosts: [ "http://10.10.100.1:9200","http://10.10.100.2:9200","http://10.10.100.3:9200" ]
  71. monitoring.ui.container.elasticsearch.enabled: true

（5）所有组件的部署方式全部为docker-compose形式编排部署，docker-compose.yml文件所在路径/root/elk_docker_compose/，编排内容：

  1. ~]# mkdir /data/elk
  2. ~]# cat /root/elk_docker_compose/docker-compose.yml
  3.  
  4. version: '2.1'
  5. services:
  6.   elasticsearch:
  7.     image: elasticsearch:7.7.0
  8.     container_name: elasticsearch
  9.     environment:
  10.       ES_JAVA_OPTS: -Xms1g -Xmx1g
  11.     network_mode: host
  12.     volumes:
  13.       - /data/es/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  14.       - /data/es/data:/usr/share/elasticsearch/data
  15.     logging:
  16.       driver: json-file
  17.  
  18.   kibana:
  19.     image: kibana:7.7.0
  20.     container_name: kibana
  21.     depends_on:
  22.       - elasticsearch
  23.     volumes:
  24.       - /data/kibana/conf/kibana.yml:/usr/share/kibana/config/kibana.yml
  25.     logging:
  26.       driver: json-file
  27.     ports:
  28.       - 5601:5601
  29.  
  30.   logstash:
  31.     image: logstash:7.7.0
  32.     container_name: logstash
  33.     volumes:
  34.       - /data/logstash/conf/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
  35.       - /data/logstash/conf/logstash.yml:/usr/share/logstash/config/logstash.yml
  36.     depends_on:
  37.       - elasticsearch
  38.     logging:
  39.       driver: json-file
  40.     ports:
  41.       - 4560:4560
  42.  
  43.   zookeeper:
  44.     image: zookeeper:3.4.13
  45.     container_name: zookeeper
  46.     environment:
  47.       ZOO_PORT: 2181
  48.       ZOO_DATA_DIR: /data/zookeeper/data
  49.       ZOO_DATA_LOG_DIR: /data/zookeeper/logs
  50.       ZOO_MY_ID: 1
  51.       ZOO_SERVERS: "server.1=10.10.100.1:2888:3888 server.2=10.10.100.2:2888:3888 server.3=10.10.100.3:2888:3888"
  52.     volumes:
  53.       - /data/zookeeper:/data/zookeeper
  54.     network_mode: host
  55.     logging:
  56.       driver: json-file
  57.  
  58.   kafka:
  59.     image: kafka:2.12-2.5.0
  60.     container_name: kafka
  61.     depends_on:
  62.       - zookeeper
  63.     environment:
  64.       KAFKA_BROKER_ID: 1
  65.       KAFKA_PORT: 9092
  66.       KAFKA_HEAP_OPTS: "-Xms1g -Xmx1g"
  67.       KAFKA_HOST_NAME: 10.10.100.145
  68.       KAFKA_ADVERTISED_HOST_NAME: 10.10.100.1
  69.       KAFKA_LOG_DIRS: /data/kafka
  70.       KAFKA_ZOOKEEPER_CONNECT: 10.10.100.1:2181,10.10.100.2:2181,10.10.100.3:2181
  71.     network_mode: host
  72.     volumes:
  73.       - /data:/data
  74.     logging:
  75.       driver: json-file

（6）启动服务

  1. #开始部署(三台节点分别修改配置文件和docker-compose配置)
  2. ~]# docker-compose up -d
  3. #停止运行的容器实例
  4. ~]# docker-compose stop
  5. #单独启动容器
  6. ~]# docker-compose up -d kafka

（7）验证集群各组件服务状态

  1. (1) 验证zookeeper：
  2. ]# docker exec -it zookeeper bash
  3. bash-4.4# zkServer.sh status
  4. ZooKeeper JMX enabled by default
  5. Using config: /conf/zoo.cfg
  6. Mode: follower
  7.  
  8. (2) 验证kafka：
  9. ]# docker exec -it kafka bash
  10. bash-4.4# kafka-topics.sh --list --zookeeper 10.10.100.1:2181
  11. __consumer_offsets
  12. system-log
  13.  
  14. (3) 验证elasticsearch
  15. ]# curl '10.10.100.1:9200/_cat/nodes?v'ip            
  16. heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
  17. 10.10.100.1           57          81   0    0.37    0.15     0.09 dilmrt    *      master2
  18. 10.10.100.2           34          83   0    0.11    0.10     0.06 dilmrt    -      master1
  19. 10.10.100.3           24          81   0    0.03    0.06     0.06 dilmrt    -      master3
  20.  
  21.  
  22. (4) 验证kibana
  23. 浏览器打开http://10.10.100.1:5601

三：日志收集

（1）以nginx日志为例，安装filebeat日志采集器

apt-get install filebeat

（2）配置filebeat向kafka写数据

  1. 启用 Nginx 模块:
  2. sudo filebeat modules enable nginx
  3. 配置 Nginx 模块: 编辑 /etc/filebeat/modules.d/nginx.yml，确保日志路径正确。例如：
  4. - module: nginxaccess:enabled: truevar.paths: ["/var/log/nginx/access.log*"]
  5.   error:enabled: truevar.paths: ["/var/log/nginx/error.log*"]
  6. 设置输出为 Kafka: 在 filebeat.yml 文件中，配置输出到 Kafka：
  7. output.kafka:
  8.   # Kafka 服务地址
  9.   hosts: ["10.10.100.1:9092", "10.10.100.2:9092", "10.10.100.3:9092"]
  10.   topic: "system-log"
  11.   partition.round_robin:
  12.     reachable_only: false
  13.   required_acks: 1
  14.   compression: gzip
  15.   max_message_bytes: 1000000
  16. 重启 Filebeat:
  17. sudo systemctl restart filebeat

（3）配置验证：使用 Filebeat 的配置测试命令来验证配置文件的正确性：

filebeat test config

（4）连接测试：可以测试 Filebeat 到 Kafka 的连接：

filebeat test output

（5）登录kibana控制台查看nginx日志是否已正常收集到

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/从前慢现在也慢/article/detail/569441
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号