当前位置:   article > 正文

ELK+Filebeat+Kafka+Zookeeper

ELK+Filebeat+Kafka+Zookeeper

本实验基于ELFK已经搭好的情况下 ELK日志分析

架构解析

  1. 第一层、数据采集层
  2. 数据采集层位于最左边的业务服务器集群上,在每个业务服务器上面安装了filebeat做日志收集,然后把采集到的原始日志发送到Kafka+zookeeper集群上。
  3. 第二层、消息队列层
  4. 原始日志发送到Kafka+zookeeper集群上后,会进行集中存储,此时,filbeat是消息的生产者,存储的消息可以随时被消费。
  5. 第三层、数据分析层
  6. Logstash作为消费者,会去Kafka+zookeeper集群节点实时拉取原始日志,然后将获取到的原始日志根据规则进行分析、清洗、过滤,最后将清洗好的日志转发至Elasticsearch集群。
  7. 第四层、数据持久化存储
  8. Elasticsearch集群在接收到logstash发送过来的数据后,执行写磁盘,建索引库等操作,最后将结构化的数据存储到Elasticsearch集群上。
  9. 第五层、数据查询、展示层
  10. Kibana是一个可视化的数据展示平台,当有数据检索请求时,它从Elasticsearch集群上读取数据,然后进行可视化出图和多维度分析。

搭建ELK+Filebeat+Kafka+Zookeeper

  1. zIP: 所属集群: 端口:
  2. 192.168.83.11 Elasticsearch+Kibana+kafka+zookeeper+nginx反向代理 9100 9200 5601 9092 3288 8080 都可以安装filebeat
  3. 192.168.83.12 Elasticsearch+Logstash+kafka+zookeeper+filebeat+nginx反向代理 9100 9200 9600 9092 3288 随机 8080
  4. 192.168.83.13 Elasticsearch+kafka+zookeeper+nginx反向代理 z 9100 9200 9092 3288

  1. root@elk2 ~]# netstat -antp |grep filebeat
  2. tcp 1 0 192.168.83.12:40348 192.168.83.11:9092 CLOSE_WAIT 6975/filebeat
  3. tcp 0 0 192.168.83.12:51220 192.168.83.12:9092 ESTABLISHED 6975/filebeat

1.3台机子安装zookeeper

 wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz --no-check-certificate

1.1 解压安装zookeeper软件包

  1. cd /opt
  2. 上传apache-zookeeper-3.8.0-bin.tar.gz包
  3. tar zxf apache-zookeeper-3.8.0-bin.tar.gz 解包
  4. mv apache-zookeeper-3.8.0-bin /usr/local/zookeeper-3.8.0 #将解压的目录剪切到/usr/local/
  5. cd /usr/local/zookeeper-3.8.0/conf/
  6. cp zoo_sample.cfg zoo.cfg 备份复制模板配置文件为zoo.cfg

1.2 修改Zookeeper配置配置文件

  1. cd /usr/local/zookeeper-3.8.0/conf #进入zookeeper配置文件汇总
  2. ls 后可以看到zoo_sample.cfg模板配置文件
  3. cp zoo_sample.cfg zoo.cfg 复制模板配置文件为zoo.cfg
  4. mkdir -p /usr/local/zookeeper-3.8.0/data
  5. mkdir -p dataLogDir=/usr/local/zookeeper-3.8.0/1ogs
  6. vim zoo.cfg
  7. # The number of milliseconds of each tick
  8. tickTime=2000
  9. # The number of ticks that the initial
  10. # synchronization phase can take
  11. initLimit=10
  12. # The number of ticks that can pass between
  13. # sending a request and getting an acknowledgement
  14. syncLimit=5
  15. # the directory where the snapshot is stored.
  16. # do not use /tmp for storage, /tmp here is just
  17. # example sakes.
  18. dataDir=/usr/local/zookeeper-3.8.0/data
  19. dataLogDir=/usr/local/zookeeper-3.8.0/1ogs
  20. # the port at which the clients will connect
  21. clientPort=2181
  22. # the maximum number of client connections.
  23. # increase this if you need to handle more clients
  24. #maxClientCnxns=60
  25. #
  26. # Be sure to read the maintenance section of the
  27. # administrator guide before turning on autopurge.
  28. #
  29. # https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
  30. #
  31. # The number of snapshots to retain in dataDir
  32. #autopurge.snapRetainCount=3
  33. # Purge task interval in hours
  34. # Set to "0" to disable auto purge feature
  35. #autopurge.purgeInterval=1
  36. ## Metrics Providers
  37. #
  38. # https://prometheus.io Metrics Exporter
  39. #metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
  40. #metricsProvider.httpHost=0.0.0.0
  41. #metricsProvider.httpPort=7000
  42. #metricsProvider.exportJvmInfo=true
  43. server.1=192.168.83.11:3188:3288
  44. server.2=192.168.83.12:3188:3288
  45. server.3=192.168.83.13:3188:3288
  1. scp zoo.cfg elk2:/usr/local/zookeeper-3.8.0/conf/zoo.cfg
  2. scp zoo.cfg elk3:/usr/local/zookeeper-3.8.0/conf/zoo.cfg

1.3 设置myid号以及启动脚本 到这里就不要设置同步了,下面的操作,做好一台机器一台机器的配置。

  1. echo 1 >/usr/local/zookeeper-3.8.0/data/myid
  2. # node1上配置
  3. echo 2 >/usr/local/zookeeper-3.8.0/data/myid
  4. #node2上配置
  5. echo 3 >/usr/local/zookeeper-3.8.0/data/myid
  6. #node3上配置

1.4 两种启动zookeeper的方法

  1. cd /usr/local/zookeeper-3.8.0/bin
  2. ls
  3. ./zkServer.sh start #启动 一次性启动三台,,才可以看状态
  4. ./zkServer.sh status #查看状态
  1. [root@elk1 bin]# ./zkServer.sh status
  2. /usr/bin/java
  3. ZooKeeper JMX enabled by default
  4. Using config: /usr/local/zookeeper-3.8.0/bin/../conf/zoo.cfg
  5. Client port found: 2181. Client address: localhost. Client SSL: false.
  6. Mode: follower
  7. [root@elk2 bin]# ./zkServer.sh status
  8. /usr/bin/java
  9. ZooKeeper JMX enabled by default
  10. Using config: /usr/local/zookeeper-3.8.0/bin/../conf/zoo.cfg
  11. Client port found: 2181. Client address: localhost. Client SSL: false.
  12. Mode: leader
  13. [root@elk3 bin]# ./zkServer.sh status
  14. /usr/bin/java
  15. ZooKeeper JMX enabled by default
  16. Using config: /usr/local/zookeeper-3.8.0/bin/../conf/zoo.cfg
  17. Client port found: 2181. Client address: localhost. Client SSL: false.
  18. Mode: follower

1.5脚本启动 推荐

  1. 2种启动
  2. 3台节点需要执行的脚本
  3. #//配置启动脚本,脚本在开启启动执行的目录中创建
  4. vim /etc/init.d/zookeeper
  5. #!/bin/bash
  6. #chkconfig:2345 20 90
  7. #description:Zookeeper Service Control Script
  8. ZK_HOME='/usr/local/zookeeper-3.8.0'
  9. case $1 in
  10. start)
  11. echo "----------zookeeper启动----------"
  12. $ZK_HOME/bin/zkServer.sh start
  13. ;;
  14. stop)
  15. echo "---------- zookeeper停止-----------"
  16. $ZK_HOME/bin/zkServer.sh stop
  17. ;;
  18. restart)
  19. echo "---------- zookeeper 重启------------"
  20. $ZK_HOME/bin/zkServer.sh restart
  21. ;;
  22. status)
  23. echo "---------- zookeeper 状态------------"
  24. $ZK_HOME/bin/zkServer.sh status
  25. ;;
  26. *)
  27. echo "Usage: $0 {start|stop|restart|status}"
  28. esac
  29. cd /usr/local/zookeeper-3.8.0/bin
  30. 在节点1服务操作
  31. chmod +x /etc/init.d/zookeeper
  32. chkconfig --add zookeeper #加入到系统管理
  33. service zookeeper start 启动服务
  34. service zookeeper status 查看状态后 是 follower
  35. 在节点2服务操作
  36. chmod +x /etc/init.d/zookeeper
  37. chkconfig --add zookeeper #加入到系统管理
  38. service zookeeper start 启动服务
  39. service zookeeper status 查看状态后 是 leader 第二台启动的,他是leader
  40. 在节点3服务操作
  41. chmod +x /etc/init.d/zookeeper
  42. chkconfig --add zookeeper #加入到系统管理
  43. service zookeeper start 启动服务
  44. service zookeeper status 查看状态后 是 follower

2. 安装 kafka(3台机子都要操作)

  1. #下载kafka
  2. cd /opt
  3. wget http://archive.apache.org/dist/kafka/2.7.1/kafka_2.13-2.7.1.tgz
  4. 上传kafka_2.13-2.7.1.tgz到/opt
  5. tar zxf kafka_2.13-2.7.1.tgz
  6. mv kafka_2.13-2.7.1 /usr/local/kafka

2.2 修改配置文件

  1. cd /usr/local/kafka/config/
  2. cp server.properties server.properties.bak
  3. vim server.properties
  4. 192.168.83.11配置
  5. broker.id=1
  6. listeners=PLAINTEXT://192.168.83.11:9092
  7. zookeeper.connect=192.168.83.11:2181,192.168.83.12:2181,192.168.83.13:2181
  8. 192.168.83.13配置
  9. broker.id=2
  10. listeners=PLAINTEXT://192.168.83.12:9092
  11. zookeeper.connect=192.168.83.11:2181,192.168.83.12:2181,192.168.83.13:21810:2181
  12. 192.168.83.13配置
  13. broker.id=3
  14. listeners=PLAINTEXT://192.168.83.13:9092
  15. zookeeper.connect=192.168.83.11:2181,192.168.83.12:2181,192.168.83.13:2181

2.3 将相关命令加入到系统环境当中

  1. vim /etc/profile 末行加入
  2. export KAFKA_HOME=/usr/local/kafka
  3. export PATH=$PATH:$KAFKA_HOME/bin
  4. source /etc/profile
  5. [root@elk1 config]# scp /etc/profile elk2:/etc/profile
  6. profile 100% 1888 1.4MB/s 00:00
  7. [root@elk1 config]# scp /etc/profile elk3:/etc/profile
  8. profile

2.3 将相关命令加入到系统环境当中

  1. cd /usr/local/kafka/config/
  2. kafka-server-start.sh -daemon server.properties
  3. netstat -antp | grep 9092

2.4Kafka 命令行操作

  1. 创建topic
  2. kafka-topics.sh --create --zookeeper 192.168.121.10:2181,192.168.121.12:2181,192.168.121.14:2181 --replication-factor 2 --partitions 3 --topic test
  3. –zookeeper:定义 zookeeper 集群服务器地址,如果有多个 IP 地址使用逗号分割,一般使用一个 IP 即可
  4. –replication-factor:定义分区副本数,1 代表单副本,建议为 2
  5. –partitions:定义分区数
  6. –topic:定义 topic 名称
  7. 查看当前服务器中的所有 topic
  8. kafka-topics.sh --list --zookeeper 192.168.121.10:2181,192.168.121.12:2181,192.168.121.14:2181
  9. 查看某个 topic 的详情
  10. kafka-topics.sh --describe --zookeeper 192.168.121.10:2181,192.168.121.12:2181,192.168.121.14:2181
  11. 发布消息
  12. kafka-console-producer.sh --broker-list 192.168.121.10:9092,192.168.121.12:9092,192.168.121.14:9092 --topic test
  13. 消费消息
  14. kafka-console-consumer.sh --bootstrap-server 192.168.121.10:9092,192.168.121.12:9092,192.168.121.14:9092 --topic test --from-beginning
  15. –from-beginning:会把主题中以往所有的数据都读取出来
  16. 修改分区数
  17. kafka-topics.sh
  18. --zookeeper 192.168.80.10:2181,192.168.80.11:2181,192.168.80.12:2181 --alter --topic test --partitions 6
  19. 删除 topic
  20. kafka-topics.sh
  21. --delete --zookeeper 192.168.80.10:2181,192.168.80.11:2181,192.168.80.12:2181 --topic test

3.配置数据采集层filebeat

3.1 定制日志格式

  1. 3.1 定制日志格式
  2. [root@elk2 ~]# vim /etc/nginx/nginx.conf
  3. user nginx;
  4. worker_processes auto;
  5. error_log /var/log/nginx/error.log notice;
  6. pid /var/run/nginx.pid;
  7. events
  8. {
  9. worker_connections 1024;
  10. }
  11. http
  12. {
  13. include /etc/nginx/mime.types;
  14. default_type application/octet-stream;
  15. # log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '
  16. # '$status $body_bytes_sent "$http_referer" '
  17. # '"$http_user_agent" "$upstream_addr" $request_time';
  18. # access_log /var/log/nginx/access.log main2;
  19. log_format json '{"@timestamp":"$time_iso8601",'
  20. '"@version":"1",'
  21. '"client":"$remote_addr",'
  22. '"url":"$uri",'
  23. '"status":"$status",'
  24. '"domain":"$host",'
  25. '"host":"$server_addr",'
  26. '"size":$body_bytes_sent,'
  27. '"responsetime":$request_time,'
  28. '"referer": "$http_referer",'
  29. '"ua": "$http_user_agent"'
  30. '}';
  31. access_log /var/log/nginx/access.log json;
  32. sendfile on;
  33. #tcp_nopush on;
  34. keepalive_timeout 65;
  35. #gzip on;
  36. upstream elasticsearch
  37. {
  38. zone elasticsearch 64K;
  39. server 192.168.83.11:9200;
  40. server 192.168.83.12:9200;
  41. server 192.168.83.13:9200;
  42. }
  43. server
  44. {
  45. listen 8080;
  46. server_name localhost;
  47. location /
  48. {
  49. proxy_pass http://elasticsearch;
  50. root html;
  51. index index.html index.htm;
  52. }
  53. }
  54. include /etc/nginx/conf.d/*.conf;
  55. }

3.2安装filebeat

  1. [root@elk2 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
  2. [root@elk2 ~]# rpm -ivh filebeat-6.0.0-x86_64.rpm

3.3 修改配置文件filebeat.yml

  1. [root@elk2 ~]# vim /etc/filebeat/filebeat.yml
  2. enabled: true
  3. paths:
  4. - /var/log/nginx/*.log
  5. #-------------------------- Elasticsearch output ------------------------------
  6. output.kafka:
  7. # Array of hosts to connect to.
  8. hosts: ["192.168.83.11:9092","192.168.83.12:9092","192.168.83.13:9092"] #145
  9. topic: "nginx-es"

3.4 启动filebeat

[root@elk2 ~]# systemctl restart filebeat 

4、所有组件部署完成之后,开始配置部署

4.1 在kafka上创建一个话题nginx-es

kafka-topics.sh --create --zookeeper 192.168.83.11:2181,192.168.83.12:2181,192.168.83.13:2181 --replication-factor 1 --partitions 1 --topic nginx-es 

4.2 修改logstash的配置文件

  1. [root@elk2 ~]# vim /etc/logstash/conf.d/nginxlog.conf
  2. input{
  3. kafka{
  4. topics=>"nginx-es"
  5. codec=>"json"
  6. decorate_events=>true
  7. bootstrap_servers=>"192.168.83.11:9092,192.168.83.12:9092,192.168.83.13:9092"
  8. }
  9. }
  10. output {
  11. elasticsearch {
  12. hosts=>["192.168.83.11:9200","192.168.83.12:9200","192.168.83.13:9200"]
  13. index=>'nginx-log-%{+YYYY-MM-dd}'
  14. }
  15. }
  16. 重启logstash
  17. systemctl restart logstash

4.3 验证网页

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/Guff_9hys/article/detail/809430
推荐阅读
相关标签
  

闽ICP备14008679号