devbox
菜鸟追梦旅行
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

centos7 yum安装EFK收集日志

作者：菜鸟追梦旅行 | 2024-05-14 18:05:12

赞

踩

centos7 yum安装EFK收集日志

配置清华源：

Elasticsearch安装

yum安装方式Elasticsearch

1.导入Elasticsearch GPG KEY
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch


2.添加elasticsearch的yum repo文件，使用清华的yum源镜像
默认安装最新版本，如需要指定版本，请下载指定rpm包，并配置java环境变量，放到/usr/bin下

配置好yum源，执行yum localinstall elasticsearch-6.7.0.rpm  -y

  1. cd /etc/yum.repos.d
  2. vi elasticsearch.repo
  3.  
  4. [elasticsearch-6.x]
  5. name=Elasticsearch repository for 6.x packages
  6. baseurl=https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/
  7. gpgcheck=1
  8. gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
  9. enabled=1
  10. autorefresh=1
  11. type=rpm-md

3.修该配置文件

  1. vim /etc/elasticsearch/elasticsearch.yml
  2. 修改55行
  3. 55 network.host: 0.0.0.0

 

  1. vim  /etc/elasticsearch/jvm.options
  2. 修改jvm参数
  3. 22 -Xms128m
  4. 23 -Xmx128m

4.一个进程在VMAs(虚拟内存区域)创建内存映射最大数量  

  1. vim /etc/sysctl.conf
  2. vm.max_map_count=655360

5. #配置生效

sysctl -p

6.启动es

  1. systemctl start elasticsearch
  2. tailf -200 /var/log/elasticsearch/elasticsearch.log
7.通过访问进行测试，看到如下信息，就说明ES启动成功了 
  1. { 
  2.     "name": "dSQV6I8", 
  3.     "cluster_name": "elasticsearch", 
  4.     "cluster_uuid": "v5GPTWAtT5emxFdjigFg-w", 
  5.     "version": { 
  6.             "number": "6.5.4", 
  7.             "build_flavor": "default", 
  8.             "build_type": "tar", 
  9.             "build_hash": "d2ef93d",
  10.             "build_date": "2018-12-17T21:17:40.758843Z", 
  11.             "build_snapshot": false, "lucene_version": "7.5.0", 
  12.             "minimum_wire_compatibility_version": "5.6.0", 
  13.             "minimum_index_compatibility_version": "5.0.0" 
  14.      },
  15.      "tagline": "You Know, for Search" 
  16. }

安装elasticsearch-head

由于ES官方并没有为ES提供界面管理工具，仅仅是提供了后台的服务。elasticsearch-head是一个为ES开发的一个页
面客户端工具，其源码托管于GitHub，地址为：https://github.com/mobz/elasticsearch-head
 
head提供了4种安装方式：
 
源码安装，通过npm run start启动（不推荐）
通过docker安装（推荐）
通过chrome插件安装（推荐）
通过ES的plugin方式安装（不推荐）
 

通过docker安装

  1. #拉取镜像 docker pull mobz/elasticsearch-head:5 
  2. #创建容器 docker create --name elasticsearch-head -p 9100:9100 mobz/elasticsearch-head:5 
  3. #启动容器 docker start elasticsearch-head

通过浏览器进行访问：

注意：
由于前后端分离开发，所以会存在跨域问题，需要在服务端做CORS的配置，如下：
  
  1. vim elasticsearch.yml
  2. http.cors.enabled: true
  3. http.cors.allow-origin: "*"

解决head插件提交请求406问题方法：

  1. docker exec -it elasticsearch-head /bin/bash
  2. sed -i 's#application/x-www-form-urlencoded#application/json;charset=UTF-8#g' ./_site/vendor.js
  3. docker restart elasticsearch-head

配置了x-pack安全认证之后，就显示连接失败了。。。

通过chrome插件的方式安装不存在该问题。

打开chrome的应用商店，即可安装https://chrome.google.com/webstore/detail/elasticsearch-head/ffmkiejjmecolpfloofpjologoblkegm

配置安装kibana

还是基于清华源进行指定版本的yum本地安装

配置好yum源 
yum localinstall kibana-6.7.0-x86_64.rpm -y

编辑配置文件

  1. vim /etc/kibana/kibana.yml
  2. server.port: 5601
  3. server.host: "10.4.7.101"
  4. elasticsearch.hosts: ["http://10.4.7.101:9200"]

启动kibana

  1. systemctl start kibana
  2. systemctl enable kibana

访问kinaba

http://10.4.7.101:5601/

x-pack破解为铂金版

参考文章链接

https://blog.csdn.net/weixin_45396564/article/details/103420345

yum安装方式破解x-pack填坑大集合

1、上传许可证书的时候上传到/etc/elasticsearch/下（权限660），并修改属主属组

2、启动x-pack安全插件报错 
curl -H "Content-Type:application/json" -XPOST http://10.4.7.101:9200/_xpack/license/start_trial?acknowledge=true

报错： 

（命令粘贴的有问题，空格该删的，删一删，重新敲空格）
正确返回结果：
 
 
3、为es创建用户名密码 
bin/elasticsearch-setup-passwords interactive

报错：

需要在elasticsearch的配置文件配置： 
xpack.security.enabled: true

重启es

4、导入ssl证书

证书文件一定放在/etc/elasticsearch/并设置660权限，修改属主属组，否则会有文件读取权限问题

5、导入证书

curl -XPUT -u elastic 'http://10.4.7.101:9200/_license?acknowledge=true' -H "Content-Type: application/json" -d @/etc/elasticsearch/kc-lisence.json

报错：

证书路径不对，或者证书不存在，提示证书内容为空
 
报错：
 
 
不能都是双引号包起来
 
正确返回结果：
 
 
 
6、/etc/elasticsearch/elasticsearch.yml 
network.host: 10.4.7.101,127.0.0.1

生成ssl证书的时候写的ip地址如果为主机ip的话，配置文件要按如上编写

报错：

展示下破解完成的成果
 
一定要记录刚刚设置的那就个用户的密码，不同的权限登录kibana使用
 
 

日志收集配置

1.filebeat收集Nginx的json格式日志

  1. 1.普通Nginx日志不足的地方：
  2. - 日志都在一个value里，不能拆分单独显示和搜索
  3. - 索引名称没有意义
  4.  
  5. 2.理想中的情况
  6. {
  7.     $remote_addr : 192.168.12.254
  8.     - : -
  9.     $remote_user : -
  10.     [$time_local]: [10/Sep/2019:10:52:08 +0800]
  11.     $request: GET /jhdgsjfgjhshj HTTP/1.0
  12.     $status : 404
  13.     $body_bytes_sent : 153
  14.     $http_referer : -
  15.     $http_user_agent :ApacheBench/2.3
  16.     $http_x_forwarded_for:-
  17. }
  18.  
  19. 3.目标
  20. 将Nginx日志转换成json格式
  21.  
  22. 4.修改nginx配置文件使日志转换成json
  23. log_format json '{ "time_local": "$time_local", '
  24.                           '"remote_addr": "$remote_addr", '
  25.                           '"referer": "$http_referer", '
  26.                           '"request": "$request", '
  27.                           '"status": $status, '
  28.                           '"bytes": $body_bytes_sent, '
  29.                           '"agent": "$http_user_agent", '
  30.                           '"x_forwarded": "$http_x_forwarded_for", '
  31.                           '"up_addr": "$upstream_addr",'
  32.                           '"up_host": "$upstream_http_host",'
  33.                           '"upstream_time": "$upstream_response_time",'
  34.                           '"request_time": "$request_time"'
  35.     ' }';
  36.     access_log  /var/log/nginx/access.log  json;
  37.  
  38. #清空旧日志
  39. [root@db01 ~]# > /var/log/nginx/access.log
  40.  
  41. #检查并重启nginx
  42. [root@db01 ~]# nginx -t
  43. [root@db01 ~]# systemctl restart nginx 
  44.  
  45. 5.修改filebeat配置文件
  46. cat >/etc/filebeat/filebeat.yml<<EOF
  47. filebeat.inputs:
  48. - type: log
  49.   enabled: true
  50.   paths:
  51.     - /var/log/nginx/access.log
  52.   json.keys_under_root: true
  53.   json.overwrite_keys: true
  54. output.elasticsearch:
  55.   hosts: ["10.0.0.51:9200"]
  56. EOF
  57.  
  58. 6.删除旧的ES索引
  59. es-head >> filebeat-6.6.0-2019.11.15 >> 动作 >>删除 
  61. 7.删除kibana里面的日日志信息
  62.  
  63. 8.重启filebeat
  64. [root@db01 ~]# systemctl restart filebeat
  65.  
  66. 9.curl 一下nginx,并在es-head插件查看
  67. [root@db01 ~]# curl 127.0.0.1
  68. db01-www

2.filebeat自定义ES索引名称

  1. 1.理想中的索引名称
  2. filebeat-6.6.0-2020.02.13
  3. nginx-6.6.0-2019.11.15
  4.  
  5. 2.filebeat配置
  6. [root@db01 ~]# cat >/etc/filebeat/filebeat.yml<<EOF
  7. filebeat.inputs:
  8. - type: log
  9.   enabled: true
  10.   paths:
  11.     - /var/log/nginx/access.log
  12.   json.keys_under_root: true
  13.   json.overwrite_keys: true
  14.  
  15. output.elasticsearch:
  16.   hosts: ["10.0.0.51:9200"]
  17.   index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
  18.  
  19. setup.template.name: "nginx"
  20. setup.template.pattern: "nginx-*"
  21. setup.template.enabled: false
  22. setup.template.overwrite: true
  23. EOF
  24.  
  25. 3.重启filebeat
  26. [root@db01 ~]# systemctl restart filebeat
  27.  
  28. 4.生成新日志并检查
  29. [root@db01 ~]# curl 127.0.0.1
  30.  
  31. 5.es-head插件查看并在中kibana添加

 

3.filebeat按照服务类型拆分索引

  1. 1.理想中的情况：
  2. nginx-access-6.6.0-2020.02
  3. nginx-error-6.6.0-2020.02
  4.  
  5. 2.filebeat配置
  6. #第一种方法：
  7. [root@db01 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
  8. filebeat.inputs:
  9. - type: log
  10.   enabled: true
  11.   paths:
  12.     - /var/log/nginx/access.log
  13.   json.keys_under_root: true
  14.   json.overwrite_keys: true
  15.  
  16. - type: log
  17.   enabled: true
  18.   paths:
  19.     - /var/log/nginx/error.log
  20.  
  21. output.elasticsearch:
  22.   hosts: ["10.0.0.51:9200"]
  23.   indices:
  24.     - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
  25.       when.contains:
  26.         source: "/var/log/nginx/access.log"
  27.     - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
  28.       when.contains:
  29.         source: "/var/log/nginx/error.log"
  30.     
  31. setup.template.name: "nginx"
  32. setup.template.pattern: "nginx-*"
  33. setup.template.enabled: false
  34. setup.template.overwrite: true
  35. EOF
  36.  
  37. #第二种方法：
  38. [root@db01 ~]# cat >/etc/filebeat/filebeat.yml <<EOF   
  39. filebeat.inputs:
  40. - type: log
  41.   enabled: true
  42.   paths:
  43.     - /var/log/nginx/access.log
  44.   json.keys_under_root: true
  45.   json.overwrite_keys: true
  46.   tags: ["access"]
  47.  
  48. - type: log
  49.   enabled: true
  50.   paths:
  51.     - /var/log/nginx/error.log
  52.   tags: ["error"]
  53.  
  54. output.elasticsearch:
  55.   hosts: ["10.0.0.51:9200"]
  56.   indices:
  57.     - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
  58.       when.contains:
  59.         tags: "access"
  60.     - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
  61.       when.contains:
  62.         tags: "error"
  63.     
  64. setup.template.name: "nginx"
  65. setup.template.pattern: "nginx-*"
  66. setup.template.enabled: false
  67. setup.template.overwrite: true
  68. EOF
  69.  
  70. 3.重启filebeat
  71. [root@db01 ~]# systemctl restart filebeat 
  72.  
  73. 4.生成正确和错误的测试数据
  74. [root@db01 ~]# curl 127.0.0.1/haahha
  75. [root@db01 ~]# curl 127.0.0.1
  76.  
  77. 5.检查是否生成对应的索引
  78. nginx-access-6.6.0-2020.02
  79. nginx-error-6.6.0-2020.02

 

4.收集多台服务器nginx日志

  1. 1.在别的服务器上面安装nginx
  2. #更换官方源
  3. [root@db02 ~]# cat /etc/yum.repos.d/nginx.repo
  4. [nginx-stable]
  5. name=nginx stable repo
  6. baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
  7. gpgcheck=1
  8. enabled=1
  9. gpgkey=https://nginx.org/keys/nginx_signing.key
  10. module_hotfixes=true
  11.  
  12. [nginx-mainline]
  13. name=nginx mainline repo
  14. baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
  15. gpgcheck=1
  16. enabled=0
  17. gpgkey=https://nginx.org/keys/nginx_signing.key
  18. module_hotfixes=true
  19.  
  20. #安装nginx
  21. [root@db02 ~]# yum install nginx -y
  22.  
  23. 2.复制db01的nginx的配置文件
  24. [root@db02 ~]# scp 10.0.0.51:/etc/nginx/nginx.conf /etc/nginx/nginx.conf
  25. [root@db02 ~]# scp 10.0.0.51:/etc/nginx/conf.d/www.conf /etc/nginx/conf.d/
  26.  
  27. 3.创建测试页面
  28. [root@db02 ~]# mkdir /code/www/ -p
  29. [root@db02 ~]# echo "db02-www" > /code/www/index.html
  30.  
  31. 4.重启nginx
  32. [root@db02 ~]# >/var/log/nginx/access.log
  33. [root@db02 ~]# >/var/log/nginx/error.log
  34. [root@db02 ~]# nginx -t 
  35. [root@db02 ~]# systemctl restart nginx
  36.  
  37. 5.安装filebeat
  38. [root@db02 ~]# rpm -ivh filebeat-6.6.0-x86_64.rpm
  39.  
  40. 6.复制filebeat配置文件
  41. [root@db02 ~]# scp 10.0.0.51:/etc/filebeat/filebeat.yml /etc/filebeat/
  42.  
  43. 7.启动filebeat
  44. [root@db02 ~]# systemctl restart filebeat
  45.  
  46. 8.生成测试数据
  47. [root@db02 ~]# curl 127.0.0.1/22222222222222
  48. [root@db02 ~]# curl 127.0.0.1
  49.  
  1. #收集nginx完整的filebeat配置
  2. [root@db01]# cat /etc/filebeat/filebeat.yml 
  3. filebeat.inputs:
  4. - type: log
  5.   enabled: true
  6.   paths:
  7.     - /var/log/nginx/access.log
  8.   json.keys_under_root: true
  9.   json.overwrite_keys: true
  10.  
  11. - type: log
  12.   enabled: true
  13.   paths:
  14.     - /var/log/nginx/error.log
  15.  
  16. output.elasticsearch:
  17.   hosts: ["10.0.0.51:9200"]
  18.   indices:
  19.     - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
  20.       when.contains:
  21.         source: "/var/log/nginx/access.log"
  22.     - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
  23.       when.contains:
  24.         source: "/var/log/nginx/error.log"
  25.     
  26. setup.template.name: "nginx"
  27. setup.template.pattern: "nginx-*"
  28. setup.template.enabled: false
  29. setup.template.overwrite: true

5.filebeat收集tomcat的json日志

1.安装tomcat、filebeat

tomcat略

在tomcat服务器安装filebeat

filebeat

yum localinstall filebeat-6.7.0-x86_64.rpm -y

2.配置tomcat日志格式为json

  1. [root@web01 ~]# /data/apache-tomcat-8.0.33/bin/shutdown.sh
  2. [root@web01 ~]# vim /data/apache-tomcat-8.0.33/conf/server.xml
  3. 修改参数日志格式 
  4. 	       pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>

3.启动tomcat

/data/apache-tomcat-8.0.33/bin/startup.sh

4.配置filebeat

  1. cat >/etc/filebeat/filebeat.yml <<EOF
  2. filebeat.inputs:
  3. - type: log
  4.   enabled: true
  5.   paths:
  6.     - /data/apache-tomcat-8.0.33/logs/localhost_access_log.*.txt
  7.   json.keys_under_root: true
  8.   json.overwrite_keys: true
  9.   tags: ["tomcat"]
  10.  
  11. output.elasticsearch:
  12.   hosts: ["10.4.7.101:9200"]
  13.   index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"
  14.   username: "elastic"
  15.   password: "123456"
  16. xpack.monitoring.enabled: true
  17. xpack.monitoring.elasticsearch:
  18. setup.template.name: "tomcat"
  19. setup.template.pattern: "tomcat_*"
  20. setup.template.enabled: false
  21. setup.template.overwrite: true
  22. EOF

5.重启filebeat 

systemctl restart filebeat

6.访问tomcat查看是否有数据和x-pack监控生成

6.filebeat收集java多行匹配模式

1.filebeat配置文件

  1. [root@db01 ~]# cat /etc/filebeat/filebeat.yml 
  2. filebeat.inputs:
  3. - type: log
  4.   enabled: true
  5.   paths:
  6.     - /var/log/elasticsearch/elasticsearch.log
  7.   multiline.pattern: '^\['
  8.   multiline.negate: true
  9.   multiline.match: after
  10.  
  11. output.elasticsearch:
  12.   hosts: ["10.0.0.51:9200"]
  13.   index: "es-%{[beat.version]}-%{+yyyy.MM}"
  14.   username: "elastic"
  15.   password: "123456"
  16. xpack.monitoring.enabled: true
  17. xpack.monitoring.elasticsearch:
  18. setup.template.name: "tomcat"
  19. setup.template.pattern: "tomcat_*"
  20. setup.template.enabled: false
  21. setup.template.overwrite: true

配置完成重启filebeat 

2.filebeat收集tomcat的catalina.out和access访问日志

  1. filebeat.inputs:
  2. - type: log
  3.   enabled: true
  4.   paths:
  5.     - /data/apache-tomcat-8.0.33/logs/localhost_access_log.*.txt
  6.   json.keys_under_root: true
  7.   json.overwrite_keys: true
  8.   tags: ["access"]
  9. - type: log
  10.   enabled: true
  11.   paths:
  12.     - /data/apache-tomcat-8.0.33/logs/catalina.out
  13.   tags: ["catalina"]
  14. output.elasticsearch:
  15.   hosts: ["10.4.7.101:9200"]
  16.   indices:
  17.     - index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"
  18.       when.contains:
  19.         tags: "access"
  20.     - index: "tomcat_catalina-%{[beat.version]}-%{+yyyy.MM}"
  21.       when.contains:
  22.         tags: "catalina"
  23.   username: "elastic"
  24.   password: "123456"
  25. xpack.monitoring.enabled: true
  26. xpack.monitoring.elasticsearch:
  27. setup.template.name: "tomcat"
  28. setup.template.pattern: "tomcat_*"
  29. setup.template.enabled: false
  30. setup.template.overwrite: true

配置完成重启filebeat

7.filbeat使用模块收集nginx日志

  1. 1.清空并把nginx日志恢复成普通格式
  2. #清空日志
  3. [root@db01 ~]# > /var/log/nginx/access.log
  4.  
  5. #编辑配置文件
  6. [root@db01 ~]# vim /etc/nginx/nginx.conf
  7.     log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
  8.                       '$status $body_bytes_sent "$http_referer" '
  9.                       '"$http_user_agent" "$http_x_forwarded_for"';
  10.     access_log  /var/log/nginx/access.log  main;
  11.  
  12. #检查并重启
  13. [root@db01 ~]# nginx -t
  14. [root@db01 ~]# systemctl restart nginx
  15.  
  16. 2.访问并检查日志是否为普通格式
  17. [root@db01 ~]# curl 127.0.0.1
  18. [root@db01 ~]# tail -f /var/log/nginx/access.log
  19.  
  20. 3.配置filebeat配置文件支持模块
  21. [root@db01 ~]# cat /etc/filebeat/filebeat.yml
  22. filebeat.config.modules:
  23.   path: ${path.config}/modules.d/*.yml
  24.   reload.enabled: true 
  25.   reload.period: 10s
  26.  
  27. output.elasticsearch:
  28.   hosts: ["10.0.0.51:9200"]
  29.   indices:
  30.   - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
  31.     when.contains:
  32.       event.dataset: "nginx.access"
  33.   - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
  34.     when.contains:
  35.       event.dataset: "nginx.error"
  36.  
  37. setup.template.name: "nginx"
  38. setup.template.pattern: "nginx-*"
  39. setup.template.enabled: false
  40. setup.template.overwrite: true
  41.  
  42. 4.激活filebeat的nginx模块
  43. [root@db01 ~]# filebeat modules enable nginx
  44. [root@db01 ~]# filebeat modules list
  45. [root@db01 ~]# ll /etc/filebeat/modules.d/nginx.yml 
  46. -rw-r--r-- 1 root root 369 Jan 24  2019 /etc/filebeat/modules.d/nginx.yml
  47.  
  48. 5.配置filebeat的nginx模块配置文件
  49. [root@db01 ~]# cat >/etc/filebeat/modules.d/nginx.yml <<EOF    
  50. - module: nginx
  51.   access:
  52.     enabled: true
  53.     var.paths: ["/var/log/nginx/access.log"]
  54.  
  55.   error:
  56.     enabled: true
  57.     var.paths: ["/var/log/nginx/error.log"]
  58. EOF
  59.  
  60.  
  61. 6.es安装filebeat的nginx模块必要插件并重启
  62. #上传插件
  63. [root@db01 ~]# ll
  64. -rw-r--r--  1 root root 33255554 Jan  8 08:15 ingest-geoip-6.6.0.zip
  65. -rw-r--r--  1 root root    62173 Jan  8 08:15 ingest-user-agent-6.6.0.zip
  66.  
  67. #切换目录并安装插件
  68. [root@db01 ~]# cd /usr/share/elasticsearch/
  69. [root@db01 ~]# ./bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip 
  70. 					注意安装时候需要输入  “y” 确认
  71. [root@db01 ~]# ./bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
  72. [root@db01 ~]# systemctl restart elasticsearch
  73.  
  74. 7.重启filebeat
  75. [root@db01 ~]# systemctl restart filebeat
  76.  
  77. 8.删除es-head插件中原有nginx的数据和ibana中的ngixn数据
  78.   生成新的日志数据，es-head插件更新查看，kibana添加

8.filebeat使用模块收集mysql慢日志

  1. #二进制安装
  2.  
  3. 1.下载或上传软件包
  4. wget https://downloads.mysql.com/archives/get/file/mysql-5.6.44-linux-glibc2.12-x86_64.tar.gz
  5.  
  6. 2.解压
  7. [root@db01 ~]# tar xf mysql-5.6.44-linux-glibc2.12-x86_64.tar.gz 
  8. [root@db01 ~]# ll
  9. total 321404
  10. drwxr-xr-x  13 root root       191 Oct 31 04:31 mysql-5.6.44-linux-glibc2.12-x86_64
  11. -rw-r--r--   1 root root 329105487 Oct 30 10:23 mysql-5.6.44-linux-glibc2.12-x86_64.tar.gz
  12.  
  13.  
  14. 3.安装依赖软件包
  15. [root@db01 ~]# yum install -y autoconf libaio*
  16.  
  17. 4.创建 mysql 用户
  18. [root@db01 ~]# useradd mysql -s /sbin/nologin -M
  19. [root@db01 ~]# id mysql
  20. uid=1000(mysql) gid=1000(mysql) groups=1000(mysql)
  21.  
  22.  
  23. 5.将解压后的软件包目录移动到   /opt  目录下面并更改文件名
  24. [root@db01 ~]# mv mysql-5.6.44-linux-glibc2.12-x86_64 /opt/mysql-5.6.44
  25. [root@db01 ~]# cd /opt/mysql-5.6.44/
  26. [root@db01 /opt/mysql-5.6.44]# ll
  27. total 40
  28. drwxr-xr-x  2 root root   4096 Oct 31 04:31 bin
  29. -rw-r--r--  1 7161 31415 17987 Mar 15  2019 COPYING
  30. drwxr-xr-x  3 root root     18 Oct 31 04:30 data
  31. drwxr-xr-x  2 root root     55 Oct 31 04:30 docs
  32. drwxr-xr-x  3 root root   4096 Oct 31 04:30 include
  33. drwxr-xr-x  3 root root    316 Oct 31 04:31 lib
  34. drwxr-xr-x  4 root root     30 Oct 31 04:30 man
  35. drwxr-xr-x 10 root root    291 Oct 31 04:30 mysql-test
  36. -rw-r--r--  1 7161 31415  2496 Mar 15  2019 README
  37. drwxr-xr-x  2 root root     30 Oct 31 04:30 scripts
  38. drwxr-xr-x 28 root root   4096 Oct 31 04:31 share
  39. drwxr-xr-x  4 root root   4096 Oct 31 04:31 sql-bench
  40. drwxr-xr-x  2 root root    136 Oct 31 04:30 support-files
  41.  
  42. 6.制作软连接
  43. [root@db01 ~]# ln -s /opt/mysql-5.6.44/ /opt/mysql
  44. [root@db01 ~]# ll /opt/mysql
  45. lrwxrwxrwx 1 root root 18 Oct 31 04:37 /opt/mysql -> /opt/mysql-5.6.44/
  46.  
  47.  
  48. 7.拷贝启动脚本
  49. [root@db01 /opt/mysql-5.6.44]# cd /opt/mysql-5.6.44/support-files/
  50. [root@db01 /opt/mysql-5.6.44/support-files]# cp mysql.server /etc/init.d/mysqld
  51. [root@db01 /opt/mysql-5.6.44/support-files]# ll /etc/init.d/mysqld
  52. -rwxr-xr-x 1 root root 10565 Oct 31 04:40 /etc/init.d/mysqld
  53.  
  54.  
  55. 8.拷贝配置文件
  56. [root@db01 /opt/mysql-5.6.44/support-files]# cp my-default.cnf /etc/my.cnf
  57. cp: overwrite ‘/etc/my.cnf’? y
  58. [root@db01 /opt/mysql-5.6.44/support-files]# ll /etc/my.cnf
  59. -rw-r--r--. 1 root root 1126 Oct 31 04:41 /etc/my.cnf
  60.  
  61.  
  62. 9.初始化数据库
  63. [root@db01 /opt/mysql-5.6.44/support-files]# cd ../scripts/
  64. [root@db01 /opt/mysql-5.6.44/scripts]# ll
  65. total 36
  66. -rwxr-xr-x 1 7161 31415 34558 Mar 15  2019 mysql_install_db
  67. [root@db01 /opt/mysql-5.6.44/scripts]# ./mysql_install_db --basedir=/opt/mysql --datadir=/opt/mysql/data --user=mysql
  68. #只要有两个ok就行
  69.  
  70. 10.授权 mysql 目录
  71. [root@db01 /opt/mysql-5.6.44/scripts]# chown -R mysql.mysql /opt/mysql-5.6.44/
  72. [root@db01 /opt/mysql-5.6.44/scripts]# ll /opt/
  73. total 0
  74. lrwxrwxrwx  1 mysql mysql  18 Oct 31 04:37 mysql -> /opt/mysql-5.6.44/
  75. drwxr-xr-x 13 mysql mysql 223 Oct 31 04:43 mysql-5.6.44
  76.  
  77.  
  78. 11.修改 mysql 启动脚本和程序
  79. [root@db01 /opt/mysql-5.6.44/scripts]# sed -i 's#/usr/local#/opt#g' /etc/init.d/mysqld /opt/mysql/bin/mysqld_safe
  80.  
  81.  
  82. 12.启动 mysqkl
  83. [root@db01 /opt/mysql-5.6.44/scripts]# /etc/init.d/mysqld start
  84. Starting MySQL.Logging to '/opt/mysql/data/db01.err'.
  85.  SUCCESS!
  86.  
  87.  
  88. 13.添加环境变量
  89. [root@db01 /opt/mysql-5.6.44/scripts]# vim /etc/profile.d/mysql.sh
  90. export PATH="/opt/mysql/bin:$PATH"
  91. [root@db01 /opt/mysql-5.6.44/scripts]# source /etc/profile.d/mysql.sh
  92.  
  93.  
  94. 14.登录mysql数据库
  95. [root@db01 /opt/mysql-5.6.44/scripts]# mysql
  96. Welcome to the MySQL monitor.  Commands end with ; or \g.
  97. Your MySQL connection id is 1
  98. Server version: 5.6.44 MySQL Community Server (GPL)
  99.  
  100. Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
  101.  
  102. Oracle is a registered trademark of Oracle Corporation and/or its
  103. affiliates. Other names may be trademarks of their respective
  104. owners.
  105.  
  106. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
  107.  
  108. mysql> 
  109.  
  110. ==============================================================================
  111.  
  112. #filebeat使用模块收集mysql慢日志
  113.  
  114. 1.配置mysql错误日志和慢日志路径
  115. 编辑my.cnf
  116. [root@db01 ~]# vim /etc/my.cnf
  117. [mysqld]
  118. slow_query_log=ON
  119. slow_query_log_file=/opt/mysql/data/slow.log
  120. long_query_time=1
  121.  
  122. 2.重启mysql并制造慢日志
  123. [root@db01 ~]# /etc/init.d/mysqld restart
  124.  
  125. 3.慢日志制造语句
  126. mysql<
  127. select sleep(2) user,host from mysql.user ;
  128.  
  129. 4.确认慢日志和错误日志确实有生成
  130. [root@db01 ~]#  mysql  -e "show variables like '%slow_query_log%'"
  131.  
  132. +---------------------+----------------------------------+
  133. | Variable_name       | Value                                   |
  134. +---------------------+----------------------------------+
  135. | slow_query_log      | ON                                       |
  136. | slow_query_log_file | /opt/mysql/data/slow.log |
  137. +---------------------+----------------------------------+
  138.  
  139. 5.激活filebeat的mysql模块
  140. [root@db01 ~]# filebeat modules enable mysql
  141.  
  142. 6.配置mysql的模块
  143. [root@db01 ~]# cat /etc/filebeat/modules.d/mysql.yml 
  144. - module: mysql
  145.   # Error logs
  146.   error:
  147.     enabled: true
  148.     var.paths: ["/opt/mysql/data/db01.err"]
  149.  
  150.   # Slow logs
  151.   slowlog:
  152.     enabled: true
  153.     var.paths: ["/opt/mysql/data/slow.log"]
  154.  
  155. 7.配置filebeat根据日志类型做判断
  156. [root@db01 ~]# cat /etc/filebeat/filebeat.yml 
  157. filebeat.config.modules:
  158.   path: ${path.config}/modules.d/*.yml
  159.   reload.enabled: true
  160.   reload.period: 10s
  161.  
  162. output.elasticsearch:
  163.   hosts: ["10.0.0.51:9200"]
  164.   indices:
  165.   - index: "mysql_slow-%{[beat.version]}-%{+yyyy.MM}"
  166.     when.contains:
  167.       source: "/opt/mysql/data/slow.log"
  168.   - index: "mysql_error-%{[beat.version]}-%{+yyyy.MM}"
  169.     when.contains:
  170.       source: "/opt/mysql/data/db01.err"
  171.  
  172. setup.template.name: "mysql"
  173. setup.template.pattern: "mysql-*"
  174. setup.template.enabled: false
  175. setup.template.overwrite: true
  176.  
  177. 8.重启filebeat
  178. [root@db01 ~]# systemctl restart filebeat
  179.  
  180. 9.生成慢日志数据
  181. mysql> select sleep(2) user,host from mysql.user ;
  182. +------+-----------+
  183. | user | host      |
  184. +------+-----------+
  185. |    0 | 127.0.0.1 |
  186. |    0 | ::1       |
  187. |    0 | db01      |
  188. |    0 | db01      |
  189. |    0 | localhost |
  190. |    0 | localhost |
  191. +------+-----------+
  192. 6 rows in set (12.01 sec)
  193.  
  194. 10.登录es-head插件查询和kibana添加查询

收集docker日志

1.filebeat收集docker类型日志 （ 普通版本）

  1. 1.安装dockder
  2. [root@db02 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
  3. [root@db02 ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
  4. [root@db02 ~]# sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
  5. [root@db02 ~]# yum makecache fast
  6. [root@db02 ~]# yum install docker-ce -y
  7. [root@db02 ~]# mkdir -p /etc/docker
  8. [root@db02 ~]# tee /etc/docker/daemon.json <<-'EOF'
  9. {
  10.   "registry-mirrors": ["https://ig2l319y.mirror.aliyuncs.com"]
  11. }
  12. EOF
  13. [root@db02 ~]# systemctl daemon-reload
  14. [root@db02 ~]# systemctl restart docker
  15.  
  16. 2.启动2个Nginx容器并访问测试
  17. [root@db02 ~]# docker run -d -p 80:80 nginx
  18. [root@db02 ~]# docker run -d -p 8080:80 nginx 
  19.  
  20. 3.测试数据是否能通
  21. [root@db02 ~]# curl 10.0.0.52
  22. [root@db02 ~]# curl 10.0.0.52:8080
  23.  
  24. 4.配置filebeat
  25. [root@db02 ~]# cat /etc/filebeat/filebeat.yml 
  26. filebeat.inputs:
  27. - type: docker
  28.   containers.ids: 
  29.     - '*'
  30.  
  31. output.elasticsearch:
  32.   hosts: ["10.0.0.51:9200"]
  33.   index: "docker-%{[beat.version]}-%{+yyyy.MM}"
  34.     
  35. setup.template.name: "docker"
  36. setup.template.pattern: "docker-*"
  37. setup.template.enabled: false
  38. setup.template.overwrite: true
  39.  
  40. 5.重启filebeat
  41. [root@db02 ~]# systemctl restart filebeat 
  42.  
  43. 6.重启es
  44. [root@db02 ~]# systemctl restart elasticsearch
  45.  
  46. 7.访问生成测试数据
  47. [root@db02 ~]# curl 10.0.0.52/1111111111
  48. [root@db02 ~]# curl 10.0.0.52:8080/2222222222
  49.  
  50. 8.登录es-head插件查询和kibana添加

2.filebeat收集docker日志使用docker-compose按服务拆分索引

  1. 1.假设的场景
  2. nginx容器 80端口
  3. toncat容器 8080端口
  4.  
  5. 2.理想中的索引名称
  6. docker-nginx-6.6.0-2020.02
  7. docker-tomcat-6.6.0-2020.02
  8.  
  9. 3.理想的日志记录格式
  10. nginx容器日志：
  11. {
  12. 	"log": "xxxxxx",
  13. 	"stream": "stdout",
  14. 	"time": "xxxx"，
  15. 	"service": "nginx"
  16. }
  17.  
  18. tomcat容器日志：
  19. {
  20. 	"log": "xxxxxx",
  21. 	"stream": "stdout",
  22. 	"time": "xxxx"，
  23. 	"service": "tomcat"
  24. }
  25.  
  26. 4.docker-compose配置
  27. [root@db02 ~]# yum install docker-compose -y
  28. [root@db02 ~]# cat >docker-compose.yml<<EOF
  29. version: '3'
  30. services:
  31.   nginx:
  32.     image: nginx:latest
  33.     labels:
  34.       service: nginx
  35.     logging:
  36.       options:
  37.         labels: "service"
  38.     ports:
  39.       - "80:80"
  40.   tomcat:
  41.     image: nginx:latest
  42.     labels:
  43.       service: tomcat 
  44.     logging:
  45.       options:
  46.         labels: "service"
  47.     ports:
  48.       - "8080:80"
  49. EOF
  50.  
  51.  
  52. 5.删除旧的容器
  53. [root@db02 ~]# docker stop $(docker ps -q)
  54. [root@db02 ~]# docker rm $(docker ps -qa)
  55.  
  56. 6.启动容器
  57. [root@db02 ~]# docker-compose up -d
  58.  
  59. 7.配置filebeat
  60. [root@db02 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
  61. filebeat.inputs:
  62. - type: log 
  63.   enabled: true
  64.   paths:
  65.     - /var/lib/docker/containers/*/*-json.log
  66.   json.keys_under_root: true
  67.   json.overwrite_keys: true
  68.  
  69. output.elasticsearch:
  70.   hosts: ["10.0.0.51:9200"]
  71.   indices:
  72.     - index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM}"
  73.       when.contains:
  74.         attrs.service: "nginx"
  75.     - index: "docker-tomcat-%{[beat.version]}-%{+yyyy.MM}"
  76.       when.contains:
  77.         attrs.service: "tomcat"
  78.  
  79. setup.template.name: "docker"
  80. setup.template.pattern: "docker-*"
  81. setup.template.enabled: false
  82. setup.template.overwrite: true
  83. EOF
  84.  
  85. 8.重启filebeat
  86. [root@db02 ~]# systemctl restart filebeat
  87.  
  88. 9.生成访问日志
  89. [root@db02 ~]# curl 127.0.0.1/nginxxxxxxxxxxx
  90. [root@db02 ~]# curl 127.0.0.1:8080/dbbbbbbbbb
  91.  
  92. 10.es-head插件查看

3.filebeat收集docker日志 ,按照日志类型，access/error拆分

  1. 1.之前收集的docker日志目前不完善的地方
  2. 正常日志和报错日志放在一个索引里了
  3.  
  4. 2.理想中的索引名称
  5. docker-nginx-access-6.6.0-2020.02
  6. docker-nginx-error-6.6.0-2020.02
  7. docker-db-access-6.6.0-2020.02
  8. docker-db-error-6.6.0-2020.02
  9.  
  10. 3.filebeat配置文件
  11. [root@db02 ~]# cat >/etc/filebeat/filebeat.yml <<EOF   
  12. filebeat.inputs:
  13. - type: log 
  14.   enabled: true
  15.   paths:
  16.     - /var/lib/docker/containers/*/*-json.log
  17.   json.keys_under_root: true
  18.   json.overwrite_keys: true
  19.  
  20. output.elasticsearch:
  21.   hosts: ["10.0.0.51:9200"]
  22.   indices:
  23.     - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
  24.       when.contains:
  25.         attrs.service: "nginx"
  26.         stream: "stdout"
  27.     - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
  28.       when.contains:
  29.         attrs.service: "nginx"
  30.         stream: "stderr"
  31.  
  32.     - index: "docker-tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
  33.       when.contains:
  34.         attrs.service: "tomcat"
  35.         stream: "stdout"
  36.     - index: "docker-tomcat-error-%{[beat.version]}-%{+yyyy.MM}"
  37.       when.contains:
  38.         attrs.service: "tomcat"
  39.         stream: "stderr"
  40.  
  41. setup.template.name: "docker"
  42. setup.template.pattern: "docker-*"
  43. setup.template.enabled: false
  44. setup.template.overwrite: true
  45. EOF
  46.  
  47. 4.重启filebeat	
  48. [root@db02 ~]# systemctl restart filebeat 
  49.  
  50. 5.生成测试数据
  51. [root@db02 ~]# curl 127.0.0.1/nginxxxxxxxxxxx
  52. [root@db02 ~]# curl 127.0.0.1:8080/dbbbbbbbbb
  53.  
  54. 6.登录es-head插件查看

4.filebeat收集docker日志优化版

  1. 1.需求分析
  2. json格式并且按照下列索引生成
  3. docker-nginx-access-6.6.0-2020.02
  4. docker-tomcat-access-6.6.0-2020.02
  5. docker-tomcat-error-6.6.0-2020.02
  6. docker-nginx-error-6.6.0-2020.02
  7.  
  8.  
  9. 2.停止并且删除以前的容器
  10. [root@db02 ~]# docker stop $(docker ps -qa)
  11. [root@db02 ~]# docker rm $(docker ps -qa)
  12.  
  13. 3.创建新容器并将容器内的日志映射出来
  14. [root@db02 ~]# docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx/ nginx
  15. [root@db02 ~]# docker run -d -p 8080:80 -v /opt/tomcat:/var/log/nginx/ nginx
  16. [root@db02 ~]# ll /opt/
  17. drwxr-xr-x 2 root root 41 Mar  1 10:24 nginx
  18. drwxr-xr-x 2 root root 41 Mar  1 10:25 tomcat
  19.  
  20.  
  21. 4.准备json格式的nginx配置文件，将其他机器的nginx的配置文件复制到本台服务器上面
  22. [root@db02 ~]# scp 10.0.0.51:/etc/nginx/nginx.conf /root/
  23. [root@db02 ~]# ll
  24. -rw-r--r--  1 root root    1358 Mar  1 10:27 nginx.conf
  25.  
  26. #将日志格式个更改为json格式
  27. [root@db02 ~]# grep "access_log" nginx.conf 
  28.     access_log  /var/log/nginx/access.log  json;
  29.  
  30. 5.拷贝到容器里并重启
  31. #查看容器id
  32. [root@db02 ~]# docker ps
  33.  
  34. [root@db02 ~]# docker cp nginx.conf Nginx容器的ID:/etc/nginx/
  35. [root@db02 ~]# docker cp nginx.conf tomcat容器的ID:/etc/nginx/
  36. [root@db02 ~]# docker stop $(docker ps -qa)
  37. [root@db02 ~]# docker start Nginx容器的ID
  38. [root@db02 ~]# docker start tomcat容器的ID
  39.  
  40.  
  41. 6.删除ES已经存在的索引（ 在 es-head 插件中删除 ）
  42.  
  43.  
  44. 7.配置filebeat配置文件
  45. [root@db02 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
  46. filebeat.inputs:
  47. - type: log 
  48.   enabled: true
  49.   paths:
  50.     - /opt/nginx/access.log
  51.   json.keys_under_root: true
  52.   json.overwrite_keys: true
  53.   tags: ["nginx_access"]
  54.  
  55. - type: log 
  56.   enabled: true
  57.   paths:
  58.     - /opt/nginx/error.log
  59.   tags: ["nginx_err"]
  60.  
  61. - type: log 
  62.   enabled: true
  63.   paths:
  64.     - /opt/tomcat/access.log
  65.   json.keys_under_root: true
  66.   json.overwrite_keys: true
  67.   tags: ["tomcat_access"]
  68.  
  69. - type: log 
  70.   enabled: true
  71.   paths:
  72.     - /opt/tomcat/error.log
  73.   tags: ["tomcat_err"]
  74.  
  75. output.elasticsearch:
  76.   hosts: ["10.0.0.51:9200"]
  77.   indices:
  78.     - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
  79.       when.contains:
  80.         tags: "nginx_access"
  81.  
  82.     - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
  83.       when.contains:
  84.         tags: "nginx_err"
  85.  
  86.     - index: "docker-tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
  87.       when.contains:
  88.         tags: "tomcat_access"
  89.  
  90.     - index: "docker-tomcat-error-%{[beat.version]}-%{+yyyy.MM}"
  91.       when.contains:
  92.         tags: "tomcat_err"
  93.  
  94. setup.template.name: "docker"
  95. setup.template.pattern: "docker-*"
  96. setup.template.enabled: false
  97. setup.template.overwrite: true
  98. EOF
  99.  
  100. 8.重启filebeat
  101. [root@db02 ~]# systemctl restart filebeat
  102.  
  103. 9.访问并测试
  104. [root@db02 ~]# curl 127.0.0.1/hahaha
  105. [root@db02 ~]# curl 127.0.0.1:8080/hahaha
  106. [root@db02 ~]# cat /opt/nginx/access.log
  107. [root@db02 ~]# cat /opt/tomcat/access.log
  108.  
  109. 9.es-head查看

 

使用redis优化方案

1.filebeat引入redis缓存 (redis 单节点)

filebeat收集日志传给redis，因为redis和es不能直接通信，需要中间件logstash从redis中取数据传给es，es在传给kibana展示数据

  1. 1.安装redis
  2. [root@db01 ~]# yum install redis 
  3. [root@db01 ~]# sed -i 's#^bind 127.0.0.1#bind 127.0.0.1 10.0.0.51#' /etc/redis.conf
  4. [root@db01 ~]# systemctl start redis 
  5. [root@db01 ~]# netstat -lntup|grep redis 
  6. [root@db01 ~]# redis-cli -h 10.0.0.51
  7.  
  8. 2.停止docker容器
  9. [root@db01 ~]# docker stop $(docker ps -q)
  10.  
  11. 3.停止filebeat
  12. [root@db01 ~]# systemctl stop filebeat 
  13.  
  14. 4.删除旧的ES索引
  15.  
  16. 5.确认nginx日志为json格式
  17. [root@db01 ~]#  grep "access_log" nginx.conf
  18. access_log  /var/log/nginx/access.log  json;
  19.  
  20. 6.修改filebeat配置文件
  21. [root@db01 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
  22. filebeat.inputs:
  23. - type: log
  24.   enabled: true 
  25.   paths:
  26.     - /var/log/nginx/access.log
  27.   json.keys_under_root: true
  28.   json.overwrite_keys: true
  29.   tags: ["access"]
  30.  
  31. - type: log
  32.   enabled: true 
  33.   paths:
  34.     - /var/log/nginx/error.log
  35.   tags: ["error"]
  36.  
  37. output.redis:
  38.   hosts: ["10.0.0.51"]
  39.   keys:
  40.     - key: "nginx_access"
  41.       when.contains:
  42.         tags: "access"
  43.     - key: "nginx_error"
  44.       when.contains:
  45.         tags: "error"
  46.  
  47. setup.template.name: "nginx"
  48. setup.template.pattern: "nginx_*"
  49. setup.template.enabled: false
  50. setup.template.overwrite: true
  51. EOF
  52.  
  53. 7.重启filebaet和nginx
  54. [root@db01 ~]# systemctl restart nginx 
  55. [root@db01 ~]# systemctl restart filebeat
  56.  
  57. 8.生成测试数据
  58. [root@db01 ~]# curl 127.0.0.1/haha
  59.  
  60. 9.检查
  61. [root@db01 ~]# redis-cli -h 10.0.0.51
  62. keys * 
  63. TYPE nginx_access
  64. LLEN nginx_access
  65. LRANGE nginx_access 0 -1 
  66. 确认是否为json格式
  67.  
  68. 10.安装logstash
  69. [root@db01 ~]# rpm -ivh jdk-8u102-linux-x64.rpm 
  70. [root@db01 ~]# rpm -ivh logstash-6.6.0.rpm
  71.  
  72.  
  73. 11.配置redis将数据传给logstash的配置文件
  74. [root@db01 ~]# cat >/etc/logstash/conf.d/redis.conf<<EOF 
  75. input {
  76.   redis {
  77.     host => "10.0.0.51"
  78.     port => "6379"
  79.     db => "0"
  80.     key => "nginx_access"
  81.     data_type => "list"
  82.   }
  83.   redis {
  84.     host => "10.0.0.51"
  85.     port => "6379"
  86.     db => "0"
  87.     key => "nginx_error"
  88.     data_type => "list"
  89.   }
  90. }
  91.  
  92. filter {
  93.   mutate {
  94.     convert => ["upstream_time", "float"]
  95.     convert => ["request_time", "float"]
  96.   }
  97. }
  98.  
  99. output {
  100.    stdout {}
  101.    if "access" in [tags] {
  102.       elasticsearch {
  103.         hosts => "http://10.0.0.51:9200"
  104.         manage_template => false
  105.         index => "nginx_access-%{+yyyy.MM}"
  106.       }
  107.     }
  108.     if "error" in [tags] {
  109.       elasticsearch {
  110.         hosts => "http://10.0.0.51:9200"
  111.         manage_template => false
  112.         index => "nginx_error-%{+yyyy.MM}"
  113.       }
  114.     }
  115. }
  116. EOF
  117.  
  118. 12.前台启动测试
  119. [root@db01 ~]#  /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf 
  120.  
  121. 13.检查
  122. logstash输出的内容有没有解析成json
  123. es-head上有没有索引生成
  124. redis里的列表数据有没有在减少
  125.  
  126. 14.将logstash放在后台运行
  127. ctrl+c
  128. [root@db01 ~]#  systemctl start logstash
  129. 听风扇声音，开始转的时候表示logstash启动了
  130.  
  131. 15.后台启动后生成数据并在es-head中查看

2.filebeat引入redis完善方案 （使用两台服务器完成redis高可用）

因为filebeat只支持把数据传到redis单节点上面（filebeat不支持传输给redis哨兵或集群，logstash也不支持从redis哨兵或集群里读取数据），所以在filebeat和redis之前配置nginx代理服务，引用keepalives高可用来完成转换，即在redis前面使用nginx做keepalived，假如节点1挂掉了，节点2还能接收filebeat数据传给redis，logstash也连接keepalived，

  1. 1.前提条件
  2. - filebeat不支持传输给redis哨兵或集群
  3. - logstash也不支持从redis哨兵或集群里读取数据
  4.  
  5. 2.安装配置redis（db01、db02安装）
  6. [root@db01 ~]# yum install redis -y
  7. [root@db02 ~]# yum install redis -y
  8. [root@db01 ~]# sed -i 's#^bind 127.0.0.1#bind 127.0.0.1 10.0.0.51#' /etc/redis.conf
  9. [root@db02 ~]# sed -i 's#^bind 127.0.0.1#bind 127.0.0.1 10.0.0.52#' /etc/redis.conf
  10. [root@db01 ~]# systemctl start redis
  11. [root@db02 ~]# systemctl start redis
  12.  
  13. 3.安装配置nginx
  14. 配置官方源
  15. [root@db01 ~]# yum install nginx -y
  16. [root@db02 ~]# yum install nginx -y
  17. 放在nginx.conf最后一行的}后面，不要放在conf.d里面
  18. stream {
  19.   upstream redis {
  20.       server 10.0.0.51:6379 max_fails=2 fail_timeout=10s;
  21.       server 10.0.0.52:6379 max_fails=2 fail_timeout=10s backup;
  22.   }
  23.   
  24.   server {
  25.           listen 6380;
  26.           proxy_connect_timeout 1s;
  27.           proxy_timeout 3s;
  28.           proxy_pass redis;
  29.   }
  30. }
  31.  
  32. #检查并启动nginx
  33. [root@db01 ~]# nginx -t
  34. [root@db02 ~]# nginx -t
  35. [root@db01 ~]# systemctl start nginx 
  36. [root@db02 ~]# systemctl start nginx 
  37.  
  38. 4.安装配置keepalived
  39. [root@db01 ~]# yum install keepalived -y
  40. [root@db02 ~]# yum install keepalived -y
  41.  
  42. #db01的配置  =======（# 虚拟ip 10.0.100）
  43. [root@db01 ~]# cat /etc/keepalived/keepalived.conf 
  44. global_defs {
  45.     router_id db01
  46. }
  47. vrrp_instance VI_1 {
  48.     state MASTER
  49.         interface eth0
  50.         virtual_router_id 50
  51.         priority 150
  52.         advert_int 1
  53.         authentication {
  54.             auth_type PASS
  55.             auth_pass 1111
  56.         }
  57.         virtual_ipaddress {
  58.             10.0.0.100
  59.         }
  60. }
  61.  
  62. #db02的配置 =======（# 虚拟ip 10.0.100）
  63. [root@db02 ~]# cat /etc/keepalived/keepalived.conf 
  64. global_defs {
  65.     router_id db02
  66. }
  67. vrrp_instance VI_1 {
  68.     state BACKUP
  69.     interface eth0
  70.     virtual_router_id 50
  71.     priority 100
  72.     advert_int 1
  73.     authentication {
  74.         auth_type PASS
  75.         auth_pass 1111
  76.     }
  77.     virtual_ipaddress {
  78.         10.0.0.100
  79.     }
  80. }
  81.  
  82. [root@db01 ~]# systemctl start keepalived 
  83. [root@db02 ~]# systemctl start keepalived 
  84. [root@db01 ~]# ip addr |grep 10.0.0.100
  85.  
  86. 5.测试访问能否代理到redis
  87. [root@db01 ~]# redis-cli -h 10.0.0.100 -p 6380
  88. #把db01的redis停掉，测试还能不能连接redis
  89.  
  90. 6.配置filebeat #（只在一台器机器上执行即可）
  91. [root@db01 ~]#  cat >/etc/filebeat/filebeat.yml <<EOF
  92. filebeat.inputs:
  93. - type: log
  94.   enabled: true 
  95.   paths:
  96.     - /var/log/nginx/access.log
  97.   json.keys_under_root: true
  98.   json.overwrite_keys: true
  99.   tags: ["access"]
  100.  
  101. - type: log
  102.   enabled: true 
  103.   paths:
  104.     - /var/log/nginx/error.log
  105.   tags: ["error"]
  106.  
  107. output.redis:
  108.   hosts: ["10.0.0.100:6380"]     #注意此处ip为虚拟ip：10.0.0.100
  109.   keys:
  110.     - key: "nginx_access"
  111.       when.contains:
  112.         tags: "access"
  113.     - key: "nginx_error"
  114.       when.contains:
  115.         tags: "error"
  116.  
  117. setup.template.name: "nginx"
  118. setup.template.pattern: "nginx_*"
  119. setup.template.enabled: false
  120. setup.template.overwrite: true
  121. EOF
  122.  
  123. 7.测试访问filebeat能否传输到redis
  124. [root@db01 ~]# curl 127.0.0.1/haha
  125. [root@db01 ~]# redis-cli -h 10.0.0.51 #应该有数据
  126. [root@db02 ~]# redis-cli -h 10.0.0.52 #应该没数据
  127. [root@db01 ~]# redis-cli -h 10.0.0.100 -p 6380 #应该有数据
  128.  
  129. 8.配置logstash 
  130. [root@db01 ~]# cat >/etc/logstash/conf.d/redis.conf<<EOF 
  131. input {
  132.   redis {
  133.     host => "10.0.0.100"     #注意此处ip为虚拟ip：10.0.0.100
  134.     port => "6380"
  135.     db => "0"
  136.     key => "nginx_access"
  137.     data_type => "list"
  138.   }
  139.   redis {
  140.     host => "10.0.0.100"      #注意此处ip为虚拟ip：10.0.0.100
  141.     port => "6380"
  142.     db => "0"
  143.     key => "nginx_error"
  144.     data_type => "list"
  145.   }
  146. }
  147.  
  148. filter {
  149.   mutate {
  150.     convert => ["upstream_time", "float"]
  151.     convert => ["request_time", "float"]
  152.   }
  153. }
  154.  
  155. output {
  156.    stdout {}
  157.    if "access" in [tags] {
  158.       elasticsearch {
  159.         hosts => "http://10.0.0.51:9200"
  160.         manage_template => false
  161.         index => "nginx_access-%{+yyyy.MM}"
  162.       }
  163.     }
  164.     if "error" in [tags] {
  165.       elasticsearch {
  166.         hosts => "http://10.0.0.51:9200"
  167.         manage_template => false
  168.         index => "nginx_error-%{+yyyy.MM}"
  169.       }
  170.     }
  171. }
  172. EOF
  173.  
  174. 9.启动测试
  175. /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf
  176.  
  177. #测试完毕后台启动
  178. systemctl start logstash
  179.  
  180. 10.最终测试
  181. ab -n 10000 -c 100 10.0.0.100/
  182. 检查es-head上索引条目是否为10000条
  183. 关闭db01的redis，在访问，测试logstash正不正常
  184. 恢复db01的redis，再测试
  185.  
  186. 11.登录es-head查看日志数据

3.filebeat引入redis优化方案

  1. 1.新增加一个日志路径需要修改4个地方：
  2. - filebat 2个位置
  3. - logstash 2个位置
  4.  
  5. 2.优化之后需要修改的地方2个地方
  6. - filebat 1个位置
  7. - logstash 1个位置
  8.  
  9. 3.filebeat配置文件
  10. cat /etc/filebeat/filebeat.yml
  11. filebeat.inputs:
  12. - type: log
  13.   enabled: true 
  14.   paths:
  15.     - /var/log/nginx/access.log
  16.   json.keys_under_root: true
  17.   json.overwrite_keys: true
  18.   tags: ["access"]
  19.  
  20. - type: log
  21.   enabled: true 
  22.   paths:
  23.     - /var/log/nginx/error.log
  24.   tags: ["error"]
  25.  
  26.  
  27. output.redis:
  28.   hosts: ["10.0.0.100:6380"]
  29.   key: "nginx_log"
  30.  
  31. setup.template.name: "nginx"
  32. setup.template.pattern: "nginx_*"
  33. setup.template.enabled: false
  34. setup.template.overwrite: true
  35.  
  36. 4.优化后的logstash
  37.  cat /etc/logstash/conf.d/redis.conf
  38. input {
  39.   redis {
  40.     host => "10.0.0.100"
  41.     port => "6380"
  42.     db => "0"
  43.     key => "nginx_log"
  44.     data_type => "list"
  45.   }
  46. }
  47.  
  48. filter {
  49.   mutate {
  50.     convert => ["upstream_time", "float"]
  51.     convert => ["request_time", "float"]
  52.   }
  53. }
  54.  
  55. output {
  56.    stdout {}
  57.    if "access" in [tags] {
  58.       elasticsearch {
  59.         hosts => "http://10.0.0.51:9200"
  60.         manage_template => false
  61.         index => "nginx_access-%{+yyyy.MM}"
  62.       }
  63.     }
  64.     if "error" in [tags] {
  65.       elasticsearch {
  66.         hosts => "http://10.0.0.51:9200"
  67.         manage_template => false
  68.         index => "nginx_error-%{+yyyy.MM}"
  69.       }
  70.     }
  71. }

使用kafka缓存方案/kibana画图

1.ELK使用kafka作为缓存

  1. #============注意es和kibana需要先启动、zook和kafak页需要java环境=============#
  2.  
  3.  
  4. 0.配置密钥和host解析    #解析需要三台都配置
  5. [root@db01 ~]# cat >/etc/hosts<<EOF
  6. 10.0.0.51 db01
  7. 10.0.0.52 db02
  8. 10.0.0.53 db03
  9. EOF
  10. #生成秘钥对并分发秘钥
  11. [root@db01 ~]# ssh-keygen
  12. [root@db01 ~]# ssh-copy-id 10.0.0.52
  13. [root@db01 ~]# ssh-copy-id 10.0.0.53
  14.  
  15. 1.安装zook
  16. ###db01操作
  17. [root@db01 ~]# yum install -y rsync
  18. [root@db01 ~]# cd /data/soft
  19. [root@db01 ~]# tar zxf zookeeper-3.4.11.tar.gz -C /opt/
  20. [root@db01 ~]# ln -s /opt/zookeeper-3.4.11/ /opt/zookeeper
  21. [root@db01 ~]# mkdir -p /data/zookeeper
  22. [root@db01 ~]# cat >/opt/zookeeper/conf/zoo.cfg<<EOF
  23. tickTime=2000
  24. initLimit=10
  25. syncLimit=5
  26. dataDir=/data/zookeeper
  27. clientPort=2181
  28. server.1=10.0.0.51:2888:3888
  29. server.2=10.0.0.52:2888:3888
  30. server.3=10.0.0.53:2888:3888
  31. EOF
  32. [root@db01 ~]# echo "1" > /data/zookeeper/myid
  33. [root@db01 ~]# cat /data/zookeeper/myid
  34. 1
  35. [root@db01 ~]# rsync -avz /opt/zookeeper* 10.0.0.52:/opt/
  36. [root@db01 ~]# rsync -avz /opt/zookeeper* 10.0.0.53:/opt/
  37.  
  38. ###db02操作
  39. [root@db02 ~]# yum install -y rsync
  40. [root@db02 ~]# mkdir -p /data/zookeeper
  41. [root@db02 ~]# echo "2" > /data/zookeeper/myid
  42. [root@db02 ~]# cat /data/zookeeper/myid
  43. 2
  44.  
  45. ###db03操作
  46. [root@db03 ~]# yum install -y rsync
  47. [root@db03 ~]# mkdir -p /data/zookeeper
  48. [root@db03 ~]# echo "3" > /data/zookeeper/myid
  49. [root@db03 ~]# cat /data/zookeeper/myid
  50. 3
  51.  
  52. 2.启动zookeeper（三台机器都需要启动）
  53. [root@db01 ~]# /opt/zookeeper/bin/zkServer.sh start
  54. [root@db02 ~]# /opt/zookeeper/bin/zkServer.sh start
  55. [root@db03 ~]# /opt/zookeeper/bin/zkServer.sh start
  56.  
  57. 3.检查启动是否成功（三台机器都需要启动）
  58. [root@db01 ~]# /opt/zookeeper/bin/zkServer.sh status
  59. [root@db02 ~]# /opt/zookeeper/bin/zkServer.sh status
  60. [root@db03 ~]# /opt/zookeeper/bin/zkServer.sh status
  61.  
  62. #如果启动正常mode应该是
  63. 2个follower
  64. 1个leader
  65.  
  66. 4.测试zookeeper通讯是否正常
  67. 在一个节点上执行,创建一个频道
  68. /opt/zookeeper/bin/zkCli.sh -server 10.0.0.51:2181
  69. create /test "hello"
  70.  
  71. 在其他节点上看能否接收到
  72. /opt/zookeeper/bin/zkCli.sh -server 10.0.0.52:2181
  73. get /test
  74.  
  75. 5.安装kafka
  76. ###db01操作
  77. [root@db01 ~]# cd /data/soft/
  78. [root@db01 ~]# tar zxf kafka_2.11-1.0.0.tgz -C /opt/
  79. [root@db01 ~]# ln -s /opt/kafka_2.11-1.0.0/ /opt/kafka
  80. [root@db01 ~]# mkdir /opt/kafka/logs
  81. [root@db01 ~]# cat >/opt/kafka/config/server.properties<<EOF
  82. broker.id=1
  83. listeners=PLAINTEXT://10.0.0.51:9092
  84. num.network.threads=3
  85. num.io.threads=8
  86. socket.send.buffer.bytes=102400
  87. socket.receive.buffer.bytes=102400
  88. socket.request.max.bytes=104857600
  89. log.dirs=/opt/kafka/logs
  90. num.partitions=1
  91. num.recovery.threads.per.data.dir=1
  92. offsets.topic.replication.factor=1
  93. transaction.state.log.replication.factor=1
  94. transaction.state.log.min.isr=1
  95. log.retention.hours=24
  96. log.segment.bytes=1073741824
  97. log.retention.check.interval.ms=300000
  98. zookeeper.connect=10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181
  99. zookeeper.connection.timeout.ms=6000
  100. group.initial.rebalance.delay.ms=0
  101. EOF
  102. [root@db01 ~]# rsync -avz /opt/kafka* 10.0.0.52:/opt/
  103. [root@db01 ~]# rsync -avz /opt/kafka* 10.0.0.53:/opt/
  104.  
  105.  
  106. ###db02操作
  107. [root@db02 ~]# sed -i "s#10.0.0.51:9092#10.0.0.52:9092#g" /opt/kafka/config/server.properties
  108. [root@db02 ~]# sed -i "s#broker.id=1#broker.id=2#g" /opt/kafka/config/server.properties
  109.  
  110. ###db03操作
  111. [root@db03 ~]# sed -i "s#10.0.0.51:9092#10.0.0.53:9092#g" /opt/kafka/config/server.properties
  112. [root@db03 ~]# sed -i "s#broker.id=1#broker.id=3#g" /opt/kafka/config/server.properties
  113.  
  114.  
  115. 6.先前台启动kafka测试 （三台机器都需要启动）
  116. [root@db01 ~]# /opt/kafka/bin/kafka-server-start.sh  /opt/kafka/config/server.properties
  117. [root@db02 ~]# /opt/kafka/bin/kafka-server-start.sh  /opt/kafka/config/server.properties
  118. [root@db03 ~]# /opt/kafka/bin/kafka-server-start.sh  /opt/kafka/config/server.properties
  119.  
  120. 7.检查是否启动 （三台机器都需要启动）
  121. jps
  122.  
  123. 8.kafka前台启动测试命令发送消息
  124. 创建命令
  125. /opt/kafka/bin/kafka-topics.sh --create --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --partitions 3 --replication-factor 3 --topic  messagetest
  126.  
  127. 测试获取所有的频道
  128. /opt/kafka/bin/kafka-topics.sh  --list --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181
  129.  
  130. 测试发送消息
  131. /opt/kafka/bin/kafka-console-producer.sh --broker-list  10.0.0.51:9092,10.0.0.52:9092,10.0.0.53:9092 --topic  messagetest
  132.  
  133. 其他节点测试接收
  134. /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --topic messagetest --from-beginning
  135.  
  136. 9.测试成功之后,可以放在后台启动  （三台都启动）
  137. 按ctrl + c 停止kafka的前台启动，切换到后台启动
  138. [root@db01 ~]# /opt/kafka/bin/kafka-server-start.sh  -daemon /opt/kafka/config/server.properties
  139. [root@db02 ~]# /opt/kafka/bin/kafka-server-start.sh  -daemon /opt/kafka/config/server.properties
  140. [root@db03 ~]# /opt/kafka/bin/kafka-server-start.sh  -daemon /opt/kafka/config/server.properties
  141.  
  142. 10.配置filebeat
  143. [root@db01 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
  144. filebeat.inputs:
  145. - type: log
  146.   enabled: true 
  147.   paths:
  148.     - /var/log/nginx/access.log
  149.   json.keys_under_root: true
  150.   json.overwrite_keys: true
  151.   tags: ["access"]
  152.  
  153. - type: log
  154.   enabled: true 
  155.   paths:
  156.     - /var/log/nginx/error.log
  157.   tags: ["error"]
  158.  
  159. output.kafka:
  160.   hosts: ["10.0.0.51:9092", "10.0.0.52:9092", "10.0.0.53:9092"]
  161.   topic: 'filebeat'
  162.  
  163. setup.template.name: "nginx"
  164. setup.template.pattern: "nginx_*"
  165. setup.template.enabled: false
  166. setup.template.overwrite: true
  167. EOF
  168.  
  169. 重启filebeat
  170. [root@db01 ~]# systemctl restart filebeat 
  171.  
  172. 11.访问并检查kafka里有没有收到日志
  173. [root@db01 ~]# curl 10.0.0.51
  174.  
  175. #获取filebeat的频道
  176. [root@db01 ~]#  /opt/kafka/bin/kafka-topics.sh  --list --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181
  177.  
  178. #接收filebeat频道发来的消息
  179. [root@db01 ~]#  /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --topic filebeat --from-beginning
  180.  
  181.  
  182. 12.logstash配置文件
  183. [root@db01 ~]# cat > /etc/logstash/conf.d/kafka.conf<<EOF 
  184. input {
  185.   kafka{
  186.     bootstrap_servers=>["10.0.0.51:9092,10.0.0.52:9092,10.0.0.53:9092"]
  187.     topics=>["filebeat"]
  188.     group_id=>"logstash"
  189.     codec => "json"
  190.   }
  191. }
  192.  
  193. filter {
  194.   mutate {
  195.     convert => ["upstream_time", "float"]
  196.     convert => ["request_time", "float"]
  197.   }
  198. }
  199.  
  200. output {
  201.    stdout {}
  202.    if "access" in [tags] {
  203.       elasticsearch {
  204.         hosts => "http://10.0.0.51:9200"
  205.         manage_template => false
  206.         index => "nginx_access-%{+yyyy.MM}"
  207.       }
  208.     }
  209.     if "error" in [tags] {
  210.       elasticsearch {
  211.         hosts => "http://10.0.0.51:9200"
  212.         manage_template => false
  213.         index => "nginx_error-%{+yyyy.MM}"
  214.       }
  215.     }
  216. }
  217. EOF
  218.  
  219. 13.前台启动logatash测试
  220. #先清空ES以前生成的索引
  221. [root@db01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka.conf 
  222.  
  223. 生成访问日志
  224. [root@db01 ~]# curl 127.0.0.1

测试：

原数据：

1.停掉db03的zookeeper

  1. #听到zookeeper
  2. [root@db03 ~]# /opt/zookeeper/bin/zkServer.sh stop
  3. ZooKeeper JMX enabled by default
  4. Using config: /opt/zookeeper/bin/../conf/zoo.cfg
  5. Stopping zookeeper ... STOPPED
  6.  
  7. #查看jps，原来3个
  8. [root@db03 ~]# jps
  9. 71553 Kafka
  10. 72851 Jps
  11.  
  12. #测试生成数据====db01测试
  13. [root@db01 ~]# curl 127.0.0.1
  14. db01-www
  15.  
  16. #登录es-head查看

2.停掉db02的zookeeper

  1. #查看jps数据
  2. [root@db02 ~]# jps
  3. 74467 QuorumPeerMain
  4. 78053 Jps
  5. 76628 Kafka
  6.  
  7. #停掉db02的zookeeper
  8. [root@db02 ~]# /opt/zookeeper/bin/zkServer.sh stop
  9. ZooKeeper JMX enabled by default
  10. Using config: /opt/zookeeper/bin/../conf/zoo.cfg
  11. Stopping zookeeper ... STOPPED
  12.  
  13. #查看jps，剩两条
  14. [root@db02 ~]# jps
  15. 78210 Jps
  16. 76628 Kafka
  17.  
  18. #测试生成数据====db01测试
  19. [root@db01 ~]# curl 127.0.0.1
  20. db01-www
  21.  
  22. #登录es-head查看

3.停掉db01的kafa

  1. #查看jps数据
  2. [root@db01 ~]# jps
  3. 76902 Kafka
  4. 48472 Logstash
  5. 78089 Logstash
  6. 79034 Jps
  7. 74509 QuorumPeerMain
  8.  
  9. #停掉db01的kafa
  10. [root@db01 ~]# /opt/kafka/bin/kafka-server-stop.sh 
  11.  
  12. #查看jps数据
  13. [root@db01 ~]# jps
  14. 79251 Jps
  15. 48472 Logstash
  16. 78089 Logstash
  17. 74509 QuorumPeerMain
  18.  
  19. #测试生成数据====db01测试
  20. [root@db01 ~]# curl 127.0.0.1
  21. db01-www
  22.  
  23. #登录es-head查看

  1. #总结kafka实验
  2. 1.前提条件
  3. - kafka和zook都是基于java的，所以需要java环境
  4. - 这俩比较吃资源，内存得够
  5.  
  6. 2.安装zook注意
  7. - 每台机器的myid要不一样，而且要和配置文件里的id对应上
  8. - 启动测试，角色为leader和follower
  9. - 测试发送和接受消息 
  10.  
  11. 3.安装kafka注意
  12. - kafka依赖于zook，所以如果zook不正常,kafka不能工作
  13. - kafka配置文件里要配上zook的所有IP的列表
  14. - kafka配置文件里要注意，写自己的IP地址
  15. - kafka配置文件里要注意，自己的ID是zook里配置的myid
  16. - kafka启动要看日志出现started才算是成功
  17.  
  18. 4.测试zook和kafka
  19. - 一端发送消息
  20. - 两端能实时接收消息
  21.  
  22. 5.配置filebeat
  23. - output要配上kafka的所有的IP列表
  24.  
  25. 6.配置logstash
  26. - input要写上所有的kafka的IP列表，别忘了[]
  27. - 前台启动测试成功后再后台启动
  28.  
  29. 7.毁灭测试结果
  30. - 只要还有1个zook和1个kafka节点，就能正常收集日志

2.kibana画图展示

 
 
 
 

 

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/菜鸟追梦旅行/article/detail/569443
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号