devbox
盐析白兔
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Elasticsearch 集群日志收集搭建_es 日志

作者：盐析白兔 | 2024-07-16 23:11:41

赞

踩

es 日志

Elasticsearch-7.2.0+Logstash-7.2.0+Kibana-7.2.0+-Filebeat-7.6.0

第一台集群内网ip：10.0.0.223

ES配置文件：/es_data/es/elasticsearch-7.2.0/config/elasticsearch.yml

ES启动命令：/es_data/es/elasticsearch-7.2.0/bin/elasticsearch

  1. cluster.name: es-search
  2. node.name: node-machine-name
  3.  
  4. node.master: true
  5. node.data: true
  6.  
  7. path.data: /es_data/data/es
  8. path.logs: /es_data/log/es
  9.  
  10. transport.tcp.port: 9300
  11. transport.tcp.compress: true
  12. http.port: 9200
  13. network.host: 10.0.0.223
  14.  
  15. http.cors.enabled: true
  16. http.cors.allow-origin: "*"
  17. http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
  18. http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"
  19.  
  20. cluster.initial_master_nodes: ["10.0.0.223","10.0.1.9","10.0.1.10"]
  21. discovery.seed_hosts: ["10.0.0.223", "10.0.1.10", "10.0.1.9"]
  22.  
  23. gateway.recover_after_nodes: 2
  24. gateway.expected_nodes: 2
  25. gateway.recover_after_time: 5m
  26.  
  27. action.destructive_requires_name: false
  28. cluster.routing.allocation.disk.threshold_enabled: true
  29. cluster.routing.allocation.disk.watermark.low: 20gb
  30. cluster.routing.allocation.disk.watermark.high: 10gb
  31. cluster.routing.allocation.disk.watermark.flood_stage: 5gb
  32.  
  33. # 需求锁住物理内存，是：true、否：false
  34. bootstrap.memory_lock: false
  35. # SecComp检测，是：true、否：false
  36. bootstrap.system_call_filter: false

Kibana配置文件：/es_data/es/kibana-7.2.0/config/kibana.yml

启动命令：/es_data/es/kibana-7.2.0/bin/kibana

  1. server.port: 5601
  2.  
  3. server.host: "localhost"
  4.  
  5. server.basePath: ""
  6.  
  7. server.rewriteBasePath: false
  8.  
  9. elasticsearch.hosts:["http://10.0.0.223:9200","http://10.0.1.9:9200","http://10.0.1.10:9200"]
  10.  
  11. kibana.index: ".kibana"
  12.  
  13. i18n.locale: "zh-CN"

Kibana nginx   代理配置文件

  1. server {
  2.       listen       80;
  3.       server_name www.elasticsearch.com;
  4.       client_max_body_size 1000m;
  5.  
  6.       location / {
  7.               proxy_read_timeout 300;
  8.               proxy_connect_timeout 300;
  9.               proxy_redirect off;
  10.               proxy_http_version 1.1;
  11.               proxy_set_header Host $http_host;
  12.               proxy_set_header X-Real-IP $remote_addr;
  13.               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  14.               proxy_set_header X-Forwarded-Proto http;
  15.  
  16.               proxy_pass http://10.0.0.223:9200;
  17.       }
  18. }

第二台ES集群内网ip：10.0.1.10

ES配置文件：/es_data/es/elasticsearch-7.2.0/config/elasticsearch.yml

  1. cluster.name: es-search
  2. node.name: node-machine-name
  3.  
  4. node.master: true
  5. node.data: true
  6. path.data: /es_data/data/es
  7. path.logs: /es_data/log/es
  8.  
  9. transport.tcp.port: 9300
  10. transport.tcp.compress: true
  11.  
  12. http.port: 9200
  13. network.host: 10.0.1.10
  14.  
  15. http.cors.enabled: true
  16. http.cors.allow-origin: "*"
  17. http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
  18. http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"
  19.  
  20. cluster.initial_master_nodes: ["10.0.0.223","10.0.1.9","10.0.1.10"]
  21. discovery.seed_hosts: ["10.0.0.223", "10.0.1.10", "10.0.1.9"]
  22.  
  23. gateway.recover_after_nodes: 2
  24. gateway.expected_nodes: 2
  25. gateway.recover_after_time: 5m
  26.  
  27. action.destructive_requires_name: false
  28. cluster.routing.allocation.disk.threshold_enabled: true
  29. cluster.routing.allocation.disk.watermark.low: 20gb
  30. cluster.routing.allocation.disk.watermark.high: 10gb
  31. cluster.routing.allocation.disk.watermark.flood_stage: 5gb
  32.  
  33. # 需求锁住物理内存，是：true、否：false
  34. bootstrap.memory_lock: false
  35. # SecComp检测，是：true、否：false
  36. bootstrap.system_call_filter: false
  37.  
  38.

Logstatsh 接收器配置

启动命令：/es_data/es/logstash-7.2.0/bin/logstash -f /es_data/es/logstash-7.2.0/conf.d/es_log.conf --path.data=/es_data/data/logstash/es_log/

配置文件：/es_data/es/logstash-7.2.0/conf.d/es_log.conf

  1. input {
  2.         beats {
  3.                 port => 5044
  4.         }
  5. }
  6.  
  7. filter {
  8.         mutate {
  9.                 split => ["message", "|"]
  10.         }
  11.  
  12.         if [message][3] =~ '[0-9a-z]{40}' {
  13.                 mutate {
  14.                         add_field => { "log_time" => "%{[message][0]}"}
  15.                         add_field => { "log_level" => "%{[message][1]}"}
  16.                         add_field => { "log_process_id" => "%{[message][2]}"}
  17.                         add_field => { "log_session" => "%{[message][3]}"}
  18.                         add_field => { "log_file_name" => "%{[message][6]}"}
  19.                         add_field => { "log_func_name" => "%{[message][7]}"}
  20.                         add_field => { "log_line" => "%{[message][8]}"}
  21.                 }
  22.                 mutate {
  23.                         update => { "message" => "%{[message][9]}" }
  24.                 }
  25.  
  26.         }
  27.         else if [message][2] =~ '[0-9a-z]+-[0-9a-z]+-[0-9a-z]+-[0-9a-z]+-[0-9a-z]' {
  28.                 mutate {
  29.                         add_field => { "log_time" => "%{[message][0]}"}
  30.                         add_field => { "log_level" => "%{[message][1]}"}
  31.                         add_field => { "log_process_id" => "%{[message][3]}"}
  32.                         add_field => { "log_session" => "%{[message][2]}"}
  33.                         add_field => { "log_thread_id" => "%{[message][4]}"}
  34.                         add_field => { "log_file_name" => "%{[message][5]}"}
  35.                         add_field => { "log_func_name" => "%{[message][6]}"}
  36.                         add_field => { "log_line" => "%{[message][7]}"}
  37.                 }
  38.                 mutate {
  39.                         update => { "message" => "%{[message][8]}" }
  40.                 }
  41.  
  42.         }
  43.         else
  44.         {
  45.                 mutate {
  46.                         split => ["message", ","]
  47.                 }
  48.                 mutate {
  49.                         add_field => { "log_time" => "%{[message][0]}"}
  50.                         add_field => { "log_level" => "%{[message][1]}"}
  51.                         add_field => { "log_process_id" => "%{[message][2]}"}
  52.                 }
  53.                 mutate {
  54.                         update => { "message" => "%{[message][3]}" }
  55.                 }
  56.         }
  57.  
  58.         mutate {
  59.                 strip => ["log_time"]
  60.         }
  61. }
  62.  
  63. output {
  64.         elasticsearch {
  65.                 hosts => ["10.0.0.223:9200","10.0.1.10:9200","10.0.1.9:9200"]
  66.                 index => "supervisor-log-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  67.         }
  68. }

第三台ES集群内网ip：10.0.1.9

ES配置文件：/es_data/elasticsearch-7.2.0/config/elasticsearch.yml

  1. cluster.name: es-search
  2. node.name: node-machine-name
  3.  
  4. node.master: true
  5. node.data: true
  6. path.data: /es_data/data/es
  7. path.logs: /es_data/log/es
  8.  
  9. transport.tcp.port: 9300
  10. transport.tcp.compress: true
  11. http.port: 9200
  12. network.host: 10.0.1.9
  13.  
  14. # 增加新的参数，这样head插件可以访问es (5.x版本，如果没有可以自己手动加)
  15. http.cors.enabled: true
  16. http.cors.allow-origin: "*"
  17. http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
  18. http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"
  19.  
  20. cluster.initial_master_nodes: ["10.0.0.223","10.0.1.9","10.0.1.10"]
  21. discovery.seed_hosts: ["10.0.0.223", "10.0.1.10", "10.0.1.9"]
  22.  
  23. gateway.recover_after_nodes: 2
  24. gateway.expected_nodes: 2
  25. gateway.recover_after_time: 5m
  26.  
  27. action.destructive_requires_name: false
  28. cluster.routing.allocation.disk.threshold_enabled: true
  29. cluster.routing.allocation.disk.watermark.low: 20gb
  30. cluster.routing.allocation.disk.watermark.high: 10gb
  31. cluster.routing.allocation.disk.watermark.flood_stage: 5gb
  32.  
  33. # 需求锁住物理内存，是：true、否：false
  34. bootstrap.memory_lock: false
  35. # SecComp检测，是：true、否：false
  36. bootstrap.system_call_filter: false

第四台具体生产日志的地方

Filebeat配置：/www/filebeat/filebeat.yml

启动命令：/www/filebeat/filebeat -e -c /www/filebeat/filebeat.yml

  1. - type: log
  2.   enabled: true
  3.   document_type: "supervisor"
  4.   exclude_files: ["filebeat-out.log$"]
  5.   paths:
  6.    - /var/log/supervisor/*.log
  7.   fields:
  8.     type: supervisor
  9.   encoding: plain
  10.   input_type: log
  11.   multiline.pattern: '^\s\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}'
  12.   multiline.negate: true
  13.   multiline.match: after
  14.  
  15. # Filebeat modules
  16. filebeat.config.modules:
  17.   path: ${path.config}/modules.d/*.yml
  18.   reload.enabled: false
  19.  
  20. # Elasticsearch template setting
  21. setup.template.settings:
  22.   index.number_of_shards: 1
  23.   index.number_of_replicas: 0
  24.  
  25. setup.template.overwrite: true
  26. setup.template.name: "machine-name"
  27. setup.template.pattern: "machine-name*"
  28.  
  29. # 生命周期管理
  30. setup.ilm.enabled: true
  31. setup.ilm.rollover_alias: "machine-name"
  32. setup.ilm.pattern: "{now/d}-000001"
  33. setup.ilm.policy_name: "machine-name-policy"
  34.  
  35. # Logstash output
  36. output.logstash:
  37.     enabled: true
  38.     hosts: ["10.0.1.10:5044"]
  39.     worker: 1
  40.     compression_level: 3
  41.     loadbalance: true
  42.     pipelining: 0
  43.     index: 'log_index'
  44.  
  45. # Processors
  46. processors:
  47.   - add_host_metadata: ~
  48.   - add_cloud_metadata: ~
  49.   - add_docker_metadata: ~
  50.   - add_kubernetes_metadata: ~

Kibana 配置：

设置-索引模式：创建索引模式 supervisor-log-*

上面的索引管理里面就可以看到已经匹配的索引了

创建生命周期策略-操作匹配索引模版，自动移除周期外的日志，保证集群健康运行

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/盐析白兔/article/detail/836683
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号