当前位置:   article > 正文

监控系统日志采集 - Filebeat_filebeat采集日志

filebeat采集日志

Filebeat 是什么?

Filebeat是本地文件的日志数据采集器,可监控日志目录或特定日志文件(tail file),并将它们转发给Elasticsearch或Logstatsh进行索引、kafka等。

是使用 Golang 实现的轻量型日志采集器,也是 Elasticsearch stack 里面的一员。本质上是一个 agent ,可以安装在各个节点上,根据配置读取对应位置的日志,并上报到相应的地方去。

工作原理

Filebeat 由两个主要组件组成:harvester 和 prospector。

  • 采集器 harvester 的主要职责是读取单个文件的内容。读取每个文件,并将内容发送到 the output。
  • 查找器 prospector 的主要职责是管理 harvester 并找到所有要读取的文件来源。如果输入类型为日志,则查找器将查找路径匹配的所有文件,并为每个文件启动一个 harvester。

安装

  1. # 1.安装
  2. cd /usr/local
  3. wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.16.3-linux-x86_64.tar.gz
  4. tar -zxf filebeat-7.16.3-linux-x86_64.tar.gz
  5. rm -f filebeat-7.16.3-linux-x86_64.tar.gz
  6. mv filebeat-7.16.3-linux-x86_64 filebeat
  7. cd filebeat
  8. mkdir -p conf/inputs.d
  9. // 不要默认配置了,把默认配置进行重命名留存
  10. mv filebeat.yml filebeat.example.yml
  11. # 2.采集配置 ==============================
  12. # 配置内容详见下面的"配置"
  13. # 2.1)主配置
  14. vim conf/filebeat.yml
  15. # 2.2)子配置配置
  16. vim conf/inputs.d/nginx.yml
  17. vim conf/inputs.d/runtime.yml
  18. # 3.启动 filebeat
  19. nohup ./filebeat -e -c conf/filebeat.yml & # root 后台守护进程启动

调试说明

  1. # 调试说明 ========================================
  2. # 1)修改主配置 output为如下内容:
  3. output.console:
  4. pretty: true
  5. # 2)执行下面命令,如果屏幕有数据输出说明程序启动正常
  6. ./filebeat -e -c conf/filebeat.yml

 配置

1. 主配置 filebeat/conf/filebeat.yml

  1. # ============================== Filebeat inputs ===============================
  2. filebeat.config.inputs:
  3. enabled: true
  4. path: ${path.config}/conf/inputs.d/*.yml
  5. reload.enabled: true
  6. reload.period: 10s
  7. # ============================== Filebeat logging ===============================
  8. logging.level: info
  9. logging.to_files: true
  10. logging.files:
  11. rotateonstartup: false
  12. path: /usr/local/filebeat/logs/filebeat.log
  13. name: filebeat
  14. # rotateeverybytes: 104857600
  15. interval: 24h
  16. keepfiles: 2
  17. permissions: 0644
  18. # ============================== Filebeat modules ==============================
  19. #filebeat.config.modules:
  20. # path: ${path.config}/modules.d/*.yml
  21. # reload.enabled: false
  22. # reload.period: 10s
  23. # ================================== Outputs ===================================
  24. output.kafka:
  25. hosts: ["xx.xx.xx.xx:9092", "xx.xx.xx.xx:9092", "xx.xx.xx.xx:9092"]
  26. topic: 'pc_app-%{[app_id]}-%{[app_type]}-%{[app_env]}-%{[log_type]}-%{[data_type]}'
  27. required_acks: 1

注Note:

topic 如果没有 需要先申请

2. 子配置

Note: 只需修改 paths 和 fields 字段

3. Nginx配置

filebeat/conf/inputs.d/nginx.yml
  1. - type: log
  2. paths:
  3. #- /usr/local/nginx/logs/access-json.log
  4. encoding: utf8
  5. scan_frequency: 5s
  6. fields:
  7. app_id: nginx
  8. app_type: prod_official
  9. app_env: prod
  10. log_type: access
  11. data_type: json
  12. fields_under_root: true
  13. - type: log
  14. paths:
  15. #- /usr/local/nginx/logs/error.log
  16. encoding: utf8
  17. scan_frequency: 5s
  18. fields:
  19. app_id: nginx
  20. app_type: prod_official
  21. app_env : prod
  22. log_type: error
  23. data_type: raw
  24. fields_under_root: true

4. 业务代码日志

filebeat/conf/inputs.d/runtime.yml
  1. - type: log
  2. paths:
  3. # - /var/www/20230103/api/runtime/logs/runtime*.log
  4. # - /var/www/20230103/app/runtime/logs/runtime*.log
  5. # - /var/www/20230103/crm/runtime/logs/runtime*.log
  6. # - /var/www/20230103/console/runtime/logs/runtime*.log
  7. # - /var/www/20230103/pc/runtime/logs/runtime*.log
  8. encoding: utf8
  9. scan_frequency: 5s
  10. fields:
  11. app_id: runtime
  12. app_type: official
  13. app_env: yf
  14. log_type: all
  15. data_type: raw
  16. fields_under_root: true
  17. - type: log
  18. paths:
  19. # - /var/www/prod/api/runtime/logs/runtime*.log
  20. # - /var/www/prod/app/runtime/logs/runtime*.log
  21. # - /var/www/prod/crm/runtime/logs/runtime*.log
  22. # - /var/www/prod/console/runtime/logs/runtime*.log
  23. # - /var/www/prod/pc/runtime/logs/runtime*.log
  24. encoding: utf8
  25. scan_frequency: 5s
  26. fields:
  27. app_id: runtime
  28. app_type: official
  29. app_env: prod
  30. log_type: all
  31. data_type: raw
  32. fields_under_root: true

5. Request请求日志

filebeat/conf/inputs.d/downstream.yml
  1. - type: log
  2. paths:
  3. - /var/www/prod/api/runtime/logs/downstream*.log
  4. - /var/www/prod/app/runtime/logs/downstream*.log
  5. - /var/www/prod/crm/runtime/logs/downstream*.log
  6. - /var/www/prod/console/runtime/logs/downstream*.log
  7. - /var/www/prod/pc/runtime/logs/downstream*.log
  8. encoding: utf8
  9. scan_frequency: 5s
  10. fields:
  11. app_id: downstream
  12. app_type: official
  13. app_env: prod
  14. log_type: all
  15. data_type: json
  16. fields_under_root: true

6. php-fpm

filebeat/conf/inputs.d/php-fpm.yml
  1. - type: log
  2. paths:
  3. - /usr/local/php/log/fpm_error.www.log
  4. encoding: utf8
  5. scan_frequency: 5s
  6. fields:
  7. app_id: phpfpm
  8. app_type: official
  9. app_env: all
  10. log_type: error
  11. data_type: raw
  12. fields_under_root: true

 日志格式示例

Nginx格式设置

  1. # 日志格式
  2. log_format json '{"@timestamp":"$time_iso8601",'
  3. '"host":"$server_addr",'
  4. '"clientip":"$remote_addr",'
  5. '"size":$body_bytes_sent,'
  6. '"responsetime":$request_time,'
  7. '"upstreamtime":"$upstream_response_time",'
  8. '"upstreamhost":"$upstream_addr",'
  9. '"http_host":"$host",'
  10. '"url":"$request",'
  11. '"referer":"$http_referer",'
  12. '"agent":"$http_user_agent",'
  13. '"http_x_forwarded_for":"$http_x_forwarded_for",'
  14. '"request_body":"-",'
  15. '"http_cookie":"-",'
  16. '"http_user_agent":"$http_user_agent",'
  17. '"status":"$status"}';
  18. # 示例
  19. {"@timestamp":"2022-03-30T15:48:50+08:00","host":"192.168.2.22","clientip":"100.125.68.198","size":1552,"responsetime":0.005,"upstreamtime":"0.005","upstreamhost":"127.0.0.1:9000","http_host":"passport.xiaofenglang.com","url":"POST /passport/prod/selectUser HTTP/1.1","referer":"-","agent":"-","http_x_forwarded_for":"10.15.1.84","request_body":"-","http_cookie":"-","http_user_agent":"-","status":"200"}

总结:

首先配置修改设定 filebeat 配置,看手机哪些日志,filebeat 会把收集到日志推送到 kafka内,最后入ElasticSearch内进行存储。

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/不正经/article/detail/706268
推荐阅读
相关标签
  

闽ICP备14008679号