filebeat收集日志_filebeat mutate

ES集群

curl -XGET 'http://localhost:9200/_nodes/procese?human&pretty'                       #查看集群名称
curl -XGET 'http://localhost:9200/_nodes/_all/info/jvm,process?human&pretty'         #查看集群所有信息
curl -XGET 'http://localhost:9200/_cat/nodes?human&pretty'                           #查看当前节点个数
curl -XPUT 'localhost:9200/_cat/indices?pretty'                                      #查看集群健康状态
1
2
3
4

1.
2个节点,master设置为2的时候,一台出现故障导致集群不可用
解决方案:
把还存活的节点的配置文件集群选举相关的选项注释掉或者改成1
discovery.zen.minimum_master_nodes: 1
重启服务

2.一开始就把
discovery.zen.minimum_master_nodes: 1这里设置为1
当一台发生故障时，两个节点独立，数据不一致会导致查询结果不一致
解决办法：找出不一致的数据,清空一个节点,以另一个节点的数据为准，然后手动插入修改后的数据

因此，最后觉得设置discovery.zen.minimum_master_nodes: 2这里为2比较妥当。

1
2
3
4
5
6
7
8
9
10
11
12
13
14

1.默认数据分配：
5分片
1副本

2.监控状态
1）监控集群健康状态 不是 green
or
2）监控集群节点数量 不是 3
curl -s -XPUT  'http://10.0.0.51:9200/_cat/nodes?human&pretty' |wc -l 
不满足上面条件其中之一就报警

3.极限损坏：
3节点
最多只能是坏2台节点

4.在动作那里先不要删除，可以先关闭，等过一段不用到再删除。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

nginx收集多个日志

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
   enabled: true
   paths:
     - /var/log/nginx/error.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["error"] 
 
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  indices:
    - index: "nginx_access-%{[agent.version]}-%{+yyyy.MM.}"
      when.contains:
        tags: "access"
    - index: "nginx_error-%{[agent.version]}-%{+yyyy.MM.}"
      when.contains:
        tags: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enable: false
setup.template.overwrite: true
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

nginx jons格式

https://www.cnblogs.com/sky-cheng/p/11187534.html
1

收集tomcat 并且改为josn格式

vim server.xml
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="tomcat_access_log" suffix=".log"
               pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>
1
2
3
4

filebeat配置文件

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true 
  tags: ["nginx"]   
 
- type: log
  enabled: true 
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
  json.keys_under_root: true
  json.overwrite_keys: true 
  tags: ["tomcat"]  

output.elasticsearch:
  hosts: ["192.168.8.128:9200"]
  indices:
    - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.}"
      when.contains:
        tags: "nginx"
    - index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM.}"
      when.contains:
        tags: "tomcat"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enable: false
setup.template.overwrite: true   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

匹配多行

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true 
  tags: ["nginx"]   
 
- type: log
  enabled: true 
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
  json.keys_under_root: true
  json.overwrite_keys: true 
  tags: ["tomcat"]  

- type: log
  enabled: true 
  paths:
    - /var/log/elasticsearch/my-cluster-2020-08-25.log
  tags: ["es"]
  multiline.pattern: '^\['  #收集java的日志
  multiline.negate: true
  multiline.match: after 
output.elasticsearch:
  hosts: ["192.168.8.128:9200"]
  indices:
    - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.}"
      when.contains:
        tags: "nginx"
    - index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM.}"
      when.contains:
        tags: "tomcat"

    - index: "es-%{[beat.version]}-%{+yyyy.MM.}"
      when.contains:
        tags: "es"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enable: false
setup.template.overwrite: true   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

filebeat 收集多个日志，传给redis
都用一个reids keys存储

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true 
  tags: ["access"]   

- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  json.keys_under_root: true
  json.overwrite_keys: true 
  tags: ["error"]  
 
output.redis:
  hosts: ["192.168.8.129"]
  key: "filebeat"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

logstach收集reids传过来的多个key

input {
            redis {
                host => "192.168.8.129"
                port => "6379"
                db => "0"
                data_type => "list"
                key => "filebeat"
                } 
}

filter{
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output { 
   stdout {}
     if "access" in [tags] {
       elasticsearch {
         hosts => "http://192.168.8.128:9200"
         manage_template => false
         index=> "nginx_access-%{+yyyy.MM.dd}"
      }
   }
     if "error" in [tags] {
       elasticsearch {
         hosts => "http://192.168.8.128:9200"
         manage_template => false
         index => "nginx_error-%{+yyyy.MM.dd}"
      }
    }
} 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34