ELK——监控nginx日志（alert报警）_nginx alert

环境准备

准备两台centos7
配置：2核心2G内存
ip地址：

192.168.153.179：

所需安装服务：

• jdk
• kibana
• elasticsearch

主机名称（方便理解）
即：elasticearsh+kibana

• ek

192.168.153.178:
所需安装服务：

• jdk
• logstash

主机名称：
即logstash：

• log

开始操作

1、安装包上传到/usr/loca/src（我是放在这里，各位学者自愿选择上传路径即可）
ek主机操作：

[root@ek ELK]# ls
elasticsearch-6.6.2.rpm  jdk-8u131-linux-x64_.rpm  kibana-6.6.2-x86_64.rpm
[root@ek ELK]# pwd
/usr/local/src/ELK
1
2
3
4

log主机操作：

[root@log ELK]# ls
jdk-8u131-linux-x64_.rpm  logstash-6.6.0.rpm
[root@log ELK]# ls
jdk-8u131-linux-x64_.rpm  logstash-6.6.0.rpm
[root@log ELK]# pwd
/usr/local/src/ELK
1
2
3
4
5
6

2、关闭防火墙

两台上做相同操作：

[root@ek ELK]# systemctl stop firewalld
[root@ek ELK]# setenforce 0
1
2

3、时间同步
两台上做相同操作：

[root@ek ELK]# ntpdate pool.ntp.org
1

如无此命令：安装如下

[root@ek ELK]# rpm -qa |grep ntpdate
ntpdate-4.2.6p5-28.el7.centos.x86_64
1
2

4、安装jdk

[root@ek ELK]# rpm -ivh jdk-8u131-linux-x64_.rpm 
1

验证：

[root@ek ELK]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
1
2
3
4

5、安装elasticsearch
ek主机上操作：

[root@ek ELK]# rpm -ivh elasticsearch-6.6.2.rpm 
1

配置如下：

[root@ek elasticsearch]# pwd
/etc/elasticsearch
[root@ek elasticsearch]# grep -v "#" elasticsearch.yml 
cluster.name: node
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.153.179
http.port: 9200
1
2
3
4
5
6
7
8
9

运行elasticsearch服务并设置开机自启动：

[root@ek elasticsearch]# systemctl start elasticsearch
[root@ek elasticsearch]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
1
2
3

查看端口并检查服务是否正常运行：
如下视为成功

[root@ek elasticsearch]# ss -nltp|grep java
LISTEN     0      128     ::ffff:192.168.153.179:9200                    :::*                   users:(("java",pid=15248,fd=204))
LISTEN     0      128     ::ffff:192.168.153.179:9300                    :::*                   users:(("java",pid=15248,fd=191))
[root@ek elasticsearch]# ss -nltp|grep java
LISTEN     0      128     ::ffff:192.168.153.179:9200                    :::*                   users:(("java",pid=15248,fd=204))
LISTEN     0      128     ::ffff:192.168.153.179:9300                    :::*                   users:(("java",pid=15248,fd=191))
[root@ek elasticsearch]# tailf /var/log/elasticsearch/node.log 
[2020-09-18T09:27:07,577][INFO ][o.e.g.GatewayService     ] [node-1] recovered [0] indices into cluster_state
[2020-09-18T09:27:08,297][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watches] for index patterns [.watches*]
[2020-09-18T09:27:08,692][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2020-09-18T09:27:08,742][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2020-09-18T09:27:08,816][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2020-09-18T09:27:08,891][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2020-09-18T09:27:08,950][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2020-09-18T09:27:08,999][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2020-09-18T09:27:09,052][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2020-09-18T09:27:09,227][INFO ][o.e.l.LicenseService     ] [node-1] license [bfd054c1-3152-42d9-bb0f-ce904f9e462f] mode [basic] - valid
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

6、安装logstash
log主机操作：

[root@log ELK]# rpm -ivh logstash-6.6.0.rpm
1

7、安装nginx并启动
log主机操作：

使用yum源的方式去安装nginx

[root@log ELK]# yum -y install epel-release
[root@log ELK]# yum -y install nginx
[root@log ELK]# nginx
1
2
3

安装ab测压工具，之后需要使用

[root@log ELK]# yum -y install httpd-tools
1

8、编辑nginx.conf文件和正则
log主机操作：

[root@log ELK]# cat /etc/logstash/conf.d/nginx.conf 
input{
	file{
		path => "/var/log/nginx/access.log"
		type => "nginx-log"
		start_position => "beginning"
	}	
}
filter{
	grok{
		match => {"message" => "%{NGX}"}	
	}
}
output{
	elasticsearch{
		hosts => "192.168.153.179:9200"
		index => "nginx_log-%{+YYYY.MM.dd}"
	}
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

上传正则路径和文件到/usr/local/src下

[root@log src]# pwd
/usr/local/src
[root@log src]# ls
ELK nginx_reguler_log_path.txt nginx_reguler_log.txt
1
2
3
4

将nginx_reguler_log.txt文件内容移动到此目录下即可并改名为nginx

[root@log src]# cat nginx_reguler_log_path.txt 
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nginx
[root@log src]# mv nginx_reguler_log.txt /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nginx
[root@log src]# cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nginx 
NGX %{IPORHOST:client_ip} (%{USER:ident}|- ) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)" %{NUMBER:status} (?:%{NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" "%{GREEDYDATA:agent}"
1
2
3
4
5

9、给/var/log权限
log主机操作：

[root@log conf.d]# chmod -R 777 /var/log
1

10、启动logstash

[root@log src]# systemctl start logstash
1

等待一段时间后监测9600端口是否启动

[root@log src]# ss -nltp|grep 9600
LISTEN     0      50        ::ffff:127.0.0.1:9600                    :::*                   users:(("java",pid=62130,fd=89))
1
2

ab测压

[root@log conf.d]# ab -n10 -c10 http://192.168.153.179/index.html
1

11、安装kibana
es主机操作：

[root@ek ELK三剑客]# yum -y install kibana-6.6.2-x86_64.rpm
1

修改kibana主配置文件：

[root@ek ELK三剑客]# grep -Ev '#|^$' /etc/kibana/kibana.yml 
server.port: 5601
server.host: "192.168.153.179"
elasticsearch.hosts: ["http://192.168.153.179:9200"]
1
2
3
4

• server.port：kibana服务端端口号
• server.host: kibana服务端主机IP
• elasticsearch.hosts： elasticsearch主机IP

命令检测nginx索引

[root@ek ELK三剑客]# curl -X GET http://192.168.153.179:9200/_cat/indices?v
health status index                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1            O38zv0b8RzORzBYO1gFW8Q   1   0          1            0      5.1kb          5.1kb
yellow open   nginx_log-2020.09.18 H-skwNRQRTi5RYQO7aOtAA   5   1         21            0     68.5kb         68.5kb
1
2
3
4

12、访问浏览器查看nginx索引

警告：如果出现此图，可以重新测压

alert环境开始部署

全部在log主机操作：

1 ：安装python3 环境

[root@log alter]# yum -y install gcc gcc-c++ openssl-devel
1

到alert目录下解压python软件包并切换到此目录下编译安装

[root@log alter]# ls
Python-3.6.2.tgz  v0.2.1_elasticalert.tar.gz
[root@log alter]# pwd
/usr/local/src/alter
[root@log alter]# tar xf Python-3.6.2.tgz 
[root@log alter]# cd Python-3.6.2
[root@log Python-3.6.2]# ./configure --prefix=/usr/local/python3 --with-openssl && make && make install
1
2
3
4
5
6
7

2、设置软链接

[root@log Python-3.6.2]# rm -rf /usr/bin/python
[root@log Python-3.6.2]# ln -s /usr/local/python3/bin/python3.6 /usr/bin/python
[root@log Python-3.6.2]# ln -s /usr/local/python3/bin/pip3.6 /usr/bin/pip
1
2
3

3、修复yum命令

[root@log ~]# sed -i 's/python/python2/' /usr/bin/yum 
[root@log ~]# sed -i 's/python/python2/' /usr/libexec/urlgrabber-ext-down 
1
2

4、安装alert 插件

解压改名并安装依赖

[root@log alter]# ls
Python-3.6.2  Python-3.6.2.tgz  v0.2.1_elasticalert.tar.gz
[root@log alter]# pwd
/usr/local/src/alter
[root@log alter]# tar xf v0.2.1_elasticalert.tar.gz 
[root@log alter]# mv elastalert-0.2.1/ /usr/local/elastalert
[root@log alter]# cd /usr/local/elastalert/
[root@log elastalert]# pip install -r requirements.txt
1
2
3
4
5
6
7
8

升个级

[root@log elastalert]# pip install --upgrade pip
1

执行如下命令（生成四个命令）

[root@log elastalert]# python setup.py install
1

创建软链接

[root@log ~]# ln -s /usr/local/python3/bin/elastalert* /usr/bin/
1

直接调用命令即可

lrwxrwxrwx. 1 root root        33 9月  19 12:10 elastalert -> /usr/local/python3/bin/elastalert
lrwxrwxrwx. 1 root root        46 9月  19 12:10 elastalert-create-index -> /usr/local/python3/bin/elastalert-create-index
lrwxrwxrwx. 1 root root        50 9月  19 12:10 elastalert-rule-from-kibana -> /usr/local/python3/bin/elastalert-rule-from-kibana
lrwxrwxrwx. 1 root root        43 9月  19 12:10 elastalert-test-rule -> /usr/local/python3/bin/elastalert-test-rule
1
2
3
4

5、设置elastalert 索引

[root@log ~]# elastalert-create-index 
Enter Elasticsearch host: 192.168.153.179
Enter Elasticsearch port: 9200
Use SSL? t/f: f
Enter optional basic-auth username (or leave blank): 
Enter optional basic-auth password (or leave blank): 
Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request): 
New index name? (Default elastalert_status) 
New alias name? (Default elastalert_alerts) 
Name of existing index to copy? (Default None) 
Traceback (most recent call last):
1
2
3
4
5
6
7
8
9
10
11

• Enter Elasticsearch host: 192.168.153.179 #输入elasticsearch主机IP
• Enter Elasticsearch port: 9200 #输入elasticsearch监听端口
• Use SSL? t/f: f #输入f（表示不启用ssl）
• 之后一路回车即可

6、设置 alert的主配置文件config.yaml

更改名称

[root@log elastalert]# pwd
/usr/local/elastalert
[root@log elastalert]# mv config.yaml.example config.yaml
1
2
3

配置详情

[root@log elastalert]# grep -Ev '#|^$' config.yaml 
rules_folder: example_rules
run_every:
  minutes: 1
buffer_time:
  minutes: 15
es_host: 192.168.153.179
es_port: 9200
writeback_index: elastalert_status
writeback_alias: elastalert_alerts
alert_time_limit:
  days: 2
1
2
3
4
5
6
7
8
9
10
11
12

配置详解
此处非代码块，只是为了清晰展示格式

rules_folder: example_rules # 用来放置 告警规则的
run_every:
  minutes: 1   #设置告警执行的频率（一分钟运行一次！！）
buffer_time:
  minutes: 15  # 设置请求里时间字段的范围（举个例子：15:30-15.45分区间的log信息。）
es_host: 192.168.53.179   # elasticsearch 的主机信息
es_port: 9200   # es的端口信息
writeback_index: elastalert_status  # 创建的index 名称
alert_time_limit:
  days: 2	# 失败重试的时间限制
1
2
3
4
5
6
7
8
9
10

7、设置告警规则

复制一个nginx的yaml文件

[root@log example_rules]# pwd
/usr/local/elastalert/example_rules
[root@log example_rules]# cp example_frequency.yaml nginx_frequency.yaml
1
2
3

配置详情

[root@log example_rules]# grep -Ev '#|^$' nginx_frequency.yaml 
es_host: 192.168.153.179
es_port: 9200
name: nginx frequency rule
type: frequency
index: nginx_log*
num_events: 5
timeframe:
  hours: 1
filter:
- term:
    status: "404"
alert:
- "email"
email:
- "2418993252@qq.com"
smtp_host: smtp.qq.com
smtp_port: 25
smtp_auth_file: /usr/local/elastalert/email_auth.yaml
from_addr: 2418993252@qq.com
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

配置详解
此处非代码块

es_host: 192.168.153.179 # elasticsearch主机信息
es_port: 9200  # elasticsearch监听的端口号
name: nginx frequency rule  # 设置告警规则的名称
type: frequency # 设置告警规则的类型（频率）
index: nginx_log*  # 设置监听的index 名称
num_events: 5  # 设置在限定的时间内，触发的次数
timeframe:
  hours: 1   # 设置限定时间
filter:
  - regexp:
      message: ".*"   #表示message 字段下，只要有内容，并且在1小时内触发了5次就告警！！
alert:
- "email"   # 设置邮件告警

email:
- "347042860@qq.com"
- "1770432354@qq.com"
- "15035993639@163.com"  # 设置接收告警的邮箱地址
smtp_host: smtp.qq.com  # 设置smtp的地址
smtp_port: 25   #设置smtp监听端口号
smtp_auth_file: /usr/local/elastalert/email_auth.yaml  # 设置smtp 验证信息
from_addr: 2418993252@qq.com   # 设置发送邮件的邮箱地址
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

需要写一个文件
此处写自己的邮箱和授权码

[root@log elastalert]# pwd
/usr/local/elastalert
[root@log elastalert]# cat email_auth.yaml 
user: "2418993252@qq.com"
password: "pcojgcyggptsdjjh"
1
2
3
4
5

8、验证邮件是否存在并且可以正常发送 linux下使用自带mail发送邮件（超简单 有意向者可以访问我之前对于mailx命令使用的简单介绍，我便直接发送邮件

[root@log ~]# rpm -qa |grep mailx
1

此处，笔者是没有安装此软件的，安装软件

[root@log ~]# yum -y install mailx
1

发送测试邮件服务配置是否正常

[root@log ~]# echo "yes/no" |mail -s "test" 2418993252@qq.com
1

9、nginx 日志里状态码包含404的则触发告警
非代码块

filter:
- term:
    status: "404"
1
2
3

10、运行alert 服务（开启两个会话口测压测试是否报警）
会话1

[root@log elastalert]# elastalert --config /usr/local/elastalert/config.yaml --rule /usr/local/elastalert/example_rules/nginx_frequency.yaml --verbose
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999755 seconds
1
2
3
4
5

会话2：测压时测压修改为错误测压页面，导致404错误出现

[root@log ~]# ab -n100 -c100 http://192.168.153.178/indasdex.htmla
1

出现如下邮件报警，视为成功

到这里，我们的ELK监控nginx日志加alert报警就大功告成了！之后还会有ELFK相关的博客上线，敬请期待…