Prometheus实战篇:Prometheus监控redis

准备环境

docker-compose安装redis

docker-compose.yaml

version: '3'
services:
   redis:
 	image:redis:5
 	container_name: redis
 	command: redis-server --requirepass 123456 --maxmemory 512mb
 	 restart: always
 	 volumes:
 	 	- /data/redis/data: /data
 	 port:
 	 - "6379:6379"
1
2
3
4
5
6
7
8
9
10
11

docker-compose up -d
1

监控redis

docker安装exporter

docker直接运行

docker run -d --name redis_exporter -p 9121:9121 oliver006/redis_exporter --redis.addr redis://localhost:6379 --redis.password '123456'
1

docker-compose方式

cat >docker-compose.yaml <<FOF
version: '3.3'
services:
 redis_exproter:
 	image:oliver006/redis-exporter
 	container_name: redis_exporter
 	restart: always
 	environment:
 		REDIS_ADDR:"localhost:6379"
 		REDIS_PASSWORD: 123456
 	port:
 	 - "9113:9113"
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13

启动

docker-compose up -d
1

检查

查看正在运行的容器
docker ps

或者:

查看redis_exporter容器的运行日志
docker logs -f redis_exporter
1
2
3
4
5
6
7

参数解释

metrics地址

安装好Exporter后会暴露一个/metrics结尾的服务

Prometheus配置

配置Prometheus去采集(拉取)nginx_exporter的监控样本数据

cd /data/docker-prometheus

# 在scrapc_configs(搜刮配置):下面增加如下配置:
cat >prometheus/prometheus.yml <<FOF
 - job_name: 'redis_exporter'
   static_configs:
   - targets: ['localhost:9121']
   	 labels:
   	 	instance: test服务器 
EOF
1
2
3
4
5
6
7
8
9
10

重新加载配置

curl -x POST http://localhost:9090/-/reload
1

检查

常用的redis监控指标

redis_up												服务器是否在线
redis_uptime_in_seconds									运行时长,单位s
rate(redis_cpu_sys_seconds_total[1m])+rate(redis_cpu_user_seconds_total[1m])		占用CPU核数
redis_memory_used_bytes									占用内存量
redis_memory_max_bytes									限制的最大内存,如果没限制则为0
delta(redis_net_input_bytes_total[1m])	 				网络接受的bytes
delta(redis_net_output_bytes_total[1m])	 				网络发送的bytes


redis_connected_clients									客户端连接数
redis_connected_clients / redis_config_maxclients
	连接数使用率
redis_rejected_connections_total
	拒绝的客户端连接数
redis_connected_slaves
	slave连接数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

触发器配置

由于之前的触发器是全部写在了一个yml里面就是alert.yam,这样随着后面配置的触发器越来越多最终会变得难以维护.这里我们让它去读rules目录下所有的yml文件即可

Prometheus配置

rule_files:
	- "alert.yml"
	- "rules/*.yml"
1
2
3

配置redis触发器

因为是单机所以未配置集群的触发器

cat >prometheus/redis.yml <<FOF
groups:
- name: redis
	rules:
    - alert: RedisDown
      expr: redis_up ==0
      for: 0m
      labels:
        severity: critical
      annotations:
          summary: "redis Down,实例:{{$labels.instance }}"
          description: "Redis实例 is down"
    - alert: RedisMissingBackup
      expr: time() - redis_rdb_last_save_timestamp_seconds > 60 * 60 * 24
      for: 0m
      labels:
        severity: critical
      annotations:
          summary: "redis 备份丢失,实例:{{$labels.instance }}"
          description: "Redis 24小时未备份"
    - alert: RedisOutOfConfigredMaxmemory
      expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 >90
      for: 2m
      labels:
        severity: warning
      annotations:
          summary: "redis超出配置的最大内存,实例:{{$labels.instance }}"
          description: "Redis内存使用超过配置的最大内存的90%"
    - alert: RedisTooManyConnections
      expr: redis_connected_clients > 100
      for: 2m
      labels:
        severity: warning
      annotations:
          summary: "redis连接数过多,实例:{{$labels.instance }}"
          description: "Redis当前连接数为:{{ $value }}"
    - alert: RedisNotEnoughConnections
      expr: redis_connected_clients < 1
      for: 2m
      labels:
        severity: warning
      annotations:
          summary: "redis没有足够的连接,实例:{{$labels.instance }}"
          description: "Redis当前连接数为:{{ $value }}"
    - alert: RedisRejectedConnections
      expr: increase(redis_rejected_connections_total[1m]) > 0
      for: 0m
      labels:
        severity: critical
      annotations:
          summary: "redis有拒绝连接,实例:{{$labels.instance }}"
          description: "与Redis的某些连接被拒绝:{{ $value }}"
  		
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

一定记住这里需要仔细校对yaml语法,最好是能去在线验证yaml语法的网站上看看.yaml语法还是比较严格的一点缩进错误都不能有

检查配置

docker exec -it prometheus promtool check config /etc/prometheus/prometheus.yml
1

重新加载配置

curl -x POST http://localhost:9090/-/reload
1

检查

http://localhost:9090/alerts?search=

或:

http://localhost:9090/rules

dashboard

grafana展示Prometheus从redis_exporter收集到的数据

id :11835