SpringBoot集成系统监控和告警工具包prometheus

prometheus以开源软件的形式进行研发的系统监控和告警工具包
Grafana 是一个监控仪表系统，它是由 Grafana Labs 公司开源的的一个系统监测工具，只需要提供需要监控的数据，它就可以帮助生成各种可视化仪表，同时它还有报警功能，可以在系统出现问题时发出通知。

环境搭建：

这里采用docker-compose搭建测试环境，具体配置如下

docker-compose-prometheus.yml

# 镜像版本请自行选择 https://hub.docker.com/search?q=&type=image
version: "3"

# 网桥 -> 方便相互通讯
networks:
  prometheus:
    ipam:
      driver: default
      config:
        - subnet: "172.22.0.0/24"

services:
  # 开源的系统监控和报警系统
  prometheus:
    image: registry.cn-hangzhou.aliyuncs.com/zhengqing/prometheus:v2.34.0             # 原镜像`prom/prometheus:v2.34.0`
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    command: "--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus"
    ports:
      - "9090:9090"
    depends_on:
      - node-exporter
    networks:
      prometheus:
        ipv4_address: 172.22.0.11

  # 采集服务器层面的运行指标
  node-exporter:
    image: registry.cn-hangzhou.aliyuncs.com/zhengqing/node-exporter:v1.3.1           # 原镜像`prom/node-exporter:v1.3.1`
    container_name: prometheus-node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    networks:
      prometheus:
        ipv4_address: 172.22.0.22

  # 用于UI展示
  # https://grafana.com/docs/grafana/latest/installation/docker
  grafana:
    image: registry.cn-hangzhou.aliyuncs.com/zhengqing/grafana:8.0.0               # 原镜像`grafana/grafana:8.0.0`
    container_name: prometheus-grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - "./prometheus/grafana/grafana.ini:/etc/grafana/grafana.ini" # 邮箱配置
#      - "./prometheus/grafana/grafana-storage:/var/lib/grafana"
#      - "./prometheus/grafana/public:/usr/share/grafana/public" # 这里面可处理汉化包 可参考 https://github.com/WangHL0927/grafana-chinese
#      - "./prometheus/grafana/conf:/usr/share/grafana/conf"
#      - "./prometheus/grafana/log:/var/log/grafana"
#      - "/etc/localtime:/etc/localtime"
    environment:
      GF_EXPLORE_ENABLED: "true"
      GF_SECURITY_ADMIN_PASSWORD: "admin"
      GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource,alexanderzobnin-zabbix-app"
      # 持久化到mysql数据库
      GF_DATABASE_URL: "mysql://root:root@172.22.0.34:3306/grafana" # TODO 修改
    depends_on:
      - prometheus
      - mysql
    networks:
      prometheus:
        ipv4_address: 172.22.0.33

  # mysql数据库 => 用于grafana持久化数据
  mysql:
    image: registry.cn-hangzhou.aliyuncs.com/zhengqing/mysql:5.7
    container_name: prometheus-mysql
    restart: unless-stopped
    volumes:
      - "./prometheus/mysql5.7/my.cnf:/etc/mysql/my.cnf"
      - "./prometheus/mysql5.7/data:/var/lib/mysql"
      - "./prometheus/mysql5.7/log/mysql/error.log:/var/log/mysql/error.log"
    environment:
      TZ: Asia/Shanghai
      LANG: en_US.UTF-8
      MYSQL_ROOT_PASSWORD: root         # 设置root用户密码
      MYSQL_DATABASE: grafana           # 初始化数据库grafana
    ports:
      - "3306:3306"
    networks:
      prometheus:
        ipv4_address: 172.22.0.34
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

启动测试环境：docker-compose-prometheus.yml 需修改grafana中配置的mysql连接信息 prometheus.yml 自行配置

# 运行
docker-compose -f docker-compose-prometheus.yml -p prometheus up -d
# 查看grafana日志
docker logs -fn10 prometheus-grafana
1
2
3
4

grafana访问地址：http://ip地址:3000 默认登录账号密码：admin/admin

prometheus访问地址: http://ip地址:9090

exporter访问地址: http://ip地址:9100/metrics

一、依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>springboot-demo</artifactId>
        <groupId>com.et</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>prometheus</artifactId>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>
    <dependencies>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-autoconfigure</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-core</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>

    </dependencies>
</project>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

二、配置文件和启动类

server.port=8088
spring.application.name=springboot2-prometheus
management.endpoints.web.exposure.include=*
management.metrics.tags.application=${spring.application.name}
1
2
3
4

@SpringBootApplication
public class DemoApplication {

   public static void main(String[] args) {
      SpringApplication.run(DemoApplication.class, args);
   }
}
1
2
3
4
5
6
7

三、PrometheusCustomMonitor类

@Component
public class PrometheusCustomMonitor {

    private Counter requestErrorCount;
    private Counter orderCount;
    private DistributionSummary amountSum;
    private AtomicInteger failCaseNum;

    private final MeterRegistry registry;

    @Autowired
    public PrometheusCustomMonitor(MeterRegistry registry) {
        this.registry = registry;
    }

    @PostConstruct
    private void init() {
        requestErrorCount = registry.counter("requests_error_total", "status", "error");
        orderCount = registry.counter("order_request_count", "order", "test-svc");
        amountSum = registry.summary("order_amount_sum", "orderAmount", "test-svc");
        failCaseNum = registry.gauge("fail_case_num", new AtomicInteger(0));
    }

    public Counter getRequestErrorCount() {
        return requestErrorCount;
    }

    public Counter getOrderCount() {
        return orderCount;
    }

    public DistributionSummary getAmountSum() {
        return amountSum;
    }

    public AtomicInteger getFailCaseNum() {
        return failCaseNum;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

四、GlobalExceptionHandler类

@ControllerAdvice
public class GlobalExceptionHandler {
    @Resource
    private PrometheusCustomMonitor monitor;

    @ResponseBody
    @ExceptionHandler(value = Exception.class)
    public String handle(Exception e) {
        monitor.getRequestErrorCount().increment();
        return "error, message: " + e.getMessage();
    }
}
1
2
3
4
5
6
7
8
9
10
11
12

五、TestController

@RestController
public class TestController {
    @Resource
    private PrometheusCustomMonitor monitor;

    @RequestMapping("/order")
    public String order(@RequestParam(defaultValue = "0") String flag) throws Exception {
        // 统计下单次数
        monitor.getOrderCount().increment();
        if ("1".equals(flag)) {
            throw new Exception("出错啦");
        }
        Random random = new Random();
        int amount = random.nextInt(100);
        // 统计金额
        monitor.getAmountSum().record(amount);

        monitor.getFailCaseNum().set(amount);
        return "下单成功, 金额: " + amount;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

测试：

启动springboot服务，，访问http://localhost:8089/order和http://localhost:8089/order?flag=1模拟下单成功和失败的情况
然后我们访问http://localhost:8088/actuator/prometheus，可以看到我们自定义指标已经被 /prometheus 端点暴露出来

# TYPE order_request_count_total counter
order_request_count_total{application="springboot2-prometheus",order="test-svc",} 34.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads{application="springboot2-prometheus",} 18.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes{application="springboot2-prometheus",} 7250.0
# HELP order_amount_sum_max  
# TYPE order_amount_sum_max gauge
order_amount_sum_max{application="springboot2-prometheus",orderAmount="test-svc",} 97.0
# HELP order_amount_sum  
# TYPE order_amount_sum summary
order_amount_sum_count{application="springboot2-prometheus",orderAmount="test-svc",} 34.0
order_amount_sum_sum{application="springboot2-prometheus",orderAmount="test-svc",} 1482.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

在grafana上配置自定义的监控参数

访问 http://localhost:3000/login ，初始账号/密码为：admin/admin

①、配置数据源
点击左侧齿轮Configuration中Add Data Source，会看到如下界面：

选择Prometheus 当做数据源，这里我们就配置一下Prometheus 的访问地址，点击 Save & Test

②、创建按监控Dashboard
点击导航栏上的 + 按钮，并点击Dashboard，将会看到类似如下的界面

点击+ Add new panel

最终效果：