当前位置:   article > 正文

prometheus 监控告警系统(对接飞书告警)_prometheus-operator 飞书告警

prometheus-operator 飞书告警

Prometheus 是一套开源的系统监控报警框架,非常适合大规模集群的监控。它也是第二个加入CNCF的项目,受欢迎度仅次于 Kubernetes 的项目。本文讲解完整prometheus 监控和告警服务的搭建。
prometheus 监控是当下主流监控系统,它是多个服务组合使用的体系。整体架构预览如下:
image.png

本篇教程监控系统搭建,包括的服务有:

  1. prometheus 监控的主体,负责数据汇总,保存,监控数据,产生告警信息
  2. exporter 监控的采集端,负责数据采集
  3. grafana 数据可视化,负责以丰富的页面展示采集到的数据
  4. alertmanager 告警管理,负责告警信息处理,包括告警周期,消息优先级等
  5. prometheusAlert 告警的具体发送端,负责配置告警模板,发出告警信息

除了监控采集节点,其他服务均通过docker-compose部署。部署系统信息:

  1. 系统:ubuntu20.04
  2. 服务器IP:172.16.9.124
  3. docker版本:20.10.21
  4. docker-compose版本:1.29.2
  5. 配置文件路径:/root/prometheus

部署prometheus

prometheus主要负责数据采集和存储,提供PromQL查询语言的支持。部署prometheus分为两个步骤:

  1. 准备配置文件
  2. 启动prometheus

准备配置文件

整个体系的配置文件在/root/prometheus,首先新建prometheus服务的配置文件路径 /root/prometheus/prometheus,并在这个目录下新建:

  • config 用于放置服务主要配置文件 prometheus.yml
  • data 用于放置服务的数据库文件
root@ubuntu-System-Product-Name:~/prometheus# tree . -L 3
.
├── docker-compose.yaml
└── prometheus
    ├── config
    │   └── prometheus.yml
    └── data
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

新建prometheus.yml,prometheus服务的主配置文件

global:
  scrape_interval:     30s # 每30s采集一次数据
  evaluation_interval: 30s # 每30s做一次告警检测


scrape_configs:
  # 配置prometheus服务本身
  - job_name: prometheus
    static_configs:
      - targets: ['172.16.9.124:9090']
        labels:
          instance: prometheus

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13

修改 data 目录的文件权限,让容器有权限在data目录里生成数据相关数据

chmod 777 data
  • 1

创建 docker-compse.yml

version: '3'
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    ports:
      - "9090:9090"
    volumes:
      - /root/prometheus/prometheus/config:/etc/prometheus
      - /root/prometheus/prometheus/data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15

参数说明:
command:

  • –config.file=/etc/prometheus/prometheus.yml   指定使用的配置文件
  • –storage.tsdb.path=/prometheus    指定时序数据库的路径
  • –web.enable-lifecycle 支持配置热加载

volumes:

  • /root/prometheus/prometheus/config:/etc/prometheus   映射配置文件所在目录
  • /root/prometheus/prometheus/data:/prometheus   映射数据库路径参数

启动prometheus

启动 docker-compse docker-compose up -d
查看日志:

root@ubuntu-System-Product-Name:~/prometheus# docker ps
CONTAINER ID   IMAGE                                           COMMAND                  CREATED         STATUS                 PORTS                                                        NAMES
776772d69b20   prom/prometheus                                 "/bin/prometheus --c…"   5 minutes ago   Up 5 minutes           0.0.0.0:9090->9090/tcp, :::9090->9090/tcp                    prometheus

  • 1
  • 2
  • 3
  • 4

查看容器的日志:
docker logs -f 776

ts=2023-12-25T10:21:17.560Z caller=main.go:478 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2023-12-25T10:21:17.560Z caller=main.go:515 level=info msg="Starting prometheus" version="(version=2.32.1, branch=HEAD, revision=41f1a8125e664985dd30674e5bdf6b683eff5d32)"
ts=2023-12-25T10:21:17.561Z caller=main.go:520 level=info build_context="(go=go1.17.5, user=root@54b6dbd48b97, date=20211217-22:08:06)"
ts=2023-12-25T10:21:17.561Z caller=main.go:521 level=info host_details="(Linux 5.15.0-56-generic #62~20.04.1-Ubuntu SMP Tue Nov 22 21:24:20 UTC 2022 x86_64 776772d69b20 (none))"
ts=2023-12-25T10:21:17.561Z caller=main.go:522 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2023-12-25T10:21:17.561Z caller=main.go:523 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-12-25T10:21:17.562Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2023-12-25T10:21:17.562Z caller=main.go:924 level=info msg="Starting TSDB ..."
ts=2023-12-25T10:21:17.562Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2023-12-25T10:21:17.564Z caller=head.go:488 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2023-12-25T10:21:17.564Z caller=head.go:522 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.305µs
ts=2023-12-25T10:21:17.564Z caller=head.go:528 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2023-12-25T10:21:17.564Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
ts=2023-12-25T10:21:17.564Z caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
ts=2023-12-25T10:21:17.564Z caller=head.go:605 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=14.305µs wal_replay_duration=301.534µs total_replay_duration=327.342µs
ts=2023-12-25T10:21:17.565Z caller=main.go:945 level=info fs_type=EXT4_SUPER_MAGIC
ts=2023-12-25T10:21:17.565Z caller=main.go:948 level=info msg="TSDB started"
ts=2023-12-25T10:21:17.565Z caller=main.go:1129 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2023-12-25T10:21:17.565Z caller=main.go:1166 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=217.62µs db_storage=555ns remote_storage=860ns web_handler=182ns query_engine=371ns scrape=90.382µs scrape_sd=10.238µs notify=450ns notify_sd=788ns rules=737ns
ts=2023-12-25T10:21:17.565Z caller=main.go:897 level=info msg="Server is ready to receive web requests."

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21

日志很重要!日志很重要!日志很重要!声明:本文内容由网友自发贡献,转载请注明出处:【wpsshop】

推荐阅读
相关标签