prometheus

 一、认识普罗米修斯

时序数据库介绍

数据库分类 :

关系型 mysql,oracle,sql server,sybase,db2,access 等

非关系型 (nosql)

key-value memcache redis etcd 文档型 mongodb elasticsearch

列式 hbase

时序 prometheus

图形数据库 Neo4j

时间序列数据 (TimeSeries Data) : 按照时间顺序记录系统、设备状态变化的数据被称为时序数据 .

时序数据主要的特点：

数据带有时间属性，且数据量随着时间递增

大都为插入操作较多且无更新的需求，插入数据多，每秒钟插入数据可到达千万甚至是上亿条

分析过去时序数据可以做成多纬度报表，揭示其趋势性、规律性、异常性

分析时序数据趋势可以做大数据分析，机器学习，实现预测和预警

能够按照条件筛选数据 , 也可以按照时间范围统计 , 聚合 , 展示数据

常见应用场景 :

无人驾驶车辆运行中要记录的经度，纬度，速度，方向，旁边物体的距离等等。每时每刻都要将数

据记录下来做分析。

某一个地区的各车辆的行驶轨迹数据

传统证券行业实时交易数据

实时运维监控数据等

prometheus 主要特性

Prometheus 的主要特性有 :

1. 多维度数据模型

2. 灵活的查询语言

3. 不依赖分布式存储，单个服务器节点是自主的

4. 以 HTTP 方式，通过 pull 模型拉去时间序列数据

5. 也可以通过中间网关支持 push 模型

6. 通过服务发现或者静态配置 , 来发现目标服务对象

7. 支持多种多样的图表和界面展示 pormetheus 原理架构图

二、 prometheus 监控

 

主机	IP
server.example.com	192.168.35.142
agent1.example.com	192.168.35.143

//各自配置好主机名,两台都互相绑定IP与主机名
[root@server ~]# vim /etc/hosts
192.168.35.142 server.example.com       server
192.168.35.143 agent1.example.com       agent1
 
//解压安装prometheus
[root@server ~]# yum -y install lrzsz tar
[root@server ~]# rz -E
rz waiting to receive.
[root@server ~]# ls
anaconda-ks.cfg  prometheus-2.54.0.linux-amd64.tar.gz
[root@server ~]# tar -zxvf prometheus-2.54.0.linux-amd64.tar.gz -C /usr/local/
 
//配置文件说明
[root@server ~]# cd /usr/local/
[root@server local]# ls
bin  games    lib    libexec                        sbin   src
etc  include  lib64  prometheus-2.54.0.linux-amd64  share
[root@server local]# mv prometheus-2.54.0.linux-amd64/ prometheus
[root@server local]# ls
bin  games    lib    libexec     sbin   src
etc  include  lib64  prometheus  share
[root@server local]# cd prometheus/
[root@server prometheus]# ls
console_libraries  LICENSE  prometheus      promtool
consoles           NOTICE   prometheus.yml        
[root@server prometheus]# egrep -n : /usr/local/prometheus/prometheus.yml | awk -F '#' '{print $1}'
2:global: //全局配置段
3: scrape_interval: 15s //每15s抓取(采集)数据一次
4: evaluation_interval: //15s 每15秒计算一次规则
8:alerting: Alertmanager  //报警相关
9: alertmanagers:
10: - static_configs:
11: - targets:
12:
15:rule_files:  //规则文件列表
19:
21:scrape_configs:   //抓取的配置文件(也就是监控的实例)
23: - job_name: 'prometheus'    //监控的实例名称
28: static_configs:
29: - targets: ['localhost:9090']   //监控的实例IP与端口,在这里为监控服务器本身
 
//启动prometheus，放入后台运行
[root@server prometheus]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
[root@server prometheus]# ss -anlt
State  Recv-Q Send-Q  Local Address:Port    Peer Address:Port Process 
LISTEN 0      5           127.0.0.1:25151        0.0.0.0:*            
LISTEN 0      128           0.0.0.0:22           0.0.0.0:*            
LISTEN 0      5             0.0.0.0:873          0.0.0.0:*            
LISTEN 0      511                 *:443                *:*            
LISTEN 0      511                 *:80                 *:*            
LISTEN 0      128              [::]:22              [::]:*            
LISTEN 0      4096                *:9090               *:*            
LISTEN 0      5                [::]:873             [::]:*            

通过http://服务器IP:9090/metrics可以查看到监控的数据  

//在远程linux主机(被监控端agent1)上安装node_exporter组件
[root@agent1 ~]# rz -E
rz waiting to receive.
[root@agent1 ~]# tar -zxcf node_exporter-1.8.2.linux-amd64.tar.gz -C /usr/local/
 
//修改名字
[root@agent1 ~]# cd /usr/local/
[root@agent1 local]# ls
bin  games    lib    libexec                          sbin   src
etc  include  lib64  node_exporter-1.8.2.linux-amd64  share
[root@agent1 local]# mv node_exporter-1.8.2.linux-amd64/ node_exporter[root@agent1 local]# ls
bin  games    lib    libexec        sbin   src
etc  include  lib64  node_exporter  share
[root@agent1 local]# cd node_exporter/
[root@agent1 node_exporter]# ls
LICENSE  node_exporter  NOTICE
 
//启动node_exporter，并放入后台运行
[root@agent1 node_exporter]# nohup /usr/local/node_exporter/node_exporter &
[1] 831
[root@agent1 node_exporter]# nohup: ignoring input and appending output to 'nohup.out'
[root@agent1 node_exporter]# ss -ant
State  Recv-Q Send-Q  Local Address:Port   Peer Address:Port  Process 
LISTEN 0      128           0.0.0.0:22          0.0.0.0:*             
ESTAB  0      0      192.168.35.143:22     192.168.35.1:57584         
LISTEN 0      4096                *:9100              *:*             
LISTEN 0      128              [::]:22             [::]:*             

浏览器访问http://被监控端IP:9100/metrics就可以查看到node_exporter在被监控端收集的

//修改配置文件，最后加上这三行，取一个job名称来代表被监控的机器
[root@server prometheus]# vim prometheus.yml  
- job_name: "agent1"    
    static_configs:
      - targets: ["192.168.35.143:9100"] //这里改成被监控机器的IP，后面端口接9100
 
 
//杀掉进程
[root@server prometheus]# ps -ef | grep prometheus
root         942     896  0 15:36 pts/0    00:00:01 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root        1013     896  0 16:13 pts/0    00:00:00 grep --color=autoprometheus
[root@server prometheus]# kill -9 942
[root@server prometheus]# ps -ef | grep prometheus
root        1017     896  0 16:14 pts/0    00:00:00 grep --color=autoprometheus
 
//重启服务
[root@server prometheus]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
[root@server prometheus]# ps -ef | grep prometheus
root        1018     896  0 16:14 pts/0    00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root        1025     896  0 16:14 pts/0    00:00:00 grep --color=autoprometheus

 回到web管理界面 --》点Status --》点Targets --》可以看到多了一台监控目标

//修改配置文件，最后加上这三行，取一个job名称来代表被监控的机器
[root@server prometheus]# vim prometheus.yml
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "agent1"
    static_configs:
      - targets: ["192.168.35.143:9100"]
  - job_name: "server"
    static_configs:
      - targets: ["192.168.35.142:9100"]
 
// 杀掉进程
[root@server prometheus]# ps -ef | grep prometheus
root        1018     896  0 16:14 pts/0    00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root        1118     896  0 16:33 pts/0    00:00:00 grep --color=autoprometheus
[root@server prometheus]# kill -9 1018
[root@server prometheus]# ps -ef | grep prometheus
root        1150     896  0 16:58 pts/0    00:00:00 grep --color=autoprometheus
 
//重启服务
[root@server prometheus]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
[root@server prometheus]# ps -ef | grep prometheus
root        1151     896  0 16:59 pts/0    00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root        1174     896  0 17:06 pts/0    00:00:00 grep --color=autoprometheus

 回到web管理界面 --》点Status --》点Targets --》可以看到多了一台监控目标

//agent1.example.com
//解压安装mysql数据包
[root@agent1 ~]# rz -E
rz waiting to receive.
[root@agent1 ~]# ls
anaconda-ks.cfg
mysqld_exporter-0.15.1.linux-amd64.tar.gz
node_exporter-1.8.2.linux-amd64.tar.gz
[root@agent1 ~]# tar -zxvf mysqld_exporter-0.15.1.linux-amd64.tar.gz -C /usr/local/
mysqld_exporter-0.15.1.linux-amd64/
mysqld_exporter-0.15.1.linux-amd64/LICENSE
mysqld_exporter-0.15.1.linux-amd64/mysqld_exporter
mysqld_exporter-0.15.1.linux-amd64/NOTICE
 
//修改名字
[root@agent1 ~]# cd /usr/local/
[root@agent1 local]# ls
bin    include  libexec                             sbin
etc    lib      mysqld_exporter-0.15.1.linux-amd64  share
games  lib64    node_exporter                       src
[root@agent1 local]# mv mysqld_exporter-0.15.1.linux-amd64/ mysqld_exporter
 
//安装mysql依赖包
[root@agent1 local]# yum -y install mariadb-server mariadb
[root@agent1 local]# systemctl restart mariadb.service 
[root@agent1 local]# systemctl enable mariadb.service 
Created symlink /etc/systemd/system/mysql.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/mysqld.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /usr/lib/systemd/system/mariadb.service.
 
//启动MySQL并授权
[root@agent1 local]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.5.22-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]> 
MariaDB [(none)]> grant all ON *.* to 'mysql_monitor'@'localhost' identified by
    -> 'redhat';
Query OK, 0 rows affected (0.001 sec)
 
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> exit
Bye
//创建连接mariadb配置文件
[root@agent1 local]# vim /usr/local/mysqld_exporter/.my.cnf
[client]
user=mysql_monitor
password=redhat
 
// 启动mysqld_exporter并验证9104端口
[root@agent1 local]# nohup /usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/.my.cnf &
[2] 3005
[root@agent1 local]# nohup: ignoring input and appending output to 'nohup.out'
[root@agent1 local]# ss -antl
State  Recv-Q Send-Q   Local Address:Port   Peer Address:Port Process 
LISTEN 0      128            0.0.0.0:22          0.0.0.0:*            
LISTEN 0      4096                 *:9100              *:*            
LISTEN 0      4096                 *:9104              *:*            
LISTEN 0      128               [::]:22             [::]:*            
 
//server.example.com
//修改配置文件，最后加上这三行，取一个job名称来代表被监控的机器
[root@server prometheus]# vim prometheus.yml
     static_configs:
      - targets: ["localhost:9090"]
  - job_name: "agent1"
    static_configs:
      - targets: ["192.168.35.143:9100"]
  - job_name: "server"
    static_configs:
      - targets: ["192.168.35.142:9100"]
  - job_name: "agent1-mariadb"
    static_configs:
      - targets: ["192.168.35.143:9104"]
 
//杀掉进程
[root@server prometheus]# ps -ef | grep prometheus
root        1151     896  0 16:59 pts/0    00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root        1206     896  0 17:27 pts/0    00:00:00 grep --color=autoprometheus
[root@server prometheus]# kill -9 1151
 
//重启服务 
[root@server prometheus]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
[root@server prometheus]# ps -ef | grep prometheus
root        1211     896  1 17:28 pts/0    00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root        1218     896  0 17:28 pts/0    00:00:00 grep --color=autoprometheus

回到web管理界面 --》点Status --》点Targets --》可以看到多了一台监控目标