轻量级日志管理工具Loki_loki日志导出

优点 ：
Loki 的架构非常简单，使用了和 Prometheus 一样的标签来作为索引，通过这些标签既可以查询日志的内容也可以查询到监控的数据，不但减少了两种查询之间的切换成本，也极大地降低了日志索引的存储。
与 ELK 相比，消耗的成本更低，具有成本效益。
在日志的收集以及可视化上可以连用 Grafana，实现在日志上的筛选以及查看上下行的功能。
缺点 ：
技术比较新颖，相对应的论坛不是非常活跃。
功能单一，只针对日志的查看，筛选有好的表现，对于数据的处理以及清洗没有 ELK 强大，同时与 ELK 相比，对于后期，ELK 可以连用各种技术进行日志的大数据处理，但是 loki 不行。

***Loki 组成***
Loki 是主服务器，负责存储日志和处理查询。
Promtail 是代理，负责收集日志并将其发送给 Loki 。
Grafana 用于 UI 展示。

***组件说明***
Promtail 作为采集器，类比filebeat
loki 相当于服务端，类比es
loki进程包含四种角色
    querier 查询器
    inester 日志存储器
    query-frontend 前置查询器
    distributor 写入分发器
可以通过 loki 二进制的 -target 参数指定运行角色
1
2
3
4
5
6
7
8
9
10
11
12
13
14

wget  https://github.com/grafana/loki/releases/download/v2.2.1/loki-linux-amd64.zip
wget https://github.com/grafana/loki/releases/download/v2.2.1/promtail-linux-amd64.zip
1
2

安装 Promtail

$ mkdir /opt/app/{promtail,loki} -pv
 
 
# promtail配置文件
$ cat <<EOF> /opt/app/promtail/promtail.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0
 
 
positions:
  filename: /var/log/positions.yaml # This location needs to be writeable by promtail.
 
 
client:
  url: http://localhost:3100/loki/api/v1/push
 
 
scrape_configs:
 - job_name: system
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: varlogs
      host: yourhost
      __path__: /var/log/*.log
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

安装loki

$ mkdir /opt/app/{promtail,loki} -pv
 
 
# promtail配置文件
$ cat <<EOF> /opt/app/loki/loki.yaml
auth_enabled: false
 
 
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
 
 
ingester:
  wal:
    enabled: true
    dir: /opt/app/loki/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled
 
 
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
 
 
storage_config:
  boltdb_shipper:
    active_index_directory: /opt/app/loki/boltdb-shipper-active
    cache_location: /opt/app/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /opt/app/loki/chunks
 
 
compactor:
  working_directory: /opt/app/loki/boltdb-shipper-compactor
  shared_store: filesystem
 
 
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
 
 
chunk_store_config:
  max_look_back_period: 0s
 
 
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
 
 
ruler:
  storage:
    type: local
    local:
      directory: /opt/app/loki/rules
  rule_path: /opt/app/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

nohup ./promtail-linux-amd64 -config.file=/opt/app/promtail/promtail.yaml &
nohup ./loki-linux-amd64 -config.file=/opt/app/loki/loki.yaml &

多个索引标签

scrape_configs:
 - job_name: system
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: syslog
      __path__: /var/log/syslog
 - job_name: system
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: apache
      __path__: /var/log/apache.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17