ELK日志分析系统_es 适合存储系统日志嘛

一、ELK日志分析系统简介

1、日志服务器的优缺点

优点
提高安全性
集中存放日志
缺点
对日志的分析困难

2、ELK

日志简化分析的管理工具，由Elasticsearch(ES)、Logstash、Kibana三个开源工具组成，官方网站: https://www.elastic.co/products

• ES(nosql非关数据库)：存储功能和索引
• Logstash(收集日志)：到应用服务器上拿取log,并进行格式转换后输出到es中
  通过input功能来收集/采集log
• filter过滤器：格式化数据
• output输出：日志输出到es数据库内
• Kibana(展示工具)：将es内的数据在浏览器展示出来，通过UI界面展示(可以根据自己的需求对日志进行处理，方便查阅读取)

2.1、Logstash管理包含四种工具

• Packetbeat ( 搜集网络流量数据)
• Topbeat(搜集系统、进程和文件系统级别的CPU和内存使用情况等数据)
• Filebeat (搜集文件数据)，相较于Logstash是轻量级工具
• Winlogbeat (搜集Windows事件日志数据)

2.2、日志处理步骤

• Logstash收集AppServer产生的Log,并将log进行集中化管理
• 将日志格式化(Logstash) 并存放到ElasticSearch集群中
• 对格式化后的数据进行索引|和存储( Elasticsearch)
• Kibana则从Es集群中查询数据生成图表，再返回给browsers

3.Elasticsearch的基础核心概念

• 接近实时(NRT)
  elasticsearch是一个接近实时的搜索平台，这意味着，从索引一个文档直到这个文档能够被搜索到有一个轻微的延迟(通常是1秒)

• 集群(cluster)
  一个集群就是由一个或多个节点组织在一起，它们共同持有整个的数据，并一起提供索引和搜索功能。其中一个节点为主节点，这个主节点是可以通过选举产生的，并提供跨节点的联合索引和搜索的功能。集群有一个唯一性标示的名字，默认是elasticsearch
  集群名字很重要，每个节点是基于集群名字加入到其集群中的。因此，确保在不同环境中使用不同的集群名字。一个集群可以只有一个节点。强烈建议在配置elasticsearch时， 配置成集群模式。es 具有集群机制，节点通过集群名称加入到集群中，同时在集群中的节点会有一个自己的唯一 身份标识(自己的名称) .

• 节点(node)
  节点就是一台单一的服务器，是集群的一部分，存储数据并参与集群的索引和搜索功能。像集群一样，节点也是通过名字来标识，默认是在节点启动时随机分配的字符名。当然，你可以自己定义。该名字也很重要，在集群中用于识别服务器对应的节点。

• 索引
  一个索引就是一个拥有积分相似特征的文档的集合，由一个名字来标识（必须全部是小写字母），索引相对于关系型数据库的库

• 类型
  在一个索引种，可以定义一种或多种类型，一个类型是索引的一个逻辑上的分类/分区，其语义完全自定义。类型相对于关系型数据库的表

• 文档
  一个文档是一个可被索引的基础信息单元，文档相对于关系型数据库的列

• 分片和副本
  ①分片的主要原因
  a：水平分割扩展、增大存储量
  b：分布式并行分片操作，提高性能和吞吐量
  ②副本的主要原因
  a： 高可用性，以应对分片或节点故障
  b： qps性能、增大吞吐量，搜索可以并行在所有副本上执行
  ③分片和副本的数量可以在索引创建的时候指定，在索引创建之后，可以动态地改变副本的数量，但是不能改变分片的数量

4.logstash介绍

4.1 主要组件

• Shipper:日志收集者
  负责监控本地日志文件的变化，及时把日志文件的最新内容收集起来。通常，远程代理端(agent) 只需要运行这个组件
• Indexer:日志存储者
  负责接收日志并写入到本地文件
• Broker:日志Hub
  负责连接多个Shipper和多个Indexer
• Searchandstorage
  允许对事件进行搜索和存储
• Web Interface
  基于Web的展示界面

正是由于以上组件在LogStash架构中可独立部署，才提供了更好的集群扩展性

4.2 主机分类

• 代理主机(agent host) :作为事件的传递者(shipper)，将各种日志数据发送至中心主机;只需运行Logstash 代理(agent)程序;
• 中心主机(central host) :可运行包括中间转发器(Broker)、索引器(Indexer) 、搜索和存储器(Search and Storage )

Web界面端“(Web Interface) 在内的各个组件，以实现对日志数据的接收、处理和存储

5.Kibana介绍

Kibana是一个 针对Elasticsearch的开源分析及可视化平台，用来搜索、查看交互存储在Elasticsearch索引中的数据。
使用Kibana，可以通过各种图表进行高级数据分析及展示。Kibana让海量数据更容易理解。它操作简单，基于浏览器的用户界面可以快速创建仪表板(dashboard) 实时显示Elasticsearch查询动态。设置Kibana非常简单。无需编写代码，几分钟内就可
以完成Kibana安装并启动Elasticsearch索引监测。
主要功能:

• Elasticsearch无缝之集成。Kibana架构为Elast icsearch定制， 可以将任何结构化和非结构化数据加入Elast icsearch索引。Kibana还充分利用了Elasticsearch强大的搜索和分析功能。
• 整合你的数据。Kibana能够更好地处理海量数据，并据此创建柱形图、折线图、散点图、直方图、饼图和地图。
• 复杂数据分析。Kibana提升了Elasticsearch分析能力，能够更加智能地分析数据，执行数学转换并且根据要求对数据切
  割分块。
• 让更多团队成员受益。强大的数据库可视化接口让各业务岗位都能够从数据集合受益。
• 接口灵活，分享更容易。使用Kibana可以更加方便地创建、保存、分享数据，并将可视化数据快速交流。
• 配置简单。Kibana的配置和启用非常简单，用户体验非常友好。Kibana自带Web服务器，可以快速启动运行。
• 可视化多数据源。Kibana可 以非常方便地把来自Logstash、ES-Hadoop、 Beats或第三方技术的数据整合到Elasticsearch，支持的第三方技术包括Apache Flume、 Fluentd等
• 简单数据导出。Kibana可以方便地导出感兴趣的数据，与其它数据集合并融合后快速建模分析，发现新结果。

二、配置ELK日志分析系统

node1  192.168.30.7  主要软件：Elasticsearch kibana
node2  192.168.30.8  主要软件： Elasticsearch
web    192.168.30.9  主要软件： logstash apache
1
2
3

1.配置elasticsearch环境

[root@node1 ~]# cat /etc/hosts  
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.30.7 node1     //添加映射
192.168.30.8 node2

1
2
3
4
5
6

[root@node1 ~]# tar xf jdk-8u91-linux-x64.tar.gz -C /usr/local/   //直接解压jdk
[root@node1 ~]# source /etc/profile
[root@node1 ~]# java -version    //查看java版本
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
[root@node1 ~]# 

1
2
3
4
5
6
7
8

2.部署elasticsearch软件

2.1 安装软件

[root@node1 opt]# rpm -ivh elasticsearch-5.5.0.rpm 
警告：elasticsearch-5.5.0.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
正在升级/安装...
   1:elasticsearch-0:5.5.0-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

1
2
3
4
5
6
7
8
9
10
11
12
13

2.2 加载系统服务

[root@node1 opt]# systemctl daemon-reload 
[root@node1 opt]# systemctl enable elasticsearch.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@node1 opt]# 

1
2
3
4
5

2.3 修改主配置文件

[root@node1 elasticsearch]# cp -a elasticsearch.yml elasticsearch.yml.bak //拷贝一份文件备份
[root@node1 elasticsearch]# vim elasticsearch.yml 
[root@node1 elasticsearch]# grep -v "^#" elasticsearch.yml   //过滤修改字段，，node2节点修改node.name为node2
cluster.name: my-elk-cluster   //集群名称
node.name: node1               //节点名称
path.data: /data/elk_data      //数据存放路径     
path.logs: /var/log/elasticsearch/    //日志存放路径
network.host: 0.0.0.0              //绑定IP
http.port: 9200                    //侦听端口
discovery.zen.ping.unicast.hosts: ["node1", "node2"]  //集群发现通过单播实现

1
2
3
4
5
6
7
8
9
10
11

2.4 创建数据存放文件路径并授权

[root@node1 elasticsearch]# mkdir -p /data/elk_data
[root@node1 elasticsearch]# chown elasticsearch /data/elk_data/
[root@node1 elasticsearch]# ll /data/
总用量 0
drwxr-xr-x 2 elasticsearch elasticsearch 6 8月  13 19:27 elk_data
[root@node1 elasticsearch]# 

1
2
3
4
5
6
7

2.5 启动elasticsearch

[root@node1 ~]# systemctl start elasticsearch.service  //启动
[root@node1 ~]# netstat -antp | grep 9200      //监听9200端口
tcp6       0      0 :::9200                 :::*                    LISTEN      22434/java
1
2
3

2.6 查看节点信息

[root@node1 ~]# curl 192.168.30.7:9200
{
  "name" : "node1",
  "cluster_name" : "my-elk-cluster",
  "cluster_uuid" : "ani4shRfQWyF6e_O7lxeHQ",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}
[root@node1 ~]# curl 192.168.30.8:9200
{
  "name" : "node2",
  "cluster_name" : "my-elk-cluster",
  "cluster_uuid" : "ani4shRfQWyF6e_O7lxeHQ",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}
[root@node1 ~]# 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

2.7 查看集群状态信息

[root@node1 ~]# curl 192.168.30.7:9200/_cluster/health?pretty
{
  "cluster_name" : "my-elk-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[root@node1 ~]# curl 192.168.30.8:9200/_cluster/health?pretty
{
  "cluster_name" : "my-elk-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[root@node1 ~]# 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

3.安装elasticsearch-head插件

3.1 安装node组件

[root@node1 node-v8.2.1]# ./configure && make -j3 && make install

1
2

3.2 安装phantomjs前端框架

[root@node1 src]# tar xf phantomjs-2.1.1-linux-x86_64.tar.bz2 
[root@node1 src]# cd phantomjs-2.1.1-linux-x86_64/bin/
[root@node1 bin]# cp phantomjs /usr/local/bin/

1
2
3
4

3.3 安装elasticsearch-head数据可视化工具

[root@node1 src]# tar xf elasticsearch-head.tar.gz 
[root@node1 src]# cd elasticsearch-head/
[root@node1 elasticsearch-head]# npm install
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/karma/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression

up to date in 4.842s
1
2
3
4
5
6
7
8
9

3.4 修改主配置文件

[root@node1 elasticsearch-head]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: node1
path.data: /data/elk_data
path.logs: /var/log/elasticsearch/
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"]
http.cors.enabled: true        //开启跨越访问支持
http.cors.allow-origin: "*"    //跨域范根允许的域名地址
[root@node1 elasticsearch-head]# systemctl restart elasticsearch.service 

1
2
3
4
5
6
7
8
9
10
11
12

3.5 启动elasticsearch-head

[root@node1 elasticsearch-head]# npm run start &  //后台启动
[1] 69107
[root@node1 elasticsearch-head]# 
> elasticsearch-head@0.0.0 start /usr/local/src/elasticsearch-head
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100

1
2
3
4
5
6
7
8
9
10

3.6 创建索引

[root@node1 elasticsearch-head]# curl -XPUT 'localhost:9200/index-demo/test/1?pretty' -H 'content-Type:application/json' -d '{"user":"zhangsan","mesg":"hello world"}'
{
  "_index" : "index-demo",
  "_type" : "test",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "created" : true
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

3.7 查看创建的索引

4 安装logstash，采集日志输出到elasticsearch

4.1 安装apache服务

[root@apache ~]# yum -y install httpd
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
软件包 httpd-2.4.6-97.el7.centos.x86_64 已安装并且是最新版本
无须任何处理

1
2
3
4
5
6
7
8
9

4.2 查看java环境

[root@apache ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

1
2
3
4
5

4.3 安装logstash

[root@apache opt]# rpm -ivh logstash-5.5.1.rpm 
警告：logstash-5.5.1.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:logstash-1:5.5.1-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
[root@apache opt]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/
[root@apache opt]# systemctl enable logstash.service   //开机自启
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
1
2
3
4
5
6
7
8
9
10

4.4 测试功能是否正常

[root@apache opt]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/
[root@apache opt]# systemctl enable logstash.service
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
[root@apache opt]# logstash -e 'input { stdin{} } output { stdout{} }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
21:22:16.731 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
21:22:16.735 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
21:22:16.754 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"c2b6db1e-f069-4bd0-8fc3-8e2112c4963e", :path=>"/usr/share/logstash/data/uuid"}
21:22:16.896 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
21:22:16.913 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
21:22:16.956 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com
2021-08-13T13:22:30.957Z apache www.baidu.com
www.sina.com
2021-08-13T13:22:42.551Z apache www.sina.com
^C21:23:02.142 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting down the agent.
21:23:02.151 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

4.5 使用rubydebug显示详细输出

[root@apache opt]# logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug} }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
21:26:10.737 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
21:26:10.761 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
21:26:10.797 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com
{
    "@timestamp" => 2021-08-13T13:26:17.238Z,
      "@version" => "1",
          "host" => "apache",
       "message" => "www.baidu.com"
}
www.sina.com
{
    "@timestamp" => 2021-08-13T13:26:23.062Z,
      "@version" => "1",
          "host" => "apache",
       "message" => "www.sina.com"
}
^C21:26:24.388 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting down the agent.
21:26:24.398 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

4.6 对接测试

[root@apache opt]# logstash -e 'input { stdin{} } output {  elasticsearch { hosts=>["192.168.30.7:9200"]} }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
21:37:24.408 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.30.7:9200/]}}
21:37:24.413 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.30.7:9200/, :path=>"/"}
21:37:24.547 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0xb0ac142>}
21:37:24.550 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Using mapping template from {:path=>nil}
21:37:24.760 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
21:37:24.769 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Installing elasticsearch template to _template/logstash
21:37:24.859 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x6044ebed>]}
21:37:24.864 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
21:37:24.883 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
21:37:24.920 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com
www.sina.com
^C21:37:45.016 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting down the agent.
21:37:45.024 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

4.7 查看索引信息

5 对接配置

[root@apache conf.d]# cat /etc/logstash/conf.d/systemctl.conf 
input {
	file{
	    path => "/var/log/messages" //收集数据的路径
	    type => "system"            //类型
	    start_position => "beginning"    //从开头收集数据
	    }
	}
output {
	elasticsearch {
	    hosts => ["192.168.30.7:9200"]     //输出到
	    index => "system-%{+YYYY.MM.dd}"    //索引
   	    }
	}
[root@apache conf.d]# systemctl restart logstash.service 


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

6.安装kibana

6.1 安装

[root@node1 src]# rpm -ivh kibana-5.5.1-x86_64.rpm 
警告：kibana-5.5.1-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:kibana-5.5.1-1                   ################################# [100%]
[root@node1 src]# 

1
2
3
4
5
6
7

6.2 修改配置文件

[root@node1 kibana]# grep -v "^#" kibana.yml
server.port: 5601  //kibana打开的端口
server.host: "0.0.0.0"   //侦听的地址
elasticsearch.url: "http://192.168.30.7:9200"  //和elasticsearch建立联系
kibana.index: ".kibana"  //添加.kibana索引

1
2
3
4
5
6

6.3 启动

[root@node1 kibana]# systemctl start kibana.service 
[root@node1 kibana]# systemctl enable kibana.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[root@node1 kibana]# 

1
2
3
4
5

6.4 查看kibana

7.对接apache的日志

7.1 对接apache日志

[root@apache conf.d]# cat apache.conf 
input {
	file{
	    path => "/etc/httpd/logs/access_log"
	    type => "access"
	    start_position => "beginning"
	    }
	file{
	    path => "/etc/httpd/logs/erroe_log"
	    type => "error"
	    start_position => "beginning"
	    }
	}
output {
	if [type] == "access" {
	elasticsearch {
            hosts => ["192.168.30.7:9200"]
	    index => "apache_access-%{+YYYY.MM.dd}"
   	    }
	}
	if [type] == "error" {
	elasticsearch {
            hosts => ["192.168.30.7:9200"]
	    index => "apache_error-%{+YYYY.MM.dd}"
   	    }
	}
	}
[root@apache conf.d]# 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

7.2 指定配置文件，查看索引

[root@apache conf.d]# /usr/share/logstash/bin/logstash -f apache.conf 

1
2

7.3 创建索引名称

7.4 查看信息（如果看不到信息，可以更换浏览器）

小结

日志处理步骤

• 将日志进行集中化管理
• 将日志格式化（Logstash） 并输出到 Elasticsearch
• 对格式化后的数据进行索引和存储（Elasticsearch）
• 前端数据的展示 （Kibana）