日志采集客户端 filebeat 安装部署_filebeat 采集到hive

linux
----------------
1. 创建yum 源文件

[root@localhost ~]# vim /etc/yum.repos.d/elk-elasticsearch.repo
[elastic-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

2. 安装 和启动服务

yum install filebeat

systemctl start filebeat
systemctl status filebeat

3. 配置文件    
配置文件：cat  /etc/filebeat/filebeat.yml

    filebeat.prospectors:
    -
      fields: {log_type: "web-proxy-nginx-access"}
      paths:
        - /data/logs/nginx/access/www2.access.log
    -
      fields: {log_type: "web-proxy-nginx-error"}
      paths:
        - /data/logs/nginx/error/www2.error.log

output.kafka:
  hosts: ["59.188.25.225:9092"]
  topic: 'web-proxy-nginx-log'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

shipper:
  tags: ["web-proxy-nginx-filebeat"]

4.启动和关闭：
    启动：nohup /home/elk/filebeat/filebeat -e -c /home/elk/filebeat/filebeat.yml  &
    关闭：kill -9 `ps aux|grep filebeat|head -1|awk -F" " '{print $2}'`
    查看进程：ps aux |grep filebeat

window
-------------------
1. 下载
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.1-windows-x86_64.zip
2. 解压到C:\Program Files\Filebeat
3. 添加服务
   以管理员允许powershell
   cd 'C:\Program Files\Filebeat'
   set-executionpolicy Unrestricted
   .\install-service-filebeat.ps1
   set-executionpolicy Restricted
4. 配置文件: filebeat.yml
    filebeat.prospectors:
    -
      fields: {log_type: "web-proxy-nginx-access"}
      paths:
        - D:\data\www\nginx-1.4.7\logs\web2.access.log
    -
      fields: {log_type: "web-proxy-nginx-error"}
      paths:
        - D:\data\www\nginx-1.4.7\logs\web2.error.log

output.kafka:
  hosts: ["59.188.25.2xx:9092"]
  topic: 'web-proxy-nginx-log'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

shipper:
  tags: ["web-proxy-nginx-filebeat"]    
5. 启动
   power shell下执行：Start-Service filebeat

让filebeat重新全部读取日志
-------------------------------

filebeat会在data/registry中记录已经读取的日志，如果想要filebeat全部重新读取，删除data/registry文件，然后重启filebeat

转载于https://www.cnblogs.com/zhaojonjon/p/7289498.html