devbox
不正经
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Docker elacticsearch+filebeat+logstash+kibana docker-compose一键启动 日志管理系统 本地安装 详细版_docker elasticsearch + logstash + filebeat + kiban

作者：不正经 | 2024-05-22 11:40:06

赞

踩

docker elasticsearch + logstash + filebeat + kibanaa

思路 （使用离线本地搭建）

1.安装docker以及docker-compose（自行安装 版本不低即可）

 安装docker：

(4条消息) Centos7安装Docker_玩物丧志的快乐的博客-CSDN博客_centos7 docker

安装docker-compose：

(4条消息) 安装docker-compose的两种方式_沙漠之鹰的博客-CSDN博客_如何安装docker-compose

2.创建网络并查看（开始elkf安装搭建）

修改系统参数：

进入目录   /etc/sysctl.conf 加入

vm.max_map_count=655360  

进入目录   /etc/security/limits.conf 加入

*               soft     nofile        65535
*               hard     nofile        65535
*               soft     nproc         65535
*               hard     nproc         65535
*               soft     memlock       unlimited
*               hard     memlock       unlimited

创建网络并查看

docker network create elk
docker network ls







3.编写一个docker-compose.yml文件进行服务定义和运行

创建一个主目录存放安装文件在进入该目录

mkdir /root/elkf

cd elkf

 vi /root/elkf/docker-compose.yml

version: "3"                                            #docker-compose版本

services:                                                #需要定义运行的服务
  nginx:
    restart: always
    image: nginx                                    
    container_name: nginx
    hostname: nginx
    ports:                                                #映射端口主机到容器
     - 80:80
    volumes:                                          #卷挂载路径主机到容器
     - /var/log/nginx:/var/log/nginx

  filebeat:
    restart: always
    depends_on:
     - "nginx"
    build:
      context: ./filebeat
      dockerfile: Dockerfile
    container_name: filebeat
    hostname: filebeat
    volumes:
     - /var/log/nginx:/var/log/nginx

  elasticsearch:
    restart: always
    depends_on:
     - "nginx"
    build:
      context: ./elasticsearch
      dockerfile: Dockerfile
    container_name: elasticsearch
    hostname: elasticsearch
    ports:
     - 9200:9200
     - 9300:9300
    volumes:
     - /var/log/elasticsearch:/var/log/elasticsearch

  logstash:
    restart: always
    depends_on:
     - "nginx"
    build:
      context: ./logstash
      dockerfile: Dockerfile
    container_name: logstash
    hostname: logstash
    ports:
     - 5044:5044
    volumes:
     - /opt/logstash/conf:/opt/logstash/conf

  kibana:
    restart: always
    depends_on:
     - "nginx"
    build:
      context: ./kibana
      dockerfile: Dockerfile
    container_name: kibana
    hostname: kibana
    ports:
     - 5601:5601
    

networks:                                             #定义添加的网络
  default:
    external:
      name: elk
 

这里边 就定义了需要运行的elkf的四个服务 

4.构建镜像

软件包下载地址  

http://www.haojiang.online/other/download.tar.gz   （ps:这里我已经打包好了全部4个安装包）

1、构建elacticsearch镜像

cd /root/elkf/

mkdir elasticsearch

cd elasticsearch

 创建Dockerfile文件并写入信息

vi Dockerfile

FROM centos:7.9.2009
MAINTAINER wzlu
RUN yum -y install java-1.8.0-openjdk vim telnet lsof
ADD elasticsearch-6.1.0.tar.gz /usr/local/
RUN cd /usr/local/elasticsearch-6.1.0/config
RUN mkdir -p /data/behavior/log-node1
RUN mkdir /var/log/elasticsearch
COPY elasticsearch.yml /usr/local/elasticsearch-6.1.0/config/
RUN useradd es && chown -R es:es /usr/local/elasticsearch-6.1.0
RUN chmod +x /usr/local/elasticsearch-6.1.0/bin/*
RUN chown -R es:es /var/log/elasticsearch/
RUN chown -R es:es /data/behavior/log-node1
EXPOSE 9200
EXPOSE 9300
CMD su es /usr/local/elasticsearch-6.1.0/bin/elasticsearch

 创建一个elasticsearch.yml文件 写入 如下信息

  vi elasticsearch.yml

  1. # ======================== Elasticsearch Configuration =========================
  2. #
  3. # NOTE: Elasticsearch comes with reasonable defaults for most settings.
  4. #       Before you set out to tweak and tune the configuration, make sure you
  5. #       understand what are you trying to accomplish and the consequences.
  6. #
  7. # The primary way of configuring a node is via this file. This template lists
  8. # the most important settings you may want to configure for a production cluster.
  9. #
  10. # Please consult the documentation for further information on configuration options:
  11. # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
  12. #
  13. # ---------------------------------- Cluster -----------------------------------
  14. #
  15. # Use a descriptive name for your cluster:
  16. #
  17. cluster.name: my-elk
  18. #
  19. # ------------------------------------ Node ------------------------------------
  20. #
  21. # Use a descriptive name for the node:
  22. #
  23. node.name: node-1
  24. #
  25. # Add custom attributes to the node:
  26. #
  27. #node.attr.rack: r1
  28. #
  29. # ----------------------------------- Paths ------------------------------------
  30. #
  31. # Path to directory where to store the data (separate multiple locations by comma):
  32. #
  33. #path.data: /path/to/data
  34. #
  35. # Path to log files:
  36. #
  37. #path.logs: /path/to/logs
  38. #
  39. # ----------------------------------- Memory -----------------------------------
  40. #
  41. # Lock the memory on startup:
  42. #
  43. #bootstrap.memory_lock: true
  44. #
  45. # Make sure that the heap size is set to about half the memory available
  46. # on the system and that the owner of the process is allowed to use this
  47. # limit.
  48. #
  49. # Elasticsearch performs poorly when the system is swapping the memory.
  50. #
  51. # ---------------------------------- Network -----------------------------------
  52. #
  53. # Set the bind address to a specific IP (IPv4 or IPv6):
  54. #
  55. network.host: 0.0.0.0
  56.  
  57. #
  58. # Set a custom port for HTTP:
  59. #
  60. http.port: 9200
  61. #
  62. # For more information, consult the network module documentation.
  63. #
  64. # --------------------------------- Discovery ----------------------------------
  65. #
  66. # Pass an initial list of hosts to perform discovery when new node is started:
  67. # The default list of hosts is ["127.0.0.1", "[::1]"]
  68. #
  69. #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
  70. #
  71. # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
  72. #
  73. #discovery.zen.minimum_master_nodes: 
  74. #
  75. # For more information, consult the zen discovery module documentation.
  76. #
  77. # ---------------------------------- Gateway -----------------------------------
  78. #
  79. # Block initial recovery after a full cluster restart until N nodes are started:
  80. #
  81. #gateway.recover_after_nodes: 3
  82. #
  83. # For more information, consult the gateway module documentation.
  84. #
  85. # ---------------------------------- Various -----------------------------------
  86. #
  87. # Require explicit names when deleting indices:
  88. #
  89. #action.destructive_requires_name: true

再导入elacticsearch的安装包

elactissearch下目录结构

2、构建logstash镜像

cd /root/elkf/

mkdir logstash

cd logstash

 写入logstash配置文件

 mkdir -p /opt/logstash/conf

vim /opt/logstash/conf/logstash-nginx-log.conf

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
  beats { port => 5044 }
}
 
filter {
    date { match => [ "timestamp","yyyy-MM-dd HH:mm:ss,SSS" ] }
    if "nginx-access" in [tags] {
    grok {
        match => {
            "message" => '%{IP:remote_addr} - (%{WORD:remote_user}|-)

"%{WORD:method} %{NOTSPACE:request} HTTP/%{NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}'
        }
          }
     urldecode {
            all_fields => true
        }
    date {
            match => [ "time_local" , "dd/MMM/YYYY:HH:mm:ss Z" ]
          }
    }
}
 
output {
  if "nginx-access" in [tags] {
    elasticsearch {
      hosts => [ "elasticsearch:9200" ]
      manage_template => false
      index => "nginx-access-%{+YYYY.MM.dd}"
    }
  }
}
 

创建Dockerfile文件并写入信息

vi Dockerfile 

FROM centos:7.9.2009
MAINTAINER wzlu
RUN yum -y install java-1.8.0-openjdk vim telnet lsof
ADD logstash-6.1.0.tar.gz /usr/local/
RUN cd /usr/local/logstash-6.1.0
ADD run.sh /run.sh
RUN chmod 755 /*.sh
EXPOSE 5044
CMD ["/run.sh"]

 编写执行的脚本文件

#!/bin/bash
/usr/local/logstash-6.1.0/bin/logstash -f /opt/logstash/conf/logstash-nginx-log.conf

注意：一定要与刚才存放的logstash配置文件路径一致

再导入logstash的安装包 

logstash下目录结构 

3、构建kibana镜像

cd /root/elkf/

mkdir kibana

cd kibana

创建Dockerfile文件并写入信息

vi Dockerfile

FROM centos:7.9.2009
MAINTAINER wzlu
RUN yum -y install java-1.8.0-openjdk vim telnet lsof
ADD kibana-6.1.0-linux-x86_64.tar.gz /usr/local/
RUN cd /usr/local/kibana-6.1.0-linux-x86_64
COPY kibana.yml /usr/local/kibana-6.1.0-linux-x86_64/config/
EXPOSE 5601
CMD ["/usr/local/kibana-6.1.0-linux-x86_64/bin/kibana"]

创建kibana.yml文件 在写入一下信息

   vi  kibana.yml     (复制即可)

  1. # Kibana is served by a back end server. This setting specifies the port to use.
  2. server.port: 5601
  3.  
  4. # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
  5. # The default is 'localhost', which usually means remote machines will not be able to connect.
  6. # To allow connections from remote users, set this parameter to a non-loopback address.
  7. server.host: "0.0.0.0"
  8.  
  9. # Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
  10. # the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
  11. # to Kibana. This setting cannot end in a slash.
  12. #server.basePath: ""
  13.  
  14. # The maximum payload size in bytes for incoming server requests.
  15. #server.maxPayloadBytes: 1048576
  16.  
  17. # The Kibana server's name.  This is used for display purposes.
  18. server.name: "kibana"
  19.  
  20. # The URL of the Elasticsearch instance to use for all your queries.
  21. elasticsearch.url: "http://elasticsearch:9200"
  22.  
  23. # When this setting's value is true Kibana uses the hostname specified in the server.host
  24. # setting. When the value of this setting is false, Kibana uses the hostname of the host
  25. # that connects to this Kibana instance.
  26. #elasticsearch.preserveHost: true
  27.  
  28. # Kibana uses an index in Elasticsearch to store saved searches, visualizations and
  29. # dashboards. Kibana creates a new index if the index doesn't already exist.
  30. #kibana.index: ".kibana"
  31.  
  32. # The default application to load.
  33. #kibana.defaultAppId: "home"
  34.  
  35. # If your Elasticsearch is protected with basic authentication, these settings provide
  36. # the username and password that the Kibana server uses to perform maintenance on the Kibana
  37. # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
  38. # is proxied through the Kibana server.
  39. #elasticsearch.username: "user"
  40. #elasticsearch.password: "pass"
  41.  
  42. # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
  43. # These settings enable SSL for outgoing requests from the Kibana server to the browser.
  44. #server.ssl.enabled: false
  45. #server.ssl.certificate: /path/to/your/server.crt
  46. #server.ssl.key: /path/to/your/server.key
  47.  
  48. # Optional settings that provide the paths to the PEM-format SSL certificate and key files.
  49. # These files validate that your Elasticsearch backend uses the same key files.
  50. #elasticsearch.ssl.certificate: /path/to/your/client.crt
  51. #elasticsearch.ssl.key: /path/to/your/client.key
  52.  
  53. # Optional setting that enables you to specify a path to the PEM file for the certificate
  54. # authority for your Elasticsearch instance.
  55. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
  56.  
  57. # To disregard the validity of SSL certificates, change this setting's value to 'none'.
  58. #elasticsearch.ssl.verificationMode: full
  59.  
  60. # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
  61. # the elasticsearch.requestTimeout setting.
  62. #elasticsearch.pingTimeout: 1500
  63.  
  64. # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
  65. # must be a positive integer.
  66. #elasticsearch.requestTimeout: 30000
  67.  
  68. # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
  69. # headers, set this value to [] (an empty list).
  70. #elasticsearch.requestHeadersWhitelist: [ authorization ]
  71.  
  72. # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
  73. # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
  74. #elasticsearch.customHeaders: {}
  75.  
  76. # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
  77. #elasticsearch.shardTimeout: 0
  78.  
  79. # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
  80. #elasticsearch.startupTimeout: 5000
  81.  
  82. # Specifies the path where Kibana creates the process ID file.
  83. #pid.file: /var/run/kibana.pid
  84.  
  85. # Enables you specify a file where Kibana stores log output.
  86. #logging.dest: stdout
  87.  
  88. # Set the value of this setting to true to suppress all logging output.
  89. #logging.silent: false
  90.  
  91. # Set the value of this setting to true to suppress all logging output other than error messages.
  92. #logging.quiet: false
  93.  
  94. # Set the value of this setting to true to log all events, including system usage information
  95. # and all requests.
  96. #logging.verbose: false
  97.  
  98. # Set the interval in milliseconds to sample system and process performance
  99. # metrics. Minimum is 100ms. Defaults to 5000.
  100. #ops.interval: 5000
  101.  
  102. # The default locale. This locale can be used in certain circumstances to substitute any missing
  103. # translations.
  104. #i18n.defaultLocale: "en"

再导入kibana的安装包

kibana下目录结构

4、构建filebeat镜像

cd /root/elkf/

mkdir filebeat

cd filebeat

 创建Dockerfile文件并写入信息

vi Dockerfile

FROM centos:7.9.2009
MAINTAINER wzlu
RUN yum -y install java-1.8.0-openjdk vim telnet lsof
ADD filebeat-6.1.0-linux-x86_64.tar.gz /usr/local/
RUN cd /usr/local/filebeat-6.1.0-linux-x86_64
COPY filebeat.yml /usr/local/filebeat-6.1.0-linux-x86_64
ADD run.sh /run.sh
RUN chmod 755 /*.sh
CMD ["/run.sh"]

 编写执行的脚本文件

#!/bin/bash
/usr/local/filebeat-6.1.0-linux-x86_64/filebeat -e -c /usr/local/filebeat-6.1.0-linux-x86_64/filebeat.yml

 创建filebeat.yml文件  写入如下信息

   vi filebeat.yml  

  1. ###################### Filebeat Configuration Example #########################
  2.  
  3. # This file is an example configuration file highlighting only the most common
  4. # options. The filebeat.reference.yml file from the same directory contains all the
  5. # supported options with more comments. You can use it as a reference.
  6. #
  7. # You can find the full configuration reference here:
  8. # https://www.elastic.co/guide/en/beats/filebeat/index.html
  9.  
  10. # For more available modules and options, please see the filebeat.reference.yml sample
  11. # configuration file.
  12.  
  13. #=========================== Filebeat prospectors =============================
  14.  
  15. filebeat.prospectors:
  16.  
  17. # Each - is a prospector. Most options can be set at the prospector level, so
  18. # you can use different prospectors for various configurations.
  19. # Below are the prospector specific configurations.
  20.  
  21. - type: log
  22.  
  23.   # Change to true to enable this prospector configuration.
  24.   enabled: true
  25.  
  26.   # Paths that should be crawled and fetched. Glob based paths.
  27.   paths:
  28.     - /var/log/nginx/access.log
  29.     #- c:\programdata\elasticsearch\logs\*
  30.  
  31.   tags: ["nginx-access"]
  32.   clean_*: true
  33.   # Exclude lines. A list of regular expressions to match. It drops the lines that are
  34.   # matching any regular expression from the list.
  35.   #exclude_lines: ['^DBG']
  36.  
  37.   # Include lines. A list of regular expressions to match. It exports the lines that are
  38.   # matching any regular expression from the list.
  39.   #include_lines: ['^ERR', '^WARN']
  40.  
  41.   # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  42.   # are matching any regular expression from the list. By default, no files are dropped.
  43.   #exclude_files: ['.gz$']
  44.  
  45.   # Optional additional fields. These fields can be freely picked
  46.   # to add additional information to the crawled log files for filtering
  47.   #fields:
  48.   #  level: debug
  49.   #  review: 1
  50.  
  51.   ### Multiline options
  52.  
  53.   # Mutiline can be used for log messages spanning multiple lines. This is common
  54.   # for Java Stack Traces or C-Line Continuation
  55.  
  56.   # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  57.   #multiline.pattern: ^\[
  58.  
  59.   # Defines if the pattern set under pattern should be negated or not. Default is false.
  60.   #multiline.negate: false
  61.  
  62.   # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  63.   # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  64.   # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  65.   #multiline.match: after
  66.  
  67.  
  68. #============================= Filebeat modules ===============================
  69.  
  70. filebeat.config.modules:
  71.   # Glob pattern for configuration loading
  72.   path: ${path.config}/modules.d/*.yml
  73.  
  74.   # Set to true to enable config reloading
  75.   reload.enabled: false
  76.  
  77.   # Period on which files under path should be checked for changes
  78.   #reload.period: 10s
  79.  
  80. #==================== Elasticsearch template setting ==========================
  81.  
  82. setup.template.settings:
  83.   index.number_of_shards: 3
  84.   #index.codec: best_compression
  85.   #_source.enabled: false
  86.  
  87. #================================ General =====================================
  88.  
  89. # The name of the shipper that publishes the network data. It can be used to group
  90. # all the transactions sent by a single shipper in the web interface.
  91. #name:
  92.  
  93. # The tags of the shipper are included in their own field with each
  94. # transaction published.
  95. #tags: ["nginx-access"]
  96.  
  97. # Optional fields that you can specify to add additional information to the
  98. # output.
  99. #fields:
  100. #  env: staging
  101.  
  102.  
  103. #============================== Dashboards =====================================
  104. # These settings control loading the sample dashboards to the Kibana index. Loading
  105. # the dashboards is disabled by default and can be enabled either by setting the
  106. # options here, or by using the `-setup` CLI flag or the `setup` command.
  107. #setup.dashboards.enabled: false
  108.  
  109. # The URL from where to download the dashboards archive. By default this URL
  110. # has a value which is computed based on the Beat name and version. For released
  111. # versions, this URL points to the dashboard archive on the artifacts.elastic.co
  112. # website.
  113. #setup.dashboards.url:
  114.  
  115. #============================== Kibana =====================================
  116.  
  117. # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
  118. # This requires a Kibana endpoint configuration.
  119. setup.kibana:
  120.  
  121.   # Kibana Host
  122.   # Scheme and port can be left out and will be set to the default (http and 5601)
  123.   # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  124.   # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  125.   #host: "localhost:5601"
  126.  
  127. #============================= Elastic Cloud ==================================
  128.  
  129. # These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).
  130.  
  131. # The cloud.id setting overwrites the `output.elasticsearch.hosts` and
  132. # `setup.kibana.host` options.
  133. # You can find the `cloud.id` in the Elastic Cloud web UI.
  134. #cloud.id:
  135.  
  136. # The cloud.auth setting overwrites the `output.elasticsearch.username` and
  137. # `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
  138. #cloud.auth:
  139.  
  140. #================================ Outputs =====================================
  141.  
  142. # Configure what output to use when sending the data collected by the beat.
  143.  
  144. #-------------------------- Elasticsearch output ------------------------------
  145. #output.elasticsearch:
  146.   # Array of hosts to connect to.
  147.   #hosts: ["localhost:9200"]
  148.  
  149.   # Optional protocol and basic auth credentials.
  150.   #protocol: "https"
  151.   #username: "elastic"
  152.   #password: "changeme"
  153.  
  154. #----------------------------- Logstash output --------------------------------
  155. output.logstash:
  156.   # The Logstash hosts
  157.   hosts: ["logstash:5044"]
  158.  
  159.   # Optional SSL. By default is off.
  160.   # List of root certificates for HTTPS server verifications
  161.   #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  162.  
  163.   # Certificate for SSL client authentication
  164.   #ssl.certificate: "/etc/pki/client/cert.pem"
  165.  
  166.   # Client Certificate Key
  167.   #ssl.key: "/etc/pki/client/cert.key"
  168.  
  169. #================================ Logging =====================================
  170.  
  171. # Sets log level. The default log level is info.
  172. # Available log levels are: critical, error, warning, info, debug
  173. #logging.level: debug
  174.  
  175. # At debug level, you can selectively enable logging only for some components.
  176. # To enable all selectors use ["*"]. Examples of other selectors are "beat",
  177. # "publish", "service".
  178. #logging.selectors: ["*"]

 导入filebeat离线安装包

filebeat目录下结构

5.一键部署启动

elkf目录结构

 先拉一个新的nginx镜像

docker pull nginx

然后使用docker-compose命令一键部署           （ps:注意检查容器端口是否被占用）

 docker-compose up -d

查看容器状态

docker-compose ps

获取日志信息

watch -n 2 curl -k 192.168.25.100（本机ip）

6.登录kibana查看日志

登录    http://本机IP地址:5601  查看

登录成功

查看日志

 ​​​

elkf日志系统搭建成功

如果您们发现里边有什么错误和问题可以联系作者欢迎指正，原创，谢谢支持！

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/不正经/article/detail/608089
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号