devbox
木道寻08
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

kubernetes EFK日志管理系统

作者：木道寻08 | 2024-07-06 11:16:16

赞

踩

efk operator

1. kubernetes 日志管理方案介绍:

  在kubernetes集群中一般使用EFK日志解决方案，所谓的EFK分布代表了elasticsearch、fluentd、kibana.

  fluentd主要功能是日志收集客户端，主要用途是收集每一个pod的日志，同时也会收集操作系统层面的日志，比如kubelet组件、docker服务、apiserver等组件的日志；

因此flunetd组件一般是以DaemonSet的资源类型运行在k8s之上，包括k8s master节点都需要运行fluentd容器；

  elasticsearch是日志存储功能，主要用来存储收集来的日志生成索引。一般都是要创建成ES集群。标准的ES集群一般分为Client端、Master管理端、DATA数据端。Client端主要负责对外提供接口，也就是访问ES集群就是访问Client的服务请求。然后Client再把服务转发给Master服务器，最后由Master把数据写入到DATA服务器。

  kibana主要用途是日志展示和管理搜索等。将ES集群中的日志索引通过图形界面展示出来，可以实时查看日志的内容和根据关键字搜索日志；

  有的解决方案是这样的. fluentd--->logstash(日志格式转换等)--->Redis缓存--->ES集群存储--->kibana展示。其中用到的logstash是用来做日志的过滤和格式转换的，而redis用来做缓存，主要是因为ES的写入速度很慢，但是fluentd的发送日志速度很快，如果中间没有缓存的话，可能会造成数据的延迟或者丢失；

  在k8s容器中收集日志也有集中类型，一般fluentd的日志收集客户端只能收集STDOUT(标准输出)的日志， 什么是标准输出？就是你可以通过kubectl logs -f Pod_name 查询到的日志输出就是标准输出。但是有些JAVA应用程序并没有把业务日志输出到STDOUT，而是输出到一个文件里面。
大家应该知道容器里面是不能写入文件内容的，因为容器一旦重启内容就会消失，除非通过挂载共享存储的方式才能保存文件中的日志；但是文件中的日志fluentd检测不到怎么办，这个时候就要用到sidecar容器来实现把业务日志的文件中的内容输出到标准输出；或者也可以直接修改log4j配置文件将应用日志直接输出到ES；
边角容器这种方法也有一个明显的缺陷，就是日志不仅会在原容器文件中保留下来，还会通过 stdout 输出后占用磁盘空间，这样无形中就增加了一倍磁盘空间。

2. 使用Helm安装fluentd、ES、kibana:

  由于前面几篇博文已经详细描述过helm怎么安装Redis、RabbitMQ、Mysql、Jenkins等步骤，这里就不详细描述Helm的使用方法了。
  反正大家只要记住几点就行:

  1. helm的官方网站是: https://github.com/helm/charts/tree/master/stable/ 你需要的应用都在这个目录下面；
  2. 一般的流程就是helm search 应用名---> helm fetch 应用名--->根据官方文档配置value.yaml--->启动helm应用;

  接下来我主要是把这几个应用的value.yaml文件给大家重点介绍一下：

  1. image:
  2.   repository: k8s.harbor.maimaiti.site/system/fluentd
  3. ## Specify an imagePullPolicy (Required)
  4. ## It's recommended to change this to 'Always' if the image tag is 'latest'
  5. ## ref: http://kubernetes.io/docs/user-guide/images/#updating-images
  6.   tag: v2.5.1
  7.   pullPolicy: IfNotPresent
  8.   ## Optionally specify an array of imagePullSecrets.
  9.   ## Secrets must be manually created in the namespace.
  10.   ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  11.   ##
  12.   # pullSecrets:
  13.   #   - myRegistrKeySecretName
  15. ## If using AWS Elasticsearch, all requests to ES need to be signed regardless of whether
  16. ## one is using Cognito or not. By setting this to true, this chart will install a sidecar
  17. ## proxy that takes care of signing all requests being sent to the AWS ES Domain.
  18. awsSigningSidecar:
  19.   enabled: false
  20.   image:
  21.     repository: abutaha/aws-es-proxy
  22.     tag: 0.9
  24. # Specify to use specific priorityClass for pods
  25. # ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
  26. # If a Pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority
  27. # Pods to make scheduling of the pending Pod possible.
  28. priorityClassName: ""
  30. ## Configure resource requests and limits
  31. ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
  32. ##
  33. resources: {}
  34.   # limits:
  35.   #   cpu: 100m
  36.   #   memory: 500Mi
  37.   # requests:
  38.   #   cpu: 100m
  39.   #   memory: 200Mi
  41. elasticsearch:
  42.   host: 'elasticsearch-client.kube-system'
  43.   port: 9200
  44.   scheme: 'http'
  45.   ssl_version: TLSv1_2
  46.   user: ""
  47.   password: ""
  48.   buffer_chunk_limit: 2M
  49.   buffer_queue_limit: 8
  50.   logstash_prefix: 'logstash'
  52. # If you want to add custom environment variables, use the env dict
  53. # You can then reference these in your config file e.g.:
  54. #     user "#{ENV['OUTPUT_USER']}"
  55. env:
  56.   # OUTPUT_USER: my_user
  57.   # LIVENESS_THRESHOLD_SECONDS: 300
  58.   # STUCK_THRESHOLD_SECONDS: 900
  60. # If you want to add custom environment variables from secrets, use the secret list
  61. secret:
  62. # - name: ELASTICSEARCH_PASSWORD
  63. #   secret_name: elasticsearch
  64. #   secret_key: password
  66. rbac:
  67.   create: true
  69. serviceAccount:
  70.   # Specifies whether a ServiceAccount should be created
  71.   create: true
  72.   # The name of the ServiceAccount to use.
  73.   # If not set and create is true, a name is generated using the fullname template
  74.   name: ""
  76. ## Specify if a Pod Security Policy for node-exporter must be created
  77. ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  78. ##
  79. podSecurityPolicy:
  80.   enabled: false
  81.   annotations: {}
  82.     ## Specify pod annotations
  83.     ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#apparmor
  84.     ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp
  85.     ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#sysctl
  86.     ##
  87.     # seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  88.     # seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
  89.     # apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
  91. livenessProbe:
  92.   enabled: true
  94. annotations: {}
  96. podAnnotations:
  97.   prometheus.io/scrape: "true"
  98.   prometheus.io/port: "24231"
  100. ## DaemonSet update strategy
  101. ## Ref: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
  102. updateStrategy:
  103.   type: RollingUpdate
  105. tolerations:
  106.   - key: node-role.kubernetes.io/master
  107.     operator: Exists
  108.     effect: NoSchedule
  110. affinity: {}
  111.   # nodeAffinity:
  112.   #   requiredDuringSchedulingIgnoredDuringExecution:
  113.   #     nodeSelectorTerms:
  114.   #     - matchExpressions:
  115.   #       - key: node-role.kubernetes.io/master
  116.   #         operator: DoesNotExist
  118. nodeSelector: {}
  120. service:
  121.   type: ClusterIP
  122.   ports:
  123.     - name: "monitor-agent"
  124.       port: 24231
  126. serviceMonitor:
  127.   ## If true, a ServiceMonitor CRD is created for a prometheus operator
  128.   ## https://github.com/coreos/prometheus-operator
  129.   ##
  130.   enabled: false
  131.   interval: 10s
  132.   path: /metrics
  133.   labels: {}
  135. prometheusRule:
  136.   ## If true, a PrometheusRule CRD is created for a prometheus operator
  137.   ## https://github.com/coreos/prometheus-operator
  138.   ##
  139.   enabled: false
  140.   prometheusNamespace: monitoring
  141.   labels: {}
  142.   #  role: alert-rules
  144. configMaps:
  145.   system.conf: |-
  146.     <system>
  147.       root_dir /tmp/fluentd-buffers/
  148.     </system>
  149.   containers.input.conf: |-
  150.     # This configuration file for Fluentd / td-agent is used
  151.     # to watch changes to Docker log files. The kubelet creates symlinks that
  152.     # capture the pod name, namespace, container name & Docker container ID
  153.     # to the docker logs for pods in the /var/log/containers directory on the host.
  154.     # If running this fluentd configuration in a Docker container, the /var/log
  155.     # directory should be mounted in the container.
  156.     #
  157.     # These logs are then submitted to Elasticsearch which assumes the
  158.     # installation of the fluent-plugin-elasticsearch & the
  159.     # fluent-plugin-kubernetes_metadata_filter plugins.
  160.     # See https://github.com/uken/fluent-plugin-elasticsearch &
  161.     # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for
  162.     # more information about the plugins.
  163.     #
  164.     # Example
  165.     # =======
  166.     # A line in the Docker log file might look like this JSON:
  167.     #
  168.     # {"log":"2014/09/25 21:15:03 Got request with path wombat\n",
  169.     #  "stream":"stderr",
  170.     #   "time":"2014-09-25T21:15:03.499185026Z"}
  171.     #
  172.     # The time_format specification below makes sure we properly
  173.     # parse the time format produced by Docker. This will be
  174.     # submitted to Elasticsearch and should appear like:
  175.     # $ curl 'http://elasticsearch-logging:9200/_search?pretty'
  176.     # ...
  177.     # {
  178.     #      "_index" : "logstash-2014.09.25",
  179.     #      "_type" : "fluentd",
  180.     #      "_id" : "VBrbor2QTuGpsQyTCdfzqA",
  181.     #      "_score" : 1.0,
  182.     #      "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n",
  183.     #                 "stream":"stderr","tag":"docker.container.all",
  184.     #                 "@timestamp":"2014-09-25T22:45:50+00:00"}
  185.     #    },
  186.     # ...
  187.     #
  188.     # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log
  189.     # record & add labels to the log record if properly configured. This enables users
  190.     # to filter & search logs on any metadata.
  191.     # For example a Docker container's logs might be in the directory:
  192.     #
  193.     #  /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b
  194.     #
  195.     # and in the file:
  196.     #
  197.     #  997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
  198.     #
  199.     # where 997599971ee6... is the Docker ID of the running container.
  200.     # The Kubernetes kubelet makes a symbolic link to this file on the host machine
  201.     # in the /var/log/containers directory which includes the pod name and the Kubernetes
  202.     # container name:
  203.     #
  204.     #    synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
  205.     #    ->
  206.     #    /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
  207.     #
  208.     # The /var/log directory on the host is mapped to the /var/log directory in the container
  209.     # running this instance of Fluentd and we end up collecting the file:
  210.     #
  211.     #   /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
  212.     #
  213.     # This results in the tag:
  214.     #
  215.     #  var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
  216.     #
  217.     # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name
  218.     # which are added to the log message as a kubernetes field object & the Docker container ID
  219.     # is also added under the docker field object.
  220.     # The final tag is:
  221.     #
  222.     #   kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
  223.     #
  224.     # And the final log record look like:
  225.     #
  226.     # {
  227.     #   "log":"2014/09/25 21:15:03 Got request with path wombat\n",
  228.     #   "stream":"stderr",
  229.     #   "time":"2014-09-25T21:15:03.499185026Z",
  230.     #   "kubernetes": {
  231.     #     "namespace": "default",
  232.     #     "pod_name": "synthetic-logger-0.25lps-pod",
  233.     #     "container_name": "synth-lgr"
  234.     #   },
  235.     #   "docker": {
  236.     #     "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b"
  237.     #   }
  238.     # }
  239.     #
  240.     # This makes it easier for users to search for logs by pod name or by
  241.     # the name of the Kubernetes container regardless of how many times the
  242.     # Kubernetes pod has been restarted (resulting in a several Docker container IDs).
  243.     # Json Log Example:
  244.     # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}
  245.     # CRI Log Example:
  246.     # 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here
  247.     <source>
  248.       @id fluentd-containers.log
  249.       @type tail
  250.       path /var/log/containers/*.log
  251.       pos_file /var/log/containers.log.pos
  252.       tag raw.kubernetes.*
  253.       read_from_head true
  254.       <parse>
  255.         @type multi_format
  256.         <pattern>
  257.           format json
  258.           time_key time
  259.           time_format %Y-%m-%dT%H:%M:%S.%NZ
  260.         </pattern>
  261.         <pattern>
  262.           format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
  263.           time_format %Y-%m-%dT%H:%M:%S.%N%:z
  264.         </pattern>
  265.       </parse>
  266.     </source>
  267.     <source>
  268.       @id nginxtest1.log
  269.       @type tail
  270.       path /var/log/containers/nginxtest1-*.log
  271.       pos_file /var/log/nginxtest1.log.pos
  272.       tag nginxtest1
  273.       read_from_head true
  274.       <parse>
  275.         @type multi_format
  276.         <pattern>
  277.           format json
  278.           time_key time
  279.           time_format %Y-%m-%dT%H:%M:%S.%NZ
  280.         </pattern>
  281.         <pattern>
  282.           format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
  283.           time_format %Y-%m-%dT%H:%M:%S.%N%:z
  284.         </pattern>
  285.       </parse>
  286.     </source>
  287.     <source>
  288.       @id httpdtest1.log
  289.       @type tail
  290.       path /var/log/containers/httpdtest1-*.log
  291.       pos_file /var/log/httpdtest1.log.pos
  292.       tag httpdtest1
  293.       read_from_head true
  294.       <parse>
  295.         @type multi_format
  296.         <pattern>
  297.           format json
  298.           time_key time
  299.           time_format %Y-%m-%dT%H:%M:%S.%NZ
  300.         </pattern>
  301.         <pattern>
  302.           format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
  303.           time_format %Y-%m-%dT%H:%M:%S.%N%:z
  304.         </pattern>
  305.       </parse>
  306.     </source>
  307.  
  308.     # Detect exceptions in the log output and forward them as one log entry.
  309.     <match raw.kubernetes.**>
  310.       @id raw.kubernetes
  311.       @type detect_exceptions
  312.       remove_tag_prefix raw
  313.       message log
  314.       stream stream
  315.       multiline_flush_interval 5
  316.       max_bytes 500000
  317.       max_lines 1000
  318.     </match>
  319.  
  320.     # Concatenate multi-line logs
  321.     <filter **>
  322.       @id filter_concat
  323.       @type concat
  324.       key message
  325.       multiline_end_regexp /\n$/
  326.       separator ""
  327.     </filter>
  328.  
  329.     # Enriches records with Kubernetes metadata
  330.     <filter kubernetes.**>
  331.       @id filter_kubernetes_metadata
  332.       @type kubernetes_metadata
  333.     </filter>
  334.  
  335.     # Fixes json fields in Elasticsearch
  336.     <filter kubernetes.**>
  337.       @id filter_parser
  338.       @type parser
  339.       key_name log
  340.       reserve_data true
  341.       remove_key_name_field true
  342.       <parse>
  343.         @type multi_format
  344.         <pattern>
  345.           format json
  346.         </pattern>
  347.         <pattern>
  348.           format none
  349.         </pattern>
  350.       </parse>
  351.     </filter>
  352.  
  353.   system.input.conf: |-
  354.     # Example:
  355.     # 2015-12-21 23:17:22,066 [salt.state       ][INFO    ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081
  356.     <source>
  357.       @id minion
  358.       @type tail
  359.       format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
  360.       time_format %Y-%m-%d %H:%M:%S
  361.       path /var/log/salt/minion
  362.       pos_file /var/log/salt.pos
  363.       tag salt
  364.     </source>
  365.  
  366.     # Example:
  367.     # Dec 21 23:17:22 gke-foo-1-1-4b5cbd14-node-4eoj startupscript: Finished running startup script /var/run/google.startup.script
  368.     <source>
  369.       @id startupscript.log
  370.       @type tail
  371.       format syslog
  372.       path /var/log/startupscript.log
  373.       pos_file /var/log/startupscript.log.pos
  374.       tag startupscript
  375.     </source>
  376.  
  377.     # Examples:
  378.     # time="2016-02-04T06:51:03.053580605Z" level=info msg="GET /containers/json"
  379.     # time="2016-02-04T07:53:57.505612354Z" level=error msg="HTTP Error" err="No such image: -f" statusCode=404
  380.     # TODO(random-liu): Remove this after cri container runtime rolls out.
  381.     <source>
  382.       @id docker.log
  383.       @type tail
  384.       format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
  385.       path /var/log/docker.log
  386.       pos_file /var/log/docker.log.pos
  387.       tag docker
  388.     </source>
  389.  
  390.     # Example:
  391.     # 2016/02/04 06:52:38 filePurge: successfully removed file /var/etcd/data/member/wal/00000000000006d0-00000000010a23d1.wal
  392.     <source>
  393.       @id etcd.log
  394.       @type tail
  395.       # Not parsing this, because it doesn't have anything particularly useful to
  396.       # parse out of it (like severities).
  397.       format none
  398.       path /var/log/etcd.log
  399.       pos_file /var/log/etcd.log.pos
  400.       tag etcd
  401.     </source>
  403.     # Multi-line parsing is required for all the kube logs because very large log
  404.     # statements, such as those that include entire object bodies, get split into
  405.     # multiple lines by glog.
  406.     # Example:
  407.     # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537]
  408.     <source>
  409.       @id kubelet.log
  410.       @type tail
  411.       format multiline
  412.       multiline_flush_interval 5s
  413.       format_firstline /^\w\d{4}/
  414.       format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  415.       time_format %m%d %H:%M:%S.%N
  416.       path /var/log/kubelet.log
  417.       pos_file /var/log/kubelet.log.pos
  418.       tag kubelet
  419.     </source>
  421.     # Example:
  422.     # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
  423.     <source>
  424.       @id kube-proxy.log
  425.       @type tail
  426.       format multiline
  427.       multiline_flush_interval 5s
  428.       format_firstline /^\w\d{4}/
  429.       format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  430.       time_format %m%d %H:%M:%S.%N
  431.       path /var/log/kube-proxy.log
  432.       pos_file /var/log/kube-proxy.log.pos
  433.       tag kube-proxy
  434.     </source>
  436.     # Example:
  437.     # I0204 07:00:19.604280       5 handlers.go:131] GET /api/v1/nodes: (1.624207ms) 200 [[kube-controller-manager/v1.1.3 (linux/amd64) kubernetes/6a81b50] 127.0.0.1:38266]
  438.     <source>
  439.       @id kube-apiserver.log
  440.       @type tail
  441.       format multiline
  442.       multiline_flush_interval 5s
  443.       format_firstline /^\w\d{4}/
  444.       format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  445.       time_format %m%d %H:%M:%S.%N
  446.       path /var/log/kube-apiserver.log
  447.       pos_file /var/log/kube-apiserver.log.pos
  448.       tag kube-apiserver
  449.     </source>
  451.     # Example:
  452.     # I0204 06:55:31.872680       5 servicecontroller.go:277] LB already exists and doesn't need update for service kube-system/kube-ui
  453.     <source>
  454.       @id kube-controller-manager.log
  455.       @type tail
  456.       format multiline
  457.       multiline_flush_interval 5s
  458.       format_firstline /^\w\d{4}/
  459.       format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  460.       time_format %m%d %H:%M:%S.%N
  461.       path /var/log/kube-controller-manager.log
  462.       pos_file /var/log/kube-controller-manager.log.pos
  463.       tag kube-controller-manager
  464.     </source>
  465.  
  466.     # Example:
  467.     # W0204 06:49:18.239674       7 reflector.go:245] pkg/scheduler/factory/factory.go:193: watch of *api.Service ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [2578313/2577886]) [2579312]
  468.     <source>
  469.       @id kube-scheduler.log
  470.       @type tail
  471.       format multiline
  472.       multiline_flush_interval 5s
  473.       format_firstline /^\w\d{4}/
  474.       format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  475.       time_format %m%d %H:%M:%S.%N
  476.       path /var/log/kube-scheduler.log
  477.       pos_file /var/log/kube-scheduler.log.pos
  478.       tag kube-scheduler
  479.     </source>
  480.  
  481.     # Example:
  482.     # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf
  483.     <source>
  484.       @id glbc.log
  485.       @type tail
  486.       format multiline
  487.       multiline_flush_interval 5s
  488.       format_firstline /^\w\d{4}/
  489.       format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  490.       time_format %m%d %H:%M:%S.%N
  491.       path /var/log/glbc.log
  492.       pos_file /var/log/glbc.log.pos
  493.       tag glbc
  494.     </source>
  495.  
  496.     # Example:
  497.     # TODO Add a proper example here.
  498.     <source>
  499.       @id cluster-autoscaler.log
  500.       @type tail
  501.       format multiline
  502.       multiline_flush_interval 5s
  503.       format_firstline /^\w\d{4}/
  504.       format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  505.       time_format %m%d %H:%M:%S.%N
  506.       path /var/log/cluster-autoscaler.log
  507.       pos_file /var/log/cluster-autoscaler.log.pos
  508.       tag cluster-autoscaler
  509.     </source>
  510.  
  511.     # Logs from systemd-journal for interesting services.
  512.     # TODO(random-liu): Remove this after cri container runtime rolls out.
  513.     <source>
  514.       @id journald-docker
  515.       @type systemd
  516.       matches [{ "_SYSTEMD_UNIT": "docker.service" }]
  517.       <storage>
  518.         @type local
  519.         persistent true
  520.         path /var/log/journald-docker.pos
  521.       </storage>
  522.       read_from_head true
  523.       tag docker
  524.     </source>
  525.  
  526.     <source>
  527.       @id journald-container-runtime
  528.       @type systemd
  529.       matches [{ "_SYSTEMD_UNIT": "{{ fluentd_container_runtime_service }}.service" }]
  530.       <storage>
  531.         @type local
  532.         persistent true
  533.         path /var/log/journald-container-runtime.pos
  534.       </storage>
  535.       read_from_head true
  536.       tag container-runtime
  537.     </source>
  538.  
  539.     <source>
  540.       @id journald-kubelet
  541.       @type systemd
  542.       matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
  543.       <storage>
  544.         @type local
  545.         persistent true
  546.         path /var/log/journald-kubelet.pos
  547.       </storage>
  548.       read_from_head true
  549.       tag kubelet
  550.     </source>
  551.  
  552.     <source>
  553.       @id journald-node-problem-detector
  554.       @type systemd
  555.       matches [{ "_SYSTEMD_UNIT": "node-problem-detector.service" }]
  556.       <storage>
  557.         @type local
  558.         persistent true
  559.         path /var/log/journald-node-problem-detector.pos
  560.       </storage>
  561.       read_from_head true
  562.       tag node-problem-detector
  563.     </source>
  564.  
  565.     <source>
  566.       @id kernel
  567.       @type systemd
  568.       matches [{ "_TRANSPORT": "kernel" }]
  569.       <storage>
  570.         @type local
  571.         persistent true
  572.         path /var/log/kernel.pos
  573.       </storage>
  574.       <entry>
  575.         fields_strip_underscores true
  576.         fields_lowercase true
  577.       </entry>
  578.       read_from_head true
  579.       tag kernel
  580.     </source>
  581.  
  582.   forward.input.conf: |-
  583.     # Takes the messages sent over TCP
  584.     <source>
  585.       @id forward
  586.       @type forward
  587.     </source>
  588.  
  589.   monitoring.conf: |-
  590.     # Prometheus Exporter Plugin
  591.     # input plugin that exports metrics
  592.     <source>
  593.       @id prometheus
  594.       @type prometheus
  595.     </source>
  596.  
  597.     <source>
  598.       @id monitor_agent
  599.       @type monitor_agent
  600.     </source>
  601.  
  602.     # input plugin that collects metrics from MonitorAgent
  603.     <source>
  604.       @id prometheus_monitor
  605.       @type prometheus_monitor
  606.       <labels>
  607.         host ${hostname}
  608.       </labels>
  609.     </source>
  610.  
  611.     # input plugin that collects metrics for output plugin
  612.     <source>
  613.       @id prometheus_output_monitor
  614.       @type prometheus_output_monitor
  615.       <labels>
  616.         host ${hostname}
  617.       </labels>
  618.     </source>
  619.  
  620.     # input plugin that collects metrics for in_tail plugin
  621.     <source>
  622.       @id prometheus_tail_monitor
  623.       @type prometheus_tail_monitor
  624.       <labels>
  625.         host ${hostname}
  626.       </labels>
  627.     </source>
  628.  
  629.   output.conf: |-
  630.     <match nginxtest1>
  631.       @id nginxtest1
  632.       @type elasticsearch
  633.       @log_level info
  634.       include_tag_key true
  635.       type_name _doc
  636.       host "#{ENV['OUTPUT_HOST']}"
  637.       port "#{ENV['OUTPUT_PORT']}"
  638.       scheme "#{ENV['OUTPUT_SCHEME']}"
  639.       ssl_version "#{ENV['OUTPUT_SSL_VERSION']}"
  640.       ssl_verify true
  641.       user "#{ENV['OUTPUT_USER']}"
  642.       password "#{ENV['OUTPUT_PASSWORD']}"
  643.       logstash_format true
  644.       logstash_prefix nginxtest1
  645.       reconnect_on_error true
  646.       <buffer>
  647.         @type file
  648.         path /var/log/fluentd-buffers/nginxtest1.buffer
  649.         flush_mode interval
  650.         retry_type exponential_backoff
  651.         flush_thread_count 2
  652.         flush_interval 5s
  653.         retry_forever
  654.         retry_max_interval 30
  655.         chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"
  656.         queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"
  657.         overflow_action block
  658.       </buffer>
  659.     </match>
  660.     <match httpdtest1>
  661.       @id httpdtest1
  662.       @type elasticsearch
  663.       @log_level info
  664.       include_tag_key true
  665.       type_name _doc
  666.       host "#{ENV['OUTPUT_HOST']}"
  667.       port "#{ENV['OUTPUT_PORT']}"
  668.       scheme "#{ENV['OUTPUT_SCHEME']}"
  669.       ssl_version "#{ENV['OUTPUT_SSL_VERSION']}"
  670.       ssl_verify true
  671.       user "#{ENV['OUTPUT_USER']}"
  672.       password "#{ENV['OUTPUT_PASSWORD']}"
  673.       logstash_format true
  674.       logstash_prefix httpdtest1
  675.       reconnect_on_error true
  676.       <buffer>
  677.         @type file
  678.         path /var/log/fluentd-buffers/httpdtest1.buffer
  679.         flush_mode interval
  680.         retry_type exponential_backoff
  681.         flush_thread_count 2
  682.         flush_interval 5s
  683.         retry_forever
  684.         retry_max_interval 30
  685.         chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"
  686.         queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"
  687.         overflow_action block
  688.       </buffer>
  689.     </match>
  690.     <match **>
  691.       @id elasticsearch
  692.       @type elasticsearch
  693.       @log_level info
  694.       include_tag_key true
  695.       type_name _doc
  696.       host "#{ENV['OUTPUT_HOST']}"
  697.       port "#{ENV['OUTPUT_PORT']}"
  698.       scheme "#{ENV['OUTPUT_SCHEME']}"
  699.       ssl_version "#{ENV['OUTPUT_SSL_VERSION']}"
  700.       ssl_verify true
  701.       user "#{ENV['OUTPUT_USER']}"
  702.       password "#{ENV['OUTPUT_PASSWORD']}"
  703.       logstash_format true
  704.       logstash_prefix "#{ENV['LOGSTASH_PREFIX']}"
  705.       reconnect_on_error true
  706.       <buffer>
  707.         @type file
  708.         path /var/log/fluentd-buffers/kubernetes.system.buffer
  709.         flush_mode interval
  710.         retry_type exponential_backoff
  711.         flush_thread_count 2
  712.         flush_interval 5s
  713.         retry_forever
  714.         retry_max_interval 30
  715.         chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"
  716.         queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"
  717.         overflow_action block
  718.       </buffer>
  719.     </match>
  720.  
  721. # extraVolumes:
  722. #   - name: es-certs
  723. #     secret:
  724. #       defaultMode: 420
  725. #       secretName: es-certs
  726. # extraVolumeMounts:
  727. #   - name: es-certs
  728. #     mountPath: /certs
  729. #     readOnly: true

  fluentd的配置是最关键也是最复杂的，能不能日志获取成功主要看fluentd配置的是否正确;配置的重点包括如下:

  1. 按照惯例将所有的镜像改成私服镜像；
  2. 连接Es集群的配置段，重点是这里连接的是client的service；
  3. 定义了fluentd的service
  4. 重点在configMap的配置，也就是fluentd的配置文件：定义缓存目录，因为fluentd也有一个缓存文件，日志先是写入到缓存文件然后再发送给ES，如果缓存文件设置的太小容易造成堵塞；
  5. fluentd默认的配置里面已经包含了容器的日志收集/var/log/containers/*.log 因为我们的容器只要是STDOUT输出的日志，默认都会在宿主机的/var/log/containers/Pod名称开始的日志名
    只需要将每个宿主机的这个目录挂载到fluentd容器里面，fluentd容器就会采集到宿主机上面运行的所有的容器日志
  6. 如果使用fluentd的默认配置的话，所有的日志都会收集到一个索引文件，也就是默认名称为logstash-年月日的索引文件，在Kibana上面创建索引后，所有的日志都汇总在一起，如果需要
    查看单独的POD的日志，就需要自己输入查询的条件，比如按照POD名或者容器名查询。
  7. 所以我们做了一个测试，我的k8s容器里面运行了nginx应用和apahce httpd应用。然后两个应用分布设置了两个不同的索引，使用不同的TAG；这个也是我们这个EFK日志解决方案的关键之处；
  8. 除了pod的日志，fluentd的配置文件中也默认收集了kube-controller-manager、 kube-scheduler、 kube-apiserver、kube-proxy、kubelet、etcd、docker等服务的日志；
  9. output.conf文件主要针对不同的源（比如nginx和httpd两个源）做了不同的索引前缀logstash_prefix，默认的logstash_prefix就是logstash
  10. 定义输出的时候还有几个参数可能需要优化，如flush_thread_count flush_interval chunk_limit_size等关于fluentd本地缓存的参数；调整的好就不至于日志d
  1. LAST DEPLOYED: Tue Apr 30 17:55:30 2019
  2. NAMESPACE: kube-system
  3. STATUS: DEPLOYED
  4.  
  5. RESOURCES:
  6. ==> v1/ConfigMap
  7. NAME                   DATA  AGE
  8. fluentd-elasticsearch  6     2d
  9.  
  10. ==> v1/ServiceAccount
  11. NAME                   SECRETS  AGE
  12. fluentd-elasticsearch  1        2d
  13.  
  14. ==> v1/ClusterRole
  15. NAME                   AGE
  16. fluentd-elasticsearch  2d
  17.  
  18. ==> v1/ClusterRoleBinding
  19. NAME                   AGE
  20. fluentd-elasticsearch  2d
  21.  
  22. ==> v1/Service
  23. NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)    AGE
  24. fluentd-elasticsearch  ClusterIP  10.200.108.50  <none>       24231/TCP  2d
  25.  
  26. ==> v1/DaemonSet
  27. NAME                   DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
  28. fluentd-elasticsearch  8        8        8      8           8          <none>         2d
  29.  
  30. ==> v1/Pod(related)
  31. NAME                         READY  STATUS   RESTARTS  AGE
  32. fluentd-elasticsearch-2trp8  1/1    RunningE0502 18:09:33.641998   29738 portforward.go:303] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:39598->127.0.0.1:43154: write tcp4 127.0.0.1:39598->127.0.0.1:43154: write: broken pipe
  33.   0         2d
  34. fluentd-elasticsearch-2xgtb  1/1    Running  0         2d
  35. fluentd-elasticsearch-589jc  1/1    Running  0         2d
  36. fluentd-elasticsearch-ctkv8  1/1    Running  0         2d
  37. fluentd-elasticsearch-d5dvz  1/1    Running  0         2d
  38. fluentd-elasticsearch-kgdxp  1/1    Running  0         2d
  39. fluentd-elasticsearch-r2c8h  1/1    Running  0         2d
  40. fluentd-elasticsearch-z8p7b  1/1    Running  0         2d
  41.  
  42. NOTES:
  43. 1. To verify that Fluentd has started, run:
  44.  
  45.   kubectl --namespace=kube-system get pods -l "app.kubernetes.io/name=fluentd-elasticsearch,app.kubernetes.io/instance=fluentd-elasticsearch"
  46.  
  47. THIS APPLICATION CAPTURES ALL CONSOLE OUTPUT AND FORWARDS IT TO elasticsearch . Anything that might be identifying,
  48. including things like IP addresses, container images, and object names will NOT be anonymized.
  49. 2. Get the application URL by running these commands:
  50.   export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=fluentd-elasticsearch,app.kubernetes.io/instance=fluentd-elasticsearch" -o jsonpath="{.items[0].metadata.name}")
  51.   echo "Visit http://127.0.0.1:8080 to use your application"
  52.   kubectl port-forward $POD_NAME 8080:80

  下面的ES集群的value.yaml配置

  1. # Default values for elasticsearch.
  2. # This is a YAML-formatted file.
  3. # Declare variables to be passed into your templates.
  4. appVersion: "6.7.0"
  5.  
  6. ## Define serviceAccount names for components. Defaults to component's fully qualified name.
  7. ##
  8. serviceAccounts:
  9.   client:
  10.     create: true
  11.     name:
  12.   master:
  13.     create: true
  14.     name:
  15.   data:
  16.     create: true
  17.     name:
  18.  
  19. ## Specify if a Pod Security Policy for node-exporter must be created
  20. ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  21. ##
  22. podSecurityPolicy:
  23.   enabled: false
  24.   annotations: {}
  25.     ## Specify pod annotations
  26.     ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#apparmor
  27.     ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp
  28.     ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#sysctl
  29.     ##
  30.     # seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  31.     # seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
  32.     # apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
  33.  
  34. image:
  35.   # repository: "k8s.harbor.maimaiti.site/system/elasticsearch-oss"
  36.   repository: "k8s.harbor.maimaiti.site/system/elasticsearch"
  37.   tag: "6.7.0"
  38.   pullPolicy: "IfNotPresent"
  39.   # If specified, use these secrets to access the image
  40.   # pullSecrets:
  41.   #   - registry-secret
  42.  
  43. testFramework:
  44.   image: "dduportal/bats"
  45.   tag: "0.4.0"
  46.  
  47. initImage:
  48.   repository: "busybox"
  49.   tag: "latest"
  50.   pullPolicy: "Always"
  51.  
  52. cluster:
  53.   name: "elasticsearch"
  54.   # If you want X-Pack installed, switch to an image that includes it, enable this option and toggle the features you want
  55.   # enabled in the environment variables outlined in the README
  56.   xpackEnable: true
  57.   # Some settings must be placed in a keystore, so they need to be mounted in from a secret.
  58.   # Use this setting to specify the name of the secret
  59.   # keystoreSecret: eskeystore
  60.   config: {}
  61.   # Custom parameters, as string, to be added to ES_JAVA_OPTS environment variable
  62.   additionalJavaOpts: ""
  63.   # Command to run at the end of deployment
  64.   bootstrapShellCommand: ""
  65.   env:
  66.     # IMPORTANT: https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#minimum_master_nodes
  67.     # To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting so that each master-eligible
  68.     # node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.
  69.     MINIMUM_MASTER_NODES: "2"
  70.   # List of plugins to install via dedicated init container
  71.   plugins: []
  72.     # - ingest-attachment
  73.     # - mapper-size
  74.  
  75. client:
  76.   name: client
  77.   replicas: 2
  78.   serviceType: ClusterIP
  79.   ## If coupled with serviceType = "NodePort", this will set a specific nodePort to the client HTTP port
  80.   # httpNodePort: 30920
  81.   loadBalancerIP: {}
  82.   loadBalancerSourceRanges: {}
  83. ## (dict) If specified, apply these annotations to the client service
  84. #  serviceAnnotations:
  85. #    example: client-svc-foo
  86.   heapSize: "512m"
  87.   # additionalJavaOpts: "-XX:MaxRAM=512m"
  88.   antiAffinity: "soft"
  89.   nodeAffinity: {}
  90.   nodeSelector: {}
  91.   tolerations: []
  92.   initResources: {}
  93.     # limits:
  94.     #   cpu: "25m"
  95.     #   # memory: "128Mi"
  96.     # requests:
  97.     #   cpu: "25m"
  98.     #   memory: "128Mi"
  99.   resources:
  100.     limits:
  101.       cpu: "1"
  102.       # memory: "1024Mi"
  103.     requests:
  104.       cpu: "25m"
  105.       memory: "512Mi"
  106.   priorityClassName: ""
  107.   ## (dict) If specified, apply these annotations to each client Pod
  108.   # podAnnotations:
  109.   #   example: client-foo
  110.   podDisruptionBudget:
  111.     enabled: false
  112.     minAvailable: 1
  113.     # maxUnavailable: 1
  114.   ingress:
  115.     enabled: false
  116.     # user: NAME
  117.     # password: PASSWORD
  118.     annotations: {}
  119.       # kubernetes.io/ingress.class: nginx
  120.       # kubernetes.io/tls-acme: "true"
  121.     path: /
  122.     hosts:
  123.       - chart-example.local
  124.     tls: []
  125.     #  - secretName: chart-example-tls
  126.     #    hosts:
  127.     #      - chart-example.local
  128.  
  129. master:
  130.   name: master
  131.   exposeHttp: false
  132.   replicas: 3
  133.   heapSize: "512m"
  134.   # additionalJavaOpts: "-XX:MaxRAM=512m"
  135.   persistence:
  136.     enabled: false
  137.     accessMode: ReadWriteOnce
  138.     name: data
  139.     size: "4Gi"
  140.     storageClass: "dynamic"
  141.   readinessProbe:
  142.     httpGet:
  143.       path: /_cluster/health?local=true
  144.       port: 9200
  145.     initialDelaySeconds: 5
  146.   antiAffinity: "soft"
  147.   nodeAffinity: {}
  148.   nodeSelector: {}
  149.   tolerations: []
  150.   initResources: {}
  151.     # limits:
  152.     #   cpu: "25m"
  153.     #   # memory: "128Mi"
  154.     # requests:
  155.     #   cpu: "25m"
  156.     #   memory: "128Mi"
  157.   resources:
  158.     limits:
  159.       cpu: "1"
  160.       # memory: "1024Mi"
  161.     requests:
  162.       cpu: "25m"
  163.       memory: "512Mi"
  164.   priorityClassName: ""
  165.   ## (dict) If specified, apply these annotations to each master Pod
  166.   # podAnnotations:
  167.   #   example: master-foo
  168.   podManagementPolicy: OrderedReady
  169.   podDisruptionBudget:
  170.     enabled: false
  171.     minAvailable: 2  # Same as `cluster.env.MINIMUM_MASTER_NODES`
  172.     # maxUnavailable: 1
  173.   updateStrategy:
  174.     type: OnDelete
  175.  
  176. data:
  177.   name: data
  178.   exposeHttp: false
  179.   replicas: 2
  180.   heapSize: "1536m"
  181.   # additionalJavaOpts: "-XX:MaxRAM=1536m"
  182.   persistence:
  183.     enabled: false
  184.     accessMode: ReadWriteOnce
  185.     name: data
  186.     size: "30Gi"
  187.     storageClass: "dynamic"
  188.   readinessProbe:
  189.     httpGet:
  190.       path: /_cluster/health?local=true
  191.       port: 9200
  192.     initialDelaySeconds: 5
  193.   terminationGracePeriodSeconds: 3600
  194.   antiAffinity: "soft"
  195.   nodeAffinity: {}
  196.   nodeSelector: {}
  197.   tolerations: []
  198.   initResources: {}
  199.     # limits:
  200.     #   cpu: "25m"
  201.     #   # memory: "128Mi"
  202.     # requests:
  203.     #   cpu: "25m"
  204.     #   memory: "128Mi"
  205.   resources:
  206.     limits:
  207.       cpu: "1"
  208.       # memory: "2048Mi"
  209.     requests:
  210.       cpu: "25m"
  211.       memory: "1536Mi"
  212.   priorityClassName: ""
  213.   ## (dict) If specified, apply these annotations to each data Pod
  214.   # podAnnotations:
  215.   #   example: data-foo
  216.   podDisruptionBudget:
  217.     enabled: false
  218.     # minAvailable: 1
  219.     maxUnavailable: 1
  220.   podManagementPolicy: OrderedReady
  221.   updateStrategy:
  222.     type: OnDelete
  223.   hooks:  # post-start and pre-stop hooks
  224.     drain:  # drain the node before stopping it and re-integrate it into the cluster after start
  225.       enabled: true
  226.  
  227. ## Sysctl init container to setup vm.max_map_count
  228. # see https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
  229. # and https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html#mlockall
  230. sysctlInitContainer:
  231.   enabled: true
  232. ## Additional init containers
  233. extraInitContainers: |

  关于ES集群的配置，主要包含如下:

  1. 经镜像都改为私服镜像；
  2. 定义ES集群的名字；
  3. 定义client服务Pod的名字，副本数，jvm内存使用量;
  4. 定义master服务Pod的名字,副本数，master一般为奇数，3或者5，最好开启持久化存储，使用ceph RBD的sc
  5. 定义data节点的POD的名字，副本数，最好开启持久化存储；注意数据节点这里开启了hooks。就是数据节点
    的停止和启动，需要有顺序，并且在脱离集群之前需要做一些配置；
    6、 kibana和fluented连接的都是elasticsearch-client的svc;9200是提供服务的端口，9300是集群端口。还有一个
    elasticsearch-discovery的无头服务的用途就是每个POD的名称需要固定下来，及时重启了ES节点的名字也不会变化。 
  1. [root@master-01 fluentd-elasticsearch]# helm status elasticsearch
  2. LAST DEPLOYED: Tue Apr 30 17:17:13 2019
  3. NAMESPACE: kube-system
  4. STATUS: DEPLOYED
  5.  
  6. RESOURCES:
  7. ==> v1/ConfigMap
  8. NAME                DATA  AGE
  9. elasticsearch       4     2d
  10. elasticsearch-test  1     2d
  11.  
  12. ==> v1/ServiceAccount
  13. NAME                  SECRETS  AGE
  14. elasticsearch-client  1        2d
  15. elasticsearch-data    1        2d
  16. elasticsearch-master  1        2d
  17.  
  18. ==> v1/Service
  19. NAME                     TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
  20. elasticsearch-client     ClusterIP  10.200.180.10  <none>       9200/TCP  2d
  21. elasticsearch-discovery  ClusterIP  None           <none>       9300/TCP  2d
  22.  
  23. ==> v1beta1/Deployment
  24. NAME                  DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
  25. elasticsearch-client  2        2        2           2          2d
  26.  
  27. ==> v1beta1/StatefulSet
  28. NAME                  DESIRED  CURRENT  AGE
  29. elasticsearch-data    2        2        2d
  30. elasticsearch-master  3        3        2d
  31.  
  32. ==> v1/Pod(related)
  33. NAME                                   READY  STATUS   RESTARTS  AGE
  34. elasticsearch-client-6bb89766f9-wfbxh  1/1    Running  0         2d
  35. elasticsearch-client-6bb89766f9-xvz6c  1/1    Running  0         2d
  36. elasticsearch-data-0                   1/1    Running  0         2d
  37. elasticsearch-data-1                   1/1    Running  0         2d
  38. elasticsearch-master-0                 1/1    Running  0         2d
  39. elasticsearch-master-1                 1/1    Running  0         2d
  40. elasticsearch-master-2                 1/1    Running  0         2d
  41.  
  42. NOTES:
  43. The elasticsearch cluster has been installed.
  44.  
  45. Elasticsearch can be accessed:
  46.  
  47.   * Within your cluster, at the following DNS name at port 9200:
  49.     elasticsearch-client.kube-system.svc
  50.  
  51.   * From outside the cluster, run these commands in the same shell:
  53.     export POD_NAME=$(kubectl get pods --namespace kube-system -l "app=elasticsearch,component=client,release=elasticsearch" -o jsonpath="{.items[0].metadata.name}")
  54.     echo "Visit http://127.0.0.1:9200 to use Elasticsearch"
  55.     kubectl port-forward --namespace kube-system $POD_NAME 9200:9200
  1. image:
  2.   # repository: "k8s.harbor.maimaiti.site/system/kibana-oss"
  3.   repository: "k8s.harbor.maimaiti.site/system/kibana"
  4.   tag: "6.7.0"
  5.   pullPolicy: "IfNotPresent"
  6.  
  7. testFramework:
  8.   image: "dduportal/bats"
  9.   tag: "0.4.0"
  10.  
  11. commandline:
  12.   args: []
  13.  
  14. env: {}
  15.   # All Kibana configuration options are adjustable via env vars.
  16.   # To adjust a config option to an env var uppercase + replace `.` with `_`
  17.   # Ref: https://www.elastic.co/guide/en/kibana/current/settings.html
  18.   #
  19.   # ELASTICSEARCH_URL: http://elasticsearch-client:9200
  20.   # SERVER_PORT: 5601
  21.   # LOGGING_VERBOSE: "true"
  22.   # SERVER_DEFAULTROUTE: "/app/kibana"
  23.  
  24. files:
  25.   kibana.yml:
  26.     ## Default Kibana configuration from kibana-docker.
  27.     server.name: kibana
  28.     server.host: "0"
  29.     elasticsearch.url: http://elasticsearch-client.kube-system:9200
  30.  
  31.     ## Custom config properties below
  32.     ## Ref: https://www.elastic.co/guide/en/kibana/current/settings.html
  33.     # server.port: 5601
  34.     # logging.verbose: "true"
  35.     # server.defaultRoute: "/app/kibana"
  36.  
  37. deployment:
  38.   annotations: {}
  39.  
  40. service:
  41.   type: NodePort
  42.   nodePort: 30001
  43.   # clusterIP: None
  44.   # portName: kibana-svc
  45.   externalPort: 443
  46.   internalPort: 5601
  47.   # authProxyPort: 5602 To be used with authProxyEnabled and a proxy extraContainer
  48.   ## External IP addresses of service
  49.   ## Default: nil
  50.   ##
  51.   # externalIPs:
  52.   # - 192.168.0.1
  53.   #
  54.   ## LoadBalancer IP if service.type is LoadBalancer
  55.   ## Default: nil
  56.   ##
  57.   # loadBalancerIP: 10.2.2.2
  58.   annotations: {}
  59.     # Annotation example: setup ssl with aws cert when service.type is LoadBalancer
  60.     # service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:EXAMPLE_CERT
  61.   labels: {}
  62.     ## Label example: show service URL in `kubectl cluster-info`
  63.     # kubernetes.io/cluster-service: "true"
  64.   ## Limit load balancer source ips to list of CIDRs (where available)
  65.   # loadBalancerSourceRanges: []
  66.   selector: {}
  67.  
  68. ingress:
  69.   enabled: false
  70.   # hosts:
  71.     # - kibana.localhost.localdomain
  72.     # - localhost.localdomain/kibana
  73.   # annotations:
  74.   #   kubernetes.io/ingress.class: nginx
  75.   #   kubernetes.io/tls-acme: "true"
  76.   # tls:
  77.     # - secretName: chart-example-tls
  78.     #   hosts:
  79.     #     - chart-example.local
  80.  
  81. serviceAccount:
  82.   # Specifies whether a service account should be created
  83.   create: true
  84.   # The name of the service account to use.
  85.   # If not set and create is true, a name is generated using the fullname template
  86.   # If set and create is false, the service account must be existing
  87.   name:
  88.  
  89. livenessProbe:
  90.   enabled: false
  91.   path: /status
  92.   initialDelaySeconds: 30
  93.   timeoutSeconds: 10
  94.  
  95. readinessProbe:
  96.   enabled: false
  97.   path: /status
  98.   initialDelaySeconds: 30
  99.   timeoutSeconds: 10
  100.   periodSeconds: 10
  101.   successThreshold: 5
  102.  
  103. # Enable an authproxy. Specify container in extraContainers
  104. authProxyEnabled: false
  105.  
  106. extraContainers: |
  107. # - name: proxy
  108. #   image: quay.io/gambol99/keycloak-proxy:latest
  109. #   args:
  110. #     - --resource=uri=/*
  111. #     - --discovery-url=https://discovery-url
  112. #     - --client-id=client
  113. #     - --client-secret=secret
  114. #     - --listen=0.0.0.0:5602
  115. #     - --upstream-url=http://127.0.0.1:5601
  116. #   ports:
  117. #     - name: web
  118. #       containerPort: 9090
  119.  
  120. extraVolumeMounts: []
  121.  
  122. extraVolumes: []
  123.  
  124. resources: {}
  125.   # limits:
  126.   #   cpu: 100m
  127.   #   memory: 300Mi
  128.   # requests:
  129.   #   cpu: 100m
  130.   #   memory: 300Mi
  131.  
  132. priorityClassName: ""
  133.  
  134. # Affinity for pod assignment
  135. # Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
  136. # affinity: {}
  137.  
  138. # Tolerations for pod assignment
  139. # Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
  140. tolerations: []
  141.  
  142. # Node labels for pod assignment
  143. # Ref: https://kubernetes.io/docs/user-guide/node-selection/
  144. nodeSelector: {}
  145.  
  146. podAnnotations: {}
  147. replicaCount: 1
  148. revisionHistoryLimit: 3
  149.  
  150. # Custom labels for pod assignment
  151. podLabels: {}
  152.  
  153. # To export a dashboard from a running Kibana 6.3.x use:
  154. # curl --user <username>:<password> -XGET https://kibana.yourdomain.com:5601/api/kibana/dashboards/export?dashboard=<some-dashboard-uuid> > my-dashboard.json
  155. # A dashboard is defined by a name and a string with the json payload or the download url
  156. dashboardImport:
  157.   enabled: false
  158.   timeout: 60
  159.   xpackauth:
  160.     enabled: true
  161.     username: myuser
  162.     password: mypass
  163.   dashboards: {}
  164.     # k8s: https://raw.githubusercontent.com/monotek/kibana-dashboards/master/k8s-fluentd-elasticsearch.json
  165.  
  166. # List of plugins to install using initContainer
  167. # NOTE : We notice that lower resource constraints given to the chart + plugins are likely not going to work well.
  168. plugins:
  169.   # set to true to enable plugins installation
  170.   enabled: false
  171.   # set to true to remove all kibana plugins before installation
  172.   reset: false
  173.   # Use <plugin_name,version,url> to add/upgrade plugin
  174.   values:
  175.     # - elastalert-kibana-plugin,1.0.1,https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.0.1/elastalert-kibana-plugin-1.0.1-6.4.2.zip
  176.     # - logtrail,0.1.31,https://github.com/sivasamyk/logtrail/releases/download/v0.1.31/logtrail-6.6.0-0.1.31.zip
  177.     # - other_plugin
  178.  
  179. persistentVolumeClaim:
  180.   # set to true to use pvc
  181.   enabled: false
  182.   # set to true to use you own pvc
  183.   existingClaim: false
  184.   annotations: {}
  185.  
  186.   accessModes:
  187.     - ReadWriteOnce
  188.   size: "5Gi"
  189.   ## If defined, storageClassName: <storageClass>
  190.   ## If set to "-", storageClassName: "", which disables dynamic provisioning
  191.   ## If undefined (the default) or set to null, no storageClassName spec is
  192.   ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  193.   ##   GKE, AWS & OpenStack)
  194.   ##
  195.   # storageClass: "-"
  196.  
  197. # default security context
  198. securityContext:
  199.   enabled: false
  200.   allowPrivilegeEscalation: false
  201.   runAsUser: 1000
  202.   fsGroup: 2000
  203.  
  204. extraConfigMapMounts: []
  205.   # - name: logtrail-configs
  206.   #   configMap: kibana-logtrail
  207.   #   mountPath: /usr/share/kibana/plugins/logtrail/logtrail.json
  208.   #   subPath: logtrail.json
  209.  
  210. # Add your own init container or uncomment and modify the given example.
  211. initContainers: {}
  212.   ## Don't start kibana till Elasticsearch is reachable.
  213.   ## Ensure that it is available at http://elasticsearch:9200
  214.   ##
  215.   # es-check:  # <- will be used as container name
  216.   #   image: "appropriate/curl:latest"
  217.   #   imagePullPolicy: "IfNotPresent"
  218.   #   command:
  219.   #     - "/bin/sh"
  220.   #     - "-c"
  221.   #     - |
  222.   #       is_down=true
  223.   #       while "$is_down"; do
  224.   #         if curl -sSf --fail-early --connect-timeout 5 http://elasticsearch:9200; then
  225.   #           is_down=false
  226.   #         else
  227.   #           sleep 5
  228.   #         fi
  229.   #       done

  关于kibana的配置就比较简单了，主要就是：

  1. 镜像换成私有仓库地址；
  2. kibana的配置文件中配置连接ES的地址，注意是连接elasticsearch-client的9200端口；
  3. kibana就部署一个deployment无状态应用就行。前面提到的ES就一定要是StatefulSet资源类型,fluentd一定要是DaemonSet资源类型；
  4. 可以定义一个ingress,因为kibana才是给用户提供访问的；
    kubernetes EFK日志管理系统
    kubernetes EFK日志管理系统
    kubernetes EFK日志管理系统
    kubernetes EFK日志管理系统
    kubernetes EFK日志管理系统
    kubernetes EFK日志管理系统
    kubernetes EFK日志管理系统
    kubernetes EFK日志管理系统

推荐关注我的个人微信公众号 “云时代IT运维”,周期性更新最新的应用运维类技术文档。关注虚拟化和容器技术、CI/CD、自动化运维等最新前沿运维技术和趋势；

kubernetes EFK日志管理系统

转载于:https://blog.51cto.com/zgui2000/2388379

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/木道寻08/article/detail/792686
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号