devbox
Monodyee
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

夜莺(Flashcat)V6监控(五)：夜莺监控k8s组件(上)

作者：Monodyee | 2024-02-28 11:47:20

赞

踩

夜莺监控k8s

   

目录

(一)Kubernetest监控体系

1.Kubernetes监控策略

(二)K8s-ApiServer组件监控

(1)我们先创建一个namespace来专门做夜莺监控采集指标

(2)创建认证授权信息rbac    

(3)使用prometheus-agent进行指标采集

① 创建Prometheus的配置文件

② 部署Prometehus Agent

(三)K8s-ControllerManager组件监控

(1)创建prometheus的配置文件

 (2)重新创建controller 的endpoints

(3)更改controller 的bind-address

(4)指标测试

(四)K8s-Scheduler组件监控

(1)创建prometheus的配置文件

 (2)配置Scheduler的service

 (3)重启prometheus-agent

  (4) 测试指标导入仪表盘

(五)K8s-Etcd组件监控

(1)更改etcd配置文件监听地址为0.0.0.0

(2)数据采集

(3)指标测试 

(六)K8s-kubelet组件监控

(1)配置Prometheus-agent configmap配置文件

(2)配置kubelet的service和endpoints

(3)测试指标

(七)K8s-KubeProxy组件监控

(1)配置Prometheus-agent configmap配置文件

(2)配置kube-proxy的endpoints

(3)更改kube-proxy的metricsbindAddress

(4)指标测试

最后的最后

    这一期我们讲一下夜莺来监控k8s组件的监控，因为k8s的组件复杂，内容多，所以我们分成上下两部分来学习，这一期我们先学习监控k8s的几大组件。首先我们先来了解认识一下k8s的架构和监控概述

(一)Kubernetest监控体系

    当我们谈及 Kubernetes 监控的时候，我们在谈论什么？显然是 Kubernetes 架构下的各个内容的监控，Kubernetes 所跑的环境、Kubernetes 本身、跑在 Kubernetes 上面的应用等等。Kubernetes 所跑的环境，可能是物理机、虚拟机，并且依赖底层的基础网络，Kubernetes 上面的应用，可能是业务应用程序，也可能是各类中间件、数据库，Kubernetes 本身，则包含很多组件，我们通过一张 Kubernetes 架构图来说明  。

     最左侧是 UI 层，包括页面 UI 以及命令行工具 kubectl，中间部分是 Kubernetes 控制面组件，右侧部分是工作负载节点，包含两个工作覆盖节点。

    k8s的这个架构我们可以大致分为两个模块来理解：

1.Master组件

    apiserver: 是Kubernetes集群中所有组件之间通信的中心组件，也是集群的前端接口。kube-apiserver负责验证和处理API请求，并将它们转发给其他组件。

    scheduler: Kubernetes Scheduler负责在Kubernetes集群中选择最合适的Node来运行新创建的Pod，考虑到节点的资源利用率、Pod的调度限制、网络位置等因素。

    controller-manager: Kubernetes Controller Manager包含多个控制器，负责监视并确保集群状态符合预期。例如，ReplicationController、NamespaceController、ServiceAccountController等等。

     etcd：etcd是Kubernetes的后端数据库，用于存储和管理Kubernetes集群状态信息，例如Pod、Service、ConfigMap等对象的配置和状态信息。

2.Slave-node组件

    kubelet：Kubelet是在每个Node上运行的代理服务，负责管理和监视该Node上的容器，并与kube-apiserver进行通信以保持节点状态最新。

    kube-proxy：Kubernetes Proxy负责为容器提供网络代理和负载均衡功能，使得容器可以访问其他Pod、Service等网络资源。

    Container Runtime：如Docker,rkt,runc等提供容器运行时环境

1.Kubernetes监控策略

     Kubernetes作为开源的容器编排工具，为用户提供了一个可以统一调度，统一管理的云操作系统。其解决如用户应用程序如何运行的问题。而一旦在生产环境中大量基于Kubernetes部署和管理应用程序后，作为系统管理员，还需要充分了解应用程序以及Kubernetes集群服务运行质量如何，通过对应用以及集群运行状态数据的收集和分析，持续优化和改进，从而提供一个安全可靠的生产运行环境。 这一小节中我们将讨论当使用Kubernetes时的监控策略该如何设计。

     从物理结构上讲Kubernetes主要用于整合和管理底层的基础设施资源，对外提供应用容器的自动化部署和管理能力，这些基础设施可能是物理机、虚拟机、云主机等等。因此，基础资源的使用直接影响当前集群的容量和应用的状态。在这部分，我们需要关注集群中各个节点的主机负载，CPU使用率、内存使用率、存储空间以及网络吞吐等监控指标。

     从自身架构上讲，kube-apiserver是Kubernetes提供所有服务的入口，无论是外部的客户端还是集群内部的组件都直接与kube-apiserver进行通讯。因此，kube-apiserver的并发和吞吐量直接决定了集群性能的好坏。其次，对于外部用户而言，Kubernetes是否能够快速的完成pod的调度以及启动，是影响其使用体验的关键因素。而这个过程主要由kube-scheduler负责完成调度工作，而kubelet完成pod的创建和启动工作。因此在Kubernetes集群本身我们需要评价其自身的服务质量，主要关注在Kubernetes的API响应时间，以及Pod的启动时间等指标上。  

     Kubernetes的最终目标还是需要为业务服务，因此我们还需要能够监控应用容器的资源使用情况。对于内置了对Prometheus支持的应用程序，也要支持从这些应用程序中采集内部的监控指标。最后，结合黑盒监控模式，对集群中部署的服务进行探测，从而当应用发生故障后，能够快速处理和恢复。

    综上所述，我们需要综合使用白盒监控和黑盒监控模式，建立从基础设施，Kubernetes核心组件，应用容器等全面的监控体系。

在白盒监控层面我们需要关注：

  • 基础设施层（Node）：为整个集群和应用提供运行时资源，需要通过各节点的kubelet获取节点的基本状态，同时通过在节点上部署Node Exporter获取节点的资源使用情况；
  • 容器基础设施（Container）：为应用提供运行时环境，Kubelet内置了对cAdvisor的支持，用户可以直接通过Kubelet组件获取给节点上容器相关监控指标；
  • 用户应用（Pod）：Pod中会包含一组容器，它们一起工作，并且对外提供一个（或者一组）功能。如果用户部署的应用程序内置了对Prometheus的支持，那么我们还应该采集这些Pod暴露的监控指标；
  • Kubernetes组件：获取并监控Kubernetes核心组件的运行状态，确保平台自身的稳定运行。

而在黑盒监控层面，则主要需要关注以下：

  • 内部服务负载均衡（Service）：在集群内，通过Service在集群暴露应用功能，集群内应用和应用之间访问时提供内部的负载均衡。通过Blackbox Exporter探测Service的可用性，确保当Service不可用时能够快速得到告警通知；
  • 外部访问入口（Ingress）：通过Ingress提供集群外的访问入口，从而可以使外部客户端能够访问到部署在Kubernetes集群内的服务。因此也需要通过Blackbox Exporter对Ingress的可用性进行探测，确保外部用户能够正常访问集群内的功能；

    说这么大家肯定有了一点初步的了解k8s的监控，那我们接下来趁热打铁，直接上实践，我们用夜莺来监控k8s的六大组件。

(二)K8s-ApiServer组件监控

    ApiServer 是 Kubernetes 架构中的核心，是所有 API 是入口，它串联所有的系统组件。

    为了方便监控管理 ApiServer，设计者们为它暴露了一系列的指标数据。当你部署完集群，默认会在default名称空间下创建一个名叫kubernetes的 service，它就是 ApiServer 的地址，当然也可以查看本机暴露的apiserver的端口ss -tlnp

  1. [root@k8s-master ~]# kubectl get service -A | grep kubernetes
  2. default        kubernetes                ClusterIP   10.96.0.1       <none>        443/TCP                  52d
  3.  
  4.  
  5. [root@k8s-master ~]# ss -tlpn | grep  apiserver
  6. LISTEN     0      128       [::]:6443                  [::]:*                   users:(("kube-apiserver",pid=2287,fd=7))

    但是当我们想要去获取抓紧metrics数据的时候，会发现我们抓紧不了，没有权限证书

  1. [root@k8s-master ~]# curl -s -k https://localhost:6443/metrics
  2. {
  3.   "kind": "Status",
  4.   "apiVersion": "v1",
  5.   "metadata": {},
  6.   "status": "Failure",
  7.   "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  8.   "reason": "Forbidden",
  9.   "details": {},
  10.   "code": 403
  11. }[root@k8s-master ~]#

    所以，要监控 ApiServer，采集到对应的指标，就需要先授权。为此，我们先准备认证信息。

(1)我们先创建一个namespace来专门做夜莺监控采集指标

[root@k8s-master ~]# kubectl create namespace flashcat

(2)创建认证授权信息rbac    

    这个yaml文件的意思就是我们创建一个账号sa名为categraf，然后给他绑定resources的verbs权限，让categraf这个账号有足够的权限来获取k8s的各个组件的指标采集

  1. vim apiserver-auth.yaml
  2.  
  3. ---
  4. apiVersion: rbac.authorization.k8s.io/v1
  5. kind: ClusterRole
  6. metadata:
  7.   name: categraf
  8. rules:
  9.   - apiGroups: [""]
  10.     resources:
  11.       - nodes
  12.       - nodes/metrics
  13.       - nodes/stats
  14.       - nodes/proxy
  15.       - services
  16.       - endpoints
  17.       - pods
  18.     verbs: ["get", "list", "watch"]
  19.   - apiGroups:
  20.       - extensions
  21.       - networking.k8s.io
  22.     resources:
  23.       - ingresses
  24.     verbs: ["get", "list", "watch"]
  25.   - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
  26.     verbs: ["get"]
  27. ---
  28. apiVersion: v1
  29. kind: ServiceAccount
  30. metadata:
  31.   name: categraf
  32.   namespace: flashcat
  33. ---
  34. apiVersion: rbac.authorization.k8s.io/v1
  35. kind: ClusterRoleBinding
  36. metadata:
  37.   name: categraf
  38. roleRef:
  39.   apiGroup: rbac.authorization.k8s.io
  40.   kind: ClusterRole
  41.   name: categraf
  42. subjects:
  43.   - kind: ServiceAccount
  44.     name: categraf
  45.     namespace: flashcat

(3)使用prometheus-agent进行指标采集

    支持 Kubernetes 服务发现的 agent 有不少，但是要说最原汁原味的还是 Prometheus 自身，Prometheus 新版本(v2.32.0)支持了 agent mode 模式，即把 Prometheus 进程当做采集器 agent，采集了数据之后通过 remote write 方式传给中心(这里使用早就准备好的 Nightingale 作为数据接收服务端)。那这里我就使用 Prometheus 的 agent mode 方式来采集 APIServer。

① 创建Prometheus的配置文件

    这里给大家解释一下这个配置文件的一些内容，给一些对普罗米修斯还不是很了解的小伙伴参考：

global：第一部分定义的一个名为global的模块

scrape_interval: 采集间隔

evaluation_interval: 评估间隔，用于控制数据的收集和处理频率

scrape_config: 第二部分定义的模块，用来配置Prometheus要监控的目标

job_name: 表示该配置是用于监控Kubernetets APIserver的

kubernetes_sd_configs: 表示指定了从kubernetest Service Discovery中获取目录对象的方式
此处使用了 role: endpoints 获取endpoint对象，也就是API server的ip地址和端口信息。

scheme：指定了网络通信协议是HTTPS

tls_config：参数指定了TLS证书的相关配置，包括是否验证服务器端证书等。

insecure_skip_verify：是一个bool类型的参数，如果为true，表示跳过对服务器端证书的验证。在生产环境中，不应该使用，因为会导致通信的不安全。正常情况下。我们需要在客户端上配置ca证书来验证服务器端证书的合法性。

authorization: 指定了认证信息的来源，这里使用了默认的kubernetest服务账号的Token。

relabel_configs：用于将原始数据标签进行变换，筛选出需要的目标数据

source_labels：定义了三个规则用来匹配标签。其中__meta_kubernetes_namespace表示Kubernetes命名空间，__meta_kubernetes_service_name表示服务名称，__meta_kubernetes_endpoint_port_name表示端口名称

action：指定该操作是保留keep，也就是保留符合指定正则表达式的标签

regex：使用正则表达式来对标签进行过滤，这里的正则表达式为default;kubernetes;http，表示要保留的目标是default命名空间下的kubernetes服务，并且端口是http。

通过这个relabel_configs块，Prometheus将采集到的来自default命名空间下的kubernetes服务，并且端口是http的数据进行保留，并将这些数据推送给后续的n9e夜莺

remote_write: 用于将普罗米修斯采集的数据写入外部存储。这里我们定义的是夜莺的地址。prometheus/v1/write是外部存储的接口路径。

  1. vim prometheus-cm.yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: prometheus-agent-conf
  6.   labels:
  7.     name: prometheus-agent-conf
  8.   namespace: flashcat
  9. data:
  10.   prometheus.yml: |-
  11.     global:
  12.       scrape_interval: 15s
  13.       evaluation_interval: 15s
  14.     scrape_configs:
  15.       - job_name: 'apiserver'
  16.         kubernetes_sd_configs:
  17.         - role: endpoints
  18.         scheme: https
  19.         tls_config:
  20.           insecure_skip_verify: true
  21.         authorization:
  22.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  23.         relabel_configs:
  24.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  25.           action: keep
  26.           regex: default;kubernetes;https
  27.     remote_write:
  28.     - url: 'http://192.168.120.17:17000/prometheus/v1/write'

② 部署Prometehus Agent

    这里我们使用deployment的方式部署

    其中--enable-feature=agent表示启动的是 agent 模式。

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: prometheus-agent
  5.   namespace: flashcat
  6.   labels:
  7.     app: prometheus-agent
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: prometheus-agent
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: prometheus-agent
  17.     spec:
  18.       serviceAccountName: categraf
  19.       containers:
  20.         - name: prometheus
  21.           image: prom/prometheus
  22.           args:
  23.             - "--config.file=/etc/prometheus/prometheus.yml"
  24.             - "--web.enable-lifecycle"
  25.             - "--enable-feature=agent"
  26.           ports:
  27.             - containerPort: 9090
  28.           resources:
  29.             requests:
  30.               cpu: 500m
  31.               memory: 500M
  32.             limits:
  33.               cpu: 1
  34.               memory: 1Gi
  35.           volumeMounts:
  36.             - name: prometheus-config-volume
  37.               mountPath: /etc/prometheus/
  38.             - name: prometheus-storage-volume
  39.               mountPath: /prometheus/
  40.       volumes:
  41.         - name: prometheus-config-volume
  42.           configMap:
  43.             defaultMode: 420
  44.             name: prometheus-agent-conf
  45.         - name: prometheus-storage-volume
  46.           emptyDir: {}

    查看是否部署成功

  1. [root@k8s-master ~]# kubectl get pod -n flashcat
  2. NAME                                READY   STATUS    RESTARTS   AGE
  3. prometheus-agent-7c8d7bc7bb-42djw   1/1     Running   0          115m

    然后可以到夜莺web页面查看指标 测试apiserver_request_total

    获取到了指标数据，后面就是合理利用指标做其他动作，比如构建面板、告警处理等。

 导入Apiserver的监控大盘，监控的json文件在categraf/apiserver-dash.json · GitHub

 直接复制导入json文件的内容即可

另外，Apiserver 的关键指标的含义也贴出来

  1. # HELP apiserver_request_duration_seconds [STABLE] Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.
  2. # TYPE apiserver_request_duration_seconds histogram
  3. apiserver响应的时间分布，按照url 和 verb 分类
  4. 一般按照instance和verb+时间 汇聚
  5.  
  6. # HELP apiserver_request_total [STABLE] Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code.
  7. # TYPE apiserver_request_total counter
  8. apiserver的请求总数，按照verb、 version、 group、resource、scope、component、 http返回码分类统计
  9.  
  10. # HELP apiserver_current_inflight_requests [STABLE] Maximal number of currently used inflight request limit of this apiserver per request kind in last second.
  11. # TYPE apiserver_current_inflight_requests gauge
  12. 最大并发请求数, 按mutating(非get list watch的请求)和readOnly(get list watch)分别限制
  13. 超过max-requests-inflight(默认值400)和max-mutating-requests-inflight(默认200)的请求会被限流
  14. apiserver变更时要注意观察，也是反馈集群容量的一个重要指标
  15.  
  16. # HELP apiserver_response_sizes [STABLE] Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component.
  17. # TYPE apiserver_response_sizes histogram
  18. apiserver 响应大小，单位byte, 按照verb、 version、 group、resource、scope、component分类统计
  19.  
  20. # HELP watch_cache_capacity [ALPHA] Total capacity of watch cache broken by resource type.
  21. # TYPE watch_cache_capacity gauge
  22. 按照资源类型统计的watch缓存大小
  23.  
  24. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  25. # TYPE process_cpu_seconds_total counter
  26. 每秒钟用户态和系统态cpu消耗时间, 计算apiserver进程的cpu的使用率
  27.  
  28. # HELP process_resident_memory_bytes Resident memory size in bytes.
  29. # TYPE process_resident_memory_bytes gauge
  30. apiserver的内存使用量(单位:Byte)
  31.  
  32. # HELP workqueue_adds_total [ALPHA] Total number of adds handled by workqueue
  33. # TYPE workqueue_adds_total counter
  34. apiserver中包含的controller的工作队列，已处理的任务总数
  35.  
  36. # HELP workqueue_depth [ALPHA] Current depth of workqueue
  37. # TYPE workqueue_depth gauge
  38. apiserver中包含的controller的工作队列深度，表示当前队列中要处理的任务的数量，数值越小越好 
  39. 例如APIServiceRegistrationController admission_quota_controller

(三)K8s-ControllerManager组件监控

     controller-manager 是 Kubernetes 控制面的组件，通常不太可能出问题，一般监控一下通用的进程指标就问题不大了，不过 controller-manager 确实也暴露了很多 /metrics 白盒指标，我们也一并梳理一下相关内容。

    监控思路跟上面一样，也是用Prometheus-Agent的方式进行采集指标

(1)创建prometheus的配置文件

    因为我们上面做apiserver的时候已经做了权限绑定和一些基础配置，所以这里我们直接添加Prometheus的配置文件添加job模块内容即可。

   这里我们可以直接打开之前创建的prometheus-cm的configmap配置文件 添加一个job关于controller-manager的即可

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: prometheus-agent-conf
  5.   labels:
  6.     name: prometheus-agent-conf
  7.   namespace: flashcat
  8. data:
  9.   prometheus.yml: |-
  10.     global:
  11.       scrape_interval: 15s
  12.       evaluation_interval: 15s
  13.  
  14.     scrape_configs:
  15.       - job_name: 'apiserver'
  16.         kubernetes_sd_configs:
  17.         - role: endpoints
  18.         scheme: https
  19.         tls_config:
  20.           insecure_skip_verify: true
  21.         authorization:
  22.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  23.         relabel_configs:
  24.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  25.           action: keep
  26.           regex: default;kubernetes;https
  27.     ## 这里添加即可以下内容即可
  28.       - job_name: 'controller-manager'
  29.         kubernetes_sd_configs:
  30.         - role: endpoints
  31.         scheme: https
  32.         tls_config:
  33.           insecure_skip_verify: true
  34.         authorization:
  35.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  36.         relabel_configs:
  37.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  38.           action: keep
  39.           regex: kube-system;kube-controller-manager;https-metrics
  40.  
  41.     remote_write:
  42.     - url: 'http://192.168.120.17:17000/prometheus/v1/write'

 (2)重新创建controller 的endpoints

    先查看一下自己有没有对应的controller-manager的endpoint 如果没有创建一个servece即可

    为什么要endpoint呢 因为我们上面Prometheus的采集规则role就是endpoint

  1. [root@k8s-master ~]# kubectl get endpoints -A | grep controller
  2.  
  3. 这里如果没有查询到就创建一个serveice文件
  4. vim controller-manager-service.yaml
  5. apiVersion: v1
  6. kind: Service
  7. metadata:
  8.   annotations:
  9.   labels:
  10.     k8s-app: kube-controller-manager
  11.   name: kube-controller-manager
  12.   namespace: kube-system
  13. spec:
  14.   clusterIP: None
  15.   ports:
  16.     - name: https-metrics
  17.       port: 10257
  18.       protocol: TCP
  19.       targetPort: 10257
  20.   selector:
  21.     component: kube-controller-manager
  22.   sessionAffinity: None
  23.   type: ClusterIP
  24.  
  25.  
  26. 运行yaml:  kubectl apply -f controller-manager-service.yaml

(3)更改controller 的bind-address

注意：如果你使用的kubeadm安装的k8s集群，需要把controller-manager的bind-address改为0.0.0.0

  1. [root@k8s-master ~]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml
  2. ....
  3. ....
  4. - --bind-address=0.0.0.0 ##找到bind-address 把127.0.0.1 改为 0.0.0.0
  5.

(4)指标测试

然后重启Prometheus-agent 的pod 重新加载Prometheus的配置文件的yaml

重启后先在夜莺的web页面查询指标，测试指标daemon_controller_rate_limiter_use

导入监控大盘，大盘链接:categraf/cm-dash.json at main · flashcatcloud/categraf · GitHub

 查看仪表盘 (怎么导入仪表盘的操作跟上面导入apiserver的仪表盘一样的，把json文件克隆进行即可)

controller-manager关键指标意思也贴出来

  1. # HELP rest_client_request_duration_seconds [ALPHA] Request latency in seconds. Broken down by verb and URL.
  2. # TYPE rest_client_request_duration_seconds histogram
  3. 请求apiserver的耗时分布，按照url+verb统计
  4.  
  5. # HELP cronjob_controller_cronjob_job_creation_skew_duration_seconds [ALPHA] Time between when a cronjob is scheduled to be run, and when the corresponding job is created
  6. # TYPE cronjob_controller_cronjob_job_creation_skew_duration_seconds histogram
  7. cronjob 创建到运行的时间分布
  8.  
  9. # HELP leader_election_master_status [ALPHA] Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
  10. # TYPE leader_election_master_status gauge
  11. 控制器的选举状态，0表示backup， 1表示master 
  12.  
  13. # HELP node_collector_zone_health [ALPHA] Gauge measuring percentage of healthy nodes per zone.
  14. # TYPE node_collector_zone_health gauge
  15. 每个zone的健康node占比
  16.  
  17. # HELP node_collector_zone_size [ALPHA] Gauge measuring number of registered Nodes per zones.
  18. # TYPE node_collector_zone_size gauge
  19. 每个zone的node数
  20.  
  21. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  22. # TYPE process_cpu_seconds_total counter
  23. cpu使用量（也可以理解为cpu使用率）
  24.  
  25. # HELP process_open_fds Number of open file descriptors.
  26. # TYPE process_open_fds gauge
  27. 控制器打开的fd数
  28.  
  29. # HELP pv_collector_bound_pv_count [ALPHA] Gauge measuring number of persistent volume currently bound
  30. # TYPE pv_collector_bound_pv_count gauge
  31. 当前绑定的pv数量
  32.  
  33. # HELP pv_collector_unbound_pvc_count [ALPHA] Gauge measuring number of persistent volume claim currently unbound
  34. # TYPE pv_collector_unbound_pvc_count gauge
  35. 当前没有绑定的pvc数量 
  36.  
  37.  
  38. # HELP pv_collector_bound_pvc_count [ALPHA] Gauge measuring number of persistent volume claim currently bound
  39. # TYPE pv_collector_bound_pvc_count gauge
  40. 当前绑定的pvc数量
  41.  
  42. # HELP pv_collector_total_pv_count [ALPHA] Gauge measuring total number of persistent volumes
  43. # TYPE pv_collector_total_pv_count gauge
  44. pv总数量
  45.  
  46.  
  47. # HELP workqueue_adds_total [ALPHA] Total number of adds handled by workqueue
  48. # TYPE workqueue_adds_total counter
  49. 各个controller已接受的任务总数
  50. 与apiserver的workqueue_adds_total指标类似
  51.  
  52. # HELP workqueue_depth [ALPHA] Current depth of workqueue
  53. # TYPE workqueue_depth gauge
  54. 各个controller队列深度，表示一个controller中的任务的数量
  55. 与apiserver的workqueue_depth类似，这个是指各个controller中队列的深度，数值越小越好
  56.  
  57. # HELP workqueue_queue_duration_seconds [ALPHA] How long in seconds an item stays in workqueue before being requested.
  58. # TYPE workqueue_queue_duration_seconds histogram
  59. 任务在队列中的等待耗时,按照控制器分别统计
  60.  
  61. # HELP workqueue_work_duration_seconds [ALPHA] How long in seconds processing an item from workqueue takes.
  62. # TYPE workqueue_work_duration_seconds histogram
  63. 任务出队到被处理完成的时间，按照控制分别统计
  64.  
  65. # HELP workqueue_retries_total [ALPHA] Total number of retries handled by workqueue
  66. # TYPE workqueue_retries_total counter
  67. 任务进入队列重试的次数
  68.  
  69. # HELP workqueue_longest_running_processor_seconds [ALPHA] How many seconds has the longest running processor for workqueue been running.
  70. # TYPE workqueue_longest_running_processor_seconds gauge
  71. 正在处理的任务中，最长耗时任务的处理时间
  72.  
  73. # HELP endpoint_slice_controller_syncs [ALPHA] Number of EndpointSlice syncs
  74. # TYPE endpoint_slice_controller_syncs counter
  75. endpoint_slice 同步的数量(1.20以上)
  76.  
  77. # HELP get_token_fail_count [ALPHA] Counter of failed Token() requests to the alternate token source
  78. # TYPE get_token_fail_count counter
  79. 获取token失败的次数
  80.  
  81. # HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
  82. # TYPE go_memstats_gc_cpu_fraction gauge
  83. controller gc的cpu使用率

(四)K8s-Scheduler组件监控

    scheduler 是 Kubernetes 的控制面组件，负责调度对象到合适的 node 上，会有一系列的规则计算和筛选，重点关注调度相关的指标。相关监控数据也是通过 /metrics 接口暴露，scheduler的暴露的端口是10259

    接下来就是采集数据了，我们还是使用 prometheus agent 来拉取数据，原汁原味的，只要在上一篇文章提供的 configmap 中增加 scheduler 相关的配置job即可

(1)创建prometheus的配置文件

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: prometheus-agent-conf
  5.   labels:
  6.     name: prometheus-agent-conf
  7.   namespace: flashcat
  8. data:
  9.   prometheus.yml: |-
  10.     global:
  11.       scrape_interval: 15s
  12.       evaluation_interval: 15s
  13.  
  14.     scrape_configs:
  15.       - job_name: 'apiserver'
  16.         kubernetes_sd_configs:
  17.         - role: endpoints
  18.         scheme: https
  19.         tls_config:
  20.           insecure_skip_verify: true
  21.         authorization:
  22.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  23.         relabel_configs:
  24.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  25.           action: keep
  26.           regex: default;kubernetes;https
  27.  
  28.       - job_name: 'controller-manager'
  29.         kubernetes_sd_configs:
  30.         - role: endpoints
  31.         scheme: https
  32.         tls_config:
  33.           insecure_skip_verify: true
  34.         authorization:
  35.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  36.         relabel_configs:
  37.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  38.           action: keep
  39.           regex: kube-system;kube-controller-manager;https-metrics
  40.   ##添加以下scheduler的job即可
  41.       - job_name: 'scheduler'
  42.         kubernetes_sd_configs:
  43.         - role: endpoints
  44.         scheme: https
  45.         tls_config:
  46.           insecure_skip_verify: true
  47.         authorization:
  48.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  49.         relabel_configs:
  50.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  51.           action: keep
  52.           regex: kube-system;kube-scheduler;https
  53.  
  54.     remote_write:
  55.     - url: 'http://192.168.120.17:17000/prometheus/v1/write'

 (2)配置Scheduler的service

    跟上面一样，首先我们要查看有没有相关的sheduler的endpoint，如果没有我们就要创建一个service来暴露

  1. [root@k8s-master ~]# kubectl get endpoints -A | grep schedu
  2. ## 如果没有我们就创建一个service的yaml
  3.  
  4. vim scheduler-service.yaml
  5. apiVersion: v1
  6. kind: Service
  7. metadata:
  8.   labels:
  9.     k8s-app: kube-scheduler
  10.   name: kube-scheduler
  11.   namespace: kube-system
  12. spec:
  13.   clusterIP: None
  14.   ports:
  15.     - name: https
  16.       port: 10259
  17.       protocol: TCP
  18.       targetPort: 10259
  19.   selector:
  20.     component: kube-scheduler
  21.   sessionAffinity: None
  22.   type: ClusterIP

 (3)重启prometheus-agent

    配置更新完configmap后要重新去apply一下configmap或者你edit更改，更改完成后如果还是无法获取指标就重启一下Prometheus-agent的pod 重新apply一下就行 ,或者curl -X POST "http://<PROMETHEUS_IP>:9090/-/reload"重载 Prometheus，这里的prometheus的ip是pod的IP，这个ip你要查看Prometheus pod的IP 可以使用kubectl get pod -o wide -n flashcat 即可。

(注意这里如果你的k8s是kubeadm安装的，也要去scheduler的manifests文件把bind-address更改为0.0.0.0)

  1. [root@k8s-master manifests]# vim /etc/kubernetes/manifests/kube-scheduler.yaml
  2. ......
  3. ......
  4. ......
  5. - --bind-address=0.0.0.0 ##找到这行更改为0.0.0.0即可

  (4) 测试指标导入仪表盘

    测试指标scheduler_scheduler_cache_size

    导入监控大盘，大盘json链接: categraf/scheduler-dash.json at main · · GitHub

这里也贴出常用scheduler关键指标意思:

  1. # HELP rest_client_request_duration_seconds [ALPHA] Request latency in seconds. Broken down by verb and URL.
  2. # TYPE rest_client_request_duration_seconds histogram
  3. 请求apiserver的延迟分布
  4.  
  5. # HELP rest_client_requests_total [ALPHA] Number of HTTP requests, partitioned by status code, method, and host.
  6. # TYPE rest_client_requests_total counter
  7. 请求apiserver的总数 ,按照host code method 统计
  8.  
  9. # HELP leader_election_master_status [ALPHA] Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
  10. # TYPE leader_election_master_status gauge
  11. 调度器的选举状态，0表示backup， 1表示master
  12.  
  13. # HELP scheduler_queue_incoming_pods_total [STABLE] Number of pods added to scheduling queues by event and queue type.
  14. # TYPE scheduler_queue_incoming_pods_total counter
  15. 进入调度队列的pod数
  16.  
  17. # HELP scheduler_preemption_attempts_total [STABLE] Total preemption attempts in the cluster till now
  18. # TYPE scheduler_preemption_attempts_total counter
  19. 调度器驱逐容器的次数
  20.  
  21. # HELP scheduler_scheduler_cache_size [ALPHA] Number of nodes, pods, and assumed (bound) pods in the scheduler cache.
  22. # TYPE scheduler_scheduler_cache_size gauge
  23. 调度器cache中node pod和绑定pod的数目
  24.  
  25. # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ.
  26. # TYPE scheduler_pending_pods gauge
  27. 调度pending的pod数量，按照queue type分别统计
  28.  
  29. # HELP scheduler_plugin_execution_duration_seconds [ALPHA] Duration for running a plugin at a specific extension point.
  30. # TYPE scheduler_plugin_execution_duration_seconds histogram
  31. 调度插件在每个扩展点的执行时间，按照extension_point+plugin+status 分别统计
  32.  
  33. # HELP scheduler_e2e_scheduling_duration_seconds [ALPHA] (Deprecated since 1.23.0) E2e scheduling latency in seconds (scheduling algorithm + binding). This metric is replaced by scheduling_attempt_duration_seconds.
  34. # TYPE scheduler_e2e_scheduling_duration_seconds histogram
  35. 调度延迟分布，1.23.0 以后会被scheduling_attempt_duration_seconds替代
  36.  
  37. # HELP scheduler_framework_extension_point_duration_seconds [STABLE] Latency for running all plugins of a specific extension point.
  38. # TYPE scheduler_framework_extension_point_duration_seconds histogram
  39. 调度框架的扩展点延迟分布,按extension_point(扩展点Bind Filter Permit PreBind/PostBind PreFilter/PostFilter Reseve)
  40. +profile（调度器）+ status(调度成功) 统计
  41.  
  42. # HELP scheduler_pod_scheduling_attempts [STABLE] Number of attempts to successfully schedule a pod.
  43. # TYPE scheduler_pod_scheduling_attempts histogram
  44. pod调度成功前，调度重试的次数分布
  45.  
  46. # HELP scheduler_schedule_attempts_total [STABLE] Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem.
  47. # TYPE scheduler_schedule_attempts_total counter
  48. 按照调度结果统计的调度重试次数。 "unschedulable" 表示无法调度，"error"表示调度器内部错误
  49.  
  50. # HELP scheduler_scheduler_goroutines [ALPHA] Number of running goroutines split by the work they do such as binding.
  51. # TYPE scheduler_scheduler_goroutines gauge
  52. 按照功能(binding filter之类)统计的goroutine数量
  53.  
  54. # HELP scheduler_scheduling_algorithm_duration_seconds [ALPHA] Scheduling algorithm latency in seconds
  55. # TYPE scheduler_scheduling_algorithm_duration_seconds histogram
  56. 调度算法的耗时分布
  57.  
  58. # HELP scheduler_scheduling_attempt_duration_seconds [STABLE] Scheduling attempt latency in seconds (scheduling algorithm + binding)
  59. # TYPE scheduler_scheduling_attempt_duration_seconds histogram
  60. 调度算法+binding的耗时分布
  61.  
  62. # HELP scheduler_scheduler_goroutines [ALPHA] Number of running goroutines split by the work they do such as binding.
  63. # TYPE scheduler_scheduler_goroutines gauge
  64. 调度器的goroutines数目

(五)K8s-Etcd组件监控

    ETCD 是 Kubernetes 控制面的重要组件和依赖，Kubernetes 的各类信息都存储在 ETCD 中，所以监控 ETCD 就显得尤为重要。ETCD 在 Kubernetes 中的架构角色如下（只与 APIServer 交互）：

ETCD 是一个类似 Zookeeper 的产品，通常由多个节点组成集群，节点之间使用 raft 协议保证一致性。ETCD 具有以下特点：

  • 每个节点都有一个角色状态，Follower、Candidate、Leader
  • 如果 Follower 找不到当前 Leader 节点的时候，就会变成 Candidate
  • 选举系统会从 Candidate 中选出 Leader
  • 所有的写操作都通过 Leader 进行
  • 一旦 Leader 从大多数 Follower 拿到 ack，该写操作就被认为是“已提交”状态
  • 只要大多数节点存活，整个 ETCD 就是存活的，个别节点挂掉不影响整个集群的可用性
  • ETCD 使用 restful 风格的 HTTP API 来操作，这使得 ETCD 的使用非常方便，这也是 ETCD 流行的一个关键因素

ETCD 这么云原生的组件，显然是内置支持了 /metrics 接口的，不过 ETCD 很讲求安全性，默认的 2379 端口的访问是要用证书的，我来测试一下先：

  1. [root@tt-fc-dev01.nj ~]# curl -k https://localhost:2379/metrics
  2. curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
  3.  
  4. [root@tt-fc-dev01.nj ~]# ls /etc/kubernetes/pki/etcd
  5. ca.crt  ca.key  healthcheck-client.crt  healthcheck-client.key  peer.crt  peer.key  server.crt  server.key
  6.  
  7. [root@tt-fc-dev01.nj ~]# curl -s --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key https://localhost:2379/metrics | head -n 6
  8. # HELP etcd_cluster_version Which version is running. 1 for 'cluster_version' label with current cluster version
  9. # TYPE etcd_cluster_version gauge
  10. etcd_cluster_version{cluster_version="3.5"} 1
  11. # HELP etcd_debugging_auth_revision The current revision of auth store.
  12. # TYPE etcd_debugging_auth_revision gauge
  13. etcd_debugging_auth_revision 1

    使用 kubeadm 安装的 Kubernetes 集群，相关证书是在 /etc/kubernetes/pki/etcd 目录下，为 curl 命令指定相关证书，是可以访问的通的。后面使用 Categraf 的 prometheus 插件直接采集相关数据即可。

    不过指标数据实在没必要做这么强的安全管控，整的挺麻烦，实际上，ETCD 也确实提供了另一个端口来获取指标数据，无需走这套证书认证机制。

(1)更改etcd配置文件监听地址为0.0.0.0

    这里我们首先去etcd的manifests文件更改监听metrics地址

  1. [root@k8s-master manifests]# vim /etc/kubernetes/manifests/etcd.yaml
  2.  
  3. ......
  4. ......
  5. - --listen-metrics-urls=http://0.0.0.0:2381  ##找到lisetn这行，把地址改为0.0.0.0

    这样改完以后我们就能直接通过2381端口来抓取metrics数据

  1. [root@tt-fc-dev01.nj ~]# curl -s localhost:2381/metrics | head -n 6
  2. # HELP etcd_cluster_version Which version is running. 1 for 'cluster_version' label with current cluster version
  3. # TYPE etcd_cluster_version gauge
  4. etcd_cluster_version{cluster_version="3.5"} 1
  5. # HELP etcd_debugging_auth_revision The current revision of auth store.
  6. # TYPE etcd_debugging_auth_revision gauge
  7. etcd_debugging_auth_revision 1

(2)数据采集

ETCD 的数据采集通常使用 3 种方式：

  • 使用 ETCD 所在宿主的 agent 直接来采集，因为 ETCD 是个静态 Pod，采用的 hostNetwork，所以 agent 直接连上去采集即可
  • 把采集器和 ETCD 做成 sidecar 的模式，ETCD 的使用其实已经越来越广泛，不只是给 Kubernetes 使用，很多业务也在使用，在 Kubernetes 里创建和管理 ETCD 也是很常见的做法，sidecar 这种模式非常干净，随着 ETCD 创建而创建，随着其销毁而销毁，省事
  • 使用服务发现机制，在中心端部署采集器，就像之前的文章中介绍的 APIServer、Controller-manager、Scheduler 等的做法，使用 Prometheus agent mode 采集监控数据，当然，这种方式的话需要有对应的 etcd endpoint，你可以自行检查一下 kubectl get endpoints -n kube-system ，如果没有，创建一下即可
  1.  
  2. [root@k8s-master manifests]# kubectl get endpoints -A | grep etcd
  3. ##如果没有对应的endpoint 就创建一个service
  4.  
  5. vim etcd-service.yaml
  6. apiVersion: v1
  7. kind: Service
  8. metadata:
  9.   namespace: kube-system
  10.   name: etcd
  11.   labels:
  12.     k8s-app: etcd
  13. spec:
  14.   selector:
  15.     component: etcd
  16.   type: ClusterIP
  17.   clusterIP: None
  18.   ports:
  19.     - name: http
  20.       port: 2381
  21.       targetPort: 2381
  22.       protocol: TCP

    更改我们直接写的configmap的Prometheus的配置文件，添加etcd job字段模块即可

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: prometheus-agent-conf
  5.   labels:
  6.     name: prometheus-agent-conf
  7.   namespace: flashcat
  8. data:
  9.   prometheus.yml: |-
  10.     global:
  11.       scrape_interval: 15s
  12.       evaluation_interval: 15s
  13.  
  14.     scrape_configs:
  15.       - job_name: 'apiserver'
  16.         kubernetes_sd_configs:
  17.         - role: endpoints
  18.         scheme: https
  19.         tls_config:
  20.           insecure_skip_verify: true
  21.         authorization:
  22.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  23.         relabel_configs:
  24.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  25.           action: keep
  26.           regex: default;kubernetes;https
  27.  
  28.       - job_name: 'controller-manager'
  29.         kubernetes_sd_configs:
  30.         - role: endpoints
  31.         scheme: https
  32.         tls_config:
  33.           insecure_skip_verify: true
  34.         authorization:
  35.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  36.         relabel_configs:
  37.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  38.           action: keep
  39.           regex: kube-system;kube-controller-manager;https-metrics
  40.       - job_name: 'scheduler'
  41.         kubernetes_sd_configs:
  42.         - role: endpoints
  43.         scheme: https
  44.         tls_config:
  45.           insecure_skip_verify: true
  46.         authorization:
  47.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  48.         relabel_configs:
  49.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  50.           action: keep
  51.           regex: kube-system;kube-scheduler;https
  52.  ##   添加以下etcd字段
  53.       - job_name: 'etcd'
  54.         kubernetes_sd_configs:
  55.         - role: endpoints
  56.         scheme: http
  57.         relabel_configs:
  58.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  59.           action: keep
  60.           regex: kube-system;etcd;http
  61.     remote_write:
  62.     - url: 'http://192.168.120.17:17000/prometheus/v1/write'

(3)指标测试 

    更改完成后 重新加载yaml文件和Prometheus-agent，然后打开夜莺的web页面指标查询，测试指标是否查询得到：etcd_cluster_version

  查询到指标后，导入监控仪表盘，仪表盘json地址：categraf/etcd-dash. · fl/categraf · GitHub

  复制json文件克隆到仪表盘

 ETCD关键指标意思含义:

  1. # HELP etcd_server_is_leader Whether or not this member is a leader. 1 if is, 0 otherwise.
  2. # TYPE etcd_server_is_leader gauge
  3. etcd leader 表示 ，1 leader 0 learner
  4.  
  5. # HELP etcd_server_health_success The total number of successful health checks
  6. # TYPE etcd_server_health_success counter
  7. etcd server 健康检查成功次数
  8.  
  9. # HELP etcd_server_health_failures The total number of failed health checks
  10. # TYPE etcd_server_health_failures counter
  11. etcd server 健康检查失败次数
  12.  
  13. # HELP etcd_disk_defrag_inflight Whether or not defrag is active on the member. 1 means active, 0 means not.
  14. # TYPE etcd_disk_defrag_inflight gauge
  15. 是否启动数据压缩，1表示压缩，0表示没有启动压缩
  16.  
  17. # HELP etcd_server_snapshot_apply_in_progress_total 1 if the server is applying the incoming snapshot. 0 if none.
  18. # TYPE etcd_server_snapshot_apply_in_progress_total gauge
  19. 是否再快照中，1 快照中，0 没有
  20.  
  21. # HELP etcd_server_leader_changes_seen_total The number of leader changes seen.
  22. # TYPE etcd_server_leader_changes_seen_total counter
  23. 集群leader切换的次数
  24.  
  25. # HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
  26. # TYPE grpc_server_handled_total counter
  27. grpc 调用总数
  28.  
  29. # HELP etcd_disk_wal_fsync_duration_seconds The latency distributions of fsync called by WAL.
  30. # TYPE etcd_disk_wal_fsync_duration_seconds histogram
  31. etcd wal同步耗时
  32.  
  33. # HELP etcd_server_proposals_failed_total The total number of failed proposals seen.
  34. # TYPE etcd_server_proposals_failed_total counter
  35. etcd proposal（提议）失败总次数(proposal就是完成raft协议的一次请求）
  36.  
  37. # HELP etcd_server_proposals_pending The current number of pending proposals to commit.
  38. # TYPE etcd_server_proposals_pending gauge
  39. etcd proposal（提议）pending总次数（proposal就是完成raft协议的一次请求）
  40.  
  41. # HELP etcd_server_read_indexes_failed_total The total number of failed read indexes seen.
  42. # TYPE etcd_server_read_indexes_failed_total counter
  43. 读取索引失败的次数统计(v3索引为所有key都建了索引，索引是为了加快range操作)
  44.  
  45. # HELP etcd_server_slow_read_indexes_total The total number of pending read indexes not in sync with leader's or timed out read index requests.
  46. # TYPE etcd_server_slow_read_indexes_total counter
  47. 读取到过期索引或者读取超时的次数
  48.  
  49. # HELP etcd_server_quota_backend_bytes Current backend storage quota size in bytes.
  50. # TYPE etcd_server_quota_backend_bytes gauge
  51. 当前后端的存储quota（db大小的上限）
  52. 通过参数quota-backend-bytes调整大小，默认2G，官方建议不超过8G
  53.  
  54. # HELP etcd_mvcc_db_total_size_in_bytes Total size of the underlying database physically allocated in bytes.
  55. # TYPE etcd_mvcc_db_total_size_in_bytes gauge
  56. etcd 分配的db大小（使用量大小+空闲大小）
  57.  
  58. # HELP etcd_mvcc_db_total_size_in_use_in_bytes Total size of the underlying database logically in use in bytes.
  59. # TYPE etcd_mvcc_db_total_size_in_use_in_bytes gauge
  60. etcd db的使用量大小
  61.  
  62. # HELP etcd_mvcc_range_total Total number of ranges seen by this member.
  63. # TYPE etcd_mvcc_range_total counter
  64. etcd执行range的数量
  65.  
  66. # HELP etcd_mvcc_put_total Total number of puts seen by this member.
  67. # TYPE etcd_mvcc_put_total counter
  68. etcd执行put的数量
  69.  
  70. # HELP etcd_mvcc_txn_total Total number of txns seen by this member.
  71. # TYPE etcd_mvcc_txn_total counter
  72. etcd实例执行事务的数量
  73.  
  74. # HELP etcd_mvcc_delete_total Total number of deletes seen by this member.
  75. # TYPE etcd_mvcc_delete_total counter
  76. etcd实例执行delete操作的数量
  77.  
  78. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  79. # TYPE process_cpu_seconds_total counter
  80. etcd cpu使用量
  81.  
  82. # HELP process_resident_memory_bytes Resident memory size in bytes.
  83. # TYPE process_resident_memory_bytes gauge
  84. etcd 内存使用量
  85.  
  86. # HELP process_open_fds Number of open file descriptors.
  87. # TYPE process_open_fds gauge
  88. etcd 打开的fd数目

(六)K8s-kubelet组件监控

    接下来我们监控的就是k8s第二个模块salve-node的组件，kubelet监听有两个固定的端口，一个是10248，一个是10250，可以用ss -ntlp | grep kubelet命令查看。

    10248是健康检测的端口，检测节点状态，可以使用curl localhost:10248/healthz查看

  1. [root@k8s-master ~]# curl localhost:10248/healthz
  2. ok

    10250是kubelet默认的端口，/metrics接口就是在这个端口下，但是你不能直接通过这个端口获取metrics的数据，因为他有认证机制。 这一期我们还是讲使用Prometheus-agent的方式来采集metrics数据，下一期我们来通过认证使用daemonset的方式部署categraf来采集。

(1)配置Prometheus-agent configmap配置文件

    跟上面的操作一样，在configmap下面添加名为kubelet的job字段即可，然后重新加载configmap的yaml文件

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: prometheus-agent-conf
  5.   labels:
  6.     name: prometheus-agent-conf
  7.   namespace: flashcat
  8. data:
  9.   prometheus.yml: |-
  10.     global:
  11.       scrape_interval: 15s
  12.       evaluation_interval: 15s
  13.  
  14.     scrape_configs:
  15.       - job_name: 'apiserver'
  16.         kubernetes_sd_configs:
  17.         - role: endpoints
  18.         scheme: https
  19.         tls_config:
  20.           insecure_skip_verify: true
  21.         authorization:
  22.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  23.         relabel_configs:
  24.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  25.           action: keep
  26.           regex: default;kubernetes;https
  27.  
  28.       - job_name: 'controller-manager'
  29.         kubernetes_sd_configs:
  30.         - role: endpoints
  31.         scheme: https
  32.         tls_config:
  33.           insecure_skip_verify: true
  34.         authorization:
  35.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  36.         relabel_configs:
  37.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  38.           action: keep
  39.           regex: kube-system;kube-controller-manager;https-metrics
  40.       - job_name: 'scheduler'
  41.         kubernetes_sd_configs:
  42.         - role: endpoints
  43.         scheme: https
  44.         tls_config:
  45.           insecure_skip_verify: true
  46.         authorization:
  47.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  48.         relabel_configs:
  49.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  50.           action: keep
  51.           regex: kube-system;kube-scheduler;https
  52.       - job_name: 'etcd'
  53.         kubernetes_sd_configs:
  54.         - role: endpoints
  55.         scheme: http
  56.         relabel_configs:
  57.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  58.           action: keep
  59.           regex: kube-system;etcd;http
  60.  ## 以下为添加的kubelete内容
  61.       - job_name: 'kubelet'
  62.         kubernetes_sd_configs:
  63.         - role: endpoints
  64.         scheme: https
  65.         tls_config:
  66.           insecure_skip_verify: true
  67.         authorization:
  68.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  69.         relabel_configs:
  70.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  71.           action: keep
  72.           regex: kube-system;kube-kubelet;https
  73.  
  74.     remote_write:
  75.     - url: 'http://192.168.120.17:17000/prometheus/v1/write'

(2)配置kubelet的service和endpoints

    跟之前一样我们要先查看本地有没有kubelet的endpoints如果没有就要添加。

  1. [root@k8s-master ~]# kubectl get endpoints -A | grep kubelet
  2.   ##如果没有就添加
  3.  
  4. vim kubelet-service.yaml
  5. apiVersion: v1
  6. kind: Service
  7. metadata:
  8.   labels:
  9.     k8s-app: kubelet
  10.   name: kube-kubelet
  11.   namespace: kube-system
  12. spec:
  13.   clusterIP: None
  14.   ports:
  15.     - name: https
  16.       port: 10250
  17.       protocol: TCP
  18.       targetPort: 10250
  19.   sessionAffinity: None
  20.   type: ClusterIP
  21. ---
  22. apiVersion: v1
  23. kind: Endpoints
  24. metadata:
  25.   labels:
  26.     k8s-app: kubelet
  27.   name: kube-kubelet
  28.   namespace: kube-system
  29. subsets:
  30.   - addresses:
  31.       - ip: 192.168.120.101
  32.       - ip: 192.168.120.102  ##这里我们自定义的endpoint，这里添加的是需要监控的k8s节点，这里我写的是master的ip地址和node的IP地址
  33.     ports:
  34.       - name: https
  35.         port: 10250
  36.         protocol: TCP

(3)测试指标

    然后打开夜莺的web页面，查看指标是否采集上。 测试指标：kubelet_running_pods

     导入仪表盘，仪表盘地址:categraf/dashboard-by-ident.json at main ·  · GitHub

     kubelet相关指标意思：

  1. # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
  2. # TYPE go_gc_duration_seconds summary
  3. gc的时间统计(summary指标)
  4.  
  5. # HELP go_goroutines Number of goroutines that currently exist.
  6. # TYPE go_goroutines gauge
  7. goroutine 数量
  8.  
  9. # HELP go_threads Number of OS threads created.
  10. # TYPE go_threads gauge
  11. 线程数量
  12.  
  13. # HELP kubelet_cgroup_manager_duration_seconds [ALPHA] Duration in seconds for cgroup manager operations. Broken down by method.
  14. # TYPE kubelet_cgroup_manager_duration_seconds histogram
  15. 操作cgroup的时长分布，按照操作类型统计
  16.  
  17. # HELP kubelet_containers_per_pod_count [ALPHA] The number of containers per pod.
  18. # TYPE kubelet_containers_per_pod_count histogram
  19. pod中container数量的统计(spec.containers的数量)
  20.  
  21. # HELP kubelet_docker_operations_duration_seconds [ALPHA] Latency in seconds of Docker operations. Broken down by operation type.
  22. # TYPE kubelet_docker_operations_duration_seconds histogram
  23. 操作docker的时长分布，按照操作类型统计
  24.  
  25. # HELP kubelet_docker_operations_errors_total [ALPHA] Cumulative number of Docker operation errors by operation type.
  26. # TYPE kubelet_docker_operations_errors_total counter
  27. 操作docker的错误累计次数，按照操作类型统计
  28.  
  29. # HELP kubelet_docker_operations_timeout_total [ALPHA] Cumulative number of Docker operation timeout by operation type.
  30. # TYPE kubelet_docker_operations_timeout_total counter
  31. 操作docker的超时统计，按照操作类型统计
  32.  
  33. # HELP kubelet_docker_operations_total [ALPHA] Cumulative number of Docker operations by operation type.
  34. # TYPE kubelet_docker_operations_total counter
  35. 操作docker的累计次数，按照操作类型统计
  36.  
  37. # HELP kubelet_eviction_stats_age_seconds [ALPHA] Time between when stats are collected, and when pod is evicted based on those stats by eviction signal
  38. # TYPE kubelet_eviction_stats_age_seconds histogram
  39. 驱逐操作的时间分布，按照驱逐信号(原因)分类统计
  40.  
  41. # HELP kubelet_evictions [ALPHA] Cumulative number of pod evictions by eviction signal
  42. # TYPE kubelet_evictions counter
  43. 驱逐次数统计，按照驱逐信号(原因)统计
  44.  
  45. # HELP kubelet_http_inflight_requests [ALPHA] Number of the inflight http requests
  46. # TYPE kubelet_http_inflight_requests gauge
  47. 请求kubelet的inflight请求数,按照method path server_type统计, 注意与每秒的request数区别开
  48.  
  49. # HELP kubelet_http_requests_duration_seconds [ALPHA] Duration in seconds to serve http requests
  50. # TYPE kubelet_http_requests_duration_seconds histogram
  51. 请求kubelet的请求时间统计, 按照method path server_type统计
  52.  
  53. # HELP kubelet_http_requests_total [ALPHA] Number of the http requests received since the server started
  54. # TYPE kubelet_http_requests_total counter
  55. 请求kubelet的请求数统计,按照method path server_type统计
  56.  
  57. # HELP kubelet_managed_ephemeral_containers [ALPHA] Current number of ephemeral containers in pods managed by this kubelet. Ephemeral containers will be ignored if disabled by the EphemeralContainers feature gate, and this number will be 0.
  58. # TYPE kubelet_managed_ephemeral_containers gauge
  59. 当前kubelet管理的临时容器数量
  60.  
  61. # HELP kubelet_network_plugin_operations_duration_seconds [ALPHA] Latency in seconds of network plugin operations. Broken down by operation type.
  62. # TYPE kubelet_network_plugin_operations_duration_seconds histogram
  63. 网络插件的操作耗时分布 ，按照操作类型(operation_type)统计, 如果 --feature-gates=EphemeralContainers=false, 否则一直为0 
  64.  
  65. # HELP kubelet_network_plugin_operations_errors_total [ALPHA] Cumulative number of network plugin operation errors by operation type.
  66. # TYPE kubelet_network_plugin_operations_errors_total counter
  67. 网络插件累计操作错误数统计，按照操作类型(operation_type)统计
  68.  
  69. # HELP kubelet_network_plugin_operations_total [ALPHA] Cumulative number of network plugin operations by operation type.
  70. # TYPE kubelet_network_plugin_operations_total counter
  71. 网络插件累计操作数统计，按照操作类型(operation_type)统计
  72.  
  73. # HELP kubelet_node_name [ALPHA] The node's name. The count is always 1.
  74. # TYPE kubelet_node_name gauge
  75. node name
  76.  
  77. # HELP kubelet_pleg_discard_events [ALPHA] The number of discard events in PLEG.
  78. # TYPE kubelet_pleg_discard_events counter
  79. PLEG(pod lifecycle event generator) 丢弃的event数统计
  80.  
  81. # HELP kubelet_pleg_last_seen_seconds [ALPHA] Timestamp in seconds when PLEG was last seen active.
  82. # TYPE kubelet_pleg_last_seen_seconds gauge
  83. PLEG上次活跃的时间戳
  84.  
  85. # HELP kubelet_pleg_relist_duration_seconds [ALPHA] Duration in seconds for relisting pods in PLEG.
  86. # TYPE kubelet_pleg_relist_duration_seconds histogram
  87. PLEG relist pod时间分布 
  88.  
  89. # HELP kubelet_pleg_relist_interval_seconds [ALPHA] Interval in seconds between relisting in PLEG.
  90. # TYPE kubelet_pleg_relist_interval_seconds histogram
  91. PLEG relist 间隔时间分布
  92.  
  93. # HELP kubelet_pod_start_duration_seconds [ALPHA] Duration in seconds for a single pod to go from pending to running.
  94. # TYPE kubelet_pod_start_duration_seconds histogram
  95. pod启动时间(从pending到running)分布, kubelet watch到pod时到pod中contianer都running后, watch各种source channel的pod变更
  96.  
  97. # HELP kubelet_pod_worker_duration_seconds [ALPHA] Duration in seconds to sync a single pod. Broken down by operation type: create, update, or sync
  98. # TYPE kubelet_pod_worker_duration_seconds histogram
  99. pod状态变化的时间分布， 按照操作类型(create update sync)统计, worker就是kubelet中处理一个pod的逻辑工作单位
  100.  
  101. # HELP kubelet_pod_worker_start_duration_seconds [ALPHA] Duration in seconds from seeing a pod to starting a worker.
  102. # TYPE kubelet_pod_worker_start_duration_seconds histogram
  103. kubelet watch到pod到worker启动的时间分布
  104.  
  105. # HELP kubelet_run_podsandbox_duration_seconds [ALPHA] Duration in seconds of the run_podsandbox operations. Broken down by RuntimeClass.Handler.
  106. # TYPE kubelet_run_podsandbox_duration_seconds histogram
  107. 启动sandbox的时间分布
  108.  
  109. # HELP kubelet_run_podsandbox_errors_total [ALPHA] Cumulative number of the run_podsandbox operation errors by RuntimeClass.Handler.
  110. # TYPE kubelet_run_podsandbox_errors_total counter
  111. 启动sanbox出现error的总数 
  112.  
  113. # HELP kubelet_running_containers [ALPHA] Number of containers currently running
  114. # TYPE kubelet_running_containers gauge
  115. 当前containers运行状态的统计, 按照container状态统计，created running exited
  116.  
  117. # HELP kubelet_running_pods [ALPHA] Number of pods that have a running pod sandbox
  118. # TYPE kubelet_running_pods gauge
  119. 当前处于running状态pod数量
  120.  
  121. # HELP kubelet_runtime_operations_duration_seconds [ALPHA] Duration in seconds of runtime operations. Broken down by operation type.
  122. # TYPE kubelet_runtime_operations_duration_seconds histogram
  123. 容器运行时的操作耗时(container在create list exec remove stop等的耗时)
  124.  
  125. # HELP kubelet_runtime_operations_errors_total [ALPHA] Cumulative number of runtime operation errors by operation type.
  126. # TYPE kubelet_runtime_operations_errors_total counter
  127. 容器运行时的操作错误数统计(按操作类型统计)
  128.  
  129. # HELP kubelet_runtime_operations_total [ALPHA] Cumulative number of runtime operations by operation type.
  130. # TYPE kubelet_runtime_operations_total counter
  131. 容器运行时的操作总数统计(按操作类型统计)
  132.  
  133. # HELP kubelet_started_containers_errors_total [ALPHA] Cumulative number of errors when starting containers
  134. # TYPE kubelet_started_containers_errors_total counter
  135. kubelet启动容器错误总数统计(按code和container_type统计)
  136. code包括ErrImagePull ErrImageInspect ErrImagePull ErrRegistryUnavailable ErrInvalidImageName等
  137. container_type一般为"container" "podsandbox"
  138.  
  139. # HELP kubelet_started_containers_total [ALPHA] Cumulative number of containers started
  140. # TYPE kubelet_started_containers_total counter
  141. kubelet启动容器总数
  142.  
  143. # HELP kubelet_started_pods_errors_total [ALPHA] Cumulative number of errors when starting pods
  144. # TYPE kubelet_started_pods_errors_total counter
  145. kubelet启动pod遇到的错误总数(只有创建sandbox遇到错误才会统计)
  146.  
  147. # HELP kubelet_started_pods_total [ALPHA] Cumulative number of pods started
  148. # TYPE kubelet_started_pods_total counter
  149. kubelet启动的pod总数 
  150.  
  151. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  152. # TYPE process_cpu_seconds_total counter
  153. 统计cpu使用率
  154.  
  155. # HELP process_max_fds Maximum number of open file descriptors.
  156. # TYPE process_max_fds gauge
  157. 允许进程打开的最大fd数
  158.  
  159. # HELP process_open_fds Number of open file descriptors.
  160. # TYPE process_open_fds gauge
  161. 当前打开的fd数量
  162.  
  163. # HELP process_resident_memory_bytes Resident memory size in bytes.
  164. # TYPE process_resident_memory_bytes gauge
  165. 进程驻留内存大小
  166.  
  167. # HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
  168. # TYPE process_start_time_seconds gauge
  169. 进程启动时间
  170.  
  171. # HELP rest_client_request_duration_seconds [ALPHA] Request latency in seconds. Broken down by verb and URL.
  172. # TYPE rest_client_request_duration_seconds histogram
  173. 请求apiserver的耗时统计(按照url和请求类型统计verb)
  174.  
  175. # HELP rest_client_requests_total [ALPHA] Number of HTTP requests, partitioned by status code, method, and host.
  176. # TYPE rest_client_requests_total counter
  177. 请求apiserver的总次数(按照返回码code和请求类型method统计)
  178.  
  179. # HELP storage_operation_duration_seconds [ALPHA] Storage operation duration
  180. # TYPE storage_operation_duration_seconds histogram
  181. 存储操作耗时(按照存储plugin(configmap emptydir hostpath 等 )和operation_name分类统计)
  182. # HELP volume_manager_total_volumes [ALPHA] Number of volumes in Volume Manager
  183. # TYPE volume_manager_total_volumes gauge
  184. 本机挂载的volume数量统计(按照plugin_name和state统计
  185. plugin_name包括"host-path" "empty-dir" "configmap" "projected")
  186. state(desired_state_of_world期状态/actual_state_of_world实际状态)

    cadivisor指标梳理

  1. # HELP container_cpu_cfs_periods_total Number of elapsed enforcement period intervals.
  2. # TYPE container_cpu_cfs_periods_total counter
  3. cfs时间片总数, 完全公平调度的时间片总数(分配到cpu的时间片数)
  4.  
  5. # HELP container_cpu_cfs_throttled_periods_total Number of throttled period intervals.
  6. # TYPE container_cpu_cfs_throttled_periods_total counter
  7. 容器被throttle的时间片总数
  8.  
  9. # HELP container_cpu_cfs_throttled_seconds_total Total time duration the container has been throttled.
  10. # TYPE container_cpu_cfs_throttled_seconds_total counter
  11. 容器被throttle的时间
  12.  
  13. # HELP container_file_descriptors Number of open file descriptors for the container.
  14. # TYPE container_file_descriptors gauge
  15. 容器打开的fd数
  16.  
  17. # HELP container_memory_usage_bytes Current memory usage in bytes, including all memory regardless of when it was accessed
  18. # TYPE container_memory_usage_bytes gauge
  19. 容器内存使用量，单位byte 
  20.  
  21. # HELP container_network_receive_bytes_total Cumulative count of bytes received
  22. # TYPE container_network_receive_bytes_total counter
  23. 容器入方向的流量
  24.  
  25. # HELP container_network_transmit_bytes_total Cumulative count of bytes transmitted
  26. # TYPE container_network_transmit_bytes_total counter
  27. 容器出方向的流量
  28.  
  29. # HELP container_spec_cpu_period CPU period of the container.
  30. # TYPE container_spec_cpu_period gauge
  31. 容器的cpu调度单位时间
  32.  
  33. # HELP container_spec_cpu_quota CPU quota of the container.
  34. # TYPE container_spec_cpu_quota gauge
  35. 容器的cpu规格 ，除以单位调度时间可以计算核数
  36.  
  37. # HELP container_spec_memory_limit_bytes Memory limit for the container.
  38. # TYPE container_spec_memory_limit_bytes gauge
  39. 容器的内存规格，单位byte
  40.  
  41. # HELP container_threads Number of threads running inside the container
  42. # TYPE container_threads gauge
  43. 容器当前的线程数
  44.  
  45. # HELP container_threads_max Maximum number of threads allowed inside the container, infinity if value is zero
  46. # TYPE container_threads_max gauge
  47. 允许容器启动的最大线程数

(七)K8s-KubeProxy组件监控

    KubeProxy 主要负责节点的网络管理，它在每个节点都会存在，是通过10249端口暴露监控指标。

    这里指标采集我们也用上面的方法，使用Prometheus-agent的方式

(1)配置Prometheus-agent configmap配置文件

    在之前的configmap的yaml文件中添加名为kube-proxy的job模块字段，添加完记得重新加载yaml文件和Prometheus-agent的pod

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: prometheus-agent-conf
  5.   labels:
  6.     name: prometheus-agent-conf
  7.   namespace: flashcat
  8. data:
  9.   prometheus.yml: |-
  10.     global:
  11.       scrape_interval: 15s
  12.       evaluation_interval: 15s
  13.  
  14.     scrape_configs:
  15.       - job_name: 'apiserver'
  16.         kubernetes_sd_configs:
  17.         - role: endpoints
  18.         scheme: https
  19.         tls_config:
  20.           insecure_skip_verify: true
  21.         authorization:
  22.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  23.         relabel_configs:
  24.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  25.           action: keep
  26.           regex: default;kubernetes;https
  27.  
  28.       - job_name: 'controller-manager'
  29.         kubernetes_sd_configs:
  30.         - role: endpoints
  31.         scheme: https
  32.         tls_config:
  33.           insecure_skip_verify: true
  34.         authorization:
  35.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  36.         relabel_configs:
  37.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  38.           action: keep
  39.           regex: kube-system;kube-controller-manager;https-metrics
  40.       - job_name: 'scheduler'
  41.         kubernetes_sd_configs:
  42.         - role: endpoints
  43.         scheme: https
  44.         tls_config:
  45.           insecure_skip_verify: true
  46.         authorization:
  47.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  48.         relabel_configs:
  49.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  50.           action: keep
  51.           regex: kube-system;kube-scheduler;https
  52.       - job_name: 'etcd'
  53.         kubernetes_sd_configs:
  54.         - role: endpoints
  55.         scheme: http
  56.         relabel_configs:
  57.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  58.           action: keep
  59.           regex: kube-system;etcd;http
  60.       - job_name: 'kubelet'
  61.         kubernetes_sd_configs:
  62.         - role: endpoints
  63.         scheme: https
  64.         tls_config:
  65.           insecure_skip_verify: true
  66.         authorization:
  67.           credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  68.         relabel_configs:
  69.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  70.           action: keep
  71.           regex: kube-system;kube-kubelet;https
  72.    ##这里是添加的模块
  73.       - job_name: 'kube-proxy'
  74.         kubernetes_sd_configs:
  75.         - role: endpoints
  76.         scheme: http
  77.         relabel_configs:
  78.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  79.           action: keep
  80.           regex: kube-system;kube-proxy;http
  81.  
  82.     remote_write:
  83.     - url: 'http://192.168.120.17:17000/prometheus/v1/write'

(2)配置kube-proxy的endpoints

    跟之前一样，先查看有没有这个kube-proxy的endpoints如果没有添加。

  1. [root@k8s-master ~]#  kubectl get endpoints -A | grep kube-pro
  2. ## 如果没有 添加service
  3.  
  4. vim kube-proxy-service.yaml
  5. apiVersion: v1
  6. kind: Service
  7. metadata:
  8.   labels:
  9.     k8s-app: proxy
  10.   name: kube-proxy
  11.   namespace: kube-system
  12. spec:
  13.   clusterIP: None
  14.   selector:
  15.     k8s-app: kube-proxy
  16.   ports:
  17.     - name: http
  18.       port: 10249
  19.       protocol: TCP
  20.       targetPort: 10249
  21.   sessionAffinity: None
  22.   type: ClusterIP

(3)更改kube-proxy的metricsbindAddress

    查看 kube-proxy 的10249端口是否绑定到127.0.0.1了，如果是就修改成0.0.0.0，通过kubectl edit cm -n kube-system kube-proxy修改metricsBindAddress即可

  1. [root@k8s-master ~]# kubectl edit cm -n kube-system kube-proxy
  2.  
  3. ......
  4. ......
  5. ......
  6.  
  7. kind: KubeProxyConfiguration
  8.     metricsBindAddress: "0.0.0.0"  ## 这里修改为0.0.0.0 即可
  9.     mode: ""
  10.     nodePortAddresses: null
  11.     oomScoreAdj: nul

(4)指标测试

在夜莺的web页面输入指标测试:
kubeproxy_network_programming_duration_seconds_bucket

 导入监控大盘, 仪表盘json文件:https://github.com/flin/inputs/kube_proxy/dashboard-by-ident.json

kube-proxy关键指标含义:

  1. # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
  2. # TYPE go_gc_duration_seconds summary
  3. gc时间
  4.  
  5. # HELP go_goroutines Number of goroutines that currently exist.
  6. # TYPE go_goroutines gauge
  7. goroutine数量
  8.  
  9. # HELP go_threads Number of OS threads created.
  10. # TYPE go_threads gauge
  11. 线程数量
  12.  
  13. # HELP kubeproxy_network_programming_duration_seconds [ALPHA] In Cluster Network Programming Latency in seconds
  14. # TYPE kubeproxy_network_programming_duration_seconds histogram
  15. service或者pod发生变化到kube-proxy规则同步完成时间指标含义较复杂，参照https://github.com/kubernetes/community/blob/master/sig-scalability/slos/network_programming_latency.md
  16.  
  17. # HELP kubeproxy_sync_proxy_rules_duration_seconds [ALPHA] SyncProxyRules latency in seconds
  18. # TYPE kubeproxy_sync_proxy_rules_duration_seconds histogram
  19. 规则同步耗时
  20.  
  21. # HELP kubeproxy_sync_proxy_rules_endpoint_changes_pending [ALPHA] Pending proxy rules Endpoint changes
  22. # TYPE kubeproxy_sync_proxy_rules_endpoint_changes_pending gauge
  23. endpoint 发生变化后规则同步pending的次数
  24.  
  25. # HELP kubeproxy_sync_proxy_rules_endpoint_changes_total [ALPHA] Cumulative proxy rules Endpoint changes
  26. # TYPE kubeproxy_sync_proxy_rules_endpoint_changes_total counter
  27. endpoint 发生变化后规则同步的总次数
  28.  
  29. # HELP kubeproxy_sync_proxy_rules_iptables_restore_failures_total [ALPHA] Cumulative proxy iptables restore failures
  30. # TYPE kubeproxy_sync_proxy_rules_iptables_restore_failures_total counter
  31. 本机上 iptables restore 失败的总次数
  32.  
  33. # HELP kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds [ALPHA] The last time a sync of proxy rules was queued
  34. # TYPE kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds gauge
  35. 最近一次规则同步的请求时间戳，如果比下一个指标 kubeproxy_sync_proxy_rules_last_timestamp_seconds 大很多，那说明同步 hung 住了
  36.  
  37. # HELP kubeproxy_sync_proxy_rules_last_timestamp_seconds [ALPHA] The last time proxy rules were successfully synced
  38. # TYPE kubeproxy_sync_proxy_rules_last_timestamp_seconds gauge
  39. 最近一次规则同步的完成时间戳
  40.  
  41. # HELP kubeproxy_sync_proxy_rules_service_changes_pending [ALPHA] Pending proxy rules Service changes
  42. # TYPE kubeproxy_sync_proxy_rules_service_changes_pending gauge
  43. service变化引起的规则同步pending数量
  44.  
  45. # HELP kubeproxy_sync_proxy_rules_service_changes_total [ALPHA] Cumulative proxy rules Service changes
  46. # TYPE kubeproxy_sync_proxy_rules_service_changes_total counter
  47. service变化引起的规则同步总数
  48.  
  49. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  50. # TYPE process_cpu_seconds_total counter
  51. 利用这个指标统计cpu使用率
  52.  
  53. # HELP process_max_fds Maximum number of open file descriptors.
  54. # TYPE process_max_fds gauge
  55. 进程可以打开的最大fd数
  56.  
  57. # HELP process_open_fds Number of open file descriptors.
  58. # TYPE process_open_fds gauge
  59. 进程当前打开的fd数
  60.  
  61. # HELP process_resident_memory_bytes Resident memory size in bytes.
  62. # TYPE process_resident_memory_bytes gauge
  63. 统计内存使用大小
  64.  
  65. # HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
  66. # TYPE process_start_time_seconds gauge
  67. 进程启动时间戳
  68.  
  69. # HELP rest_client_request_duration_seconds [ALPHA] Request latency in seconds. Broken down by verb and URL.
  70. # TYPE rest_client_request_duration_seconds histogram
  71. 请求 apiserver 的耗时(按照url和verb统计)
  72.  
  73. # HELP rest_client_requests_total [ALPHA] Number of HTTP requests, partitioned by status code, method, and host.
  74. # TYPE rest_client_requests_total counter
  75. 请求 apiserver 的总数(按照code method host统计)

最后的最后

     夜莺监控k8s的方法夜莺的官网也做了合计专栏，有兴趣的伙伴可以去看看Kubernetes监控专栏，无论是指标还是原理，都做了解释初识Kubernetes -(flashcat.cloud)。如果在部署中遇到问题欢迎在本文章留言，24小时内必回复。

     看完这一期肯定会有小伙伴会有疑问，我业务都跑pod上面，光监控这些组件没啥大用啊，我想知道总共有几个 Namespace，有几个 Service、Deployment、Statefulset，某个 Deployment 期望有几个 Pod 要运行，实际有几个 Pod 在运行，这些既有的指标就无法回答了。当然这一点肯定重中之重，这个问题我们下一期详细讲解使用使用 kube-state-metrics (KSM)监控 Kubernetes 对象,俗称KSM来监听各个Kubernetes对象的状态，生产指标暴露出来让我们查看。 还有下一期还会讲用daemonset的最佳实践方案来采集监控。

声明：本文内容由网友自发贡献，转载请注明出处：【wpsshop】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号