当前位置:   article > 正文

K8S篇之实现利用Prometheus监控pod的实时数据指标_kube-state-metrics prometheus

kube-state-metrics prometheus

一、监控部署
1、将k8s集群中kube-state-metrics指标进行收集,服务进行部署
1.1 pod性能指标(k8s集群组件自动集成)
k8s组件本身提供组件自身运行的监控指标以及容器相关的监控指标。通过cAdvisor 是一个开源的分析容器资源使用率和性能特性的代理工具,集成到 Kubelet中,当Kubelet启动时会同时启动cAdvisor,且一个cAdvisor只监控一个Node节点的信息。cAdvisor 自动查找所有在其所在节点上的容器,自动采集 CPU、内存、文件系统和网络使用的统计信息。cAdvisor 通过它所在节点机的 Root 容器,采集并分析该节点机的全面使用情况。
当然kubelet也会输出一些监控指标数据,因此pod的监控数据有kubelet和cadvisor,监控url分别为
https://NodeIP:10250/metrics
https://NodeIP:10250/metrics/cadvisor
1.2 K8S资源监控(k8s集群内部署)
kube-state-metrics是一个简单的服务,它监听Kubernetes API服务器并生成关联对象的指标。它不关注单个Kubernetes组件的运行状况,而是关注内部各种对象(如deployment、node、pod等)的运行状况。
注:先手动检查下集群,是否已经安装kube-state-metrics
在这里插入图片描述
如果集群没有安装,可参考如下步骤进行部署:

docker pull gcr.io/google_containers/kube-state-metrics:v1.6.0
// 镜像打标签,设置为当前k8s配置的镜像仓库地址
docker tag quay.io/coreos/kube-state-metrics:v1.9.0 dockerhub.kubekey.local/library/kube-state-metrics:v1.9.0
// 推进仓库
docker push dockerhub.kubekey.local/library/kube-state-metrics:v1.9.0
  • 1
  • 2
  • 3
  • 4
  • 5

1.3 编辑kube-state-metrics.yml文件

vim kube-state-metrics.yml
  • 1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
  namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - ingresses
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - statefulsets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
- apiGroups: ["policy"]
  resources:
  - poddisruptionbudgets
  verbs: ["list", "watch"]
- apiGroups: ["certificates.k8s.io"]
  resources:
  - certificatesigningrequests
  verbs: ["list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources:
  - storageclasses
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling.k8s.io"]
  resources:
  - verticalpodautoscalers
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: prometheus
---
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      # 注意,这里image地址修改为你k8s配置的仓库地址
      - image: dockerhub.kubekey.local/library/kube-state-metrics:v1.9.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 30
        name: kube-state-metrics
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 500m
            memory: 768Mi
          requests:
            cpu: 250m
            memory: 768Mi
      restartPolicy: Always
      serviceAccount: kube-state-metrics
      serviceAccountName: kube-state-metrics
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
  namespace: prometheus
spec:
  ports:
  - name: kube-state-metrics
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: kube-state-metrics
    ## 注意这里kube-state-metrics暴露类型修改为NodePort对外暴露
  type: NodePort
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163

1.4 启动yaml文件

kubectl apply -f kube-state-metrics.yaml
  • 1

在这里插入图片描述
1.5 查看pod信息

kubectl get pod -n prometheus
  • 1

在这里插入图片描述
1.6 查看service信息

kubectl get svc -n prometheus
  • 1

在这里插入图片描述
这里可以看到k8s集群对外暴露的端口为 62177
1.7 查看集群信息

kubectl get po -n prometheus -owide
  • 1

在这里插入图片描述
然后查看metrics信息
可以手动

curl k8s02:62177/metrics
  • 1

正常,数据metrics就会出现
在这里插入图片描述
二、创建token供集群外部访问
集群外部监控K8s集群,通过访问kube-apiserver来访问集群资源。通过这种方式集群外部prometheus也能自动发现k8s集群服务

# 1.创建serviceaccounts
kubectl create sa prometheus -n default
# 2.创建prometheus角色并对其绑定cluster-admin
kubectl create clusterrolebinding prometheus --clusterrole cluster-admin --serviceaccount=default:prometheus
# 3. 创建secret; k8s1.24之后默认不会为serveiceaccounts创建secret
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: prometheus-token
  namespace: default
  annotations:
    kubernetes.io/service-account.name: "prometheus"
EOF
# 4. 测试访问kube-apiserver
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
TOKEN=$(kubectl get secret  prometheus-token -n default -o jsonpath='{.data.token}' | base64 --decode)
curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
# 5. 保存token
echo $TOKEN > k8s_token
# 6. 测试访问指标
# 访问pod性能资源指标:(访问kubelet)
# 注意:master1为当前master节点的hostname,需要修改
curl $APISERVER/api/v1/nodes/master1:10250/proxy/metrics --header "Authorization: Bearer $TOKEN" --insecure
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25

三、集成Prometheus配置

vim prometheus.yml
  • 1
scrape_configs:
  - job_name: "k8s-cadvisor"
    honor_timestamps: true
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
    - api_server: https://10.142.155.202:6443
      role: node
      bearer_token_file: /prometheus/data/k8s_token
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /prometheus/data/k8s_token
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - separator: ;
      regex: (.*)
      target_label: __address__
      replacement: 10.142.155.202:6443
      action: replace
    - source_labels: [__meta_kubernetes_node_name]
      separator: ;
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}:10250/proxy/metrics/cadvisor
      action: replace
  - job_name: "kube-node-kubelet"
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /prometheus/data/k8s_token
    kubernetes_sd_configs:
    - role: node
      api_server: "https://10.142.155.202:6443"   // 修改为对应的k8s master的节点
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /prometheus/data/k8s_token
    relabel_configs:
    - target_label: __address__
      replacement: 10.142.155.202:6443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}:10250/proxy/metrics
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service_name
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54

注意:bearer_token_file: /prometheus/data/k8s_token
这里的token为上面生成的token信息,请根据目录进行配置即可

然后重启prometheus
如果是容器部署的prometheus,需要考虑映射token,可docker cp到/prometheus/data/ 即可
即可

docker restart prometheus
  • 1

3、进入prometheus界面,查看相关指标
默认情况下 prometheus url: http://IP:9090
在这里插入图片描述
4、集成grafana
导入grafana JSON ID, 747
4.1、导入node信息指标
在这里插入图片描述
load 即可
在这里插入图片描述
4.2、导入pod信息指标
JSON ID:15760
在这里插入图片描述
大盘信息即可完全展示~

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/天景科技苑/article/detail/873713
推荐阅读
相关标签
  

闽ICP备14008679号