devbox
2023面试高手
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

混合云K8s容器化应用弹性伸缩实战_cluster autoscaler cloud-provider

作者：2023面试高手 | 2024-02-27 10:02:13

赞

踩

cluster autoscaler cloud-provider
简介： 混合云K8s容器化应用弹性伸缩实战

 

image.png

 

1. 前提条件

本最佳实践的软件环境要求如下：
应用环境：
①容器服务ACK基于专有云V3.10.0版本。
②公共云云企业网服务CEN。
③公共云弹性伸缩组服务ESS。
配置条件：
1）使用专有云的容器服务或者在ECS上手动部署敏捷PaaS。
2）开通云专线，打通容器服务所在VPC与公共云上的VPC。
3）开通公共云弹性伸缩组服务（ESS）。

2. 背景信息

本实践基于K8s的业务集群运行在专有云上，对测试业务进行压力测试，主要基于以下三种产品和能力：
①利用阿里云的云企业网专线打通专有云和公共云，实现两朵云上VPC网络互通。
②利用K8s（Kubernetes）的HPA能力，实现容器的水平伸缩。
③利用K8s的Cluster Autoscaler和阿里云弹性伸缩组ESS能力实现节点的自动伸缩。

HPA（Horizontal Pod Autoscaler）是K8s的一种资源对象，能够根据CPU、内存等指标对statefulset、deployment等对象中的pod数量进行动态伸缩，使运行在上面的服务对指标的变化有一定的自适应能力。

当被测试业务指标达到上限时，触发HPA自动扩容业务pod；当业务集群无法承载更多pod时，触发公共云的ESS服务，在公共云内扩容出ECS并自动添加到专有云的K8s集群。

 

1.jpg
图 1：架构原理图

 

3. 配置HPA

本示例创建了一个支持HPA的nginx应用，创建成功后，当Pod的利用率超过本例中设置的20%利用率时，则会进行水平扩容，低于20%的时候会进行缩容。

1.若使用自建K8s集群，则通过yaml文件配置HPA

1）创建一个nginx应用，必须为应用设置request值，否则HPA不会生效。

  1. apiVersion:
  2. app/v1beta2
  3. kind: Deployment
  4. spec:
  5.   template:
  6.     metadata:
  7.       creationTimestamp: null
  8.       labels:
  9.         app: hpa-test
  10.     spec:
  11.         dnsPolicy: ClusterFirst     
  12.         terminationGracePeriodSeconds:30         
  13.         containers:
  14.         image: '192.168.**.***:5000/admin/hpa-example:v1'
  15.         imagePullPolicy: IfNotPresent
  16.         terminationMessagePolicy:File
  17.         terminationMessagePath:/dev/termination-log
  18.         name: hpa-test
  19.         resources:
  20.           requests:
  21.             cpu: //必须设置request值
  22.         securityContext: {}
  23.         restartPolicy:Always
  24.         schedulerName:default-scheduler
  25.   replicas: 1
  26.   selector: 
  27.     matchLabels:
  28.         app: hpa-test
  29.   revisionHistoryLimit: 10
  30.   strategy: 
  31.     type: RollingUpdate
  32.     rollingUpdate:
  33.         maxSurge: 25%
  34.         maxUnavailable: 25%
  35.      progressDeadlineSeconds: 600

2）创建HPA。

  1. apiVersion: autoscaling/v1
  2. kind: HorizontalPodAutoscaler
  3. metadata:
  4.  annotations:
  5.     autoscaling.alpha.kubernetes.io/conditions:'[{"type":"AbleToScale","status":"True","lastTransitionTime":"2020-04-29T06:57:28Z","reason":"ScaleDownStabilized","message":"recent
  6.     recommendations were higher than current one, applying the highest recent
  7.     recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2020-04-29T06:57:28Z","reason":"ValidMetricFound","message":"theHPA
  8.     was able to successfully calculate a replica count from cpu resource
  9.     utilization(percentage of
  10.     request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2020-04-29T06:57:28Z","reason":"DesiredWithinRange","message":"thedesired
  11.     count is within the acceptable range"}]'
  12.     autoscaling.alpha.kubernetes.io/currentmetrics:'[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":0,"currentAverageValue":"0"}}]'
  13.  
  14. creationTimestamp: 2020-04-29T06:57:13Z
  15. name: hpa-test
  16. namespace: default
  17. resourceVersion: "3092268"
  18. selfLink:
  19. /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/hpa01
  20. uid: a770ca26-89e6-11ea-a7d7-00163e0106e9
  21. spec:
  22.     maxReplicas: //设置pod数量 
  23.     minReplicas: 1
  24.     scaleTargetRef:
  25.        apiVersion: apps/v1beta2
  26.        kind: Deployment
  27.        name: centos 
  28.             targetCPUUtilizationPercentage://设置CPU阈值

2.若使用阿里云容器服务，需要在部署应用时选择配置HPA

 

2.jpg
图2：访问设置

 

4. 配置Cluster Autoscaler

资源请求（Request）的正确、合理设置，是弹性伸缩的前提条件。节点自动伸缩组件基于K8s资源调度的分配情况进行伸缩判断，节点中资源的分配通过资源请（Request）进行计算。

当Pod由于资源请求（Request）无法满足并进入等待（Pending）状态时，节点自动伸缩组件会根据弹性伸缩组配置信息中的资源规格以及约束配置，计算所需的节点数目。

如果可以满足伸缩条件，则会触发伸缩组的节点加入。而当一个节点在弹性伸缩组中且节点上Pod的资源请求低于阈值时，节点自动伸缩组件会将节点进行缩容。

1.配置弹性伸缩组ESS

1）创建ESS弹性伸缩组，记录最小实例数和最大实例数。

 

3-1.jpg
图3：修改伸缩组

 

2）创建伸缩配置，记录伸缩配置的id。

 

5.jpg
图4：伸缩配置

 

  1. #!/bin/sh
  2. yum install -y ntpdate && ntpdate -u ntp1.aliyun.com && curl http:// example.com/public/hybrid/attach_local_node_aliyun.sh | bash -s -- --docker-version 17.06.2-ce-3 --token
  3. 9s92co.y2gkocbumal4fz1z --endpoint 192.168.**.***:6443 --cluster-dns 10.254.**.**
  4. --region cn-huhehaote
  5. echo "{" > /etc/docker/daemon.json
  6. echo "\"registry-mirrors\": [" >>
  7. /etc/docker/daemon.json
  8. echo "\"https://registry-vpc.cn-huhehaote.aliyuncs.com\"" >> /etc/docker/daemon.json
  9. echo "]," >> /etc/docker/daemon.json
  10. echo "\"insecure-registries\": [\"https://192.168.**.***:5000\"]" >> /etc/docker/daemon.json
  11. echo "}" >> /etc/docker/daemon.json
  12. systemctl restart docker

2.K8s集群部署autoscaler

kubectl apply -f ca.yml

参考ca.yml创建autoscaler，注意修改如下配置与实际环境相对应。

  1. access-key-id: "TFRBSWlCSFJyeHd2QXZ6****"
  2. access-key-secret: "bGIyQ3NuejFQOWM0WjFUNjR4WTVQZzVPRXND****"
  3. region-id: "Y24taHVoZWhh****"

ca.yal代码如下：

  1. ---
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   labels:
  6.     k8s-addon: cluster-autoscaler.addons.k8s.io
  7.     k8s-app: cluster-autoscaler
  8.   name: cluster-autoscaler
  9.   namespace: kube-system
  10. ---
  11. apiVersion: rbac.authorization.k8s.io/v1
  12. kind: ClusterRole
  13. metadata:
  14.   name: cluster-autoscaler
  15.   labels:
  16.     k8s-addon: cluster-autoscaler.addons.k8s.io
  17.     k8s-app: cluster-autoscaler
  18. rules:
  19. - apiGroups: [""]
  20.   resources: ["events","endpoints"]
  21.   verbs: ["create", "patch"]
  22. - apiGroups: [""]
  23.   resources: ["pods/eviction"]
  24.   verbs: ["create"]
  25. - apiGroups: [""]
  26.   resources: ["pods/status"]
  27.   verbs: ["update"]
  28. - apiGroups: [""]
  29.   resources: ["endpoints"]
  30.   resourceNames: ["cluster-autoscaler"]
  31.   verbs: ["get","update"]
  32. - apiGroups: [""]
  33.   resources: ["nodes"]
  34.   verbs: ["watch","list","get","update"]
  35. - apiGroups: [""]
  36.   resources: ["pods","services","replicationcontrollers","persistentvolumeclaims","persistentvolumes"]
  37.   verbs: ["watch","list","get"]
  38. - apiGroups: ["extensions"]
  39.   resources: ["replicasets","daemonsets"]
  40.   verbs: ["watch","list","get"]
  41. - apiGroups: ["policy"]
  42.   resources: ["poddisruptionbudgets"]
  43.   verbs: ["watch","list"]
  44. - apiGroups: ["apps"]
  45.   resources: ["statefulsets"]
  46.   verbs: ["watch","list","get"]
  47. - apiGroups: ["storage.k8s.io"]
  48.   resources: ["storageclasses"]
  49.   verbs: ["watch","list","get"]
  50.  
  51. ---
  52. apiVersion: rbac.authorization.k8s.io/v1
  53. kind: Role
  54. metadata:
  55.   name: cluster-autoscaler
  56.   namespace: kube-system
  57.   labels:
  58.     k8s-addon: cluster-autoscaler.addons.k8s.io
  59.     k8s-app: cluster-autoscaler
  60. rules:
  61. - apiGroups: [""]
  62.   resources: ["configmaps"]
  63.   verbs: ["create","list","watch"]
  64. - apiGroups: [""]
  65.   resources: ["configmaps"]
  66.   resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
  67.   verbs: ["delete","get","update","watch"]
  68.  
  69. ---
  70. apiVersion: rbac.authorization.k8s.io/v1
  71. kind: ClusterRoleBinding
  72. metadata:
  73.   name: cluster-autoscaler
  74.   labels:
  75.     k8s-addon: cluster-autoscaler.addons.k8s.io
  76.     k8s-app: cluster-autoscaler
  77. roleRef:
  78.   apiGroup: rbac.authorization.k8s.io
  79.   kind: ClusterRole
  80.   name: cluster-autoscaler
  81. subjects:
  82.   - kind: ServiceAccount
  83.     name: cluster-autoscaler
  84.     namespace: kube-system
  85.  
  86. ---
  87. apiVersion: rbac.authorization.k8s.io/v1
  88. kind: RoleBinding
  89. metadata:
  90.   name: cluster-autoscaler
  91.   namespace: kube-system
  92.   labels:
  93.     k8s-addon: cluster-autoscaler.addons.k8s.io
  94.     k8s-app: cluster-autoscaler
  95. roleRef:
  96.   apiGroup: rbac.authorization.k8s.io
  97.   kind: Role
  98.   name: cluster-autoscaler
  99. subjects:
  100.   - kind: ServiceAccount
  101.     name: cluster-autoscaler
  102.     namespace: kube-system
  103. ---
  104. apiVersion: v1
  105. kind: Secret
  106. metadata:
  107.   name: cloud-config
  108.   namespace: kube-system
  109. type: Opaque
  110. data:
  111.   access-key-id: "TFRBSWlCSFJyeHd2********"
  112.   access-key-secret: "bGIyQ3NuejFQOWM0WjFUNjR4WTVQZzVP*********"
  113.   region-id: "Y24taHVoZW********"
  114.  
  115. ---
  116. apiVersion: apps/v1
  117. kind: Deployment
  118. metadata:
  119.   name: cluster-autoscaler
  120.   namespace: kube-system
  121.   labels:
  122.     app: cluster-autoscaler
  123. spec:
  124.   replicas: 1
  125.   selector:
  126.     matchLabels:
  127.       app: cluster-autoscaler
  128.   template:
  129.     metadata:
  130.       labels:
  131.         app: cluster-autoscaler
  132.     spec:
  133.       dnsConfig:
  134.         nameservers:
  135.           - 100.XXX.XXX.XXX
  136.           - 100.XXX.XXX.XXX
  137.       nodeSelector:
  138.         ca-key: ca-value
  139.       priorityClassName: system-cluster-critical
  140.       serviceAccountName: admin
  141.       containers:
  142.         - image: 192.XXX.XXX.XXX:XX/admin/autoscaler:v1.3.1-7369cf1
  143.           name: cluster-autoscaler
  144.           resources:
  145.             limits:
  146.               cpu: 100m
  147.               memory: 300Mi
  148.             requests:
  149.               cpu: 100m
  150.               memory: 300Mi
  151.           command:
  152.             - ./cluster-autoscaler
  153.             - '--v=5'
  154.             - '--stderrthreshold=info'
  155.             - '--cloud-provider=alicloud'
  156.             - '--scan-interval=30s'
  157.             - '--scale-down-delay-after-add=8m'
  158.             - '--scale-down-delay-after-failure=1m'
  159.             - '--scale-down-unready-time=1m'
  160.             - '--ok-total-unready-count=1000'
  161.             - '--max-empty-bulk-delete=50'
  162.             - '--expander=least-waste'
  163.             - '--leader-elect=false'
  164.             - '--scale-down-unneeded-time=8m'
  165.             - '--scale-down-utilization-threshold=0.2'
  166.             - '--scale-down-gpu-utilization-threshold=0.3'
  167.             - '--skip-nodes-with-local-storage=false'
  168.             - '--nodes=0:5:asg-hp3fbu2zeu9bg3clraqj'
  169.           imagePullPolicy: "Always"
  170.           env:
  171.             - name: ACCESS_KEY_ID
  172.               valueFrom:
  173.                 secretKeyRef:
  174.                   name: cloud-config
  175.                   key: access-key-id
  176.             - name: ACCESS_KEY_SECRET
  177.               valueFrom:
  178.                 secretKeyRef:
  179.                   name: cloud-config
  180.                   key: access-key-secret
  181.             - name: REGION_ID
  182.               valueFrom:
  183.                 secretKeyRef:
  184.                   name: cloud-config
  185.                   key: region-id

5. 执行结果

模拟业务访问：

启动busybox镜像，在pod内执行如下命令访问以上应用的service，可以同时启动多个pod增加业务负载。while true;do wget -q -O- http://hpa-test/index.html;done

观察HPA：

加压前

 

6.jpg
图 5：加压前

 

加压后
当CPU值达到阈值后，会触发pod的水平扩容。

 

7.jpg
图 6：加压后1
8.jpg
图 7：加压后2

 

观察Pod：

当集群资源不足时，新扩容出的pod处于pending状态，此时将触发cluster autoscaler，自动扩容节点。

 

9.jpg
图8：伸缩活动

原文链接

本文为阿里云原创内容，未经允许不得转载。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/2023面试高手/article/detail/151915
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号