热门标签
热门文章
- 1基于n-gram模型的中文分词_ngram分词
- 2java连接mysql增删改查（01单增删改查）_java连接mysq 增删改查
- 3论文aigc检测率为多少合格？论文ai免费润色_本科毕业论文aigc值多少合格
- 4Python中数据可视化的最新趋势和未来发展
- 5不相交集（The Disjoint Set ADT）_the array representation of a disjoint set contain
- 6Solana 与 DePIN 的双向奔赴，会带来 DePIN 之夏吗？
- 7matlab实现apriori算法源代码实验报告_apriori算法实验报告
- 8计算机毕业设计hadoop+spark知识图谱视频推荐系统 视频分析可视化大屏 视频爬虫 视频大数据分析 大数据毕业设计 机器学习 深度学习 人工智能
- 9对云原生整体解决方案的进一步复盘
- 10Java进阶-SpringCloud设计模式-工厂模式的设计与详解
当前位置: article > 正文
【k8s 高级调度--污点和容忍】_所以一定要确保controller-manager时间是准确的
作者：我家小花儿 | 2024-05-13 13:16:26
赞
踩
所以一定要确保controller-manager时间是准确的
1、调度概念
在 Kubernetes 中，调度（scheduling）指的是确保 Pod 匹配到合适的节点， 以便 kubelet 能够运行它们。 抢占（Preemption）指的是终止低优先级的 Pod 以便高优先级的 Pod 可以调度运行的过程。 驱逐（Eviction）是在资源匮乏的节点上，主动让一个或多个 Pod 失效的过程。
2、CronJob 计划任务
在k8s中周期性运行计划任务，与linux中的crontab相同
注意点：CronJob执行的时间是controllerr-manager的时间，所以一定要确保controller-manager时间是准确的。
2.1 配置文件
apiVersion: batch/v1 kind: CronJob # 定时任务 metadata: name: cron-job-test # 定时任务名字 spec: concurrencyPolicy: Allow # 并发调度策略：Allow 允许并发调度，Forbid：不允许并发执行，Replace：如果之前的任务还没执行完，就直接执行新的，放弃上一个任务 failedJobsHistoryLimit: 1 # 保留多少个失败的任务 successfulJobsHistoryLimit: 3 # 保留多少个成功的任务 suspend: false # 是否挂起任务，若为 true 则该任务不会执行 # startingDeadlineSeconds: 30 # 间隔多长时间检测失败的任务并重新执行，时间不能小于 10 schedule: "* * * * *" # 调度策略 jobTemplate: spec: template: spec: containers: - name: busybox image: busybox:1.28 imagePullPolicy: IfNotPresent command: - /bin/sh - -c - date; echo Hello from the Kubernetes cluster restartPolicy: OnFailure
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
2.2 CronJob执行
[root@k8s-master job]# kubectl create -f cron-job-pd.yaml cronjob.batch/cron-job-test created [root@k8s-master job]# kubectl get cronjobs NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cron-job-test * * * * * False 0 <none> 11s [root@k8s-master job]# kubectl get po NAME READY STATUS RESTARTS AGE configfile-po 0/1 Completed 0 26h dns-test 1/1 Running 2 (36h ago) 3d21h emptydir-volume-pod 2/2 Running 44 (58m ago) 23h fluentd-59k8k 1/1 Running 1 (36h ago) 3d3h fluentd-hhtls 1/1 Running 1 (36h ago) 3d3h host-volume-pod 1/1 Running 0 23h nfs-volume-pod-1 1/1 Running 0 21h nfs-volume-pod-2 1/1 Running 0 21h nginx-deploy-6fb8d6548-8khhv 1/1 Running 29 (54m ago) 29h nginx-deploy-6fb8d6548-fd9tx 1/1 Running 29 (54m ago) 29h nginx-sc-0 1/1 Running 0 3h20m [root@k8s-master job]# kubectl get cronjobs NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cron-job-test * * * * * False 0 42s 2m16s [root@k8s-master job]# kubectl get po NAME READY STATUS RESTARTS AGE configfile-po 0/1 Completed 0 26h cron-job-test-28484150-wkbgp 0/1 Completed 0 2m4s cron-job-test-28484151-886j6 0/1 Completed 0 64s cron-job-test-28484152-srjb4 0/1 Completed 0 4s dns-test 1/1 Running 2 (36h ago) 3d21h emptydir-volume-pod 2/2 Running 46 (35s ago) 23h fluentd-59k8k 1/1 Running 1 (36h ago) 3d3h fluentd-hhtls 1/1 Running 1 (36h ago) 3d3h host-volume-pod 1/1 Running 0 23h nfs-volume-pod-1 1/1 Running 0 21h nfs-volume-pod-2 1/1 Running 0 21h nginx-deploy-6fb8d6548-8khhv 1/1 Running 29 (56m ago) 29h nginx-deploy-6fb8d6548-fd9tx 1/1 Running 29 (56m ago) 29h nginx-sc-0 1/1 Running 0 3h22m [root@k8s-master job]# kubectl logs -f cron-job-test-28484150-wkbgp Tue Feb 27 15:50:19 UTC 2024 Hello from the Kubernetes cluster
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
3、初始化容器 InitContainer
- 相对于postStart来说，首先InitController能够保证一定在EntryPoint之前执行，而postStart不能，其次postStart更适合去执行一些命令操作，而InitController实际就是一个容器，可以在其他基础容器环境下执行更复杂的初始化功能。
3.1 在pod创建的模板中配置 initContainers 参数：
spec:
template:
spec:
initContainers:
- image: nginx:1.20
imagePullPolicy: IfNotPresent
command: [ sh,"-c","sleep 10 ；echo 'inited' >> /init.log "]
name: init-test
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
3.2 修改存在的deploy资源，如下
3.3 更新过deploy后，新的pod有个init的过程
4、污点和容忍
- 节点亲和性 是 Pod 的一种属性，它使 Pod 被吸引到一类特定的节点 （这可能出于一种偏好，也可能是硬性要求）。 污点（Taint） 则相反——它使节点能够排斥一类特定的 Pod。
- 容忍度（Toleration） 是应用于 Pod 上的。容忍度允许调度器调度带有对应污点的 Pod。 容忍度允许调度但并不保证调度：作为其功能的一部分， 调度器也会评估其他参数。
- 污点和容忍度（Toleration）相互配合，可以用来避免 Pod 被分配到不合适的节点上。 每个节点上都可以应用一个或多个污点，这表示对于那些不能容忍这些污点的 Pod， 是不会被该节点接受的。
4.1 污点（Taint）
- 污点：是标注在节点上的，给一个节点打上污点以后，k8s回认为尽量不要将Pod调度到该节点上，除非该pod上面表示可以容忍该污点，且一个节点可以打多个污点，此时需要pod容忍所有污点才会被调度该节点。
- 污点的影响：
- NoSchedule:不能容忍的pod不能被调度到该节点，但是已经存在的节点不会被驱逐。
- NoExecute:不能容忍的pod会被立即清除，能容忍的pod则会存在于节点上
- tolerationSeconds属性，如果没设置，则可以一直运行，
- tolerationSeconds:3600属性，则该pod还能继续在该节点运行3600，超过3600后会被重新调度
4.1.1 为node1节点打上污点
root@k8s-master volume]# kubectl taint node k8s-node-01 tag=test:NoSchedule node/k8s-node-01 tainted [root@k8s-master volume]# kubectl describe no k8s-node-01 Name: k8s-node-01 Roles: <none> Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux ingress=true kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-node-01 kubernetes.io/os=linux type=microsvc Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"66:39:6c:7a:92:99"} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 10.10.10.178 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/cri-dockerd.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Mon, 19 Feb 2024 22:58:42 +0800 # 这个地方可以看到这个新加的污点信息 Taints: tag=test:NoSchedule Unschedulable: false Lease: HolderIdentity: k8s-node-01 AcquireTime: <unset> RenewTime: Wed, 28 Feb 2024 01:17:51 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Mon, 26 Feb 2024 11:32:55 +0800 Mon, 26 Feb 2024 11:32:55 +0800 FlannelIsUp Flannel is running on this node MemoryPressure False Wed, 28 Feb 2024 01:17:30 +0800 Mon, 26 Feb 2024 11:32:41 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 28 Feb 2024 01:17:30 +0800 Mon, 26 Feb 2024 11:32:41 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 28 Feb 2024 01:17:30 +0800 Mon, 26 Feb 2024 11:32:41 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Wed, 28 Feb 2024 01:17:30 +0800 Tue, 27 Feb 2024 01:59:54 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.10.10.177 Hostname: k8s-node-01 Capacity: cpu: 2 ephemeral-storage: 62575768Ki hugepages-2Mi: 0 memory: 3861288Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 57669827694 hugepages-2Mi: 0 memory: 3758888Ki pods: 110 System Info: Machine ID: 9ee2b84718d0437fa9ea4380bdb34024 System UUID: A90F4D56-48C7-6739-A05A-A22B33EC7C5F Boot ID: 6cb5ab07-c82b-4404-8f48-7b9abafe52f1 Kernel Version: 3.10.0-1160.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://25.0.3 Kubelet Version: v1.25.0 Kube-Proxy Version: v1.25.0 PodCIDR: 10.2.2.0/24 PodCIDRs: 10.2.2.0/24 Non-terminated Pods: (7 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- default fluentd-59k8k 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3d4h default nginx-deploy-69ccc996f9-wqp55 100m (5%) 200m (10%) 128Mi (3%) 128Mi (3%) 61m ingress-nginx ingress-nginx-controller-jn65t 100m (5%) 0 (0%) 90Mi (2%) 0 (0%) 2d5h kube-flannel kube-flannel-ds-glkkb 100m (5%) 0 (0%) 50Mi (1%) 0 (0%) 8d kube-system coredns-c676cc86f-pdsl6 100m (5%) 0 (0%) 70Mi (1%) 170Mi (4%) 6d14h kube-system kube-proxy-n2w92 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system metrics-server-7bb86dcf48-hfpb5 100m (5%) 0 (0%) 200Mi (5%) 0 (0%) 3d3h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 500m (25%) 200m (10%) memory 538Mi (14%) 298Mi (8%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: <none>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
4.1.2 查看下master的污点
- 之前在安装ingress-nginx的时候无法安装再master节点就是因为master节点有污点
- master节点的污点是：Taints: node-role.kubernetes.io/control-plane:NoSchedule
[root@k8s-master volume]# kubectl describe no k8s-master Name: k8s-master Roles: control-plane Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux ingress=true kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-master kubernetes.io/os=linux node-role.kubernetes.io/control-plane= node.kubernetes.io/exclude-from-external-load-balancers= Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"c2:fd:ef:b4:ea:aa"} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 10.10.10.100 kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Mon, 19 Feb 2024 22:04:42 +0800 Taints: node-role.kubernetes.io/control-plane:NoSchedule Unschedulable: false Lease: HolderIdentity: k8s-master AcquireTime: <unset> RenewTime: Wed, 28 Feb 2024 01:21:44 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Thu, 22 Feb 2024 18:30:31 +0800 Thu, 22 Feb 2024 18:30:31 +0800 FlannelIsUp Flannel is running on this node MemoryPressure False Wed, 28 Feb 2024 01:18:30 +0800 Mon, 19 Feb 2024 22:04:38 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 28 Feb 2024 01:18:30 +0800 Mon, 19 Feb 2024 22:04:38 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 28 Feb 2024 01:18:30 +0800 Mon, 19 Feb 2024 22:04:38 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Wed, 28 Feb 2024 01:18:30 +0800 Mon, 19 Feb 2024 23:35:15 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.10.10.100 Hostname: k8s-master Capacity: cpu: 2 ephemeral-storage: 62575768Ki hugepages-2Mi: 0 memory: 3861288Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 57669827694 hugepages-2Mi: 0 memory: 3758888Ki pods: 110 System Info: Machine ID: 9ee2b84718d0437fa9ea4380bdb34024 System UUID: AE134D56-9F2E-B64D-9BA2-6368B1379B3A Boot ID: e0bf44a5-6a0d-4fc0-923d-f5d63089b93f Kernel Version: 3.10.0-1160.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://25.0.3 Kubelet Version: v1.25.0 Kube-Proxy Version: v1.25.0 PodCIDR: 10.2.0.0/24 PodCIDRs: 10.2.0.0/24 Non-terminated Pods: (7 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-flannel kube-flannel-ds-tpm8x 100m (5%) 0 (0%) 50Mi (1%) 0 (0%) 8d kube-system coredns-c676cc86f-q7hcw 100m (5%) 0 (0%) 70Mi (1%) 170Mi (4%) 6d14h kube-system etcd-k8s-master 100m (5%) 0 (0%) 100Mi (2%) 0 (0%) 8d kube-system kube-apiserver-k8s-master 250m (12%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system kube-controller-manager-k8s-master 200m (10%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system kube-proxy-xtllb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system kube-scheduler-k8s-master 100m (5%) 0 (0%) 0 (0%) 0 (0%) 8d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 850m (42%) 0 (0%) memory 220Mi (5%) 170Mi (4%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: <none>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
4.1.3 污点的删除
[root@k8s-master volume]# kubectl taint no k8s-master node-role.kubernetes.io/control-plane:NoSchedule-
node/k8s-master untainted
- 1
- 2
4.1.4 查看k8s上的pod节点信息
[root@k8s-master volume]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dns-test 1/1 Running 2 (37h ago) 3d22h 10.2.1.58 k8s-node-02 <none> <none>
fluentd-59k8k 1/1 Running 1 (37h ago) 3d5h 10.2.2.34 k8s-node-01 <none> <none>
fluentd-hhtls 1/1 Running 1 (37h ago) 3d4h 10.2.1.59 k8s-node-02 <none> <none>
nginx-deploy-69ccc996f9-stgcl 1/1 Running 1 (8m23s ago) 68m 10.2.1.78 k8s-node-02 <none> <none>
nginx-deploy-69ccc996f9-wqp55 1/1 Running 1 (8m8s ago) 68m 10.2.2.71 k8s-node-01 <none> <none>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
4.1.5 测试这个场景，把nginx的pod删除，是否可以再master创建
4.1.6 测试把master的污点给加上，但是污点属性是：NoExecute
- NoExecute 属性会把目前在该节点上的pod都迁移到别的节点
- 目前master给打上污点了，node1节点也给打上污点了，最终新的pod会跑到node2上。
4.2 容忍（Toleration）
- 容忍：是标注在pod上的，当pod被调度时，如果没有配置容忍，则该pod不会被调度到有污点的节点上，只有该pod上标注了满足某个节点的所有污点，则会被调度到这些节点
4.2.1 k8s-node-01上配置了污点，影响是：NoSchedule
- 污点的key-value是：tag=test
- 属性是：NoSchedule ”不能容忍的pod不能被调度到该节点，但是已经存在的节点不会被驱逐。“
[root@k8s-master volume]# kubectl describe no k8s-node-01 | grep -i tain
Taints: tag=test:NoSchedule
Container Runtime Version: docker://25.0.3
- 1
- 2
- 3
4.2.2 pod的spec下面配置容忍度影响是：NoSchedule ，操作是 Equal
- 容忍操作 operator: “Equal” 表示pod上的容忍度和节点的污点相同，才会匹配到该节点
tolerations:
- key: tag # 污点的key
value: test #污点的value
effect: "NoSchedule" # 污点产生的影响
operator: "Equal" # 表示 value与污点的value要相等，也可以设置为Exists表示存在key即可，此时可以不用配置value
- 1
- 2
- 3
- 4
- 5
4.2.3 修改deploy资源中容器的容忍值
4.2.4 查看pod的变动
- 能够容忍污点的节点上，pod会被创建到这个节点上，之前的节点并不会改变
4.2.5 修改deploy资源中容器的的容忍属性：容忍度影响是：NoSchedule ，操作是Exists
4.2.6 查看pod的变动
- 容忍影响是：NoSchedule，代表节点有污点，不能被使用，但是有容忍的操作是匹配到污点的key就可以存在，所以更新deploy后，会有pod创建到这个节点上。
4.2.7 修改deploy资源中容器的的容忍属性：容忍度影响是：NoExecute ，操作是 Exists
4.2.8 查看pod变动
4.2.9 修改deploy资源中容器的的容忍属性：容忍度影响不设置 ，操作是 Exists
4.2.10 查看pod变动
4.2.11 Pod 存在一个 effect 值为 NoExecute 的容忍度指定了可选属性 tolerationSeconds 的值，则表示在给节点添加了上述污点之后， Pod 还能继续在节点上运行的时间。
4.3 k8s内置污点
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/我家小花儿/article/detail/563737
推荐阅读
相关标签
