IT小白

这个屌丝很懒，什么也没留下！

热门标签

云原生|kubernetes|kubernetes集群升级+证书更新（Ubuntu-18.04+kubeadm）_kubeadm更新集群证书

作者：IT小白 | 2024-03-16 05:44:14

踩

kubeadm更新集群证书

前言：

kubernetes集群根据部署手法来分类，一般是两种，一种是基于kubeadm搭建的集群，一种是二进制方式搭建的集群。那么，二进制集群升级和证书更新就完全是手动处理了，而kubeadm的集群可以自动的升级集群（有一些手动的操作，但不多）和证书更新。

kubernetes集群升级的目的和意义：

1，

某些版本有安全隐患的时候需要升级集群

例如，kubernetes CVE-2022-3172 安全漏洞，kubernetes CVE-2019-11247，kubernetesCVE-2018-1002105，kubernetes CVE-2020-8559等等漏洞。

这些内容可以在阿里云查找到，链接如下：

【CVE安全】漏洞修复公告_容器服务Kubernetes版-阿里云帮助中心

一般情况下，不管是kubernetes集群还是什么其它的集群，还是什么tomcat，什么elasticsearch，ssh等等各类软件，当然也包括各个操作系统，比如centos7.Ubuntu，debian等等所有软件类，规避各种各样的安全漏洞基本都是通过升级版本的方式来完成的。

多说一句，在软件制作者的角度来说，升级软件版本的动力一个是安全漏洞的压力，一个是更多的，更新的功能以及更美观的软件界面。在软件使用者的角度来说，升级软件版本的动力第一个是安全漏洞，其次才是更新的功能，最后才是美观的软件界面。

2，

有更加强大的功能

例如，kubernetes集群从最初的1.1版本发展到现在的1.26版本，无疑功能是更多了，某些版本也更加的好用，易用，是能够提升生产效率的。那么，很多的新功能无疑是需要升级集群版本才可以做到的。

3，

有更加美观的界面

例如，kubernetes的组件dashboard，每个版本界面都是不一样的，虽然总体的风格类似，但，其中的某个web界面总能打动某些人的心，这些也是需要通过升级版本（这里是组件的版本）才可以做到的。

4，

对于kubernetes集群来说，kubernetes集群升级会同时刷新集群内的证书期限，某些时候，这个比较特定的机制也是kubernetes升级的一个小小动力吧。

OK，以上是对kubernetes集群升级的目的以及升级的用途做了一个简单的总结，下面将就一个Ubuntu-18.04版本的操作系统内部署的kubernetes-1.22.0集群做一个小小的升级---从1.22.0升级到1.22.2，说明如何升级一个kubernetes集群并更新集群的所有证书。

一，

环境介绍

操作系统：Ubuntu-18.04

集群架构：三个节点，一主两从节点，kubernetes集群初始版本1.22.0，kubeadm部署的集群

IP地址：192.168.123.150，192.168.123.151，192.168.123.152

集群状态：三个节点由于证书过期，全部挂掉的

二，

kubernetes集群和证书的关系

由于kubernetes集群出于安全方面的考虑，因此，从一开始发布就基于RBAC（权限管理系统），到1.16还是1.17版本，这个不太记得了，集群基本是默认使用RBAC了，这就使得集群内的各个组件之间的通信都是基于证书的，例如kubelet的配置文件（这里随便找了一个kubelet的配置文件）：


 cat /etc/kubernetes/kubelet.conf 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1USXdPREEyTXpJek1Wb1hEVE14TVRJd05qQTJNekl6TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT0N4Ck14ekVGcTZEeVZBTjNkelU0MUFUVU94VEgvd1dlUmZkS2F2L0lDSmFjR3VQQXZQOG9LUTc3L1BucDlnMHdsa2YKZFVTWW1WclpEVFR0dHN6YWRPdDlkWWFRTHI0MkM1MlNWWEU4eEl2MSt3MXo3QURYek04N2FISXlCZXVqbm1INwptS3lYdFlyR3I0UmxIM1d4TGU1YmRCYk03QkMrSTRndUZmNThHVFJ3N1QrclpJYXpqcDRPd1pVeFZGRm0rd0Y4ClZTZ2s1VVZXMGxtZ05mamt4WjZPbk1EcDBBREdDZ2JUZkVkazdmdlpGTVFkUkFMU2dmVmNGdGtWcG5xWjBjYVMKRDZaVHBwTWNiNkVqV2JNd1dnQ2F1eFRmNTF5dkhFdTFRTXdXa2Y1V1NZWEsyMmRoN3VjbHlMOGRra1NYaERWKwpEWjR2cmlhR3JZR21UMFQzbTRVQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZMOGFOS1lFM2M1LytqOTlqci9XeXVwNTF2cllNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBTGZCakdEUEo1Q1BVUUlRTkZmOApxMzgwU3hWWjBVMERZWXNGdFg0SW5SbmxCWGFtNkF5VUNMUGJYQkxnNDdCMThhZWN4aTZzVmpFeVZDdFpWUU9ZClZZd3Ftd1RkZ3VlK01sL1hKTGcyZDFVYjZBSVhSVnF2VExVTGt2ck56NVh6RHRjZElCdUlwUXRRUFBVS013NXoKYmZQanhxNndNeks1Z1htWUVONnI1ZzJTR1lNdEc5UGlhMFppcHJFY2lLaUNybW5TN3plaHBVOS9taUFEbWZzaQpld2E2RyszbHlBc3JISFRraTZWMUtNVVJUN3BWWnpFWFJUNElmK25kRDd5N2FwcS9lN3FLMkwwVDd1NnA0WVVuCmowQkdLcWNscGkwRzZzVU54NGxKdkN3aDZJbkh6UEN3OC9BRk9yTzdXRTRWVmM1N0JJVjZrQlFzVmJmaEVnN2kKM0JzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.123.150:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:k8s-master
  name: system:node:k8s-master@kubernetes
current-context: system:node:k8s-master@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:k8s-master
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKVENDQWcyZ0F3SUJBZ0lJS0h2czRzK3FaR0l3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRFeU1EZ3dOak15TXpGYUZ3MHlNekV5TVRJeE1EQXdNamRhTURneApGVEFUQmdOVkJBb1RESE41YzNSbGJUcHViMlJsY3pFZk1CMEdBMVVFQXhNV2MzbHpkR1Z0T201dlpHVTZhemh6CkxXMWhjM1JsY2pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHRQVkZIdWJ1KzgKNkJCZ0I4UGJadzV0OGF2S3VxeklQMHRPN2tRbVN0ZkpTbjNxdjFHS0xwL2U4UllLbkpuUFRzaytYKzRPcitxSwoyeDhXQWxKUGtjQUFXNEJxUlZOT3R5WDhaZHRYc3J2d3VVaGRoZEZkU3dFV3ZzaWRVOEJJamRwQktKVkN0dHR0CkJPQ0hobXBSY0VyQ1JqYkt2Zy90WE9YcDE1cmx0aS92ZHJHaXpKRUt2cUJQZElpVU05UUh3WEZqdmFqMXFnT2YKaUpraTRlZDBnZ3AwSmtxM0grSXp6MFZNUG9YdTJXdnBTTU81dmtGbi9Lay92bUxhaHRwaTlJeCtPN1VMYUxTNQpsdEVGaVJXQnEyT1R2N3YzMndEVUJUcFdwN25wWlU0WE9ra3ltaW5VT3E3ZGhVZWxrQTMwWVhucjB0MklwOWI4ClgxRVNuSEtKMHMwQ0F3RUFBYU5XTUZRd0RnWURWUjBQQVFIL0JBUURBZ1dnTUJNR0ExVWRKUVFNTUFvR0NDc0cKQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0h3WURWUjBqQkJnd0ZvQVV2eG8wcGdUZHpuLzZQMzJPdjliSwo2bm5XK3Rnd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFORk5MVXA5SzRMM0s0NTRXbmp2OHprNVVhREU0ZGovCnpmRXBRdHRWSHRLV3VreHZTRWRRTG15RGZURDFoNm8veGFqNkc3Wk1YbmFWUTQ1WmkzZ3F5ck1ibTdETHhxWDYKV0pLdkZkNUJNY2F4YW16dWhoN0I4R2xrMkNsNUZsK3Z0QnUxREtya293blpydFBTeGFaVjhsUmo2bmFHU1k4RQo4RVVqUWN5VXF3Z2duZWwwanNoaFVKOGdKMHV0MXN5UVAxWEJJcEpsTEZ5b0dDQmNuWkFvdE9oWnFWTWwxcTdOClQ5aVZEVy9IZ2xPbll0WktTbXREN2JvMk4rSDZxNmhaUVFJWmVzWVJxNUoxV0IwclR5SkkzbHJEV3J2QWRrcDEKTUZrekJJK3d2WHF2MEdtTFNYNzRudU4wZnY2K0VvQkdUakFVbkNIdUxvQ0RQNmQxTVBYL3ZZcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBdTA5VVVlNXU3N3pvRUdBSHc5dG5EbTN4cThxNnJNZy9TMDd1UkNaSzE4bEtmZXEvClVZb3VuOTd4RmdxY21jOU95VDVmN2c2djZvcmJIeFlDVWsrUndBQmJnR3BGVTA2M0pmeGwyMWV5dS9DNVNGMkYKMFYxTEFSYSt5SjFUd0VpTjJrRW9sVUsyMjIwRTRJZUdhbEZ3U3NKR05zcStEKzFjNWVuWG11VzJMKzkyc2FMTQprUXErb0U5MGlKUXoxQWZCY1dPOXFQV3FBNStJbVNMaDUzU0NDblFtU3JjZjRqUFBSVXcraGU3WmErbEl3N20rClFXZjhxVCsrWXRxRzJtTDBqSDQ3dFF0b3RMbVcwUVdKRllHclk1Ty91L2ZiQU5RRk9sYW51ZWxsVGhjNlNUS2EKS2RRNnJ0MkZSNldRRGZSaGVldlMzWWluMXZ4ZlVSS2Njb25TelFJREFRQUJBb0lCQVFDcHNWUFZxaW9zM1RwcwpZMk9GaDhhVXB2d3p3OFZjOVVtS1EyYk9yTlpQS2hobmZQMTR0TFJLdCtJNE1zTHZBWVlDQVpWTkNWZE1LQ0lkCnhvV3g1azVINE1zRXlzSWxtQUdLMDErLzJIS2ZtNVZ3UHZJVjIrd3dmMWUyVGZuckVKQWFzNzg5Z2lSQkpFSXYKMi9mbGFBUlFaakxRUHRyemVQb1pmTUdNbmlGd3lIVVJVQWZkL0U0QlFGNS8zUWVFeEpEVkNtaEZEOG5YRXlqRwpLSG5CSlI1TGFCeFpPcmh0bDVqckRZRGxtaWcyZGNPY21TOU5xN2xCUVExb3NmR1FhbjlTODQ1VnlFdjQvcWZrCjAwZkpJN0JpSStDcWNtM0lHRmhNMG5lNXlvVE8zSEo4Sy94bTdpRmxIcVg1cXZaYjJnc2hTRzZIR1E4YjZPV2UKT09id0xxZnRBb0dCQU1VNisvaEdsaVJrNGd4S3NmUE12dERsV3hGTFVWaTdlK1BGQlJ6Zm5FRlh3U055dlI4ZgpYTmlWZStUOFQvcTgwUnZrTmQ1QVNnd1NBTkwraUZnN0R4SXIzUC9GSkc5SjYwcURYMDZzZi9tKzM5c2VzZFp0Ck9Fanl6UXBmNnVtRGVKMHhvNDlrVWtiTkNvQUNPamE4QndDeFdRcjJHb0ZyL0ptZGEra0JteWRUQW9HQkFQTWYKbDkzbXo1QUpiVkE3cVRWckRiVDNDWEsxRjQyV0RuQ0FDRU5kcE8vZlhOZWtuWEhWcjZiRUl6UTluelQ4R2hmOApUUlBtb1VJTVNVVGpHZmdUMTlkbTliODdZb2NkMENCd0pZejlwcmtRaUpMdm13QS82cytQalkvVUwvSHdrT05QCllvcm9YclB5WVpIeHNjMW54di9lRWJic0UyWjdVSXpXbGpxK0lIbGZBb0dBYlN1OUZTeGRKMEFBTDdXWTBzNWUKUU5yemtac1RKLzUvRVJDWlIrWXVZNnpqWjIrM1oyYkF5ZEhVaG1kekRlTStEQ1pCK3dleTlRTnlHVmh5dUFQWQp6OElmemlPZGkweHJSUTk2emQyRjZRUFNmVU44Uktpb0l4amlqZitSMURmRnA1MDJYOFMwRmlTZ3owSnNYcWV0CmFLRENIT01rd01hNVIzNXZvTVlXejZrQ2dZRUE1TWdiR2ZhRDVjL3BMUEluaFp3SzF2c015Z045ZVgvMmNJa2EKdllIV250ODZ0N1l4YnBpZDVUbDJ3MGNsbFMrU3duVnFkc3ExZnJpZkRoTURNZjVDUTNHZzJXWmhqakpRMHVXVgpnSHFFdEd2SmlUT3VVV3JVWktONm5Ca1pVUHVHN0ZDY3M0aDg3YXF0aEMvRG1ENEs5bVlibDEzSjE4czgvbnRECi9WMUNvOU1DZ1lFQWlCb2tPTEFBNlFDbHNYOXcwNlMrZ1pXT0FpbDlRRXhHalhVR2o0ZEtielpibmgrWnkwbUoKNW9LRHpydTFKeGdoa1JtQ0ZiZmx6aHpMMktWU2xIbXNPYWZDSU1JSHJSZTdic0gvdjg2SVpkYXlnTWxLckJ2LwpXUVMxdmJoQjVvNGdwRkE5aG0rMFcwTW9ZOUVsaERPUG52ajFxT2lTRVArN0dXdDAxOW5HdWdzPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

可以看到，此配置文件携带了三个证书和api-server通信认证，也就是说，如果此配置文件没有证书或者证书过期了，那么，kubelet服务奖不会成功启动的。

同样的，controller-manager服务也携带了证书来与api-server通信，同样的，如果此配置文件内的证书是不对的，或者证书过期了，controller-manager服务也不会成功启动，即使这个服务是存在于静态pod内。


 cat /etc/kubernetes/controller-manager.conf 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1USXdPREEyTXpJek1Wb1hEVE14TVRJd05qQTJNekl6TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT0N4Ck14ekVGcTZEeVZBTjNkelU0MUFUVU94VEgvd1dlUmZkS2F2L0lDSmFjR3VQQXZQOG9LUTc3L1BucDlnMHdsa2YKZFVTWW1WclpEVFR0dHN6YWRPdDlkWWFRTHI0MkM1MlNWWEU4eEl2MSt3MXo3QURYek04N2FISXlCZXVqbm1INwptS3lYdFlyR3I0UmxIM1d4TGU1YmRCYk03QkMrSTRndUZmNThHVFJ3N1QrclpJYXpqcDRPd1pVeFZGRm0rd0Y4ClZTZ2s1VVZXMGxtZ05mamt4WjZPbk1EcDBBREdDZ2JUZkVkazdmdlpGTVFkUkFMU2dmVmNGdGtWcG5xWjBjYVMKRDZaVHBwTWNiNkVqV2JNd1dnQ2F1eFRmNTF5dkhFdTFRTXdXa2Y1V1NZWEsyMmRoN3VjbHlMOGRra1NYaERWKwpEWjR2cmlhR3JZR21UMFQzbTRVQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZMOGFOS1lFM2M1LytqOTlqci9XeXVwNTF2cllNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBTGZCakdEUEo1Q1BVUUlRTkZmOApxMzgwU3hWWjBVMERZWXNGdFg0SW5SbmxCWGFtNkF5VUNMUGJYQkxnNDdCMThhZWN4aTZzVmpFeVZDdFpWUU9ZClZZd3Ftd1RkZ3VlK01sL1hKTGcyZDFVYjZBSVhSVnF2VExVTGt2ck56NVh6RHRjZElCdUlwUXRRUFBVS013NXoKYmZQanhxNndNeks1Z1htWUVONnI1ZzJTR1lNdEc5UGlhMFppcHJFY2lLaUNybW5TN3plaHBVOS9taUFEbWZzaQpld2E2RyszbHlBc3JISFRraTZWMUtNVVJUN3BWWnpFWFJUNElmK25kRDd5N2FwcS9lN3FLMkwwVDd1NnA0WVVuCmowQkdLcWNscGkwRzZzVU54NGxKdkN3aDZJbkh6UEN3OC9BRk9yTzdXRTRWVmM1N0JJVjZrQlFzVmJmaEVnN2kKM0JzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.123.150:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:kube-controller-manager
  name: system:kube-controller-manager@kubernetes
current-context: system:kube-controller-manager@kubernetes
kind: Config
preferences: {}
users:
- name: system:kube-controller-manager
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGakNDQWY2Z0F3SUJBZ0lJVXJjNDVZUVF1NE13RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TVRFeU1EZ3dOak15TXpGYUZ3MHlNekV5TVRJeE1EQXdNamRhTUNreApKekFsQmdOVkJBTVRIbk41YzNSbGJUcHJkV0psTFdOdmJuUnliMnhzWlhJdGJXRnVZV2RsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUtkU2d1d1NDMVhVSmlvRjIvbFhaNHQwWjBQRVEvc0QKWk9ZTzdPdlUwbHVSYngzcW1VaEN6NjlPbFBtY0h0OUllRHl2a09qYzFBZ2FlZWwzR3VnSjlKZ2xFM0RmZm5hegpZVVBrOXVzTVBIQjZtMUhNR2ZjUVloZkhLUG9TNmk5bEVWZUhKOXEvOTVaSUdGKzVMR0F2eXVBUWg1NmZYT1hrCjRZRTRsSmYzRGhzdGRNeEtDNXVZTXpxazR3RmlNVkNxYkRhdlVqc3VmZzhhYlFQQWFVQ3NieWFFMm04RVllOGgKb2VkTkdVdml5WkxrQUM2ckM4bGVIeGNBbjB2ZlVvYXU3UzJFdWpNK01jV3RLZ1o4NmVVdjJaU2dXemFCa1VWZAo4QlRxK0VDOUtlb1pWRHJnUlBpejVub3FsVDBYTTJacUVJK01Ud1lTRWt3aTJ5WlpqNS9yWFQ4Q0F3RUFBYU5XCk1GUXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIKL3dRQ01BQXdId1lEVlIwakJCZ3dGb0FVdnhvMHBnVGR6bi82UDMyT3Y5Yks2bm5XK3Rnd0RRWUpLb1pJaHZjTgpBUUVMQlFBRGdnRUJBR0d3NGxaaTZDSzB5YWpsOGNlUEg3a20vY1VLcGFFTld1aWJyVTQyNEVDM1A0UkFZZlJXCmRranZjNm9JbVhzR2V2T2Y0ZWlJU0dxaWNOb254N2RkWUxLY2tDaWxLQmcwS2hyZGFRT3o3N3ZCQitvamczbmgKMHByb05oYW12dkVpc0lUY212cmdzNTZqMk1Id2lUK3ZHeXFHbWxPOG9TRHZmWVFnMUVqTkRxWlVEd0g3OFlHYwowT0h5cXU3SW1hYngvKzdWOGcvMmlBS3NEVVVja3I3UHVMWWI3RlA0ZlZvVjlDWkIzVHI3bXFRQ2FrUmJmMnF1CjUvd3pEMG9lYjFBeHV0aUFSVjBlM2JBZUxXV0tqckEyNW9ISVBCRW1zTEFQSmtlMDVlRk9LK05ZUHBMdjBNU04KVnlzaXEzcVl0RkxZSzRaN0kyaGgrKzc5MXM4Y2g1TDNFeGM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBcDFLQzdCSUxWZFFtS2dYYitWZG5pM1JuUThSRCt3Tms1ZzdzNjlUU1c1RnZIZXFaClNFTFByMDZVK1p3ZTMwaDRQSytRNk56VUNCcDU2WGNhNkFuMG1DVVRjTjkrZHJOaFErVDI2d3c4Y0hxYlVjd1oKOXhCaUY4Y28raExxTDJVUlY0Y24yci8zbGtnWVg3a3NZQy9LNEJDSG5wOWM1ZVRoZ1RpVWwvY09HeTEwekVvTAptNWd6T3FUakFXSXhVS3BzTnE5U095NStEeHB0QThCcFFLeHZKb1RhYndSaDd5R2g1MDBaUytMSmt1UUFMcXNMCnlWNGZGd0NmUzk5U2hxN3RMWVM2TXo0eHhhMHFCbnpwNVMvWmxLQmJOb0dSUlYzd0ZPcjRRTDBwNmhsVU91QkUKK0xQbWVpcVZQUmN6Wm1vUWo0eFBCaElTVENMYkpsbVBuK3RkUHdJREFRQUJBb0lCQUZ0Vm9QMjRBOVFBRUMwVQpNYlZ6enFQREVMTmZLVFNWNzdmZElkckJ1Mm9jZ3lremJDU1R3OGFRQUtZWVlJbkZoMHlwRVZMcmFCcGNTWHYxCmRneC9rcktTV29CY255MndVVUc4ZEVSdDAzZ2FsVG9iVFhrZHlrM3NleU8ydTNyUGtwM1N1eUNmZFVqbFpkaXEKdmR4cmVqVEJFU2EzR3dDcTVhV2grd3JRNHpSVnc5eERTVGF0MXA1cS82UFRPcHhkWElJRWhHQVd4SEE2RnVnSgpuR2RrMWFxT1ZweFFSZVZ1QVluNVlYemZ0MXByU0wwdWQwOFRNd3FaRUowa0dabUdIODJDV0tTRnQ4cEs1MnJ2CnpURDFnWE5hZzhrNGF3czhrQXpnT1lWek8wMHR2SjdVQjNFcVBSSThKQVJucWpyTGpNaTRqeDVMZGFQZkc2VUQKMERwNFdsRUNnWUVBelRYT2FJdUQ2QUhkaTNPaFlpQ2puOUwwM3ZOUGY1c1JtRmJzclc2UFFxdEVTTit2aVFxMApFMVREU2REM3BpSndqSC9zQTNhRW5ISStRZVYxT1ZFbVR3YkxpaFhOVWtYTWo2OWRmVHdxYTd1MnJEWm0zQnF6CkVrYUI3N0FHb3BvR2hRajBHcGMzTnY5MDZwN3orekVEKzFSajFaeGtsaXVXQ2NDMzViVlZqejBDZ1lFQTBMd2QKa1VYdGxhK0hWb2dicFJsRlE2dTVtbVFzRXpMYWdrQkFoRDFqYkpORWtVL1o0RjJGNkwwVTN0SWtWNWRyRG9XRQowbUo5TmdELzhPWVdXbTZSdVo1bVhPZlRSOVdkc2k5MGpUNU9NRkJHbHVMdlUyNllhTEhUdmMwbHdiQlVPelJECjJvb2ZEZURjU3Y2d1oxSVU3QlQ2YTA4dWtON0FpNVR6ZWhrQ1ppc0NnWUVBdWVNeHRKWWN5TDlYMW9qSitiK2oKT0pXNTUzUHo0WjJ3bEpTNUZHbUFNRjVBSHRzeGdTeEc3dlByYXlSMkVQSkZqYUFiUlEvSkZJYVFTdFQyR1JPZgpaaHE3cWJ3U0g2TEdxS21zUUZPT0FjVXF0bGtaVit4L3BlQmt0NkIyZ2ppUUMxYU8rTDlkN3QzOUpNTVVNOGkwCjJLZ2JQMWJKN3haUWRVa3p6RXMwMCtrQ2dZQlJraUlQNG43dEx4STVpNmthQk4wZmk5MVZhMjRaOXBhVHJoNUkKVDJFcVRnYk9ycURiWUZEeldlanRCcnd6Q3JaSWozOFBaSFBBQmZYL0l6dDdEWmlmTERxZWRlNElOWCtSNFordgpqcmlwZ3NXRE01NEpRY0FIc2U2b1RxSkJwZkhVelNEekoyVHBYSVZhUFZ1Y2xPUWVPamgrZFF3aWl4bzlzZkRRCk56UEx6d0tCZ1FDaVU2QUZ4NjdPdW1jRGtwK21tRXpYNWpsQmtlZk9XakJlNlZ4ZGwzNjA1TjJ6YS9CbDlINzEKZ3pjdlZhQTY5RC9uVG50bE1xaWhnUm1NTGdEMGovOWkrV0VkMzR2Y0JTOHI2M1VZZVRvUWRoakJCZC8yeUM5NAppQ0ZiNlZ0dGRrSmg5SEhkV1pTRkxwQ0FmTnVlMVRxclA5L1RobTNRaTFjM2lZZVJUblh0YkE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

三，

报错信息

集群很明显没有启动


root@k8s-master:~# kubectl get no
The connection to the server 192.168.123.150:6443 was refused - did you specify the right host or port?

系统日志报错如下；

12月8号证书已经过期。


ec 12 23:11:18 k8s-master kubelet[2750]: I1212 23:11:18.900395    2750 server.go:868] "Client rotation is on, will bootstrap in background"
Dec 12 23:11:18 k8s-master kubelet[2750]: E1212 23:11:18.903330    2750 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2022-12-08 06:32:35 +0000 UTC
Dec 12 23:11:18 k8s-master kubelet[2750]: E1212 23:11:18.905482    2750 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or

查看证书时间，再次确认一哈：


root@k8s-master:~# kubeadm  certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 08, 2022 06:32 UTC   <invalid>                               no      
apiserver                  Dec 08, 2022 06:32 UTC   <invalid>       ca                      no      
apiserver-etcd-client      Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   Dec 08, 2022 06:32 UTC   <invalid>       ca                      no      
controller-manager.conf    Dec 08, 2022 06:32 UTC   <invalid>                               no      
etcd-healthcheck-client    Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
etcd-server                Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         Dec 08, 2022 06:32 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             Dec 08, 2022 06:32 UTC   <invalid>                               no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 06, 2031 06:32 UTC   8y              no      
etcd-ca                 Dec 06, 2031 06:32 UTC   8y              no      
front-proxy-ca          Dec 06, 2031 06:32 UTC   8y              no

四，

kubernetes集群的升级

添加阿里云的apt源：


cat >/etc/apt/sources.list.d/kubernetes.list <<EOF
# 阿里源
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe 
multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe 
multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe 
multiverse
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

更新apt源：

sudo apt-get update

输出如下：

有警告，可以不处理，也可以处理一哈，这里的警告是说两个文件的源重复配置，冲突了，因此，删除/etc/apt/sources.list这个文件即可。


root@k8s-master:~# sudo apt-get update
Hit:1 http://mirrors.aliyun.com/ubuntu bionic InRelease                 
Hit:2 http://mirrors.aliyun.com/ubuntu bionic-updates InRelease                                                                  
Hit:3 https://mirrors.aliyun.com/docker-ce/linux/ubuntu bionic InRelease                                                         
Hit:4 http://mirrors.aliyun.com/ubuntu bionic-backports InRelease                                       
Hit:5 http://mirrors.aliyun.com/ubuntu bionic-security InRelease                                                                  
Hit:6 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease                           
Get:7 http://mirrors.aliyun.com/ubuntu bionic-proposed InRelease [242 kB]                             
Get:8 http://mirrors.aliyun.com/ubuntu bionic/universe Sources [9,051 kB]       
Get:9 http://mirrors.aliyun.com/ubuntu bionic/restricted Sources [5,324 B]                                                                                                                                                                 
Get:10 http://mirrors.aliyun.com/ubuntu bionic/main Sources [829 kB]                                                                                                                                                                       
Get:11 http://mirrors.aliyun.com/ubuntu bionic/multiverse Sources [181 kB]                                                                                                                                                                 
Get:12 http://mirrors.aliyun.com/ubuntu bionic-updates/restricted Sources [33.1 kB]                                                                                                                                                        
Get:13 http://mirrors.aliyun.com/ubuntu bionic-updates/universe Sources [486 kB]                                                                                                                                                           
Get:14 http://mirrors.aliyun.com/ubuntu bionic-updates/multiverse Sources [17.2 kB]                                                                                                                                                        
Get:15 http://mirrors.aliyun.com/ubuntu bionic-updates/main Sources [537 kB]                                                                                                                                                               
Get:16 http://mirrors.aliyun.com/ubuntu bionic-backports/universe Sources [6,600 B]                                                                                                                                                        
Get:17 http://mirrors.aliyun.com/ubuntu bionic-backports/main Sources [10.5 kB]                                                                                                                                                            
Get:18 http://mirrors.aliyun.com/ubuntu bionic-security/restricted Sources [30.2 kB]                                                                                                                                                       
Get:19 http://mirrors.aliyun.com/ubuntu bionic-security/multiverse Sources [10.6 kB]                                                                                                                                                       
Get:20 http://mirrors.aliyun.com/ubuntu bionic-security/main Sources [288 kB]                                                                                                                                                              
Get:21 http://mirrors.aliyun.com/ubuntu bionic-security/universe Sources [309 kB]                                                                                                                                                          
Get:22 http://mirrors.aliyun.com/ubuntu bionic-proposed/restricted Sources [8,164 B]                                                                                                                                                       
Get:23 http://mirrors.aliyun.com/ubuntu bionic-proposed/universe Sources [9,428 B]                                                                                                                                                         
Get:24 http://mirrors.aliyun.com/ubuntu bionic-proposed/main Sources [75.6 kB]                                                                                                                                                             
Get:25 http://mirrors.aliyun.com/ubuntu bionic-proposed/main amd64 Packages [145 kB]                                                                                                                                                       
Get:26 http://mirrors.aliyun.com/ubuntu bionic-proposed/main Translation-en [32.2 kB]                                                                                                                                                      
Get:27 http://mirrors.aliyun.com/ubuntu bionic-proposed/restricted amd64 Packages [132 kB]                                                                                                                                                 
Get:28 http://mirrors.aliyun.com/ubuntu bionic-proposed/restricted Translation-en [18.5 kB]                                                                                                                                                
Get:29 http://mirrors.aliyun.com/ubuntu bionic-proposed/universe amd64 Packages [11.0 kB]                                                                                                                                                  
Get:30 http://mirrors.aliyun.com/ubuntu bionic-proposed/universe Translation-en [6,676 B]                                                                                                                                                  
Fetched 12.5 MB in 14s (878 kB/s)                                                                                                                                                                                                          
Reading package lists... Done
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:3 and /etc/apt/sources.list.d/kubernetes.list:2
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:3 and /etc/apt/sources.list.d/kubernetes.list:2

2，

重新安装kubelet，kubeadm，kubectl

sudo apt-get install kubelet=1.22.2-00  kubeadm=1.22.2-00 kubectl=1.22.2-00  -y

3，

开始升级

先更新证书以启动kubelet

kubeadm  certs renew all

输出如下：


[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
 
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
 
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

以上输出最后提示需要重启restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd，在此之前先看看证书是否更新：

已经更新了，但kubelet的配置文件等等这些还是使用的旧证书，因此，此时的kubelet等服务还是不能启动的状态


root@k8s-master:~# kubeadm  certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 12, 2023 15:29 UTC   364d                                    no      
apiserver                  Dec 12, 2023 15:29 UTC   364d            ca                      no      
apiserver-etcd-client      Dec 12, 2023 15:29 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Dec 12, 2023 15:29 UTC   364d            ca                      no      
controller-manager.conf    Dec 12, 2023 15:29 UTC   364d                                    no      
etcd-healthcheck-client    Dec 12, 2023 15:29 UTC   364d            etcd-ca                 no      
etcd-peer                  Dec 12, 2023 15:29 UTC   364d            etcd-ca                 no      
etcd-server                Dec 12, 2023 15:29 UTC   364d            etcd-ca                 no      
front-proxy-client         Dec 12, 2023 15:29 UTC   364d            front-proxy-ca          no      
scheduler.conf             Dec 12, 2023 15:29 UTC   364d                                    no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 06, 2031 06:32 UTC   8y              no      
etcd-ca                 Dec 06, 2031 06:32 UTC   8y              no      
front-proxy-ca          Dec 06, 2031 06:32 UTC   8y              no

因此，这个时候需要删除这些服务的配置文件，使用kubeadm重新生成这些文件：


root@k8s-master:~# rm -rf /etc/kubernetes/*.conf
root@k8s-master:~# kubeadm  init phase kubeconfig all
I1212 23:35:49.775848   19629 version.go:255] remote version is much newer: v1.26.0; falling back to: stable-1.22
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

如上所示，可以看到新的配置文件都使用的新的证书了，此时kubelet可以重启了。


root@k8s-master:~# systemctl restart kubelet
root@k8s-master:~# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Mon 2022-12-12 23:36:57 CST; 2s ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 21307 (kubelet)
    Tasks: 20 (limit: 2210)
   CGroup: /system.slice/kubelet.service
           ├─21307 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=registry.
           └─21589 /opt/cni/bin/calico
 
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316164   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"policysync\" (UniqueName: \"kubernetes.io/host-path/5ef5e743-ee71-4c80-a5
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316204   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flexvolume-dir\" (UniqueName: \"kubernetes.io/host-path/c4992d6a9ff2341f1
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316279   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwxm\" (UniqueName: \"kubernetes.io/projected/8ad3a63e-e
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316323   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/5ef5e743-ee71-4c80-
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316364   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6q6\" (UniqueName: \"kubernetes.io/projected/5ef5e743-e
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316407   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeconfig\" (UniqueName: \"kubernetes.io/host-path/c4992d6a9ff2341f1f1b0
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316447   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-run-calico\" (UniqueName: \"kubernetes.io/host-path/5ef5e743-ee71-4c8
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316497   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-local-net-dir\" (UniqueName: \"kubernetes.io/host-path/5ef5e743-ee71
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316540   21307 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-bin-dir\" (UniqueName: \"kubernetes.io/host-path/5ef5e743-ee71-4c80-a
Dec 12 23:36:59 k8s-master kubelet[21307]: I1212 23:36:59.316562   21307 reconciler.go:157] "Reconciler: start to sync state"

此时查看节点状态发现报错，因为是使用的.kube目录下congfig，这个文件是使用的旧的证书，因此，删除它，重新引用变量，并覆盖旧的config：


root@k8s-master:~# kubectl get no 
error: You must be logged in to the server (Unauthorized)

OK，master节点的正式就更新完了，那么，工作节点的证书怎么处理呢？

解决方案为：由于整个集群是kubeadm搭建的，而etcd是静态pod 形式存在在master节点的，因此，master节点恢复后，确认etcd正常后，工作节点重新加入集群即可：

删除工作节点：


root@k8s-master:~# kubectl delete nodes k8s-node1 
node "k8s-node1" deleted
root@k8s-master:~# kubectl delete nodes k8s-node2
node "k8s-node2" deleted

生成加入节点命令：


root@k8s-master:~# kubeadm token create --print-join-command
kubeadm join 192.168.123.150:6443 --token 692e4m.o8njp7guix9w5jne --discovery-token-ca-cert-hash sha256:fb346dffae444c802ffeaee5269375b3727c05d92a4365231772de414cbd6923

在工作节点重设节点并重新加入节点（151和152节点执行）：


root@k8s-node1:~# kubeadm reset -f
[preflight] Running pre-flight checks
W1212 23:52:30.989714   84567 removeetcdmember.go:80] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
 
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
 
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.
 
If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
root@k8s-node1:~# kubeadm join 192.168.123.150:6443 --token 692e4m.o8njp7guix9w5jne --discovery-token-ca-cert-hash sha256:fb346dffae444c802ffeaee5269375b3727c05d92a4365231772de414cbd6923
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.


root@k8s-master:~# export KUBECONFIG=/etc/kubernetes/admin.conf 
root@k8s-master:~# kubectl get no 
NAME         STATUS     ROLES                  AGE    VERSION
k8s-master   Ready      control-plane,master   369d   v1.22.2
k8s-node1    NotReady   <none>                 369d   v1.22.0
k8s-node2    NotReady   <none>                 369d   v1.22.0
root@k8s-master:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y

此时，回到master节点，查看节点状态，可以看到恢复正常了：


root@k8s-master:~# kubectl get no
NAME         STATUS   ROLES                  AGE     VERSION
k8s-master   Ready    control-plane,master   369d    v1.22.2
k8s-node1    Ready    <none>                 2m18s   v1.22.0
k8s-node2    Ready    <none>                 12s     v1.22.0

查看pod，也都是正常的状态了：


root@k8s-master:~# kubectl get po -A
NAMESPACE       NAME                                       READY   STATUS    RESTARTS        AGE
default         front-end-6f94965fd9-dq7t8                 1/1     Running   0               27m
default         guestbook-86bb8f5bc9-mcdvg                 1/1     Running   0               27m
default         guestbook-86bb8f5bc9-zh7zq                 1/1     Running   0               27m
default         nfs-client-provisioner-56dd5765dc-gp6mz    1/1     Running   0               27m

五，

集群升级

先升级kubeadm到1.22.10


root@k8s-master:~# apt-get install kubeadm=1.22.10-00
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  cri-tools
The following packages will be upgraded:
  cri-tools kubeadm
2 upgraded, 0 newly installed, 0 to remove and 193 not upgraded.
Need to get 26.7 MB of archives.
After this operation, 19.9 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 cri-tools amd64 1.25.0-00 [17.9 MB]
Get:2 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubeadm amd64 1.22.10-00 [8,728 kB]                                                                                                                           
Fetched 26.7 MB in 51s (522 kB/s)                                                                                                                                                                                                          
(Reading database ... 67719 files and directories currently installed.)
Preparing to unpack .../cri-tools_1.25.0-00_amd64.deb ...
Unpacking cri-tools (1.25.0-00) over (1.19.0-00) ...
Preparing to unpack .../kubeadm_1.22.10-00_amd64.deb ...
Unpacking kubeadm (1.22.10-00) over (1.22.2-00) ...
Setting up cri-tools (1.25.0-00) ...
Setting up kubeadm (1.22.10-00) ...

在升级kubernetes集群：

kubeadm upgrade apply v1.22.10

输出如下：


root@k8s-master:~# kubeadm upgrade apply v1.22.10
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.22.10"
[upgrade/versions] Cluster version: v1.22.0
[upgrade/versions] kubeadm version: v1.22.10
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.22.10"...
Static pod: kube-apiserver-k8s-master hash: 2761ce4ea551870599dbc22df4805251
Static pod: kube-controller-manager-k8s-master hash: c4992d6a9ff2341f1f1b0d3058a62049
Static pod: kube-scheduler-k8s-master hash: 938652c36b8ab3b7a6345373ea6e1ded
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-master hash: 942d6def80e349c32e518bf1fb533795
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-12-13-00-10-26/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-k8s-master hash: 942d6def80e349c32e518bf1fb533795
Static pod: etcd-k8s-master hash: 942d6def80e349c32e518bf1fb533795
Static pod: etcd-k8s-master hash: ee7d79d2b2967f03af72732ecda2b44f
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests149634972"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-12-13-00-10-26/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8s-master hash: 2761ce4ea551870599dbc22df4805251
Static pod: kube-apiserver-k8s-master hash: 2761ce4ea551870599dbc22df4805251
Static pod: kube-apiserver-k8s-master hash: 2761ce4ea551870599dbc22df4805251
Static pod: kube-apiserver-k8s-master hash: d2601c13ace3af023db083125c56d47b
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-12-13-00-10-26/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8s-master hash: c4992d6a9ff2341f1f1b0d3058a62049
Static pod: kube-controller-manager-k8s-master hash: 648269e02b16780e315b096eec7eaa5d
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-12-13-00-10-26/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8s-master hash: 938652c36b8ab3b7a6345373ea6e1ded
Static pod: kube-scheduler-k8s-master hash: ec4c9f7722e075d30583bde88d591749
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated)
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
 
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.22.10". Enjoy!
 
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

提示kubelet也应该升级：


root@k8s-master:~# apt-get install kubelet=1.22.10-00
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  kubelet
1 upgraded, 0 newly installed, 0 to remove and 193 not upgraded.
Need to get 19.2 MB of archives.
After this operation, 32.1 MB disk space will be freed.
Get:1 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubelet amd64 1.22.10-00 [19.2 MB]
Fetched 19.2 MB in 42s (453 kB/s)                                                                                                                                                                                                          
(Reading database ... 67719 files and directories currently installed.)
Preparing to unpack .../kubelet_1.22.10-00_amd64.deb ...
Unpacking kubelet (1.22.10-00) over (1.22.2-00) ...
Setting up kubelet (1.22.10-00) ...

证书的时间也刷新了：

（为什么一开始不直接升级呢？因为升级的时候需要集群是正常运行的，但前面证书是已经过期，集群宕机状态了，没办法升级）


root@k8s-master:~# kubeadm  certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 12, 2023 16:11 UTC   364d            ca                      no      
apiserver                  Dec 12, 2023 16:11 UTC   364d            ca                      no      
apiserver-etcd-client      Dec 12, 2023 16:11 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Dec 12, 2023 16:11 UTC   364d            ca                      no      
controller-manager.conf    Dec 12, 2023 16:11 UTC   364d            ca                      no      
etcd-healthcheck-client    Dec 12, 2023 16:10 UTC   364d            etcd-ca                 no      
etcd-peer                  Dec 12, 2023 16:10 UTC   364d            etcd-ca                 no      
etcd-server                Dec 12, 2023 16:10 UTC   364d            etcd-ca                 no      
front-proxy-client         Dec 12, 2023 16:11 UTC   364d            front-proxy-ca          no      
scheduler.conf             Dec 12, 2023 16:11 UTC   364d            ca                      no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 06, 2031 06:32 UTC   8y              no      
etcd-ca                 Dec 06, 2031 06:32 UTC   8y              no      
front-proxy-ca          Dec 06, 2031 06:32 UTC   8y              no

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/IT小白/article/detail/247446