k8s 集群重启报错：The connection to the server 192.168.92.26:6443 was refused_--pod-infra-container-image will not be pruned by

[root@k8s-master ~]# kubectl get node
The connection to the server 192.168.92.26:6443 was refused - did you specify the right host or port?
1
2

查到是kubelet进程没有启动

[root@k8s-master ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since 六 2024-03-23 11:31:48 CST; 734ms ago
     Docs: https://kubernetes.io/docs/
  Process: 3023 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 3023 (code=exited, status=1/FAILURE)

3月 23 11:31:48 k8s-master systemd[1]: Unit kubelet.service entered failed state.
3月 23 11:31:48 k8s-master systemd[1]: kubelet.service failed.
1
2
3
4
5
6
7
8
9
10
11
12

查看kubelet日志

[root@k8s-master ~]# journalctl -u kubelet -n 50
1

-- Logs begin at 六 2024-03-23 11:24:05 CST, end at 六 2024-03-23 11:32:39 CST. --
3月 23 11:32:08 k8s-master systemd[1]: Unit kubelet.service entered failed state.
3月 23 11:32:08 k8s-master systemd[1]: kubelet.service failed.
3月 23 11:32:19 k8s-master systemd[1]: kubelet.service holdoff time over, scheduling restart.
3月 23 11:32:19 k8s-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
3月 23 11:32:19 k8s-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
3月 23 11:32:19 k8s-master kubelet[3109]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the
3月 23 11:32:19 k8s-master kubelet[3109]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will 
3月 23 11:32:19 k8s-master kubelet[3109]: I0323 11:32:19.252789    3109 server.go:203] "--pod-infra-container-image will not be pruned by the image garbage collec
3月 23 11:32:19 k8s-master kubelet[3109]: I0323 11:32:19.261821    3109 server.go:467] "Kubelet version" kubeletVersion="v1.28.2"
3月 23 11:32:19 k8s-master kubelet[3109]: I0323 11:32:19.261896    3109 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
3月 23 11:32:19 k8s-master kubelet[3109]: I0323 11:32:19.262392    3109 server.go:895] "Client rotation is on, will bootstrap in background"
3月 23 11:32:19 k8s-master kubelet[3109]: I0323 11:32:19.265353    3109 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-
3月 23 11:32:19 k8s-master kubelet[3109]: I0323 11:32:19.267282    3109 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernet
3月 23 11:32:19 k8s-master kubelet[3109]: I0323 11:32:19.276086    3109 server.go:725] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaultin
3月 23 11:32:19 k8s-master kubelet[3109]: E0323 11:32:19.276458    3109 run.go:74] "command failed" err="failed to run Kubelet: running with swap on is not suppor
3月 23 11:32:19 k8s-master systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
3月 23 11:32:19 k8s-master systemd[1]: Unit kubelet.service entered failed state.
3月 23 11:32:19 k8s-master systemd[1]: kubelet.service failed.
3月 23 11:32:29 k8s-master systemd[1]: kubelet.service holdoff time over, scheduling restart.
3月 23 11:32:29 k8s-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
3月 23 11:32:29 k8s-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
3月 23 11:32:29 k8s-master kubelet[3134]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the
3月 23 11:32:29 k8s-master kubelet[3134]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will 
3月 23 11:32:29 k8s-master kubelet[3134]: I0323 11:32:29.495156    3134 server.go:203] "--pod-infra-container-image will not be pruned by the image garbage collec
3月 23 11:32:29 k8s-master kubelet[3134]: I0323 11:32:29.503858    3134 server.go:467] "Kubelet version" kubeletVersion="v1.28.2"
3月 23 11:32:29 k8s-master kubelet[3134]: I0323 11:32:29.503934    3134 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
3月 23 11:32:29 k8s-master kubelet[3134]: I0323 11:32:29.504374    3134 server.go:895] "Client rotation is on, will bootstrap in background"
3月 23 11:32:29 k8s-master kubelet[3134]: I0323 11:32:29.509014    3134 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-
3月 23 11:32:29 k8s-master kubelet[3134]: I0323 11:32:29.512142    3134 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernet
3月 23 11:32:29 k8s-master kubelet[3134]: I0323 11:32:29.541685    3134 server.go:725] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaultin
3月 23 11:32:29 k8s-master kubelet[3134]: E0323 11:32:29.542625    3134 run.go:74] "command failed" err="failed to run Kubelet: running with swap on is not suppor
3月 23 11:32:29 k8s-master systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
3月 23 11:32:29 k8s-master systemd[1]: Unit kubelet.service entered failed state.
3月 23 11:32:29 k8s-master systemd[1]: kubelet.service failed.
3月 23 11:32:39 k8s-master systemd[1]: kubelet.service holdoff time over, scheduling restart.
3月 23 11:32:39 k8s-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

日志里面有一条报错很关键： command failed" err="failed to run Kubelet: running with swap on is not suppor

说明服务器上swap分区没有关闭

[root@k8s-master ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.8G        281M        1.1G        9.5M        435M        1.4G
Swap:          2.0G          0B        2.0G
1
2
3
4

关闭swap分区，并重启kubelet
关闭swap分区

[root@k8s-master ~]# swapoff -a
1

注释掉/etc/fstab 关于swap分区的自动挂载信息

[root@k8s-master ~]# cat -n /etc/fstab 
     1	
     2	#
     3	# /etc/fstab
     4	# Created by anaconda on Fri Jan  5 07:44:09 2024
     5	#
     6	# Accessible filesystems, by reference, are maintained under '/dev/disk'
     7	# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
     8	#
     9	/dev/mapper/centos-root /                       xfs     defaults        0 0
    10	UUID=31669ea6-db8d-4cb0-8fee-b6f833aa15e3 /boot                   xfs     defaults        0 0
    11	/dev/mapper/centos-swap swap                    swap    defaults        0 0
1
2
3
4
5
6
7
8
9
10
11
12

[root@k8s-master ~]# sed -i  '11s/^/#/g' /etc/fstab 
[root@k8s-master ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Fri Jan  5 07:44:09 2024
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=31669ea6-db8d-4cb0-8fee-b6f833aa15e3 /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0
1
2
3
4
5
6
7
8
9
10
11
12
13

重启kubelet

[root@k8s-master ~]# systemctl restart kubelet
[root@k8s-master ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 六 2024-03-23 11:50:55 CST; 7s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 10941 (kubelet)
    Tasks: 11
   Memory: 30.7M
   CGroup: /system.slice/kubelet.service
           └─10941 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/k...

3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.350904   10941 topology_manager.go:215] "Topology Admit Handler" podUID="7b2a2e08-d9c0-4131...6c-sm85q"
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.351073   10941 topology_manager.go:215] "Topology Admit Handler" podUID="24593fe2-2171-4784...xy-bzqd2"
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.351194   10941 topology_manager.go:215] "Topology Admit Handler" podUID="44175e7b-8496-40ca...ds-hvsm5"
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.448530   10941 desired_state_of_world_populator.go:159] "Finished populating initial desire...of world"
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.480322   10941 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started f...
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.481079   10941 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started f...
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.481331   10941 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started f...
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.481390   10941 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started f...
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.481431   10941 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started f...
3月 23 11:50:56 k8s-master kubelet[10941]: I0323 11:50:56.481539   10941 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started f...
Hint: Some lines were ellipsized, use -l to show in full.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

[root@k8s-master ~]# kubectl get nodes
NAME   STATUS   ROLES           AGE   VERSION
node   Ready    control-plane   37h   v1.28.2
1
2
3

已经看出k8s 集权目前正常运行。