K8S学习之容器探测 livenessProbe、readinessProbe、startupProbe、lifecycle_container failed liveness probe, will be restarted

容器探测

所谓容器探测就是我们在里面设置了一些探针，或者传感器来获取相应的数据用来判断容器是否启动、存活、就绪的标准；
目前k8s支持的探针有三种：livenessProbe、readinessProbe、startupProbe。（可以混用）
探测类型有三种：exec、httpGet、tcpSocket。我们平时定义的时候只需要定义任意一个即可。如果探测失败，kubelet杀掉该容器，并根据容器的重启策略做相应的处理（readinessProbe不会重启pod）。

命令行查看探针

[root@master ~]# kubectl explain pods.spec.containers
...
livenessProbe        <Object>
     Periodic probe of container liveness. Container will be restarted if the
     probe fails. Cannot be updated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
readinessProbe       <Object>
     Periodic probe of container service readiness. Container will be removed
     from service endpoints if the probe fails. Cannot be updated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
startupProbe <Object>
     StartupProbe indicates that the Pod has successfully initialized. If
     specified, no other probes are executed until this completes successfully.
     If this probe fails, the Pod will be restarted, just as if the
     livenessProbe failed. This can be used to provide different probe
     parameters at the beginning of a Pod's lifecycle, when it might take a long
     time to load data or warm a cache, than during steady-state operation. This
     cannot be updated. This is a beta feature enabled by the StartupProbe
     feature flag. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

查看livenessprobe

[root@master ~]# kubectl explain pods.spec.containers.livenessProbe
KIND:     Pod
VERSION:  v1

RESOURCE: livenessProbe <Object>

DESCRIPTION:
     Periodic probe of container liveness. Container will be restarted if the
     probe fails. Cannot be updated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes

     Probe describes a health check to be performed against a container to
     determine whether it is alive or ready to receive traffic.

FIELDS:
   exec 		<Object>
     One and only one of the following should be specified. Exec specifies the
     action to take.

   failureThreshold     <integer>
     Minimum consecutive failures for the probe to be considered failed after
     having succeeded. Defaults to 3. Minimum value is 1.

   httpGet      <Object>
     HTTPGet specifies the http request to perform.

   initialDelaySeconds  <integer>
     Number of seconds after the container has started before liveness probes
     are initiated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes

   periodSeconds        <integer>
     How often (in seconds) to perform the probe. Default to 10 seconds. Minimum
     value is 1.

   successThreshold     <integer>
     Minimum consecutive successes for the probe to be considered successful
     after having failed. Defaults to 1. Must be 1 for liveness and startup.
     Minimum value is 1.

   tcpSocket    <Object>
     TCPSocket specifies an action involving a TCP port. TCP hooks not yet
     supported

   timeoutSeconds       <integer>
     Number of seconds after which the probe times out. Defaults to 1 second.
     Minimum value is 1. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

参数简介

   failureThreshold	    <integer>       失败判断，默认失败3次
   initialDelaySeconds	<integer>		初始化延迟等待时间
   periodSeconds		<integer>		周期间隔时长，默认10s
   successThreshold     <integer> 		启动几秒后，认为成功，默认1s
   timeoutSeconds		<integer>		探测后的响应时间，默认1s
1
2
3
4
5

ReadinessProbe、startupprobe各项参数解释同上面的livenessProbe

查看livenessProbe下的exec用法

[root@master ~]# kubectl explain pods.spec.containers.livenessProbe.exec
KIND:     Pod
VERSION:  v1

RESOURCE: exec <Object>

DESCRIPTION:
     One and only one of the following should be specified. Exec specifies the
     action to take.

     ExecAction describes a "run in container" action.

FIELDS:
   command      <[]string>
     Command is the command line to execute inside the container, the working
     directory for the command is root ('/') in the container's filesystem. The
     command is simply exec'd, it is not run inside a shell, so traditional
     shell instructions ('|', etc) won't work. To use a shell, you need to
     explicitly call out to that shell. Exit status of 0 is treated as
     live/healthy and non-zero is unhealthy.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

[root@master ~]# vim liveness-exec.yaml

apiVersion: v1
kind: Pod
metadata:
  name: liveness-exec-container
  namespace: default
spec:
  containers:
  - name: liveness-exec-container
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: ["/bin/sh","-c","touch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 3600"]
    livenessProbe:
       exec:
         command: ["test","-e","/tmp/healthy"]    #检测文件是否存在，判断存活
       initialDelaySeconds: 1
       periodSeconds: 3

[root@master ~]# kubectl get pod
NAME                      READY   STATUS    RESTARTS   AGE
liveness-exec-container   1/1     Running   0          7s
[root@master ~]# kubectl describe pod liveness-exec-container
...
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  16s   default-scheduler  Successfully assigned default/liveness-exec-container to node1
  Normal  Pulled     16s   kubelet            Container image "busybox:latest" already present on machine
  Normal  Created    16s   kubelet            Created container liveness-exec-container
  Normal  Started    15s   kubelet            Started container liveness-exec-container
此时可以看到容器启动成功，等待一段时间再次查看，会发现Liveness probe failed，容器重启
[root@master ~]# kubectl describe pod liveness-exec-container
...
  Restart Count:  3
...
Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  40s              default-scheduler  Successfully assigned default/liveness-exec-container to node1
  Normal   Pulled     40s              kubelet            Container image "busybox:latest" already present on machine
  Normal   Created    40s              kubelet            Created container liveness-exec-container
  Normal   Started    39s              kubelet            Started container liveness-exec-container
  Warning  Unhealthy  2s (x3 over 8s)  kubelet            Liveness probe failed:
  Normal   Killing    2s               kubelet            Container liveness-exec-container failed liveness probe, will be restarted	

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

查看livenessProbe下的httpGet的使用

[root@master ~]# kubectl explain pods.spec.containers.livenessProbe.httpGet
KIND:     Pod
VERSION:  v1

RESOURCE: httpGet <Object>

DESCRIPTION:
     HTTPGet specifies the http request to perform.

     HTTPGetAction describes an action based on HTTP Get requests.

FIELDS:
   host <string>
     Host name to connect to, defaults to the pod IP. You probably want to set
     "Host" in httpHeaders instead.

   httpHeaders  <[]Object>
     Custom headers to set in the request. HTTP allows repeated headers.

   path <string>
     Path to access on the HTTP server.

   port <string> -required-
     Name or number of the port to access on the container. Number must be in
     the range 1 to 65535. Name must be an IANA_SVC_NAME.

   scheme       <string>
     Scheme to use for connecting to the host. Defaults to HTTP.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

[root@master ~]# vim liveness-httpget.yaml

apiVersion: v1
kind: Pod
metadata:
  name: liveness-httpget-pod
  namespace: default
spec:
  containers:
  - name: liveness-httpget-container
    image: ikubernetes/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      httpGet:
        port: http
        path: /index.html
      initialDelaySeconds: 1
      periodSeconds: 3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

查看pod的信息，此时pod 能够访问

[root@master ceshi]# kubectl get pod
NAME                      READY   STATUS        RESTARTS   AGE
liveness-httpget-pod      1/1     Running       0          18s
[root@master ceshi]# kubectl get pod -owide
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
liveness-httpget-pod   1/1     Running   0          23s   10.244.1.34   node1   <none>           <none>
[root@master ceshi]# curl 10.244.1.34
Hello MyApp 
[root@master ~]#  kubectl describe pod liveness-httpget-pod
...
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  1m30s  default-scheduler  Successfully assigned default/liveness-httpget-pod to node1
  Normal  Pulled     1m30s  kubelet            Container image "ikubernetes/myapp:v1" already present on machine
  Normal  Created    1m30s  kubelet            Created container liveness-httpget-container
  Normal  Started    1m30s  kubelet            Started container liveness-httpget-container
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

进入容器删除livenessProbe的探测的文件index.html。退出稍等一下，观察会发现pod，依然能访问

[root@master ~]# kubectl exec -it liveness-httpget-pod -- /bin/sh
/ # rm -rf /usr/share/nginx/html/index.html
/ # exit
[root@master ~]#  curl 10.244.1.34
Hello MyApp 
[root@master ~]# kubectl get pod -owide
NAME                   READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
liveness-httpget-pod   1/1     Running   1          6m25s   10.244.1.34   node1   <none>           <none>

[root@master ~]#  kubectl describe pod liveness-httpget-pod
...
    Restart Count:  1
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  5m11s               default-scheduler  Successfully assigned default/liveness-httpget-pod to node1
  Normal   Pulled     6s (x2 over 5m10s)  kubelet            Container image "ikubernetes/myapp:v1" already present on machine
  Normal   Created    6s (x2 over 5m10s)  kubelet            Created container liveness-httpget-container
  Normal   Started    6s (x2 over 5m10s)  kubelet            Started container liveness-httpget-container
  Warning  Unhealthy  6s (x3 over 12s)    kubelet            Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    6s                  kubelet            Container liveness-httpget-container failed liveness probe, will be restarted
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

readnessProbe（容器就绪性探测）使用

[root@master ~]# vim readliness-httpget.yaml

apiVersion: v1
kind: Pod
metadata:
  name: readiness-httpget-pod
  namespace: default
spec:
  containers:
  - name: readiness-httpget-container
    image: ikubernetes/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: http
      containerPort: 80
    readinessProbe:
      httpGet:
        port: http
        path: /index.html
      initialDelaySeconds: 1
      periodSeconds: 3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

此时查看pod

[root@master ~]# kubectl apply -f readliness-httpget.yaml
pod/readiness-httpget-pod created
[root@master ~]# kubectl get pod -owide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
readiness-httpget-pod   1/1     Running   0          26s   10.244.1.35   node1   <none>           <none>
[root@master ~]# curl 10.244.1.35
Hello MyApp
1
2
3
4
5
6
7

容器就绪性状态检测index.html是否存在，如果把index.html删除，那么容器处于不正常运行，**但不会重启。**此时
kubectl get pod的READY状态是0/1
kubectl descript pod 显示404报错访问的网页是默认的，不是Hello MyApp。
kubectl logs readiness-httpget-pod 显示没有这个文件

[root@master ~]# kubectl exec -it readiness-httpget-pod -- /bin/sh
/ # rm -rf /usr/share/nginx/html/index.html
/ # exit
[root@master ~]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
readiness-httpget-pod   0/1     Running   0          5m3s
[root@master ~]# kubectl describe pod readiness-httpget-pod
...
Events:
...
  Warning  Unhealthy  3m1s (x53 over 5m37s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 404
[root@master ~]# kubectl logs readiness-httpget-pod
   open() "/usr/share/nginx/html/index.html" failed (2: No such file or directory), 
1
2
3
4
5
6
7
8
9
10
11
12
13

进入容器再次创建index.html时，会发现容器就绪，可以正常访问

[root@master ~]# kubectl exec -it readiness-httpget-pod -- /bin/sh
/ # echo "Hello MyApp" >> /usr/share/nginx/html/index.html
/ # exit
[root@master ~]# kubectl get pod 
NAME                    READY   STATUS    RESTARTS   AGE
readiness-httpget-pod   1/1     Running   0          10m
[root@master ~]# curl 10.244.1.35
Hello MyApp
1
2
3
4
5
6
7
8

3.lifecycle生命周期讲解
[root@master ~]# kubectl explain pods.spec.containers.lifecycle
PostStart ：容器创建成功后，运行前的任务，用于资源部署、环境准备等。如果操作失败，容器根据重启策略决定是否重启，
PreStop ：在容器被终止前的任务，用于优雅关闭应用程序、通知其他系统等等。
[root@master ~]# kubectl explain pods.spec.containers.lifecycle.postStart
同样也是exec、httpGet、tcpSocket三种方式

postStart例子 初始化操作使用的比较多，实际生产使用频次较少

这里前钩子修改了index.html文件，访问时是新的界面home
[root@master ~]# vim poststart-nginx.yaml

apiVersion: v1
kind: Pod
metadata:
  name: poststart-nginx
  namespace: default
spec:
  containers:
  - name: poststart-nginx
    image: nginx
    imagePullPolicy: IfNotPresent
    lifecycle:
       postStart:
         exec:
           command: ["/bin/sh","-c","echo home >> /usr/share/nginx/html/index.html"]
    ports:
    - name: http
      containerPort: 80
[root@master ~]# kubectl get pod -owide
NAME              READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
poststart-nginx   1/1     Running   0          5s    10.244.1.36   node1   <none>           <none>
[root@master ~]# curl 10.244.1.36
home
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

prestop举例：

PreStop在整个生命周期中比较有用，实用场景也比较多。 比如：
1.关闭前等待某一个状态完成；
2.关闭前同步一些状态（数据）到其他的地方；
3.关闭前通知某一个系统或者更新一个状态；

[root@master ~]# vim preStop-nginx.yaml 这里实现了nginx的优雅退出

apiVersion: v1
kind: Pod
metadata:
  name: prestop-nginx
  namespace: default
spec:
  containers:
  - name: prestop-nginx
    image: nginx
    imagePullPolicy: IfNotPresent
    lifecycle:
       preStop:
         exec:
           command: ["/usr/local/nginx/sbin/nginx","-s","quit"]
    ports:
    - name: http
      containerPort: 80
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

混合配置，这里有个参考的，可以看看
https://www.cnblogs.com/superlinux/p/14933961.html