小丑西瓜9

这个屌丝很懒，什么也没留下！

热门标签

helm安装mysql

作者：小丑西瓜9 | 2024-05-02 10:44:42

踩

helm 安装mysql

导航：


一、前提
1.版本信息-2.查看方法-3.查看k8s集群节点状态-4.配置docker仓库下载加速
二、安装部署helm
1.下载软件包-2.安装helm-3.初始化
三、使用helm安装mysql
1.搜索mysql的helm包-2.准备后端存储-3.准备docker image-4.配置集群内的kube-dns-5.安装mysql的helm包-6.查看mysql的有状态应用、pod、svc、pvc等k8s资源-7.客户端验证
四、故障排除
五、参考资料

一、前提

使用helm安装有状态应用mysql主备集群，首先确保你已经搭建好一个3节点的k8s集群(如何搭建k8s集群、k8s集群网络配置flannel请参考百度)。

1.版本信息


k8s: v1.5.2
docker: 1.13.1
flannel:0.7.1
helm: v3.0.0

2.查看方法


kubectl version
docker version
flanneld --version
helm version

3.查看k8s集群节点状态


[root@master ~]# kubectl get node
NAME      STATUS    AGE
master    Ready     4d
node2     Ready     18h
node3     Ready     4m
[root@master ~]#

4.配置docker仓库下载加速

使用163的国内docker仓库，提高docker image的下载速度。


[root@master ~]# cat /etc/docker/daemon.json 
{"registry-mirrors": ["http://hub-mirror.c.163.com"]}
[root@master ~]# systemctl restart docker   #重启docker服务生效
[root@master ~]#

二、安装部署helm

使用helm的v3版本，v3版本不用配置服务端组件 Tiller。

1.下载软件包

下载helm的二进制包。
helm安装mysql
下载地址：
https://space.dingtalk.com/s/gwHOAF91cwLOABfU0gPaACBkNjc5YmMwMDUzZmQ0YWQwYjg2MmNhZWRiMGU3YmU1ZA 密码: 5kEK

2.安装helm

使用xshell或者winscp将安装包传输到linux中。


[root@master ~]# tar -tvf helm-v3.0.0-alpha.1-linux-amd64.tar.gz     #查看压缩包中的内容
drwxr-xr-x circleci/circleci 0 2019-05-16 05:44 linux-amd64/
-rwxr-xr-x circleci/circleci 39771936 2019-05-16 05:42 linux-amd64/helm
-rw-r--r-- circleci/circleci    11373 2019-05-16 05:44 linux-amd64/LICENSE
-rw-r--r-- circleci/circleci     3159 2019-05-16 05:44 linux-amd64/README.md
[root@master ~]#

将二进制文件复制到$PATH路径下即可使用。


[root@master ~]# which helm      #查看helm二进制文件所在的路径，将helm复制到$PATH中即可直接使用，例如将helm放在/usr/bin/helm的路径
/usr/bin/helm
[root@master ~]# helm --help
The Kubernetes package manager
 
To begin working with Helm, run the 'helm init' command:
 
    $ helm init
 
This will set up any necessary local configuration.
 
Common actions from this point include:
 
- helm search:    search for charts
- helm fetch:     download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts
 
Environment:
  $HELM_HOME          set an alternative location for Helm files. By default, these are stored in ~/.helm
  $HELM_DRIVER        set the backend storage driver. Values are: configmap, secret, memory
  $HELM_NO_PLUGINS    disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.
  $KUBECONFIG         set an alternative Kubernetes configuration file (default "~/.kube/config")
 
Usage:
  helm [command]
 
Available Commands:
  chart       push, pull, tag, or remove Helm charts
  completion  Generate autocompletions script for the specified shell (bash or zsh)
  create      create a new chart with the given name
  dependency  manage a chart's dependencies
  get         download a named release
  help        Help about any command
  history     fetch release history
  home        displays the location of HELM_HOME
  init        initialize Helm client
  install     install a chart
  lint        examines a chart for possible issues
  list        list releases
  package     package a chart directory into a chart archive
  plugin      add, list, or remove Helm plugins
  pull        download a chart from a repository and (optionally) unpack it in local directory
  registry    login to or logout from a registry
  repo        add, list, remove, update, and index chart repositories
  rollback    roll back a release to a previous revision
  search      search for a keyword in charts
  show        inspect a chart
  status      displays the status of the named release
  template    locally render templates
  test        test a release or cleanup test artifacts
  uninstall   uninstall a release
  upgrade     upgrade a release
  verify      verify that a chart at the given path has been signed and is valid
  version     print the client version information
Flags:
      --debug                 enable verbose output
  -h, --help                  help for helm
      --home string           location of your Helm config. Overrides $HELM_HOME (default "/root/.helm")
      --kube-context string   name of the kubeconfig context to use
      --kubeconfig string     path to the kubeconfig file
  -n, --namespace string      namespace scope for this request
Use "helm [command] --help" for more information about a command.
[root@master ~]#

3.初始化

使用helm前，需要初始化。


[root@docker repository]# helm init --stable-repo-url=https://apphub.aliyuncs.com     #配置helm使用的仓库为国内阿里云地址，国外的访问不了，而且比较慢
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://apphub.aliyuncs.com 
$HELM_HOME has been configured at /root/.helm.
Happy Helming!
[root@docker repository]# ls
cache  local  repositories.yaml
[root@docker repository]# cat repositories.yaml     #helm初始化完后会有一个配置文件生成，下面是repositories.yaml的内容
apiVersion: v1
generated: "2019-08-25T07:08:38.917953914+08:00"
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/stable-index.yaml
  certFile: ""
  keyFile: ""
  name: stable
  password: ""
  url: https://apphub.aliyuncs.com
  username: ""

三、使用helm安装mysql

使用helm安装有状态应用mysql，使用到的docker镜像有：


docker.io/mysql:latest
docker.io/ist0ne/xtrabackup:latest

构建mysql的1主2从，主备集群，使用xtrabackup工具进行数据同步。

1.搜索mysql的helm包


[root@docker ~]# helm search mysql     #搜索mysql的helm包
NAME                                CHART VERSION   APP VERSION DESCRIPTION                                       
apphub/mysql                        6.3.1           8.0.17      Chart to create a Highly available MySQL cluster  
apphub/mysqldump                    2.6.0           2.4.1       A Helm chart to help backup MySQL databases usi...
apphub/mysqlha                      1.0.0           5.7.13      MySQL cluster with a single master and zero or ...
apphub/prometheus-mysql-exporter    0.5.1           v0.11.0     A Helm chart for prometheus mysql exporter with...
stable/mysql                        6.3.1           8.0.17      Chart to create a Highly available MySQL cluster  
stable/mysqldump                    2.6.0           2.4.1       A Helm chart to help backup MySQL databases usi...
stable/mysqlha                      1.0.0           5.7.13      MySQL cluster with a single master and zero or ...
stable/prometheus-mysql-exporter    0.5.1           v0.11.0     A Helm chart for prometheus mysql exporter with...
apphub/percona                      1.1.0           5.7.17      free, fully compatible, enhanced, open source d...
apphub/percona-xtradb-cluster       1.0.1           5.7.19      free, fully compatible, enhanced, open source d...
apphub/phpmyadmin                   3.0.1           4.9.0-1     phpMyAdmin is an mysql administration frontend    
stable/percona                      1.1.0           5.7.17      free, fully compatible, enhanced, open source d...
stable/percona-xtradb-cluster       1.0.1           5.7.19      free, fully compatible, enhanced, open source d...
stable/phpmyadmin                   3.0.1           4.9.0-1     phpMyAdmin is an mysql administration frontend    
apphub/mariadb                      6.8.1           10.3.17     Fast, reliable, scalable, and easy to use open-...
apphub/mariadb-galera               0.2.1           10.3.17     MariaDB Galera is a multi-master database clust...
stable/mariadb                      6.8.1           10.3.17     Fast, reliable, scalable, and easy to use open-...
stable/mariadb-galera               0.2.1           10.3.17     MariaDB Galera is a multi-master database clust...

2.准备后端存储

由于mysql的helm包需要使用pv，而开源的k8s集群默认没有配置pv后端存储，因此需要手动添加nfs服务器(如何搭建nfs服务器请参考百度)为后端存储。
nfs配置：


[root@master ~]# cat /etc/exports
/nfsdata/pv001  *(rw,no_root_squash)   #添加no_root_squash不压缩root权限，因为容器里会使用root权限写入数据，否则后面会有权限不足的报错
/nfsdata/pv002  *(rw,no_root_squash)
/nfsdata/pv003  *(rw,no_root_squash)
[root@master ~]# ls -ld /nfsdata/      #配置目录的权限为nfsnobody
drwxr-xr-x 5 nfsnobody nfsnobody 45 Aug 26 01:18 /nfsdata/
[root@master ~]# ls -ll /nfsdata/
total 0
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv001
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv002
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv003
[root@master ~]#

创建pv的yaml文件：


[root@master hrr]# cat pv.yml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv003
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce   #accessModes需要和pvc的模式一样
  nfs:
    path: /nfsdata/pv003
    server: 192.168.222.129
[root@master hrr]#

创建pv:


[root@docker hrr]# kubectl create -f pv.yml 
persistentvolume "pv003" created
[root@docker hrr]#

查看pv：


[root@master hrr]# kubectl get pv    #Available表示没有被PVC使用
NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM     REASON    AGE
pv001     10Gi       RWO           Retain          Available                       26s
pv002     10Gi       RWO           Retain          Available                       14s
pv003     10Gi       RWO           Retain          Available                       3s
[root@master hrr]#
[root@master hrr]# kubectl get pv     #Bound表示被PVC使用
NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                          REASON    AGE
pv001     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-0             32s
pv002     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-1             17s
pv003     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-2             4s
[root@master hrr]#

3.准备docker image

在安装mysql的helm包时需要mysql与xtrabackup的docker镜像。


[root@docker docker]# docker pull docker.io/ist0ne/xtrabackup   #下载xtrabackup镜像
Using default tag: latest
Trying to pull repository docker.io/ist0ne/xtrabackup ... 
latest: Pulling from docker.io/ist0ne/xtrabackup
386a066cd84a: Pull complete 
40f175e652e1: Pull complete 
5eb6b5905d55: Pull complete 
Digest: sha256:e5e5519b874363f1bf35541cef2caed5803645937e29834817be51b30aabc479
Status: Downloaded newer image for docker.io/ist0ne/xtrabackup:latest
[root@docker docker]# 
[root@docker docker]# docker pull docker.io/mysql   #下载mysql镜像

查看本地的镜像：


[root@master hrr]# docker images   
REPOSITORY                                                  TAG                 IMAGE ID            CREATED             SIZE
192.168.222.129:5000/nginx                                  latest              5a3221f0137b        9 days ago          126 MB
docker.io/nginx                                             latest              5a3221f0137b        9 days ago          126 MB
docker.io/mysql                                             latest              62a9f311b99c        11 days ago         445 MB
mysql                                                       5.7.13              62a9f311b99c        11 days ago         445 MB
docker.io/ist0ne/xtrabackup                                 latest              c415dbd7af07        2 years ago         265 MB
registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup   1.0                 c415dbd7af07        2 years ago         265 MB
192.168.222.129:5000/pod-infrastructure                     latest              f9d5de079539        5 years ago         240 kB
docker.io/kubernetes/pause                                  latest              f9d5de079539        5 years ago         240 kB
[root@master hrr]#

将镜像打tag，用于后面创建mysql的helm包：


[root@docker hrr]# docker tag docker.io/mysql:latest mysql:5.7.13
[root@docker docker]# docker tag docker.io/ist0ne/xtrabackup:latest registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0

将docker镜像导出，用于导入到其它k8s节点中：


[root@master ~]# docker save mysql:5.7.13 -o mysql5713.tgz  #将docker镜像导出，名为mysql5713.tgz
[root@master ~]# docker save registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 -o xtrabackup10.tgz 
[root@master ~]# scp mysql5713.tgz xtrabackup10.tgz root@192.168.222.130:
root@192.168.222.130's password: 
mysql5713.tgz                                                                                      100%  429MB  35.7MB/s   00:12    
xtrabackup10.tgz                                                                                   100%  260MB  43.4MB/s   00:06    
[root@master ~]#

在其它k8s节点中导入docker镜像：


[root@node2 ~]# docker load -i  mysql5713.tgz   #将导出的docker镜像mysql5713.tgz导入
8fa655db5360: Loading layer [==================================================>] 58.48 MB/58.48 MB
f5741d086b76: Loading layer [==================================================>] 338.4 kB/338.4 kB
9f77b78f01a7: Loading layer [==================================================>] 10.44 MB/10.44 MB
7f33ce1066af: Loading layer [==================================================>] 4.472 MB/4.472 MB
1dd5f3e365cf: Loading layer [==================================================>] 1.536 kB/1.536 kB
65430c57aee2: Loading layer [==================================================>] 46.15 MB/46.15 MB
bb1ef34119b2: Loading layer [==================================================>] 32.77 kB/32.77 kB
995fa4bb8afe: Loading layer [==================================================>] 3.584 kB/3.584 kB
bdebd19e878f: Loading layer [==================================================>] 329.8 MB/329.8 MB
79a46f1eb9d0: Loading layer [==================================================>] 5.632 kB/5.632 kB
5b96d5043191: Loading layer [==================================================>] 9.728 kB/9.728 kB
647229410363: Loading layer [==================================================>] 1.536 kB/1.536 kB
Loaded image: mysql:5.7.13
[root@node2 ~]# docker load -i  xtrabackup10.tgz 
fe4c16cbf7a4: Loading layer [==================================================>] 128.9 MB/128.9 MB
82d548d175dd: Loading layer [==================================================>] 9.216 kB/9.216 kB
f85c58969eb0: Loading layer [==================================================>]   144 MB/144 MB
Loaded image: registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0
[root@node2 ~]#

node3做同样的操作。

4.配置集群内的kube-dns

5.安装mysql的helm包


[root@master ~]#  helm install mysql apphub/mysqlha
NAME: mysql
LAST DEPLOYED: 2019-08-25 09:43:33.16471802 +0800 CST m=+0.898633546
NAMESPACE: default
STATUS: deployed
 
NOTES:
The MySQL cluster is comprised of 3 MySQL pods: 1 master and 2 slaves. Each instance is accessible within the cluster through:
 
    <pod-name>.mysql-mysqlha
 
`mysql-mysqlha-0.mysql-mysqlha` is designated as the master and where all writes should be executed against. Read queries can be executed against the `mysql-mysqlha-readonly` service which distributes connections across all MySQL pods.
 
To connect to your database:
 
1. Obtain the root password: 
 
    kubectl get secret --namespace default mysql-mysqlha -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo
 
2. Run a pod to use as a client:
 
    kubectl run mysql-client --image=mysql:5.7.13 -it --rm --restart='Never' --namespace default -- /bin/sh
 
3. To connect to Master service (read/write):
 
    mysql -h mysql-mysqlha-0.mysql-mysqlha -u root -p
 
4. To connect to slave service (read-only):
 
   mysql -h mysql-mysqlha-readonly -u root -p
[root@master ~]#

安装完后，在本地会有mysql的helm包：


[root@master archive]# pwd
/root/.helm/cache/archive
[root@master archive]# ls
guestbook-0.2.0.tgz  mysqlha-1.0.0.tgz
[root@master archive]# 
[root@master archive]# tar -tvf mysqlha-1.0.0.tgz 
-rwxr-xr-x 0/0             378 2019-08-22 13:20 mysqlha/Chart.yaml
-rwxr-xr-x 0/0            2561 2019-08-22 13:20 mysqlha/values.yaml
-rwxr-xr-x 0/0            1216 2019-08-22 13:20 mysqlha/templates/NOTES.txt
-rwxr-xr-x 0/0             961 2019-08-22 13:20 mysqlha/templates/_helpers.tpl
-rwxr-xr-x 0/0             756 2019-08-22 13:20 mysqlha/templates/configmap.yaml
-rwxr-xr-x 0/0             967 2019-08-22 13:20 mysqlha/templates/secret.yaml
-rwxr-xr-x 0/0           10905 2019-08-22 13:20 mysqlha/templates/statefulset.yaml
-rwxr-xr-x 0/0            1203 2019-08-22 13:20 mysqlha/templates/svc.yaml
-rwxr-xr-x 0/0              56 2019-08-22 13:20 mysqlha/OWNERS
-rwxr-xr-x 0/0            6724 2019-08-22 13:20 mysqlha/README.md
[root@master archive]#

将mysqlha-1.0.0.tgz解压，查看文件内容：


[root@master mysqlha]# ls
Chart.yaml  OWNERS  README.md  templates  values.yaml
[root@master mysqlha]# cat README.md
# MySQL - Single Master, Multiple Slaves
 
[MySQL](https://MySQL.org) is one of the most popular database servers in the world. Notable users include Wikipedia, Facebook and Google.
 
## Introduction
 
This chart bootstraps a single master and multiple slave MySQL deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager. Largely inspired by this [tutorial](https://kubernetes.io/docs/tutorials/stateful-application/run-replicated-stateful-application/), further work was made to 'production-ize' the example.
 
## Prerequisites
 
- Kubernetes 1.6+
- PV provisioner support in the underlying infrastructure
 
## Installing the Chart
 
To install the chart with the release name `my-release`:
 
$ helm install --name my-release <helm-repo>/mysqlha
 
The command deploys MySQL cluster on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
 
### Uninstall
 
To uninstall/delete the `my-release` deployment:
 
$ helm delete my-release
 
## Configuration
 
The following table lists the configurable parameters of the MySQL chart and their default values.
 
| Parameter                                    | Description                                       | Default                                |
| -----------------------------------------    | ------------------------------------------------- | -------------------------------------- |
| `mysqlImage`                                 | `mysql` image and tag.                            | `mysql:5.7.13`                         |
| `xtraBackupImage`                            | `xtrabackup` image and tag.                       | `gcr.io/google-samples/xtrabackup:1.0` |
| `imagePullPolicy`                            | Image pull policy.                                | `IfNotPresent`                         |
| `nameOverride`                               | `String to partially override mysqlha.fullname template with a string (will prepend the release name)` | `nil` |
| `fullnameOverride`                           | `String to fully override mysqlha.fullname template with a string`                 | `nil` |
| `replicaCount`                               | Number of MySQL replicas                          | 3                                      |
| `mysqlRootPassword`                          | Password for the `root` user.                     | Randomly generated                     |
| `mysqlUser`                                  | Username of new user to create.                   | `nil`                                  |
| `mysqlPassword`                              | Password for the new user.                        | Randomly generated                     |
| `mysqlReplicationUser`                       | Username for replication user                     | `repl`                                 |
| `mysqlReplicationPassword`                   | Password for replication user.                    | Randomly generated                     |
| `mysqlDatabase`                              | Name of the new Database to create                | `nil`                                  |
| `configFiles.master.cnf`                     | Master configuration file                         | See `values.yaml`                      |
| `configFiles.slave.cnf`                      | Slave configuration file                          | See `values.yaml`                      |
| `podAnnotations`                             | Pod annotations                                   | `nil`                                  |
| `schedulerName`                              | Name of the k8s scheduler (other than default)    | `nil`                                  |
| `persistence.enabled`                        | Create a volume to store data                     | true                                   |
| `persistence.size`                           | Size of persistent volume claim                   | 10Gi                                   |
| `persistence.storageClass`                   | Type of persistent volume claim                   | `nil`                                  |
| `persistence.accessModes`                    | Persistent volume access modes                    | `[ReadWriteOnce]`                      |
| `persistence.annotations`                    | Persistent volume annotations                     | `{}`                                   |
| `resources`                                  | CPU/Memory resource requests/limits               | Memory: `128Mi`, CPU: `100m`           |
| `metrics.enabled`                            | Start a side-car prometheus exporter              | false                                  |
| `metrics.image`                              | Exporter image                                    | `prom/mysqld-exporter`                 |
| `metrics.imageTag`                           | Exporter image                                    | `v0.10.0`                              |
| `metrics.imagePullPolicy`                    | Exporter image pull policy                        | `IfNotPresent`                         |
| `metrics.resources`                          | Exporter resource requests/limit                  | See `values.yaml`                      |
| `metrics.livenessProbe.initialDelaySeconds`  | Delay before metrics liveness probe is initiated  | 15                                     |
| `metrics.livenessProbe.timeoutSeconds`       | When the probe times out                          | 5                                      |
| `metrics.readinessProbe.initialDelaySeconds` | Delay before metrics readiness probe is initiated | 5                                      |
| `metrics.readinessProbe.timeoutSeconds`      | When the probe times out                          | 1                                      |
 
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
 
## Persistence
 
The [MySQL](https://hub.docker.com/_/mysql/) image stores the MySQL data and configurations at the `/var/lib/mysql` path of the container.
 
By default persistence is enabled, and a PersistentVolumeClaim is created and mounted in that directory. As a result, a persistent volume will need to be defined:
 
# https://kubernetes.io/docs/user-guide/persistent-volumes/#azure-disk
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: kubernetes.io/azure-disk
parameters:
  skuName: Premium_LRS
  location: westus
 
In order to disable this functionality you can change the values.yaml to disable persistence and use an emptyDir instead.

查看values.yaml文件：


[root@master mysqlha]# cat values.yaml 
## mysql image version
## ref: https://hub.docker.com/r/library/mysql/tags/
##
mysqlImage: mysql:5.7.13     #前面下载的mysql镜像
xtraBackupImage: registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0      #前面下载的xtrabackup镜像
 
## Specify an imagePullPolicy (Required)
## It's recommended to change this to 'Always' if the image tag is 'latest'
## ref: http://kubernetes.io/docs/user-guide/images/#updating-images
##
imagePullPolicy: IfNotPresent
 
## String to partially override orangehrm.fullname template (will maintain the release name)
##
# nameOverride: ""
## String to fully override orangehrm.fullname template
##
# fullnameOverride: ""
 
mysqlha:
  replicaCount: 3
 
  ## Password for MySQL root user
  ##
  # mysqlRootPassword: ## Default: random 10 character string
 
  ## Username/password for MySQL replication user
  ##
  mysqlReplicationUser: repl
  # mysqlReplicationPassword:
 
  ## Create a database user
  ##
  # mysqlUser:
  # mysqlPassword: ## Default: random 10 character string
 
  ## Allow unauthenticated access, uncomment to enable
  ##
  # mysqlAllowEmptyPassword: true
 
  ## Create database with name and grant all permissions to user on startup, if needed
  # mysqlDatabase:
 
  ## Configuration files for the master and slaves
  ##
  configFiles:
    master.cnf: |
      # Apply this config only on the master.
      [mysqld]
      log-bin
      skip_name_resolve
    slave.cnf: |
      # Apply this config only on slaves.
      [mysqld]
      super-read-only
      skip_name_resolve
 
  podAnnotations: {}
 
## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:
 
## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: true
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, azure-disk on
  ##   Azure, standard on GKE, AWS & OpenStack)
  ##
  # storageClass: "-"
  accessModes:
  - ReadWriteOnce
  size: 10Gi
  annotations: {}
 
resources:
  requests:
    cpu: 100m
    memory: 128Mi
 
metrics:
  enabled: false
  image: prom/mysqld-exporter
  imageTag: v0.10.0
  annotations: {}
 
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  readinessProbe:
    initialDelaySeconds: 5
    timeoutSeconds: 1
  resources:
    requests:
      cpu: 100m
      memory: 100Mi
[root@master mysqlha]#

6.查看mysql的有状态应用、pod、svc、pvc等k8s资源


[root@master hrr]# kubectl get statefulset 
NAME            DESIRED   CURRENT   AGE
mysql-mysqlha   3         1         18s
[root@master hrr]# kubectl get svc
NAME                     CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes               10.254.0.1      <none>        443/TCP    4d
mysql-mysqlha            None            <none>        3306/TCP   27s
mysql-mysqlha-readonly   10.254.54.188   <none>        3306/TCP   27s
[root@master hrr]# kubectl get pod
NAME              READY     STATUS    RESTARTS   AGE
mysql-mysqlha-0   2/2       Running   0          45s
mysql-mysqlha-1   2/2       Running   0          24s
mysql-mysqlha-2   1/2       Running   0          4s
[root@master hrr]# kubectl get pv
NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                          REASON    AGE
pv001     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-0             2m
pv002     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-2             2m
pv003     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-1             1m
[root@master hrr]# kubectl get pvc
NAME                   STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
data-mysql-mysqlha-0   Bound     pv001     10Gi       RWO           51s
data-mysql-mysqlha-1   Bound     pv003     10Gi       RWO           51s
data-mysql-mysqlha-2   Bound     pv002     10Gi       RWO           51s
[root@master hrr]# kubectl get configmap 
NAME            DATA      AGE
mysql-mysqlha   4         54s
[root@master hrr]# kubectl get secret
NAME            TYPE              DATA      AGE
mysql-mysqlha   Opaque            2         51m
mysql.v1        helm.sh/release   1         51m
[root@master hrr]#
[root@master hrr]# kubectl get pod -owide
NAME              READY     STATUS    RESTARTS   AGE       IP            NODE
mysql-mysqlha-0   2/2       Running   0          1m        172.17.93.2   node3
mysql-mysqlha-1   2/2       Running   0          41s       172.17.78.2   node2
mysql-mysqlha-2   2/2       Running   0          21s       172.17.86.2   master

查看第一个节点的docker容器


[root@master hrr]# docker ps
CONTAINER ID        IMAGE                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
41ec983706b8        registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0   "/bin/bash -c 'set..."   25 seconds ago      Up 24 seconds                           k8s_xtrabackup.516a4a61_mysql-mysqlha-2_default_387e4cdb-c772-11e9-83c0-000c296be0a4_72aba107
691066dcf870        mysql:5.7.13                                                    "docker-entrypoint..."   25 seconds ago      Up 24 seconds                           k8s_mysql.b9eef278_mysql-mysqlha-2_default_387e4cdb-c772-11e9-83c0-000c296be0a4_894896de
d83f031101ef        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 26 seconds ago      Up 25 seconds                           k8s_POD.62b22cd9_mysql-mysqlha-2_default_387e4cdb-c772-11e9-83c0-000c296be0a4_c46314b7

查看第二个节点的docker容器


[root@node2 ~]# docker ps
CONTAINER ID        IMAGE                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
211a5deed23f        registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0   "/bin/bash -c 'set..."   52 seconds ago      Up 51 seconds                           k8s_xtrabackup.516a4a61_mysql-mysqlha-1_default_2c7419c8-c772-11e9-83c0-000c296be0a4_982995a3
51d9e7cd3d1d        mysql:5.7.13                                                    "docker-entrypoint..."   52 seconds ago      Up 51 seconds                           k8s_mysql.b9eef278_mysql-mysqlha-1_default_2c7419c8-c772-11e9-83c0-000c296be0a4_3acd8bf2
71bb21654249        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 52 seconds ago      Up 51 seconds                           k8s_POD.62b22cd9_mysql-mysqlha-1_default_2c7419c8-c772-11e9-83c0-000c296be0a4_cce640c3
[root@node2 ~]#

查看第三个节点的docker容器


[root@node3 ~]# docker ps
CONTAINER ID        IMAGE                                                           COMMAND                  CREATED              STATUS              PORTS               NAMES
d159b422bcdf        registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0   "/bin/bash -c 'set..."   About a minute ago   Up About a minute                       k8s_xtrabackup.516a4a61_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_63f2a879
6aae997c995d        mysql:5.7.13                                                    "docker-entrypoint..."   About a minute ago   Up About a minute                       k8s_mysql.b9eef278_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_7dee904b
40695f86aaf7        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 About a minute ago   Up About a minute                       k8s_POD.62b22cd9_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_58c67c1e
[root@node3 ~]#

查看pvc里的数据


[root@master hrr]# ll /nfsdata/pv001
total 4
drwxr-xr-x 6 polkitd input 4096 Aug 26 03:54 mysql
[root@master hrr]# ll /nfsdata/pv001/mysql/
total 177164
-rw-r----- 1 polkitd input       56 Aug 26 03:54 auto.cnf
-rw-r----- 1 polkitd input  3091158 Aug 26 03:54 binlog.000001
-rw-r----- 1 polkitd input      155 Aug 26 03:54 binlog.000002
-rw-r----- 1 polkitd input       32 Aug 26 03:54 binlog.index
-rw------- 1 polkitd input     1676 Aug 26 03:54 ca-key.pem
-rw-r--r-- 1 polkitd input     1112 Aug 26 03:54 ca.pem
-rw-r--r-- 1 polkitd input     1112 Aug 26 03:54 client-cert.pem
-rw------- 1 polkitd input     1680 Aug 26 03:54 client-key.pem
-rw-r----- 1 polkitd input     5419 Aug 26 03:54 ib_buffer_pool
-rw-r----- 1 polkitd input 12582912 Aug 26 03:54 ibdata1
-rw-r----- 1 polkitd input 50331648 Aug 26 03:54 ib_logfile0
-rw-r----- 1 polkitd input 50331648 Aug 26 03:54 ib_logfile1
-rw-r----- 1 polkitd input 12582912 Aug 26 03:55 ibtmp1
drwxr-x--- 2 polkitd input      187 Aug 26 03:54 #innodb_temp
drwxr-x--- 2 polkitd input      143 Aug 26 03:54 mysql
-rw-r----- 1 polkitd input 29360128 Aug 26 03:54 mysql.ibd
drwxr-x--- 2 polkitd input     8192 Aug 26 03:54 performance_schema
-rw------- 1 polkitd input     1680 Aug 26 03:54 private_key.pem
-rw-r--r-- 1 polkitd input      452 Aug 26 03:54 public_key.pem
-rw-r--r-- 1 polkitd input     1112 Aug 26 03:54 server-cert.pem
-rw------- 1 polkitd input     1680 Aug 26 03:54 server-key.pem
drwxr-x--- 2 polkitd input       28 Aug 26 03:54 sys
-rw-r----- 1 polkitd input 12582912 Aug 26 03:54 undo_001
-rw-r----- 1 polkitd input 10485760 Aug 26 03:54 undo_002

查看svc


[root@master hrr]# kubectl describe svc mysql-mysqlha
Name:           mysql-mysqlha
Namespace:      default
Labels:         app=mysql-mysqlha
            chart=mysqlha-1.0.0
            heritage=Helm
            release=mysql
Selector:       app=mysql-mysqlha
Type:           ClusterIP
IP:         None
Port:           mysql-mysqlha   3306/TCP
Endpoints:      172.17.78.2:3306,172.17.86.2:3306,172.17.93.2:3306
Session Affinity:   None
No events.
[root@master hrr]# kubectl describe svc mysql-mysqlha-readonly 
Name:           mysql-mysqlha-readonly
Namespace:      default
Labels:         app=mysql-mysqlha
            chart=mysqlha-1.0.0
            heritage=Helm
            release=mysql
Selector:       app=mysql-mysqlha
Type:           ClusterIP
IP:         10.254.54.188
Port:           mysql-mysqlha   3306/TCP
Endpoints:      172.17.78.2:3306,172.17.86.2:3306,172.17.93.2:3306
Session Affinity:   None
No events.
[root@master hrr]#

查看有状态应用stateful


[root@master hrr]# kubectl get statefulset -oyaml
apiVersion: v1
items:
- apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    creationTimestamp: 2019-08-25T19:54:18Z
    generation: 1
    labels:
      app: mysql-mysqlha
      chart: mysqlha-1.0.0
      heritage: Helm
      release: mysql
    name: mysql-mysqlha
    namespace: default
    resourceVersion: "94760"
    selfLink: /apis/apps/v1beta1/namespaces/default/statefulsets/mysql-mysqlha
    uid: 1fa6f230-c772-11e9-83c0-000c296be0a4
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: mysql-mysqlha
    serviceName: mysql-mysqlha
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: mysql-mysqlha
      spec:
        containers:
        - env:
          - name: MYSQL_DATABASE
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                key: mysql-root-password
                name: mysql-mysqlha
          - name: MYSQL_REPLICATION_USER
            value: repl
          - name: MYSQL_REPLICATION_PASSWORD
            valueFrom:
              secretKeyRef:
                key: mysql-replication-password
                name: mysql-mysqlha
          image: mysql:5.7.13
          imagePullPolicy: IfNotPresent
          livenessProbe:
            exec:
              command:
              - /bin/sh
              - -c
              - mysqladmin ping -h 127.0.0.1 -u root -p${MYSQL_ROOT_PASSWORD}
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          name: mysql
          ports:
          - containerPort: 3306
            name: mysql
            protocol: TCP
          readinessProbe:
            exec:
              command:
              - /bin/sh
              - -c
              - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
              - mysql -h 127.0.0.1 -u root -e "SELECT 1"
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
          terminationMessagePath: /dev/termination-log
          volumeMounts:
          - mountPath: /var/lib/mysql
            name: data
            subPath: mysql
          - mountPath: /etc/mysql/conf.d
            name: conf
        - args:
          - -c
          - |
            set -ex
 
            echo "Waiting for mysqld to be ready (accepting connections)"
            until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 5; done
 
            # Create replication user
            cd /mnt/scripts
            # file exists and is not empty with -s
            if [[ -s create-replication-user.sh  ]]; then
              ls -la
              ./create-replication-user.sh
            fi
 
            cd /var/lib/mysql
            # Determine binlog position of cloned data, if any.
            if [[ -f xtrabackup_slave_info ]]; then
              # XtraBackup already generated a partial "CHANGE MASTER TO" query
              # because we're cloning from an existing slave.
              cp xtrabackup_slave_info change_master_to.sql.in
            elif [[ -f xtrabackup_binlog_info ]]; then
              # We're cloning directly from master. Parse binlog position.
              [[ $(cat xtrabackup_binlog_info) =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
              echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
                    MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
            fi
 
            # Check if we need to complete a clone by starting replication.
            if [[ -f change_master_to.sql.in ]]; then
 
              # In case of container restart, attempt this at-most-once.
              cp change_master_to.sql.in change_master_to.sql.orig
              mysql -h 127.0.0.1 --verbose<<EOF
              STOP SLAVE IO_THREAD;
              $(<change_master_to.sql.orig),
              MASTER_HOST='mysql-mysqlha-0.mysql-mysqlha',
              MASTER_USER='${MYSQL_REPLICATION_USER}',
              MASTER_PASSWORD='${MYSQL_REPLICATION_PASSWORD}',
              MASTER_CONNECT_RETRY=10;
              START SLAVE;
            EOF
            fi
 
            # Start a server to send backups when requested by peers.
            exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
              "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=${MYSQL_REPLICATION_USER} --password=${MYSQL_REPLICATION_PASSWORD}"
          command:
          - /bin/bash
          env:
          - name: MYSQL_PWD
            valueFrom:
              secretKeyRef:
                key: mysql-root-password
                name: mysql-mysqlha
          - name: MYSQL_REPLICATION_USER
            value: repl
          - name: MYSQL_REPLICATION_PASSWORD
            valueFrom:
              secretKeyRef:
                key: mysql-replication-password
                name: mysql-mysqlha
          image: registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0
          imagePullPolicy: IfNotPresent
          name: xtrabackup
          ports:
          - containerPort: 3307
            name: xtrabackup
            protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          volumeMounts:
          - mountPath: /var/lib/mysql
            name: data
            subPath: mysql
          - mountPath: /etc/mysql/conf.d
            name: conf
          - mountPath: /mnt/scripts
            name: scripts
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        securityContext: {}
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: conf
        - configMap:
            defaultMode: 420
            name: mysql-mysqlha
          name: config-map
        - emptyDir: {}
          name: scripts
    volumeClaimTemplates:
    - metadata:
        creationTimestamp: null
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
      status:
        phase: Pending
  status:
    replicas: 3
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""
[root@master hrr]#

7.客户端验证

创建mysql-client容器，参考前面“ helm install mysql apphub/mysqlha”中的


[root@master hrr]# kubectl run mysql-client --image=mysql:5.7.13 -it --rm --restart='Never' --namespace default -- /bin/sh
Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter.
 Error attaching, falling back to logs: error dialing backend: dial tcp: lookup node3 on 192.168.222.2:53: no such host
Error from server: Get https://node3:10250/containerLogs/default/mysql-client/mysql-client: dial tcp: lookup node3 on 192.168.222.2:53: no such host
[root@master hrr]# 
[root@master hrr]# kubectl get pod -owide
NAME              READY     STATUS    RESTARTS   AGE       IP            NODE
mysql-client      1/1       Running   0          48s       172.17.93.3   node3
mysql-mysqlha-0   2/2       Running   0          54m       172.17.93.2   node3
mysql-mysqlha-1   2/2       Running   0          53m       172.17.78.2   node2
mysql-mysqlha-2   2/2       Running   0          53m       172.17.86.2   master
[root@master hrr]#

在node3上查看


[root@node3 ~]# docker ps
CONTAINER ID        IMAGE                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
8261abe363b0        mysql:5.7.13                                                    "docker-entrypoint..."   56 seconds ago      Up 55 seconds                           k8s_mysql-client.d32e2e5_mysql-client_default_90905092-c779-11e9-83c0-000c296be0a4_0d4f202f
1fac37badaa8        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 56 seconds ago      Up 56 seconds                           k8s_POD.cace165_mysql-client_default_90905092-c779-11e9-83c0-000c296be0a4_a0f9ab22
d159b422bcdf        registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0   "/bin/bash -c 'set..."   54 minutes ago      Up 54 minutes                           k8s_xtrabackup.516a4a61_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_63f2a879
6aae997c995d        mysql:5.7.13                                                    "docker-entrypoint..."   54 minutes ago      Up 54 minutes                           k8s_mysql.b9eef278_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_7dee904b
40695f86aaf7        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 54 minutes ago      Up 54 minutes                           k8s_POD.62b22cd9_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_58c67c1e
[root@node3 ~]#

写入数据


[root@node3 ~]# docker exec -ti 8261abe363b0 sh
# mysql -h172.17.93.2 -uroot
ERROR 1045 (28000): Access denied for user 'root'@'172.17.93.3' (using password: NO)
# mysql -h172.17.93.2 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1018
Server version: 8.0.17 MySQL Community Server - GPL
 
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
 
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.01 sec)
 
mysql> CREATE DATABASE test;
Query OK, 1 row affected (0.02 sec)
 
mysql> CREATE TABLE test.messages (message VARCHAR(250));
Query OK, 0 rows affected (0.03 sec)
 
mysql> INSERT INTO test.messages VALUES ('hello');
Query OK, 1 row affected (0.03 sec)
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.00 sec)
 
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
mysql> select * from messages;
+---------+
| message |
+---------+
| hello   |
+---------+
1 row in set (0.01 sec)
 
mysql> exit
Bye
#

连接svc查看，这个是read-only的，可以看到是轮询的，查询了2次才到有test中的库去


# mysql -h10.254.54.188 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1058
Server version: 8.0.17 MySQL Community Server - GPL
 
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
 
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)
 
mysql> exit
Bye
# mysql -h10.254.54.188 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1054
Server version: 8.0.17 MySQL Community Server - GPL
 
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
 
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.01 sec)
 
mysql> exit
Bye
# mysql -h10.254.54.188 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1072
Server version: 8.0.17 MySQL Community Server - GPL
 
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
 
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.00 sec)
 
mysql> exit
Bye
#

四、故障排除

1.helm init报错

故障现象：


[root@docker ~]# helm init
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Error: https://kubernetes-charts.storage.googleapis.com is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: net/http: TLS handshake timeout
[root@docker ~]#

可能原因：
可能国内访问不了https://kubernetes-charts.storage.googleapis.com/index.yaml，并且网速慢。
解决方法：
在初始化时，使用--stable-repo-url参数，将仓库地址改为阿里云的helm仓库地址。
helm init --stable-repo-url=https://apphub.aliyuncs.com

2.Pending、PersistentVolumeClaim is not bound

故障现象：


[root@docker ~]# kubectl get pod
NAME              READY     STATUS    RESTARTS   AGE
mysql-mysqlha-0   0/2       Pending   0          35s
nginx             1/1       Running   0          2m
[root@docker ~]#
[root@docker ~]# kubectl describe pod mysql-mysqlha-0
……
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  1m        4s      8   {default-scheduler }            Warning     FailedScheduling    SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "data-mysql-mysqlha-0", which is unexpected.

可能原因：
第一次使用helm安装mysql时，没有提前创建pv，导致报pvc没有bound的错。
解决方法：
查看pvc的模式，创建pv，创建过程请参考前面。

3.RunContainerError、Failed to pull image "mysql:5.7.13"

故障现象：


[root@docker hrr]# kubectl get pod
NAME              READY     STATUS              RESTARTS   AGE
mysql-mysqlha-0   0/2       RunContainerError   0          19s
nginx             1/1       Running             1          1h
[root@docker hrr]# 
[root@docker ~]# kubectl describe pod mysql-mysqlha-0
……
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath           Type        Reason          Message
  --------- --------    -----   ----            -------------           --------    ------          -------
  18m       5m      49  {default-scheduler }                    Warning     FailedScheduling    SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "data-mysql-mysqlha-0", which is unexpected.
  4m        4m      1   {default-scheduler }                    Normal      Scheduled       Successfully assigned mysql-mysqlha-0 to master
  4m        4m      1   {kubelet master}                    Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
  4m        4m      1   {kubelet master}    spec.containers{mysql}      Normal      Pulling         pulling image "mysql:5.7.13"
  10s       10s     1   {kubelet master}    spec.containers{mysql}      Warning     Failed          Failed to pull image "mysql:5.7.13": Get https://registry-1.docker.io/v2/library/mysql/manifests/5.7.13: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fmysql%3Apull&service=registry.docker.io: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  10s       10s     1   {kubelet master}    spec.containers{xtrabackup} Normal      Pulling         pulling image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0"
  9s        9s      1   {kubelet master}    spec.containers{xtrabackup} Warning     Failed          Failed to pull image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0": image pull failed for registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0, this may be because there are no credentials on this request.  details: (repository registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup not found: does not exist or no pull access)
  9s        9s      1   {kubelet master}                    Warning     FailedSync      Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with ErrImagePull: "Get https://registry-1.docker.io/v2/library/mysql/manifests/5.7.13: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fmysql%3Apull&service=registry.docker.io: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
, failed to "StartContainer" for "xtrabackup" with ErrImagePull: "image pull failed for registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0, this may be because there are no credentials on this request.  details: (repository registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup not found: does not exist or no pull access)"
]
  8s    8s  1   {kubelet master}    spec.containers{mysql}      Normal  BackOff     Back-off pulling image "mysql:5.7.13"
  8s    8s  1   {kubelet master}    spec.containers{xtrabackup} Normal  BackOff     Back-off pulling image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0"
  8s    8s  1   {kubelet master}                    Warning FailedSync  Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with ImagePullBackOff: "Back-off pulling image \"mysql:5.7.13\""
, failed to "StartContainer" for "xtrabackup" with ImagePullBackOff: "Back-off pulling image \"registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0\""
]
[root@docker hrr]#

可能原因：
下载不到docker image，本地没有mysql镜像。
解决方法：
手动下载mysql镜像，将镜像docker tag为配置中的名字，如何手动下载，请参考上面。

4.CrashLoopBackOff、permission denied

故障现象：


[root@docker docker]# kubectl get pod
NAME              READY     STATUS              RESTARTS   AGE
mysql-mysqlha-0   0/2       RunContainerError   0          9s
nginx             1/1       Running             2          1h
[root@docker docker]#
[root@docker docker]# kubectl describe pod mysql-mysqlha-0 
……
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath           Type        Reason          Message
  --------- --------    -----   ----            -------------           --------    ------          -------
  15s       15s     1   {default-scheduler }                    Normal      Scheduled       Successfully assigned mysql-mysqlha-0 to master
  14s       14s     1   {kubelet master}                    Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
  14s       3s      3   {kubelet master}    spec.containers{mysql}      Normal      Pulled          Container image "mysql:5.7.13" already present on machine
  14s       3s      3   {kubelet master}    spec.containers{xtrabackup} Normal      Pulled          Container image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0" already present on machine
  14s       3s      3   {kubelet master}                    Warning     FailedSync      Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
, failed to "StartContainer" for "xtrabackup" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
]
[root@docker docker]#
[root@docker docker]# kubectl get pod
NAME              READY     STATUS             RESTARTS   AGE
mysql-mysqlha-0   1/2       CrashLoopBackOff   2          5m
nginx             1/1       Running            2          1h
[root@docker docker]#
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  5m        5m      1   {default-scheduler }            Normal      Scheduled   Successfully assigned mysql-mysqlha-0 to master
  1m        1m      1   {kubelet master}            Warning     FailedSync  Error syncing pod, skipping: [failed to "StartContainer" for "xtrabackup" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
, failed to "StartContainer" for "mysql" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
]
  5m    44s 23  {kubelet master}        Warning FailedSync  Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
, failed to "StartContainer" for "xtrabackup" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
]
  32s   32s 1   {kubelet master}    spec.containers{mysql}      Normal  Started     Started container with docker id d2f5e2c645b8
  32s   32s 1   {kubelet master}    spec.containers{xtrabackup} Normal  Started     Started container with docker id 6f2fb901a587
  5m    32s 25  {kubelet master}    spec.containers{xtrabackup} Normal  Pulled      Container image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0" already present on machine
  32s   32s 1   {kubelet master}    spec.containers{mysql}      Normal  Created     Created container with docker id d2f5e2c645b8; Security:[seccomp=unconfined]
  32s   32s 1   {kubelet master}    spec.containers{xtrabackup} Normal  Created     Created container with docker id 6f2fb901a587; Security:[seccomp=unconfined]
  30s   30s 1   {kubelet master}    spec.containers{mysql}      Normal  Created     Created container with docker id 104f64db6e8c; Security:[seccomp=unconfined]
  30s   30s 1   {kubelet master}    spec.containers{mysql}      Normal  Started     Started container with docker id 104f64db6e8c
  28s   26s 3   {kubelet master}                    Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 10s restarting failed container=mysql pod=mysql-mysqlha-0_default(18f319d5-c6d2-11e9-baf7-000c296be0a4)"
 
  5m    11s 27  {kubelet master}    spec.containers{mysql}  Normal  Pulled          Container image "mysql:5.7.13" already present on machine
  5m    11s 5   {kubelet master}                Warning MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
  11s   11s 1   {kubelet master}    spec.containers{mysql}  Normal  Created         Created container with docker id ec8f0827294b; Security:[seccomp=unconfined]
  11s   11s 1   {kubelet master}    spec.containers{mysql}  Normal  Started         Started container with docker id ec8f0827294b
  28s   7s  5   {kubelet master}    spec.containers{mysql}  Warning BackOff         Back-off restarting failed docker container
  10s   7s  2   {kubelet master}                Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 20s restarting failed container=mysql pod=mysql-mysqlha-0_default(18f319d5-c6d2-11e9-baf7-000c296be0a4)"
[root@docker docker]#
[root@docker docker]# docker ps -a
CONTAINER ID        IMAGE                                                           COMMAND                  CREATED              STATUS                          PORTS               NAMES
24a5e40cfbe6        mysql:5.7.13                                                    "docker-entrypoint..."   About a minute ago   Exited (1) About a minute ago                       k8s_mysql.b9eef278_mysql-mysqlha-0_default_18f319d5-c6d2-11e9-baf7-000c296be0a4_f37f9154
6f2fb901a587        registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0   "/bin/bash -c 'set..."   5 minutes ago        Up 5 minutes                                        k8s_xtrabackup.516a4a61_mysql-mysqlha-0_default_18f319d5-c6d2-11e9-baf7-000c296be0a4_4c633d64
61a82b9ad65a        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 10 minutes ago       Up 10 minutes                                       k8s_POD.62b22cd9_mysql-mysqlha-0_default_18f319d5-c6d2-11e9-baf7-000c296be0a4_9f7337a5
dc6071e8f613        192.168.222.129:5000/nginx                                      "nginx -g 'daemon ..."   26 minutes ago       Up 26 minutes                                       k8s_nginx.3556dff8_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_9d257b60
15929311253f        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 26 minutes ago       Up 26 minutes                                       k8s_POD.cace165_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_d9ad1e0f
48f464ad9c9b        192.168.222.129:5000/pod-infrastructure:latest                  "/pause"                 37 minutes ago       Exited (2) 26 minutes ago                           k8s_POD.cace165_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_eb3bebdb
c01c41763b06        192.168.222.129:5000/nginx                                      "nginx -g 'daemon ..."   About an hour ago    Exited (0) 37 minutes ago                           k8s_nginx.3556dff8_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_0848bc8b
[root@docker docker]# docker logs 24a5e40cfbe6
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
[root@docker docker]#

可能原因：
docker对pvc没有权限，由于这里使用的是nfs作后端存储，所以可能是nfs目录的权限没放开。
解决方法：
将nfs目录权限设置为chown nfsnobody:nfsnobody -R /nfsdata，并且/etc/exportfs中设置no_root_squash，不压缩root权限，即在docker可以使用任何用户写入数据，具体操作步骤请参考上面。

5.MissingClusterDNS

故障现象：


[root@master ~]# kubectl describe pod mysql-mysqlha-0
……
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath           Type        Reason          Message
  --------- --------    -----   ----            -------------           --------    ------          -------
  56m       56m     1   {default-scheduler }                    Normal      Scheduled       Successfully assigned mysql-mysqlha-0 to master
  56m       56m     3   {kubelet master}                    Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
  56m       56m     1   {kubelet master}    spec.containers{mysql}      Normal      Pulled          Container image "mysql:5.7.13" already present on machine
  56m       56m     1   {kubelet master}    spec.containers{mysql}      Normal      Created         Created container with docker id 31013a7822ae; Security:[seccomp=unconfined]
  56m       56m     1   {kubelet master}    spec.containers{mysql}      Normal      Started         Started container with docker id 31013a7822ae
  56m       56m     1   {kubelet master}    spec.containers{xtrabackup} Normal      Pulled          Container image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0" already present on machine
  56m       56m     1   {kubelet master}    spec.containers{xtrabackup} Normal      Created         Created container with docker id a5f2f49c1eef; Security:[seccomp=unconfined]
  56m       56m     1   {kubelet master}    spec.containers{xtrabackup} Normal      Started         Started container with docker id a5f2f49c1eef

可能原因：
没有配置kube-dns。
解决方法：
请参考上面配置kube-dns。

故障现象：
可能原因：
解决方法：

五、参考资料

K8s Helm Charts 一键安装云原生应用指南：
https://mp.weixin.qq.com/s/qeLA2F4vNGUDYVdQSHIWtQ
mysql有状态服务部署：
https://segmentfault.com/a/1190000012244714
记k8s的一次pv&pvc创建过程：
https://staight.github.io/2018/08/18/%E8%AE%B0k8s%E7%9A%84%E4%B8%80%E6%AC%A1pv&pvc%E5%88%9B%E5%BB%BA%E8%BF%87%E7%A8%8B/
nfs挂载Permission denied问题解决：
https://blog.csdn.net/weixin_42256397/article/details/97283989

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小丑西瓜9/article/detail/523646