devbox
AllinToyou
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

基于K8S搭建企业级高可用集群

作者：AllinToyou | 2024-05-16 13:10:33

赞

踩

基于K8S搭建企业级高可用集群

目录

1.项目架构图

1.1项目介绍

1.2项目说明

2.项目步骤

1.规划设计整个集群的架构，部署k8s单master的集群环境，1个master2个node点、相关的服务器。

1.1 部署k8s集群

2.部署jumpserver堡垒机，接入node节点

3.将自己的开发的web接口系统制作成镜像，部署到k8s里作为web应用，采用HPA技术，当cpu使用率达到50%的时候，进行水平扩缩，最小20个，最多40个pod。

3.1使用HPA的配置文件如下

4.部署nfs服务器，为整个web集群提供数据，让所有的web 业务pod都取访问，通过pv和pvc、卷挂载实现。

4.1搭建nfs服务器

5.部署主从复制的Mysql集群，以及Redis集群提供数据库服务

6.安装promethues对所有集群（cpu，内存，网络带宽，web服务，数据库服务，磁盘IO等）进行监控包括k8s集群

7.使用ingress给web业务做负载均衡

8.使用探针（liveless、readiness、startup）的httpGet和exec方法对web业务pod进行监控，一旦出现问题马上重启，增强业务pod的可靠性。

9.对整个k8s集群和相关的服务器进行压力测试

1.项目架构图

1.1项目介绍

该架构图主要从安全，稳定，效率，成本四个角度出发

1.安全方面：在业务入口处，使用阿里云的WAF防火墙产品，以及阿里云的DDOS高防包，可以有效地防止sql注入，DDos攻击等百分之95的安全攻击

2.稳定方面：考虑到项目部署在一个机房的一个地区。如果这个地区出现故障将导致整体项目不可用，所以采用同一地区，双机房的部署方式，项目的数据库和redis互相连通，在一个机房出现问题时，可以快速切换到另一个机房，最短的时间恢复业务

3.效率方面：如果说业务机房在广州，考虑到用户量的增大，用户分布在全国各地，新疆，河北这一片区域的用户，访问卡顿问题，购买云产商阿里云的CDN服务，可以避免出现这种地域访问卡顿问题，同时整体统一使用Prometheus＋Grafana的监控告警，以及阿里云的SLS日志产品，对整个双地区的日志进行收集，方便快速定位日志

4.成本方面：在保障业务高可用的情况下，可以有效的节约成本，减少损失

1.2项目说明

由于本地虚拟机搭建，本项目没有购买CDN,WAF,DDOS，SLS等服务，将会搭建一套k8s集群，在生产环境可以使用以上服务

2.项目步骤

1.规划设计整个集群的架构，部署k8s单master的集群环境，1个master2个node点、相关的服务器。

1.1 部署k8s集群

  1. 注意：以下操作标注每台和master执行
  2.  
  3. 1.关闭防火墙和selinux（每台）
  4. [root@scmaster ~]# service firewalld stop
  5. Redirecting to /bin/systemctl stop firewalld.service
  6.  
  7. [root@scmaster ~]# systemctl disable firewalld
  8. Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
  9. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
  10.  
  11. [root@scmaster ~]# vim /etc/selinux/config 
  12. SELINUX=disabled
  13.  
  14. [root@scmaster ~]# getenforce 
  15. Enforcing
  16.  
  17. 2.安装docker（每台）
  18. 每台节点服务器上都操作master和node上进行
  19. 卸载原来安装过的docker，如果没有安装可以不需要卸载
  20. yum remove docker \
  21.                   docker-client \
  22.                   docker-client-latest \
  23.                   docker-common \
  24.                   docker-latest \
  25.                   docker-latest-logrotate \
  26.                   docker-logrotate \
  27.                   docker-engine
  28.  
  29. 安装yum相关的工具，下载docker-ce.repo文件
  30. [root@cali ~]#  yum install -y yum-utils -y
  31.  
  32. [root@cali ~]#yum-config-manager \
  33.     --add-repo \
  34.     https://download.docker.com/linux/centos/docker-ce.repo
  35.  
  36. 下载docker-ce.repo文件存放在/etc/yum.repos.d
  37. [root@cali yum.repos.d]# pwd
  38. /etc/yum.repos.d
  39. [root@cali yum.repos.d]# ls
  40. CentOS-Base.repo  CentOS-Debuginfo.repo  CentOS-Media.repo    CentOS-Vault.repo          docker-ce.repo
  41. CentOS-CR.repo    CentOS-fasttrack.repo  CentOS-Sources.repo  CentOS-x86_64-kernel.repo  nginx.repo
  42.  
  43. 安装docker-ce软件
  44. [root@cali yum.repos.d]#yum install docker-ce-20.10.18 docker-ce-cli-20.10.18 containerd.io docker-compose-plugin -y
  45. container engine 容器引擎
  46. docker是一个容器管理的软件
  47. docker-ce 是服务器端软件 server
  48. docker-ce-cli 是客户端软件 client
  49. docker-compose-plugin 是compose插件，用来批量启动很多容器，在单台机器上
  50. containerd.io  底层用来启动容器的
  51.  
  52.  
  53. [root@scmaster ~]# docker --version
  54. Docker version 20.10.18, build b40c2f6
  55.  
  56. 4.启动docker服务（每台）
  57. [root@scmaster ~]# systemctl start docker
  58.  
  59. [root@scmaster ~]# ps aux|grep docker
  60. root      53288  1.5  2.3 1149960 43264 ?       Ssl  15:11   0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
  61. root      53410  0.0  0.0 112824   984 pts/0    S+   15:11   0:00 grep --color=auto docker
  62.  
  63. 5.设置docker服务开机启动
  64. [root@scmaster ~]# systemctl enable docker
  65. Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
  66.  
  67. 3.配置 Docker使用systemd作为默认Cgroup驱动
  68. 每台服务器上都要操作，master和node上都要操作执行下面的脚本，会产生 /etc/docker/daemon.json文件
  69. 	cat <<EOF > /etc/docker/daemon.json
  70. 	{
  71. 	   "exec-opts": ["native.cgroupdriver=systemd"]
  72. 	}
  73. 	EOF
  74. 	
  75. #重启docker
  76. [root@scmaster docker]# systemctl restart docker
  77.  
  78.  
  79. [root@web1 yum.repos.d]# cat /etc/docker/daemon.json 
  80. {
  81.    "exec-opts": ["native.cgroupdriver=systemd"]
  82. }
  83.  
  84. 关闭swap分区
  85. 	因为k8s不想使用swap分区来存储数据，使用swap会降低性能
  86. 	每台服务器都需要操作
  87. 	swapoff -a # 临时关闭
  88. 	sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab #永久关闭
  89.  
  90. 修改主机名（每台）
  91. [root@prometheus ~]# cat >> /etc/hosts << EOF
  92. 192.168.159.137 prometheus
  93. 192.168.159.131 web1
  94. 192.168.159.132 web2
  95. 192.168.159.133 web3
  96. EOF
  97.  
  98. 修改内核参数（每台）
  99. cat <<EOF >>  /etc/sysctl.conf  
  100. net.bridge.bridge-nf-call-ip6tables = 1
  101. net.bridge.bridge-nf-call-iptables = 1
  102. net.ipv4.ip_nonlocal_bind = 1
  103. net.ipv4.ip_forward = 1
  104. vm.swappiness=0
  105. EOF
  106.  
  107. cat /etc/sysctl.conf （每台）
  108. # sysctl settings are defined through files in
  109. # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
  110. #
  111. # Vendors settings live in /usr/lib/sysctl.d/.
  112. # To override a whole file, create a new file with the same in
  113. # /etc/sysctl.d/ and put new settings there. To override
  114. # only specific settings, add a file with a lexically later
  115. # name in /etc/sysctl.d/ and put new settings there.
  116. #
  117. # For more information, see sysctl.conf(5) and sysctl.d(5).
  118. net.bridge.bridge-nf-call-ip6tables = 1
  119. net.bridge.bridge-nf-call-iptables = 1
  120. net.ipv4.ip_nonlocal_bind = 1
  121. net.ipv4.ip_forward = 1
  122. vm.swappiness=0
  123.  
  124. 每台机器都做
  125. [root@prometheus ~]# sysctl -p
  126. net.bridge.bridge-nf-call-ip6tables = 1
  127. net.bridge.bridge-nf-call-iptables = 1
  128. net.ipv4.ip_nonlocal_bind = 1
  129. net.ipv4.ip_forward = 1
  130. vm.swappiness = 0
  131.  
  132. 6.安装kubeadm,kubelet和kubectl（每台）
  133. kubeadm --》k8s的管理程序--》在master上运行的--》建立整个k8s集群，背后是执行了大量的脚本，帮助我们去启动k8s
  134.  
  135. kubelet --》在node节点上用来管理容器的--》管理docker，告诉docker程序去启动容器，管理docker容器，告诉docker程序去启动pod
  136. 一个在集群中每个节点（node）上运行的代理。 它保证容器（containers）都运行在 Pod 中。
  137.  
  138. kubectl --》在master上用来给node节点发号施令的程序，用来控制node节点的，告诉它们做什么事情的，是命令行操作的工具
  139.  
  140. # 添加kubernetes YUM软件源（每台）
  141. 	cat > /etc/yum.repos.d/kubernetes.repo << EOF
  142. 	[kubernetes]
  143. 	name=Kubernetes
  144. 	baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
  145. 	enabled=1
  146. 	gpgcheck=0
  147. 	repo_gpgcheck=0
  148. 	gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  149. 	EOF
  150.  
  151. #安装kubeadm,kubelet和kubectl ,并且指定版本（每台）
  152.  
  153. yum install -y kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6  
  154. --》最好指定版本，因为1.24的版本默认的容器运行时环境不是docker了
  155.  
  156. 下面这个网站有解决方法	
  157. https://www.docker.com/blog/dockershim-not-needed-docker-desktop-with-kubernetes-1-24/
  158. 	
  159. #设置开机自启，因为kubelet是k8s在node节点上的代理，必须开机要运行的（每台）
  160. 	systemctl enable  kubelet
  161.  
  162. #提前准备coredns:1.8.4的镜像，后面需要使用,需要在每台机器上下载镜像
  163. [root@master ~]#  docker pull  coredns/coredns:1.8.4
  164.  
  165. [root@master ~]# docker tag coredns/coredns:1.8.4 registry.aliyuncs.com/google_containers/coredns:v1.8.4
  166.  
  167. 部署Kubernetes Master
  168. master主机执行
  169. #初始化操作在master服务器上执行
  170. 	[root@master ~]#kubeadm init \
  171. 	--apiserver-advertise-address=192.168.237.180 \ 
  172. 	--image-repository registry.aliyuncs.com/google_containers \
  173. 	--service-cidr=10.1.0.0/16 \
  174. 	--pod-network-cidr=10.244.0.0/16
  175. 	#192.168.92.132 是master的ip
  176. 	#      --service-cidr string                  Use alternative range of IP address for service VIPs. (default "10.96.0.0/12")  服务发布暴露--》dnat
  177. 	#      --pod-network-cidr string              Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
  178.  
  179.  
  180.  
  181. 	
  182. Your Kubernetes control-plane has initialized successfully!
  183.  
  184. To start using your cluster, you need to run the following as a regular user
  185.  
  186. (在master上执行这三个命令)
  187.   mkdir -p $HOME/.kube
  188.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  189.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  190.  
  191. Alternatively, if you are the root user, you can run:
  192. （再执行这个命令）
  193.   export KUBECONFIG=/etc/kubernetes/admin.conf
  194.  
  195. You should now deploy a pod network to the cluster.
  196. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  197.   https://kubernetes.io/docs/concepts/cluster-administration/addons/
  198.  
  199. Then you can join any number of worker nodes by running the following on eac
  200. （最后复制产生的下面些命令去node）
  201. kubeadm join 192.168.159.137:6443 --token 4aw7ag.a8qzacdm0jcd3hxs \
  202. 	--discovery-token-ca-cert-hash sha256:09511c561866693a9e7f574c1162b3bc28c7 
  203.  
  204. 加入k8s集群（node服务器）

2.部署jumpserver堡垒机，接入node节点

1.下载Jumpserver官网提供的文件一键部署(提前部署阿里YUM)

curl -sSL https://github.com/jumpserver/jumpserver/releases/download/v2.24.0/quick_start.sh | bash

2.启动jumpserver服务

  1. cd /opt/jumpserver-installer-v2.24.0 ##(默认安装目录)
  2. #启动
  3. ./jmsctl.sh start ## 或者 ./jmsctl.sh restart/down/uninstall来管理服务的状态

3.查看镜像和容器是否存在

  1. docker images 
  2. ##验证
  3. REPOSITORY           TAG       IMAGE ID       CREATED        SIZE
  4. jumpserver/mariadb   10.6      aac2cf878de9   2 months ago   405MB
  5. jumpserver/redis     6.2       48da0c367062   2 months ago   113MB
  6. jumpserver/web       v2.24.0   a9046484de3d   6 months ago   416MB
  7. jumpserver/core      v2.24.0   83b8321cf9e0   6 months ago   1.84GB
  8. jumpserver/koko      v2.24.0   708386a1290e   6 months ago   770MB
  9. jumpserver/lion      v2.24.0   81602523a0ac   6 months ago   351MB
  10. jumpserver/magnus    v2.24.0   e0a90a2217ad   6 months ago   157MB
  11.  
  12. docker ps | grep jumpser
  13. ##验证 查看容器是否健康
  14. 1c26e0acbc8e   jumpserver/core:v2.24.0     "./entrypoint.sh sta…"   8 hours ago    Up 8 hours (healthy)   8070/tcp, 8080/tcp                                                                                                           
  15. d544ec4a155d   jumpserver/magnus:v2.24.0   "./entrypoint.sh"        8 hours ago    Up 8 hours (healthy)   15211-15212/tcp, 0.0.0.0:33060-33061->33060-33061/tcp, :::33060-33061->33060-33061/tcp, 54320/tcp, 0.0.0.0:63790->63790/tcp, :::63790->63790/tcp   jms_magnus
  16. 1d409d8d4a62   jumpserver/lion:v2.24.0     "./entrypoint.sh"        8 hours ago    Up 8 hours (healthy)   4822/tcp
  17. b6bbd8bf21e8   jumpserver/koko:v2.24.0     "./entrypoint.sh"        8 hours ago    Up 8 hours (healthy)   0.0.0.0:2222->2222/tcp, :::2222->2222/tcp, 5000/tcp                                                               
  18. 5774d0475eef   jumpserver/web:v2.24.0      "/docker-entrypoint.…"   8 hours ago    Up 8 hours (healthy)   0.0.0.0:80->80/tcp, :::80->80/tcp                                                                                   
  19. 18c1f9eecbaf   jumpserver/core:v2.24.0     "./entrypoint.sh sta…"   8 hours ago    Up 8 hours (healthy)   8070/tcp, 8080/tcp                                                                                                           
  20. 2767e8938563   jumpserver/mariadb:10.6     "docker-entrypoint.s…"   19 hours ago   Up 8 hours (healthy)   3306/tcp
  21. 635f74cc0e43   jumpserver/redis:6.2        "docker-entrypoint.s…"   19 hours ago   Up 8 hours (healthy)   6379/tcp

4.访问当前部署虚拟机的IP地址

  1. http://192.168.xxx.xxx
  2. 默认用户: admin  默认密码: admin (超级用户)

5.接入Master和Node节点（参考jumpserver使用教程）

3.将自己的开发的web接口系统制作成镜像，部署到k8s里作为web应用，采用HPA技术，当cpu使用率达到50%的时候，进行水平扩缩，最小20个，最多40个pod。

3.1使用HPA的配置文件如下

  1. [root@k8s-master HPA]# cat hpa.yaml 
  2. apiVersion: autoscaling/v2beta2
  3. kind: HorizontalPodAutoscaler
  4. metadata:
  5.   name: nginx-hpa
  6. spec:
  7.   scaleTargetRef:
  8.     apiVersion: apps/v1
  9.     kind: Deployment
  10.     name: nginx
  11.   minReplicas: 20
  12.   maxReplicas: 40
  13.   metrics:
  14.   - type: Resource
  15.     resource:
  16.       name: cpu
  17.       target:
  18.         type: Utilization
  19.         averageUtilization: 50
  1. [root@k8s-master HPA]# cat nginx.yaml 
  2. apiVersion: apps/v1 
  3. kind: Deployment
  4. metadata:
  5.   name: nginx
  6.   labels:
  7.     app: nginx
  8. spec:
  9.   replicas: 2
  10.   selector:
  11.     matchLabels:
  12.       app: nginx  
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: nginx
  17.     spec:
  18.       containers:
  19.       - name: nginx
  20.         image: nginx:latest # replace it with your exactly <image_name:tags>
  21.         ports:
  22.         - containerPort: 80
  23.         resources:
  24.           requests:                         ##必须设置，不然HPA无法运行。
  25.             cpu: 500m
  1. [root@k8s-master HPA]# kubectl get hpa
  2. NAME        REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  3. nginx-hpa   Deployment/nginx   0%/50%    20        40        3          20h

4.部署nfs服务器，为整个web集群提供数据，让所有的web 业务pod都取访问，通过pv和pvc、卷挂载实现。

4.1搭建nfs服务器

  1. 使用nfs的数据流程
  2. pod-->volume-->pvc-->pv-->nfs
  3.  
  4. 在每个节点都要安装nfs服务器
  5. yum install nfs-utils -y
  6.  
  7. [root@master 55]# service nfs restart
  8. 2.设置共享目录
  9. [root@master 55]# vim /etc/exports
  10. [root@master 55]# cat /etc/exports
  11. /sc/web   192.168.2.0/24(rw,no_root_squash,sync)
  12. [root@master 55]# 
  13. 3.新建共享目录和index.html网页
  14. [root@master 55]# mkdir /sc/web -p
  15. [root@master 55]# cd /sc/web/
  16. [root@master web]# echo "welcome to sanchuang" >index.html
  17. [root@master web]# ls
  18. index.html
  19. [root@master web]# cat index.html 
  20. welcome to sanchuang
  21. [root@master web]#
  22. 4.刷新nfs或者重新输出共享目录
  23. [root@master web]# exportfs  -a  输出所有共享目录
  24. [root@master web]# exportfs  -v  显示输出的共享目录
  25. /sc/web         192.168.2.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
  26. [root@master web]# exportfs  -r 重新输出所有的共享目录
  27. [root@master web]# exportfs  -v
  28. /sc/web         192.168.2.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
  29. [root@master web]# 
  30. [root@master web]# service nfs restart  重启nfs服务
  31. Redirecting to /bin/systemctl restart nfs.service
  32.  
  33. 测试是否成功
  34. [root@web1 ~]# mkdir /sc
  35. [root@web1 ~]# mount 192.168.159.134:/web /sc
  36. [root@web1 ~]# df -Th
  37. 192.168.159.134:/web    nfs4       17G  2.6G   15G   16% /sc
  38.  
  39. 5.创建pv使用nfs服务器上的共享目录
  40. [root@master 55]# vim nfs-pv.yaml
  41. [root@master 55]# cat nfs-pv.yaml 
  42. apiVersion: v1
  43. kind: PersistentVolume
  44. metadata:
  45.   name: sc-nginx-pv
  46.   labels:
  47.     type: sc-nginx-pv
  48. spec:
  49.   capacity:
  50.     storage: 10Gi
  51.   accessModes:
  52.     - ReadWriteMany
  53.   storageClassName: nfs         #pv对应的名字
  54.   nfs:
  55.     path: "/sc/web"       #nfs共享的目录
  56.     server: 192.168.2.130   #nfs服务器的ip地址
  57.     readOnly: false   #访问模式
  58.  
  59. [root@prometheus pv]# kubectl get pv
  60. NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                   STORAGECLASS   REASON   AGE
  61. sc-nginx-pv      10Gi       RWX            Retain           Available                           nfs                     32s
  62.  
  63. 6.创建pvc使用pv
  64. [root@master 55]# vim pvc-nfs.yaml
  65. [root@master 55]# cat pvc-nfs.yaml 
  66. apiVersion: v1
  67. kind: PersistentVolumeClaim
  68. metadata:
  69.   name: sc-nginx-pvc
  70. spec:
  71.   accessModes:
  72.   - ReadWriteMany      
  73.   resources:
  74.      requests:
  75.        storage: 1Gi
  76.   storageClassName: nfs #使用nfs类型的pv
  77.  
  78. [root@prometheus pv]# kubectl apply -f pvc-nfs.yaml 
  79. persistentvolumeclaim/sc-nginx-pvc created
  80. [root@prometheus pv]# kubectl get pvc
  81. NAME            STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  82. sc-nginx-pvc    Bound    sc-nginx-pv      10Gi       RWX            nfs            4s
  83.  
  84.  
  85. 7.创建pod使用pvc
  86. [root@master 55]# cat pod-nfs.yaml 
  87. apiVersion: v1
  88. kind: Pod
  89. metadata:
  90.   name: sc-pv-pod-nfs
  91. spec:
  92.   volumes:
  93.     - name: sc-pv-storage-nfs
  94.       persistentVolumeClaim:
  95.         claimName: sc-nginx-pvc
  96.   containers:
  97.     - name: sc-pv-container-nfs
  98.       image: nginx
  99.       ports:
  100.         - containerPort: 80
  101.           name: "http-server"
  102.       volumeMounts:
  103.         - mountPath: "/usr/share/nginx/html"
  104.           name: sc-pv-storage-nfs
  105.  
  106. [root@prometheus pv]# kubectl apply -f pod.yaml 
  107. pod/sc-pv-pod-nfs created
  108.  
  109. [root@prometheus pv]# kubectl get pod
  110. sc-pv-pod-nfs          1/1     Running   0             28s
  111.  
  112. 测试，结果发现内容为nfs1中储存的
  113. [root@prometheus pv]# curl 10.244.3.84
  114. wel come to qiuchao feizi feizi
  115.  
  116. 修改nfs的内容，再次访问
  117. [root@prometheus pv]# curl 10.244.3.84
  118. wel come to qiuchao feizi feizi
  119. qiuchaochao

5.部署主从复制的Mysql集群，以及Redis集群提供数据库服务

  1. 主从复制实验
  2.  
  3. 准备的机器为：web1，web3
  4.  
  5. 因为web3开启了二进制日志，选web3为主服务器，web1为从
  6.  
  7. 配置web3二进制日志
  8. [root@web3 backup]# cat /etc/my.cnf
  9. [mysqld_safe]
  10.  
  11. [client]
  12. socket=/data/mysql/mysql.sock
  13.  
  14. [mysqld]
  15. socket=/data/mysql/mysql.sock
  16. port = 3306
  17. open_files_limit = 8192
  18. innodb_buffer_pool_size = 512M
  19. character-set-server=utf8
  20. log_bin
  21. server_id = 1
  22. expire_logs_days = 3
  23.  
  24. [mysql]
  25. auto-rehash
  26. prompt=\u@\d \R:\m  mysql>
  27.  
  28. 检查是否打开
  29. root@(none) 11:44  mysql>show variables like 'log_bin';
  30. +---------------+-------+
  31. | Variable_name | Value |
  32. +---------------+-------+
  33. | log_bin       | ON    |
  34. +---------------+-------+
  35. 1 row in set (0.02 sec)
  36.  
  37. 保持主与从的数据一致性
  38. 在主服务器上备份数据
  39. [root@web3 backup]# mysqldump -uroot -p'Sanchuang123#' --all-databases > all_db.SAQL 
  40. mysqldump: [Warning] Using a password on the command line interface can be insecure.
  41. [root@web3 backup]# ls
  42. all_db.SAQL  check_pwd.sh  my.cnf  passwd  qiuchao.sql
  43.  
  44. 传到从服务器上/root目录
  45. [root@web3 backup]# scp all_db.SAQL root@192.168.159.131:/root
  46. all_db.SAQL                                     100%  885KB  66.2MB/s   00:00
  47.  
  48. 在从服务器上
  49. [root@web1 ~]# mysql -uroot -p'Sanchuang123#' < all_db.SAQL 
  50. mysql: [Warning] Using a password on the command line interface can be insecure.
  51.  
  52. 主服务器清除二进制文件
  53. root@(none) 15:17  mysql>reset master;
  54. Query OK, 0 rows affected (0.01 sec)
  55.  
  56. root@(none) 15:18  mysql>show master status;
  57. +-----------------+----------+--------------+------------------+-------------------+
  58. | File            | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
  59. +-----------------+----------+--------------+------------------+-------------------+
  60. | web3-bin.000001 |      154 |              |                  |                   |
  61. +-----------------+----------+--------------+------------------+-------------------+
  62. 1 row in set (0.00 sec)
  63.  
  64. 主服务器新建一个授权用户给从服务器复制二进制日志
  65. grant replication slave on *.* to 'ren'@'192.168.92.%' identified by 'Sanchuang123#';
  66. Query OK, 0 rows affected, 1 warning (0.00 sec)
  67.  
  68. 在从服务器上配置master info的信息
  69. root@(none) 15:16  mysql>CHANGE MASTER TO MASTER_HOST='192.168.92.134' ,
  70.       MASTER_USER='ren',
  71.       MASTER_PASSWORD='Sanchuang123#',
  72.       MASTER_PORT=3306,
  73.       MASTER_LOG_FILE='nginx-lb1-bin.000001',
  74.       MASTER_LOG_POS=154;
  75. Query OK, 0 rows affected, 2 warnings (0.02 sec)
  76.  
  77. 从服务器查看slave信息
  78. show slave status\G;
  79.                Slave_IO_State: 
  80.                   Master_Host: 192.168.159.133
  81.                   Master_User: renxj
  82.                   Master_Port: 3306
  83.                 Connect_Retry: 60
  84.               Master_Log_File: web3-bin.000001
  85.           Read_Master_Log_Pos: 154
  86.                Relay_Log_File: web1-relay-bin.000001
  87.                 Relay_Log_Pos: 4
  88.         Relay_Master_Log_File: web3-bin.000001
  89.              Slave_IO_Running: No
  90.             Slave_SQL_Running: No
  91.  
  92. 从服务器启动slave
  93. root@(none) 15:31  mysql> start slave;
  94.  
  95. root@(none) 15:31  mysql>show slave status\G;
  96. *************************** 1. row ***************************
  97.                Slave_IO_State: Waiting for master to send event
  98.                   Master_Host: 192.168.159.133
  99.                   Master_User: renxj
  100.                   Master_Port: 3306
  101.                 Connect_Retry: 60
  102.               Master_Log_File: web3-bin.000001
  103.           Read_Master_Log_Pos: 450
  104.                Relay_Log_File: web1-relay-bin.000002
  105.                 Relay_Log_Pos: 615
  106.         Relay_Master_Log_File: web3-bin.000001
  107.              Slave_IO_Running: Yes
  108.             Slave_SQL_Running: Yes

6.安装promethues对所有集群（cpu，内存，网络带宽，web服务，数据库服务，磁盘IO等）进行监控包括k8s集群

  1. [root@k8s-master prometheus-grafana]# kubectl get pod -n monitor-sa
  2. NAME                                  READY   STATUS    RESTARTS        AGE
  3. monitoring-grafana-7bf44d7b64-9hsks   1/1     Running   1 (5h36m ago)   3d1h
  4. node-exporter-bz8x5                   1/1     Running   6 (5h36m ago)   4d1h
  5. node-exporter-hrh2m                   1/1     Running   3 (5h36m ago)   4d1h
  6. node-exporter-mf5d6                   1/1     Running   4 (5h36m ago)   4d1h
  7. prometheus-server-6885985cfc-fr7k9    1/1     Running   1 (5h36m ago)   3d1h
  8.  
  9. [root@k8s-master prometheus-grafana]# kubectl get service -n monitor-sa
  10. NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
  11. alertmanager         ClusterIP   10.1.245.239   <none>        80/TCP           4d5h
  12. monitoring-grafana   NodePort    10.1.92.54     <none>        80:30503/TCP     4d1h
  13. prometheus           NodePort    10.1.240.51    <none>        9090:31066/TCP   4d23h

prometheus ＋ Grafana效果图

 

7.使用ingress给web业务做负载均衡

  1. 使用yaml文件方式去安装部署ingress controller
  2. wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.7.0/deploy/static/provider/cloud/deploy.yaml
  3.  
  4. [root@prometheus ingress]# ls
  5. deploy.yaml （新版配置文件）
  6.  
  7. 在冯老师新加坡的服务器上去下载镜像，然后导出，传回国内使用
  8. [root@k8smaster 4-4]# ls
  9. ingress-controller-deploy.yaml        
  10. kube-webhook-certgen-v1.1.0.tar.gz  
  11. sc-ingress.yaml
  12. ingress-nginx-controllerv1.1.0.tar.gz 
  13. nginx-deployment-nginx-svc-2.yaml   
  14. sc-nginx-svc-1.yaml
  15.  
  16. ingress-controller-deploy.yaml   是部署ingress controller使用的yaml文件
  17. ingress-nginx-controllerv1.1.0.tar.gz    ingress-nginx-controller镜像
  18. kube-webhook-certgen-v1.1.0.tar.gz       kube-webhook-certgen镜像
  19. sc-ingress.yaml 创建ingress的配置文件
  20. sc-nginx-svc-1.yaml  启动sc-nginx-svc服务和相关pod的yaml
  21. nginx-deployment-nginx-svc-2.yaml  启动sc-nginx-svc-2服务和相关pod的yaml
  22.  
  23. 1.将镜像部署到所有的node节点上
  24.  
  25. 2.导入镜像（所有机器）
  26.  
  27.  
  28. [root@web1 ~]# docker images|grep nginx-ingress
  29. registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller   v1.1.0     ae1a7201ec95   16 months ago   285MB
  30. [root@web1 ~]# docker images|grep kube-webhook
  31. registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen       v1.1.1     c41e9fcadf5a   18 months ago   47.7MB
  32.  
  33. 3.使用ingress-controller-deploy.yaml  文件去启动ingress  controller
  34. [root@prometheus 2]# kubectl apply -f ingress-controller-deploy.yaml 
  35. namespace/ingress-nginx unchanged
  36. serviceaccount/ingress-nginx unchanged
  37. configmap/ingress-nginx-controller unchanged
  38. clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
  39. clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
  40. role.rbac.authorization.k8s.io/ingress-nginx unchanged
  41. rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
  42. service/ingress-nginx-controller-admission unchanged
  43. service/ingress-nginx-controller unchanged
  44. deployment.apps/ingress-nginx-controller configured
  45. ingressclass.networking.k8s.io/nginx unchanged
  46. validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
  47. serviceaccount/ingress-nginx-admission unchanged
  48. clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
  49. clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
  50. role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
  51. rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
  52. job.batch/ingress-nginx-admission-create unchanged
  53. job.batch/ingress-nginx-admission-patch unchanged
  54.  
  55. [root@prometheus 2]#  kubectl get svc -n ingress-nginx
  56. NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
  57. ingress-nginx-controller             NodePort    10.1.11.51     <none>        80:31493/TCP,443:30366/TCP   103s
  58. ingress-nginx-controller-admission   ClusterIP   10.1.243.244   <none>        443/TCP                      103s
  59.  
  60. [root@prometheus 2]#  kubectl get pod -n ingress-nginx
  61. NAME                                        READY   STATUS      RESTARTS   AGE
  62. ingress-nginx-admission-create-g4vcz        0/1     Completed   0          2m24s
  63. ingress-nginx-admission-patch-wqlh6         0/1     Completed   1          2m24s
  64. ingress-nginx-controller-7cd558c647-b5pzx   0/1     Pending     0          2m25s
  65. ingress-nginx-controller-7cd558c647-wl9n4   1/1     Running     0          2m25s
  66.  
  67. [root@prometheus 2]# kubectl get service -n ingress-nginx
  68. NAME                                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
  69. ingress-nginx-controller             NodePort    10.1.82.223   <none>        80:32222/TCP,443:31427/TCP   112s
  70. ingress-nginx-controller-admission   ClusterIP   10.1.35.142   <none>        443/TCP                      113s
  71.  
  72. [root@prometheus 2]# cat sc-ingress.yaml 
  73. apiVersion: networking.k8s.io/v1
  74. kind: Ingress
  75. metadata:
  76.   name: sc-ingress
  77.   annotations:
  78.     kubernets.io/ingress.class: nginx
  79. spec:
  80.   ingressClassName: nginx
  81.   rules:
  82.   - host: www.feng.com
  83.     http:
  84.       paths:
  85.       - pathType: Prefix
  86.         path: /
  87.         backend:
  88.           service:
  89.             name: sc-nginx-svc  要绑定的服务
  90.             port:
  91.               number: 80
  92.   - host: www.zhang.com
  93.     http:
  94.       paths:
  95.       - pathType: Prefix
  96.         path: /
  97.         backend:
  98.           service:
  99.             name: sc-nginx-svc-2
  100.             port:
  101.               number: 80
  102.  
  103.  
  104.  
  105. 创建pod和发布服务一
  106. [root@k8smaster 4-4]# cat sc-nginx-svc-1.yaml 
  107. apiVersion: apps/v1
  108. kind: Deployment 
  109. metadata:
  110.   name: sc-nginx-deploy
  111.   labels:
  112.     app: sc-nginx-feng
  113. spec:
  114.   replicas: 3
  115.   selector:
  116.     matchLabels:
  117.       app: sc-nginx-feng
  118.   template:
  119.     metadata: 
  120.       labels:
  121.         app: sc-nginx-feng
  122.     spec:
  123.       containers:
  124.       - name: sc-nginx-feng
  125.         image: nginx
  126.         imagePullPolicy: IfNotPresent
  127.         ports:
  128.         - containerPort: 80
  129. ---
  130. apiVersion: v1
  131. kind: Service
  132. metadata:
  133.   name:  sc-nginx-svc
  134.   labels:
  135.     app: sc-nginx-svc
  136. spec:
  137.   selector:
  138.     app: sc-nginx-feng
  139.   ports:
  140.   - name: name-of-service-port
  141.     protocol: TCP
  142.     port: 80
  143.     targetPort: 80
  144.  
  145. [root@prometheus 2]# kubectl apply -f sc-nginx-svc-1.yaml 
  146. deployment.apps/sc-nginx-deploy created
  147. service/sc-nginx-svc created
  148. [root@prometheus 2]# kubectl get svc
  149. NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
  150. kubernetes     ClusterIP   10.1.0.1      <none>        443/TCP   33m
  151. sc-nginx-svc   ClusterIP   10.1.29.219   <none>        80/TCP    12s
  152.  
  153. [root@prometheus 2]# kubectl apply -f nginx-deployment-nginx-svc-2.yaml 
  154. deployment.apps/nginx-deployment created
  155. service/sc-nginx-svc-2 created
  156.  
  157. [root@prometheus 2]# kubectl get svc
  158. NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
  159. kubernetes       ClusterIP   10.1.0.1       <none>        443/TCP   51m
  160. sc-nginx-svc     ClusterIP   10.1.29.219    <none>        80/TCP    17m
  161. sc-nginx-svc-2   ClusterIP   10.1.183.100   <none>        80/TCP    7s
  162.  
  163.  
  164. [root@prometheus 2]# kubectl get ingress
  165. NAME         CLASS   HOSTS                        ADDRESS                           PORTS   AGE
  166. sc-ingress   nginx   www.feng.com,www.zhang.com   192.168.92.140,192.168.92.142   80      20m
  167.  
  168. [root@prometheus 2]# kubectl get pod -n ingress-nginx
  169. NAME                                        READY   STATUS      RESTARTS   AGE
  170. ingress-nginx-admission-create-7l4v5        0/1     Completed   0          13m
  171. ingress-nginx-admission-patch-bxpfq         0/1     Completed   1          13m
  172. ingress-nginx-controller-7cd558c647-jctwf   1/1     Running     0          13m
  173. ingress-nginx-controller-7cd558c647-sws4b   1/1     Running     0          4m29s
  174.  
  175.  
  176. 获取ingress controller对应的service暴露宿主机的端口，访问宿主机和相关端口，就可以验证ingress controller是否能进行负载均衡
  177.  
  178. [root@prometheus 2]# kubectl get service -n ingress-nginx
  179. NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
  180. ingress-nginx-controller             NodePort    10.1.236.56    <none>        80:30910/TCP,443:30609/TCP   26m
  181. ingress-nginx-controller-admission   ClusterIP   10.1.187.223   <none>        443/TCP                      26m
  182.  
  183. 在宿主机上使用域名进行访问
  184. [root@nfs ~]# cat /etc/hosts
  185. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  186. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  187. 192.168.159.134 NFS
  188. 192.168.92.140 www.feng.com
  189. 192.168.92.142 www.zhang.com
  190.  
  191. 因为我们是基于域名做的负载均衡的配置，所有必须要在浏览器里使用域名去访问，不能使用ip地址
  192. 同时ingress controller做负载均衡的时候是基于http协议的，7层负载均衡
  193.  
  194. [root@nfs ~]# curl www.feng.com
  195. <!DOCTYPE html>
  196. <html>
  197. <head>
  198. <title>Welcome to nginx!</title>
  199. <style>
  200. html { color-scheme: light dark; }
  201. body { width: 35em; margin: 0 auto;
  202. font-family: Tahoma, Verdana, Arial, sans-serif; }
  203. </style>
  204. </head>
  205. <body>
  206. <h1>Welcome to nginx!</h1>
  207. <p>If you see this page, the nginx web server is successfully installed and
  208. working. Further configuration is required.</p>
  209.  
  210. <p>For
  211. online documentation and support please refer to
  212. <a href="http://nginx.org/">nginx.org</a>.<br/>
  213. Commercial support is available at
  214. <a href="http://nginx.com/">nginx.com</a>.</p>
  215.  
  216. <p><em>Thank you for using nginx.</em></p>
  217. </body>
  218. </html>
  219.  
  220.  
  221. 访问zhang.con
  222. [root@nfs ~]# curl www.zhang.com
  223. <html>
  224. <head><title>503 Service Temporarily Unavailable</title></head>
  225. <body>
  226. <center><h1>503 Service Temporarily Unavailable</h1></center>
  227. <hr><center>nginx</center>
  228. </body>
  229. </html>
  230.  
  231. service2使用了pv＋pvc+nfs
  232. [root@prometheus 2]# kubectl get pv
  233. NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE
  234. sc-nginx-pv      10Gi       RWX            Retain           Bound    default/sc-nginx-pvc    nfs                     21h
  235. task-pv-volume   10Gi       RWO            Retain           Bound    default/task-pv-claim   manual                  25h
  236.  
  237. 刷新一下nfs服务器，最终成功
  238. [root@nfs ~]# curl www.zhang.com
  239. wel come to qiuchao feizi feizi
  240. qiuchaochao

8.使用探针（liveless、readiness、startup）的httpGet和exec方法对web业务pod进行监控，一旦出现问题马上重启，增强业务pod的可靠性。

  1. apiVersion: apps/v1 
  2. kind: Deployment
  3. metadata:
  4.   name: nginx
  5.   labels:
  6.     app: nginx
  7. spec:
  8.   replicas: 2
  9.   selector:
  10.     matchLabels:
  11.       app: nginx  
  12.   template:
  13.     metadata:
  14.       labels:
  15.         app: nginx
  16.     spec:
  17.       containers:
  18.       - name: nginx
  19.         image: nginx:latest # replace it with your exactly <image_name:tags>
  20.         ports:
  21.         - containerPort: 80
  22.         resources:
  23.           requests:                         ##必须设置，不然HPA无法运行。
  24.             cpu: 500m
  25.       livenessProbe:
  26.           tcpSocket:
  27.             port: 80
  28.           initialDelaySeconds: 15
  29.           periodSeconds: 20

9.对整个k8s集群和相关的服务器进行压力测试

使用ab压力测试

  1. 安装ab软件
  2. [root@lb conf]# yum install httpd-tools -y
  3.  
  4. -n：表示请求数
  5.  
  6. -c：表示并发数
  7.  
  8. [root@lb conf]# ab -c 100 -n 1000 http://192.168.159.135/
  9. 并发数100，请求数1000访问负载均衡器
  10. 100人访问，访问1000次
  11. Requests per second:    3206.62 [#/sec] (mean)
  12. Time per request:       31.185 [ms] (mean)
  13. Time per request:       0.312 [ms] (mean, across all concurrent requests)

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/AllinToyou/article/detail/578879
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号