devbox
菜鸟追梦旅行
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

二进制安装Kubernetes（k8s） v1.26.0 IPv4/IPv6双栈

作者：菜鸟追梦旅行 | 2024-04-15 17:21:39

赞

踩

vjkttlv

二进制安装Kubernetes（k8s） v1.26.0 IPv4/IPv6双栈

https://github.com/cby-chen/Kubernetes 开源不易，帮忙点个star，谢谢了

介绍

kubernetes（k8s）二进制高可用安装部署，支持IPv4+IPv6双栈。

我使用IPV6的目的是在公网进行访问，所以我配置了IPV6静态地址。

若您没有IPV6环境，或者不想使用IPv6，不对主机进行配置IPv6地址即可。

不配置IPV6，不影响后续，不过集群依旧是支持IPv6的。为后期留有扩展可能性。

若不要IPv6 ，不给网卡配置IPv6即可，不要对IPv6相关配置删除或操作，否则会出问题。

强烈建议在Github上查看文档 ！！！！！！

Github出问题会更新文档，并且后续尽可能第一时间更新新版本文档 ！！！

手动项目地址：https://github.com/cby-chen/Kubernetes

1.环境

主机名称IP地址说明软件
Master01192.168.1.61master节点kube-apiserver、kube-controller-manager、kube-scheduler、etcd、
kubelet、kube-proxy、nfs-client、haproxy、keepalived、nginx
Master02192.168.1.62master节点kube-apiserver、kube-controller-manager、kube-scheduler、etcd、
kubelet、kube-proxy、nfs-client、haproxy、keepalived、nginx
Master03192.168.1.63master节点kube-apiserver、kube-controller-manager、kube-scheduler、etcd、
kubelet、kube-proxy、nfs-client、haproxy、keepalived、nginx
Node01192.168.1.64node节点kubelet、kube-proxy、nfs-client、nginx
Node02192.168.1.65node节点kubelet、kube-proxy、nfs-client、nginx

192.168.8.66VIP
软件版本
kernel6.0.11
CentOS 8v8、 v7、Ubuntu
kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxyv1.25.4
etcdv3.5.6
containerdv1.6.10
dockerv20.10.21
cfsslv1.6.3
cniv1.1.1
crictlv1.26.0
haproxyv1.8.27
keepalivedv2.1.5

网段

物理主机：192.168.1.0/24

service：10.96.0.0/12

pod：172.16.0.0/12

安装包已经整理好：https://github.com/cby-chen/Kubernetes/releases/download/v1.26.0/kubernetes-v1.26.0.tar

1.1.k8s基础系统环境配置

1.2.配置IP 

  1. ssh root@192.168.1.143 "nmcli con mod eth0 ipv4.addresses 192.168.1.61/24; nmcli con mod eth0 ipv4.gateway 192.168.1.1; nmcli con mod eth0 ipv4.method manual; nmcli con mod eth0 ipv4.dns "8.8.8.8"; nmcli con up eth0"
  2. ssh root@192.168.1.144 "nmcli con mod eth0 ipv4.addresses 192.168.1.62/24; nmcli con mod eth0 ipv4.gateway 192.168.1.1; nmcli con mod eth0 ipv4.method manual; nmcli con mod eth0 ipv4.dns "8.8.8.8"; nmcli con up eth0"
  3. ssh root@192.168.1.145 "nmcli con mod eth0 ipv4.addresses 192.168.1.63/24; nmcli con mod eth0 ipv4.gateway 192.168.1.1; nmcli con mod eth0 ipv4.method manual; nmcli con mod eth0 ipv4.dns "8.8.8.8"; nmcli con up eth0"
  4. ssh root@192.168.1.146 "nmcli con mod eth0 ipv4.addresses 192.168.1.64/24; nmcli con mod eth0 ipv4.gateway 192.168.1.1; nmcli con mod eth0 ipv4.method manual; nmcli con mod eth0 ipv4.dns "8.8.8.8"; nmcli con up eth0"
  5. ssh root@192.168.1.148 "nmcli con mod eth0 ipv4.addresses 192.168.1.65/24; nmcli con mod eth0 ipv4.gateway 192.168.1.1; nmcli con mod eth0 ipv4.method manual; nmcli con mod eth0 ipv4.dns "8.8.8.8"; nmcli con up eth0"
  6.  
  7. # 没有IPv6选择不配置即可
  8. ssh root@192.168.1.61 "nmcli con mod eth0 ipv6.addresses fc00:43f4:1eea:1::10; nmcli con mod eth0 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod eth0 ipv6.method manual; nmcli con mod eth0 ipv6.dns "2400:3200::1"; nmcli con up eth0"
  9. ssh root@192.168.1.62 "nmcli con mod eth0 ipv6.addresses fc00:43f4:1eea:1::20; nmcli con mod eth0 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod eth0 ipv6.method manual; nmcli con mod eth0 ipv6.dns "2400:3200::1"; nmcli con up eth0"
  10. ssh root@192.168.1.63 "nmcli con mod eth0 ipv6.addresses fc00:43f4:1eea:1::30; nmcli con mod eth0 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod eth0 ipv6.method manual; nmcli con mod eth0 ipv6.dns "2400:3200::1"; nmcli con up eth0"
  11. ssh root@192.168.1.64 "nmcli con mod eth0 ipv6.addresses fc00:43f4:1eea:1::40; nmcli con mod eth0 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod eth0 ipv6.method manual; nmcli con mod eth0 ipv6.dns "2400:3200::1"; nmcli con up eth0"
  12. ssh root@192.168.1.65 "nmcli con mod eth0 ipv6.addresses fc00:43f4:1eea:1::50; nmcli con mod eth0 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod eth0 ipv6.method manual; nmcli con mod eth0 ipv6.dns "2400:3200::1"; nmcli con up eth0"
  13.  
  14. # 查看网卡配置
  15. [root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
  16. TYPE=Ethernet
  17. PROXY_METHOD=none
  18. BROWSER_ONLY=no
  19. BOOTPROTO=none
  20. DEFROUTE=yes
  21. IPV4_FAILURE_FATAL=no
  22. IPV6INIT=yes
  23. IPV6_AUTOCONF=no
  24. IPV6_DEFROUTE=yes
  25. IPV6_FAILURE_FATAL=no
  26. IPV6_ADDR_GEN_MODE=stable-privacy
  27. NAME=eth0
  28. UUID=424fd260-c480-4899-97e6-6fc9722031e8
  29. DEVICE=eth0
  30. ONBOOT=yes
  31. IPADDR=192.168.1.61
  32. PREFIX=24
  33. GATEWAY=192.168.8.1
  34. DNS1=8.8.8.8
  35. IPV6ADDR=fc00:43f4:1eea:1::10/128
  36. IPV6_DEFAULTGW=fc00:43f4:1eea:1::1
  37. DNS2=2400:3200::1
  38. [root@localhost ~]#

1.3.设置主机名 

  1. hostnamectl set-hostname k8s-master01
  2. hostnamectl set-hostname k8s-master02
  3. hostnamectl set-hostname k8s-master03
  4. hostnamectl set-hostname k8s-node01
  5. hostnamectl set-hostname k8s-node02

1.4.配置yum源 

  1. # 对于 Ubuntu
  2. sed -i 's/cn.archive.ubuntu.com/mirrors.ustc.edu.cn/g' /etc/apt/sources.list
  3.  
  4. # 对于 CentOS 7
  5. sudo sed -e 's|^mirrorlist=|#mirrorlist=|g' \
  6.          -e 's|^#baseurl=http://mirror.centos.org|baseurl=https://mirrors.tuna.tsinghua.edu.cn|g' \
  7.          -i.bak \
  8.          /etc/yum.repos.d/CentOS-*.repo
  9.  
  10. # 对于 CentOS 8
  11. sudo sed -e 's|^mirrorlist=|#mirrorlist=|g' \
  12.          -e 's|^#baseurl=http://mirror.centos.org/$contentdir|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos|g' \
  13.          -i.bak \
  14.          /etc/yum.repos.d/CentOS-*.repo
  15.  
  16. # 对于私有仓库
  17. sed -e 's|^mirrorlist=|#mirrorlist=|g' -e 's|^#baseurl=http://mirror.centos.org/\$contentdir|baseurl=http://192.168.1.123/centos|g' -i.bak  /etc/yum.repos.d/CentOS-*.repo

1.5.安装一些必备工具 

  1. # 对于 Ubuntu
  2. apt update && apt upgrade -y && apt install -y wget psmisc vim net-tools nfs-kernel-server telnet lvm2 git tar curl
  3.  
  4. # 对于 CentOS 7
  5. yum update -y && yum -y install  wget psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git tar curl
  6.  
  7. # 对于 CentOS 8
  8. yum update -y && yum -y install wget psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git network-scripts tar curl

1.6.选择性下载需要工具 

  1. 1.下载kubernetes1.26.+的二进制包
  2. github二进制包下载地址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md
  3.  
  4. wget https://dl.k8s.io/v1.26.0/kubernetes-server-linux-amd64.tar.gz
  5.  
  6. 2.下载etcdctl二进制包
  7. github二进制包下载地址：https://github.com/etcd-io/etcd/releases
  8.  
  9. wget https://ghproxy.com/https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz
  10.  
  11. 3.docker二进制包下载
  12. 二进制包下载地址：https://download.docker.com/linux/static/stable/x86_64/
  13.  
  14. wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.21.tgz
  15.  
  16. 4.下载cri-docker 
  17. 二进制包下载地址：https://github.com/Mirantis/cri-dockerd/releases/
  18.  
  19. wget  https://ghproxy.com/https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.6/cri-dockerd-0.2.6.amd64.tgz
  20.  
  21. 4.containerd下载时下载带cni插件的二进制包。
  22. github下载地址：https://github.com/containerd/containerd/releases
  23.  
  24. wget https://ghproxy.com/https://github.com/containerd/containerd/releases/download/v1.6.10/cri-containerd-cni-1.6.10-linux-amd64.tar.gz
  25.  
  26. 5.下载cfssl二进制包
  27. github二进制包下载地址：https://github.com/cloudflare/cfssl/releases
  28.  
  29. wget https://ghproxy.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssl_1.6.3_linux_amd64
  30. wget https://ghproxy.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssljson_1.6.3_linux_amd64
  31. wget https://ghproxy.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssl-certinfo_1.6.3_linux_amd64
  32.  
  33. 6.cni插件下载
  34. github下载地址：https://github.com/containernetworking/plugins/releases
  35.  
  36. wget https://ghproxy.com/https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
  37.  
  38. 7.crictl客户端二进制下载
  39. github下载：https://github.com/kubernetes-sigs/cri-tools/releases
  40.  
  41. wget https://ghproxy.com/https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.26.0/crictl-v1.26.0-linux-amd64.tar.gz

1.7.关闭防火墙 

  1. # Ubuntu忽略，CentOS执行
  2. systemctl disable --now firewalld

1.8.关闭SELinux 

  1. # Ubuntu忽略，CentOS执行
  2. setenforce 0
  3. sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

1.9.关闭交换分区 

  1. sed -ri 's/.*swap.*/#&/' /etc/fstab
  2. swapoff -a && sysctl -w vm.swappiness=0
  3.  
  4. cat /etc/fstab
  5. # /dev/mapper/centos-swap swap                    swap    defaults        0 0

1.10.网络配置（俩种方式二选一） 

  1. # Ubuntu忽略，CentOS执行
  2.  
  3. # 方式一
  4. # systemctl disable --now NetworkManager
  5. # systemctl start network && systemctl enable network
  6.  
  7. # 方式二
  8. cat > /etc/NetworkManager/conf.d/calico.conf << EOF 
  9. [keyfile]
  10. unmanaged-devices=interface-name:cali*;interface-name:tunl*
  11. EOF
  12. systemctl restart NetworkManager

1.11.进行时间同步 

  1. # 服务端
  2. # apt install chrony -y
  3. yum install chrony -y
  4. cat > /etc/chrony.conf << EOF 
  5. pool ntp.aliyun.com iburst
  6. driftfile /var/lib/chrony/drift
  7. makestep 1.0 3
  8. rtcsync
  9. allow 192.168.1.0/24
  10. local stratum 10
  11. keyfile /etc/chrony.keys
  12. leapsectz right/UTC
  13. logdir /var/log/chrony
  14. EOF
  15.  
  16. systemctl restart chronyd ; systemctl enable chronyd
  17.  
  18. # 客户端
  19. # apt install chrony -y
  20. yum install chrony -y
  21. cat > /etc/chrony.conf << EOF 
  22. pool 192.168.1.61 iburst
  23. driftfile /var/lib/chrony/drift
  24. makestep 1.0 3
  25. rtcsync
  26. keyfile /etc/chrony.keys
  27. leapsectz right/UTC
  28. logdir /var/log/chrony
  29. EOF
  30.  
  31. systemctl restart chronyd ; systemctl enable chronyd
  32.  
  33. #使用客户端进行验证
  34. chronyc sources -v

1.12.配置ulimit 

  1. ulimit -SHn 65535
  2. cat >> /etc/security/limits.conf <<EOF
  3. * soft nofile 655360
  4. * hard nofile 131072
  5. * soft nproc 655350
  6. * hard nproc 655350
  7. * seft memlock unlimited
  8. * hard memlock unlimitedd
  9. EOF

1.13.配置免密登录 

  1. # apt install -y sshpass
  2. yum install -y sshpass
  3. ssh-keygen -f /root/.ssh/id_rsa -P ''
  4. export IP="192.168.1.61 192.168.1.62 192.168.1.63 192.168.1.64 192.168.1.65"
  5. export SSHPASS=123123
  6. for HOST in $IP;do
  7.      sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $HOST
  8. done

1.14.添加启用源 

  1. # Ubuntu忽略，CentOS执行
  2.  
  3. # 为 RHEL-8或 CentOS-8配置源
  4. yum install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm -y 
  5. sed -i "s@mirrorlist@#mirrorlist@g" /etc/yum.repos.d/elrepo.repo 
  6. sed -i "s@elrepo.org/linux@mirrors.tuna.tsinghua.edu.cn/elrepo@g" /etc/yum.repos.d/elrepo.repo 
  7.  
  8. # 为 RHEL-7 SL-7 或 CentOS-7 安装 ELRepo 
  9. yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm -y 
  10. sed -i "s@mirrorlist@#mirrorlist@g" /etc/yum.repos.d/elrepo.repo 
  11. sed -i "s@elrepo.org/linux@mirrors.tuna.tsinghua.edu.cn/elrepo@g" /etc/yum.repos.d/elrepo.repo 
  12.  
  13. # 查看可用安装包
  14. yum  --disablerepo="*"  --enablerepo="elrepo-kernel"  list  available

1.15.升级内核至4.18版本以上 

  1. # Ubuntu忽略，CentOS执行
  2.  
  3. # 安装最新的内核
  4. # 我这里选择的是稳定版kernel-ml   如需更新长期维护版本kernel-lt  
  5. yum -y --enablerepo=elrepo-kernel  install  kernel-ml
  6.  
  7. # 查看已安装那些内核
  8. rpm -qa | grep kernel
  9.  
  10. # 查看默认内核
  11. grubby --default-kernel
  12.  
  13. # 若不是最新的使用命令设置
  14. grubby --set-default $(ls /boot/vmlinuz-* | grep elrepo)
  15.  
  16. # 重启生效
  17. reboot
  18.  
  19. # v8 整合命令为：
  20. yum install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm -y ; sed -i "s@mirrorlist@#mirrorlist@g" /etc/yum.repos.d/elrepo.repo ; sed -i "s@elrepo.org/linux@mirrors.tuna.tsinghua.edu.cn/elrepo@g" /etc/yum.repos.d/elrepo.repo ; yum  --disablerepo="*"  --enablerepo="elrepo-kernel"  list  available -y ; yum  --enablerepo=elrepo-kernel  install  kernel-ml -y ; grubby --default-kernel ; reboot 
  21.  
  22. # v7 整合命令为：
  23. yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm -y ; sed -i "s@mirrorlist@#mirrorlist@g" /etc/yum.repos.d/elrepo.repo ; sed -i "s@elrepo.org/linux@mirrors.tuna.tsinghua.edu.cn/elrepo@g" /etc/yum.repos.d/elrepo.repo ; yum  --disablerepo="*"  --enablerepo="elrepo-kernel"  list  available -y ; yum  --enablerepo=elrepo-kernel  install  kernel-ml -y ; grubby --set-default $(ls /boot/vmlinuz-* | grep elrepo) ; grubby --default-kernel ; reboot

1.16.安装ipvsadm 

  1. # 对于 Ubuntu
  2. # apt install ipvsadm ipset sysstat conntrack -y
  3.  
  4. # 对于 CentOS
  5. yum install ipvsadm ipset sysstat conntrack libseccomp -y
  6. cat >> /etc/modules-load.d/ipvs.conf <<EOF 
  7. ip_vs
  8. ip_vs_rr
  9. ip_vs_wrr
  10. ip_vs_sh
  11. nf_conntrack
  12. ip_tables
  13. ip_set
  14. xt_set
  15. ipt_set
  16. ipt_rpfilter
  17. ipt_REJECT
  18. ipip
  19. EOF
  20.  
  21. systemctl restart systemd-modules-load.service
  22.  
  23. lsmod | grep -e ip_vs -e nf_conntrack
  24. ip_vs_sh               16384  0
  25. ip_vs_wrr              16384  0
  26. ip_vs_rr               16384  0
  27. ip_vs                 180224  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
  28. nf_conntrack          176128  1 ip_vs
  29. nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
  30. nf_defrag_ipv4         16384  1 nf_conntrack
  31. libcrc32c              16384  3 nf_conntrack,xfs,ip_vs

1.17.修改内核参数 

  1. cat <<EOF > /etc/sysctl.d/k8s.conf
  2. net.ipv4.ip_forward = 1
  3. net.bridge.bridge-nf-call-iptables = 1
  4. fs.may_detach_mounts = 1
  5. vm.overcommit_memory=1
  6. vm.panic_on_oom=0
  7. fs.inotify.max_user_watches=89100
  8. fs.file-max=52706963
  9. fs.nr_open=52706963
  10. net.netfilter.nf_conntrack_max=2310720
  11.  
  12. net.ipv4.tcp_keepalive_time = 600
  13. net.ipv4.tcp_keepalive_probes = 3
  14. net.ipv4.tcp_keepalive_intvl =15
  15. net.ipv4.tcp_max_tw_buckets = 36000
  16. net.ipv4.tcp_tw_reuse = 1
  17. net.ipv4.tcp_max_orphans = 327680
  18. net.ipv4.tcp_orphan_retries = 3
  19. net.ipv4.tcp_syncookies = 1
  20. net.ipv4.tcp_max_syn_backlog = 16384
  21. net.ipv4.ip_conntrack_max = 65536
  22. net.ipv4.tcp_max_syn_backlog = 16384
  23. net.ipv4.tcp_timestamps = 0
  24. net.core.somaxconn = 16384
  25.  
  26. net.ipv6.conf.all.disable_ipv6 = 0
  27. net.ipv6.conf.default.disable_ipv6 = 0
  28. net.ipv6.conf.lo.disable_ipv6 = 0
  29. net.ipv6.conf.all.forwarding = 1
  30. EOF
  31.  
  32. sysctl --system

1.18.所有节点配置hosts本地解析 

  1. cat > /etc/hosts <<EOF
  2. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  3. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  4.  
  5.  
  6. 192.168.1.61 k8s-master01
  7. 192.168.1.62 k8s-master02
  8. 192.168.1.63 k8s-master03
  9. 192.168.1.64 k8s-node01
  10. 192.168.1.65 k8s-node02
  11. 192.168.8.66 lb-vip
  12. EOF

2.k8s基本组件安装

注意 ：  2.1 和 2.2 二选其一即可

2.1.安装Containerd作为Runtime （推荐）

  1. # wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
  2.  
  3. cd kubernetes-v1.26.0/cby/
  4.  
  5. #创建cni插件所需目录
  6. mkdir -p /etc/cni/net.d /opt/cni/bin 
  7. #解压cni二进制包
  8. tar xf cni-plugins-linux-amd64-v*.tgz -C /opt/cni/bin/
  9.  
  10. # wget https://github.com/containerd/containerd/releases/download/v1.6.8/cri-containerd-cni-1.6.8-linux-amd64.tar.gz
  11.  
  12. #解压
  13. tar -xzf cri-containerd-cni-*-linux-amd64.tar.gz -C /
  14.  
  15. #创建服务启动文件
  16. cat > /etc/systemd/system/containerd.service <<EOF
  17. [Unit]
  18. Description=containerd container runtime
  19. Documentation=https://containerd.io
  20. After=network.target local-fs.target
  21.  
  22. [Service]
  23. ExecStartPre=-/sbin/modprobe overlay
  24. ExecStart=/usr/local/bin/containerd
  25. Type=notify
  26. Delegate=yes
  27. KillMode=process
  28. Restart=always
  29. RestartSec=5
  30. LimitNPROC=infinity
  31. LimitCORE=infinity
  32. LimitNOFILE=infinity
  33. TasksMax=infinity
  34. OOMScoreAdjust=-999
  35.  
  36. [Install]
  37. WantedBy=multi-user.target
  38. EOF

2.1.1配置Containerd所需的模块 

  1. cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
  2. overlay
  3. br_netfilter
  4. EOF

2.1.2加载模块 

systemctl restart systemd-modules-load.service

2.1.3配置Containerd所需的内核 

  1. cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
  2. net.bridge.bridge-nf-call-iptables  = 1
  3. net.ipv4.ip_forward                 = 1
  4. net.bridge.bridge-nf-call-ip6tables = 1
  5. EOF
  6.  
  7. # 加载内核
  8.  
  9. sysctl --system

2.1.4创建Containerd的配置文件 

  1. # 创建默认配置文件
  2. mkdir -p /etc/containerd
  3. containerd config default | tee /etc/containerd/config.toml
  4.  
  5. # 修改Containerd的配置文件
  6. sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
  7. cat /etc/containerd/config.toml | grep SystemdCgroup
  8.  
  9. sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml
  10. cat /etc/containerd/config.toml | grep sandbox_image
  11.  
  12. sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml
  13. cat /etc/containerd/config.toml | grep certs.d
  14.  
  15. mkdir /etc/containerd/certs.d/docker.io -pv
  16.  
  17. cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
  18. server = "https://docker.io"
  19. [host."https://hub-mirror.c.163.com"]
  20.   capabilities = ["pull", "resolve"]
  21. EOF

2.1.5启动并设置为开机启动 

  1. systemctl daemon-reload
  2. systemctl enable --now containerd
  3. systemctl restart containerd

2.1.6配置crictl客户端连接的运行时位置 

  1. # wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.24.2/crictl-v1.24.2-linux-amd64.tar.gz
  2.  
  3. #解压
  4. tar xf crictl-v*-linux-amd64.tar.gz -C /usr/bin/
  5. #生成配置文件
  6. cat > /etc/crictl.yaml <<EOF
  7. runtime-endpoint: unix:///run/containerd/containerd.sock
  8. image-endpoint: unix:///run/containerd/containerd.sock
  9. timeout: 10
  10. debug: false
  11. EOF
  12.  
  13. #测试
  14. systemctl restart  containerd
  15. crictl info

2.2 安装docker作为Runtime （暂不支持）

v1.26.0 暂时不支持docker方式

2.2.1 安装docker 

  1. # 二进制包下载地址：https://download.docker.com/linux/static/stable/x86_64/
  2. # wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.21.tgz
  3.  
  4. #解压
  5. tar xf docker-*.tgz 
  6. #拷贝二进制文件
  7. cp docker/* /usr/bin/
  8. #创建containerd的service文件,并且启动
  9. cat >/etc/systemd/system/containerd.service <<EOF
  10. [Unit]
  11. Description=containerd container runtime
  12. Documentation=https://containerd.io
  13. After=network.target local-fs.target
  16. [Service]
  17. ExecStartPre=-/sbin/modprobe overlay
  18. ExecStart=/usr/bin/containerd
  19. Type=notify
  20. Delegate=yes
  21. KillMode=process
  22. Restart=always
  23. RestartSec=5
  24. LimitNPROC=infinity
  25. LimitCORE=infinity
  26. LimitNOFILE=1048576
  27. TasksMax=infinity
  28. OOMScoreAdjust=-999
  31. [Install]
  32. WantedBy=multi-user.target
  33. EOF
  36. systemctl enable --now containerd.service
  38. #准备docker的service文件
  39. cat > /etc/systemd/system/docker.service <<EOF
  40. [Unit]
  41. Description=Docker Application Container Engine
  42. Documentation=https://docs.docker.com
  43. After=network-online.target firewalld.service containerd.service
  44. Wants=network-online.target
  45. Requires=docker.socket containerd.service
  48. [Service]
  49. Type=notify
  50. ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
  51. ExecReload=/bin/kill -s HUP $MAINPID
  52. TimeoutSec=0
  53. RestartSec=2
  54. Restart=always
  55. StartLimitBurst=3
  56. StartLimitInterval=60s
  57. LimitNOFILE=infinity
  58. LimitNPROC=infinity
  59. LimitCORE=infinity
  60. TasksMax=infinity
  61. Delegate=yes
  62. KillMode=process
  63. OOMScoreAdjust=-500
  66. [Install]
  67. WantedBy=multi-user.target
  68. EOF
  69. #准备docker的socket文件
  70. cat > /etc/systemd/system/docker.socket <<EOF
  71. [Unit]
  72. Description=Docker Socket for the API
  75. [Socket]
  76. ListenStream=/var/run/docker.sock
  77. SocketMode=0660
  78. SocketUser=root
  79. SocketGroup=docker
  82. [Install]
  83. WantedBy=sockets.target
  84. EOF
  86. #创建docker组
  87. groupadd docker
  88. #启动docker
  89. systemctl enable --now docker.socket  && systemctl enable --now docker.service
  90. #验证
  91. docker info
  94. cat >/etc/docker/daemon.json <<EOF
  95. {
  96.   "exec-opts": ["native.cgroupdriver=systemd"],
  97.   "registry-mirrors": [
  98.     "https://docker.mirrors.ustc.edu.cn",
  99.     "http://hub-mirror.c.163.com"
  100.   ],
  101.   "max-concurrent-downloads": 10,
  102.   "log-driver": "json-file",
  103.   "log-level": "warn",
  104.   "log-opts": {
  105.     "max-size": "10m",
  106.     "max-file": "3"
  107.     },
  108.   "data-root": "/var/lib/docker"
  109. }
  110. EOF
  111. systemctl restart docker

2.2.2 安装cri-docker 

  1. # 由于1.24以及更高版本不支持docker所以安装cri-docker
  2. # 下载cri-docker 
  3. # wget  https://ghproxy.com/https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.5/cri-dockerd-0.2.5.amd64.tgz
  4.  
  5. # 解压cri-docker
  6. tar xvf cri-dockerd-*.amd64.tgz 
  7. cp cri-dockerd/cri-dockerd  /usr/bin/
  8.  
  9. # 写入启动配置文件
  10. cat >  /usr/lib/systemd/system/cri-docker.service <<EOF
  11. [Unit]
  12. Description=CRI Interface for Docker Application Container Engine
  13. Documentation=https://docs.mirantis.com
  14. After=network-online.target firewalld.service docker.service
  15. Wants=network-online.target
  16. Requires=cri-docker.socket
  17.  
  18. [Service]
  19. Type=notify
  20. ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
  21. ExecReload=/bin/kill -s HUP $MAINPID
  22. TimeoutSec=0
  23. RestartSec=2
  24. Restart=always
  25.  
  26. StartLimitBurst=3
  27.  
  28. StartLimitInterval=60s
  29.  
  30. LimitNOFILE=infinity
  31. LimitNPROC=infinity
  32. LimitCORE=infinity
  33.  
  34. TasksMax=infinity
  35. Delegate=yes
  36. KillMode=process
  37.  
  38. [Install]
  39. WantedBy=multi-user.target
  40. EOF
  41.  
  42. # 写入socket配置文件
  43. cat > /usr/lib/systemd/system/cri-docker.socket <<EOF
  44. [Unit]
  45. Description=CRI Docker Socket for the API
  46. PartOf=cri-docker.service
  47.  
  48. [Socket]
  49. ListenStream=%t/cri-dockerd.sock
  50. SocketMode=0660
  51. SocketUser=root
  52. SocketGroup=docker
  53.  
  54. [Install]
  55. WantedBy=sockets.target
  56. EOF
  57.  
  58. # 进行启动cri-docker
  59. systemctl daemon-reload ; systemctl enable cri-docker --now

2.3.k8s与etcd下载及安装（仅在master01操作）

2.3.1解压k8s安装包 

  1. # 下载安装包
  2. # wget https://dl.k8s.io/v1.25.4/kubernetes-server-linux-amd64.tar.gz
  3. # wget https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz
  4.  
  5. # 解压k8s安装文件
  6. cd cby
  7. tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
  8.  
  9. # 解压etcd安装文件
  10. tar -xf etcd*.tar.gz && mv etcd-*/etcd /usr/local/bin/ && mv etcd-*/etcdctl /usr/local/bin/
  11.  
  12. # 查看/usr/local/bin下内容
  13.  
  14. ls /usr/local/bin/
  15. containerd               crictl       etcdctl                  kube-proxy
  16. containerd-shim          critest      kube-apiserver           kube-scheduler
  17. containerd-shim-runc-v1  ctd-decoder  kube-controller-manager
  18. containerd-shim-runc-v2  ctr          kubectl
  19. containerd-stress        etcd         kubelet

2.3.2查看版本 

  1. [root@k8s-master01 ~]#  kubelet --version
  2. Kubernetes v1.26.0
  3. [root@k8s-master01 ~]# etcdctl version
  4. etcdctl version: 3.5.6
  5. API version: 3.5
  6. [root@k8s-master01 ~]#

2.3.3将组件发送至其他k8s节点 

  1. Master='k8s-master02 k8s-master03'
  2. Work='k8s-node01 k8s-node02'
  3.  
  4. for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
  5.  
  6. for NODE in $Work; do     scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
  7.  
  8. mkdir -p /opt/cni/bin

2.3创建证书相关文件

  1. mkdir pki
  2. cd pki
  3. cat > admin-csr.json << EOF 
  4. {
  5.   "CN": "admin",
  6.   "key": {
  7.     "algo": "rsa",
  8.     "size": 2048
  9.   },
  10.   "names": [
  11.     {
  12.       "C": "CN",
  13.       "ST": "Beijing",
  14.       "L": "Beijing",
  15.       "O": "system:masters",
  16.       "OU": "Kubernetes-manual"
  17.     }
  18.   ]
  19. }
  20. EOF
  21.  
  22. cat > ca-config.json << EOF 
  23. {
  24.   "signing": {
  25.     "default": {
  26.       "expiry": "876000h"
  27.     },
  28.     "profiles": {
  29.       "kubernetes": {
  30.         "usages": [
  31.             "signing",
  32.             "key encipherment",
  33.             "server auth",
  34.             "client auth"
  35.         ],
  36.         "expiry": "876000h"
  37.       }
  38.     }
  39.   }
  40. }
  41. EOF
  42.  
  43. cat > etcd-ca-csr.json  << EOF 
  44. {
  45.   "CN": "etcd",
  46.   "key": {
  47.     "algo": "rsa",
  48.     "size": 2048
  49.   },
  50.   "names": [
  51.     {
  52.       "C": "CN",
  53.       "ST": "Beijing",
  54.       "L": "Beijing",
  55.       "O": "etcd",
  56.       "OU": "Etcd Security"
  57.     }
  58.   ],
  59.   "ca": {
  60.     "expiry": "876000h"
  61.   }
  62. }
  63. EOF
  64.  
  65. cat > front-proxy-ca-csr.json  << EOF 
  66. {
  67.   "CN": "kubernetes",
  68.   "key": {
  69.      "algo": "rsa",
  70.      "size": 2048
  71.   },
  72.   "ca": {
  73.     "expiry": "876000h"
  74.   }
  75. }
  76. EOF
  77.  
  78. cat > kubelet-csr.json  << EOF 
  79. {
  80.   "CN": "system:node:\$NODE",
  81.   "key": {
  82.     "algo": "rsa",
  83.     "size": 2048
  84.   },
  85.   "names": [
  86.     {
  87.       "C": "CN",
  88.       "L": "Beijing",
  89.       "ST": "Beijing",
  90.       "O": "system:nodes",
  91.       "OU": "Kubernetes-manual"
  92.     }
  93.   ]
  94. }
  95. EOF
  96.  
  97. cat > manager-csr.json << EOF 
  98. {
  99.   "CN": "system:kube-controller-manager",
  100.   "key": {
  101.     "algo": "rsa",
  102.     "size": 2048
  103.   },
  104.   "names": [
  105.     {
  106.       "C": "CN",
  107.       "ST": "Beijing",
  108.       "L": "Beijing",
  109.       "O": "system:kube-controller-manager",
  110.       "OU": "Kubernetes-manual"
  111.     }
  112.   ]
  113. }
  114. EOF
  115.  
  116. cat > apiserver-csr.json << EOF 
  117. {
  118.   "CN": "kube-apiserver",
  119.   "key": {
  120.     "algo": "rsa",
  121.     "size": 2048
  122.   },
  123.   "names": [
  124.     {
  125.       "C": "CN",
  126.       "ST": "Beijing",
  127.       "L": "Beijing",
  128.       "O": "Kubernetes",
  129.       "OU": "Kubernetes-manual"
  130.     }
  131.   ]
  132. }
  133. EOF
  134.  
  135.  
  136. cat > ca-csr.json   << EOF 
  137. {
  138.   "CN": "kubernetes",
  139.   "key": {
  140.     "algo": "rsa",
  141.     "size": 2048
  142.   },
  143.   "names": [
  144.     {
  145.       "C": "CN",
  146.       "ST": "Beijing",
  147.       "L": "Beijing",
  148.       "O": "Kubernetes",
  149.       "OU": "Kubernetes-manual"
  150.     }
  151.   ],
  152.   "ca": {
  153.     "expiry": "876000h"
  154.   }
  155. }
  156. EOF
  157.  
  158. cat > etcd-csr.json << EOF 
  159. {
  160.   "CN": "etcd",
  161.   "key": {
  162.     "algo": "rsa",
  163.     "size": 2048
  164.   },
  165.   "names": [
  166.     {
  167.       "C": "CN",
  168.       "ST": "Beijing",
  169.       "L": "Beijing",
  170.       "O": "etcd",
  171.       "OU": "Etcd Security"
  172.     }
  173.   ]
  174. }
  175. EOF
  176.  
  177.  
  178. cat > front-proxy-client-csr.json  << EOF 
  179. {
  180.   "CN": "front-proxy-client",
  181.   "key": {
  182.      "algo": "rsa",
  183.      "size": 2048
  184.   }
  185. }
  186. EOF
  187.  
  188.  
  189. cat > kube-proxy-csr.json  << EOF 
  190. {
  191.   "CN": "system:kube-proxy",
  192.   "key": {
  193.     "algo": "rsa",
  194.     "size": 2048
  195.   },
  196.   "names": [
  197.     {
  198.       "C": "CN",
  199.       "ST": "Beijing",
  200.       "L": "Beijing",
  201.       "O": "system:kube-proxy",
  202.       "OU": "Kubernetes-manual"
  203.     }
  204.   ]
  205. }
  206. EOF
  207.  
  208.  
  209. cat > scheduler-csr.json << EOF 
  210. {
  211.   "CN": "system:kube-scheduler",
  212.   "key": {
  213.     "algo": "rsa",
  214.     "size": 2048
  215.   },
  216.   "names": [
  217.     {
  218.       "C": "CN",
  219.       "ST": "Beijing",
  220.       "L": "Beijing",
  221.       "O": "system:kube-scheduler",
  222.       "OU": "Kubernetes-manual"
  223.     }
  224.   ]
  225. }
  226. EOF
  227.  
  228. cd ..
  229. mkdir bootstrap
  230. cd bootstrap
  231. cat > bootstrap.secret.yaml << EOF 
  232. apiVersion: v1
  233. kind: Secret
  234. metadata:
  235.   name: bootstrap-token-c8ad9c
  236.   namespace: kube-system
  237. type: bootstrap.kubernetes.io/token
  238. stringData:
  239.   description: "The default bootstrap token generated by 'kubelet '."
  240.   token-id: c8ad9c
  241.   token-secret: 2e4d610cf3e7426e
  242.   usage-bootstrap-authentication: "true"
  243.   usage-bootstrap-signing: "true"
  244.   auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
  245.  
  246. ---
  247. apiVersion: rbac.authorization.k8s.io/v1
  248. kind: ClusterRoleBinding
  249. metadata:
  250.   name: kubelet-bootstrap
  251. roleRef:
  252.   apiGroup: rbac.authorization.k8s.io
  253.   kind: ClusterRole
  254.   name: system:node-bootstrapper
  255. subjects:
  256. - apiGroup: rbac.authorization.k8s.io
  257.   kind: Group
  258.   name: system:bootstrappers:default-node-token
  259. ---
  260. apiVersion: rbac.authorization.k8s.io/v1
  261. kind: ClusterRoleBinding
  262. metadata:
  263.   name: node-autoapprove-bootstrap
  264. roleRef:
  265.   apiGroup: rbac.authorization.k8s.io
  266.   kind: ClusterRole
  267.   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  268. subjects:
  269. - apiGroup: rbac.authorization.k8s.io
  270.   kind: Group
  271.   name: system:bootstrappers:default-node-token
  272. ---
  273. apiVersion: rbac.authorization.k8s.io/v1
  274. kind: ClusterRoleBinding
  275. metadata:
  276.   name: node-autoapprove-certificate-rotation
  277. roleRef:
  278.   apiGroup: rbac.authorization.k8s.io
  279.   kind: ClusterRole
  280.   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  281. subjects:
  282. - apiGroup: rbac.authorization.k8s.io
  283.   kind: Group
  284.   name: system:nodes
  285. ---
  286. apiVersion: rbac.authorization.k8s.io/v1
  287. kind: ClusterRole
  288. metadata:
  289.   annotations:
  290.     rbac.authorization.kubernetes.io/autoupdate: "true"
  291.   labels:
  292.     kubernetes.io/bootstrapping: rbac-defaults
  293.   name: system:kube-apiserver-to-kubelet
  294. rules:
  295.   - apiGroups:
  296.       - ""
  297.     resources:
  298.       - nodes/proxy
  299.       - nodes/stats
  300.       - nodes/log
  301.       - nodes/spec
  302.       - nodes/metrics
  303.     verbs:
  304.       - "*"
  305. ---
  306. apiVersion: rbac.authorization.k8s.io/v1
  307. kind: ClusterRoleBinding
  308. metadata:
  309.   name: system:kube-apiserver
  310.   namespace: ""
  311. roleRef:
  312.   apiGroup: rbac.authorization.k8s.io
  313.   kind: ClusterRole
  314.   name: system:kube-apiserver-to-kubelet
  315. subjects:
  316.   - apiGroup: rbac.authorization.k8s.io
  317.     kind: User
  318.     name: kube-apiserver
  319. EOF
  320.  
  321.  
  322. cd ..
  323. mkdir coredns
  324. cd coredns
  325. cat > coredns.yaml << EOF 
  326. apiVersion: v1
  327. kind: ServiceAccount
  328. metadata:
  329.   name: coredns
  330.   namespace: kube-system
  331. ---
  332. apiVersion: rbac.authorization.k8s.io/v1
  333. kind: ClusterRole
  334. metadata:
  335.   labels:
  336.     kubernetes.io/bootstrapping: rbac-defaults
  337.   name: system:coredns
  338. rules:
  339.   - apiGroups:
  340.     - ""
  341.     resources:
  342.     - endpoints
  343.     - services
  344.     - pods
  345.     - namespaces
  346.     verbs:
  347.     - list
  348.     - watch
  349.   - apiGroups:
  350.     - discovery.k8s.io
  351.     resources:
  352.     - endpointslices
  353.     verbs:
  354.     - list
  355.     - watch
  356. ---
  357. apiVersion: rbac.authorization.k8s.io/v1
  358. kind: ClusterRoleBinding
  359. metadata:
  360.   annotations:
  361.     rbac.authorization.kubernetes.io/autoupdate: "true"
  362.   labels:
  363.     kubernetes.io/bootstrapping: rbac-defaults
  364.   name: system:coredns
  365. roleRef:
  366.   apiGroup: rbac.authorization.k8s.io
  367.   kind: ClusterRole
  368.   name: system:coredns
  369. subjects:
  370. - kind: ServiceAccount
  371.   name: coredns
  372.   namespace: kube-system
  373. ---
  374. apiVersion: v1
  375. kind: ConfigMap
  376. metadata:
  377.   name: coredns
  378.   namespace: kube-system
  379. data:
  380.   Corefile: |
  381.     .:53 {
  382.         errors
  383.         health {
  384.           lameduck 5s
  385.         }
  386.         ready
  387.         kubernetes cluster.local in-addr.arpa ip6.arpa {
  388.           fallthrough in-addr.arpa ip6.arpa
  389.         }
  390.         prometheus :9153
  391.         forward . /etc/resolv.conf {
  392.           max_concurrent 1000
  393.         }
  394.         cache 30
  395.         loop
  396.         reload
  397.         loadbalance
  398.     }
  399. ---
  400. apiVersion: apps/v1
  401. kind: Deployment
  402. metadata:
  403.   name: coredns
  404.   namespace: kube-system
  405.   labels:
  406.     k8s-app: kube-dns
  407.     kubernetes.io/name: "CoreDNS"
  408. spec:
  409.   # replicas: not specified here:
  410.   # 1. Default is 1.
  411.   # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  412.   strategy:
  413.     type: RollingUpdate
  414.     rollingUpdate:
  415.       maxUnavailable: 1
  416.   selector:
  417.     matchLabels:
  418.       k8s-app: kube-dns
  419.   template:
  420.     metadata:
  421.       labels:
  422.         k8s-app: kube-dns
  423.     spec:
  424.       priorityClassName: system-cluster-critical
  425.       serviceAccountName: coredns
  426.       tolerations:
  427.         - key: "CriticalAddonsOnly"
  428.           operator: "Exists"
  429.       nodeSelector:
  430.         kubernetes.io/os: linux
  431.       affinity:
  432.          podAntiAffinity:
  433.            preferredDuringSchedulingIgnoredDuringExecution:
  434.            - weight: 100
  435.              podAffinityTerm:
  436.                labelSelector:
  437.                  matchExpressions:
  438.                    - key: k8s-app
  439.                      operator: In
  440.                      values: ["kube-dns"]
  441.                topologyKey: kubernetes.io/hostname
  442.       containers:
  443.       - name: coredns
  444.         image: registry.cn-beijing.aliyuncs.com/dotbalo/coredns:1.8.6 
  445.         imagePullPolicy: IfNotPresent
  446.         resources:
  447.           limits:
  448.             memory: 170Mi
  449.           requests:
  450.             cpu: 100m
  451.             memory: 70Mi
  452.         args: [ "-conf", "/etc/coredns/Corefile" ]
  453.         volumeMounts:
  454.         - name: config-volume
  455.           mountPath: /etc/coredns
  456.           readOnly: true
  457.         ports:
  458.         - containerPort: 53
  459.           name: dns
  460.           protocol: UDP
  461.         - containerPort: 53
  462.           name: dns-tcp
  463.           protocol: TCP
  464.         - containerPort: 9153
  465.           name: metrics
  466.           protocol: TCP
  467.         securityContext:
  468.           allowPrivilegeEscalation: false
  469.           capabilities:
  470.             add:
  471.             - NET_BIND_SERVICE
  472.             drop:
  473.             - all
  474.           readOnlyRootFilesystem: true
  475.         livenessProbe:
  476.           httpGet:
  477.             path: /health
  478.             port: 8080
  479.             scheme: HTTP
  480.           initialDelaySeconds: 60
  481.           timeoutSeconds: 5
  482.           successThreshold: 1
  483.           failureThreshold: 5
  484.         readinessProbe:
  485.           httpGet:
  486.             path: /ready
  487.             port: 8181
  488.             scheme: HTTP
  489.       dnsPolicy: Default
  490.       volumes:
  491.         - name: config-volume
  492.           configMap:
  493.             name: coredns
  494.             items:
  495.             - key: Corefile
  496.               path: Corefile
  497. ---
  498. apiVersion: v1
  499. kind: Service
  500. metadata:
  501.   name: kube-dns
  502.   namespace: kube-system
  503.   annotations:
  504.     prometheus.io/port: "9153"
  505.     prometheus.io/scrape: "true"
  506.   labels:
  507.     k8s-app: kube-dns
  508.     kubernetes.io/cluster-service: "true"
  509.     kubernetes.io/name: "CoreDNS"
  510. spec:
  511.   selector:
  512.     k8s-app: kube-dns
  513.   clusterIP: 10.96.0.10 
  514.   ports:
  515.   - name: dns
  516.     port: 53
  517.     protocol: UDP
  518.   - name: dns-tcp
  519.     port: 53
  520.     protocol: TCP
  521.   - name: metrics
  522.     port: 9153
  523.     protocol: TCP
  524. EOF
  525.  
  526.  
  527. cd ..
  528. mkdir metrics-server
  529. cd metrics-server
  530. cat > metrics-server.yaml << EOF 
  531. apiVersion: v1
  532. kind: ServiceAccount
  533. metadata:
  534.   labels:
  535.     k8s-app: metrics-server
  536.   name: metrics-server
  537.   namespace: kube-system
  538. ---
  539. apiVersion: rbac.authorization.k8s.io/v1
  540. kind: ClusterRole
  541. metadata:
  542.   labels:
  543.     k8s-app: metrics-server
  544.     rbac.authorization.k8s.io/aggregate-to-admin: "true"
  545.     rbac.authorization.k8s.io/aggregate-to-edit: "true"
  546.     rbac.authorization.k8s.io/aggregate-to-view: "true"
  547.   name: system:aggregated-metrics-reader
  548. rules:
  549. - apiGroups:
  550.   - metrics.k8s.io
  551.   resources:
  552.   - pods
  553.   - nodes
  554.   verbs:
  555.   - get
  556.   - list
  557.   - watch
  558. ---
  559. apiVersion: rbac.authorization.k8s.io/v1
  560. kind: ClusterRole
  561. metadata:
  562.   labels:
  563.     k8s-app: metrics-server
  564.   name: system:metrics-server
  565. rules:
  566. - apiGroups:
  567.   - ""
  568.   resources:
  569.   - pods
  570.   - nodes
  571.   - nodes/stats
  572.   - namespaces
  573.   - configmaps
  574.   verbs:
  575.   - get
  576.   - list
  577.   - watch
  578. ---
  579. apiVersion: rbac.authorization.k8s.io/v1
  580. kind: RoleBinding
  581. metadata:
  582.   labels:
  583.     k8s-app: metrics-server
  584.   name: metrics-server-auth-reader
  585.   namespace: kube-system
  586. roleRef:
  587.   apiGroup: rbac.authorization.k8s.io
  588.   kind: Role
  589.   name: extension-apiserver-authentication-reader
  590. subjects:
  591. - kind: ServiceAccount
  592.   name: metrics-server
  593.   namespace: kube-system
  594. ---
  595. apiVersion: rbac.authorization.k8s.io/v1
  596. kind: ClusterRoleBinding
  597. metadata:
  598.   labels:
  599.     k8s-app: metrics-server
  600.   name: metrics-server:system:auth-delegator
  601. roleRef:
  602.   apiGroup: rbac.authorization.k8s.io
  603.   kind: ClusterRole
  604.   name: system:auth-delegator
  605. subjects:
  606. - kind: ServiceAccount
  607.   name: metrics-server
  608.   namespace: kube-system
  609. ---
  610. apiVersion: rbac.authorization.k8s.io/v1
  611. kind: ClusterRoleBinding
  612. metadata:
  613.   labels:
  614.     k8s-app: metrics-server
  615.   name: system:metrics-server
  616. roleRef:
  617.   apiGroup: rbac.authorization.k8s.io
  618.   kind: ClusterRole
  619.   name: system:metrics-server
  620. subjects:
  621. - kind: ServiceAccount
  622.   name: metrics-server
  623.   namespace: kube-system
  624. ---
  625. apiVersion: v1
  626. kind: Service
  627. metadata:
  628.   labels:
  629.     k8s-app: metrics-server
  630.   name: metrics-server
  631.   namespace: kube-system
  632. spec:
  633.   ports:
  634.   - name: https
  635.     port: 443
  636.     protocol: TCP
  637.     targetPort: https
  638.   selector:
  639.     k8s-app: metrics-server
  640. ---
  641. apiVersion: apps/v1
  642. kind: Deployment
  643. metadata:
  644.   labels:
  645.     k8s-app: metrics-server
  646.   name: metrics-server
  647.   namespace: kube-system
  648. spec:
  649.   selector:
  650.     matchLabels:
  651.       k8s-app: metrics-server
  652.   strategy:
  653.     rollingUpdate:
  654.       maxUnavailable: 0
  655.   template:
  656.     metadata:
  657.       labels:
  658.         k8s-app: metrics-server
  659.     spec:
  660.       containers:
  661.       - args:
  662.         - --cert-dir=/tmp
  663.         - --secure-port=4443
  664.         - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  665.         - --kubelet-use-node-status-port
  666.         - --metric-resolution=15s
  667.         - --kubelet-insecure-tls
  668.         - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem # change to front-proxy-ca.crt for kubeadm
  669.         - --requestheader-username-headers=X-Remote-User
  670.         - --requestheader-group-headers=X-Remote-Group
  671.         - --requestheader-extra-headers-prefix=X-Remote-Extra-
  672.         image: registry.cn-beijing.aliyuncs.com/dotbalo/metrics-server:0.5.0
  673.         imagePullPolicy: IfNotPresent
  674.         livenessProbe:
  675.           failureThreshold: 3
  676.           httpGet:
  677.             path: /livez
  678.             port: https
  679.             scheme: HTTPS
  680.           periodSeconds: 10
  681.         name: metrics-server
  682.         ports:
  683.         - containerPort: 4443
  684.           name: https
  685.           protocol: TCP
  686.         readinessProbe:
  687.           failureThreshold: 3
  688.           httpGet:
  689.             path: /readyz
  690.             port: https
  691.             scheme: HTTPS
  692.           initialDelaySeconds: 20
  693.           periodSeconds: 10
  694.         resources:
  695.           requests:
  696.             cpu: 100m
  697.             memory: 200Mi
  698.         securityContext:
  699.           readOnlyRootFilesystem: true
  700.           runAsNonRoot: true
  701.           runAsUser: 1000
  702.         volumeMounts:
  703.         - mountPath: /tmp
  704.           name: tmp-dir
  705.         - name: ca-ssl
  706.           mountPath: /etc/kubernetes/pki
  707.       nodeSelector:
  708.         kubernetes.io/os: linux
  709.       priorityClassName: system-cluster-critical
  710.       serviceAccountName: metrics-server
  711.       volumes:
  712.       - emptyDir: {}
  713.         name: tmp-dir
  714.       - name: ca-ssl
  715.         hostPath:
  716.           path: /etc/kubernetes/pki
  717.  
  718. ---
  719. apiVersion: apiregistration.k8s.io/v1
  720. kind: APIService
  721. metadata:
  722.   labels:
  723.     k8s-app: metrics-server
  724.   name: v1beta1.metrics.k8s.io
  725. spec:
  726.   group: metrics.k8s.io
  727.   groupPriorityMinimum: 100
  728.   insecureSkipTLSVerify: true
  729.   service:
  730.     name: metrics-server
  731.     namespace: kube-system
  732.   version: v1beta1
  733.   versionPriority: 100
  734. EOF

3.相关证书生成

  1. # master01节点下载证书生成工具
  2. # wget "https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.2_linux_amd64" -O /usr/local/bin/cfssl
  3. # wget "https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.2_linux_amd64" -O /usr/local/bin/cfssljson
  4.  
  5. # 软件包内有
  6. cp cfssl_*_linux_amd64 /usr/local/bin/cfssl
  7. cp cfssljson_*_linux_amd64 /usr/local/bin/cfssljson
  8.  
  9. chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

3.1.生成etcd证书

特别说明除外，以下操作在所有master节点操作

3.1.1所有master节点创建证书存放目录 

mkdir /etc/etcd/ssl -p

3.1.2master01节点生成etcd证书 

  1. cd pki
  2. # 生成etcd证书和etcd证书的key（如果你觉得以后可能会扩容，可以在ip那多写几个预留出来）
  3. # 若没有IPv6 可删除可保留 
  4. cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca
  5. cfssl gencert \
  6.    -ca=/etc/etcd/ssl/etcd-ca.pem \
  7.    -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
  8.    -config=ca-config.json \
  9.    -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.1.61,192.168.1.62,192.168.1.63,fc00:43f4:1eea:1::10,fc00:43f4:1eea:1::20,fc00:43f4:1eea:1::30 \
  10.    -profile=kubernetes \
  11.    etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

3.1.3将证书复制到其他节点 

  1. Master='k8s-master02 k8s-master03'
  2. for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done

3.2.生成k8s相关证书

特别说明除外，以下操作在所有master节点操作

3.2.1所有k8s节点创建证书存放目录 

mkdir -p /etc/kubernetes/pki

3.2.2master01节点生成k8s证书 

  1. cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
  2.  
  3. # 生成一个根证书 ，多写了一些IP作为预留IP，为将来添加node做准备
  4. # 10.96.0.1是service网段的第一个地址，需要计算，192.168.8.66为高可用vip地址
  5. # 若没有IPv6 可删除可保留 
  6.  
  7. cfssl gencert   \
  8. -ca=/etc/kubernetes/pki/ca.pem   \
  9. -ca-key=/etc/kubernetes/pki/ca-key.pem   \
  10. -config=ca-config.json   \
  11. -hostname=10.96.0.1,192.168.8.66,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,x.oiox.cn,k.oiox.cn,l.oiox.cn,o.oiox.cn,192.168.1.61,192.168.1.62,192.168.1.63,192.168.1.64,192.168.1.65,192.168.8.66,192.168.1.67,192.168.1.68,192.168.1.69,192.168.1.70,fc00:43f4:1eea:1::10,fc00:43f4:1eea:1::20,fc00:43f4:1eea:1::30,fc00:43f4:1eea:1::40,fc00:43f4:1eea:1::50,fc00:43f4:1eea:1::60,fc00:43f4:1eea:1::70,fc00:43f4:1eea:1::80,fc00:43f4:1eea:1::90,fc00:43f4:1eea:1::100   \
  12. -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

3.2.3生成apiserver聚合证书 

  1. cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca 
  2.  
  3. # 有一个警告，可以忽略
  4.  
  5. cfssl gencert  \
  6. -ca=/etc/kubernetes/pki/front-proxy-ca.pem   \
  7. -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   \
  8. -config=ca-config.json   \
  9. -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

3.2.4生成controller-manage的证书

在《5.高可用配置》选择使用那种高可用方案
若使用 haproxy、keepalived 那么为 --server=https://192.168.8.66:8443
若使用 nginx方案，那么为 --server=https://127.0.0.1:8443

  1. cfssl gencert \
  2.    -ca=/etc/kubernetes/pki/ca.pem \
  3.    -ca-key=/etc/kubernetes/pki/ca-key.pem \
  4.    -config=ca-config.json \
  5.    -profile=kubernetes \
  6.    manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
  7.  
  8. # 设置一个集群项
  9.  
  10. # 在《5.高可用配置》选择使用那种高可用方案
  11. # 若使用 haproxy、keepalived 那么为 `--server=https://192.168.8.66:8443`
  12. # 若使用 nginx方案，那么为 `--server=https://127.0.0.1:8443`
  13.  
  14. kubectl config set-cluster kubernetes \
  15.      --certificate-authority=/etc/kubernetes/pki/ca.pem \
  16.      --embed-certs=true \
  17.      --server=https://127.0.0.1:8443 \
  18.      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
  19.  
  20. # 设置一个环境项，一个上下文
  21.  
  22. kubectl config set-context system:kube-controller-manager@kubernetes \
  23.     --cluster=kubernetes \
  24.     --user=system:kube-controller-manager \
  25.     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
  26.  
  27. # 设置一个用户项
  28.  
  29. kubectl config set-credentials system:kube-controller-manager \
  30.      --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
  31.      --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
  32.      --embed-certs=true \
  33.      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
  34.  
  35. # 设置默认环境
  36.  
  37. kubectl config use-context system:kube-controller-manager@kubernetes \
  38.      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
  39.  
  40. cfssl gencert \
  41.    -ca=/etc/kubernetes/pki/ca.pem \
  42.    -ca-key=/etc/kubernetes/pki/ca-key.pem \
  43.    -config=ca-config.json \
  44.    -profile=kubernetes \
  45.    scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
  46.  
  47. # 在《5.高可用配置》选择使用那种高可用方案
  48. # 若使用 haproxy、keepalived 那么为 `--server=https://192.168.8.66:8443`
  49. # 若使用 nginx方案，那么为 `--server=https://127.0.0.1:8443`
  50.  
  51. kubectl config set-cluster kubernetes \
  52.      --certificate-authority=/etc/kubernetes/pki/ca.pem \
  53.      --embed-certs=true \
  54.      --server=https://127.0.0.1:8443 \
  55.      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
  56.  
  57. kubectl config set-credentials system:kube-scheduler \
  58.      --client-certificate=/etc/kubernetes/pki/scheduler.pem \
  59.      --client-key=/etc/kubernetes/pki/scheduler-key.pem \
  60.      --embed-certs=true \
  61.      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
  62.  
  63. kubectl config set-context system:kube-scheduler@kubernetes \
  64.      --cluster=kubernetes \
  65.      --user=system:kube-scheduler \
  66.      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
  67.  
  68. kubectl config use-context system:kube-scheduler@kubernetes \
  69.      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
  70.  
  71. cfssl gencert \
  72.    -ca=/etc/kubernetes/pki/ca.pem \
  73.    -ca-key=/etc/kubernetes/pki/ca-key.pem \
  74.    -config=ca-config.json \
  75.    -profile=kubernetes \
  76.    admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
  77.  
  78. # 在《5.高可用配置》选择使用那种高可用方案
  79. # 若使用 haproxy、keepalived 那么为 `--server=https://192.168.8.66:8443`
  80. # 若使用 nginx方案，那么为 `--server=https://127.0.0.1:8443`
  81.  
  82. kubectl config set-cluster kubernetes     \
  83.   --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  84.   --embed-certs=true     \
  85.   --server=https://127.0.0.1:8443     \
  86.   --kubeconfig=/etc/kubernetes/admin.kubeconfig
  87.  
  88. kubectl config set-credentials kubernetes-admin  \
  89.   --client-certificate=/etc/kubernetes/pki/admin.pem     \
  90.   --client-key=/etc/kubernetes/pki/admin-key.pem     \
  91.   --embed-certs=true     \
  92.   --kubeconfig=/etc/kubernetes/admin.kubeconfig
  93.  
  94. kubectl config set-context kubernetes-admin@kubernetes    \
  95.   --cluster=kubernetes     \
  96.   --user=kubernetes-admin     \
  97.   --kubeconfig=/etc/kubernetes/admin.kubeconfig
  98.  
  99. kubectl config use-context kubernetes-admin@kubernetes  --kubeconfig=/etc/kubernetes/admin.kubeconfig

3.2.5创建kube-proxy证书

在《5.高可用配置》选择使用那种高可用方案
若使用 haproxy、keepalived 那么为 --server=https://192.168.8.66:8443
若使用 nginx方案，那么为 --server=https://127.0.0.1:8443

  1. cfssl gencert \
  2.    -ca=/etc/kubernetes/pki/ca.pem \
  3.    -ca-key=/etc/kubernetes/pki/ca-key.pem \
  4.    -config=ca-config.json \
  5.    -profile=kubernetes \
  6.    kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy
  7.  
  8. # 在《5.高可用配置》选择使用那种高可用方案
  9. # 若使用 haproxy、keepalived 那么为 `--server=https://192.168.8.66:8443`
  10. # 若使用 nginx方案，那么为 `--server=https://127.0.0.1:8443`
  11.  
  12. kubectl config set-cluster kubernetes     \
  13.   --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  14.   --embed-certs=true     \
  15.   --server=https://127.0.0.1:8443     \
  16.   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
  17.  
  18. kubectl config set-credentials kube-proxy  \
  19.   --client-certificate=/etc/kubernetes/pki/kube-proxy.pem     \
  20.   --client-key=/etc/kubernetes/pki/kube-proxy-key.pem     \
  21.   --embed-certs=true     \
  22.   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
  23.  
  24. kubectl config set-context kube-proxy@kubernetes    \
  25.   --cluster=kubernetes     \
  26.   --user=kube-proxy     \
  27.   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
  28.  
  29. kubectl config use-context kube-proxy@kubernetes  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

3.2.5创建ServiceAccount Key ——secret 

  1. openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
  2. openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

3.2.6将证书发送到其他master节点 

  1. #其他节点创建目录
  2. # mkdir  /etc/kubernetes/pki/ -p
  3.  
  4. for NODE in k8s-master02 k8s-master03; do  for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do  scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done;  for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do  scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done

3.2.7查看证书 

  1. ls /etc/kubernetes/pki/
  2. admin.csr          controller-manager.csr      kube-proxy.csr
  3. admin-key.pem      controller-manager-key.pem  kube-proxy-key.pem
  4. admin.pem          controller-manager.pem      kube-proxy.pem
  5. apiserver.csr      front-proxy-ca.csr          sa.key
  6. apiserver-key.pem  front-proxy-ca-key.pem      sa.pub
  7. apiserver.pem      front-proxy-ca.pem          scheduler.csr
  8. ca.csr             front-proxy-client.csr      scheduler-key.pem
  9. ca-key.pem         front-proxy-client-key.pem  scheduler.pem
  10. ca.pem             front-proxy-client.pem
  11.  
  12. # 一共26个就对了
  13. ls /etc/kubernetes/pki/ |wc -l
  14. 26

4.k8s系统组件配置

4.1.etcd配置

4.1.1master01配置 

  1. # 如果要用IPv6那么把IPv4地址修改为IPv6即可
  2. cat > /etc/etcd/etcd.config.yml << EOF 
  3. name: 'k8s-master01'
  4. data-dir: /var/lib/etcd
  5. wal-dir: /var/lib/etcd/wal
  6. snapshot-count: 5000
  7. heartbeat-interval: 100
  8. election-timeout: 1000
  9. quota-backend-bytes: 0
  10. listen-peer-urls: 'https://192.168.1.61:2380'
  11. listen-client-urls: 'https://192.168.1.61:2379,http://127.0.0.1:2379'
  12. max-snapshots: 3
  13. max-wals: 5
  14. cors:
  15. initial-advertise-peer-urls: 'https://192.168.1.61:2380'
  16. advertise-client-urls: 'https://192.168.1.61:2379'
  17. discovery:
  18. discovery-fallback: 'proxy'
  19. discovery-proxy:
  20. discovery-srv:
  21. initial-cluster: 'k8s-master01=https://192.168.1.61:2380,k8s-master02=https://192.168.1.62:2380,k8s-master03=https://192.168.1.63:2380'
  22. initial-cluster-token: 'etcd-k8s-cluster'
  23. initial-cluster-state: 'new'
  24. strict-reconfig-check: false
  25. enable-v2: true
  26. enable-pprof: true
  27. proxy: 'off'
  28. proxy-failure-wait: 5000
  29. proxy-refresh-interval: 30000
  30. proxy-dial-timeout: 1000
  31. proxy-write-timeout: 5000
  32. proxy-read-timeout: 0
  33. client-transport-security:
  34.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  35.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  36.   client-cert-auth: true
  37.   trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  38.   auto-tls: true
  39. peer-transport-security:
  40.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  41.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  42.   peer-client-cert-auth: true
  43.   trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  44.   auto-tls: true
  45. debug: false
  46. log-package-levels:
  47. log-outputs: [default]
  48. force-new-cluster: false
  49. EOF

4.1.2master02配置 

  1. # 如果要用IPv6那么把IPv4地址修改为IPv6即可
  2. cat > /etc/etcd/etcd.config.yml << EOF 
  3. name: 'k8s-master02'
  4. data-dir: /var/lib/etcd
  5. wal-dir: /var/lib/etcd/wal
  6. snapshot-count: 5000
  7. heartbeat-interval: 100
  8. election-timeout: 1000
  9. quota-backend-bytes: 0
  10. listen-peer-urls: 'https://192.168.1.62:2380'
  11. listen-client-urls: 'https://192.168.1.62:2379,http://127.0.0.1:2379'
  12. max-snapshots: 3
  13. max-wals: 5
  14. cors:
  15. initial-advertise-peer-urls: 'https://192.168.1.62:2380'
  16. advertise-client-urls: 'https://192.168.1.62:2379'
  17. discovery:
  18. discovery-fallback: 'proxy'
  19. discovery-proxy:
  20. discovery-srv:
  21. initial-cluster: 'k8s-master01=https://192.168.1.61:2380,k8s-master02=https://192.168.1.62:2380,k8s-master03=https://192.168.1.63:2380'
  22. initial-cluster-token: 'etcd-k8s-cluster'
  23. initial-cluster-state: 'new'
  24. strict-reconfig-check: false
  25. enable-v2: true
  26. enable-pprof: true
  27. proxy: 'off'
  28. proxy-failure-wait: 5000
  29. proxy-refresh-interval: 30000
  30. proxy-dial-timeout: 1000
  31. proxy-write-timeout: 5000
  32. proxy-read-timeout: 0
  33. client-transport-security:
  34.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  35.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  36.   client-cert-auth: true
  37.   trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  38.   auto-tls: true
  39. peer-transport-security:
  40.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  41.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  42.   peer-client-cert-auth: true
  43.   trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  44.   auto-tls: true
  45. debug: false
  46. log-package-levels:
  47. log-outputs: [default]
  48. force-new-cluster: false
  49. EOF

4.1.3master03配置 

  1. # 如果要用IPv6那么把IPv4地址修改为IPv6即可
  2. cat > /etc/etcd/etcd.config.yml << EOF 
  3. name: 'k8s-master03'
  4. data-dir: /var/lib/etcd
  5. wal-dir: /var/lib/etcd/wal
  6. snapshot-count: 5000
  7. heartbeat-interval: 100
  8. election-timeout: 1000
  9. quota-backend-bytes: 0
  10. listen-peer-urls: 'https://192.168.1.63:2380'
  11. listen-client-urls: 'https://192.168.1.63:2379,http://127.0.0.1:2379'
  12. max-snapshots: 3
  13. max-wals: 5
  14. cors:
  15. initial-advertise-peer-urls: 'https://192.168.1.63:2380'
  16. advertise-client-urls: 'https://192.168.1.63:2379'
  17. discovery:
  18. discovery-fallback: 'proxy'
  19. discovery-proxy:
  20. discovery-srv:
  21. initial-cluster: 'k8s-master01=https://192.168.1.61:2380,k8s-master02=https://192.168.1.62:2380,k8s-master03=https://192.168.1.63:2380'
  22. initial-cluster-token: 'etcd-k8s-cluster'
  23. initial-cluster-state: 'new'
  24. strict-reconfig-check: false
  25. enable-v2: true
  26. enable-pprof: true
  27. proxy: 'off'
  28. proxy-failure-wait: 5000
  29. proxy-refresh-interval: 30000
  30. proxy-dial-timeout: 1000
  31. proxy-write-timeout: 5000
  32. proxy-read-timeout: 0
  33. client-transport-security:
  34.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  35.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  36.   client-cert-auth: true
  37.   trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  38.   auto-tls: true
  39. peer-transport-security:
  40.   cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  41.   key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  42.   peer-client-cert-auth: true
  43.   trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  44.   auto-tls: true
  45. debug: false
  46. log-package-levels:
  47. log-outputs: [default]
  48. force-new-cluster: false
  49. EOF

4.2.创建service（所有master节点操作）

4.2.1创建etcd.service并启动 

  1. cat > /usr/lib/systemd/system/etcd.service << EOF
  2.  
  3. [Unit]
  4. Description=Etcd Service
  5. Documentation=https://coreos.com/etcd/docs/latest/
  6. After=network.target
  7.  
  8. [Service]
  9. Type=notify
  10. ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
  11. Restart=on-failure
  12. RestartSec=10
  13. LimitNOFILE=65536
  14.  
  15. [Install]
  16. WantedBy=multi-user.target
  17. Alias=etcd3.service
  18.  
  19. EOF

4.2.2创建etcd证书目录 

  1. mkdir /etc/kubernetes/pki/etcd
  2. ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
  3. systemctl daemon-reload
  4. systemctl enable --now etcd

4.2.3查看etcd状态 

  1. # 如果要用IPv6那么把IPv4地址修改为IPv6即可
  2. export ETCDCTL_API=3
  3. etcdctl --endpoints="192.168.1.63:2379,192.168.1.62:2379,192.168.1.61:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
  4. +----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  5. |    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
  6. +----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  7. | 192.168.1.63:2379 | c0c8142615b9523f |   3.5.6 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
  8. | 192.168.1.62:2379 | de8396604d2c160d |   3.5.6 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
  9. | 192.168.1.61:2379 | 33c9d6df0037ab97 |   3.5.6 |   20 kB |      true |      false |         2 |          9 |                  9 |        |
  10. +----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  11.  
  12. [root@k8s-master01 pki]#

5.高可用配置（在Master服务器上操作）

注意* 5.1.1 和5.1.2 二选一即可

选择使用那种高可用方案

在《3.2.生成k8s相关证书》

若使用 nginx方案，那么为 --server=https://127.0.0.1:8443
若使用 haproxy、keepalived 那么为 --server=https://192.168.8.66:8443

5.1 NGINX高可用方案 （推荐）

5.1.1自己手动编译

在所有节点执行

  1. # 安装编译环境
  2. yum install gcc -y
  3.  
  4. # 下载解压nginx二进制文件
  5. wget http://nginx.org/download/nginx-1.22.1.tar.gz
  6. tar xvf nginx-*.tar.gz
  7. cd nginx-*
  8.  
  9. # 进行编译
  10. ./configure --with-stream --without-http --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  11. make && make install

5.1.2使用我编译好的 

  1. # 使用我编译好的
  2.  
  3. cd kubernetes-v1.26.0/cby
  4. # 拷贝我编译好的nginx
  5. node='k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02'
  6. for NODE in $node; do scp nginx.tar $NODE:/usr/local/; done
  7.  
  8. # 其他节点上执行
  9. cd /usr/local/
  10. tar xvf nginx.tar

5.1.3写入启动配置

在所有主机上执行

  1. # 写入nginx配置文件
  2. cat > /usr/local/nginx/conf/kube-nginx.conf <<EOF
  3. worker_processes 1;
  4. events {
  5.     worker_connections  1024;
  6. }
  7. stream {
  8.     upstream backend {
  9.         least_conn;
  10.         hash $remote_addr consistent;
  11.         server 192.168.1.61:6443        max_fails=3 fail_timeout=30s;
  12.         server 192.168.1.62:6443        max_fails=3 fail_timeout=30s;
  13.         server 192.168.1.63:6443        max_fails=3 fail_timeout=30s;
  14.     }
  15.     server {
  16.         listen 127.0.0.1:8443;
  17.         proxy_connect_timeout 1s;
  18.         proxy_pass backend;
  19.     }
  20. }
  21. EOF
  22.  
  23. # 写入启动配置文件
  24. cat > /etc/systemd/system/kube-nginx.service <<EOF
  25. [Unit]
  26. Description=kube-apiserver nginx proxy
  27. After=network.target
  28. After=network-online.target
  29. Wants=network-online.target
  30.  
  31. [Service]
  32. Type=forking
  33. ExecStartPre=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -t
  34. ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx
  35. ExecReload=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -s reload
  36. PrivateTmp=true
  37. Restart=always
  38. RestartSec=5
  39. StartLimitInterval=0
  40. LimitNOFILE=65536
  41.  
  42. [Install]
  43. WantedBy=multi-user.target
  44. EOF
  45.  
  46. # 设置开机自启
  47. systemctl enable --now  kube-nginx 
  48. systemctl restart kube-nginx
  49. systemctl status kube-nginx

5.2 keepalived和haproxy 高可用方案 （不推荐）

5.2.1安装keepalived和haproxy服务 

  1. systemctl disable --now firewalld
  2.  
  3. setenforce 0
  4. sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
  5.  
  6. yum -y install keepalived haproxy

5.2.2修改haproxy配置文件（两台配置文件一样） 

  1. # cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
  2.  
  3. cat >/etc/haproxy/haproxy.cfg<<"EOF"
  4. global
  5.  maxconn 2000
  6.  ulimit-n 16384
  7.  log 127.0.0.1 local0 err
  8.  stats timeout 30s
  9.  
  10. defaults
  11.  log global
  12.  mode http
  13.  option httplog
  14.  timeout connect 5000
  15.  timeout client 50000
  16.  timeout server 50000
  17.  timeout http-request 15s
  18.  timeout http-keep-alive 15s
  19.  
  20.  
  21. frontend monitor-in
  22.  bind *:33305
  23.  mode http
  24.  option httplog
  25.  monitor-uri /monitor
  26.  
  27. frontend k8s-master
  28.  bind 0.0.0.0:8443
  29.  bind 127.0.0.1:8443
  30.  mode tcp
  31.  option tcplog
  32.  tcp-request inspect-delay 5s
  33.  default_backend k8s-master
  34.  
  35.  
  36. backend k8s-master
  37.  mode tcp
  38.  option tcplog
  39.  option tcp-check
  40.  balance roundrobin
  41.  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  42.  server  k8s-master01  192.168.1.61:6443 check
  43.  server  k8s-master02  192.168.1.62:6443 check
  44.  server  k8s-master03  192.168.1.63:6443 check
  45. EOF

5.2.3Master01配置keepalived master节点 

  1. #cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
  2.  
  3. cat > /etc/keepalived/keepalived.conf << EOF
  4. ! Configuration File for keepalived
  5.  
  6. global_defs {
  7.     router_id LVS_DEVEL
  8. }
  9. vrrp_script chk_apiserver {
  10.     script "/etc/keepalived/check_apiserver.sh"
  11.     interval 5 
  12.     weight -5
  13.     fall 2
  14.     rise 1
  15. }
  16. vrrp_instance VI_1 {
  17.     state MASTER
  18.     # 注意网卡名
  19.     interface eth0 
  20.     mcast_src_ip 192.168.1.61
  21.     virtual_router_id 51
  22.     priority 100
  23.     nopreempt
  24.     advert_int 2
  25.     authentication {
  26.         auth_type PASS
  27.         auth_pass K8SHA_KA_AUTH
  28.     }
  29.     virtual_ipaddress {
  30.         192.168.8.66
  31.     }
  32.     track_script {
  33.       chk_apiserver 
  34. } }
  35.  
  36. EOF

5.2.4Master02配置keepalived backup节点 

  1. # cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
  2.  
  3. cat > /etc/keepalived/keepalived.conf << EOF
  4. ! Configuration File for keepalived
  5.  
  6. global_defs {
  7.     router_id LVS_DEVEL
  8. }
  9. vrrp_script chk_apiserver {
  10.     script "/etc/keepalived/check_apiserver.sh"
  11.     interval 5 
  12.     weight -5
  13.     fall 2
  14.     rise 1
  15.  
  16. }
  17. vrrp_instance VI_1 {
  18.     state BACKUP
  19.     # 注意网卡名
  20.     interface eth0
  21.     mcast_src_ip 192.168.1.62
  22.     virtual_router_id 51
  23.     priority 80
  24.     nopreempt
  25.     advert_int 2
  26.     authentication {
  27.         auth_type PASS
  28.         auth_pass K8SHA_KA_AUTH
  29.     }
  30.     virtual_ipaddress {
  31.         192.168.8.66
  32.     }
  33.     track_script {
  34.       chk_apiserver 
  35. } }
  36.  
  37. EOF

5.2.5Master03配置keepalived backup节点 

  1. # cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
  2.  
  3. cat > /etc/keepalived/keepalived.conf << EOF
  4. ! Configuration File for keepalived
  5.  
  6. global_defs {
  7.     router_id LVS_DEVEL
  8. }
  9. vrrp_script chk_apiserver {
  10.     script "/etc/keepalived/check_apiserver.sh"
  11.     interval 5 
  12.     weight -5
  13.     fall 2
  14.     rise 1
  15.  
  16. }
  17. vrrp_instance VI_1 {
  18.     state BACKUP
  19.     # 注意网卡名
  20.     interface eth0
  21.     mcast_src_ip 192.168.1.63
  22.     virtual_router_id 51
  23.     priority 50
  24.     nopreempt
  25.     advert_int 2
  26.     authentication {
  27.         auth_type PASS
  28.         auth_pass K8SHA_KA_AUTH
  29.     }
  30.     virtual_ipaddress {
  31.         192.168.8.66
  32.     }
  33.     track_script {
  34.       chk_apiserver 
  35. } }
  36.  
  37. EOF

5.2.6健康检查脚本配置（两台lb主机） 

  1. cat >  /etc/keepalived/check_apiserver.sh << EOF
  2. #!/bin/bash
  3.  
  4. err=0
  5. for k in \$(seq 1 3)
  6. do
  7.     check_code=\$(pgrep haproxy)
  8.     if [[ \$check_code == "" ]]; then
  9.         err=\$(expr \$err + 1)
  10.         sleep 1
  11.         continue
  12.     else
  13.         err=0
  14.         break
  15.     fi
  16. done
  17.  
  18. if [[ \$err != "0" ]]; then
  19.     echo "systemctl stop keepalived"
  20.     /usr/bin/systemctl stop keepalived
  21.     exit 1
  22. else
  23.     exit 0
  24. fi
  25. EOF
  26.  
  27. # 给脚本授权
  28.  
  29. chmod +x /etc/keepalived/check_apiserver.sh

5.2.7启动服务 

  1. systemctl daemon-reload
  2. systemctl enable --now haproxy
  3. systemctl enable --now keepalived

5.2.8测试高可用 

  1. # 能ping同
  2.  
  3. [root@k8s-node02 ~]# ping 192.168.8.66
  4.  
  5. # 能telnet访问
  6.  
  7. [root@k8s-node02 ~]# telnet 192.168.8.66 8443
  8.  
  9. # 关闭主节点，看vip是否漂移到备节点

6.k8s组件配置（区别于第4点）

所有k8s节点创建以下目录

mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

6.1.创建apiserver（所有master节点）

6.1.1master01节点配置 

  1. cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
  2.  
  3. [Unit]
  4. Description=Kubernetes API Server
  5. Documentation=https://github.com/kubernetes/kubernetes
  6. After=network.target
  7.  
  8. [Service]
  9. ExecStart=/usr/local/bin/kube-apiserver \\
  10.       --v=2  \\
  11.       --allow-privileged=true  \\
  12.       --bind-address=0.0.0.0  \\
  13.       --secure-port=6443  \\
  14.       --advertise-address=192.168.1.61 \\
  15.       --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
  16.       --service-node-port-range=30000-32767  \\
  17.       --etcd-servers=https://192.168.1.61:2379,https://192.168.1.62:2379,https://192.168.1.63:2379 \\
  18.       --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
  19.       --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
  20.       --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
  21.       --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
  22.       --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
  23.       --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
  24.       --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
  25.       --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
  26.       --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
  27.       --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
  28.       --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
  29.       --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
  30.       --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
  31.       --authorization-mode=Node,RBAC  \\
  32.       --enable-bootstrap-token-auth=true  \\
  33.       --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
  34.       --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
  35.       --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
  36.       --requestheader-allowed-names=aggregator  \\
  37.       --requestheader-group-headers=X-Remote-Group  \\
  38.       --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
  39.       --requestheader-username-headers=X-Remote-User \\
  40.       --enable-aggregator-routing=true
  41.       # --feature-gates=IPv6DualStack=true
  42.       # --token-auth-file=/etc/kubernetes/token.csv
  43.  
  44. Restart=on-failure
  45. RestartSec=10s
  46. LimitNOFILE=65535
  47.  
  48. [Install]
  49. WantedBy=multi-user.target
  50.  
  51. EOF

6.1.2master02节点配置 

  1. cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
  2. [Unit]
  3. Description=Kubernetes API Server
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=network.target
  6.  
  7. [Service]
  8. ExecStart=/usr/local/bin/kube-apiserver \\
  9.       --v=2  \\
  10.       --allow-privileged=true  \\
  11.       --bind-address=0.0.0.0  \\
  12.       --secure-port=6443  \\
  13.       --advertise-address=192.168.1.62 \\
  14.       --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
  15.       --service-node-port-range=30000-32767  \\
  16.       --etcd-servers=https://192.168.1.61:2379,https://192.168.1.62:2379,https://192.168.1.63:2379 \\
  17.       --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
  18.       --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
  19.       --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
  20.       --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
  21.       --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
  22.       --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
  23.       --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
  24.       --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
  25.       --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
  26.       --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
  27.       --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
  28.       --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
  29.       --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
  30.       --authorization-mode=Node,RBAC  \\
  31.       --enable-bootstrap-token-auth=true  \\
  32.       --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
  33.       --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
  34.       --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
  35.       --requestheader-allowed-names=aggregator  \\
  36.       --requestheader-group-headers=X-Remote-Group  \\
  37.       --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
  38.       --requestheader-username-headers=X-Remote-User \\
  39.       --enable-aggregator-routing=true
  40.       # --feature-gates=IPv6DualStack=true
  41.       # --token-auth-file=/etc/kubernetes/token.csv
  42.  
  43. Restart=on-failure
  44. RestartSec=10s
  45. LimitNOFILE=65535
  46.  
  47. [Install]
  48. WantedBy=multi-user.target
  49.  
  50. EOF

6.1.3master03节点配置 

  1. cat > /usr/lib/systemd/system/kube-apiserver.service  << EOF
  2.  
  3. [Unit]
  4. Description=Kubernetes API Server
  5. Documentation=https://github.com/kubernetes/kubernetes
  6. After=network.target
  7.  
  8. [Service]
  9. ExecStart=/usr/local/bin/kube-apiserver \\
  10.       --v=2  \\
  11.       --allow-privileged=true  \\
  12.       --bind-address=0.0.0.0  \\
  13.       --secure-port=6443  \\
  14.       --advertise-address=192.168.1.63 \\
  15.       --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
  16.       --service-node-port-range=30000-32767  \\
  17.       --etcd-servers=https://192.168.1.61:2379,https://192.168.1.62:2379,https://192.168.1.63:2379 \\
  18.       --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
  19.       --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
  20.       --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
  21.       --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
  22.       --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
  23.       --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
  24.       --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
  25.       --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
  26.       --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
  27.       --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
  28.       --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
  29.       --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
  30.       --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
  31.       --authorization-mode=Node,RBAC  \\
  32.       --enable-bootstrap-token-auth=true  \\
  33.       --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
  34.       --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
  35.       --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
  36.       --requestheader-allowed-names=aggregator  \\
  37.       --requestheader-group-headers=X-Remote-Group  \\
  38.       --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
  39.       --requestheader-username-headers=X-Remote-User \\
  40.       --enable-aggregator-routing=true
  41.       # --feature-gates=IPv6DualStack=true
  42.       # --token-auth-file=/etc/kubernetes/token.csv
  43.  
  44. Restart=on-failure
  45. RestartSec=10s
  46. LimitNOFILE=65535
  47.  
  48. [Install]
  49. WantedBy=multi-user.target
  50.  
  51. EOF

6.1.4启动apiserver（所有master节点） 

  1. systemctl daemon-reload && systemctl enable --now kube-apiserver
  2.  
  3. # 注意查看状态是否启动正常
  4. # systemctl status kube-apiserver

6.2.配置kube-controller-manager service

  1. # 所有master节点配置，且配置相同
  2. # 172.16.0.0/12为pod网段，按需求设置你自己的网段
  3.  
  4. cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
  5.  
  6. [Unit]
  7. Description=Kubernetes Controller Manager
  8. Documentation=https://github.com/kubernetes/kubernetes
  9. After=network.target
  10.  
  11. [Service]
  12. ExecStart=/usr/local/bin/kube-controller-manager \\
  13.       --v=2 \\
  14.       --bind-address=127.0.0.1 \\
  15.       --root-ca-file=/etc/kubernetes/pki/ca.pem \\
  16.       --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \\
  17.       --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \\
  18.       --service-account-private-key-file=/etc/kubernetes/pki/sa.key \\
  19.       --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \\
  20.       --leader-elect=true \\
  21.       --use-service-account-credentials=true \\
  22.       --node-monitor-grace-period=40s \\
  23.       --node-monitor-period=5s \\
  24.       --pod-eviction-timeout=2m0s \\
  25.       --controllers=*,bootstrapsigner,tokencleaner \\
  26.       --allocate-node-cidrs=true \\
  27.       --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\
  28.       --cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\
  29.       --node-cidr-mask-size-ipv4=24 \\
  30.       --node-cidr-mask-size-ipv6=120 \\
  31.       --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem 
  32.       # --feature-gates=IPv6DualStack=true
  33.  
  34. Restart=always
  35. RestartSec=10s
  36.  
  37. [Install]
  38. WantedBy=multi-user.target
  39.  
  40. EOF

6.2.1启动kube-controller-manager，并查看状态 

  1. systemctl daemon-reload
  2. systemctl enable --now kube-controller-manager
  3. # systemctl  status kube-controller-manager

6.3.配置kube-scheduler service

6.3.1所有master节点配置，且配置相同 

  1. cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
  2.  
  3. [Unit]
  4. Description=Kubernetes Scheduler
  5. Documentation=https://github.com/kubernetes/kubernetes
  6. After=network.target
  7.  
  8. [Service]
  9. ExecStart=/usr/local/bin/kube-scheduler \\
  10.       --v=2 \\
  11.       --bind-address=127.0.0.1 \\
  12.       --leader-elect=true \\
  13.       --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
  14.  
  15. Restart=always
  16. RestartSec=10s
  17.  
  18. [Install]
  19. WantedBy=multi-user.target
  20.  
  21. EOF

6.3.2启动并查看服务状态 

  1. systemctl daemon-reload
  2. systemctl enable --now kube-scheduler
  3. # systemctl status kube-scheduler

7.TLS Bootstrapping配置

7.1在master01上配置

  1. # 在《5.高可用配置》选择使用那种高可用方案
  2. # 若使用 haproxy、keepalived 那么为 `--server=https://192.168.8.66:8443`
  3. # 若使用 nginx方案，那么为 `--server=https://127.0.0.1:8443`
  4.  
  5. cd bootstrap
  6.  
  7. kubectl config set-cluster kubernetes     \
  8. --certificate-authority=/etc/kubernetes/pki/ca.pem     \
  9. --embed-certs=true     --server=https://127.0.0.1:8443     \
  10. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
  11.  
  12. kubectl config set-credentials tls-bootstrap-token-user     \
  13. --token=c8ad9c.2e4d610cf3e7426e \
  14. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
  15.  
  16. kubectl config set-context tls-bootstrap-token-user@kubernetes     \
  17. --cluster=kubernetes     \
  18. --user=tls-bootstrap-token-user     \
  19. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
  20.  
  21. kubectl config use-context tls-bootstrap-token-user@kubernetes     \
  22. --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
  23.  
  24. # token的位置在bootstrap.secret.yaml，如果修改的话到这个文件修改
  25. mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

7.2查看集群状态，没问题的话继续后续操作

  1. kubectl get cs
  2. Warning: v1 ComponentStatus is deprecated in v1.19+
  3. NAME                 STATUS    MESSAGE                         ERROR
  4. scheduler            Healthy   ok                              
  5. controller-manager   Healthy   ok                              
  6. etcd-0               Healthy   {"health":"true","reason":""}   
  7. etcd-2               Healthy   {"health":"true","reason":""}   
  8. etcd-1               Healthy   {"health":"true","reason":""} 
  9.  
  10. # 切记执行，别忘记！！！
  11. kubectl create -f bootstrap.secret.yaml

8.node节点配置

8.1.在master01上将证书复制到node节点

  1. cd /etc/kubernetes/
  2.  
  3. for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do ssh $NODE mkdir -p /etc/kubernetes/pki; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done

8.2.kubelet配置

注意 ： 8.2.1 和 8.2.2 需要和 上方 2.1 和 2.2 对应起来

8.2.1当使用docker作为Runtime（暂不支持）

v1.26.0 暂时不支持docker方式

  1. cat > /usr/lib/systemd/system/kubelet.service << EOF
  2.  
  3. [Unit]
  4. Description=Kubernetes Kubelet
  5. Documentation=https://github.com/kubernetes/kubernetes
  6.  
  7. [Service]
  8. ExecStart=/usr/local/bin/kubelet \\
  9.     --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig  \\
  10.     --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
  11.     --config=/etc/kubernetes/kubelet-conf.yml \\
  12.     --container-runtime-endpoint=unix:///run/cri-dockerd.sock  \\
  13.     --node-labels=node.kubernetes.io/node=
  14.  
  15. [Install]
  16. WantedBy=multi-user.target
  17. EOF

8.2.2当使用Containerd作为Runtime （推荐） 

  1. mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/
  2.  
  3. # 所有k8s节点配置kubelet service
  4. cat > /usr/lib/systemd/system/kubelet.service << EOF
  5.  
  6. [Unit]
  7. Description=Kubernetes Kubelet
  8. Documentation=https://github.com/kubernetes/kubernetes
  9. After=containerd.service
  10. Requires=containerd.service
  11.  
  12. [Service]
  13. ExecStart=/usr/local/bin/kubelet \\
  14.     --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig  \\
  15.     --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
  16.     --config=/etc/kubernetes/kubelet-conf.yml \\
  17.     --container-runtime-endpoint=unix:///run/containerd/containerd.sock  \\
  18.     --node-labels=node.kubernetes.io/node=
  19.     # --feature-gates=IPv6DualStack=true
  20.     # --container-runtime=remote
  21.     # --runtime-request-timeout=15m
  22.     # --cgroup-driver=systemd
  23.  
  24. [Install]
  25. WantedBy=multi-user.target
  26. EOF

8.2.3所有k8s节点创建kubelet的配置文件 

  1. cat > /etc/kubernetes/kubelet-conf.yml <<EOF
  2. apiVersion: kubelet.config.k8s.io/v1beta1
  3. kind: KubeletConfiguration
  4. address: 0.0.0.0
  5. port: 10250
  6. readOnlyPort: 10255
  7. authentication:
  8.   anonymous:
  9.     enabled: false
  10.   webhook:
  11.     cacheTTL: 2m0s
  12.     enabled: true
  13.   x509:
  14.     clientCAFile: /etc/kubernetes/pki/ca.pem
  15. authorization:
  16.   mode: Webhook
  17.   webhook:
  18.     cacheAuthorizedTTL: 5m0s
  19.     cacheUnauthorizedTTL: 30s
  20. cgroupDriver: systemd
  21. cgroupsPerQOS: true
  22. clusterDNS:
  23. - 10.96.0.10
  24. clusterDomain: cluster.local
  25. containerLogMaxFiles: 5
  26. containerLogMaxSize: 10Mi
  27. contentType: application/vnd.kubernetes.protobuf
  28. cpuCFSQuota: true
  29. cpuManagerPolicy: none
  30. cpuManagerReconcilePeriod: 10s
  31. enableControllerAttachDetach: true
  32. enableDebuggingHandlers: true
  33. enforceNodeAllocatable:
  34. - pods
  35. eventBurst: 10
  36. eventRecordQPS: 5
  37. evictionHard:
  38.   imagefs.available: 15%
  39.   memory.available: 100Mi
  40.   nodefs.available: 10%
  41.   nodefs.inodesFree: 5%
  42. evictionPressureTransitionPeriod: 5m0s
  43. failSwapOn: true
  44. fileCheckFrequency: 20s
  45. hairpinMode: promiscuous-bridge
  46. healthzBindAddress: 127.0.0.1
  47. healthzPort: 10248
  48. httpCheckFrequency: 20s
  49. imageGCHighThresholdPercent: 85
  50. imageGCLowThresholdPercent: 80
  51. imageMinimumGCAge: 2m0s
  52. iptablesDropBit: 15
  53. iptablesMasqueradeBit: 14
  54. kubeAPIBurst: 10
  55. kubeAPIQPS: 5
  56. makeIPTablesUtilChains: true
  57. maxOpenFiles: 1000000
  58. maxPods: 110
  59. nodeStatusUpdateFrequency: 10s
  60. oomScoreAdj: -999
  61. podPidsLimit: -1
  62. registryBurst: 10
  63. registryPullQPS: 5
  64. resolvConf: /etc/resolv.conf
  65. rotateCertificates: true
  66. runtimeRequestTimeout: 2m0s
  67. serializeImagePulls: true
  68. staticPodPath: /etc/kubernetes/manifests
  69. streamingConnectionIdleTimeout: 4h0m0s
  70. syncFrequency: 1m0s
  71. volumeStatsAggPeriod: 1m0s
  72. EOF

8.2.4启动kubelet 

  1. systemctl daemon-reload
  2. systemctl restart kubelet
  3. systemctl enable --now kubelet

8.2.5查看集群 

  1. [root@k8s-master01 ~]# kubectl  get node
  2. NAME           STATUS     ROLES    AGE   VERSION
  3. k8s-master01   Ready    <none>   18s   v1.26.0
  4. k8s-master02   Ready    <none>   16s   v1.26.0
  5. k8s-master03   Ready    <none>   16s   v1.26.0
  6. k8s-node01     Ready    <none>   14s   v1.26.0
  7. k8s-node02     Ready    <none>   14s   v1.26.0
  8. [root@k8s-master01 ~]#

8.3.kube-proxy配置

8.3.1将kubeconfig发送至其他节点 

  1. for NODE in k8s-master02 k8s-master03; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done
  2.  
  3. for NODE in k8s-node01 k8s-node02; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig;  done

8.3.2所有k8s节点添加kube-proxy的service文件 

  1. cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
  2. [Unit]
  3. Description=Kubernetes Kube Proxy
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=network.target
  6.  
  7. [Service]
  8. ExecStart=/usr/local/bin/kube-proxy \\
  9.   --config=/etc/kubernetes/kube-proxy.yaml \\
  10.   --v=2
  11.  
  12. Restart=always
  13. RestartSec=10s
  14.  
  15. [Install]
  16. WantedBy=multi-user.target
  17.  
  18. EOF

8.3.3所有k8s节点添加kube-proxy的配置 

  1. cat > /etc/kubernetes/kube-proxy.yaml << EOF
  2. apiVersion: kubeproxy.config.k8s.io/v1alpha1
  3. bindAddress: 0.0.0.0
  4. clientConnection:
  5.   acceptContentTypes: ""
  6.   burst: 10
  7.   contentType: application/vnd.kubernetes.protobuf
  8.   kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  9.   qps: 5
  10. clusterCIDR: 172.16.0.0/12,fc00:2222::/112
  11. configSyncPeriod: 15m0s
  12. conntrack:
  13.   max: null
  14.   maxPerCore: 32768
  15.   min: 131072
  16.   tcpCloseWaitTimeout: 1h0m0s
  17.   tcpEstablishedTimeout: 24h0m0s
  18. enableProfiling: false
  19. healthzBindAddress: 0.0.0.0:10256
  20. hostnameOverride: ""
  21. iptables:
  22.   masqueradeAll: false
  23.   masqueradeBit: 14
  24.   minSyncPeriod: 0s
  25.   syncPeriod: 30s
  26. ipvs:
  27.   masqueradeAll: true
  28.   minSyncPeriod: 5s
  29.   scheduler: "rr"
  30.   syncPeriod: 30s
  31. kind: KubeProxyConfiguration
  32. metricsBindAddress: 127.0.0.1:10249
  33. mode: "ipvs"
  34. nodePortAddresses: null
  35. oomScoreAdj: -999
  36. portRange: ""
  37. udpIdleTimeout: 250ms
  38.  
  39. EOF

8.3.4启动kube-proxy 

  1. systemctl daemon-reload
  2.  systemctl restart kube-proxy
  3.  systemctl enable --now kube-proxy

9.安装网络插件

注意 9.1 和 9.2 二选其一即可，建议在此处创建好快照后在进行操作，后续出问题可以回滚

** centos7 要升级libseccomp 不然 无法安装网络插件**

  1. # https://github.com/opencontainers/runc/releases
  2. # 升级runc
  3. wget https://ghproxy.com/https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64
  4. install -m 755 runc.amd64 /usr/local/sbin/runc
  5. cp -p /usr/local/sbin/runc  /usr/local/bin/runc
  6. cp -p /usr/local/sbin/runc  /usr/bin/runc
  7.  
  8. #下载高于2.4以上的包
  9. yum -y install http://rpmfind.net/linux/centos/8-stream/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm
  10.  
  11.  
  12. #查看当前版本
  13. [root@k8s-master-1 ~]# rpm -qa | grep libseccomp
  14. libseccomp-2.5.1-1.el8.x86_64

9.1安装Calico

9.1.1更改calico网段 

  1. # 本地没有公网 IPv6 使用 calico.yaml
  2. kubectl apply -f calico.yaml
  3.  
  4. # 本地有公网 IPv6 使用 calico-ipv6.yaml 
  5. # kubectl apply -f calico-ipv6.yaml 
  6.  
  7. # 若docker镜像拉不下来，可以使用我的仓库
  8. # sed -i "s#docker.io/calico/#registry.cn-hangzhou.aliyuncs.com/chenby/#g" calico.yaml 
  9. # sed -i "s#docker.io/calico/#registry.cn-hangzhou.aliyuncs.com/chenby/#g" calico-ipv6.yaml

9.1.2查看容器状态 

  1. # calico 初始化会很慢 需要耐心等待一下，大约十分钟左右
  2. [root@k8s-master01 ~]# kubectl  get pod -A
  3. NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
  4. kube-system   calico-kube-controllers-6747f75cdc-fbvvc   1/1     Running   0          61s
  5. kube-system   calico-node-fs7hl                          1/1     Running   0          61s
  6. kube-system   calico-node-jqz58                          1/1     Running   0          61s
  7. kube-system   calico-node-khjlg                          1/1     Running   0          61s
  8. kube-system   calico-node-wmf8q                          1/1     Running   0          61s
  9. kube-system   calico-node-xc6gn                          1/1     Running   0          61s
  10. kube-system   calico-typha-6cdc4b4fbc-57snb              1/1     Running   0          61s

9.2 安装cilium

9.2.1 安装helm 

  1. # [root@k8s-master01 ~]# curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
  2. # [root@k8s-master01 ~]# chmod 700 get_helm.sh
  3. # [root@k8s-master01 ~]# ./get_helm.sh
  4.  
  5. wget https://get.helm.sh/helm-canary-linux-amd64.tar.gz
  6. tar xvf helm-canary-linux-amd64.tar.tar
  7. cp linux-amd64/helm /usr/local/bin/

9.2.2 安装cilium 

  1. # 添加源
  2. helm repo add cilium https://helm.cilium.io
  3.  
  4. # 默认参数安装
  5. helm install cilium cilium/cilium --namespace kube-system
  6.  
  7. # 启用ipv6
  8. # helm install cilium cilium/cilium --namespace kube-system --set ipv6.enabled=true
  9.  
  10. # 启用路由信息和监控插件
  11. # helm install cilium cilium/cilium --namespace kube-system --set hubble.relay.enabled=true --set hubble.ui.enabled=true --set prometheus.enabled=true --set operator.prometheus.enabled=true --set hubble.enabled=true --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

9.2.3 查看 

  1. [root@k8s-master01 ~]# kubectl  get pod -A | grep cil
  2. kube-system   cilium-gmr6c                       1/1     Running       0             5m3s
  3. kube-system   cilium-kzgdj                       1/1     Running       0             5m3s
  4. kube-system   cilium-operator-69b677f97c-6pw4k   1/1     Running       0             5m3s
  5. kube-system   cilium-operator-69b677f97c-xzzdk   1/1     Running       0             5m3s
  6. kube-system   cilium-q2rnr                       1/1     Running       0             5m3s
  7. kube-system   cilium-smx5v                       1/1     Running       0             5m3s
  8. kube-system   cilium-tdjq4                       1/1     Running       0             5m3s
  9. [root@k8s-master01 ~]#

9.2.4 下载专属监控面板 

  1. [root@k8s-master01 yaml]# wget https://raw.githubusercontent.com/cilium/cilium/1.12.1/examples/kubernetes/addons/prometheus/monitoring-example.yaml
  2. [root@k8s-master01 yaml]#
  3. [root@k8s-master01 yaml]# kubectl  apply -f monitoring-example.yaml
  4. namespace/cilium-monitoring created
  5. serviceaccount/prometheus-k8s created
  6. configmap/grafana-config created
  7. configmap/grafana-cilium-dashboard created
  8. configmap/grafana-cilium-operator-dashboard created
  9. configmap/grafana-hubble-dashboard created
  10. configmap/prometheus created
  11. clusterrole.rbac.authorization.k8s.io/prometheus created
  12. clusterrolebinding.rbac.authorization.k8s.io/prometheus created
  13. service/grafana created
  14. service/prometheus created
  15. deployment.apps/grafana created
  16. deployment.apps/prometheus created
  17. [root@k8s-master01 yaml]#

9.2.5 下载部署测试用例 

  1. [root@k8s-master01 yaml]# wget https://raw.githubusercontent.com/cilium/cilium/master/examples/kubernetes/connectivity-check/connectivity-check.yaml
  2.  
  3. [root@k8s-master01 yaml]# sed -i "s#google.com#oiox.cn#g" connectivity-check.yaml
  4.  
  5. [root@k8s-master01 yaml]# kubectl  apply -f connectivity-check.yaml
  6. deployment.apps/echo-a created
  7. deployment.apps/echo-b created
  8. deployment.apps/echo-b-host created
  9. deployment.apps/pod-to-a created
  10. deployment.apps/pod-to-external-1111 created
  11. deployment.apps/pod-to-a-denied-cnp created
  12. deployment.apps/pod-to-a-allowed-cnp created
  13. deployment.apps/pod-to-external-fqdn-allow-google-cnp created
  14. deployment.apps/pod-to-b-multi-node-clusterip created
  15. deployment.apps/pod-to-b-multi-node-headless created
  16. deployment.apps/host-to-b-multi-node-clusterip created
  17. deployment.apps/host-to-b-multi-node-headless created
  18. deployment.apps/pod-to-b-multi-node-nodeport created
  19. deployment.apps/pod-to-b-intra-node-nodeport created
  20. service/echo-a created
  21. service/echo-b created
  22. service/echo-b-headless created
  23. service/echo-b-host-headless created
  24. ciliumnetworkpolicy.cilium.io/pod-to-a-denied-cnp created
  25. ciliumnetworkpolicy.cilium.io/pod-to-a-allowed-cnp created
  26. ciliumnetworkpolicy.cilium.io/pod-to-external-fqdn-allow-google-cnp created
  27. [root@k8s-master01 yaml]#

9.2.6 查看pod 

  1. [root@k8s-master01 yaml]# kubectl  get pod -A
  2. NAMESPACE           NAME                                                     READY   STATUS    RESTARTS      AGE
  3. cilium-monitoring   grafana-59957b9549-6zzqh                                 1/1     Running   0             10m
  4. cilium-monitoring   prometheus-7c8c9684bb-4v9cl                              1/1     Running   0             10m
  5. default             chenby-75b5d7fbfb-7zjsr                                  1/1     Running   0             27h
  6. default             chenby-75b5d7fbfb-hbvr8                                  1/1     Running   0             27h
  7. default             chenby-75b5d7fbfb-ppbzg                                  1/1     Running   0             27h
  8. default             echo-a-6799dff547-pnx6w                                  1/1     Running   0             10m
  9. default             echo-b-fc47b659c-4bdg9                                   1/1     Running   0             10m
  10. default             echo-b-host-67fcfd59b7-28r9s                             1/1     Running   0             10m
  11. default             host-to-b-multi-node-clusterip-69c57975d6-z4j2z          1/1     Running   0             10m
  12. default             host-to-b-multi-node-headless-865899f7bb-frrmc           1/1     Running   0             10m
  13. default             pod-to-a-allowed-cnp-5f9d7d4b9d-hcd8x                    1/1     Running   0             10m
  14. default             pod-to-a-denied-cnp-65cc5ff97b-2rzb8                     1/1     Running   0             10m
  15. default             pod-to-a-dfc64f564-p7xcn                                 1/1     Running   0             10m
  16. default             pod-to-b-intra-node-nodeport-677868746b-trk2l            1/1     Running   0             10m
  17. default             pod-to-b-multi-node-clusterip-76bbbc677b-knfq2           1/1     Running   0             10m
  18. default             pod-to-b-multi-node-headless-698c6579fd-mmvd7            1/1     Running   0             10m
  19. default             pod-to-b-multi-node-nodeport-5dc4b8cfd6-8dxmz            1/1     Running   0             10m
  20. default             pod-to-external-1111-8459965778-pjt9b                    1/1     Running   0             10m
  21. default             pod-to-external-fqdn-allow-google-cnp-64df9fb89b-l9l4q   1/1     Running   0             10m
  22. kube-system         cilium-7rfj6                                             1/1     Running   0             56s
  23. kube-system         cilium-d4cch                                             1/1     Running   0             56s
  24. kube-system         cilium-h5x8r                                             1/1     Running   0             56s
  25. kube-system         cilium-operator-5dbddb6dbf-flpl5                         1/1     Running   0             56s
  26. kube-system         cilium-operator-5dbddb6dbf-gcznc                         1/1     Running   0             56s
  27. kube-system         cilium-t2xlz                                             1/1     Running   0             56s
  28. kube-system         cilium-z65z7                                             1/1     Running   0             56s
  29. kube-system         coredns-665475b9f8-jkqn8                                 1/1     Running   1 (36h ago)   36h
  30. kube-system         hubble-relay-59d8575-9pl9z                               1/1     Running   0             56s
  31. kube-system         hubble-ui-64d4995d57-nsv9j                               2/2     Running   0             56s
  32. kube-system         metrics-server-776f58c94b-c6zgs                          1/1     Running   1 (36h ago)   37h
  33. [root@k8s-master01 yaml]#

9.2.7 修改为NodePort 

  1. [root@k8s-master01 yaml]# kubectl  edit svc  -n kube-system hubble-ui
  2. service/hubble-ui edited
  3. [root@k8s-master01 yaml]#
  4. [root@k8s-master01 yaml]# kubectl  edit svc  -n cilium-monitoring grafana
  5. service/grafana edited
  6. [root@k8s-master01 yaml]#
  7. [root@k8s-master01 yaml]# kubectl  edit svc  -n cilium-monitoring prometheus
  8. service/prometheus edited
  9. [root@k8s-master01 yaml]#
  10.  
  11. type: NodePort

9.2.8 查看端口 

  1. [root@k8s-master01 yaml]# kubectl get svc -A | grep monit
  2. cilium-monitoring   grafana                NodePort    10.100.250.17    <none>        3000:30707/TCP           15m
  3. cilium-monitoring   prometheus             NodePort    10.100.131.243   <none>        9090:31155/TCP           15m
  4. [root@k8s-master01 yaml]#
  5. [root@k8s-master01 yaml]# kubectl get svc -A | grep hubble
  6. kube-system         hubble-metrics         ClusterIP   None             <none>        9965/TCP                 5m12s
  7. kube-system         hubble-peer            ClusterIP   10.100.150.29    <none>        443/TCP                  5m12s
  8. kube-system         hubble-relay           ClusterIP   10.109.251.34    <none>        80/TCP                   5m12s
  9. kube-system         hubble-ui              NodePort    10.102.253.59    <none>        80:31219/TCP             5m12s
  10. [root@k8s-master01 yaml]#

9.2.9 访问 

  1. http://192.168.1.61:30707
  2. http://192.168.1.61:31155
  3. http://192.168.1.61:31219

10.安装CoreDNS

10.1以下步骤只在master01操作

10.1.1修改文件 

  1. cd coredns/
  2. cat coredns.yaml | grep clusterIP:
  3.   clusterIP: 10.96.0.10

10.1.2安装 

  1. kubectl  create -f coredns.yaml 
  2. serviceaccount/coredns created
  3. clusterrole.rbac.authorization.k8s.io/system:coredns created
  4. clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
  5. configmap/coredns created
  6. deployment.apps/coredns created
  7. service/kube-dns created

11.安装Metrics Server

11.1以下步骤只在master01操作

11.1.1安装Metrics-server

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率

  1. # 安装metrics server
  2. cd metrics-server/
  3.  
  4. kubectl  apply -f metrics-server.yaml

11.1.2稍等片刻查看状态 

  1. kubectl  top node
  2. NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
  3. k8s-master01   154m         1%     1715Mi          21%       
  4. k8s-master02   151m         1%     1274Mi          16%       
  5. k8s-master03   523m         6%     1345Mi          17%       
  6. k8s-node01     84m          1%     671Mi           8%        
  7. k8s-node02     73m          0%     727Mi           9%        
  8. k8s-node03     96m          1%     769Mi           9%        
  9. k8s-node04     68m          0%     673Mi           8%        
  10. k8s-node05     82m          1%     679Mi           8%

12.集群验证

12.1部署pod资源

  1. cat<<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. kind: Pod
  4. metadata:
  5.   name: busybox
  6.   namespace: default
  7. spec:
  8.   containers:
  9.   - name: busybox
  10.     image: docker.io/library/busybox:1.28
  11.     command:
  12.       - sleep
  13.       - "3600"
  14.     imagePullPolicy: IfNotPresent
  15.   restartPolicy: Always
  16. EOF
  17.  
  18. # 查看
  19. kubectl  get pod
  20. NAME      READY   STATUS    RESTARTS   AGE
  21. busybox   1/1     Running   0          17s

12.2用pod解析默认命名空间中的kubernetes

  1. kubectl get svc
  2. NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
  3. kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   17h
  4.  
  5. kubectl exec  busybox -n default -- nslookup kubernetes
  6. 3Server:    10.96.0.10
  7. Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
  8.  
  9. Name:      kubernetes
  10. Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

12.3测试跨命名空间是否可以解析

  1. kubectl exec  busybox -n default -- nslookup kube-dns.kube-system
  2. Server:    10.96.0.10
  3. Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
  4.  
  5. Name:      kube-dns.kube-system
  6. Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

12.4每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53

  1. telnet 10.96.0.1 443
  2. Trying 10.96.0.1...
  3. Connected to 10.96.0.1.
  4. Escape character is '^]'.
  5.  
  6.  telnet 10.96.0.10 53
  7. Trying 10.96.0.10...
  8. Connected to 10.96.0.10.
  9. Escape character is '^]'.
  10.  
  11. curl 10.96.0.10:53
  12. curl: (52) Empty reply from server

12.5Pod和Pod之前要能通

  1. kubectl get po -owide
  2. NAME      READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
  3. busybox   1/1     Running   0          17m   172.27.14.193   k8s-node02   <none>           <none>
  4.  
  5.  kubectl get po -n kube-system -owide
  6. NAME                                       READY   STATUS    RESTARTS      AGE   IP               NODE           NOMINATED NODE   READINESS GATES
  7. calico-kube-controllers-5dffd5886b-4blh6   1/1     Running   0             77m   172.25.244.193   k8s-master01   <none>           <none>
  8. calico-node-fvbdq                          1/1     Running   1 (75m ago)   77m   192.168.1.61     k8s-master01   <none>           <none>
  9. calico-node-g8nqd                          1/1     Running   0             77m   192.168.1.64     k8s-node01     <none>           <none>
  10. calico-node-mdps8                          1/1     Running   0             77m   192.168.1.65     k8s-node02     <none>           <none>
  11. calico-node-nf4nt                          1/1     Running   0             77m   192.168.1.63     k8s-master03   <none>           <none>
  12. calico-node-sq2ml                          1/1     Running   0             77m   192.168.1.62     k8s-master02   <none>           <none>
  13. calico-typha-8445487f56-mg6p8              1/1     Running   0             77m   192.168.1.65     k8s-node02     <none>           <none>
  14. calico-typha-8445487f56-pxbpj              1/1     Running   0             77m   192.168.1.61     k8s-master01   <none>           <none>
  15. calico-typha-8445487f56-tnssl              1/1     Running   0             77m   192.168.1.64     k8s-node01     <none>           <none>
  16. coredns-5db5696c7-67h79                    1/1     Running   0             63m   172.25.92.65     k8s-master02   <none>           <none>
  17. metrics-server-6bf7dcd649-5fhrw            1/1     Running   0             61m   172.18.195.1     k8s-master03   <none>           <none>
  18.  
  19. # 进入busybox ping其他节点上的pod
  20.  
  21. kubectl exec -ti busybox -- sh
  22. / # ping 192.168.1.64
  23. PING 192.168.1.64 (192.168.1.64): 56 data bytes
  24. 64 bytes from 192.168.1.64: seq=0 ttl=63 time=0.358 ms
  25. 64 bytes from 192.168.1.64: seq=1 ttl=63 time=0.668 ms
  26. 64 bytes from 192.168.1.64: seq=2 ttl=63 time=0.637 ms
  27. 64 bytes from 192.168.1.64: seq=3 ttl=63 time=0.624 ms
  28. 64 bytes from 192.168.1.64: seq=4 ttl=63 time=0.907 ms
  29.  
  30. # 可以连通证明这个pod是可以跨命名空间和跨主机通信的

12.6创建三个副本，可以看到3个副本分布在不同的节点上（用完可以删了）

  1. cat > deployments.yaml << EOF
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: nginx-deployment
  6.   labels:
  7.     app: nginx
  8. spec:
  9.   replicas: 3
  10.   selector:
  11.     matchLabels:
  12.       app: nginx
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: nginx
  17.     spec:
  18.       containers:
  19.       - name: nginx
  20.         image: docker.io/library/nginx:1.14.2
  21.         ports:
  22.         - containerPort: 80
  23.  
  24. EOF
  25.  
  26. kubectl  apply -f deployments.yaml 
  27. deployment.apps/nginx-deployment created
  28.  
  29. kubectl  get pod 
  30. NAME                               READY   STATUS    RESTARTS   AGE
  31. busybox                            1/1     Running   0          6m25s
  32. nginx-deployment-9456bbbf9-4bmvk   1/1     Running   0          8s
  33. nginx-deployment-9456bbbf9-9rcdk   1/1     Running   0          8s
  34. nginx-deployment-9456bbbf9-dqv8s   1/1     Running   0          8s
  35.  
  36. # 删除nginx
  37.  
  38. [root@k8s-master01 ~]# kubectl delete -f deployments.yaml

13.安装dashboard

  1. helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
  2. helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard

13.1更改dashboard的svc为NodePort，如果已是请忽略

  1. kubectl edit svc kubernetes-dashboard
  2.   type: NodePort

13.2查看端口号

  1. kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
  2. NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
  3. kubernetes-dashboard   NodePort   10.108.120.110   <none>        443:30034/TCP   34s

13.3创建token

  1. kubectl -n kubernetes-dashboard create token admin-user
  2. eyJhbGciOiJSUzI1NiIsImtpZCI6IkFZWENLUmZQWTViWUF4UV81NWJNb0JEa0I4R2hQMHVac2J3RDM3RHJLcFEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjcwNjc0MzY1LCJpYXQiOjE2NzA2NzA3NjUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiODkyODRjNGUtYzk0My00ODkzLWE2ZjctNTYxZWJhMzE2NjkwIn19LCJuYmYiOjE2NzA2NzA3NjUsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.DFxzS802Iu0lldikjhyp2diZSpVAUoSTbOjerH2t7ToM0TMoPQdcdDyvBTcNlIew3F01u4D6atNV7J36IGAnHEX0Q_cYAb00jINjy1YXGz0gRhRE0hMrXay2-Qqo6tAORTLUVWrctW6r0li5q90rkBjr5q06Lt5BTpUhbhbgLQQJWwiEVseCpUEikxD6wGnB1tCamFyjs3sa-YnhhqCR8wUAZcTaeVbMxCuHVAuSqnIkxat9nyxGcsjn7sqmBqYjjOGxp5nhHPDj03TWmSJlb_Csc7pvLsB9LYm0IbER4xDwtLZwMAjYWRbjKxbkUp4L9v5CZ4PbIHap9qQp1FXreA

13.3登录dashboard

https://192.168.1.61:30034/

14.ingress安装

14.1执行部署

  1. cd ingress/
  2.  
  3. kubectl  apply -f deploy.yaml 
  4. kubectl  apply -f backend.yaml 
  5.  
  6. # 等创建完成后在执行：
  7. kubectl  apply -f ingress-demo-app.yaml 
  8.  
  9. kubectl  get ingress
  10. NAME               CLASS   HOSTS                            ADDRESS     PORTS   AGE
  11. ingress-host-bar   nginx   hello.chenby.cn,demo.chenby.cn   192.168.1.62   80      7s

14.2过滤查看ingress端口

  1. [root@hello ~/yaml]# kubectl  get svc -A | grep ingress
  2. ingress-nginx          ingress-nginx-controller             NodePort    10.104.231.36    <none>        80:32636/TCP,443:30579/TCP   104s
  3. ingress-nginx          ingress-nginx-controller-admission   ClusterIP   10.101.85.88     <none>        443/TCP                      105s
  4. [root@hello ~/yaml]#

15.IPv6测试

  1. #部署应用
  2.  
  3. cat<<EOF | kubectl apply -f -
  4. apiVersion: apps/v1
  5. kind: Deployment
  6. metadata:
  7.   name: chenby
  8. spec:
  9.   replicas: 3
  10.   selector:
  11.     matchLabels:
  12.       app: chenby
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: chenby
  17.     spec:
  18.       containers:
  19.       - name: chenby
  20.         image: docker.io/library/nginx
  21.         resources:
  22.           limits:
  23.             memory: "128Mi"
  24.             cpu: "500m"
  25.         ports:
  26.         - containerPort: 80
  27.  
  28. ---
  29. apiVersion: v1
  30. kind: Service
  31. metadata:
  32.   name: chenby
  33. spec:
  34.   ipFamilyPolicy: PreferDualStack
  35.   ipFamilies:
  36.   - IPv6
  37.   - IPv4
  38.   type: NodePort
  39.   selector:
  40.     app: chenby
  41.   ports:
  42.   - port: 80
  43.     targetPort: 80
  44. EOF
  45.  
  46.  
  47. #查看端口
  48. [root@k8s-master01 ~]# kubectl  get svc
  49. NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
  50. chenby         NodePort    fd00::a29c       <none>        80:30779/TCP   5s
  51. [root@k8s-master01 ~]# 
  52.  
  53. #使用内网访问
  54. [root@localhost yaml]# curl -I http://[fd00::a29c]
  55. HTTP/1.1 200 OK
  56. Server: nginx/1.21.6
  57. Date: Thu, 05 May 2022 10:20:35 GMT
  58. Content-Type: text/html
  59. Content-Length: 615
  60. Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT
  61. Connection: keep-alive
  62. ETag: "61f01158-267"
  63. Accept-Ranges: bytes
  64.  
  65. [root@localhost yaml]# curl -I http://192.168.1.61:30779
  66. HTTP/1.1 200 OK
  67. Server: nginx/1.21.6
  68. Date: Thu, 05 May 2022 10:20:59 GMT
  69. Content-Type: text/html
  70. Content-Length: 615
  71. Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT
  72. Connection: keep-alive
  73. ETag: "61f01158-267"
  74. Accept-Ranges: bytes
  75.  
  76. [root@localhost yaml]# 
  77.  
  78. #使用公网访问
  79. [root@localhost yaml]# curl -I http://[2409:8a10:9e18:9020::10]:30779
  80. HTTP/1.1 200 OK
  81. Server: nginx/1.21.6
  82. Date: Thu, 05 May 2022 10:20:54 GMT
  83. Content-Type: text/html
  84. Content-Length: 615
  85. Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT
  86. Connection: keep-alive
  87. ETag: "61f01158-267"
  88. Accept-Ranges: bytes

16.安装命令行自动补全功能

  1. yum install bash-completion -y
  2. source /usr/share/bash-completion/bash_completion
  3. source <(kubectl completion bash)
  4. echo "source <(kubectl completion bash)" >> ~/.bashrc

关于

https://www.oiox.cn/

https://www.oiox.cn/index.php/start-page.html

CSDN、GitHub、知乎、开源中国、思否、掘金、简书、华为云、阿里云、腾讯云、哔哩哔哩、今日头条、新浪微博、个人博客

全网可搜《小陈运维》

文章主要发布于微信公众号：《Linux运维交流社区》

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/菜鸟追梦旅行/article/detail/429106
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号