赞
踩
这里使用pve虚拟机搭建三台centos机器,搭建过程参考: Centos篇-Centos Minimal安装
CPU:2C
内存:2G
存储:64G
操作系统:Centos 7.9
内核版本:6.2.0-1.el7.elrepo.x86_64
master:192.168.1.12
node1:192.168.1.13
node2:192.168.1.14
已经为三台目标机设置了ssh公钥登录,实现方式参考:Centos篇-Centos ssh公钥登录
systemctl stop firewalld && systemctl disable firewalld && iptables -F
firewalld和iptables是前后有两代方案,一般这俩可以先关闭,然后在部署k8s时如果确定使用iptables,可以再开启,如果使用其他方式,则firewalld和iptables均保持关闭就可以。
sed -i 's/enforcing/disabled/' /etc/selinux/config && setenforce 0
Selinux是内核级别的一个安全模块,通过安全上下文的方式控制应用服务的权限,是应用和操作系统之间的一道ACL,但不是所有的程序都会去适配这个模块,不适配的话开着也不起作用,何况还有规则影响到正常的服务。比如权限等等相关的问题,所以一般要关闭掉。
临时关闭指令:
swapoff -a
永久关闭指令:
sed -ri 's/.*swap.*/#&/' /etc/fstab
可以先临时关闭,再永久关闭。
Swap会导致docker的运行不正常,性能下降,是个bug,现在不知道是否有修复,但是一般关闭swap已经成了通用方案。
因为很多主机的名称是一样的,可能是安装时默认的,也可能是虚拟机克隆的,所以这里设置hostname来进行区分,如果像我一样在安装系统时就定义了不同的主机名,那么可以忽略。
master:
hostnamectl set-hostname centos-k8s-master.local
node1:
hostnamectl set-hostname centos-k8s-node1.local
node2:
hostnamectl set-hostname centos-k8s-node2.local
vim /etc/hosts 添加如下内容
192.168.1.12 centos-k8s-master.local
192.168.1.13 centos-k8s-node1.local
192.168.1.14 centos-k8s-node2.local
这样就可以根据主机的hostname名解析ip了。
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
如果kube-proxy 模式为ip_vs则必须加载,本文采用iptables
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
设置开机启动加载:
cat > /etc/modules-load.d/ip_vs.conf << EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
EOF
这里我是更新了自己系统的内核,但是不是必须的,可以忽略此步骤。
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
当ELRepo仓库启用后,可以使用下面的指令列出可以使用的内核包:
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
安装稳定版本内核:
yum --enablerepo=elrepo-kernel install kernel-ml -y
设置GRUB默认的内核版本:L;
grub2-set-default 0
init 6
使用下面的命令查看内核版本是否更新成功:
uname -r
打开并编辑/etc/default/grub并设置
GRUB_DEFAULT=0。意思是 GRUB 初始化页面的第一个内核将作为默认内核。
GRUB_TIMEOUT=5
GRUB_DEFAULT=0
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
再重启完成。
yum install wget -y
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
列出所有支持的docker版本:
yum list docker-ce.x86_64 --showduplicates |sort
选择一个版本进行安装,此处我安装的是23.0.1-1.el7
yum -y install docker-ce-23.0.1-1.el7 docker-ce-cli-23.0.1-1.el7
编辑/etc/docker/daemon.json
mkdir /etc/docker/
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://gqs7xcfd.mirror.aliyuncs.com","https://hub-mirror.c.163.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
systemctl daemon-reload && systemctl enable docker && systemctl start docker
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
因为kubeadm、kubelet、kubectl需要安装同版本,所以一般查看kubelet就可以了,其余的使用同版本就ok
yum list kubelet --showduplicates
yum install -y kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6
systemctl enable kubelet
kubeadm init \
--kubernetes-version 1.23.6 \
--apiserver-advertise-address=192.168.1.12 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=10.245.0.0/16 \
--image-repository registry.aliyuncs.com/google_containers
-kubernetes-version版本一定要填与上面一致,我这里是1.23.6。
–service-cidr 指定service网络,不能和node网络冲突
–pod-network-cidr 指定pod网络,不能和node网络、service网络冲突
–image-repository registry.aliyuncs.com/google_containers 指定镜像源,由于默认拉取镜像地址k8s.gcr.io国内无法访问,这里指定阿里云镜像仓库地址。
这里可以直接使用下面的指令提前拉取镜像:
kubeadm --kubernetes-version 1.23.6 config images list
一般使用阿里云的镜像源后就慢不到哪里去了。
初始化结束后会在屏幕上出现两条重要的打印:
1、类似于initialized successfully!
2、kubeadm join xxx.xxx.xxx.xxx:6443 --token …
打印1是证明master节点初始化成功,打印2是用来加入集群的命令,在woker节点上使用,这里就不贴出来了。
这里是还有三条指令,也是在初始化成功后输出的,是用来创建集群kubeconfig文件使用的命令:
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
在各woker节点也要走一遍上面的流程走一遍(除了初始化master节点)
使用在初始化master节点成功后输出的加入集群命令,将woker节点加入集群,我的命令是:
kubeadm join 192.168.1.12:6443 --token 37xfed.jqnw7nsll8jhdfl8 \
--discovery-token-ca-cert-hash sha256:27b08986844ca2306353287507370ce7ba0caef7d4608860db4c85c762149bf6
在所有woker节点上都执行这条命令。
kubectl get nodes
差不多这个样子
但是我这个是已经成功运行之后的,在安装网络插件之前应该都是NotReady状态。
从官网下载flannel yaml文件:
wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
或者直接使用我的:
apiVersion: v1 kind: Namespace metadata: labels: k8s-app: flannel pod-security.kubernetes.io/enforce: privileged name: kube-flannel --- apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: flannel name: flannel namespace: kube-flannel --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: flannel name: flannel rules: - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - get - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch - apiGroups: - networking.k8s.io resources: - clustercidrs verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: flannel name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-flannel --- apiVersion: v1 data: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.245.0.0/16", "Backend": { "Type": "vxlan" } } kind: ConfigMap metadata: labels: app: flannel k8s-app: flannel tier: node name: kube-flannel-cfg namespace: kube-flannel --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel k8s-app: flannel tier: node name: kube-flannel-ds namespace: kube-flannel spec: selector: matchLabels: app: flannel k8s-app: flannel template: metadata: labels: app: flannel k8s-app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: EVENT_QUEUE_DEPTH value: "5000" image: docker.io/flannel/flannel:v0.21.2 name: kube-flannel resources: requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: docker.io/flannel/flannel-cni-plugin:v1.1.2 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: docker.io/flannel/flannel:v0.21.2 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock
官网下载的yaml文件需要注意:net-conf.json内部的Network要与初始化master节点的pod-network-cidr保持一致。
官网默认下载的是:10.244.0.0/16
我修改为了:10.245.0.0/16
执行yaml文件:
kubectl apply -f kube-flannel.yaml
kubectl -n kube-system get pods -o wide
等待flannel节点ready。
等flanner部署成功后,应该所有节点都ready了。
如果centos的内核高于4,则可以忽略:
kubectl get cm kube-proxy -n kube-system -o yaml | sed 's/mode: ""/mode: "iptables"/' | kubectl apply -f -
kubectl -n kube-system rollout restart daemonsets.apps kube-proxy
kubectl -n kube-system rollout restart daemonsets.apps kube-flannel-ds
先检查k8s集群的api server是否有启用API Aggregator
ps -ef |grep apiserver
对比:
ps -ef | grep apiserver | grep enable-aggregator-routing
如果输出一片空白就需要修改下api server
[root@centos-k8s-master ~]# ps -ef | grep apiserver
root 772 20181 0 12:17 pts/0 00:00:00 grep --color=auto apiserver
root 1999 1909 2 Feb22 ? 01:39:39 kube-apiserver --advertise-address=192.168.1.12 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
[root@centos-k8s-master ~]# ps -ef | grep apiserver | grep enable-aggregator-routing
[root@centos-k8s-master ~]#
默认是没有开启的,此时需要修改k8s apiserver的配置文件
sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
在spec.containers.command增加 --enable-aggregator-routing=true
api server 会自动重启,稍后用命令验证一下:
[root@centos-k8s-master ~]# ps -ef | grep apiserver | grep enable-aggregator-routing
root 4010 3984 11 12:22 ? 00:00:03 kube-apiserver --advertise-address=192.168.1.12 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --enable-aggregator-routing=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
[root@centos-k8s-master ~]#
先下载安装文件,直接用最新版本:wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
修改下载下来的 components.yaml(重命名为metrics-server.yaml), 增加 --kubelet-insecure-tls,而且将默认的官方源替换成了阿里云的源,不然一直拉取镜像失败。
image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.6.2
apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server rbac.authorization.k8s.io/aggregate-to-admin: "true" rbac.authorization.k8s.io/aggregate-to-edit: "true" rbac.authorization.k8s.io/aggregate-to-view: "true" name: system:aggregated-metrics-reader rules: - apiGroups: - metrics.k8s.io resources: - pods - nodes verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server name: system:metrics-server rules: - apiGroups: - "" resources: - nodes/metrics verbs: - get - apiGroups: - "" resources: - pods - nodes verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server-auth-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server:system:auth-delegator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: system:metrics-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: v1 kind: Service metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: ports: - name: https port: 443 protocol: TCP targetPort: https selector: k8s-app: metrics-server --- apiVersion: apps/v1 kind: Deployment metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: selector: matchLabels: k8s-app: metrics-server strategy: rollingUpdate: maxUnavailable: 0 template: metadata: labels: k8s-app: metrics-server spec: containers: - args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --kubelet-use-node-status-port - --metric-resolution=15s - --kubelet-insecure-tls image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.6.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 name: metrics-server ports: - containerPort: 4443 name: https protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /readyz port: https scheme: HTTPS initialDelaySeconds: 20 periodSeconds: 10 resources: requests: cpu: 100m memory: 200Mi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 volumeMounts: - mountPath: /tmp name: tmp-dir nodeSelector: kubernetes.io/os: linux priorityClassName: system-cluster-critical serviceAccountName: metrics-server volumes: - emptyDir: {} name: tmp-dir --- apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: labels: k8s-app: metrics-server name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: name: metrics-server namespace: kube-system version: v1beta1 versionPriority: 100
执行部署命令:
kubectl apply -f metrics-server.yaml
简单试用一下:
[root@centos-k8s-master ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6d8c4cb4d-874c6 1/1 Running 0 2d14h coredns-6d8c4cb4d-vrxjv 1/1 Running 0 2d14h etcd-centos-k8s-master.local 1/1 Running 1 (2d13h ago) 2d14h kube-apiserver-centos-k8s-master.local 1/1 Running 0 26m kube-controller-manager-centos-k8s-master.local 1/1 Running 2 (26m ago) 2d14h kube-flannel-ds-bv4rv 1/1 Running 0 2d13h kube-flannel-ds-tv254 1/1 Running 0 2d13h kube-flannel-ds-v9hjg 1/1 Running 0 2d13h kube-proxy-5wfn9 1/1 Running 0 2d13h kube-proxy-lwwzf 1/1 Running 0 2d13h kube-proxy-w6vxq 1/1 Running 0 2d13h kube-scheduler-centos-k8s-master.local 1/1 Running 2 (26m ago) 2d14h metrics-server-55db4f88bc-rbddg 1/1 Running 0 5m46s [root@centos-k8s-master ~]# kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% centos-k8s-master.local 85m 4% 1329Mi 70% centos-k8s-node1.local 28m 1% 581Mi 30% centos-k8s-node2.local 28m 1% 614Mi 32% [root@centos-k8s-master ~]#
命令:scp -r root@192.168.1.12:/root/.kube/ /root/
命令格式:scp [OPTIONS] [[user@]src_host:]file [[user@]dest_host:]file
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。