Kubeadm 安装 Kubernetes1.12 全流程_failed to load admin kubeconfig: open /root/.kube/

kubeadm 安装k8s 1.12

产品需要从k8s 1.6升级到1.12.3最新稳定版，目前kubeadm已经GA，所以这次也准备用上，替代以前的大量ansible脚本工作，由于要支持旧业务的heapster等组件，升级过程有点曲折，特此记录。

1 安装环境准备

1.1检查以下两个文件的输出：

#确保这两个值为1
cat /proc/sys/net/bridge/bridge-nf-call-ip6tables

cat /proc/sys/net/bridge/bridge-nf-call-iptables   

#如果这两个值不为1，则进行如下操作：

#创建/etc/sysctl.d/k8s.conf文件，添加如下内容：

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_forward = 1

#执行下面两条命令使修改生效。

modprobe br_netfilter

sysctl -p /etc/sysctl.d/k8s.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

1.2 关闭selinux,firewalld,swap等

1.3 ntp配置

• 服务器默认已经安装了ntpd和ntpdate服务。如果没有安装，请自行yum安装。

systemctl enable ntpd

systemctl start ntpd
1
2
3

• 配置/etc/ntp.conf

  其中主节点配置如下：
  

• 从节点配置如下：
  

• 注意统一时区为上海

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

• 在从节点上同步时间:
  

• 查看ntp状态

  主节点：
  
  从节点：
  

2 安装docker

yum install -y yum-utils device-mapper-persistent-data

yum install -y docker-ce-18.06.0.ce-3.el7.x86_64 

#安装docker 18.06版本，如果之前安装过更高版本的docker，可能在yum时报错conflict，需要rpm -e 指定的版本

1
2
3
4
5
6

3 k8s环境准备

3.1 将阿里云的kubernetes yum源 sync到本地

3.2 引用自身repo

这里省略了安装nginx等过程，请自行查找自建yum源方法~

3.3 配置hosts，以及各节点SSH免密

3.4 安装kubeadm,kubectl,kubelet

yum install -y kubeadm-1.12.3-0.x86_64 kubectl-1.12.3-0.x86_64 kubelet-1.12.3-0.x86_64

3.5 配置kubelet

systemctl enable kubelet

注意：这里只enable,不可start

通过观察kubeadm的配置文件，可以看出kubelet启动时的启动参数都从哪里读

3.6 kubeadm init安装主节点

kubeadm init --kubernetes-version=v1.12.3 --pod-network-cidr=172.30.0.0/16  --service-cidr=10.96.0.0/16 --ignore-preflight-errors=Swap
#也可以使用kubeadm config images pull 来预先下载镜像
1
2

由于Kubeadm安装的master节点，默认带有taint，不接受一般的pod调度，所以coredns pod无法创建成功

将admin的kubeconfig文件拷贝到各节点的 /root/.kube目录下

cp /etc/kubernetes/admin.conf /root/.kube/config

kubectl taint nodes node1 node-role.kubernetes.io/master- 

#可以去掉污点，让master节点参与工作负载 

#安装flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# flannel 修改为如下配置
1
2
3
4
5
6
7
8
9
10

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "ipMasq": false,
      "plugins": [
        {
          "type": "flannel",
          "ipMasq": false,
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "172.30.0.0/16",
      "Backend": {
        "Type": "host-gw"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.10.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq=false
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162

需要在主节点开启非安全端口，需要在/etc/kubernetes/manifests/kube-apiserver.yaml 进行如下配置：

3.7 从节点安装

从节点安装kubeadm,kubelet,kubectl与主节点相同；

# 从节点joib到集群
kubeadm join 10.6.6.190:6443 --token i3r115.q4ucaz0uio489438 --discovery-token-ca-cert-hash sha256:eb9daf9067902d23069c3910caefc5651d3156809fec18afee17e158c57f6afe --ignore-preflight-errors=Swap 

#如果从节点的镜像不是预先导入的，则也需要通过代理拉取镜像。
#如果忘记了token和ca hash，通过以下命令生成新token和查找hash
kubeadm token create
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
1
2
3
4
5
6
7

主节点scp ~/.kube/config 到从节点对应目录

3.8 heapster安装

wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml 
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml 
wget <https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml> 
1
2
3

#注意！HeapSter会出错 报错信息如下：
E0823 04:43:05.012122       1 manager.go:101] Error in scraping containers from kubelet:192.168.1.67:10255: failed to get all container stats from Kubelet URL "http://192.168.1.67:10 255/stats/container/": Post http://192.168.1.67:10255/stats/container/: dial tcp 192.168.1.67:10255: getsockopt: connection refused

#解决 在yaml文件中进行如下配置
- --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true
- --sink=influxdb:http://monitoring-influxdb.kube-system.svc.cluster.local:8086
1
2
3
4
5
6

influxDb会因为dns的原因，解析错误的地址和8088端口，去尝试绑定

解决：

再次注意：

k8s 1.12版本的 proxy api接口改变了

原来是/api/v1/proxy/namespaces/xxx/pods/xxx

现在是/api/v1/namespaces/xxx/pods/xxx/proxy

通过kubectl cluster-info 可以获取服务列表

访问一个nginx服务

此时heapster的访问路径改为：

curl 10.6.6.190:8080/api/v1/namespaces/kube-system/services/heapster/proxy/api/v1/model/nodes/lct-k8s-1/metrics/cpu/usage

3.9 删除节点操作

#在master节点上执行：

kubectl drain node2 --delete-local-data --force --ignore-daemonsets

kubectl delete node node2

#在node2上执行：

kubeadm reset

ifconfig cni0 down

ip link delete cni0

ifconfig flannel.1 down

ip link delete flannel.1

rm -rf /var/lib/cni/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

3.10 解决CNI0 SNAT问题

# 添加一条规则，放在首位
iptables -t nat -I POSTROUTING 1 -s 172.30.0.0/16 -d 172.30.0.0/16 -j ACCEPT

#删除此条规则

iptables -t nat -D POSTROUTING  -s 172.30.0.0/16 -d 172.30.0.0/16 -j ACCEPT
1
2
3
4
5
6

3.11 添加gpu支持

• 安装nv-docker

# If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers 
docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f sudo yum remove nvidia-docker 

# Add the package repositories distribution=$(. /etc/os-release;echo $ID$VERSION_ID) 
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \ sudo tee /etc/yum.repos.d/nvidia-docker.repo 

# Install nvidia-docker2 and reload the Docker daemon configuration 
sudo yum install -y nvidia-docker2 sudo pkill -SIGHUP dockerd 

# Test nvidia-smi with the latest official CUDA image 
docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi
1
2
3
4
5
6
7
8
9
10
11

• 安装device-plugin

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.11/nvidia-device-plugin.yml

• kubelet配置

--feature-gates=DevicePlugins=true

• 重启kubelet服务