devbox
繁依Fanyi0
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

编译安装kubernetes 1.15.3

作者：繁依Fanyi0 | 2024-05-13 03:16:09

赞

踩

afpcjhn

环境：

操作系统： win10 on Ubuntu 18.04.3 LTS (GNU/Linux 4.4.0-18362-Microsoft x86_64)
GO 版本 go version go1.12.9 linux/amd64
GCC 版本 gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
make 版本 Make 4.1
部署服务器： etcd 192.168.30.31 kubernetes 192.168.30.32
部署环境：centos 7.6
业务部署目录：/apps/业务
K8S 网段 10.66.0.0/16
docker pod 网段 10.67.0.0/16
使用kube-router 提供网络服务

go 环境部署

  1. cd /usr/local/src
  2. wget https://dl.google.com/go/go1.12.9.linux-amd64.tar.gz
  3. tar -xvf go1.12.9.linux-amd64.tar.gz
  4. mv go ../
  5. vi /etc/profile
  6. export GOPATH=/mnt/e/work/go
  7. export GOBIN=/mnt/e/work/go/bin
  8. PATH=$PATH:/usr/local/go/bin:$HOME/bin:$GOBIN
  9. export PATH
  10. source /etc/profile
  11. go version 
  12. root@Qist:~# go version
  13. go version go1.12.9 linux/amd64
  14. 创建go 工作目录BIN目录
  15. mkdir -p /mnt/e/work/go/{bin,src,pkg}

安装编译kubernetes 1.15.3 依赖

apt -y install make gcc

编译kubernetes 1.15.3

  1. wget https://github.com/kubernetes/kubernetes/archive/v1.15.3.tar.gz
  2. tar -xvf v1.15.3.tar.gz
  3. cd kubernetes-1.15.3/
  4. make
  5. cd ./_output/local/bin/linux/amd64
  6. mkdir -p /mnt/e/work/k8s/bin
  7. cp -pdr kube* /mnt/e/work/k8s/bin/

编译证书生成工具

  1. go get  github.com/cloudflare/cfssl/cmd/cfssl
  2. go get  github.com/cloudflare/cfssl/cmd/cfssljson

部署etcd

  1. #etcd 使用二进制方式部署编译依赖会用到墙外的东西环境设置比较麻烦 etcd 节点服务器操作
  2. wget https://github.com/etcd-io/etcd/releases/download/v3.4.0/etcd-v3.4.0-linux-amd64.tar.gz
  3. mkdir -p /apps/etcd/{bin,conf,ssl,data}
  4. # 数据存储目录
  5. mkdir -p /apps/etcd/data/default.etcd
  6. # 创建 etcd 用户
  7.  useradd etcd -s /sbin/nologin -M
  8.  # 解压etcd
  9.  tar -xvf  etcd-v3.4.0-linux-amd64.tar.gz
  10.  # cp 可执行文件到工作目录
  11.  cd etcd-v3.4.0-linux-amd64/
  12.  cp -pdr etcd  etcdctl /apps/etcd/bin/
  13.  # etcd 证书 在win on Ubuntu 操作
  14.  mkdir -p /apps/work/k8s/cfssl/ && \
  15. cat << EOF | tee /apps/work/k8s/cfssl/ca-config.json
  16. {
  17. "signing": {
  18. "default": {
  19. "expiry": "87600h"
  20. },
  21. "profiles": {
  22. "kubernetes": {
  23. "usages": [
  24. "signing",
  25. "key encipherment",
  26. "server auth",
  27. "client auth"
  28. ],
  29. "expiry": "87600h"
  30. }
  31. }
  32. }
  33. }
  34. EOF
  35. # 创建etcd ca证书配置
  36. mkdir -p /apps/work/k8s/cfssl/etcd
  37. cat << EOF | tee /apps/work/k8s/cfssl/etcd/etcd-ca-csr.json
  38. {
  39. "CN": "etcd",
  40. "key": {
  41. "algo": "rsa",
  42. "size": 2048
  43. },
  44. "names": [
  45. {
  46. "C": "CN",
  47. "ST": "GuangDong",
  48. "L": "GuangZhou",
  49. "O": "cluster",
  50. "OU": "cluster"
  51. }
  52. ]
  53. }
  54. EOF
  55. # 生成 ETCD CA 证书和私钥
  56. mkdir -p /apps/work/k8s/cfssl/pki/etcd
  57. cfssl gencert -initca /apps/work/k8s/cfssl/etcd/etcd-ca-csr.json | cfssljson -bare /apps/work/k8s/cfssl/pki/etcd/etcd-ca
  58. # 创建 ETCD Server 证书 
  59. export ETCD_SERVER_IPS=" \
  60. \"192.168.30.31\" \
  61. " && \
  62. export ETCD_SERVER_HOSTNAMES=" \
  63. \"etcd\" \
  64. " && \
  65. cat << EOF | tee /apps/work/k8s/cfssl/etcd/etcd_server.json
  66. {
  67. "CN": "etcd",
  68. "hosts": [
  69. "127.0.0.1",
  70. ${ETCD_SERVER_IPS},
  71. ${ETCD_SERVER_HOSTNAMES}
  72. ],
  73. "key": {
  74. "algo": "rsa",
  75. "size": 2048
  76. },
  77. "names": [
  78. {
  79. "C": "CN",
  80. "ST": "GuangDong",
  81. "L": "GuangZhou",
  82. "O": "cluster",
  83. "OU": "cluster"
  84. }
  85. ]
  86. }
  87. EOF
  88. # 生成 ETCD Server 证书和私钥
  89. cfssl gencert \
  90. -ca=/apps/work/k8s/cfssl/pki/etcd/etcd-ca.pem \
  91. -ca-key=/apps/work/k8s/cfssl/pki/etcd/etcd-ca-key.pem \
  92. -config=/apps/work/k8s/cfssl/ca-config.json \
  93. -profile=kubernetes \
  94. /apps/work/k8s/cfssl/etcd/etcd_server.json | \
  95. cfssljson -bare /apps/work/k8s/cfssl/pki/etcd/etcd_server
  96. # 创建 ETCD Member 证书
  97. export ETCD_MEMBER_1_IP=" \
  98.     \"192.168.30.31\" \
  99. " && \
  100. export ETCD_MEMBER_1_HOSTNAMES="etcd\
  101. " && \
  102. cat << EOF | tee /apps/work/k8s/cfssl/etcd/${ETCD_MEMBER_1_HOSTNAMES}.json
  103. {
  104.   "CN": "etcd",
  105.   "hosts": [
  106.     "127.0.0.1",
  107.     ${ETCD_MEMBER_1_IP},
  108.     "${ETCD_MEMBER_1_HOSTNAMES}"
  109.   ],
  110.   "key": {
  111.     "algo": "rsa",
  112.     "size": 2048
  113.   },
  114.   "names": [
  115.     {
  116.       "C": "CN",
  117.       "ST": "GuangDong",
  118.       "L": "GuangZhou",
  119.       "O": "cluster",
  120.       "OU": "cluster"
  121.     }
  122.   ]
  123. }
  124. EOF
  125. ##### 生成 ETCD Member 1 证书和私钥
  126. cfssl gencert \
  127.     -ca=/apps/work/k8s/cfssl/pki/etcd/etcd-ca.pem \
  128.     -ca-key=/apps/work/k8s/cfssl/pki/etcd/etcd-ca-key.pem \
  129.     -config=/apps/work/k8s/cfssl/ca-config.json \
  130.     -profile=kubernetes \
  131.     /apps/work/k8s/cfssl/etcd/${ETCD_MEMBER_1_HOSTNAMES}.json | \
  132.     cfssljson -bare /apps/work/k8s/cfssl/pki/etcd/etcd_member_${ETCD_MEMBER_1_HOSTNAMES}
  133. # 创建 ETCD Client 配置文件
  134. cat << EOF | tee /apps/work/k8s/cfssl/etcd/etcd_client.json
  135. {
  136. "CN": "client",
  137. "hosts": [""], 
  138. "key": {
  139. "algo": "rsa",
  140. "size": 2048
  141. },
  142. "names": [
  143. {
  144. "C": "CN",
  145. "ST": "GuangDong",
  146. "L": "GuangZhou",
  147. "O": "cluster",
  148. "OU": "cluster"
  149. }
  150. ]
  151. }
  152. EOF
  153. #生成 ETCD Client 证书和私钥
  154.  
  155. cfssl gencert \
  156. -ca=/apps/work/k8s/cfssl/pki/etcd/etcd-ca.pem \
  157. -ca-key=/apps/work/k8s/cfssl/pki/etcd/etcd-ca-key.pem \
  158. -config=/apps/work/k8s/cfssl/ca-config.json \
  159. -profile=kubernetes \
  160. /apps/work/k8s/cfssl/etcd/etcd_client.json | \
  161. cfssljson -bare /apps/work/k8s/cfssl/pki/etcd/etcd_client
  162. # 复制证书到etcd 节点服务器
  163. scp -r /apps/work/k8s/cfssl/pki/etcd/*  192.168.30.31:/apps/etcd/ssl
  164. # 配置etcd 启动文件 etcd 服务器操作
  165. vi /apps/etcd/conf/etcd
  166. ETCD_OPTS="--name=etcd \
  167.            --data-dir=/apps/etcd/data/default.etcd \
  168.            --listen-peer-urls=https://192.168.30.31:2380 \
  169.            --listen-client-urls=https://192.168.30.31:2379,https://127.0.0.1:2379 \
  170.            --advertise-client-urls=https://192.168.30.31:2379 \
  171.            --initial-advertise-peer-urls=https://192.168.30.31:2380 \
  172.            --initial-cluster=etcd=https://192.168.30.31:2380\
  173.            --initial-cluster-token=node4=etcd=https://192.168.30.31:2380 \
  174.            --initial-cluster-state=new \
  175.            --heartbeat-interval=6000 \
  176.            --election-timeout=30000 \
  177.            --snapshot-count=5000 \
  178.            --auto-compaction-retention=1 \
  179.            --max-request-bytes=33554432 \
  180.            --quota-backend-bytes=17179869184 \
  181.            --trusted-ca-file=/apps/etcd/ssl/etcd-ca.pem \
  182.            --cert-file=/apps/etcd/ssl/etcd_server.pem \
  183.            --key-file=/apps/etcd/ssl/etcd_server-key.pem \
  184.            --peer-cert-file=/apps/etcd/ssl/etcd_member_etcd.pem \
  185.            --peer-key-file=/apps/etcd/ssl/etcd_member_etcd-key.pem \
  186.            --peer-client-cert-auth \
  187.            --peer-trusted-ca-file=/apps/etcd/ssl/etcd-ca.pem"
  188. # 配置etcd 启动service 文件
  189. vi /usr/lib/systemd/system/etcd.service
  190. [Unit]
  191. Description=Etcd Server
  192. After=network.target
  193. After=network-online.target
  194. Wants=network-online.target
  195. [Service]
  196. Type=notify
  197. LimitNOFILE=1024000
  198. LimitNPROC=1024000
  199. LimitCORE=infinity
  200. LimitMEMLOCK=infinity
  201. User=etcd
  202. Group=etcd
  203. EnvironmentFile=-/apps/etcd/conf/etcd
  204. ExecStart=/apps/etcd/bin/etcd $ETCD_OPTS
  205. Restart=on-failure
  206. [Install]
  207. WantedBy=multi-user.target
  208.  
  209. # etcd 目录 etcd 用户权限
  210. chown -R etcd.etcd /apps/etcd
  211. # 启动 etcd
  212. systemctl start etcd
  213. # 设置开机启动
  214. systemctl enable etcd
  215. # 设置环境变量
  216.  /etc/profile
  217.  export ETCDCTL_API=3
  218. export ENDPOINTS=https://192.168.30.31:2379
  219. #执行
  220. source /etc/profile
  221. vim ~/.bashrc
  222. alias etcdctl='/apps/etcd/bin/etcdctl --endpoints=${ENDPOINTS} --cacert=/apps/etcd/ssl/etcd-ca.pem'
  223. source ~/.bashrc
  224. # 查看状态
  225. etcdctl endpoint health
  226. [root@etcd ~]# etcdctl endpoint health
  227. https://192.168.30.31:2379 is healthy: successfully committed proposal: took = 16.707114ms

kube-apiserver 部署

  1. # win on Ubuntu 操作
  2. #创建kube-apiserver ca证书配置
  3. mkdir -p /apps/work/k8s/cfssl/k8s
  4. cat << EOF | tee /apps/work/k8s/cfssl/k8s/k8s-ca-csr.json
  5. {
  6. "CN": "kubernetes",
  7. "key": {
  8. "algo": "rsa",
  9. "size": 2048
  10. },
  11. "names": [
  12. {
  13. "C": "CN",
  14. "ST": "GuangDong",
  15. "L": "GuangZhou",
  16. "O": "cluster",
  17. "OU": "cluster"
  18. }
  19. ]
  20. }
  21. EOF
  22. #生成 Kubernetes CA 证书和私钥
  23.  
  24. mkdir -p /apps/work/k8s/cfssl/pki/k8s
  25. cfssl gencert -initca /apps/work/k8s/cfssl/k8s/k8s-ca-csr.json | \
  26. cfssljson -bare /apps/work/k8s/cfssl/pki/k8s/k8s-ca
  27. #创建 Kubernetes API Server 证书配置文件
  28. export K8S_APISERVER_VIP=" \
  29. \"192.168.30.32\" \
  30. " && \
  31. export K8S_APISERVER_SERVICE_CLUSTER_IP="10.66.0.1" && \
  32. export K8S_APISERVER_HOSTNAME="api.k8s.cluster.local" && \
  33. export K8S_CLUSTER_DOMAIN_SHORTNAME="cluster" && \
  34. export K8S_CLUSTER_DOMAIN_FULLNAME="cluster.local" && \
  35. cat << EOF | tee /apps/work/k8s/cfssl/k8s/k8s_apiserver.json
  36. {
  37. "CN": "kubernetes",
  38. "hosts": [
  39. "127.0.0.1",
  40. ${K8S_APISERVER_VIP}
  41. "${K8S_APISERVER_SERVICE_CLUSTER_IP}", 
  42. "${K8S_APISERVER_HOSTNAME}",
  43. "kubernetes",
  44. "kubernetes.default",
  45. "kubernetes.default.svc",
  46. "kubernetes.default.svc.${K8S_CLUSTER_DOMAIN_SHORTNAME}",
  47. "kubernetes.default.svc.${K8S_CLUSTER_DOMAIN_FULLNAME}" 
  48. ],
  49. "key": {
  50. "algo": "rsa",
  51. "size": 2048
  52. },
  53. "names": [
  54. {
  55. "C": "CN",
  56. "ST": "GuangDong",
  57. "L": "GuangZhou",
  58. "O": "cluster",
  59. "OU": "cluster"
  60. }
  61. ]
  62. }
  63. EOF
  64. #生成 Kubernetes API Server 证书和私钥
  65.  
  66. cfssl gencert \
  67. -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  68. -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  69. -config=/apps/work/k8s/cfssl/ca-config.json \
  70. -profile=kubernetes \
  71. /apps/work/k8s/cfssl/k8s/k8s_apiserver.json | \
  72. cfssljson -bare /apps/work/k8s/cfssl/pki/k8s/k8s_server
  73. # 创建 Kubernetes webhook 证书配置文件
  74. cat << EOF | tee /apps/work/k8s/cfssl/k8s/aggregator.json
  75. {
  76.   "CN": "aggregator",
  77.   "hosts": [""], 
  78.   "key": {
  79.     "algo": "rsa",
  80.     "size": 2048
  81.   },
  82.   "names": [
  83.     {
  84.       "C": "CN",
  85.       "ST": "GuangDong",
  86.       "L": "GuangZhou",
  87.       "O": "cluster",
  88.       "OU": "cluster"
  89.     }
  90.   ]
  91. }
  92. EOF
  93.  
  94. ##### 生成 Kubernetes webhook 证书和私钥
  95. cfssl gencert \
  96.     -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  97.     -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  98.     -config=/apps/work/k8s/cfssl/ca-config.json \
  99.     -profile=kubernetes \
  100.     /apps/work/k8s/cfssl/k8s/aggregator.json | \
  101.     cfssljson -bare /apps/work/k8s/cfssl/pki/k8s/aggregator
  102.     # 远程服务器创建目录
  103.     mkdir -p /apps/kubernetes/{bin,conf,config,kubelet-plugins,log,ssl}
  104.  #证书到分发到 192.168.30.32
  105.  scp -r /apps/work/k8s/cfssl/pki/k8s 192.168.30.32:/apps/kubernetes/ssl/k8s
  106.  # cp  etcd_client 证书
  107.   scp -r /apps/work/k8s/cfssl/pki/etcd/etcd_client* 192.168.30.32:/apps/kubernetes/ssl/etcd
  108.     scp -r /apps/work/k8s/cfssl/pki/etcd/etcd-ca.pem 192.168.30.32:/apps/kubernetes/ssl/etcd
  109. # 分发kubernetes 二进制文件到远程服务器把所有的二进制都cp 过去
  110. scp -r /mnt/e/work/k8s/bin/* 192.168.30.32:/apps/kubernetes/bin
  111. # 远程服务器操作192.168.30.32
  112.  # 创建 k8s 用户
  113.  useradd k8s -s /sbin/nologin -M
  114.  # encryption-config.yaml 生成
  115.  cd /apps/kubernetes/config
  116.  export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
  117.  cat > encryption-config.yaml <<EOF
  118. kind: EncryptionConfig
  119. apiVersion: v1
  120. resources:
  121.   - resources:
  122.       - secrets
  123.     providers:
  124.       - aescbc:
  125.           keys:
  126.             - name: key1
  127.               secret: ${ENCRYPTION_KEY}
  128.       - identity: {}
  129. EOF
  130. # Apiserver配置文件生成
  131. cd  /apps/kubernetes/conf
  132. vi  kube-apiserver
  133. KUBE_APISERVER_OPTS="--logtostderr=false \
  134.         --bind-address=192.168.30.32 \
  135.         --advertise-address=192.168.30.32 \
  136.         --secure-port=5443 \
  137.         --insecure-port=0 \
  138.         --service-cluster-ip-range=10.66.0.0/16 \
  139.         --service-node-port-range=30000-65000 \
  140.         --etcd-cafile=/apps/kubernetes/ssl/etcd/etcd-ca.pem \
  141.         --etcd-certfile=/apps/kubernetes/ssl/etcd/etcd_client.pem \
  142.         --etcd-keyfile=/apps/kubernetes/ssl/etcd/etcd_client-key.pem \
  143.         --etcd-prefix=/registry \
  144.         --etcd-servers=https://192.168.30.31:2379 \
  145.         --client-ca-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  146.         --tls-cert-file=/apps/kubernetes/ssl/k8s/k8s_server.pem \
  147.         --tls-private-key-file=/apps/kubernetes/ssl/k8s/k8s_server-key.pem \
  148.         --kubelet-client-certificate=/apps/kubernetes/ssl/k8s/k8s_server.pem \
  149.         --kubelet-client-key=/apps/kubernetes/ssl/k8s/k8s_server-key.pem \
  150.         --service-account-key-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  151.         --requestheader-client-ca-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  152.         --proxy-client-cert-file=/apps/kubernetes/ssl/k8s/aggregator.pem \
  153.         --proxy-client-key-file=/apps/kubernetes/ssl/k8s/aggregator-key.pem \
  154.         --requestheader-allowed-names=aggregator \
  155.         --requestheader-group-headers=X-Remote-Group \
  156.         --requestheader-extra-headers-prefix=X-Remote-Extra- \
  157.         --requestheader-username-headers=X-Remote-User \
  158.         --enable-aggregator-routing=true \
  159.         --anonymous-auth=false \
  160.         --allow-privileged=true \
  161.         --experimental-encryption-provider-config=/apps/kubernetes/config/encryption-config.yaml \
  162.         --enable-admission-plugins=AlwaysPullImages,DefaultStorageClass,DefaultTolerationSeconds,LimitRanger,NamespaceExists,NamespaceLifecycle,NodeRestriction,OwnerReferencesPermissionEnforcement,PodNodeSelector,PersistentVolumeClaimResize,PodPreset,PodTolerationRestriction,ResourceQuota,ServiceAccount,StorageObjectInUseProtection MutatingAdmissionWebhook ValidatingAdmissionWebhook \
  163.         --disable-admission-plugins=DenyEscalatingExec,ExtendedResourceToleration,ImagePolicyWebhook,LimitPodHardAntiAffinityTopology,NamespaceAutoProvision,Priority,EventRateLimit,PodSecurityPolicy \
  164.         --cors-allowed-origins=.* \
  165.         --enable-swagger-ui \
  166.         --runtime-config=api/all=true \
  167.         --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
  168.         --authorization-mode=Node,RBAC \
  169.         --apiserver-count=1 \
  170.         --audit-log-maxage=30 \
  171.         --audit-log-maxbackup=3 \
  172.         --audit-log-maxsize=100 \
  173.         --kubelet-https \
  174.         --event-ttl=1h \
  175.         --feature-gates=RotateKubeletServerCertificate=true,RotateKubeletClientCertificate=true \
  176.         --enable-bootstrap-token-auth=true \
  177.         --audit-log-path=/apps/kubernetes/log/api-server-audit.log \
  178.         --alsologtostderr=true \
  179.         --log-dir=/apps/kubernetes/log \
  180.         --v=2 \
  181.         --endpoint-reconciler-type=lease \
  182.         --max-mutating-requests-inflight=100 \
  183.         --max-requests-inflight=500 \
  184.         --target-ram-mb=6000"
  185. #  kube-apiserver systemd文件
  186. vi /usr/lib/systemd/system/kube-apiserver.service
  187. [Unit]
  188. Description=Kubernetes API Server
  189. Documentation=https://github.com/kubernetes/kubernetes
  190. [Service]
  191. Type=notify
  192. LimitNOFILE=1024000
  193. LimitNPROC=1024000
  194. LimitCORE=infinity
  195. LimitMEMLOCK=infinity
  196. EnvironmentFile=-/apps/kubernetes/conf/kube-apiserver
  197. ExecStart=/apps/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
  198. Restart=on-failure
  199. RestartSec=5
  200. User=k8s
  201. [Install]
  202. WantedBy=multi-user.target
  203. # /apps/kubernetes 目录k8s 权限
  204. chown -R k8s.k8s /apps/kubernetes
  205. # 启动kube-apiserver
  206. systemctl start kube-apiserver
  207. # 设置开机启动
  208. systemctl enable kube-apiserver
  209. #创建admin管理员证书  win on Ubuntu  操作
  210. cat << EOF | tee /apps/work/k8s/cfssl/k8s/k8s_apiserver_admin.json
  211. {
  212.   "CN": "admin",
  213.   "hosts": [""], 
  214.   "key": {
  215.     "algo": "rsa",
  216.     "size": 2048
  217.   },
  218.   "names": [
  219.     {
  220.       "C": "CN",
  221.       "ST": "GuangDong",
  222.       "L": "GuangZhou",
  223.       "O": "system:masters",
  224.       "OU": "Kubernetes-manual"
  225.     }
  226.   ]
  227. }
  228. EOF
  229.  
  230. cfssl gencert -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  231.     -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  232.     -config=/apps/work/k8s/cfssl/ca-config.json \
  233.     -profile=kubernetes \
  234.         /apps/work/k8s/cfssl/k8s/k8s_apiserver_admin.json | \
  235.         cfssljson -bare /apps/work/k8s/cfssl/pki/k8s/k8s_apiserver_admin
  236.  
  237. # 创建admin管理员登录 admin.kubeconfig
  238. export KUBE_APISERVER="https://192.168.30.32:5443"
  239. kubectl config set-cluster kubernetes \
  240. --certificate-authority=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  241. --embed-certs=true  \
  242. --server=${KUBE_APISERVER} \
  243. --kubeconfig=admin.kubeconfig
  244.  
  245.  kubectl config set-credentials admin \
  246.  --client-certificate=/apps/work/k8s/cfssl/pki/k8s/k8s_apiserver_admin.pem \
  247.  --client-key=/apps/work/k8s/cfssl/pki/k8s/k8s_apiserver_admin-key.pem \
  248.  --embed-certs=true \
  249.  --kubeconfig=admin.kubeconfig
  250.  
  251. kubectl config set-context kubernetes \
  252. --cluster=kubernetes \
  253. --user=admin \
  254. --namespace=kube-system \
  255. --kubeconfig=admin.kubeconfig
  256.  
  257. kubectl config use-context kubernetes --kubeconfig=admin.kubeconfig
  258. # cp 证书到当前用户目录
  259. cp admin.kubeconfig ~/.kube/config
  260. # 验证kube-apiserver 是否正常
  261. kubectl cluster-info 
  262. [root@]~]#kubectl cluster-info
  263. Kubernetes master is running at https://192.168.30.32:5443

kube_scheduler 部署

  1. #  生成kube_scheduler访问kube-apiserver 证书win on Ubuntu  操作
  2. cat << EOF | tee /apps/work/k8s/cfssl/k8s/k8s_scheduler.json
  3. {
  4.   "CN": "system:kube-scheduler",
  5.   "hosts": [""], 
  6.   "key": {
  7.     "algo": "rsa",
  8.     "size": 2048
  9.   },
  10.   "names": [
  11.     {
  12.       "C": "CN",
  13.       "ST": "GuangDong",
  14.       "L": "GuangZhou",
  15.       "O": "system:kube-scheduler",
  16.       "OU": "Kubernetes-manual"
  17.     }
  18.   ]
  19. }
  20. EOF
  21.  
  22. ##  生成 Kubernetes Scheduler 证书和私钥
  23. cfssl gencert \
  24.     -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  25.     -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  26.     -config=/apps/work/k8s/cfssl/ca-config.json \
  27.     -profile=kubernetes \
  28.     /apps/work/k8s/cfssl/k8s/k8s_scheduler.json | \
  29.     cfssljson -bare /apps/work/k8s/cfssl/pki/k8s/k8s_scheduler
  30. #创建kube_scheduler.kubeconfig
  31. kubectl config set-cluster kubernetes \
  32.     --certificate-authority=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  33.     --embed-certs=true \
  34.     --server=${KUBE_APISERVER} \
  35.     --kubeconfig=kube_scheduler.kubeconfig
  36.  
  37. kubectl config set-credentials system:kube-scheduler \
  38.     --client-certificate=/apps/work/k8s/cfssl/pki/k8s/k8s_scheduler.pem \
  39.     --embed-certs=true \
  40.     --client-key=/apps/work/k8s/cfssl/pki/k8s/k8s_scheduler-key.pem \
  41.     --kubeconfig=kube_scheduler.kubeconfig
  42. kubectl config set-context kubernetes \
  43.     --cluster=kubernetes \
  44.     --user=system:kube-scheduler \
  45.     --kubeconfig=kube_scheduler.kubeconfig
  46.  
  47. kubectl config use-context kubernetes --kubeconfig=kube_scheduler.kubeconfig
  48. # cp kube_scheduler.kubeconfig 到远程服务器
  49. scp kube_scheduler.kubeconfig 192.168.30.32:/apps/kubernetes/config
  50. # 远程服务器操作
  51. cd /apps/kubernetes/conf
  52. # 创建kube-scheduler 启动配置文件
  53. vi kube-scheduler
  54. KUBE_SCHEDULER_OPTS=" \
  55.                    --logtostderr=false \
  56.                    --address=0.0.0.0 \
  57.                    --leader-elect=true \
  58.                    --kubeconfig=/apps/kubernetes/config/kube_scheduler.kubeconfig \
  59.                    --authentication-kubeconfig=/apps/kubernetes/config/kube_scheduler.kubeconfig \
  60.                    --authorization-kubeconfig=/apps/kubernetes/config/kube_scheduler.kubeconfig \
  61.                    --alsologtostderr=true \
  62.                    --kube-api-qps=100 \
  63.                    --kube-api-burst=100 \
  64.                    --log-dir=/apps/kubernetes/log \
  65.                    --v=2"
  66. # 创建/kube-scheduler 启动文件
  67.  
  68. vi /usr/lib/systemd/system/kube-scheduler.service
  69. [Unit]
  70. Description=Kubernetes Scheduler
  71. Documentation=https://github.com/kubernetes/kubernetes
  72. [Service]
  73. LimitNOFILE=1024000
  74. LimitNPROC=1024000
  75. LimitCORE=infinity
  76. LimitMEMLOCK=infinity
  77. EnvironmentFile=-/apps/kubernetes/conf/kube-scheduler
  78. ExecStart=/apps/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
  79. Restart=on-failure
  80. RestartSec=5
  81. User=k8s
  82. [Install]
  83. WantedBy=multi-user.target
  84. # 给新创建文件 k8s 用户权限
  85. chown -R k8s.k8s /apps/kubernetes
  86. # 启动 kube-scheduler
  87. systemctl start kube-scheduler
  88. # 设置开启启动
  89. systemctl enable kube-scheduler
  90. # 验证状态
  91.  kubectl get cs
  92.  [root@]~]# kubectl get cs
  93. NAME                 STATUS    MESSAGE             ERROR
  94. controller-manager   Healthy   ok
  95. scheduler            Healthy   ok
  96. etcd-0               Healthy   {"health":"true"}

kube-controller-manager部署

  1. # 生成kube-controller-manager访问kube-apiserver 证书 win on Ubuntu  操作
  2. cat << EOF | tee /apps/work/k8s/cfssl/k8s/k8s_controller_manager.json
  3. {
  4.   "CN": "system:kube-controller-manager",
  5.   "hosts": [""], 
  6.   "key": {
  7.     "algo": "rsa",
  8.     "size": 2048
  9.   },
  10.   "names": [
  11.     {
  12.       "C": "CN",
  13.       "ST": "GuangDong",
  14.       "L": "GuangZhou",
  15.       "O": "system:kube-controller-manager",
  16.       "OU": "Kubernetes-manual"
  17.     }
  18.   ]
  19. }
  20. EOF
  21.  
  22. ## 生成 Kubernetes Controller Manager 证书和私钥
  23. cfssl gencert \
  24.     -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  25.     -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  26.     -config=/apps/work/k8s/cfssl/ca-config.json \
  27.     -profile=kubernetes \
  28.     /apps/work/k8s/cfssl/k8s/k8s_controller_manager.json | \
  29.     cfssljson -bare /apps/work/k8s/cfssl/pki/k8s/k8s_controller_manager
  30. # 创建kube_controller_manager.kubeconfig
  31. kubectl config set-cluster kubernetes \
  32.    --certificate-authority=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  33.    --embed-certs=true \
  34.    --server=${KUBE_APISERVER} \
  35.    --kubeconfig=kube_controller_manager.kubeconfig
  36.  
  37. kubectl config set-credentials system:kube-controller-manager \
  38.    --client-certificate=/apps/work/k8s/cfssl/pki/k8s/k8s_controller_manager.pem \
  39.    --embed-certs=true \
  40.    --client-key=/apps/work/k8s/cfssl/pki/k8s/k8s_controller_manager-key.pem \
  41.    --kubeconfig=kube_controller_manager.kubeconfig
  42.  
  43. kubectl config set-context kubernetes \
  44.    --cluster=kubernetes \
  45.    --user=system:kube-controller-manager \
  46.    --kubeconfig=kube_controller_manager.kubeconfig
  47.  
  48. kubectl config use-context kubernetes --kubeconfig=kube_controller_manager.kubeconfig
  49. # cp kube_controller_manager.kubeconfig 到远程服务器
  50. scp kube_controller_manager.kubeconfig 192.168.30.32:/apps/kubernetes/config
  51. #  远程服务器操作
  52. cd /apps/kubernetes/conf
  53. vi kube-controller-manager
  54. KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
  55.  --leader-elect=true \
  56.  --address=0.0.0.0 \
  57.  --service-cluster-ip-range=10.66.0.0/16 \
  58.  --cluster-cidr=10.67.0.0/16 \
  59.  --node-cidr-mask-size=24 \
  60.  --cluster-name=kubernetes \
  61.  --allocate-node-cidrs=true \
  62.  --kubeconfig=/apps/kubernetes/config/kube_controller_manager.kubeconfig \
  63.  --authentication-kubeconfig=/apps/kubernetes/config/kube_controller_manager.kubeconfig \
  64.  --authorization-kubeconfig=/apps/kubernetes/config/kube_controller_manager.kubeconfig \
  65.  --use-service-account-credentials=true \
  66.  --client-ca-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  67.  --requestheader-client-ca-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  68.  --node-monitor-grace-period=40s \
  69.  --node-monitor-period=5s \
  70.  --pod-eviction-timeout=5m0s \
  71.  --terminated-pod-gc-threshold=50 \
  72.  --alsologtostderr=true \
  73.  --cluster-signing-cert-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  74.  --cluster-signing-key-file=/apps/kubernetes/ssl/k8s/k8s-ca-key.pem  \
  75.  --deployment-controller-sync-period=10s \
  76.  --experimental-cluster-signing-duration=86700h0m0s \
  77.  --enable-garbage-collector=true \
  78.  --root-ca-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  79.  --service-account-private-key-file=/apps/kubernetes/ssl/k8s/k8s-ca-key.pem \
  80.  --feature-gates=RotateKubeletServerCertificate=true,RotateKubeletClientCertificate=true \
  81.  --controllers=*,bootstrapsigner,tokencleaner \
  82.  --horizontal-pod-autoscaler-use-rest-clients=true \
  83.  --horizontal-pod-autoscaler-sync-period=10s \
  84.  --flex-volume-plugin-dir=/apps/kubernetes/kubelet-plugins/volume \
  85.  --tls-cert-file=/apps/kubernetes/ssl/k8s/k8s_controller_manager.pem \
  86.  --tls-private-key-file=/apps/kubernetes/ssl/k8s/k8s_controller_manager-key.pem \
  87.  --kube-api-qps=100 \
  88.  --kube-api-burst=100 \
  89.  --log-dir=/apps/kubernetes/log \
  90.  --v=2"
  91.  # 创建启动文件kube-controller-manager
  92.  vi /usr/lib/systemd/system/kube-controller-manager.service
  93.  [Unit]
  94. Description=Kubernetes Controller Manager
  95. Documentation=https://github.com/kubernetes/kubernetes
  96. [Service]
  97. LimitNOFILE=1024000
  98. LimitNPROC=1024000
  99. LimitCORE=infinity
  100. LimitMEMLOCK=infinity
  101. EnvironmentFile=-/apps/kubernetes/conf/kube-controller-manager
  102. ExecStart=/apps/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
  103. Restart=on-failure
  104. RestartSec=5
  105. User=k8s
  106. [Install]
  107. WantedBy=multi-user.target
  108. # 给新创建文件 k8s 用户权限
  109. chown -R k8s.k8s /apps/kubernetes
  110. # 启动kube-controller-manager
  111. systemctl start kube-controller-manager
  112. # 设置开机启动
  113. systemctl enable kube-controller-manager
  114. # 验证状态
  115. [root@]~]#kubectl get cs
  116. NAME                 STATUS    MESSAGE             ERROR
  117. scheduler            Healthy   ok
  118. controller-manager   Healthy   ok
  119. etcd-0               Healthy   {"health":"true"}
  120. # 配置 kube-controller-manager，kubelet 、kube-scheduler 访问kube-api 用户授权
  121. 授予 kubernetes API 的权限
  122. kubectl create clusterrolebinding controller-node-clusterrolebing --clusterrole=system:kube-controller-manager  --user=system:kube-controller-manager
  123. kubectl create clusterrolebinding scheduler-node-clusterrolebing  --clusterrole=system:kube-scheduler --user=system:kube-scheduler
  124. kubectl create clusterrolebinding controller-manager:system:auth-delegator --user system:kube-controller-manager --clusterrole system:auth-delegator
  125. 授予 kubernetes 证书访问 kubelet API 的权限
  126. kubectl create clusterrolebinding --user system:serviceaccount:kube-system:default kube-system-cluster-admin --clusterrole cluster-admin
  127. kubectl create clusterrolebinding kubelet-node-clusterbinding --clusterrole=system:node --group=system:nodes
  128. kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

docker 部署

  1. # 远程服务器节点操作
  2. # 使用阿里源
  3. cat > /etc/yum.repos.d/docker-ce.repo << EOF
  4. [docker-ce-stable]
  5. name=Docker CE Stable - \$basearch
  6. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/\$basearch/stable
  7. enabled=1
  8. gpgcheck=1
  9. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  10.  
  11. [docker-ce-stable-debuginfo]
  12. name=Docker CE Stable - Debuginfo \$basearch
  13. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-\$basearch/stable
  14. enabled=0
  15. gpgcheck=1
  16. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  17.  
  18. [docker-ce-stable-source]
  19. name=Docker CE Stable - Sources
  20. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/stable
  21. enabled=0
  22. gpgcheck=1
  23. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  24.  
  25. [docker-ce-edge]
  26. name=Docker CE Edge - \$basearch
  27. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/\$basearch/edge
  28. enabled=0
  29. gpgcheck=1
  30. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  31.  
  32. [docker-ce-edge-debuginfo]
  33. name=Docker CE Edge - Debuginfo \$basearch
  34. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-\$basearch/edge
  35. enabled=0
  36. gpgcheck=1
  37. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  38.  
  39. [docker-ce-edge-source]
  40. name=Docker CE Edge - Sources
  41. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/edge
  42. enabled=0
  43. gpgcheck=1
  44. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  45.  
  46. [docker-ce-test]
  47. name=Docker CE Test - \$basearch
  48. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/\$basearch/test
  49. enabled=0
  50. gpgcheck=1
  51. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  52.  
  53. [docker-ce-test-debuginfo]
  54. name=Docker CE Test - Debuginfo \$basearch
  55. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-\$basearch/test
  56. enabled=0
  57. gpgcheck=1
  58. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  59.  
  60. [docker-ce-test-source]
  61. name=Docker CE Test - Sources
  62. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/test
  63. enabled=0
  64. gpgcheck=1
  65. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  66.  
  67. [docker-ce-nightly]
  68. name=Docker CE Nightly - \$basearch
  69. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/\$basearch/nightly
  70. enabled=0
  71. gpgcheck=1
  72. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  73.  
  74. [docker-ce-nightly-debuginfo]
  75. name=Docker CE Nightly - Debuginfo \$basearch
  76. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/debug-\$basearch/nightly
  77. enabled=0
  78. gpgcheck=1
  79. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  80.  
  81. [docker-ce-nightly-source]
  82. name=Docker CE Nightly - Sources
  83. baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/source/nightly
  84. enabled=0
  85. gpgcheck=1
  86. gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
  87. EOF
  88. # 安装docker依赖
  89. yum install -y    python-pip python-devel yum-utils device-mapper-persistent-data lvm2 
  90. # 安装docker
  91. yum install -y docker-ce
  92. # 修改docker 启动配置
  93. vi /lib/systemd/system/docker.service
  94. ExecStart= 修改成
  95. ExecStart=/usr/bin/dockerd -H fd:// --graph /apps/docker -H unix:///var/run/docker.sock  --max-concurrent-downloads=20 --log-opt max-size=200M --log-opt max-file=10 --default-ulimit nofile=1024000 --default-ulimit nproc=1024000
  96. # reload service 配置
  97. systemctl daemon-reload
  98. # 重启docker
  99. systemctl restart docker
  100. # 设置开机启动
  101. systemctl enable docker

安装kubelet 依赖

  1. # 远程服务器节点操作
  2. cni 插件安装
  3. mkdir -p /apps/cni/bin
  4. cd  /apps/cni/bin
  5. wget https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz
  6. tar -xvf cni-plugins-linux-amd64-v0.8.2.tgz
  7. rm -f cni-plugins-linux-amd64-v0.8.2.tgz
  8. # 创建/etc/cni/net.d 目录
  9. mkdir -p /etc/cni/net.d
  10. vi  /etc/cni/net.d/10-kuberouter.conflist
  11. {
  12.    "cniVersion":"0.3.0",
  13.    "name":"mynet",
  14.    "plugins":[
  15.       {
  16.          "name":"kubernetes",
  17.          "type":"bridge",
  18.          "bridge":"kube-bridge",
  19.          "isDefaultGateway":true,
  20.          "ipam":{
  21.             "type":"host-local"
  22.          }
  23.       },
  24.       {
  25.          "type":"portmap",
  26.          "capabilities":{
  27.             "snat":true,
  28.             "portMappings":true
  29.          }
  30.       }
  31.    ]
  32. }
  33.  
  34. # lxcfs 安装
  35. yum install -y git automake libtool fuse-devel
  36. git clone git://github.com/lxc/lxcfs 
  37. cd lxcfs/
  38. ./bootstrap.sh
  39. ./configure
  40. make
  41. make install 
  42. # 创建目录
  43. mkdir -p /var/lib/lxcfs/
  44. # 配置 lxcfs 启动文件
  45. vi /usr/lib/systemd/system/lxcfs.service
  46. [Unit]
  47. Description=FUSE filesystem for LXC
  48. ConditionVirtualization=!container
  49. Before=lxc.service
  50. Documentation=man:lxcfs(1)
  51. [Service]
  52. ExecStart=/usr/local/bin/lxcfs /var/lib/lxcfs/
  53. KillMode=process
  54. Restart=on-failure
  55. ExecStopPost=-/bin/fusermount -u /var/lib/lxcfs
  56. Delegate=yes
  57. [Install]
  58. WantedBy=multi-user.target
  59. # 启动lxcfs
  60. systemctl start lxcfs 
  61. # 设置开机启动
  62. systemctl enable lxcfs
  63. # 安装 kubelet 依赖
  64. yum install -y  epel-release
  65. yum install -y   yum-utils  ipvsadm  telnet  wget  net-tools  conntrack  ipset  jq  iptables  curl  sysstat  libseccomp  socat  nfs-utils  fuse  fuse-devel

kubelet 部署

  1. # win on Ubuntu  操作
  2. # 生成 bootstrap Token
  3. # Bootstrap Token 生成
  4. echo "$(head -c 6 /dev/urandom | md5sum | head -c 6)"."$(head -c 16 /dev/urandom | md5sum | head -c 16)"
  5. 9dad00.2ac445bf1cc5e9c2
  6. vi bootstrap.secret.yaml
  7. apiVersion: v1
  8. kind: Secret
  9. metadata:
  10.   # Name MUST be of form "bootstrap-token-<token id>"
  11.   name: bootstrap-token-9dad00
  12.   namespace: kube-system
  13.  
  14. # Type MUST be 'bootstrap.kubernetes.io/token'
  15. type: bootstrap.kubernetes.io/token
  16. stringData:
  17.   # Human readable description. Optional.
  18.   description: "The default bootstrap token generated by 'kubelet '."
  19.  
  20.   # Token ID and secret. Required.
  21.   token-id: 9dad00
  22.   token-secret: 2ac445bf1cc5e9c2
  23.  
  24.   # Allowed usages.
  25.   usage-bootstrap-authentication: "true"
  26.   usage-bootstrap-signing: "true"
  27.  
  28.   # Extra groups to authenticate the token as. Must start with "system:bootstrappers:"
  29.   auth-extra-groups: system:bootstrappers:worker,system:bootstrappers:ingress
  30. ### 创建k8s资源
  31. kubectl create -f bootstrap.secret.yaml
  32. ### 创建bootstrap.clusterrole.yaml
  33. vi bootstrap.clusterrole.yaml
  34. # A ClusterRole which instructs the CSR approver to approve a node requesting a
  35. # serving cert matching its client cert.
  36. kind: ClusterRole
  37. apiVersion: rbac.authorization.k8s.io/v1
  38. metadata:
  39.   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
  40. rules:
  41. - apiGroups: ["certificates.k8s.io"]
  42.   resources: ["certificatesigningrequests/selfnodeserver"]
  43.   verbs: ["create"]
  44.  
  45. kubectl create -f bootstrap.clusterrole.yaml
  46. ### 创建 apiserver-to-kubelet.yaml
  47. vi apiserver-to-kubelet.yaml
  48. apiVersion: rbac.authorization.k8s.io/v1
  49. kind: ClusterRole
  50. metadata:
  51.   annotations:
  52.     rbac.authorization.kubernetes.io/autoupdate: "true"
  53.   labels:
  54.     kubernetes.io/bootstrapping: rbac-defaults
  55.   name: system:kubernetes-to-kubelet
  56. rules:
  57.   - apiGroups:
  58.       - ""
  59.     resources:
  60.       - nodes/proxy
  61.       - nodes/stats
  62.       - nodes/log
  63.       - nodes/spec
  64.       - nodes/metrics
  65.     verbs:
  66.       - "*"
  67. ---
  68. apiVersion: rbac.authorization.k8s.io/v1
  69. kind: ClusterRoleBinding
  70. metadata:
  71.   name: system:kubernetes
  72.   namespace: ""
  73. roleRef:
  74.   apiGroup: rbac.authorization.k8s.io
  75.   kind: ClusterRole
  76.   name: system:kubernetes-to-kubelet
  77. subjects:
  78.   - apiGroup: rbac.authorization.k8s.io
  79.     kind: User
  80.     name: kubernetes
  81. kubectl create -f apiserver-to-kubelet.yaml
  82. ### 查看创建的token
  83. kubeadm token list
  84. # 允许 system:bootstrappers 组用户创建 CSR 请求
  85. kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
  86. # 自动批准 system:bootstrappers 组用户 TLS bootstrapping 首次申请证书的 CSR 请求
  87. kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --group=system:bootstrappers
  88. # 自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求
  89. kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
  90.  
  91. # 自动批准 system:nodes 组用户更新 kubelet 10250 api 端口证书的 CSR 请求
  92. kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes
  93. # 创建bootstrap.kubeconfig
  94. # 设置集群参数
  95. kubectl config set-cluster kubernetes \
  96.   --certificate-authority=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  97.   --embed-certs=true \
  98.   --server=${KUBE_APISERVER} \
  99.   --kubeconfig=bootstrap.kubeconfig
  100. # 设置客户端认证参数
  101. kubectl config set-credentials system:bootstrap:9dad00 \
  102.   --token=9dad00.2ac445bf1cc5e9c2 \
  103.   --kubeconfig=bootstrap.kubeconfig
  104. # 设置上下文参数
  105. kubectl config set-context default \
  106.   --cluster=kubernetes \
  107.   --user=system:bootstrap:9dad00 \
  108.   --kubeconfig=bootstrap.kubeconfig
  109. # 设置默认上下文
  110. kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
  111. # 分发 bootstrap.kubeconfig 到远程节点
  112. scp bootstrap.kubeconfig 192.168.30.32:/apps/kubernetes/conf
  113. # 创建kubelet 启动配置文件 远程节点 操作
  114. cd /apps/kubernetes/conf
  115. vi  kubelet
  116. KUBELET_OPTS="--bootstrap-kubeconfig=/apps/kubernetes/conf/bootstrap.kubeconfig \
  117.               --fail-swap-on=false \
  118.               --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/apps/cni/bin \
  119.               --kubeconfig=/apps/kubernetes/conf/kubelet.kubeconfig \
  120.               --address=192.168.30.32 \
  121.               --node-ip=192.168.30.32 \
  122.               --hostname-override=master \
  123.               --cluster-dns=10.66.0.2 \
  124.               --cluster-domain=cluster.local \
  125.               --authorization-mode=Webhook \
  126.               --authentication-token-webhook=true \
  127.               --client-ca-file=/apps/kubernetes/ssl/k8s/k8s-ca.pem \
  128.               --rotate-certificates=true \
  129.               --cgroup-driver=cgroupfs \
  130.               --healthz-port=10248 \
  131.               --healthz-bind-address=192.168.30.32 \
  132.               --cert-dir=/apps/kubernetes/ssl \
  133.               --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \
  134.               --node-labels=node-role.kubernetes.io/k8s-node=true \
  135.               --serialize-image-pulls=false \
  136.               --enforce-node-allocatable=pods,kube-reserved,system-reserved \
  137.               --pod-manifest-path=/apps/work/kubernetes/manifests \
  138.               --runtime-cgroups=/systemd/system.slice/kubelet.service \
  139.               --kube-reserved-cgroup=/systemd/system.slice/kubelet.service \
  140.               --system-reserved-cgroup=/systemd/system.slice \
  141.               --root-dir=/apps/work/kubernetes/kubelet \
  142.               --log-dir=/apps/kubernetes/log \
  143.               --alsologtostderr=true \
  144.               --logtostderr=false \
  145.               --anonymous-auth=true \
  146.               --image-gc-high-threshold=70 \
  147.               --image-gc-low-threshold=50 \
  148.               --kube-reserved=cpu=500m,memory=512Mi,ephemeral-storage=1Gi \
  149.               --system-reserved=cpu=1000m,memory=1024Mi,ephemeral-storage=1Gi \
  150.               --eviction-hard=memory.available<500Mi,nodefs.available<10% \
  151.               --serialize-image-pulls=false \
  152.               --sync-frequency=30s \
  153.               --resolv-conf=/etc/resolv.conf \
  154.               --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 \
  155.               --image-pull-progress-deadline=30s \
  156.               --v=2 \
  157.               --event-burst=30 \
  158.               --event-qps=15 \
  159.               --kube-api-burst=30 \
  160.               --kube-api-qps=15 \
  161.               --max-pods=200 \
  162.               --pods-per-core=10 \
  163.               --read-only-port=0 \
  164.               --allowed-unsafe-sysctls 'kernel.msg*,kernel.shm*,kernel.sem,fs.mqueue.*,net.*' \
  165.               --volume-plugin-dir=/apps/kubernetes/kubelet-plugins/volume"
  166. # 创建 kubelet 服务文件
  167. vi /usr/lib/systemd/system/kubelet.service
  168. [Unit]
  169. Description=Kubernetes Kubelet
  170. After=docker.service
  171. Requires=docker.service
  172. [Service]
  173. LimitNOFILE=1024000
  174. LimitNPROC=1024000
  175. LimitCORE=infinity
  176. LimitMEMLOCK=infinity
  177. EnvironmentFile=-/apps/kubernetes/conf/kubelet
  178. ExecStart=/apps/kubernetes/bin/kubelet $KUBELET_OPTS
  179. Restart=on-failure
  180. KillMode=process
  181. [Install]
  182. WantedBy=multi-user.target
  183. # 启动 kubelet
  184. systemctl start kubelet 
  185. # 设置开机启动
  186. systemctl enable kubelet
  187. #查看证书是否签发
  188. cd /apps/kubernetes/ssl/
  189. [root@master ssl]# ll
  190. total 12
  191. drwxr-xr-x 2 k8s k8s   75 Jul  4 15:06 etcd
  192. drwxr-xr-x 2 k8s k8s  310 Sep  3 11:18 k8s
  193. -rw------- 1 k8s k8s 1277 Sep  3 10:25 kubelet-client-2019-09-03-10-25-10.pem
  194. lrwxrwxrwx 1 k8s k8s   59 Sep  3 10:25 kubelet-client-current.pem -> /apps/kubernetes/ssl/kubelet-client-2019-09-03-10-25-10.pem
  195. -rw-r--r-- 1 k8s k8s 2153 Sep  3 10:25 kubelet.crt
  196. -rw------- 1 k8s k8s 1675 Sep  3 10:25 kubelet.key
  197. # 查看节点是否正常
  198. kubectl get node
  199. [root@]~]#kubectl get node
  200. NAME     STATUS   ROLES      AGE   VERSION
  201. master   Ready    k8s-node   27h   v1.15.3

kube-router 部署

  1. # win on Ubuntu  操作
  2. # 创建kube-router 访问kube-apiserver 证书
  3. cat << EOF | tee /apps/work/k8s/cfssl/k8s/kube-router.json
  4. {
  5.   "CN": "kube-router",
  6.   "hosts": [""], 
  7.   "key": {
  8.     "algo": "rsa",
  9.     "size": 2048
  10.   },
  11.   "names": [
  12.     {
  13.       "C": "CN",
  14.       "ST": "GuangDong",
  15.       "L": "GuangZhou",
  16.       "O": "system:masters",
  17.       "OU": "Kubernetes-manual"
  18.     }
  19.   ]
  20. }
  21. EOF
  22.  
  23. ## 生成 kube-router 证书和私钥
  24. cfssl gencert \
  25.         -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  26.         -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  27.         -config=/apps/work/k8s/cfssl/ca-config.json \
  28.         -profile=kubernetes \
  29.          /apps/work/k8s/cfssl/k8s/kube-router.json | \
  30.          cfssljson -bare /apps/work/k8s/cfssl/pki/k8s/kube-router
  31. # 设置集群参数
  32. kubectl config set-cluster kubernetes \
  33.   --certificate-authority=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  34.   --embed-certs=true \
  35.   --server=${KUBE_APISERVER} \
  36.   --kubeconfig=kubeconfig.conf
  37. # 设置客户端认证参数
  38.     kubectl config set-credentials kube-router \
  39.   --client-certificate=/apps/work/k8s/cfssl/pki/k8s/kube-router.pem \
  40.   --client-key=/apps/work/k8s/cfssl/pki/k8s/kube-router-key.pem \
  41.   --embed-certs=true \
  42.   --kubeconfig=kubeconfig.conf
  43. # 设置上下文参数
  44.     kubectl config set-context default \
  45.   --cluster=kubernetes \
  46.   --user=kube-router \
  47.   --kubeconfig=kubeconfig.conf
  48. # 设置默认上下文
  49. kubectl config use-context default --kubeconfig=kubeconfig.conf
  50. # 创建kube-router configmap 
  51. kubectl create configmap "kube-proxy" --from-file=kubeconfig.conf
  52. # 创建kubeadm-kuberouter-all-features-hostport.yaml
  53. vi kubeadm-kuberouter-all-features-hostport.yaml
  54. apiVersion: v1
  55. kind: ConfigMap
  56. metadata:
  57.   name: kube-router-cfg
  58.   namespace: kube-system
  59.   labels:
  60.     tier: node
  61.     k8s-app: kube-router
  62. data:
  63.   cni-conf.json: |
  64.     {
  65.        "cniVersion":"0.3.0",
  66.        "name":"mynet",
  67.        "plugins":[
  68.           {
  69.              "name":"kubernetes",
  70.              "type":"bridge",
  71.              "bridge":"kube-bridge",
  72.              "isDefaultGateway":true,
  73.              "ipam":{
  74.                 "type":"host-local"
  75.              }
  76.           },
  77.           {
  78.              "type":"portmap",
  79.              "capabilities":{
  80.                 "snat":true,
  81.                 "portMappings":true
  82.              }
  83.           }
  84.        ]
  85.     }
  86. ---
  87. apiVersion: apps/v1
  88. kind: DaemonSet
  89. metadata:
  90.   labels:
  91.     k8s-app: kube-router
  92.     tier: node
  93.   name: kube-router
  94.   namespace: kube-system
  95. spec:
  96.   selector:
  97.     matchLabels:
  98.       k8s-app: kube-router
  99.   template:
  100.     metadata:
  101.       labels:
  102.         k8s-app: kube-router
  103.         tier: node
  104.       annotations:
  105.         scheduler.alpha.kubernetes.io/critical-pod: ''
  106.     spec:
  107.       serviceAccountName: kube-router
  108.       serviceAccount: kube-router
  109.       containers:
  110.       - name: kube-router
  111.         image: docker.io/cloudnativelabs/kube-router
  112.         imagePullPolicy: Always
  113.         args:
  114.         - --run-router=true 
  115.         - --run-firewall=true 
  116.         - --run-service-proxy=true 
  117.         - --advertise-cluster-ip=true 
  118.         - --advertise-loadbalancer-ip=true 
  119.         - --advertise-pod-cidr=true 
  120.         - --advertise-external-ip=true 
  121.         - --cluster-asn=64512 
  122.         - --metrics-path=/metrics 
  123.         - --metrics-port=20241 
  124.         - --enable-cni=true 
  125.         - --enable-ibgp=true 
  126.         - --enable-overlay=true 
  127.         - --nodeport-bindon-all-ip=true 
  128.         - --nodes-full-mesh=true 
  129.         - --enable-pod-egress=true 
  130.         - --cluster-cidr=10.67.0.0/16 
  131.         - --v=2
  132.         - --kubeconfig=/var/lib/kube-router/kubeconfig
  133.         env:
  134.         - name: NODE_NAME
  135.           valueFrom:
  136.             fieldRef:
  137.               fieldPath: spec.nodeName
  138.         - name: KUBE_ROUTER_CNI_CONF_FILE
  139.           value: /etc/cni/net.d/10-kuberouter.conflist
  140.         livenessProbe:
  141.           httpGet:
  142.             path: /healthz
  143.             port: 20244
  144.           initialDelaySeconds: 10
  145.           periodSeconds: 3
  146.         resources:
  147.           requests:
  148.             cpu: 250m
  149.             memory: 250Mi
  150.         securityContext:
  151.           privileged: true
  152.         volumeMounts:
  153.         - name: lib-modules
  154.           mountPath: /lib/modules
  155.           readOnly: true
  156.         - name: cni-conf-dir
  157.           mountPath: /etc/cni/net.d
  158.         - name: kubeconfig
  159.           mountPath: /var/lib/kube-router
  160.           readOnly: true
  161.       initContainers:
  162.       - name: install-cni
  163.         image: busybox
  164.         imagePullPolicy: Always
  165.         command:
  166.         - /bin/sh
  167.         - -c
  168.         - set -e -x;
  169.           if [ ! -f /etc/cni/net.d/10-kuberouter.conflist ]; then
  170.             if [ -f /etc/cni/net.d/*.conf ]; then
  171.               rm -f /etc/cni/net.d/*.conf;
  172.             fi;
  173.             TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
  174.             cp /etc/kube-router/cni-conf.json ${TMP};
  175.             mv ${TMP} /etc/cni/net.d/10-kuberouter.conflist;
  176.           fi
  177.         volumeMounts:
  178.         - name: cni-conf-dir
  179.           mountPath: /etc/cni/net.d
  180.         - name: kube-router-cfg
  181.           mountPath: /etc/kube-router
  182.       hostNetwork: true
  183.       tolerations:
  184.       - key: CriticalAddonsOnly
  185.         operator: Exists
  186.       - effect: NoSchedule
  187.         key: node-role.kubernetes.io/master
  188.         operator: Exists
  189.       - effect: NoSchedule
  190.         key: node.kubernetes.io/not-ready
  191.         operator: Exists
  192.       - effect: NoSchedule
  193.         key: node-role.kubernetes.io/ingress
  194.         operator: Equal
  195.       volumes:
  196.       - name: lib-modules
  197.         hostPath:
  198.           path: /lib/modules
  199.       - name: cni-conf-dir
  200.         hostPath:
  201.           path: /etc/cni/net.d
  202.       - name: kube-router-cfg
  203.         configMap:
  204.           name: kube-router-cfg
  205.       - name: kubeconfig
  206.         configMap:
  207.           name: kube-proxy
  208.           items:
  209.           - key: kubeconfig.conf
  210.             path: kubeconfig
  211. ---
  212. apiVersion: v1
  213. kind: ServiceAccount
  214. metadata:
  215.   name: kube-router
  216.   namespace: kube-system
  217. ---
  218. kind: ClusterRole
  219. apiVersion: rbac.authorization.k8s.io/v1beta1
  220. metadata:
  221.   name: kube-router
  222.   namespace: kube-system
  223. rules:
  224.   - apiGroups:
  225.     - ""
  226.     resources:
  227.       - namespaces
  228.       - pods
  229.       - services
  230.       - nodes
  231.       - endpoints
  232.     verbs:
  233.       - list
  234.       - get
  235.       - watch
  236.   - apiGroups:
  237.     - "networking.k8s.io"
  238.     resources:
  239.       - networkpolicies
  240.     verbs:
  241.       - list
  242.       - get
  243.       - watch
  244.   - apiGroups:
  245.     - extensions
  246.     resources:
  247.       - networkpolicies
  248.     verbs:
  249.       - get
  250.       - list
  251.       - watch
  252. ---
  253. kind: ClusterRoleBinding
  254. apiVersion: rbac.authorization.k8s.io/v1beta1
  255. metadata:
  256.   name: kube-router
  257. roleRef:
  258.   apiGroup: rbac.authorization.k8s.io
  259.   kind: ClusterRole
  260.   name: kube-router
  261. subjects:
  262. - kind: ServiceAccount
  263.   name: kube-router
  264.   namespace: kube-system
  265. # 创建  kube-router 服务
  266. kubectl apply -f  kubeadm-kuberouter-all-features-hostport.yaml
  267. # 查看服务是否成功创建
  268. [root@]~]#kubectl get pod -A | grep kube-router
  269. kube-system            kube-router-5tmgw                           1/1     Running   0          21h
  270. # 进入192.168.30.32
  271. cat  /etc/cni/net.d/10-kuberouter.conflist
  272. [root@master ssl]# cat  /etc/cni/net.d/10-kuberouter.conflist
  273. {"cniVersion":"0.3.0","name":"mynet","plugins":[{"bridge":"kube-bridge","ipam":{"subnet":"10.67.0.0/24","type":"host-local"},"isDefaultGateway":true,"name":"kubernetes","type":"bridge"},{"capabilities":{"portMappings":true,"snat":true},"type":"portmap"}]}
  274. # 已经获取到ip段
  275. ip a| grep  kube
  276. [root@master ssl]# ip a| grep  kube
  277. 4: kube-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
  278.     inet 10.67.0.1/24 brd 10.67.0.255 scope global kube-bridge
  279. 6: kube-dummy-if: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
  280.     inet 10.66.0.1/32 brd 10.66.0.1 scope link kube-dummy-if
  281.     inet 10.66.0.2/32 brd 10.66.0.2 scope link kube-dummy-if
  282.     inet 10.66.91.125/32 brd 10.66.91.125 scope link kube-dummy-if
  283.     inet 10.66.86.10/32 brd 10.66.86.10 scope link kube-dummy-if
  284.     inet 10.66.52.216/32 brd 10.66.52.216 scope link kube-dummy-if
  285. # kube-router 部署正常

coredns 部署

  1. # win on Ubuntu  操作
  2. vi coredns.yaml
  3. # __MACHINE_GENERATED_WARNING__
  4.  
  5. apiVersion: v1
  6. kind: ServiceAccount
  7. metadata:
  8.   name: coredns
  9.   namespace: kube-system
  10.   labels:
  11.       kubernetes.io/cluster-service: "true"
  12.       addonmanager.kubernetes.io/mode: Reconcile
  13. ---
  14. apiVersion: rbac.authorization.k8s.io/v1
  15. kind: ClusterRole
  16. metadata:
  17.   labels:
  18.     kubernetes.io/bootstrapping: rbac-defaults
  19.     addonmanager.kubernetes.io/mode: Reconcile
  20.   name: system:coredns
  21. rules:
  22. - apiGroups:
  23.   - ""
  24.   resources:
  25.   - endpoints
  26.   - services
  27.   - pods
  28.   - namespaces
  29.   verbs:
  30.   - list
  31.   - watch
  32. - apiGroups:
  33.   - ""
  34.   resources:
  35.   - nodes
  36.   verbs:
  37.   - get
  38. ---
  39. apiVersion: rbac.authorization.k8s.io/v1
  40. kind: ClusterRoleBinding
  41. metadata:
  42.   annotations:
  43.     rbac.authorization.kubernetes.io/autoupdate: "true"
  44.   labels:
  45.     kubernetes.io/bootstrapping: rbac-defaults
  46.     addonmanager.kubernetes.io/mode: EnsureExists
  47.   name: system:coredns
  48. roleRef:
  49.   apiGroup: rbac.authorization.k8s.io
  50.   kind: ClusterRole
  51.   name: system:coredns
  52. subjects:
  53. - kind: ServiceAccount
  54.   name: coredns
  55.   namespace: kube-system
  56. ---
  57. apiVersion: v1
  58. kind: ConfigMap
  59. metadata:
  60.   name: coredns
  61.   namespace: kube-system
  62.   labels:
  63.       addonmanager.kubernetes.io/mode: EnsureExists
  64. data:
  65.   Corefile: |
  66.     .:53 {
  67.         errors
  68.         health
  69.         kubernetes cluster.local in-addr.arpa ip6.arpa {
  70.             pods insecure
  71.             upstream /etc/resolv.conf
  72.             fallthrough in-addr.arpa ip6.arpa
  73.         }
  74.         prometheus :9153
  75.         forward . /etc/resolv.conf
  76.         cache 30
  77.         reload
  78.         loadbalance
  79.     }
  80. ---
  81. apiVersion: apps/v1
  82. kind: Deployment
  83. metadata:
  84.   name: coredns
  85.   namespace: kube-system
  86.   labels:
  87.     k8s-app: kube-dns
  88.     kubernetes.io/cluster-service: "true"
  89.     addonmanager.kubernetes.io/mode: Reconcile
  90.     kubernetes.io/name: "CoreDNS"
  91. spec:
  92.   # replicas: not specified here:
  93.   # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  94.   # 2. Default is 1.
  95.   # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  96.   strategy:
  97.     type: RollingUpdate
  98.     rollingUpdate:
  99.       maxUnavailable: 1
  100.   selector:
  101.     matchLabels:
  102.       k8s-app: kube-dns
  103.   template:
  104.     metadata:
  105.       labels:
  106.         k8s-app: kube-dns
  107.       annotations:
  108.         seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
  109.     spec:
  110.       priorityClassName: system-cluster-critical
  111.       serviceAccountName: coredns
  112.       tolerations:
  113.         - key: "CriticalAddonsOnly"
  114.           operator: "Exists"
  115.       nodeSelector:
  116.         beta.kubernetes.io/os: linux
  117.       containers:
  118.       - name: coredns
  119.         image: coredns/coredns
  120.         imagePullPolicy: Always
  121.         resources:
  122.           limits:
  123.             memory: 170Mi
  124.           requests:
  125.             cpu: 100m
  126.             memory: 70Mi
  127.         args: [ "-conf", "/etc/coredns/Corefile" ]
  128.         volumeMounts:
  129.         - name: config-volume
  130.           mountPath: /etc/coredns
  131.           readOnly: true
  132.         ports:
  133.         - containerPort: 53
  134.           name: dns
  135.           protocol: UDP
  136.         - containerPort: 53
  137.           name: dns-tcp
  138.           protocol: TCP
  139.         - containerPort: 9153
  140.           name: metrics
  141.           protocol: TCP
  142.         livenessProbe:
  143.           httpGet:
  144.             path: /health
  145.             port: 8080
  146.             scheme: HTTP
  147.           initialDelaySeconds: 60
  148.           timeoutSeconds: 5
  149.           successThreshold: 1
  150.           failureThreshold: 5
  151.         readinessProbe:
  152.           httpGet:
  153.             path: /health
  154.             port: 8080
  155.             scheme: HTTP
  156.         securityContext:
  157.           allowPrivilegeEscalation: false
  158.           capabilities:
  159.             add:
  160.             - NET_BIND_SERVICE
  161.             drop:
  162.             - all
  163.           readOnlyRootFilesystem: true
  164.       dnsPolicy: Default
  165.       volumes:
  166.         - name: config-volume
  167.           configMap:
  168.             name: coredns
  169.             items:
  170.             - key: Corefile
  171.               path: Corefile
  172. ---
  173. apiVersion: v1
  174. kind: Service
  175. metadata:
  176.   name: kube-dns
  177.   namespace: kube-system
  178.   annotations:
  179.     prometheus.io/port: "9153"
  180.     prometheus.io/scrape: "true"
  181.   labels:
  182.     k8s-app: kube-dns
  183.     kubernetes.io/cluster-service: "true"
  184.     addonmanager.kubernetes.io/mode: Reconcile
  185.     kubernetes.io/name: "CoreDNS"
  186. spec:
  187.   selector:
  188.     k8s-app: kube-dns
  189.   clusterIP: 10.66.0.2
  190.   ports:
  191.   - name: dns
  192.     port: 53
  193.     protocol: UDP
  194.   - name: dns-tcp
  195.     port: 53
  196.     protocol: TCP
  197.   - name: metrics
  198.     port: 9153
  199.     protocol: TCP
  200. # 创建 CoreDNS dns 服务
  201. kubectl apply -f coredns.yaml
  202. # 验证服务
  203. [root@]~]#kubectl get all -A | grep coredns
  204. kube-system            pod/coredns-597b77445b-fhxvr                    1/1     Running   0          27h
  205. kube-system            deployment.apps/coredns                     1/1     1            1           27h
  206. kube-system            replicaset.apps/coredns-597b77445b                    1         1         1       27h
  207. dig @10.66.0.2 www.baidu.com
  208. [root@master ssl]# dig @10.66.0.2 www.baidu.com
  209.  
  210. ; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> @10.66.0.2 www.baidu.com
  211. ; (1 server found)
  212. ;; global options: +cmd
  213. ;; Got answer:
  214. ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40347
  215. ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 13, ADDITIONAL: 27
  216.  
  217. ;; OPT PSEUDOSECTION:
  218. ; EDNS: version: 0, flags:; udp: 4096
  219. ;; QUESTION SECTION:
  220. ;www.baidu.com.                 IN      A
  221.  
  222. ;; ANSWER SECTION:
  223. www.baidu.com.          30      IN      CNAME   www.a.shifen.com.
  224. www.a.shifen.com.       30      IN      A       14.215.177.38
  225. www.a.shifen.com.       30      IN      A       14.215.177.39
  226.  
  227. ;; AUTHORITY SECTION:
  228. com.                    30      IN      NS      h.gtld-servers.net.
  229. com.                    30      IN      NS      m.gtld-servers.net.
  230. com.                    30      IN      NS      g.gtld-servers.net.
  231. com.                    30      IN      NS      d.gtld-servers.net.
  232. com.                    30      IN      NS      a.gtld-servers.net.
  233. com.                    30      IN      NS      j.gtld-servers.net.
  234. com.                    30      IN      NS      c.gtld-servers.net.
  235. com.                    30      IN      NS      l.gtld-servers.net.
  236. com.                    30      IN      NS      b.gtld-servers.net.
  237. com.                    30      IN      NS      f.gtld-servers.net.
  238. com.                    30      IN      NS      k.gtld-servers.net.
  239. com.                    30      IN      NS      i.gtld-servers.net.
  240. com.                    30      IN      NS      e.gtld-servers.net.
  241.  
  242. ;; ADDITIONAL SECTION:
  243. e.gtld-servers.net.     30      IN      AAAA    2001:502:1ca1::30
  244. a.gtld-servers.net.     30      IN      A       192.5.6.30
  245. i.gtld-servers.net.     30      IN      AAAA    2001:503:39c1::30
  246. c.gtld-servers.net.     30      IN      A       192.26.92.30
  247. g.gtld-servers.net.     30      IN      AAAA    2001:503:eea3::30
  248. m.gtld-servers.net.     30      IN      A       192.55.83.30
  249. d.gtld-servers.net.     30      IN      A       192.31.80.30
  250. a.gtld-servers.net.     30      IN      AAAA    2001:503:a83e::2:30
  251. b.gtld-servers.net.     30      IN      A       192.33.14.30
  252. b.gtld-servers.net.     30      IN      AAAA    2001:503:231d::2:30
  253. i.gtld-servers.net.     30      IN      A       192.43.172.30
  254. d.gtld-servers.net.     30      IN      AAAA    2001:500:856e::30
  255. l.gtld-servers.net.     30      IN      A       192.41.162.30
  256. h.gtld-servers.net.     30      IN      AAAA    2001:502:8cc::30
  257. e.gtld-servers.net.     30      IN      A       192.12.94.30
  258. l.gtld-servers.net.     30      IN      AAAA    2001:500:d937::30
  259. k.gtld-servers.net.     30      IN      AAAA    2001:503:d2d::30
  260. j.gtld-servers.net.     30      IN      AAAA    2001:502:7094::30
  261. m.gtld-servers.net.     30      IN      AAAA    2001:501:b1f9::30
  262. f.gtld-servers.net.     30      IN      A       192.35.51.30
  263. g.gtld-servers.net.     30      IN      A       192.42.93.30
  264. h.gtld-servers.net.     30      IN      A       192.54.112.30
  265. j.gtld-servers.net.     30      IN      A       192.48.79.30
  266. k.gtld-servers.net.     30      IN      A       192.52.178.30
  267. c.gtld-servers.net.     30      IN      AAAA    2001:503:83eb::30
  268. f.gtld-servers.net.     30      IN      AAAA    2001:503:d414::30
  269.  
  270. ;; Query time: 6 msec
  271. ;; SERVER: 10.66.0.2#53(10.66.0.2)
  272. ;; WHEN: Wed Sep 04 14:17:05 CST 2019
  273. ;; MSG SIZE  rcvd: 897
  274. dig @10.66.0.2 kube-dns.kube-system.svc.cluster.local
  275. [root@master ssl]# dig @10.66.0.2 kube-dns.kube-system.svc.cluster.local
  276.  
  277. ; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> @10.66.0.2 kube-dns.kube-system.svc.cluster.local
  278. ; (1 server found)
  279. ;; global options: +cmd
  280. ;; Got answer:
  281. ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40471
  282. ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
  283. ;; WARNING: recursion requested but not available
  284.  
  285. ;; OPT PSEUDOSECTION:
  286. ; EDNS: version: 0, flags:; udp: 4096
  287. ;; QUESTION SECTION:
  288. ;kube-dns.kube-system.svc.cluster.local.        IN A
  289.  
  290. ;; ANSWER SECTION:
  291. kube-dns.kube-system.svc.cluster.local. 5 IN A  10.66.0.2
  292.  
  293. ;; Query time: 1 msec
  294. ;; SERVER: 10.66.0.2#53(10.66.0.2)
  295. ;; WHEN: Wed Sep 04 14:19:13 CST 2019
  296. ;; MSG SIZE  rcvd: 121
  297. 能够正常解析

traefik-https 部署

  1. # win on Ubuntu  操作 
  2.  
  3. #申请证书 请使用letsencrypt 申请免费 多域名证书
  4. #重命名证书名字
  5. tls.crt 
  6. tls.key
  7. #创建 secret
  8. kubectl -n kube-system create secret generic tls-cert --from-file=tls.key --from-file=tls.crt
  9. # 创建traefik 配置
  10. vi traefik.toml
  11. defaultEntryPoints = ["http","https"]
  12. [entryPoints]
  13.   [entryPoints.http]
  14.   address = ":80"
  15.     entryPoint = "https"
  16.   [entryPoints.https]
  17.   address = ":443"
  18.     [entryPoints.https.tls]
  19.       [[entryPoints.https.tls.certificates]]
  20.       certFile = "/certs/tls.crt"
  21.       keyFile = "/certs/tls.key"
  22. # 生成 configmap
  23. kubectl create configmap traefik-conf --from-file=traefik.toml -n kube-system
  24. # 创建traefik-rbac
  25. vi traefik-rbac.yaml
  26. ---
  27. kind: ClusterRole
  28. apiVersion: rbac.authorization.k8s.io/v1beta1
  29. metadata:
  30.   name: traefik
  31.   namespace: kube-system
  32. rules:
  33.   - apiGroups:
  34.       - ""
  35.     resources:
  36.       - services
  37.       - endpoints
  38.       - secrets
  39.     verbs:
  40.       - get
  41.       - list
  42.       - watch
  43.   - apiGroups:
  44.       - extensions
  45.     resources:
  46.       - ingresses
  47.     verbs:
  48.       - get
  49.       - list
  50.       - watch
  51.   - apiGroups:
  52.     - extensions
  53.     resources:
  54.     - ingresses/status
  55.     verbs:
  56.     - update
  57. ---
  58. kind: ClusterRoleBinding
  59. apiVersion: rbac.authorization.k8s.io/v1beta1
  60. metadata:
  61.   name: traefik
  62. roleRef:
  63.   apiGroup: rbac.authorization.k8s.io
  64.   kind: ClusterRole
  65.   name: traefik
  66. subjects:
  67. - kind: ServiceAccount
  68.   name: traefik
  69.   namespace: kube-system
  70. # traefik-deployment-https
  71. vi traefik-deployment-https.yaml
  72. ---
  73. apiVersion: v1
  74. kind: ServiceAccount
  75. metadata:
  76.   name: traefik
  77.   namespace: kube-system
  78. ---
  79. kind: Deployment
  80. apiVersion: apps/v1
  81. metadata:
  82.   name: traefik
  83.   namespace: kube-system
  84.   labels:
  85.     k8s-app: traefik
  86. spec:
  87.   replicas: 1
  88.   selector:
  89.     matchLabels:
  90.       k8s-app: traefik
  91.   template:
  92.     metadata:
  93.       labels:
  94.         k8s-app: traefik
  95.         name: traefik
  96.     spec:
  97.       serviceAccountName: traefik
  98.       terminationGracePeriodSeconds: 60
  99.       volumes:
  100.       - name: ssl
  101.         secret:
  102.           secretName: tls-cert
  103.       - name: config
  104.         configMap:
  105.           name: traefik-conf
  106.           defaultMode: 0644
  107.           items:
  108.           - key: traefik.toml
  109.             path: traefik.toml 
  110.       hostNetwork: true
  111.       dnsPolicy: ClusterFirstWithHostNet      
  112.       containers:
  113.       - image: traefik
  114.         name: traefik
  115.         imagePullPolicy: IfNotPresent
  116.         volumeMounts:
  117.         - mountPath: /certs         
  118.           name: "ssl"
  119.         - mountPath: /etc/traefik.toml
  120.           subPath: traefik.toml
  121.           name: "config"
  122.         ports:
  123.         - name: http
  124.           containerPort: 80
  125.           hostPort: 80
  126.         - name: https
  127.           containerPort: 443
  128.           hostPort: 443
  129.         - name: admin
  130.           containerPort: 8080
  131.         args:
  132.         - --api
  133.         - --web
  134.         - --api.dashboard
  135.         - --logLevel=INFO
  136.         - --web.metrics
  137.         - --metrics.prometheus
  138.         - --web.metrics.prometheus
  139.         - --kubernetes
  140.         - --traefiklog
  141.         - --traefiklog.format=json
  142.         - --accesslog
  143.         - --accesslog.format=json
  144.         - --accessLog.fields.headers.defaultMode=redact
  145.         - --insecureskipverify=true
  146.         - --configFile=/etc/traefik.toml       
  147. #      nodeSelector:
  148. #        ingress: "yes"
  149. #      tolerations:
  150. #      - effect: NoSchedule
  151. #        key: node-role.kubernetes.io/ingress
  152. #        operator: Equal
  153. ---
  154. kind: Service
  155. apiVersion: v1
  156. metadata:
  157.   labels:
  158.     k8s-app: traefik
  159.   name: traefik
  160.   namespace: kube-system
  161. spec:
  162.   selector:
  163.     k8s-app: traefik
  164.   clusterIP: None
  165.   ports:
  166.     - protocol: TCP
  167.       port: 80
  168.       name: http
  169.     - protocol: TCP
  170.       port: 443
  171.       name: https
  172.     - protocol: TCP
  173.       port: 8080
  174.       name: admin
  175.   type: ClusterIP
  176. # 或者traefik-daemonset-https
  177. ---
  178. apiVersion: v1
  179. kind: ServiceAccount
  180. metadata:
  181.   name: traefik
  182.   namespace: kube-system
  183. ---
  184. kind: DaemonSet
  185. apiVersion: apps/v1
  186. metadata:
  187.   name: traefik
  188.   namespace: kube-system
  189.   labels:
  190.     k8s-app: traefik
  191. spec:
  192.   selector:
  193.     matchLabels:
  194.       k8s-app: traefik
  195.   template:
  196.     metadata:
  197.       labels:
  198.         k8s-app: traefik
  199.         name: traefik
  200.     spec:
  201.       serviceAccountName: traefik
  202.       terminationGracePeriodSeconds: 60
  203.       volumes:
  204.       - name: ssl
  205.         secret:
  206.           secretName: tls-cert
  207.       - name: config
  208.         configMap:
  209.           name: traefik-conf
  210.           defaultMode: 0644
  211.           items:
  212.           - key: traefik.toml
  213.             path: traefik.toml
  214.       hostNetwork: true
  215.       dnsPolicy: ClusterFirstWithHostNet      
  216.       containers:
  217.       - image: traefik
  218.         name: traefik
  219.         imagePullPolicy: IfNotPresent
  220.         volumeMounts:
  221.         - mountPath: /certs         
  222.           name: "ssl"
  223.         - mountPath: /etc/traefik.toml
  224.           subPath: traefik.toml
  225.           name: "config"
  226.         ports:
  227.         - name: http
  228.           containerPort: 80
  229.           hostPort: 80
  230.         - name: https
  231.           containerPort: 443
  232.           hostPort: 443
  233.         - name: admin
  234.           containerPort: 8080
  235.         securityContext:
  236.           capabilities:
  237.             drop:
  238.             - ALL
  239.             add:
  240.             - NET_BIND_SERVICE
  241.         args:
  242.         - --api
  243.         - --web
  244.         - --api.dashboard
  245.         - --logLevel=INFO
  246.         - --web.metrics
  247.         - --metrics.prometheus
  248.         - --web.metrics.prometheus
  249.         - --kubernetes
  250.         - --traefiklog
  251.         - --traefiklog.format=json
  252.         - --accesslog
  253.         - --accesslog.format=json
  254.         - --accessLog.fields.headers.defaultMode=redact
  255.         - --insecureskipverify=true
  256.         - --configFile=/etc/traefik.toml       
  257.       nodeSelector:
  258.         ingress: "yes"
  259.       tolerations:
  260.       - effect: NoSchedule
  261.         key: node-role.kubernetes.io/ingress
  262.         operator: Equal
  263. ---
  264. kind: Service
  265. apiVersion: v1
  266. metadata:
  267.   labels:
  268.     k8s-app: traefik
  269.   name: traefik
  270.   namespace: kube-system
  271. spec:
  272.   selector:
  273.     k8s-app: traefik
  274.   clusterIP: None
  275.   ports:
  276.     - protocol: TCP
  277.       port: 80
  278.       name: http
  279.     - protocol: TCP
  280.       port: 443
  281.       name: https
  282.     - protocol: TCP
  283.       port: 8080
  284.       name: admin
  285.   type: ClusterIP
  286. #创建 traefik-dashboard
  287. vi traefik-dashboard.yam
  288. apiVersion: extensions/v1beta1
  289. kind: Ingress
  290. metadata:
  291.   name: traefik-dashboard
  292.   namespace: kube-system
  293.   annotations:
  294.     kubernetes.io/ingress.class: traefik
  295.     traefik.ingress.kubernetes.io/frontend-entry-points: http,https
  296. spec:
  297.   rules:
  298.   - host: trae.xxx.com
  299.     http:
  300.       paths:
  301.         - backend:
  302.             serviceName: traefik
  303.             servicePort: 8080
  304.   tls:
  305.    - secretName: tls-cert
  306. # 创建 服务器
  307. kubectl apply -f traefik-deployment-https.yaml
  308. kubectl apply -f traefik-rbac.yaml
  309. kubectl apply -f traefik-dashboard.yaml
  310. hosts 绑定 访问 trae.xxx.com 是否正常
  311. 能正常打开证明正常

kubernetes-dashboard 部署

  1. # win on Ubuntu  操作  
  2. # 创建 kubernetes-dashboard 使用证书
  3. cat << EOF | tee /apps/work/k8s/cfssl/k8s/dashboard.json
  4. {
  5.   "CN": "dashboard",
  6.   "hosts": [""], 
  7.   "key": {
  8.     "algo": "rsa",
  9.     "size": 2048
  10.   },
  11.   "names": [
  12.     {
  13.       "C": "CN",
  14.       "ST": "GuangDong",
  15.       "L": "GuangZhou",
  16.       "O": "cluster",
  17.       "OU": "cluster"
  18.     }
  19.   ]
  20. }
  21. EOF
  22. ##### 生成kubernetes-dashboard 证书，当然如果有外部签发的证书也可以使用
  23. cfssl gencert \
  24.         -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  25.        -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  26.         -config=/apps/work/k8s/cfssl/ca-config.json \
  27.         -profile=kubernetes \
  28.         /apps/work/k8s/cfssl/k8s/dashboard.json | \
  29.         cfssljson -bare ./dashboard
  30. # base64 加密
  31. cat dashboard.pem|base64 | tr -d '\n'
  32. cat dashboard-key.pem|base64 | tr -d '\n'
  33. # 做好记录
  34. # kubernetes-dashboard 
  35. vi kubernetes-dashboard.yaml
  36. # Copyright 2017 The Kubernetes Authors.
  37. #
  38. # Licensed under the Apache License, Version 2.0 (the "License");
  39. # you may not use this file except in compliance with the License.
  40. # You may obtain a copy of the License at
  41. #
  42. #     http://www.apache.org/licenses/LICENSE-2.0
  43. #
  44. # Unless required by applicable law or agreed to in writing, software
  45. # distributed under the License is distributed on an "AS IS" BASIS,
  46. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  47. # See the License for the specific language governing permissions and
  48. # limitations under the License.
  49.  
  50. apiVersion: v1
  51. kind: Namespace
  52. metadata:
  53.   name: kubernetes-dashboard
  54.  
  55. ---
  56.  
  57. apiVersion: v1
  58. kind: ServiceAccount
  59. metadata:
  60.   labels:
  61.     k8s-app: kubernetes-dashboard
  62.   name: kubernetes-dashboard
  63.   namespace: kubernetes-dashboard
  64. ---
  65.  
  66. kind: Service
  67. apiVersion: v1
  68. metadata:
  69.   labels:
  70.     k8s-app: kubernetes-dashboard
  71.   name: kubernetes-dashboard
  72.   namespace: kubernetes-dashboard
  73.   annotations:
  74.     kubernetes.io/ingress.class: traefik
  75. spec:
  76.   ports:
  77.     - port: 443
  78.       targetPort: 8443
  79.   selector:
  80.     k8s-app: kubernetes-dashboard
  81.  
  82. ---
  83.  
  84. apiVersion: v1
  85. kind: Secret
  86. metadata:
  87.   labels:
  88.     k8s-app: kubernetes-dashboard
  89.   name: kubernetes-dashboard-certs
  90.   namespace: kubernetes-dashboard
  91. type: Opaque
  92. data:
  93.   dashboard.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeFlFV0MxbGlqcnFzNW5vcHBxTXF0YzZSY0pnSWFJSGhGemZZUWhRQm5pK0Vjam8vCkRTUkYvY3BUOFlkTTg2MVpEV1lSN1FEelFLNmJUTmRLWXJJYmpVWHJpRFVFU01EUW13Y1VteTMzWjFpeXR6K0wKUUVmTVFvWVNReGVIY2RqUHp3bUhFS0todk9vNmxQTHNFWkMwQ3ZCamw2VHlERjhuSDEzby9kRlRVbGJhWUlGaQpPeGVIWkxMMTZKbmNLK3RVaW9ncjdLekFKMUkxTjdwOVQ1blZ5YU9PbWNCVEFnU3RJM0ZwSzdMZG1zaVU0ZEZ0CkpSSFZ0eTh5Y3dCSU9wWnhqV29mM2ROVkRrVUFsYjVtV2psU0RaQ2lhYmFYQi91NmJ0R0k3RlY2cENaUzdDVG4KeWlpUFlFSXRPSGRCT0VycGpKZWQ0bHQ5K2MvNDE3UTRIaiswdndJREFRQUJBb0lCQVFDK1daSWdjQTZRRnhScQpzVlNST1BNQjlFdXlJNlQrN0NZL2xXQUZGM2tUdHlKRVlTVEJpck0yVFprbjBFbjNGSndlVU1CNEZwRmJScTJBCm1vSWpxeHJveG5taGRjOWlPd3NTVHZtcU1kd2ZLNXBiQ0pBeDdNRE5ZS0FiTDRNbjAxazlaaVpaZnhTNG1WcksKa1hHNTRDZlYzeWR0VU5qRDJiVkFBdWQ2TVJQSDV5QWJTVktsMG9ONkRCaFV4MlYyWEo0WnRUVHE0b3R6VGYxZwp3SjNJeVFjSXl3czE2V3dkeHpuYStqVmpOYU5OQ3ZCT1BMbm9TeXZBQXZGRG9UYmUrMG1tcnZLVmlSeDBDT1FzCkUwNjFtNHY2eUExL3locndkT1BDYXN6SkpjWlYzOThJTzFKb2QxUHk3OU9aT1FpY1FEOGhwQmxqb0FSQ2JlY3QKRFFPcG5CR0JBb0dCQVBhYlJSSGpPTkxIQ25JZWlFQU1EYXNwQXo2RGxRNkQvdWNNdzROdkVPRVNVa3dvQ0p4cApwK1hJeVVzT1B1d2swTzVCcHJRcHZjdGYyWXlLZTFtR25iVUpmUVNWNGpLdWpqb0M0OWhOWk9lSE8zd0xMcnNXCkl1SU1Qeko0TjhxSzl0dUpDQ3BVYUZFVzRiN1R2OGsyK1pJWHJwN3hzNklDd01EUnpTaW9wY0hCQW9HQkFNMEgKQVl1bmdzY3hTM2JnZ05idU5sQ3lIOHBLZFVPbi95cU9IQUdYcG9vZmJUbXJiUUlWN0ZOVSszUTlYc2ErVVE0QwpUbVdFbzhabVhrL3lIV2FDVWxpRkN0ckRhTzNUZVhvb2pia1JyaDcxakFXN0pjVDRVZ1ZwcG1RakFVUW8vOWtVCmxHMUNpOTFZZy94dlV5dHlYM1BnZHJ6SnU2aWNsM1pVZ1h3dzNoWi9Bb0dBZENmY2w3bFVLWXZSTXNHSTRjb0wKb2lRMlAvclFlYjdZa05IbFFZSk9EQVdLT0E3ZlIzVkl2U1lmRWpoS2tRWWlWeWNiTTE4NTQ1SnBNUmFGVlR6ZwpDY2JIV1NLVUlkVXdic2l2czFGNUJza2V6cVdoeEVOLytNTlYvUnE5QkswQjY1UVhBWUV5aFlkbW0zQzN0RG90CndZOWdFOE83SGNONE1ScGhMUmFLeE1FQ2dZRUFoS2E5eHorUUM1VEhRSmlzZzJNSVhWbUIyLzRrdEt0akdvTnIKZDFSSStpQ3ZLSnJUSW9CUXNQSFE1em8xc2R5ODBKV0paNEZUL1MrS1lhdENmbXBmSU1xalpUcjlEcksrYTkwRgpKUEpkZDhaaTIrcGoyM2JXaW8zNmk5dGlIRmx5ZjE4alVUVzNESFVTb0NiZTVzTlBJc2ZkeXZPeXFMcjMvQ1ZjCnlaOU1jYjBDZ1lBMVp2RVM3bU42Nm10T2JpSlR3a3hhaTVvS2tHbDdHTDJkZXJFUmxsc1YrNWRCSVY4dG5DTnAKT2tjMFlMbHV2TEg4cG4zd2VCNzg5dUFCQjNXYmNKcHg0L2NIRm9oZDNhdlR0RThRVjJod0tNS2RKQVBvTHNoMgprK2lEUWd1dmFxSzNmL1RYUW43bWU3dWFqSDk3SXZldXJtWWsvVmRJY0dicnd1SVRzd0FEYWc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
  94.   dashboard.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ5ekNDQXQrZ0F3SUJBZ0lVUWRIVXdKS1JYc1ZRb2VYS1JDTjd0eVcwWU04d0RRWUpLb1pJaHZjTkFRRUwKQlFBd2JqRUxNQWtHQTFVRUJoTUNRMDR4RWpBUUJnTlZCQWdUQ1VkMVlXNW5SRzl1WnpFU01CQUdBMVVFQnhNSgpSM1ZoYm1kYWFHOTFNUkF3RGdZRFZRUUtFd2R0WkdSbllXMWxNUkF3RGdZRFZRUUxFd2R0WkdSbllXMWxNUk13CkVRWURWUVFERXdwcmRXSmxjbTVsZEdWek1CNFhEVEU1TURjd05ERXhNVE13TUZvWERUSTVNRGN3TVRFeE1UTXcKTUZvd2JURUxNQWtHQTFVRUJoTUNRMDR4RWpBUUJnTlZCQWdUQ1VkMVlXNW5SRzl1WnpFU01CQUdBMVVFQnhNSgpSM1ZoYm1kYWFHOTFNUkF3RGdZRFZRUUtFd2R0WkdSbllXMWxNUkF3RGdZRFZRUUxFd2R0WkdSbllXMWxNUkl3CkVBWURWUVFERXdsa1lYTm9ZbTloY21Rd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUIKQVFERmdSWUxXV0tPdXF6bWVpbW1veXExenBGd21BaG9nZUVYTjloQ0ZBR2VMNFJ5T2o4TkpFWDl5bFB4aDB6egpyVmtOWmhIdEFQTkFycHRNMTBwaXNodU5SZXVJTlFSSXdOQ2JCeFNiTGZkbldMSzNQNHRBUjh4Q2hoSkRGNGR4CjJNL1BDWWNRb3FHODZqcVU4dXdSa0xRSzhHT1hwUElNWHljZlhlajkwVk5TVnRwZ2dXSTdGNGRrc3ZYb21kd3IKNjFTS2lDdnNyTUFuVWpVM3VuMVBtZFhKbzQ2WndGTUNCSzBqY1drcnN0MmF5SlRoMFcwbEVkVzNMekp6QUVnNgpsbkdOYWgvZDAxVU9SUUNWdm1aYU9WSU5rS0pwdHBjSCs3cHUwWWpzVlhxa0psTHNKT2ZLS0k5Z1FpMDRkMEU0ClN1bU1sNTNpVzMzNXovalh0RGdlUDdTL0FnTUJBQUdqZ1kwd2dZb3dEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CMEcKQTFVZEpRUVdNQlFHQ0NzR0FRVUZCd01CQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUFNQjBHQTFVZApEZ1FXQkJURTl6cWx4dkErRXMrbE8zWlFEMlhubGFHRFpqQWZCZ05WSFNNRUdEQVdnQlJ4NEtjQVJjYWtSL2J4Cm13b1RCZURzK3hBb2FUQUxCZ05WSFJFRUJEQUNnZ0F3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUJnWHZwTEMKQjIybXlQaURlZnhsWGNZRzAvY0R2RXlYcTlENWtKTnBxKzFZQ0EvMlp2RDIyN1Q5VjY3aHVyTlA3T2FvSG95Tgo0MHpkR3lZTGRNV3pyZTQwVksxdC84N3pDTENzamt1ZXRCRWEwNVRqUTJhbDRhSzJ6TXl5MkJLWEpYbjlvdkhzCjJwNndvL001eklEOXl2OEhyRkZqWHM3NitTUTFzNXpOdUxuaDBET0Z1SktiZUZxSUJyNmZRbXlsb0l1VURtZjYKcGtQYkJyRnJpNHFGS0lDcVZKRCt3Z01zRFBiclVMZXF5NWlBVjNqRzJKMFgxOE4zdklCeUFwdWhZbjNudlV0TwpLREVIWkFJcFpjRWdqQ2ZLVDNyaERLL3JLN0VFZkxLcGlCdGJya3pFbjVWV3FQUFJEK3ZPU2VySldETDl1K0xyCmhEazlvZ084cmNqQzZGdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
  95. ---
  96.  
  97. apiVersion: v1
  98. kind: Secret
  99. metadata:
  100.   labels:
  101.     k8s-app: kubernetes-dashboard
  102.   name: dashboard-tls-cert
  103.   namespace: kubernetes-dashboard
  104. type: Opaque
  105. data:
  106.   tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUdYekNDQlVlZ0F3SUJBZ0lTQkdVcDlSaVAvK2lNMDVYM0FuY0FUeTg1TUEwR0NTcUdTSWIzRFFFQkN3VUEKTUVveEN6QUpCZ05WQkFZVEFsVlRNUll3RkFZRFZRUUtFdzFNWlhRbmN5QkZibU55ZVhCME1TTXdJUVlEVlFRRApFeHBNWlhRbmN5QkZibU55ZVhCMElFRjFkR2h2Y21sMGVTQllNekFlRncweE9UQTNNRGt3T1RJNU1ESmFGdzB4Ck9URXdNRGN3T1RJNU1ESmFNQll4RkRBU0JnTlZCQU1UQzIxa1pHZGhiV1V1WTI5dE1JSUNJakFOQmdrcWhraUcKOXcwQkFRRUZBQU9DQWc4QU1JSUNDZ0tDQWdFQW9mSVdOdTE4YUp1T3Jzd0JjZE9lODN0dWpXZ2dpUXl0VVYxQwpqNVhYbzNjQTM1L2ZxQXNGVHpJRGNwUmxhTGJ6SHd1d1psOWNSKzJuRENaUzI4VlhZaXcrSkQvQXpna3FzTHFJCjZ3YlFhcHNCa1lYUzRuT1UrZzhSMVgwcm52ckpickE1eHFJSWJKM002ajVLTXZ4RktvMEV3YXNBY2NiYlVGOW4KMHQ2RzNreG4zWW1Sek5HeHh1bXZ4V2prNWNkSWMza0MyT1VuRktGOG5XemJab2JiNk9PUnZSaElEWW5YdjkxdgoyMUYwQnZ0Q21GY0FEaDRqZXUrLzNKVDVLcEJkdkFHOHI3aU1wbkhKaFU1alhqTXlPRytMbkcvcnJuRzJGaXpHCmx1UHQwKzRlK0ZRSXFZY1BUM1cyTUF2ZDlzQTNEMThsUW82M00vZlMyYjNIYVNidFY0b1pmNS9zTzJNeEVPVnoKVEd1M0NxYk40TkcrZE8ycXoxYWxMQmlGZlVjNEdmUVpYRmlLaDFzazl3Qm5zeWhqYUZmdUx6bHRxMDg3STJLYQorVlRaUzFQSlJFbGduM3UwY1FmaENjelF5ZTJ3Vjl6RE9lVmUxeTBjLzZ0RWJhNllCeGR2ZGcwOFpKL0QwYTBLCnJvWlVJMW5Rc2RKeE8rQ3N1OURLYjROZzJCYnZkWVpHVWJrSCtSUDU0UUdrS1VnYnVxNVIwbXI0U1I2VUwrRE4KZjNxem81a3ZiMXVRWXFpaDZYUFVDVUVPOTNOU1Y2MTNUSUVOTUpyYjVhbGRLUkhPZlpWL201QThlUy9ibFFYcgpOV3FCRy9OL2RtckZjMmcyNGJEY3d5OXIzL3FkNy9MTWxmMVRVdzJGczR3M2x2VHJFanlwWEZhQ3BRRGxkc0xJCkYwcWVKVnNDQXdFQUFhT0NBbkV3Z2dKdE1BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFRmpBVUJnZ3IKQmdFRkJRY0RBUVlJS3dZQkJRVUhBd0l3REFZRFZSMFRBUUgvQkFJd0FEQWRCZ05WSFE0RUZnUVVHUUNXOGNFbgpaNWhVWjBDa004QW03Wjh6NGJNd0h3WURWUjBqQkJnd0ZvQVVxRXBxWXdSOTNicm0wVG0zcGtWbDcvT283S0V3CmJ3WUlLd1lCQlFVSEFRRUVZekJoTUM0R0NDc0dBUVVGQnpBQmhpSm9kSFJ3T2k4dmIyTnpjQzVwYm5RdGVETXUKYkdWMGMyVnVZM0o1Y0hRdWIzSm5NQzhHQ0NzR0FRVUZCekFDaGlOb2RIUndPaTh2WTJWeWRDNXBiblF0ZURNdQpiR1YwYzJWdVkzSjVjSFF1YjNKbkx6QWxCZ05WSFJFRUhqQWNnZzBxTG0xa1pHZGhiV1V1WTI5dGdndHRaR1JuCllXMWxMbU52YlRCTUJnTlZIU0FFUlRCRE1BZ0dCbWVCREFFQ0FUQTNCZ3NyQmdFRUFZTGZFd0VCQVRBb01DWUcKQ0NzR0FRVUZCd0lCRmhwb2RIUndPaTh2WTNCekxteGxkSE5sYm1OeWVYQjBMbTl5WnpDQ0FRWUdDaXNHQVFRQgoxbmtDQkFJRWdmY0VnZlFBOGdCM0FPSnBTNjRtNk9sQUNlaUdHN1k3ZzlRKzUvNTBpUHVranlpVEFaM2Q4ZHYrCkFBQUJhOVpIamZBQUFBUURBRWd3UmdJaEFKNXBWaDFDSEpmcTFhd2NOYmxEU2FwL1prQmVBeXU5ajcrTVhISnMKTEI3TUFpRUFwM2xLVVNCZXpiQWpodkZWSTBGR3ZFWmtzU2lYKyt3SitiZ3VLOXlaS3JBQWR3QXBQRkdXVk1nNQpaYnFxVVB4WUI5UzNiNzlZZWlseTNLVEREUFRsUlVmMGVBQUFBV3ZXUjQzd0FBQUVBd0JJTUVZQ0lRRDI1L1NHClcrWHRDa2VzaHViekZtUnRnaDUrWXMxaXpnSG5CSmtOS1Z0cE9nSWhBT1lteWJCWjV3RjZBeE5UT29WdnkyYVMKNktEdURyWmRzSVYrN251WkhFSDdNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUNjRHFwTzF3OWdNbzJGaW1GTgpwSUlxT3d1N2hsUWVURU44enY1UmFiYWtGelpvZlhURXpRcGNtSlNWRUhET25MVGpjaWpITWxtbGdIbndTM2w0CjAyWFB0akIzUWJUNFRWUHlqUGpBZ1ZvL1ZmclNJT2N5S1pKRDNJMWxLNXV1anRCdGF3Rnh3cjBYeGd1Q2k5TlUKdlQ2R0RxYnlaVVdiL1I0bXVVYzFwRzMySVJiS3BxQnZveitsaGRMNHdOb1M5YXdiUlg3LzBmUytEZUZiZ09vbgpzYnBDYTFQeFdqWHYwNloxNkF0LzBRTlVZLzExdEw4bTRDK3Q2OW5kOUt6eUdRZmdOank2NmM1RmhIODVBQkNFClJ6L3NoVkdyb1lTQkh3M1Q0c0NKZnh4dW5oK0tVZ0dvRFk5VUc5RzI2T200eHgvWFQ5OTZONTNxUytPS21iY0wKajVJMgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCgotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRWtqQ0NBM3FnQXdJQkFnSVFDZ0ZCUWdBQUFWT0ZjMm9MaGV5bkNEQU5CZ2txaGtpRzl3MEJBUXNGQURBLwpNU1F3SWdZRFZRUUtFeHRFYVdkcGRHRnNJRk5wWjI1aGRIVnlaU0JVY25WemRDQkRieTR4RnpBVkJnTlZCQU1UCkRrUlRWQ0JTYjI5MElFTkJJRmd6TUI0WERURTJNRE14TnpFMk5EQTBObG9YRFRJeE1ETXhOekUyTkRBME5sb3cKU2pFTE1Ba0dBMVVFQmhNQ1ZWTXhGakFVQmdOVkJBb1REVXhsZENkeklFVnVZM0o1Y0hReEl6QWhCZ05WQkFNVApHa3hsZENkeklFVnVZM0o1Y0hRZ1FYVjBhRzl5YVhSNUlGZ3pNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DCkFROEFNSUlCQ2dLQ0FRRUFuTk1NOEZybExrZTNjbDAzZzdOb1l6RHExelVtR1NYaHZiNDE4WENTTDdlNFMwRUYKcTZtZU5RaFk3TEVxeEdpSEM2UGpkZVRtODZkaWNicDVnV0FmMTVHYW4vUFFlR2R4eUdrT2xaSFAvdWFaNldBOApTTXgreWsxM0VpU2RSeHRhNjduc0hqY0FISnlzZTZjRjZzNUs2NzFCNVRhWXVjdjliVHlXYU44aktrS1FESVowClo4aC9wWnE0VW1FVUV6OWw2WUtIeTl2NkRsYjJob256aFQrWGhxK3czQnJ2YXcyVkZuM0VLNkJsc3BrRU5uV0EKYTZ4Szh4dVFTWGd2b3BaUEtpQWxLUVRHZE1EUU1jMlBNVGlWRnJxb003aEQ4YkVmd3pCL29ua3hFejB0TnZqagovUEl6YXJrNU1jV3Z4STBOSFdRV002cjZoQ20yMUF2QTJIM0Rrd0lEQVFBQm80SUJmVENDQVhrd0VnWURWUjBUCkFRSC9CQWd3QmdFQi93SUJBREFPQmdOVkhROEJBZjhFQkFNQ0FZWXdmd1lJS3dZQkJRVUhBUUVFY3pCeE1ESUcKQ0NzR0FRVUZCekFCaGlab2RIUndPaTh2YVhOeVp5NTBjblZ6ZEdsa0xtOWpjM0F1YVdSbGJuUnlkWE4wTG1OdgpiVEE3QmdnckJnRUZCUWN3QW9ZdmFIUjBjRG92TDJGd2NITXVhV1JsYm5SeWRYTjBMbU52YlM5eWIyOTBjeTlrCmMzUnliMjkwWTJGNE15NXdOMk13SHdZRFZSMGpCQmd3Rm9BVXhLZXhwSHNzY2ZyYjRVdVFkZi9FRldDRmlSQXcKVkFZRFZSMGdCRTB3U3pBSUJnWm5nUXdCQWdFd1B3WUxLd1lCQkFHQzN4TUJBUUV3TURBdUJnZ3JCZ0VGQlFjQwpBUllpYUhSMGNEb3ZMMk53Y3k1eWIyOTBMWGd4TG14bGRITmxibU55ZVhCMExtOXlaekE4QmdOVkhSOEVOVEF6Ck1ER2dMNkF0aGl0b2RIUndPaTh2WTNKc0xtbGtaVzUwY25WemRDNWpiMjB2UkZOVVVrOVBWRU5CV0RORFVrd3UKWTNKc01CMEdBMVVkRGdRV0JCU29TbXBqQkgzZHV1YlJPYmVtUldYdjg2anNvVEFOQmdrcWhraUc5dzBCQVFzRgpBQU9DQVFFQTNUUFhFZk5qV0RqZEdCWDdDVlcrZGxhNWNFaWxhVWNuZThJa0NKTHhXaDlLRWlrM0pIUlJIR0pvCnVNMlZjR2ZsOTZTOFRpaFJ6WnZvcm9lZDZ0aTZXcUVCbXR6dzNXb2RhdGcrVnlPZXBoNEVZcHIvMXdYS3R4OC8Kd0FwSXZKU3d0bVZpNE1GVTVhTXFyU0RFNmVhNzNNajJ0Y015bzVqTWQ2am1lV1VISzhzby9qb1dVb0hPVWd3dQpYNFBvMVFZeiszZHN6a0RxTXA0ZmtseEJ3WFJzVzEwS1h6UE1UWitzT1BBdmV5eGluZG1qa1c4bEd5K1FzUmxHClBmWitHNlo2aDdtamVtMFkraVdsa1ljVjRQSVdMMWl3Qmk4c2FDYkdTNWpOMnA4TStYK1E3VU5LRWtST2IzTjYKS09xa3FtNTdUSDJIM2VESkFrU25oNi9ETkZ1MFFnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
  107.   tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS0FJQkFBS0NBZ0VBb2ZJV051MThhSnVPcnN3QmNkT2U4M3R1aldnZ2lReXRVVjFDajVYWG8zY0EzNS9mCnFBc0ZUeklEY3BSbGFMYnpId3V3Wmw5Y1IrMm5EQ1pTMjhWWFlpdytKRC9BemdrcXNMcUk2d2JRYXBzQmtZWFMKNG5PVStnOFIxWDBybnZySmJyQTV4cUlJYkozTTZqNUtNdnhGS28wRXdhc0FjY2JiVUY5bjB0Nkcza3huM1ltUgp6Tkd4eHVtdnhXams1Y2RJYzNrQzJPVW5GS0Y4bld6YlpvYmI2T09SdlJoSURZblh2OTF2MjFGMEJ2dENtRmNBCkRoNGpldSsvM0pUNUtwQmR2QUc4cjdpTXBuSEpoVTVqWGpNeU9HK0xuRy9ycm5HMkZpekdsdVB0MCs0ZStGUUkKcVljUFQzVzJNQXZkOXNBM0QxOGxRbzYzTS9mUzJiM0hhU2J0VjRvWmY1L3NPMk14RU9WelRHdTNDcWJONE5HKwpkTzJxejFhbExCaUZmVWM0R2ZRWlhGaUtoMXNrOXdCbnN5aGphRmZ1THpsdHEwODdJMkthK1ZUWlMxUEpSRWxnCm4zdTBjUWZoQ2N6UXllMndWOXpET2VWZTF5MGMvNnRFYmE2WUJ4ZHZkZzA4WkovRDBhMEtyb1pVSTFuUXNkSngKTytDc3U5REtiNE5nMkJidmRZWkdVYmtIK1JQNTRRR2tLVWdidXE1UjBtcjRTUjZVTCtETmYzcXpvNWt2YjF1UQpZcWloNlhQVUNVRU85M05TVjYxM1RJRU5NSnJiNWFsZEtSSE9mWlYvbTVBOGVTL2JsUVhyTldxQkcvTi9kbXJGCmMyZzI0YkRjd3k5cjMvcWQ3L0xNbGYxVFV3MkZzNHczbHZUckVqeXBYRmFDcFFEbGRzTElGMHFlSlZzQ0F3RUEKQVFLQ0FnQXY5Zk13UnpzTisrdlF4cWd5M3JwM1gzbkpOU3BWakVTVUVTdVNQSTFGWXd3R0xtSGRjWTRiK3pMYwpMeWl0VDJsSEszNE5nM1pmOHZrQzl5S1k1YVBRZGt2ZERtaDZYR3FoTmswd1ZhOUpzeWhPd2JSSHpuVXpiVjBaCnZkMDZVd2x1MTQvMHpLMzBCUFBYOTZTZjN1aFpCclIrNnJiUisxT2VSUE1KbDArWDdFYmliRWlhd1F1R1hsVHAKQVB5eE5FaTNzZ0h1M0VhcnJIdXNYNzNHYW5BY1U3RW9zRlUrZFRGSktEcGxXSVVsUUNwajFYZzF0aVZKMWxFYQo4Wit0UkY0T1BQRjFsUkZLaGU1cHBXSjJWbkVzRjVUZ09xRXc0NHBLbk80Zlo5ZGFhVzRRbTBxSmNtOU5XQTRoCndwSDA3czRmcGt6eG5qU1JsbmFDZDlyandGeVBsSkJzUXNhVlFFNzlpQzJZMTRnTk9KQ0xyMXRKSEQ2ODN3bW4KS3ZNOHZpOTdHTmIybXZHeWNtZnloNVpzTFBpTWNqOFFER3VWZU53dlNESXpybnhqVkZlc0liTWt5UlZRem9IVApTTHRQbXdVR3lwRHVrMDhaZytsT0lYOC85K3lqMER3MDRqenllTVptYlFVdkd2N2lNWjFUaHdaRHF1YkJXV3J4CmtYTmJwTG9BMGxrcHh4bjdGam9Ya20zM2ZKQURjd2xWSS82WFNrSm1FaFVlZmZnaFFSMGNyVGphQVd1Qkx2Qk0KT0s5aEEzT3RTN2F0S2FDb1lvSmRrYkpHQTdWdytNNzA4NEJOTGhxM1Fyckg4S3M3Z05pdC9NN3lxSnU1alBaZgo2SE1seHNyWU9NVUhuVlk4VDkwN0Q3cS9ORUNnRThzODhnZzAyQ3JNWTFqanE4UnBpUUtDQVFFQTE2UHJaMUEwClNISS83akdmS3BETkJzQ0xrVUFxRERKSzQ0dFdJYmJBUXFhRTN1eDh3bkFlU2NjSHozbS9ScEpPSGtteHZTZlgKbTJ1Wk8veGtNTWhYK2lwOHdFOHZibzR1enVNYitTSXE3bWpialJkK1JJczJ5NHJsZVQ2NGVjRWc4R2pZckExZgpiSEI0MmhQclVTcXpxUVIwOTZocm1Lb1diU0RDZDZwOUVNeWVzT3IwTjdtQmJYVVZPazJxZGtYRlZWbHBlUDdpClFxWGdRUUI0bHgzLzJJdlpBMlhJUXlQdGJ0RWVRbmgyQ3FNM2NDMzR0VEVjZ244K0VwNG9SWmkwTTBHaUY3bXgKOTEvZHY2THZlNTR5K1pON1lXd1NFQ09ubzd5bDlvTlBZVnVGMGRiMjh0elppMThCeHJTQ2JESE1XbExvUzhWNgpXTEo0OGlSODJDYkc1d0tDQVFFQXdFRjM4KzYyeDhDU2x0blZZNlJaN0J0NEdiNEJqVWhWYXZ0NFkxUGFlbXFNCjFidFVnR2JyUnBoNHFUSEFTckUwUUZLeVZKYnlCUkJyRHIxWHU4WWRSVXQzZC92VzlIR1dPd1BKdTN2M3pLbHMKQ2xsZnpFY3J5L1l2aHAzSzlEcGR6OE1icHdueW5xcGV6b0xMNlJpL3JnK0hyTzBueXd1RSt0T2xYVFo2eUtadApHWVdTSVBWaG00NUJkc2ZxUzhnYjVvbjA0bHh3bnhxVnJvN0c0TUR6cmVEYlFhaGdyS3VuRWxwajZ4eW1PVWpBCkdCZDR3QUVrUExxNUUrRWcreDY4TkRLVTYwK29ybFhLWVhDQm5HSFZOQ3BVcmswVXkrcHFZZmFEN3VuR2VzaHMKSEwra3lXbXl5a3ErTmNKbnRXMFNSNy9sU1IvZUFhVEZyVzZVaXV0RGJRS0NBUUVBemhRYU9PNmVPSW51N016QgpScVdCT3EyeDg4cjFKQmpBRnZzbkFpc3JTOGJsZmtGVTdXREdvVTB5K3FWb0ZhSm1RMjI4RFlCUS9YZnp4aTdxCjlPL1JuQU1VbTVoUlJQOWVYbHNPZGFXZ2o1em9ETXRoNFZHRnVUbHhHZERGN1oyU3hBMysyMVlnVm5xYUZCY3IKTUxOMVpOWWNqajJITGl1R0tSNUFtcW4wd2FRN0YrcENJQ3NKTkxqSzQ2QXJnc0lrMXU4TzdCSHgyeTI0eFlZVQp1SjV6emRmQU9nNEFONkhURzY5L2twaWFmb29DeGhNNDlyZ0xmZTdxUEZLbk8vTzJhckdUbmNiWi9BWEMzb3h3Ci81dHRMYlF6R2lSMGtyWHdWSHRKdys4elltQmIzL0RtcWF4RHZueTZMdEo5UGJiTmk1aGw1VnZCRTVqa0dzeWgKL3RQNEN3S0NBUUJ2R1dZb0lKcWZkRGxCMHovdEJOeXlCRzJ5OG9vVEN1blJtT0JKQmZ3TEllZWcyMUJKb3kveQo2OGxPZk9HU1NEVFp0dkEyMGNPcUNZTFVVYmFSWERzdUFCNVp4NzdBSTZPZEZ1Tk01S2FlTG9td3NWVWF4MFlYCjUzd3ZYcUFaNG1DejN4dnJ1MlBwTEtyOHk3anFTdEw1MHgra1hxZlFQaWZxaXNQVXlkYktmT0l2RFhFVWVyaWQKRytmWXJFNUkzS3JDM3BZVStUWmJ1eEVrZm4yUEEvSE5XVk5hN2VKdjVnSDJLU1gwaCtuRzBMT3hPRjhmRlluTApUbHdGa09OdU9xU254Vk1wYUM4aUQ1R1VIVi9JN3dBMTFRQjZlVEM3Wmd0ejhQRHM3MHN6U1A2dzNrNXIxaGpyCnJhV2RpMnBDL1hUQzRiR3VRQ3dhNXcwVTNBSWJCVGxCQW9JQkFEc1RONGhvclVHNWw3MXhLZk5ibVBTbDZ6RlIKYTJ4d2U2VVZPOVZzMFpHeEdLWWJSN1VuVDBDL1FqUiswS2JsbE9leDdFY3cyMklCcmFFVzBGbXpuVnoyUW9FNwpMUE5COXhyTTFEeE56UjZEbFBUeERMcEFGWVlUcm40SWY1cjFVdVdpc2lMdmd6T2xGTlVITnN5UFJIZWNGblhUCnNhTk9JWkgrQTJ5KzF3QWdpSFZIS2JPRGRHeVFQVlQ0TXFFWkJaY2pQcmRBekNKcnloSHlYdHBqRjFSdlFEYTMKTVM3U3JVTGM4djJGQWJ1VG1QZ2R1ZHBKd1Q4dENCa2VRKzZ4YmJWN3YrZzBEMG5EWFNIZFVwNXFyUzcrTnhtVwp4NWV4UHo1VENhYXcxSnkzWjRmT1MzMTV6eHJGdmRHTmhWRXhMMzRlUVlzOHRYN0N0VWxuWkNray9zYz0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0=
  108. ---
  109.  
  110. apiVersion: v1
  111. kind: Secret
  112. metadata:
  113.   labels:
  114.     k8s-app: kubernetes-dashboard
  115.   name: kubernetes-dashboard-csrf
  116.   namespace: kubernetes-dashboard
  117. type: Opaque
  118. data:
  119.   csrf: ""
  120.  
  121. ---
  122.  
  123. apiVersion: v1
  124. kind: Secret
  125. metadata:
  126.   labels:
  127.     k8s-app: kubernetes-dashboard
  128.   name: kubernetes-dashboard-key-holder
  129.   namespace: kubernetes-dashboard
  130. type: Opaque
  131.  
  132. ---
  133.  
  134. kind: ConfigMap
  135. apiVersion: v1
  136. metadata:
  137.   labels:
  138.     k8s-app: kubernetes-dashboard
  139.   name: kubernetes-dashboard-settings
  140.   namespace: kubernetes-dashboard
  141.  
  142. ---
  143.  
  144. kind: Role
  145. apiVersion: rbac.authorization.k8s.io/v1
  146. metadata:
  147.   labels:
  148.     k8s-app: kubernetes-dashboard
  149.   name: kubernetes-dashboard
  150.   namespace: kubernetes-dashboard
  151. rules:
  152.   # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  153.   - apiGroups: [""]
  154.     resources: ["secrets"]
  155.     resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
  156.     verbs: ["get", "update", "delete"]
  157.     # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  158.   - apiGroups: [""]
  159.     resources: ["configmaps"]
  160.     resourceNames: ["kubernetes-dashboard-settings"]
  161.     verbs: ["get", "update"]
  162.     # Allow Dashboard to get metrics.
  163.   - apiGroups: [""]
  164.     resources: ["services"]
  165.     resourceNames: ["heapster", "dashboard-metrics-scraper"]
  166.     verbs: ["proxy"]
  167.   - apiGroups: [""]
  168.     resources: ["services/proxy"]
  169.     resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
  170.     verbs: ["get"]
  171.  
  172. ---
  173.  
  174. kind: ClusterRole
  175. apiVersion: rbac.authorization.k8s.io/v1
  176. metadata:
  177.   labels:
  178.     k8s-app: kubernetes-dashboard
  179.   name: kubernetes-dashboard
  180. rules:
  181.   # Allow Metrics Scraper to get metrics from the Metrics server
  182.   - apiGroups: ["metrics.k8s.io"]
  183.     resources: ["pods", "nodes"]
  184.     verbs: ["get", "list", "watch"]
  185.  
  186. ---
  187.  
  188. apiVersion: rbac.authorization.k8s.io/v1
  189. kind: RoleBinding
  190. metadata:
  191.   labels:
  192.     k8s-app: kubernetes-dashboard
  193.   name: kubernetes-dashboard
  194.   namespace: kubernetes-dashboard
  195. roleRef:
  196.   apiGroup: rbac.authorization.k8s.io
  197.   kind: Role
  198.   name: kubernetes-dashboard
  199. subjects:
  200.   - kind: ServiceAccount
  201.     name: kubernetes-dashboard
  202.     namespace: kubernetes-dashboard
  203.  
  204. ---
  205.  
  206. apiVersion: rbac.authorization.k8s.io/v1
  207. kind: ClusterRoleBinding
  208. metadata:
  209.   name: kubernetes-dashboard
  210.   namespace: kubernetes-dashboard
  211. roleRef:
  212.   apiGroup: rbac.authorization.k8s.io
  213.   kind: ClusterRole
  214.   name: kubernetes-dashboard
  215. subjects:
  216.   - kind: ServiceAccount
  217.     name: kubernetes-dashboard
  218.     namespace: kubernetes-dashboard
  219.  
  220. ---
  221.  
  222. kind: Deployment
  223. apiVersion: apps/v1
  224. metadata:
  225.   labels:
  226.     k8s-app: kubernetes-dashboard
  227.   name: kubernetes-dashboard
  228.   namespace: kubernetes-dashboard
  229. spec:
  230.   replicas: 1
  231.   revisionHistoryLimit: 10
  232.   selector:
  233.     matchLabels:
  234.       k8s-app: kubernetes-dashboard
  235.   template:
  236.     metadata:
  237.       labels:
  238.         k8s-app: kubernetes-dashboard
  239.     spec:
  240.       containers:
  241.         - name: kubernetes-dashboard
  242.           image: kubernetesui/dashboard:v2.0.0-beta4
  243.           imagePullPolicy: Always
  244.           ports:
  245.             - containerPort: 8443
  246.               protocol: TCP
  247.           args:
  248.             - --auto-generate-certificates
  249.             - --namespace=kubernetes-dashboard
  250.             - --token-ttl=43200
  251.             # Uncomment the following line to manually specify Kubernetes API server Host
  252.             # If not specified, Dashboard will attempt to auto discover the API server and connect
  253.             # to it. Uncomment only if the default does not work.
  254.             # - --apiserver-host=http://my-address:port
  255.           volumeMounts:
  256.             - name: kubernetes-dashboard-certs
  257.               mountPath: /certs
  258.               # Create on-disk volume to store exec logs
  259.             - mountPath: /tmp
  260.               name: tmp-volume
  261.           livenessProbe:
  262.             httpGet:
  263.               scheme: HTTPS
  264.               path: /
  265.               port: 8443
  266.             initialDelaySeconds: 30
  267.             timeoutSeconds: 30
  268.       volumes:
  269.         - name: kubernetes-dashboard-certs
  270.           secret:
  271.             secretName: kubernetes-dashboard-certs
  272.         - name: tmp-volume
  273.           emptyDir: {}
  274.       serviceAccountName: kubernetes-dashboard
  275.       # Comment the following tolerations if Dashboard must not be deployed on master
  276.       tolerations:
  277.         - key: node-role.kubernetes.io/master
  278.           effect: NoSchedule
  279.  
  280. ---
  281.  
  282. kind: Service
  283. apiVersion: v1
  284. metadata:
  285.   labels:
  286.     k8s-app: dashboard-metrics-scraper
  287.   name: dashboard-metrics-scraper
  288.   namespace: kubernetes-dashboard
  289. spec:
  290.   ports:
  291.     - port: 8000
  292.       targetPort: 8000
  293.   selector:
  294.     k8s-app: dashboard-metrics-scraper
  295.  
  296. ---
  297.  
  298. kind: Deployment
  299. apiVersion: apps/v1
  300. metadata:
  301.   labels:
  302.     k8s-app: dashboard-metrics-scraper
  303.   name: dashboard-metrics-scraper
  304.   namespace: kubernetes-dashboard
  305. spec:
  306.   replicas: 1
  307.   revisionHistoryLimit: 10
  308.   selector:
  309.     matchLabels:
  310.       k8s-app: dashboard-metrics-scraper
  311.   template:
  312.     metadata:
  313.       labels:
  314.         k8s-app: dashboard-metrics-scraper
  315.     spec:
  316.       containers:
  317.         - name: dashboard-metrics-scraper
  318.           image: kubernetesui/metrics-scraper:v1.0.1
  319.           ports:
  320.             - containerPort: 8000
  321.               protocol: TCP
  322.           livenessProbe:
  323.             httpGet:
  324.               scheme: HTTP
  325.               path: /
  326.               port: 8000
  327.             initialDelaySeconds: 30
  328.             timeoutSeconds: 30
  329.           volumeMounts:
  330.           - mountPath: /tmp
  331.             name: tmp-volume
  332.       serviceAccountName: kubernetes-dashboard
  333.       # Comment the following tolerations if Dashboard must not be deployed on master
  334.       tolerations:
  335.         - key: node-role.kubernetes.io/master
  336.           effect: NoSchedule
  337.       volumes:
  338.         - name: tmp-volume
  339.           emptyDir: {}
  340. ---
  341. apiVersion: extensions/v1beta1
  342. kind: Ingress
  343. metadata:
  344.   labels:
  345.     k8s-app: kubernetes-dashboard
  346.   name: kubernetes-dashboard
  347.   namespace: kubernetes-dashboard
  348.   annotations:
  349.     kubernetes.io/ingress.class: traefik
  350.     traefik.ingress.kubernetes.io/frontend-entry-points: http,https
  351.     traefik.ingress.kubernetes.io/redirect-entry-point: https
  352. spec:
  353.   rules:
  354.   - host: csdd.xxxx.com
  355.     http:
  356.       paths:
  357.         - backend:
  358.             serviceName: kubernetes-dashboard
  359.             servicePort: 443
  360.   tls:
  361.    - secretName: dashboard-tls-cert
  362. # 创建kubernetes-dashboard token 登录
  363. #  生成token
  364. kubectl create sa dashboard-admin -n kube-system
  365.  # 授权token 访问权限
  366. kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
  367. # 获取token 
  368. ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')
  369. # 获取dashboard.kubeconfig 使用token   值
  370. DASHBOARD_LOGIN_TOKEN=$(kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}')
  371. echo ${DASHBOARD_LOGIN_TOKEN}
  372. # 设置集群参数
  373. kubectl config set-cluster kubernetes \
  374.   --certificate-authority=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem \
  375.   --embed-certs=true \
  376.   --server=${KUBE_APISERVER} \
  377.   --kubeconfig=dashboard.kubeconfig
  378.  
  379. # 设置客户端认证参数，使用上面创建的 Token
  380. kubectl config set-credentials dashboard_user \
  381.   --token=${DASHBOARD_LOGIN_TOKEN} \
  382.   --kubeconfig=dashboard.kubeconfig
  383.  
  384. # 设置上下文参数
  385. kubectl config set-context default \
  386.   --cluster=kubernetes \
  387.   --user=dashboard_user \
  388.   --kubeconfig=dashboard.kubeconfig
  389.  
  390. # 设置默认上下文
  391. kubectl config use-context default --kubeconfig=dashboard.kubeconfig
  392. # 绑定hosts
  393. https://csdd.xxxx.com/#/overview?namespace=default
  394. # kubernetes-dashboard 使用metrics 显示cpu内存资源 所有要部署metrics-server

metrics-server 部署

  1. # win on Ubuntu  操作 
  2. #创建metrics-server 证书
  3. at << EOF | tee /apps/work/k8s/cfssl/k8s/metrics-server.json
  4. {
  5.   "CN": "metrics-server",
  6.   "key": {
  7.     "algo": "rsa",
  8.     "size": 2048
  9.   },
  10.   "names": [
  11.     {
  12.       "C": "CN",
  13.       "ST": "GuangDong",
  14.       "L": "GuangZhou",
  15.       "O": "cluster",
  16.       "OU": "cluster"
  17.     }
  18.   ]
  19. }
  20. EOF
  21. ### 生成证书
  22. cfssl gencert -ca=/apps/work/k8s/cfssl/pki/k8s/k8s-ca.pem -ca-key=/apps/work/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
  23.     -config=/apps/work/k8s/cfssl/ca-config.json \
  24.     -profile=kubernetes /apps/work/k8s/cfssl/k8s/metrics-server.json | cfssljson -bare ./metrics-server
  25. # 创建metrics-server-secrets.yaml
  26. # base64 加密
  27. cat metrics-server.pem|base64 | tr -d '\n'
  28. cat metrics-server-key.pem|base64 | tr -d '\n'
  29. vi metrics-server-secrets.yaml
  30. apiVersion: v1
  31. kind: Secret
  32. metadata:
  33.   labels:
  34.     k8s-app: metrics-server
  35.   name: metrics-server-certs
  36.   namespace: kube-system
  37. type: Opaque
  38. data:
  39.   metrics-server.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ3VENDQXRXZ0F3SUJBZ0lVYkloOTQ3Z3NET2gxdVplVnBoMU9GVmhYeHA0d0RRWUpLb1pJaHZjTkFRRUwKQlFBd2JqRUxNQWtHQTFVRUJoTUNRMDR4RWpBUUJnTlZCQWdUQ1VkMVlXNW5SRzl1WnpFU01CQUdBMVVFQnhNSgpSM1ZoYm1kYWFHOTFNUkF3RGdZRFZRUUtFd2R0WkdSbllXMWxNUkF3RGdZRFZRUUxFd2R0WkdSbllXMWxNUk13CkVRWURWUVFERXdwcmRXSmxjbTVsZEdWek1CNFhEVEU1TURjd05UQXpOVGd3TUZvWERUSTVNRGN3TWpBek5UZ3cKTUZvd2NqRUxNQWtHQTFVRUJoTUNRMDR4RWpBUUJnTlZCQWdUQ1VkMVlXNW5SRzl1WnpFU01CQUdBMVVFQnhNSgpSM1ZoYm1kYWFHOTFNUkF3RGdZRFZRUUtFd2R0WkdSbllXMWxNUkF3RGdZRFZRUUxFd2R0WkdSbllXMWxNUmN3CkZRWURWUVFERXc1dFpYUnlhV056TFhObGNuWmxjakNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0MKQVFvQ2dnRUJBTUlvdHhsakpLcCtyb1hGcGJJWXRuNW1mVXVyREM1bUlkb2Z0RTNSVGhNU1pPSCt0aVVoMDdQRQpnb0xPOG1lSkxaS21ob1BUYzhJWTJYVjdiZzJWWFpRVUd6MFJuMExmNWdWam5UN29yMFFnZzErYnJnZU5wWUtpCjVoNm5ISVE3ZGlKYk10cUFndm16RGR6bWdoUXd2SHBKTzh3bEQwKzRwT0VHT2VtQkNPU3BsaFhrenR3UWQ3ZHYKY2x1QUljQUdiUGF6dzI4VkJJU2F4bCtrZnZwNzIyeEkvVy9DL3pRS1JnN053UG9IaVpFWm9QcGxPY001cGpvUwpJeEdnWVZEYjB6OGlqZWR3RjZmcE9RZkFOcitvQnVONnZnMXAzd2Jud2tKTWtEQUV2MzBXZG1BUzB5STJicC9RCkJZYjU2VWxGTXI4anNoWHJ5dlVsZ3F3S0hscFh0WkVDQXdFQUFhTi9NSDB3RGdZRFZSMFBBUUgvQkFRREFnV2cKTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEVBakFBTUIwRwpBMVVkRGdRV0JCVEp6cVJBMWdIN***d3B3TG01ZEtWUHFvdG43VEFmQmdOVkhTTUVHREFXZ0JSeDRLY0FSY2FrClIvYnhtd29UQmVEcyt4QW9hVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBcTgyS0JRMWJBM1pjeG9OZGFJVVUKNjVvSkZhM3paM2k5SDk1QWZvU0dXQWQvbFU3L3B2Qkt2K0EwbS9qQ0tMS1FpQ0xXRGtHYzFUNS8xTzhPYTg0YgpDV3I5M3pOWldRb2N6UWJEaHFCRnZNUVp0azRYVVkrMjR3Yjd3cmJCRHc2QVY2R0l2bVMrYm91eFRVd29sbmRMCk5FS2EvcHNvQUtaRUFJZkJUaCtMNVpMQ09GOXFUWEMyOGtnN1czak4vMzBiYlk5UE5ObVpLcGNaNEpEVjA5aGYKU3RaTjZuOVFXK3ZDcFFoZXVISWVORlR2RnQ5bGtSMVBFYUtHUjFiWEdyeUNHOHNTeXVDc0xER1lnVlhmYVZtYgp3dTlnSG1JS2E2aDZWVmVIWitMbVFmZmxqcEdPRStKV1l1TWRPamtHYUYxdEtLUWZFelZGN3BxT0VTQXkrV3hpCnVBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  40.   metrics-server-key.pem: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBd2lpM0dXTWtxbjZ1aGNXbHNoaTJmbVo5UzZzTUxtWWgyaCswVGRGT0V4Sms0ZjYyCkpTSFRzOFNDZ3M3eVo0a3RrcWFHZzlOendoalpkWHR1RFpWZGxCUWJQUkdmUXQvbUJXT2RQdWl2UkNDRFg1dXUKQjQybGdxTG1IcWNjaER0Mklsc3kyb0NDK2JNTjNPYUNGREM4ZWtrN3pDVVBUN2lrNFFZNTZZRUk1S21XRmVUTwozQkIzdDI5eVc0QWh3QVpzOXJQRGJ4VUVoSnJHWDZSKytudmJiRWo5YjhML05BcEdEczNBK2dlSmtSbWcrbVU1Cnd6bW1PaElqRWFCaFVOdlRQeUtONTNBWHArazVCOEEydjZnRzQzcStEV25mQnVmQ1FreVFNQVMvZlJaMllCTFQKSWpadW45QUZodm5wU1VVeXZ5T3lGZXZLOVNXQ3JBb2VXbGUxa1FJREFRQUJBb0lCQUQzaEtoSGVSalpNYjZUVQp6QzFKc1FmeDlDYmttZHBEMUxBbkFKajRoekNPNFRZaHJyMkEzVzdpeDFHWFVTeHNUT2o3UjgzRjI1UFZ4YUNxCnVQVjlVRGk4ZTczbjJ1RSthSm41R0ltUE1TUytWQUJwcG5wank0Y3FFYnRkT1RwdmxRUDRHdW9Hb1RlaHVGNVoKM01WQWp5Rk9LOCt4VkFMdGJ5Y0VpL3ArbEc0RGkvOThIcUlDQngwSlhCUnJoV05lWUdZL0c3eGNWT2pCNUl5SQpPNVpoZ1I0Sk9yODloNVZ1RHdIY3E2UVlLQ2sxTktQZzc0Y3BOY2J5ZVZEM0FhYVRHd25QU3BMd0hGaElzTGpNCkllaEJqZzkrZDdyRU8xMHU0azhKWW5qYUdNMzRMM0RlTVVrck95NUMzRjY3RFNwaTJVZUhjOVh6YzViVUFwb0gKTE1zRUxuMENnWUVBenV3STRSN0FrYmZ2VnVFR0hOa0FFNjgzdTJoWk1iUFdwSHVVTGQ5UzJnUzYzODVWUFdJSApiQXp6WGZuTjh1U2dHNVVQdzUxa1VLZVppY0pORTVBWWtOVDRwMWxoTFdKSkwxSWRSdEV3VU5oblVLNlczRWlMCmJLeDhhalk3dkZDV0ZKUmRTUHJYLzViTWU4TVBCWWNTT0FkZEErZFhlaGd0K2x0WEU3LzE1cGNDZ1lFQThEVzcKOEIyZGNYLzZqMVF4UkhEc3ZGUDYxaWxTREJhMEtKUGlobnpSZWVwVHRFc0hvRitBODk2SXRDZFFMSTJZcEZyRApBU1dSSU1VQVVzVE1uMStvZFprOGovd21QRkxzUmRpSVJWZC80ZHdCTmlDNHJxdnkwQTFxUVNJUXF1MC9CcFV2ClRpMjhZeURRdHh0Wmg1d0NDQUx2a0Rqb2N4cXJzbHEwRDViTnNoY0NnWUFFNjB1U3ZuNXBlR3Z5THN0TWFFUSsKTTQ0cG9HTnVjUTRMUHZ6WTBsUlA1b2ppeFM3MWxKMEVqcEpDaTZ1L3VIdXQ3QzlCd1Q0ZlJMeTdyNmpSYkQySgpRK2JkWTV5UnphSmJ3Nkg2aXdLUkNYUDdVUXM1Rldockh2YWVOOGZYeERxdEpwSEpLRjEyTUFtUWI2U3R4dlpCCjZycmxXdHlUaEh1alZnU043YVJVNVFLQmdHbVFJN3lkTnpERy9sVDR1Z0lLNG03Tk5VSGl2TlRsTVYxWHlFZzAKR0ZiTW5PWnh4ck02NVUvRzd5ckUwQjRVU0EyS2VZSktnU0gya1hMT1crSjZSbTBQMzZhak9DWndocmNYTnFQSwpsVCtyMExoNTNzK2NiMFB4Y1UyWWE5ekNFRjJUT0V2U0c2VXdxYWllazFUZVFhSkZzQVFnamo3dmJKOGY3MXVlCmVWMFhBb0dCQUlQWDN1OTJtU3o5clpqOTN3NFNKbUk2K0tkY3BtTGg0TE5IcUdBOVJSNi80T0NWWDBHNHVtZ1YKMkxUOU1MY05CdUVFZE5nTHRHeGIzdTI5cUlLZHZpcDhBZzlTbHQvMlBIcnlsLzIzWWU0bURTWDVUOXNrVXhaRgpjVGxvN3QxSnRZN3NaRU00Vng4cWxQbDcwWXMvSWRuWmhWaFU2d1F2ZGp0TGk1UlU4L2ttCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
  41. # resource-reader.yaml
  42. vi resource-reader.yaml
  43. ---
  44. apiVersion: rbac.authorization.k8s.io/v1
  45. kind: ClusterRole
  46. metadata:
  47.   name: system:metrics-server
  48. rules:
  49. - apiGroups:
  50.   - ""
  51.   resources:
  52.   - pods
  53.   - nodes
  54.   - nodes/stats
  55.   verbs:
  56.   - get
  57.   - list
  58.   - watch
  59. ---
  60. apiVersion: rbac.authorization.k8s.io/v1
  61. kind: ClusterRoleBinding
  62. metadata:
  63.   name: system:metrics-server
  64. roleRef:
  65.   apiGroup: rbac.authorization.k8s.io
  66.   kind: ClusterRole
  67.   name: system:metrics-server
  68. subjects:
  69. - kind: ServiceAccount
  70.   name: metrics-server
  71.   namespace: kube-system
  72. # metrics-server-service.yaml
  73. vi metrics-server-service.yaml
  74. ---
  75. apiVersion: v1
  76. kind: Service
  77. metadata:
  78.   name: metrics-server
  79.   namespace: kube-system
  80.   labels:
  81.     kubernetes.io/name: "Metrics-server"
  82. spec:
  83.   selector:
  84.     k8s-app: metrics-server
  85.   ports:
  86.   - port: 443
  87.     protocol: TCP
  88.     targetPort: 443
  89. # metrics-apiservice.yaml
  90. vi metrics-apiservice.yaml
  91. ---
  92. apiVersion: apiregistration.k8s.io/v1beta1
  93. kind: APIService
  94. metadata:
  95.   name: v1beta1.metrics.k8s.io
  96. spec:
  97.   service:
  98.     name: metrics-server
  99.     namespace: kube-system
  100.   group: metrics.k8s.io
  101.   version: v1beta1
  102.   insecureSkipTLSVerify: true
  103.   groupPriorityMinimum: 100
  104.   versionPriority: 100
  105.  
  106. # auth-reader.yaml
  107. vi auth-reader.yaml
  108. ---
  109. apiVersion: rbac.authorization.k8s.io/v1beta1
  110. kind: RoleBinding
  111. metadata:
  112.   name: metrics-server-auth-reader
  113.   namespace: kube-system
  114. roleRef:
  115.   apiGroup: rbac.authorization.k8s.io
  116.   kind: Role
  117.   name: extension-apiserver-authentication-reader
  118. subjects:
  119. - kind: ServiceAccount
  120.   name: metrics-server
  121.   namespace: kube-system
  122.  
  123. #  auth-delegator.yaml
  124. vi auth-delegator.yaml
  125. ---
  126. apiVersion: rbac.authorization.k8s.io/v1beta1
  127. kind: ClusterRoleBinding
  128. metadata:
  129.   name: metrics-server:system:auth-delegator
  130. roleRef:
  131.   apiGroup: rbac.authorization.k8s.io
  132.   kind: ClusterRole
  133.   name: system:auth-delegator
  134. subjects:
  135. - kind: ServiceAccount
  136.   name: metrics-server
  137.   namespace: kube-system
  138. # aggregated-metrics-reader.yaml
  139. vi aggregated-metrics-reader.yaml
  140. kind: ClusterRole
  141. apiVersion: rbac.authorization.k8s.io/v1
  142. metadata:
  143.   name: system:aggregated-metrics-reader
  144.   labels:
  145.     rbac.authorization.k8s.io/aggregate-to-view: "true"
  146.     rbac.authorization.k8s.io/aggregate-to-edit: "true"
  147.     rbac.authorization.k8s.io/aggregate-to-admin: "true"
  148. rules:
  149. - apiGroups: ["metrics.k8s.io"]
  150.   resources: ["pods"]
  151.   verbs: ["get", "list", "watch"]
  152.  
  153. # metrics-server-deployment.yaml
  154. vi  metrics-server-deployment.yaml
  155. ---
  156. apiVersion: v1
  157. kind: ServiceAccount
  158. metadata:
  159.   name: metrics-server
  160.   namespace: kube-system
  161. ---
  162. apiVersion: apps/v1
  163. kind: Deployment
  164. metadata:
  165.   name: metrics-server
  166.   namespace: kube-system
  167.   labels:
  168.     k8s-app: metrics-server
  169. spec:
  170.   selector:
  171.     matchLabels:
  172.       k8s-app: metrics-server
  173.   template:
  174.     metadata:
  175.       name: metrics-server
  176.       labels:
  177.         k8s-app: metrics-server
  178.     spec:
  179.       serviceAccountName: metrics-server
  180.       tolerations:
  181.         - effect: NoSchedule
  182.           key: node.kubernetes.io/unschedulable
  183.           operator: Exists
  184.         - key: NoSchedule
  185.           operator: Exists
  186.           effect: NoSchedule
  187.       volumes:
  188.       # mount in tmp so we can safely use from-scratch images and/or read-only containers
  189.       - name: tmp-dir
  190.         emptyDir: {}
  191.       - name: metrics-server-certs
  192.         secret:
  193.           secretName: metrics-server-certs
  194.       containers:
  195.       - name: metrics-server
  196.         image: juestnow/metrics-server-amd64:v0.3.3
  197.         imagePullPolicy: Always
  198.         command:
  199.         - /metrics-server
  200.         - --tls-cert-file=/certs/metrics-server.pem
  201.         - --tls-private-key-file=/certs/metrics-server-key.pem
  202.         - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
  203.         - --kubelet-insecure-tls
  204.         volumeMounts:
  205.         - name: tmp-dir
  206.           mountPath: /tmp
  207.         - name: metrics-server-certs
  208.           mountPath: /certs
  209. # 创建metrics-server 服务
  210. kubectl apply -f .
  211. # 验证metrics-server 
  212. kubectl top node
  213. [root@]~]#kubectl top node
  214. NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
  215. master   177m         7%     2057Mi          35%
  216. [root@]~]#kubectl top pods -A
  217. NAMESPACE              NAME                                        CPU(cores)   MEMORY(bytes)
  218. clusterstorage         nfs-client-provisioner-5f6bc44cd7-fjr7f     3m           15Mi
  219. kube-system            coredns-597b77445b-fhxvr                    4m           23Mi
  220. kube-system            kube-router-5tmgw                           9m           16Mi
  221. kube-system            metrics-server-66d78c47-zn679               1m           14Mi
  222. kube-system            traefik-578574dfdb-dzl22                    6m           41Mi
  223. kubernetes-dashboard   dashboard-metrics-scraper-fb986f88d-rc6zs   1m           25Mi
  224. kubernetes-dashboard   kubernetes-dashboard-668c4f84bc-w6vw6       2m           40Mi
  225.  
  226. # 能够正常获取CPU 内存值

编译安装kubernetes 1.15.3
编译安装kubernetes 1.15.3

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/繁依Fanyi0/article/detail/561676
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号