k8s 安装harbor_kubernetes 安装harbor

前言
文章篇幅较长，请详细阅读，不要漏掉每一个注释，每一个细节。
文章中用到的harbor存储是hostpath，service的类型是ingress。根据harbor官网提示，需要提前创建pv和pvc，既然选择了hostpath，那么就需要将pod固定到某一个node上面，本文所有的资源副本都是1，harbor的所有pod放在同一个namespace，并且需要将namespace固定到某个node上，具体步骤见正文。

一.环境准备

2.创建namespace
cat namespace-harbor.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: harbor

kubectl apply -f namespace-harbor.yaml

3.开启准入控制器

如果是多master，所有master都要修改
–enable-admission-plugins 添加 PodNodeSelector

vim /etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.0.20:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.0.20
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,PodNodeSelector

apiserver pod会自动重启

4.命名空间加注解
kubectl edit ns harbor -o yaml

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: harbor=env
  name: harbor

5.node打标签
kubectl label node k8s-node3 harbor=env

6.创建pv和pvc
选择hostpath，需要提前准备pv和pvc，官网解释如下：

官网链接：https://goharbor.io/docs/2.7.0/install-config/harbor-ha-helm/

在k8s-node3节点创建目录，建议给/data下单独挂一块存储盘

df -h

mkdir -pv /data/harbor/{chartmuseum,database,jobservice,redis,registry,scandata,trivy}
mkdir: created directory ‘harbor’
mkdir: created directory ‘harbor/chartmuseum’
mkdir: created directory ‘harbor/database’
mkdir: created directory ‘harbor/jobservice’
mkdir: created directory ‘harbor/redis’
mkdir: created directory ‘harbor/registry’
mkdir: created directory ‘harbor/scandata’
mkdir: created directory ‘harbor/trivy’
chmod 777 -R /data/harbor   ##必须要赋权，不然pod启动会有问题

创建pv和pvc

cat harbor-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-registry-pv"
  labels:
    name: harbor-registry-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/registry
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-chartmuseum-pv"
  labels:
    name: harbor-chartmuseum-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/chartmuseum
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-jobservice-pv"
  labels:
    name: harbor-jobservice-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/jobservice
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-database-pv"
  labels:
    name: harbor-database-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/database
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-redis-pv"
  labels:
    name: harbor-redis-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/redis
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-trivy-pv"
  labels:
    name: harbor-trivy-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/trivy
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "harbor-scandata-pv"
  labels:
    name: harbor-scandata-pv
    release: stable
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/harbor/scandata
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-registry-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-registry-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-chartmuseum-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-chartmuseum-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-jobservice-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-jobservice-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-database-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-database-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-redis-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-redis-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-trivy-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-trivy-pv
      release: stable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-scandata-pvc
  namespace: harbor
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      name: harbor-scandata-pv
      release: stable

 
kubectl apply -f harbor-pv.yaml
kubectl get pv,pvc -n harbor

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                        STORAGECLASS         REASON   AGE
persistentvolume/harbor-chartmuseum-pv                      5Gi        RWO            Retain           Bound    harbor/harbor-chartmuseum-pvc                                              31h
persistentvolume/harbor-database-pv                         5Gi        RWO            Retain           Bound    harbor/harbor-database-pvc                                                 31h
persistentvolume/harbor-jobservice-pv                       5Gi        RWO            Retain           Bound    harbor/harbor-jobservice-pvc                                               31h
persistentvolume/harbor-redis-pv                            5Gi        RWO            Retain           Bound    harbor/harbor-redis-pvc                                                    31h
persistentvolume/harbor-registry-pv                         5Gi        RWO            Retain           Bound    harbor/harbor-registry-pvc                                                 31h
persistentvolume/harbor-scandata-pv                         5Gi        RWO            Retain           Bound    harbor/harbor-scandata-pvc                                                 30h
persistentvolume/harbor-trivy-pv                            5Gi        RWO            Retain           Bound    harbor/harbor-trivy-pvc                                                    31h
 
NAME                                           STATUS   VOLUME                  CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/harbor-chartmuseum-pvc   Bound    harbor-chartmuseum-pv   5Gi        RWO                           31h
persistentvolumeclaim/harbor-database-pvc      Bound    harbor-database-pv      5Gi        RWO                           31h
persistentvolumeclaim/harbor-jobservice-pvc    Bound    harbor-jobservice-pv    5Gi        RWO                           31h
persistentvolumeclaim/harbor-redis-pvc         Bound    harbor-redis-pv         5Gi        RWO                           31h
persistentvolumeclaim/harbor-registry-pvc      Bound    harbor-registry-pv      5Gi        RWO                           31h
persistentvolumeclaim/harbor-scandata-pvc      Bound    harbor-scandata-pv      5Gi        RWO                           30h
persistentvolumeclaim/harbor-trivy-pvc         Bound    harbor-trivy-pv         5Gi        RWO                           31h

二.安装harbor
1.添加harbor的helm源

helm repo add harbor https://helm.goharbor.io
helm search repo
helm pull harbor/harbor
tar zxf harbor-1.11.0.tgz
cd harbor/
ls

cert  Chart.yaml  conf  LICENSE  README.md  templates  values.yaml

2.前方高能，修改values.yaml
这里用默认类型 ingress，https协议，如果想用http协议，那么需要删除掉“expose.ingress.annotations”里面的“ssl-redirect”相关注解

vim values.yaml

expose:
  # Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer"
  # and fill the information in the corresponding section
  type: ingress
  tls:
    # Enable TLS or not.
    # Delete the "ssl-redirect" annotations in "expose.ingress.annotations" when TLS is disabled and "expose.type" is "ingress"
    # Note: if the "expose.type" is "ingress" and TLS is disabled,
    # the port must be included in the command when pulling/pushing images.
    # Refer to https://github.com/goharbor/harbor/issues/5291 for details.
    enabled: true  
    # The source of the tls certificate. Set as "auto", "secret"
    # or "none" and fill the information in the corresponding section
    # 1) auto: generate the tls certificate automatically
    # 2) secret: read the tls certificate from the specified secret.
    # The tls certificate can be generated manually or by cert manager
    # 3) none: configure no tls certificate for the ingress. If the default
    # tls certificate is configured in the ingress controller, choose this option
    certSource: auto

修改持久化配置，将pvc name添加在existingClaim后面

persistence:
  enabled: true
  # Setting it to "keep" to avoid removing PVCs during a helm delete
  # operation. Leaving it empty will delete PVCs after the chart deleted
  # (this does not apply for PVCs that are created for internal database
  # and redis components, i.e. they are never deleted automatically)
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound,
      # and specify the "subPath" if the PVC is shared with other components
      existingClaim: "harbor-registry-pvc"
      # Specify the "storageClass" used to provision the volume. Or the default
 
      # StorageClass will be used (the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    chartmuseum:
      existingClaim: "harbor-chartmuseum-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    jobservice:
      jobLog:
        existingClaim: "harbor-jobservice-pvc"
        storageClass: ""
        subPath: ""
        accessMode: ReadWriteOnce
        size: 5Gi
        annotations: {}
      scanDataExports:
        existingClaim: "harbor-scandata-pvc"
        storageClass: ""
        subPath: ""
        accessMode: ReadWriteOnce
        size: 5Gi
        annotations: {}
    # If external database is used, the following settings for database will
    # be ignored
    database:
      existingClaim: "harbor-database-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    # If external Redis is used, the following settings for Redis will
    # be ignored
    redis:
      existingClaim: "harbor-redis-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    trivy:
      existingClaim: "harbor-trivy-pvc"
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}

3.helm安装harbor

helm install harbor . -f values.yaml -n harbor

删除命令 

helm uninstall harbor -n harbor

kubectl get pod -n harbor -o wide

NAME                                    READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
harbor-chartmuseum-7b74f8b585-qcp89     1/1     Running   2          4h39m   10.244.107.231   k8s-node3   <none>           <none>
harbor-core-7fc48678c8-qcdqk            1/1     Running   2          4h39m   10.244.107.233   k8s-node3   <none>           <none>
harbor-database-0                       1/1     Running   2          4h39m   10.244.107.236   k8s-node3   <none>           <none>
harbor-jobservice-8486bb4bcb-2gjtp      1/1     Running   9          4h39m   10.244.107.237   k8s-node3   <none>           <none>
harbor-notary-server-7f7bf8f6d-zphpg    1/1     Running   2          4h39m   10.244.107.244   k8s-node3   <none>           <none>
harbor-notary-signer-5f9df848b7-skpxj   1/1     Running   2          4h39m   10.244.107.248   k8s-node3   <none>           <none>
harbor-portal-748c6db9c6-qw97j          1/1     Running   2          4h39m   10.244.107.243   k8s-node3   <none>           <none>
harbor-redis-0                          1/1     Running   2          4h39m   10.244.107.228   k8s-node3   <none>           <none>
harbor-registry-6777c99d8d-cb9rj        2/2     Running   4          4h39m   10.244.107.235   k8s-node3   <none>           <none>
harbor-trivy-0                          1/1     Running   2          4h39m   10.244.107.245   k8s-node3   <none>           <none>

4.ingress绑定ingress class

kubectl get ingress -n harbor

NAME                    CLASS    HOSTS                  ADDRESS                     PORTS     AGE
harbor-ingress          <none>   core.harbor.domain     192.168.0.21,192.168.0.22   80, 443   4h55m

kubectl edit ingress harbor-ingress -n harbor

......
spec:
  ingressClassName: nginx            ## k8s-1.20.X添加ingressClassName
  rules:
  - host: core.harbor.domain
    http:
      paths:
......
[root@k8s-master harbor]# kubectl get ingress -n harbor   ##查看添加成功
NAME                    CLASS    HOSTS                  ADDRESS                     PORTS     AGE
harbor-ingress          nginx    core.harbor.domain     192.168.0.21,192.168.0.22   80, 443   4h57m

5.服务器配置证书，不然docker login和docker pull 都会报错
在harbor-ingress 的secret里面找到data下的ca.crt

kubectl get secret harbor-ingress -n harbor -o yaml

apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRY3d4c0NtVmNPV2RpZWJTSVREcU9kakFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05Nakl4TWpJNE1EWXdOVE13V2hjTk1qTXhNakk0TURZdwpOVE13V2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRQ3g0YVBFbkJ5K2xicElsSytyZmlsdndIWkJRMzM1a3lxMThKWHhpcDl1SFQ0VkExUzEKbG45anVtQUR4Smk4QnhWZStmVlBDWW5UeGJoNWJua3pqdm9XekptOTlMRUY5UDJPdW5VZnZCWGt6VFZEK0o1TgpBZjhyMmtqMFRTck5yZVVKL3JwR01mT1V1MFRSSjg2N2g1WHg1aEpBeDNlWlk4LzRFWmZQOVFGRlFsVGRxRDFBCmlkUkRydjFCSGJIajNxVjU3ckI4L3VxMVJ1a2hjQUZodlZXKzMvdmNkelBaS0hjc0c4d3VhNml2ekExK096a1gKTFA2K2wvcnlPektWWG9kU3ZHV1RmVzJpSlFEUjJqSWliR1loSUF3SDIrZlU4MVJrdGNsS050b2hZYW5aYjdOdQpTcTlzOFB1b3Y2V1JDMWpwRUd6cjhtR2hKTEhtcHJMNlRXeDdBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVVZc2xqdmpDZ3F5R2NRZ1ZWS0s5TnIreU8yWll3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFKd1NkMkFuMmpEdHp6NUIvOEl2UzliamxVN3hLVlR0M1pWWkc0N0R5aFJXTmRpVEs1NFF3dFdhCmtCYUZrV2xTQWJaUk5iOWwyak85NEFMelRCb0ttVU04QTBlZFBQbXdvTFFaZlV2NVp6SCtGM05PenNVT1I5dFIKbnhva3lkSU14WUlpQWp1YThudXRLK0g2TklsRDRiVVA1MVZSNFJsNjBhdHhSWi81OW1HMWw1ZUNVSWtObVJyaQpwY2FQTjBBa3NzR2dCNmpxK3VXMXhPU0l1M21idVl3amdzTEpqWTFkNkNuS1IvQ1RRZnRBKzB2NFNVRTJBTFRGClg0TEFRVXFsM0F2NU8wWkxFQXAzNE5aR0xGNCtnRmJuYi84MmdYTlBuY2IzQkxOR0tKVE5zL0RGTTBkYlgwajcKY3VIMHQvT01oV2YyZ084eTNBMDB6dkIzcTVZSkIwdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURVekNDQWp1Z0F3SUJBZ0lSQU9QeW44V1FlOVZOaUlYL21LSGhrSHN3RFFZSktvWklodmNOQVFFTEJRQXcKRkRFU01CQUdBMVVFQXhNSmFHRnlZbTl5TFdOaE1CNFhEVEl5TVRJeU9EQTJNRFV6TUZvWERUSXpNVEl5T0RBMgpNRFV6TUZvd0hURWJNQmtHQTFVRUF4TVNZMjl5WlM1b1lYSmliM0l1Wkc5dFlXbHVNSUlCSWpBTkJna3Foa2lHCjl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF2dno0N0l4d2JUZDBQOFcvWngxUTVYaS9IRWZLajJmRzFlK08KQ0RMb2psSWEvbmErUlpaRzZvSVNIbGt6TVo2cTh3WmhZeTRVRTFCQi9VM1hKK2xuNHNhTkgrVllTQjdDbmFuYQpiWW5DVkp2Y091d1J3LzVaNUJzU1UreE55U0tLd3dGN0hLcDZBdklIc1puaU85aHhMNUVKRjRmbmtHY3BQZlZKCjNQTzdMY3B4ZDBVcEJ0enJyYnBWWTV1eDlmY3ZKSWNFeEJTY3ZmaGxmN3VjR0pYSDBZa0xra3RJUVAveWdRekMKQWMvUG82NDJ6VUJqNE1RWGVaRVQ2WjRxaWtLWnc4VXRGTyttZmZBU3BPT3FmMkpncDJiaU4yR1VqR044MTkxTAo1NU1BUUY3aFBzUWo0Y0k2aEVWczk3VmtXci9ZKzdxN3EyNkZnQ1IxY0p6ZVBQY1puUUlEQVFBQm80R1dNSUdUCk1BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXcKREFZRFZSMFRBUUgvQkFJd0FEQWZCZ05WSFNNRUdEQVdnQlJpeVdPK01LQ3JJWnhDQlZVb3IwMnY3STdabGpBegpCZ05WSFJFRUxEQXFnaEpqYjNKbExtaGhjbUp2Y2k1a2IyMWhhVzZDRkc1dmRHRnllUzVvWVhKaWIzSXVaRzl0CllXbHVNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUJpK3pFdFllS3A3VnRnamo1Vk9PMXUzRERLSmJFaXZYRFYKbG9ZeVRjQzdWRGg3YlF5UmhPVEtja2pJeGd6cFpubjUyczYvSHYxbkQ3MTk0UXFieE0yOGI1OEpxUEtvS0E4aQpWNHEzSE9xUGhlMlRoS1M4ZUt4anl4R0ZRYi8rQ2prdXNlTDV0bjVFbFM1SjBIRU93K2crZjk0WUdMMTFhdkNuCklnazdDbjQwTWovUUg1YTdhaC9lenNad3Y1aFJISHQrc2pyeE5TYWFTcXNjbHQ4bEV2M3VQWFVsdHBrRm1VY2IKc3hOM1lwRDg4d05LWElQSnU5bWpGMnFJUUdUYXBtME04a3lvTGZpMHlrdDlWM1I1cXoybVcwQWx5d3dob0NvMQpTTm9KbUJIOE42bnhqU1JPUzVJS01XU0RhYWtLc1ZVWGhiUjJRQVlyYVkvZ3YrWnhmZEdlCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdnZ6NDdJeHdiVGQwUDhXL1p4MVE1WGkvSEVmS2oyZkcxZStPQ0RMb2psSWEvbmErClJaWkc2b0lTSGxrek1aNnE4d1poWXk0VUUxQkIvVTNYSitsbjRzYU5IK1ZZU0I3Q25hbmFiWW5DVkp2Y091d1IKdy81WjVCc1NVK3hOeVNLS3d3RjdIS3A2QXZJSHNabmlPOWh4TDVFSkY0Zm5rR2NwUGZWSjNQTzdMY3B4ZDBVcApCdHpycmJwVlk1dXg5ZmN2SkljRXhCU2N2ZmhsZjd1Y0dKWEgwWWtMa2t0SVFQL3lnUXpDQWMvUG82NDJ6VUJqCjRNUVhlWkVUNlo0cWlrS1p3OFV0Rk8rbWZmQVNwT09xZjJKZ3AyYmlOMkdVakdOODE5MUw1NU1BUUY3aFBzUWoKNGNJNmhFVnM5N1ZrV3IvWSs3cTdxMjZGZ0NSMWNKemVQUGNablFJREFRQUJBb0lCQUEyOEVZYXBYdGhyMlg5UQprRUF4R29VMkZzQkk1V2RFdGtiaVVGdUVsYmJPYnNTSDg0ejdqcVFFc0pyZ0hVOWZNcm9CWm5XRWRjc2h4VzBhCjhQTWMxT3k1REtNVGtqSzFpNWRkQktsRjgrU3p4TFcwRGw4QzVxSUIxR0tXcHMxbEg0ZUFqc0x6KzR4RXJZcjMKSHc3Z3ROc1AyaENJSkgzOC9UbEliNFJsUHJ1L0tIV2p1SzJQWWQweFdzdDBtWVZBMytpYW9jWWJXTk1BV3p5cAo0WmR0c1Via2MrU3JPVUIvM2hrNE9iNUtkbUM3RE5aMTZJTnlSNGdpdXliSFI0Wk9JaHh1NGsyenFhVDBoU2J5CjREaUEyNytsczNWcGxETHhmanlKUVQyMEZHVW5jNTllb1pzdzBUeHZSUjM5cEFJQW9IdUhlL0ZCQW1RRnhZSEEKMS9qNUp5RUNnWUVBMnFnVDd4Sms4WFNLcStwenhDSTV4TlpTRVp0OU5WeVFZUUFzM25ubkxhemp0UUUwSGJGeQpWbk9ja1h6ZFd4Wnd5WUtUTktSb0ViSGE3V0NUVmd0T3RrR3VJWCtnenprQTRQaEI1YmhFNmp2bUhzS0laQkkvCmN5Z1lvMENZY0dZTGl5YnpRV0VLTzRPRHg1TlVxZmY2OTVRVVRGek1ORE10eSsvRnozc3lYNVVDZ1lFQTM1c3oKUmZHa0Qzd1lDTSszbWR3VldOZnRjbkNBYmw1YXppM3BWRG4yR2JRNC9GYVZkYVNmT3Z0Vk1oRUZpZkZGS2RzRApSNXJ3SmdXNmJSRUg0V1JOSEQxUzZ2WHc5ZnJJRkZFMnBRa2JZb01iV0tsUlFMNTlSYlRMQnEzQVJnV3Q4MXBYCmJtOXNNeFNBS2NBaVF6dGFxYXFTVDhOQjBHY1U0MjZSK3ZEYTcra0NnWUVBbFVRMlQzZE1KN1Q2VVgvOHhZRGoKMU9iR0liVDIxYTI1OWk0TGMxamVvMFNxTWM5L2gyR3lmeWZ2VXpaZFdpaEltSmVsN0VMcnRHQ281bkdPUXlmMgo5TjZEZytTL241YjNiWnlzUjZqeWlzQ1hTSnBlUjRwWmZFQjhDVDQ0a2twblNQZ3ZDWXU0VTRabE9LSHdJeFBoClpJL1hCNFkxOU5DWFMrZ0VMcTZZWmIwQ2dZRUExVVpsNEVlN2tOMXUyekNzVVQ5K3lPK2pWaTBDQXNOU1h5ZWgKVHFtK0Z3UXorbExuV1g4OE5QaTJhUHVkU2RYcmZ5R2JmamZFNks4OEFuMWxBOUUwVDBRYWkyc3JlcUxKSmIvVQpuQk1Vb0tDbWU2bDdpNEpsWUJBeU9kdU44ZnZHejc4U2Q0NGxLSTljTXZaRWQ5WHNBcnBqdFZwcXNza3ZQa0lmClVMZGNTOGtDZ1lCejZ2QlJqc2VlZnpySXVSRjR5UWorcUhZM2cxYXR2VDBMYlh5MWVvTk5RdW92ZXhoZmduTHoKangyVVB0Z29VMjlrMUpOWWhzMjByWVBhS1NDa2c5ZjNqUk1NNmZOc2M4QWQ5RFNJa3FaeS9rQkI0YjJPMDdKdwo2cVBRZEhyemJDK0Q2VUZhakFJMEFoVVV1WUZXbFRPaW5uLzlEc0p4Ky9CbEZSQzJFZTJtNEE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

转码

echo 'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRY3d4c0NtVmNPV2RpZWJTSVREcU9kakFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05Nakl4TWpJNE1EWXdOVE13V2hjTk1qTXhNakk0TURZdwpOVE13V2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRQ3g0YVBFbkJ5K2xicElsSytyZmlsdndIWkJRMzM1a3lxMThKWHhpcDl1SFQ0VkExUzEKbG45anVtQUR4Smk4QnhWZStmVlBDWW5UeGJoNWJua3pqdm9XekptOTlMRUY5UDJPdW5VZnZCWGt6VFZEK0o1TgpBZjhyMmtqMFRTck5yZVVKL3JwR01mT1V1MFRSSjg2N2g1WHg1aEpBeDNlWlk4LzRFWmZQOVFGRlFsVGRxRDFBCmlkUkRydjFCSGJIajNxVjU3ckI4L3VxMVJ1a2hjQUZodlZXKzMvdmNkelBaS0hjc0c4d3VhNml2ekExK096a1gKTFA2K2wvcnlPektWWG9kU3ZHV1RmVzJpSlFEUjJqSWliR1loSUF3SDIrZlU4MVJrdGNsS050b2hZYW5aYjdOdQpTcTlzOFB1b3Y2V1JDMWpwRUd6cjhtR2hKTEhtcHJMNlRXeDdBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVVZc2xqdmpDZ3F5R2NRZ1ZWS0s5TnIreU8yWll3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFKd1NkMkFuMmpEdHp6NUIvOEl2UzliamxVN3hLVlR0M1pWWkc0N0R5aFJXTmRpVEs1NFF3dFdhCmtCYUZrV2xTQWJaUk5iOWwyak85NEFMelRCb0ttVU04QTBlZFBQbXdvTFFaZlV2NVp6SCtGM05PenNVT1I5dFIKbnhva3lkSU14WUlpQWp1YThudXRLK0g2TklsRDRiVVA1MVZSNFJsNjBhdHhSWi81OW1HMWw1ZUNVSWtObVJyaQpwY2FQTjBBa3NzR2dCNmpxK3VXMXhPU0l1M21idVl3amdzTEpqWTFkNkNuS1IvQ1RRZnRBKzB2NFNVRTJBTFRGClg0TEFRVXFsM0F2NU8wWkxFQXAzNE5aR0xGNCtnRmJuYi84MmdYTlBuY2IzQkxOR0tKVE5zL0RGTTBkYlgwajcKY3VIMHQvT01oV2YyZ084eTNBMDB6dkIzcTVZSkIwdz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=' | base64 -d > ca.crt

-----BEGIN CERTIFICATE-----
MIIDEzCCAfugAwIBAgIQcwxsCmVcOWdiebSITDqOdjANBgkqhkiG9w0BAQsFADAU
MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMjIxMjI4MDYwNTMwWhcNMjMxMjI4MDYw
NTMwWjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCx4aPEnBy+lbpIlK+rfilvwHZBQ335kyq18JXxip9uHT4VA1S1
ln9jumADxJi8BxVe+fVPCYnTxbh5bnkzjvoWzJm99LEF9P2OunUfvBXkzTVD+J5N
Af8r2kj0TSrNreUJ/rpGMfOUu0TRJ867h5Xx5hJAx3eZY8/4EZfP9QFFQlTdqD1A
idRDrv1BHbHj3qV57rB8/uq1RukhcAFhvVW+3/vcdzPZKHcsG8wua6ivzA1+OzkX
LP6+l/ryOzKVXodSvGWTfW2iJQDR2jIibGYhIAwH2+fU81RktclKNtohYanZb7Nu
Sq9s8Puov6WRC1jpEGzr8mGhJLHmprL6TWx7AgMBAAGjYTBfMA4GA1UdDwEB/wQE
AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw
AwEB/zAdBgNVHQ4EFgQUYsljvjCgqyGcQgVVKK9Nr+yO2ZYwDQYJKoZIhvcNAQEL
BQADggEBAJwSd2An2jDtzz5B/8IvS9bjlU7xKVTt3ZVZG47DyhRWNdiTK54QwtWa
kBaFkWlSAbZRNb9l2jO94ALzTBoKmUM8A0edPPmwoLQZfUv5ZzH+F3NOzsUOR9tR
nxokydIMxYIiAjua8nutK+H6NIlD4bUP51VR4Rl60atxRZ/59mG1l5eCUIkNmRri
pcaPN0AkssGgB6jq+uW1xOSIu3mbuYwjgsLJjY1d6CnKR/CTQftA+0v4SUE2ALTF
X4LAQUql3Av5O0ZLEAp34NZGLF4+gFbnb/82gXNPncb3BLNGKJTNs/DFM0dbX0j7
cuH0t/OMhWf2gO8y3A00zvB3q5YJB0w=
-----END CERTIFICATE-----

将证书保存在docker目录下，创建根域名相同的子目录，这一步建议所有机器都执行

mkdir -p /etc/docker/certs.d/core.harbor.domain
cp ca.crt /etc/docker/certs.d/core.harbor.domain/

6.配置host
建议所有机器执行
将harbor域名解析到ingress-nginx的svc的clusterIP，实现内网使用harbor的功能

kubectl get svc -n ingress-nginx
NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller-admission-nginx   ClusterIP   10.106.125.71    <none>        443/TCP                      48d
ingress-nginx-controller-nginx             NodePort    10.108.233.252   <none>        80:31203/TCP,443:32292/TCP   48d
echo '10.108.233.252 core.harbor.domain' >> /etc/hosts

三.测试
1.浏览器访问harbor
修改本机host

47.92.*.*  core.harbor.domain

查看ingress-nginx暴露的https协议端口，这里我们能看到https协议的端口是32292

kubectl get svc -n ingress-nginx
NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller-admission-nginx   ClusterIP   10.106.125.71    <none>        443/TCP                      48d
ingress-nginx-controller-nginx             NodePort    10.108.233.252   <none>        80:31203/TCP,443:32292/TCP   48d

浏览器访问

2.服务器内部访问harbor
docker login https://core.harbor.domain
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded