k8s学习笔记——关于harbor与docker_harbor使用ingress 32429端口未开放

作者：从前慢现在也慢 | 2024-08-14 01:44:46

踩

harbor使用ingress 32429端口未开放

最近半个多月我被vim编译器坑死了，自己挖了个大坑，耗时20多天才爬出来。不管怎么样，爬出来了，总该庆贺一下。

我前不久安装了harbor要建立私有仓库。要创建自己的私有仓库首先要配置harbor 同docker通信，要让docker 可以通过login登录harbor，并且可以通过push上传镜像到harbor里面。

我的harbor是通过helm安装，ingress控制器使用的traefik，并将traefik的服务通过noteport暴露出来，使之所有https服务都有了统一的访问入口，端口号为32042。

参考官网Harbor docs | Configure HTTPS Access to Harbor

我做了使用自己创建的密钥访问https，密钥制作方式和官网一致，我重点说一下helm安装的harbor配置。

首先，创建自定义的secret，参考Harbor docs | Customize the Harbor Token Service 这里使用的域名和Common Name都使用 *.harbor.shell.com

将用于的docker pull 和 push images 的token密钥也做进去。


kubectl create secret generic harbor-ingress-custom \
--from-file=ca.crt \
--from-file=tls.crt \
--from-file=tls.key \
--from-file=private_key.pem \
--from-file=root.crt \
--type=kubernetes.io/tls -n harbor

修改values.yaml文件


1、certSource: secret  //这里修改为从自定义secret取密钥
 
2、 secret:
      # The name of secret which contains keys named:
      # "tls.crt" - the certificate
      # "tls.key" - the private key
      secretName: "harbor-ingress-custom"  //修改为刚才创建的secret名
      # The name of secret which contains keys named:
      # "tls.crt" - the certificate
      # "tls.key" - the private key
      # Only needed when the "expose.type" is "ingress".
      notarySecretName: "harbor-ingress-custom"  //这里同上
  ingress:
    hosts:
 
3、  ingress:
       annotations:
 
         //这里加一条
         kubernetes.io/ingress.class: "traefik"
 
4、   externalURL: https://core.harbor.shell.com:32042
    
5、然后就是设置存储storageClass ，安装的时候说过，这里不做赘述
 
6、然后在这里找到
 
# The initial password of Harbor admin. Change it from portal after launching Harbor
harborAdminPassword: "Harbor12345"
 
# The name of the secret which contains key named "ca.crt". Setting this enables the
# download link on portal to download the certificate of CA when the certificate isn't
# generated automatically
caSecretName: "harbor-ingress-custom"  // 设置这里
7、然后在这里找到
core:
  image:
    repository: goharbor/harbor-core
    tag: v2.3.1
  # set the service 
...
secretName: "harbor-ingress-custom"  //设置这里
  # The XSRF key. Will be generated automatically if it isn't specified
xsrfKey: "root.crt"   //设置这里

如果现在着急使用sudo docker login -u admin core.harbor.shell.com:32042登录话，会报类似错误

Error getting v2 registry: Get https://harbor.shell.com:32042/v2/: dial tcp 10.12.70.132:32042:
connect: no route to host"

需要配置一下docker

首先在要使用docker的客户机上配置一下/etc/hosts 将core.harbor.shell.com与ip地址绑定

然后创建mkdir -p /etc/docker/certs.d/core.harbor.shell.com:32042

将密钥拷贝进去。

如果这里拷贝的密钥和harbor加载的密钥不一致，或者你这里用自建的密钥，harbor中用helm创建的密钥，也就是certSource使用auto，会报一下类似错误

Error response from daemon: Get https://harbor.shell.com:32042/v2/: x509: certificate signed by unknown authority

最后配置/etc/docker/daemon.json

添加 "registry-mirrors": ["https://core.harbor.shell.com:32042"],
"insecure-registries": ["10.12.70.130:32042"]

其实这里如果不用https使用http也可以通过docker login登陆的，不过再往下走，使用docker push上传image会报错如下：

error parsing HTTP 405 response body: invalid character '<' looking for beginning of value: "<html>\r
\n<head><title>405 Not Allowed</title></head>\r\n<body>\r\n<center><h1>405 Not Allowed</h1></center>\r
\n<hr><center>nginx/1.19.3</center>\r\n</body>\r\n</html>

这说明push使用post方法传输是要通过https通信的。

然后 sudo systemctl daemon-reload

sudo systemctl restart docker

配置到如此，按理来说，应该一切正常了。不过我却遇到了莫名其妙的错误：

Error response from daemon: Get https://core.harbor.shell.com:32042/v2/: Get /service/token?
account=admin&client_id=docker&offline_token=true&service=harbor-registry: unsupported protocol scheme ""

但通过curl测试也可以正常通过


//sudo cp tls.crt /etc/ssl/certs/ca-certificates.crt
 
 curl -u "admin:Harbor12345" -k -i -v https://core.harbor.shell.com:32042
 
   Trying 10.12.70.130:32042...
* TCP_NODELAY set
* Connected to core.harbor.shell.com (10.12.70.130) port 32042 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=CN; ST=TianJin; L=TianJin; O=dzj; OU=dzj; CN=*.harbor.shell.com
*  start date: Sep  6 09:21:52 2021 GMT
*  expire date: Sep  4 09:21:52 2031 GMT
*  issuer: C=CN; ST=TianJin; L=TianJin; O=dzj; OU=dzj; CN=*.harbor.shell.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Basic with user 'admin'
* Using Stream ID: 1 (easy handle 0x565310e86c80)
> GET / HTTP/2
> Host: core.harbor.shell.com:32042
> authorization: Basic YWRtaW46SGFyYm9yMTIzNDU=
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
HTTP/2 200
< cache-control: no-store, no-cache, must-revalidate
cache-control: no-store, no-cache, must-revalidate
< content-type: text/html
content-type: text/html
< date: Tue, 07 Sep 2021 03:38:32 GMT
date: Tue, 07 Sep 2021 03:38:32 GMT
< etag: W/"60f55533-48f"
etag: W/"60f55533-48f"
< last-modified: Mon, 19 Jul 2021 10:34:27 GMT
last-modified: Mon, 19 Jul 2021 10:34:27 GMT
< server: nginx/1.19.3
server: nginx/1.19.3
 
<
<!doctype html>
<html>
 
<head>
    <meta charset="utf-8">
    <title>Harbor</title>
    <base href="/">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
    <link rel="preload" as="style" href="./light-theme.css?buildTimestamp=1626690627398">
    <link rel="preload" as="style" href="./dark-theme.css?buildTimestamp=1626690627398">
<link rel="stylesheet" href="styles.0588221ae7a21e1ad681.css"></head>
<body>
    <harbor-app>
        <div class="spinner spinner-lg app-loading app-loading-fixed">
            Loading...
        </div>
    </harbor-app>
<script src="runtime-es2015.3dddbb4251aaabd9f502.js" type="module"></script><script src="runtime-es5.3dddbb4251aaabd9f502.js" nomodule defer></script><script src="polyfills-es5.174f74044ee744816e0d.js" nomodule defer></script><script src="polyfills-es2015.746a3b159a979d256b42.js" type="module"></script><script src="scripts.cd39b4a7ab651b82e180.js" defer></script><script src="main-es2015.96bbe18d2039f20d8d41.js" type="module"></script><script src="main-es5.96bbe18d2039f20d8d41.js" nomodule defer></script></body>
 
* Connection #0 to host core.harbor.shell.com left intact

网上找了很久都没有答案，可能没人遇到过类似错误，于是我就在坑里被困了很久。

后来，翻阅资料，发现notary组件是用来提供各组件注册信任通信的，于是我到helm项目的安装目录harbor/conf/notary-server.json这个文件看到了

"{{ .Values.externalURL }}/service/token"

然后看了看harbor/conf/notary-signer.json文件
"level": "{{ .Values.logLevel }}"

回过头来又去查看配置文件values.yaml

发现logLevel这个配置项是顶格书写的，而externalURL配置项却空了两格，按照上面两个文件信息，可以知道这两个配置项应该是同级的，要顶格同时顶格，要空格同时空格，所以我猜测应该是externalURL的配置项前面空两格导致的错误，于是我将其顶格更新当前harbor

helm upgrade harbor harbor -n harbor

发现成功了，可以正常登录，并且正常push镜像了。

后来又尝试了一下，发现是因为vim在修改externalURL配置项时，当输入：时vim自动将此配置项向后空两格，当时没有发现，所以直至导致错误。现在终于明白了，所以以后书写yaml文件时一定要谨慎。要是遇到莫名其妙的错误，应多来检查一下yaml文件的书写规范。

后记——注意事项：

因为使用helm安装harbor，在helm配置文件values.yaml中使用自己定义的证书实现https访问，secret默认是tls.crt和tls.key，在导入docker支持的时候参考Harbor docs | Configure HTTPS Access to Harbor文章内容

中的自定义域名的


cp yourdomain.com.cert /etc/docker/certs.d/yourdomain.com/
cp yourdomain.com.key /etc/docker/certs.d/yourdomain.com/
cp ca.crt /etc/docker/certs.d/yourdomain.com/

需全部改成tls.cert、tls.key和ca. crt拷贝到


/etc/docker/certs.d/
    └── yourdomain.com:port
       ├── yourdomain.com.cert  <-- Server certificate signed by CA
       ├── yourdomain.com.key   <-- Server key signed by CA
       └── ca.crt               <-- Certificate authority that signed the registry certificate

在重新启动docker，这样再使用docker login登录，自定义的证书才可以生效。

声明：本文内容由网友自发贡献，转载请注明出处：【wpsshop】