热门标签
热门文章
- 1提示由于找不到MSVCR120.dll文件，怎么修复呢？MSVCR120.dll是什么文件？
- 2AI流程图教程，小白也能轻松画！_ai根据文件制作线性流程图
- 3离线AI聊天清华大模型(ChatGLM3)本地搭建_glm3本地部署微调及使用
- 4ubuntu安装docker及插件docker-compose（详细图文）_ubuntu安装docker-compose
- 5python列表 中添加元素append()，extend()， insert(),+list的区别_insert append
- 6【一文带你了解Git+Linux+Docker等常见命令】_linux docker git
- 7王炸，OpenAI推出免费GPT-4o模型～
- 8SCI论文为什么会被拒稿？我总结了5大原因_sci论文投稿被模板拒稿
- 9人工智能发展方向的思考：简单与复杂的对立与融合
- 10了解redis以及使用规则_redis 出入规则
当前位置: article > 正文
容器学习Day10-搭建私有镜像仓库_私有镜像仓库搭建
作者：码创造者 | 2024-07-08 01:25:35
赞
踩
私有镜像仓库搭建
前言
今天继续学习Docker镜像相关内容，了解一下如何搭建私有的镜像仓库。
在实际的生产、开发环境中，有些环境是不允许连接外网的，另外，还有代码安全、镜像拉取速度的因素，这种情况下就需要搭建本地的私有镜像仓库。
一般搭建本地私有镜像仓库有两种方式，Registry 和 Harbor，我们新装一台 Ubuntu 作为镜像仓库的服务器，具体步骤可参考容器学习Day02-VMware Workstation安装Ubuntu。当然，也可以偷个懒使用之前的虚拟机克隆一台新虚拟机。
我的实验环境：安装两台虚拟机，一台镜像仓库服务器（192.168.1.129），一台 docker 客户端（192.168.1.128），用于拉取和推送镜像测试，docker 版本20.10.20。
一、搭建Registry私有镜像仓库
Registry 是 Dcoker 官方的一个私有仓库镜像，必须要基于一个 docker 服务器，如果是新装的系统还是要先安装 docker，可参考容器学习Day07-Docker基础（一）。
1、Registry配置（http）
1）拉取 registry 镜像。
- ###拉取registry的镜像。
- root@hub:/docker# docker pull registry
- Using default tag: latest
- latest: Pulling from library/registry
- 79e9f2f55bf5: Pull complete
- 0d96da54f60b: Pull complete
- 5b27040df4a2: Pull complete
- e2ead8259a04: Pull complete
- 3790aef225b9: Pull complete
- Digest: sha256:169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375
- Status: Downloaded newer image for registry:latest
- docker.io/library/registry:latest
2）运行 registry 仓库容器。
registry 服务默认会将上传的镜像保存在容器的 /var/lib/registry 目录，我们可以在主机上创建文件系统并挂载到容器，实现持久化存储。创建文件系统可参考容器学习Day04。
- ###在创建好的文件系统中，创建目录localregistry。
- root@hub:/# df -h
- Filesystem Size Used Avail Use% Mounted on
- tmpfs 195M 940K 194M 1% /run
- /dev/mapper/ubuntu--vg-ubuntu--lv 9.8G 5.5G 3.9G 59% /
- tmpfs 971M 0 971M 0% /dev/shm
- tmpfs 5.0M 0 5.0M 0% /run/lock
- /dev/sda2 1.8G 246M 1.4G 15% /boot
- tmpfs 195M 4.0K 195M 1% /run/user/1000
- /dev/mapper/data--vg-data--lv 10G 104M 9.9G 2% /data
- root@hub:/# mkdir -p /data/localregistry
-
- ###运行容器，-p参数映射主机端口，-v参数挂载本地目录到容器，--name指定容器名字。
- root@hub:/# docker run -d -p 5000:5000 --name localregistry -v /data/localregistry/:/var/lib/registry registry
- eb0935bbb8a16a5da8e5111be99a0d167c99346ad8937840143037551d0a1899
-
- ###查看容器运行情况。
- root@hub:/# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- eb0935bbb8a1 registry "/entrypoint.sh /etc…" 51 seconds ago Up 50 seconds 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp localregistry
-
- ###查看5000端口监听情况。
- root@hub:/# netstat -an|grep 5000
- tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN
- tcp6 0 0 :::5000 :::* LISTEN
3）推送镜像到 registry。
docker 默认是使用 https 协议工作的，而 registry 服务器的响应是http协议，客户端需要添加相关配置才能与registry进行通信。
- ###查看docker客户端（192.168.1.128）上的本地镜像，将镜像从docker客户端上传到镜像仓库。
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- httpd latest dabbfbe0c57b 11 months ago 144MB
- httpd v1 dabbfbe0c57b 11 months ago 144MB
- ubuntu latest ba6acccedd29 13 months ago 72.8MB
-
- ###给要上传的镜像打标签，标签格式，镜像仓库IP:5000/镜像名称。
- root@docker:~# docker tag httpd:v1 192.168.1.129:5000/httpd:v1
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- 192.168.1.129:5000/httpd v1 dabbfbe0c57b 11 months ago 144MB
- httpd latest dabbfbe0c57b 11 months ago 144MB
- httpd v1 dabbfbe0c57b 11 months ago 144MB
- ubuntu latest ba6acccedd29 13 months ago 72.8MB
-
- ###尝试推送镜像，docker push。
- root@docker:~# docker push 192.168.1.129:5000/httpd:v1
- The push refers to repository [192.168.1.129:5000/httpd]
- Get "https://192.168.1.129:5000/v2/": http: server gave HTTP response to HTTPS client
-
- ###编辑docker客户端（192.168.1.128）的/etc/docker/daemon.json文件，如没有则创建。
- root@docker:~# vi /etc/docker/daemon.json
-
- ###编辑入以下内容。
- {
- "insecure-registries": ["192.168.1.129:5000"]
- }
-
- ###重启docker服务。
- root@docker:~# systemctl restart docker
-
- ###重新push镜像到镜像仓库（192.168.1.129）。
- root@docker:~# docker push 192.168.1.129:5000/httpd:v1
- The push refers to repository [192.168.1.129:5000/httpd]
- deefaa620a71: Pushed
- 9cff3206f9a6: Pushed
- 15e4bf5d0804: Pushed
- 1da636a1aa95: Pushed
- 2edcec3590a4: Pushed
- v1: digest: sha256:57c1e4ff150e2782a25c8cebb80b574f81f06b74944caf972f27e21b76074194 size: 1365
4）通过浏览器访问仓库地址，查看上传的镜像。
5）从私有镜像仓库拉取镜像测试。
- ###在docker客户端（192.168.1.128），查看本地镜像，清除之前的httpd镜像。
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- httpd latest dabbfbe0c57b 11 months ago 144MB
- httpd v1 dabbfbe0c57b 11 months ago 144MB
- 192.168.1.129:5000/httpd v1 dabbfbe0c57b 11 months ago 144MB
- ubuntu latest ba6acccedd29 13 months ago 72.8MB
- root@docker:~# docker rmi httpd:v1
- Untagged: httpd:v1
- root@docker:~# docker rmi httpd:latest
- Untagged: httpd:latest
- Untagged: httpd@sha256:0954cc1af252d824860b2c5dc0a10720af2b7a3d3435581ca788dff8480c7b32
- root@docker:~# docker rmi 192.168.1.129:5000/httpd:v1
- Untagged: 192.168.1.129:5000/httpd:v1
- Untagged: 192.168.1.129:5000/httpd@sha256:57c1e4ff150e2782a25c8cebb80b574f81f06b74944caf972f27e21b76074194
- Deleted: sha256:dabbfbe0c57b6e5cd4bc089818d3f664acfad496dc741c9a501e72d15e803b34
- Deleted: sha256:0e16a5a61bcb4e6b2bb2d746c2d6789d6c0b66198208b831f74b52198d744189
- Deleted: sha256:f79670638074ff7fd293e753c11ea2ca0a2d92ab516d2f6b0bac3f4c6fed5d86
- Deleted: sha256:189d55cdd18e4501032bb700a511c2d69c82fd75f1b619b5218ea6870e71e4aa
- Deleted: sha256:cb038ed3e490a8c0f195cf135ac0d27dd8d3872598b1cb858c2666f2dae95a61
- Deleted: sha256:2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- ubuntu latest ba6acccedd29 13 months ago 72.8MB
-
- ###拉取镜像。
- root@docker:~# docker pull 192.168.1.129:5000/httpd:v1
- v1: Pulling from httpd
- a2abf6c4d29d: Pull complete
- dcc4698797c8: Pull complete
- 41c22baa66ec: Pull complete
- 67283bbdd4a0: Pull complete
- d982c879c57e: Pull complete
- Digest: sha256:57c1e4ff150e2782a25c8cebb80b574f81f06b74944caf972f27e21b76074194
- Status: Downloaded newer image for 192.168.1.129:5000/httpd:v1
- 192.168.1.129:5000/httpd:v1
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- 192.168.1.129:5000/httpd v1 dabbfbe0c57b 11 months ago 144MB
- ubuntu latest ba6acccedd29 13 months ago 72.8MB
2、Registry配置（https）
配置https的话，需要有证书，学习测试使用自签名证书即可。
1）停止并销毁之前创建的registry容器。
- ###仓库服务上停止并销毁之前运行的registry容器。
- root@hub:/# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- eb0935bbb8a1 registry "/entrypoint.sh /etc…" 15 hours ago Up 15 hours 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp localregistry
- root@hub:/# docker stop localregistry
- localregistry
- root@hub:/# docker rm localregistry
- localregistry
2）生成自签名证书。
- ###仓库服务器上生成自签名证书,定义一个域名registry.dh.com，并设置为Common Name。
- root@hub:/# cd /data/
- root@hub:/data# ls
- localregistry
- root@hub:/data# mkdir certs
- root@hub:/data# openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/myhub.key -addext "subjectAltName = DNS:registry.dh.com" -x509 -days 3650 -out certs/myhub.crt
- .+.+......+..+...+...+..........+..+..........+...+..............+...+.......+.....+.......+......+...+.....+.+........+.+......+...+..+.............+..+......+...+.......+.........+.......................+.+..+...+......+...+......+.+.....+.+.........+...+..+...+......+....+......+.....+......+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.........+.+.....+...+.+.........+..+.........+.+.....+.........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+.....+..................+...+......+.+..+.......+.....+................+..+................+............+...+..+....+.........+..............+.+..+.......+...........+...................+...........................+......+..+...+...+..........+.....+......+...+.+.....+......................+...............+..+...+...................+.........+.........+..+...+.+.....+......+...+.....................+...................+..+....+......+..................+...+...+......+...+...+..+.............+.....+....+...+...+.....+....+.....................+.....+............+...+....+....................+...+......+.+......+.........+...+..+......+.......+..+....+.................+............+.+....................+.+...+.....+......+..........+..+....+...............+......+...+.....+....+..+..........+.....+......+.......+...+........+.......+.................+...+....+......+.........+...+..+.........+................+........+.+.....+....+.....+......+....+.....+.......+......+...............+.....+.......+......+..+.........+.+.................................+......+...+......+......+.....+....+.....+.........+...+...+............+...+..............................+....+...+...............+...+..............+.+.....+.+...+..+............................+...+...+..+....+...+..+......+.......+......+..+....+.....+.+.................+......+...............+.+...+..+....+.........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- .....+.....+.+.........+.....+.+..+.........+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+.......+..+.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+.........+.........+.....+......+.+.........+........+...+..........+..+............+..........+.....+....+..+...+.+.....+.+........+......+.........+.........+.......+........+.+.........+...+.........+........+.+............+...+.....+...............+......+.............+...+..+.......+........+...+.............+.........+............+.........+..+....+..............+....+........+.........+.+............+...........+.+...........+...+.+.......................+...+...+.........+...+.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- -----
- You are about to be asked to enter information that will be incorporated
- into your certificate request.
- What you are about to enter is what is called a Distinguished Name or a DN.
- There are quite a few fields but you can leave some blank
- For some fields there will be a default value,
- If you enter '.', the field will be left blank.
- -----
- Country Name (2 letter code) [AU]:CN
- State or Province Name (full name) [Some-State]:BeiJing
- Locality Name (eg, city) []:BeiJing
- Organization Name (eg, company) [Internet Widgits Pty Ltd]:DH
- Organizational Unit Name (eg, section) []:DH
- Common Name (e.g. server FQDN or YOUR name) []:registry.dh.com
- Email Address []:
- root@hub:/data# ls
- certs localregistry
-
- ###查看证书。
- root@hub:/data# ls certs/
- myhub.crt myhub.key
3）运行一个新的容器。
- ###运行容器。
- root@hub:/data# docker run -d -p 443:443 --restart=always --name localregistry -v /data/localregistry/:/var/lib/registry -v /data/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/myhub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/myhub.key registry
- 666c0fe96fde0cf14e422e5221414316413352fc4b3844023a77bcd7f2c05fb3
- root@hub:/data# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- 666c0fe96fde registry "/entrypoint.sh /etc…" 24 seconds ago Up 17 seconds 0.0.0.0:443->443/tcp, :::443->443/tcp, 5000/tcp localregistry
4）推送镜像测试。
docker 客户机增加仓库服务器的域名解析，去掉之前在 docker 客户机（192.168.1.128）/etc/docker/daemon.json 文件中添加的 "insecure-registries": ["192.168.1.129:5000"] ，再进行推送测试。
- ###docker客户机hosts文件增加仓库域名的解析。
- root@docker:~# echo 192.168.1.129 registry.dh.com >> /etc/hosts
-
- ###删除"insecure-registries": ["192.168.1.129:5000"]配置，并重启docker。
- root@docker:~# vi /etc/docker/daemon.json
-
- ###删除前内容。
- {
- "registry-mirrors": ["https://yj431vht.mirror.aliyuncs.com"],
- "insecure-registries": ["192.168.1.129:5000"]
- }
-
-
- ###删除后内容。
- {
- "registry-mirrors": ["https://yj431vht.mirror.aliyuncs.com"],
- }
-
-
- ###重启docker。
- root@docker:~# systemctl restart docker
-
- ###查看本地镜像。
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- 192.168.1.129:5000/httpd v1 dabbfbe0c57b 11 months ago 144MB
- ubuntu latest ba6acccedd29 13 months ago 72.8MB
-
- ###标记Ubuntu的镜像。
- root@docker:~# docker tag ubuntu:latest registry.dh.com/ubuntu:v1
-
- ###推送镜像。
- root@docker:~# docker push registry.dh.com/ubuntu:v1
- The push refers to repository [registry.dh.com/ubuntu]
- Get "https://registry.dh.com/v2/": x509: certificate relies on legacy Common Name field, use SANs instead
-
- ###有报错，因为自签名证书不被信任，添加证书的信任，将证书从仓库服务器拷贝到docker客户端上，如果有多个客户端，每个客户端都要处理。
- ##创建目录。
- root@docker:~# mkdir -p /etc/docker/certs.d/registry.dh.com
-
- ##拷贝证书，也可以使用其他命令，达到效果即可。注意：Ubuntu默认root不能直接ssh登录，如使用scp，需要修改ssh配置文件，修改/etc/ssh/sshd_config，增加PermitRootLogin yes，并重启sshd服务。
-
- #仓库服务器开启root登录ssh权限。
- root@hub:/data# echo -e "PermitRootLogin yes" >> /etc/ssh/sshd_config
- root@hub:/data# systemctl restart sshd
-
- #docker客户端执行scp拷贝证书。
- root@docker:~# scp 192.168.1.129:/data/certs/myhub.crt /etc/docker/certs.d/registry.dh.com/ca.crt
- root@192.168.1.129's password:
- myhub.crt 100% 2065 82.9KB/s 00:00
- ###重新推送镜像。
- root@docker:~# docker push registry.dh.com/ubuntu:v1
- The push refers to repository [registry.dh.com/ubuntu]
- 9f54eef41275: Pushed
- v1: digest: sha256:7cc0576c7c0ec2384de5cbf245f41567e922aab1b075f3e8ad565f508032df17 size: 529
-
3、Registry身份认证
1）仓库服务器上停止和销毁之前运行的registry容器。
- ###停止和销毁之前启动的registry容器。
- root@hub:/data# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- 666c0fe96fde registry "/entrypoint.sh /etc…" 26 minutes ago Up 26 minutes 0.0.0.0:443->443/tcp, :::443->443/tcp, 5000/tcp localregistry
- root@hub:/data# docker stop 666
- 666
- root@hub:/data# docker rm 666
- 666
- root@hub:/data# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2）安装认证需要的工具包。
- ###安装工具包，Ubuntu是apache2-utils，CentOS是httpd-tools包。
- root@hub:/data# apt install -y apache2-utils
3）创建存放认证信息目录。
- ###创建目录。
- root@hub:/data# mkdir registry-auth
- root@hub:/data# ls
- certs localregistry registry-auth
4）创建认证信息。
- ###创建认证信息。
- root@hub:/data# htpasswd -Bbn hubuser 123abc > registry-auth/htpasswd
- root@hub:/data# ls -l registry-auth/htpasswd
- -rw-r--r-- 1 root root 70 Nov 27 08:23 registry-auth/htpasswd
5）运行带认证的registry容器。
- ###运行带认证、支持https的registry容器。
- root@hub:/data# docker run -d -p 443:443 --restart=always --name localregistry -v /data/localregistry/:/var/lib/registry -v /data/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/myhub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/myhub.key -v /data/registry-auth:/auth -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd registry
- 2aef873145ad415bcda13324213f3e04a6291ddb0f2ba4a8c0664f95743e63ef
- root@hub:/data# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- 2aef873145ad registry "/entrypoint.sh /etc…" About a minute ago Up About a minute 0.0.0.0:443->443/tcp, :::443->443/tcp, 5000/tcp localregistry
6）docker客户机拉取镜像测试。
- ###在docker客户机上删除之前的httpd镜像。
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- 192.168.1.129:5000/httpd v1 dabbfbe0c57b 11 months ago 144MB
- registry.dh.com/ubuntu v1 ba6acccedd29 13 months ago 72.8MB
- root@docker:~# docker rmi 192.168.1.129:5000/httpd:v1
- Untagged: 192.168.1.129:5000/httpd:v1
- Untagged: 192.168.1.129:5000/httpd@sha256:57c1e4ff150e2782a25c8cebb80b574f81f06b74944caf972f27e21b76074194
- Deleted: sha256:dabbfbe0c57b6e5cd4bc089818d3f664acfad496dc741c9a501e72d15e803b34
- Deleted: sha256:0e16a5a61bcb4e6b2bb2d746c2d6789d6c0b66198208b831f74b52198d744189
- Deleted: sha256:f79670638074ff7fd293e753c11ea2ca0a2d92ab516d2f6b0bac3f4c6fed5d86
- Deleted: sha256:189d55cdd18e4501032bb700a511c2d69c82fd75f1b619b5218ea6870e71e4aa
- Deleted: sha256:cb038ed3e490a8c0f195cf135ac0d27dd8d3872598b1cb858c2666f2dae95a61
- Deleted: sha256:2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f
-
- ###拉取新镜像，会有报错，提示没有经过基础身份认证，要先认证，才能拉取。
- root@docker:~# docker pull registry.dh.com/httpd:v1
- Error response from daemon: Head "https://registry.dh.com/v2/httpd/manifests/v1": no basic auth credentials
-
- ###登陆信息认证。
- root@docker:~# docker login registry.dh.com
- Username: hubuser
- Password:
- WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
- Configure a credential helper to remove this warning. See
- https://docs.docker.com/engine/reference/commandline/login/#credentials-store
-
- Login Succeeded
-
- ###再次拉取镜像。
- root@docker:~# docker pull registry.dh.com/httpd:v1
- v1: Pulling from httpd
- a2abf6c4d29d: Pull complete
- dcc4698797c8: Pull complete
- 41c22baa66ec: Pull complete
- 67283bbdd4a0: Pull complete
- d982c879c57e: Pull complete
- Digest: sha256:57c1e4ff150e2782a25c8cebb80b574f81f06b74944caf972f27e21b76074194
- Status: Downloaded newer image for registry.dh.com/httpd:v1
- registry.dh.com/httpd:v1
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- registry.dh.com/httpd v1 dabbfbe0c57b 11 months ago 144MB
- registry.dh.com/ubuntu v1 ba6acccedd29 13 months ago 72.8MB
4、Registry离线安装
通常在生产中，内网服务器无法连接外网，可以先在连接外网的机器上拉取镜像，拉取镜像后，将镜像导出为文件，然后上传到内网服务器，进行镜像导入，导入后即可根据需求运行自己的registry容器。
- ###将在仓库服务器上拉取的registry镜像保存成文件。
- root@hub:/data# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- registry latest b8604a3fe854 12 months ago 26.2MB
- root@hub:/data# docker save -o registry.tar.gz registry:latest
- root@hub:/data# ls
- certs localregistry registry-auth registry.tar.gz
-
- ###将导出的文件拷贝到docker客户端，看能否导入。
- root@docker:~# scp -p 192.168.1.129:/data/registry.tar.gz .
- root@192.168.1.129's password:
- registry.tar.gz 100% 26MB 64.7MB/s 00:00
-
- root@docker:~# ls
- registry.tar.gz
- ###导入镜像并查看。
- root@docker:~# docker load -i registry.tar.gz
- 69715584ec78: Loading layer [==================================================>] 5.885MB/5.885MB
- ad10b481abe7: Loading layer [==================================================>] 818.7kB/818.7kB
- aa4330046b37: Loading layer [==================================================>] 20.08MB/20.08MB
- f640be0d5aad: Loading layer [==================================================>] 4.096kB/4.096kB
- aeccf26589a7: Loading layer [==================================================>] 2.048kB/2.048kB
- Loaded image: registry:latest
- root@docker:~# docker images
- REPOSITORY TAG IMAGE ID CREATED SIZE
- registry.dh.com/httpd v1 dabbfbe0c57b 11 months ago 144MB
- registry latest b8604a3fe854 12 months ago 26.2MB
- registry.dh.com/ubuntu v1 ba6acccedd29 13 months ago 72.8MB
二、搭建Harbor私有镜像仓库
1、Harbor简介
Harbor是 VMware 公司开源的企业级 Docker Registry 管理项目，Harbor 在 Docker Registry 的基础上增加了企业用户所需的权限控制、安全漏洞扫描、日志审核和远程复制等重要功能，还提供了图形管理界面及面向国内用户的中文支持。
2、Harbor安装先决条件
安装 Harbor 先决条件Harbor docs | Harbor Installation Prerequisites。
- 硬件资源
最低配置：2CPU、4G内存、40G硬盘。推荐配置：4CPU、8G内存、160G硬盘。
硬件参数是官方提供的参考值，生产中可以根据实际情况进行调整。
- 软件依赖
docker-ce：17.06.0及以上版本。
docker-compose：1.18.0及以上版本。
openssl：尽量最新版本。
- 网络端口
主机可能需要放开的端口，根据实际需要放开即可。
443：Harbor门户网站及核心API使用的HTTPS端口。
4443：仅开启镜像签名（notary）功能时需要。
80：Harbor门户网站及核心API使用的HTTP端口。
3、Harbor的安装
1）停止和删除之前启动的registry容器。
- ###停止并删除之前运行的registry容器。
- root@hub:/data# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- 2aef873145ad registry "/entrypoint.sh /etc…" 2 hours ago Up 2 hours 0.0.0.0:443->443/tcp, :::443->443/tcp, 5000/tcp localregistry
- root@hub:/data# docker stop 2aef
- 2aef
- root@hub:/data# docker rm 2aef
- 2aef
- root@hub:/data# docker ps
- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2）docker-ce前面已经安装了，openssl是系统自带组件，还需要安装docker compose，docker compose下载地址Releases · docker/compose · GitHub，当前最新版本是v2.13.0。
- ###下载当前最新版本。
- root@hub:/data# curl -L "https://github.com/docker/compose/releases/download/v2.13.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- % Total % Received % Xferd Average Speed Time Time Time Current
- Dload Upload Total Spent Left Speed
- 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
- 100 43.5M 100 43.5M 0 0 10.5M 0 0:00:04 0:00:04 --:--:-- 15.5M
-
- ###给下载好的二进制文件执行权限。
- root@hub:/data# chmod +x /usr/local/bin/docker-compose
-
- ###查看compose版本。
- root@hub:/data# docker-compose --version
- Docker Compose version v2.13.0
3）下载Harbor安装包，Harbor下载地址Releases · goharbor/harbor · GitHub，当前最新版本2.6.2版本，安装包分离线版本和在线版本，我们下载离线版本。
- ###下载Harbor。
- root@hub:/data# wget https://github.com/goharbor/harbor/releases/download/v2.6.2/harbor-offline-installer-v2.6.2.tgz
- --2022-11-27 12:52:09-- https://github.com/goharbor/harbor/releases/download/v2.6.2/harbor-offline-installer-v2.6.2.tgz
- Resolving github.com (github.com)... 20.205.243.166
- Connecting to github.com (github.com)|20.205.243.166|:443... connected.
- HTTP request sent, awaiting response... 302 Found
- Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/50613991/d492cc2a-3e93-4989-9de7-7b2f9fbfe8d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221127T125150Z&X-Amz-Expires=300&X-Amz-Signature=dac2b237424173f501590cefb8c8ba56335f6cc485cb9f3e3cfcc4150c05fb9b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=50613991&response-content-disposition=attachment%3B%20filename%3Dharbor-offline-installer-v2.6.2.tgz&response-content-type=application%2Foctet-stream [following]
- --2022-11-27 12:52:10-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/50613991/d492cc2a-3e93-4989-9de7-7b2f9fbfe8d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221127T125150Z&X-Amz-Expires=300&X-Amz-Signature=dac2b237424173f501590cefb8c8ba56335f6cc485cb9f3e3cfcc4150c05fb9b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=50613991&response-content-disposition=attachment%3B%20filename%3Dharbor-offline-installer-v2.6.2.tgz&response-content-type=application%2Foctet-stream
- Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...
- Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
- HTTP request sent, awaiting response... 200 OK
- Length: 806562682 (769M) [application/octet-stream]
- Saving to: ‘harbor-offline-installer-v2.6.2.tgz’
-
- harbor-offline-installer-v2.6.2.tgz 100%[====================================================================================================================>] 769.20M 8.43MB/s in 3m 30s
-
- 2022-11-27 12:55:41 (3.66 MB/s) - ‘harbor-offline-installer-v2.6.2.tgz’ saved [806562682/806562682]
-
- ###解压下载的文件。
- root@hub:/data# ls
- certs harbor-offline-installer-v2.6.2.tgz localregistry registry-auth registry.tar.gz
- root@hub:/data#
- root@hub:/data#
- root@hub:/data# tar -xvf harbor-offline-installer-v2.6.2.tgz
- harbor/harbor.v2.6.2.tar.gz
- harbor/prepare
- harbor/LICENSE
- harbor/install.sh
- harbor/common.sh
- harbor/harbor.yml.tmpl
4） 将harbor.yml.tmpl复制为harbor.yml，并编辑harbor.yml文件。
- ###将harbor.yml.tmpl复制为harbor.yml。
- root@hub:/data# cd harbor
- root@hub:/data/harbor# ls
- LICENSE common.sh harbor.v2.6.2.tar.gz harbor.yml.tmpl install.sh prepare
- root@hub:/data/harbor# cp -p harbor.yml.tmpl harbor.yml
harbor.yml文件主要修改的部分：
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: registry.dh.com #定义域名，和证书的CN匹配。# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80# https related config
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
certificate: /data/certs/myhub.crt #证书实际路径，使用前面章节生成的证书即可。
private_key: /data/certs/myhub.key............
....省略
............
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: myhub123 #admin用户登录密码。
database:
# The password for the root user of Harbor DB. Change this before any production use.
password: myhubdb123 #harbor数据库root用户密码。
# The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
max_idle_conns: 100
# The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
# Note: the default number of connections is 1024 for postgres of harbor.
max_open_conns: 900# The default data volume
data_volume: /data/harbordata/ #harbor数据目录。
5）修改完yml文件后，执行安装脚本，进行安装。
- ###执行安装脚本。
- root@hub:/data/harbor# ./install.sh
-
- [Step 0]: checking if docker is installed ...
-
- Note: docker version: 20.10.20
-
- [Step 1]: checking docker-compose is installed ...
-
- Note: Docker Compose version v2.6.0
-
- [Step 2]: loading Harbor images ...
- 93d5db33d7f0: Loading layer [==================================================>] 37.72MB/37.72MB
- fed5229d083f: Loading layer [==================================================>] 8.902MB/8.902MB
- 60993d7b1dd7: Loading layer [==================================================>] 3.584kB/3.584kB
- dcfafa86c82f: Loading layer [==================================================>] 2.56kB/2.56kB
- 8eecc685f414: Loading layer [==================================================>] 102.3MB/102.3MB
- d93c6405d04a: Loading layer [==================================================>] 103.1MB/103.1MB
- Loaded image: goharbor/harbor-jobservice:v2.6.2
- a279a640c793: Loading layer [==================================================>] 6.287MB/6.287MB
- 4f9d6222bbfe: Loading layer [==================================================>] 4.096kB/4.096kB
- 55ce939e7c9a: Loading layer [==================================================>] 3.072kB/3.072kB
- a631cda93151: Loading layer [==================================================>] 186MB/186MB
- 6fe15b823e2b: Loading layer [==================================================>] 12.87MB/12.87MB
- 48c9bf72ec70: Loading layer [==================================================>] 199.6MB/199.6MB
- Loaded image: goharbor/trivy-adapter-photon:v2.6.2
- a9c145b6d823: Loading layer [==================================================>] 5.759MB/5.759MB
- 7493dee46b78: Loading layer [==================================================>] 90.88MB/90.88MB
- 531fd3669887: Loading layer [==================================================>] 3.072kB/3.072kB
- 8bf93a9c7cac: Loading layer [==================================================>] 4.096kB/4.096kB
- 697af810c7b0: Loading layer [==================================================>] 91.67MB/91.67MB
- Loaded image: goharbor/chartmuseum-photon:v2.6.2
- 565c6d1e1338: Loading layer [==================================================>] 119.9MB/119.9MB
- 070770563b0d: Loading layer [==================================================>] 3.072kB/3.072kB
- 1cbd1219edb1: Loading layer [==================================================>] 59.9kB/59.9kB
- 1eafa56427f8: Loading layer [==================================================>] 61.95kB/61.95kB
- Loaded image: goharbor/redis-photon:v2.6.2
- b1e37b42b3c0: Loading layer [==================================================>] 119.1MB/119.1MB
- Loaded image: goharbor/nginx-photon:v2.6.2
- 0242e94056b7: Loading layer [==================================================>] 5.754MB/5.754MB
- f25460e36589: Loading layer [==================================================>] 8.735MB/8.735MB
- 0854e8369d2a: Loading layer [==================================================>] 14.47MB/14.47MB
- 1d58fd938f61: Loading layer [==================================================>] 29.29MB/29.29MB
- bbf9db741c3d: Loading layer [==================================================>] 22.02kB/22.02kB
- 41ea5b5c915c: Loading layer [==================================================>] 14.47MB/14.47MB
- Loaded image: goharbor/notary-signer-photon:v2.6.2
- 1a91031428df: Loading layer [==================================================>] 8.902MB/8.902MB
- 3b0b5cc72c70: Loading layer [==================================================>] 3.584kB/3.584kB
- 773ad3349c8a: Loading layer [==================================================>] 2.56kB/2.56kB
- 39140cab076f: Loading layer [==================================================>] 83.92MB/83.92MB
- 6c80f545b0fa: Loading layer [==================================================>] 5.632kB/5.632kB
- 0ccee37aa42e: Loading layer [==================================================>] 106.5kB/106.5kB
- 756d72e6b8c7: Loading layer [==================================================>] 44.03kB/44.03kB
- 01a12d2708e6: Loading layer [==================================================>] 84.87MB/84.87MB
- eab188b4f0ed: Loading layer [==================================================>] 2.56kB/2.56kB
- Loaded image: goharbor/harbor-core:v2.6.2
- 3aecbbff620d: Loading layer [==================================================>] 1.097MB/1.097MB
- 3ee789a1e97c: Loading layer [==================================================>] 5.888MB/5.888MB
- 285f58d22327: Loading layer [==================================================>] 169MB/169MB
- b47f4e789504: Loading layer [==================================================>] 16.98MB/16.98MB
- e8b4a74e7102: Loading layer [==================================================>] 4.096kB/4.096kB
- 52bc97105e75: Loading layer [==================================================>] 6.144kB/6.144kB
- 21cad868f5ac: Loading layer [==================================================>] 3.072kB/3.072kB
- 6b24213c30fd: Loading layer [==================================================>] 2.048kB/2.048kB
- 54dd774f4fcd: Loading layer [==================================================>] 2.56kB/2.56kB
- 8fcc4e1141e7: Loading layer [==================================================>] 2.56kB/2.56kB
- 47e81a37ed9a: Loading layer [==================================================>] 2.56kB/2.56kB
- 15a2f5707a75: Loading layer [==================================================>] 8.704kB/8.704kB
- Loaded image: goharbor/harbor-db:v2.6.2
- b87a3132dc28: Loading layer [==================================================>] 5.759MB/5.759MB
- 099c3879eab4: Loading layer [==================================================>] 4.096kB/4.096kB
- e111d5399557: Loading layer [==================================================>] 17.11MB/17.11MB
- 0339872ecc4b: Loading layer [==================================================>] 3.072kB/3.072kB
- 28d42c5fb14d: Loading layer [==================================================>] 29.71MB/29.71MB
- a6e47445b2a9: Loading layer [==================================================>] 47.61MB/47.61MB
- Loaded image: goharbor/harbor-registryctl:v2.6.2
- dc8141f6851b: Loading layer [==================================================>] 8.902MB/8.902MB
- 98b4e2589226: Loading layer [==================================================>] 25.08MB/25.08MB
- b413ead94415: Loading layer [==================================================>] 4.608kB/4.608kB
- 2c8d3d788084: Loading layer [==================================================>] 25.88MB/25.88MB
- Loaded image: goharbor/harbor-exporter:v2.6.2
- 393d5a0ffc78: Loading layer [==================================================>] 43.85MB/43.85MB
- a93c9af803e4: Loading layer [==================================================>] 65.97MB/65.97MB
- 7d77c2b336d0: Loading layer [==================================================>] 18.28MB/18.28MB
- a023a4db994f: Loading layer [==================================================>] 65.54kB/65.54kB
- 938defed7f01: Loading layer [==================================================>] 2.56kB/2.56kB
- 40a257e6076f: Loading layer [==================================================>] 1.536kB/1.536kB
- 0a9ee117d49e: Loading layer [==================================================>] 12.29kB/12.29kB
- fee3e00ceac7: Loading layer [==================================================>] 2.613MB/2.613MB
- aa7a1d37a974: Loading layer [==================================================>] 379.9kB/379.9kB
- Loaded image: goharbor/prepare:v2.6.2
- 43e329977d55: Loading layer [==================================================>] 5.759MB/5.759MB
- cd7f664a365e: Loading layer [==================================================>] 4.096kB/4.096kB
- d52491a2b6aa: Loading layer [==================================================>] 3.072kB/3.072kB
- 96c88da130e9: Loading layer [==================================================>] 17.11MB/17.11MB
- 3ed947e61e01: Loading layer [==================================================>] 17.9MB/17.9MB
- Loaded image: goharbor/registry-photon:v2.6.2
- c326200729cd: Loading layer [==================================================>] 5.754MB/5.754MB
- dddf8a3ea2da: Loading layer [==================================================>] 8.735MB/8.735MB
- 64881611b8c7: Loading layer [==================================================>] 15.88MB/15.88MB
- e8316cf8a6f4: Loading layer [==================================================>] 29.29MB/29.29MB
- a36ddfae8822: Loading layer [==================================================>] 22.02kB/22.02kB
- 071790bc488e: Loading layer [==================================================>] 15.88MB/15.88MB
- Loaded image: goharbor/notary-server-photon:v2.6.2
- 02fbd82c1ad9: Loading layer [==================================================>] 119.1MB/119.1MB
- 4cb31a91fcbf: Loading layer [==================================================>] 7.538MB/7.538MB
- 5c4e30d33b60: Loading layer [==================================================>] 1.185MB/1.185MB
- Loaded image: goharbor/harbor-portal:v2.6.2
- f0b226869da1: Loading layer [==================================================>] 126.9MB/126.9MB
- 375103145f01: Loading layer [==================================================>] 3.584kB/3.584kB
- 5cf0e3f47476: Loading layer [==================================================>] 3.072kB/3.072kB
- 67b68a640c0f: Loading layer [==================================================>] 2.56kB/2.56kB
- d1537b9cab8f: Loading layer [==================================================>] 3.072kB/3.072kB
- fccaf06d29a5: Loading layer [==================================================>] 3.584kB/3.584kB
- 02d47852b759: Loading layer [==================================================>] 20.99kB/20.99kB
- Loaded image: goharbor/harbor-log:v2.6.2
-
-
- [Step 3]: preparing environment ...
-
- [Step 4]: preparing harbor configs ...
- prepare base dir is set to /data/harbor
- Generated configuration file: /config/portal/nginx.conf
- Generated configuration file: /config/log/logrotate.conf
- Generated configuration file: /config/log/rsyslog_docker.conf
- Generated configuration file: /config/nginx/nginx.conf
- Generated configuration file: /config/core/env
- Generated configuration file: /config/core/app.conf
- Generated configuration file: /config/registry/config.yml
- Generated configuration file: /config/registryctl/env
- Generated configuration file: /config/registryctl/config.yml
- Generated configuration file: /config/db/env
- Generated configuration file: /config/jobservice/env
- Generated configuration file: /config/jobservice/config.yml
- Generated and saved secret to file: /data/secret/keys/secretkey
- Successfully called func: create_root_cert
- Generated configuration file: /compose_location/docker-compose.yml
- Clean up the input dir
-
-
- Note: stopping existing Harbor instance ...
- Warning: No resource found to remove for project "harbor".
-
-
- [Step 5]: starting Harbor ...
- [+] Running 10/10
- ⠿ Network harbor_harbor Created 0.2s
- ⠿ Container harbor-log Started 0.9s
- ⠿ Container registry Started 2.2s
- ⠿ Container harbor-portal Started 2.0s
- ⠿ Container redis Started 2.6s
- ⠿ Container registryctl Started 2.4s
- ⠿ Container harbor-db Started 2.6s
- ⠿ Container harbor-core Started 3.6s
- ⠿ Container nginx Started 4.6s
- ⠿ Container harbor-jobservice Started 4.5s
- ✔ ----Harbor has been installed and started successfully.----
6） 安装完后，查看容器运行状态。
- ###使用docker-compose命令查看容器。
- root@hub:/data/harbor# docker-compose ps
- NAME COMMAND SERVICE STATUS PORTS
- harbor-core "/harbor/entrypoint.…" core running (healthy)
- harbor-db "/docker-entrypoint.…" postgresql running (healthy)
- harbor-jobservice "/harbor/entrypoint.…" jobservice running (healthy)
- harbor-log "/bin/sh -c /usr/loc…" log running (healthy) 127.0.0.1:1514->10514/tcp
- harbor-portal "nginx -g 'daemon of…" portal running (healthy)
- nginx "nginx -g 'daemon of…" proxy running (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp
- redis "redis-server /etc/r…" redis running (healthy)
- registry "/home/harbor/entryp…" registry running (healthy)
- registryctl "/home/harbor/start.…" registryctl running (healthy)
7）通过浏览器访问harbor的门户网站，访问方式：https://harbor服务器地址或域名/ ，如果有域名解析使用域名也可以访问。
8）新建一个私有项目，进行镜像上传测试。
9）推送镜像到 harbor。
- ###因为是自签名证书，也还是需要证书信任这一步，参考前面章节。
- root@docker:~#scp 192.168.1.129:/data/certs/myhub.crt /etc/docker/certs.d/registry.dh.com/ca.crt
-
- ###修改镜像标签，推送到myhub项目。
- root@docker:~# docker tag registry.dh.com/httpd:v1 registry.dh.com/myhub/httpd:v1
-
- ###登录harbor。
- root@docker:~# docker login registry.dh.com
- Username: admin
- Password:
- WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
- Configure a credential helper to remove this warning. See
- https://docs.docker.com/engine/reference/commandline/login/#credentials-store
-
- Login Succeeded
-
- ###推送镜像。
- root@docker:~# docker push registry.dh.com/myhub/httpd:v1
- The push refers to repository [registry.dh.com/myhub/httpd]
- deefaa620a71: Pushed
- 9cff3206f9a6: Pushed
- 15e4bf5d0804: Pushed
- 1da636a1aa95: Pushed
- 2edcec3590a4: Pushed
- v1: digest: sha256:57c1e4ff150e2782a25c8cebb80b574f81f06b74944caf972f27e21b76074194 size: 1365
10）查看镜像已成功上传。
4、Harbor的启动和停止
进入到 harbor 的安装包目录，执行命令对 harbor 进行启停。也可以使用 -f 参数指定 yml 文件位置。
- ###进入到harbor的安装包目录。
- cd /data/harbor
-
- ###停止harbor。
- root@hub:/data/harbor# docker-compose stop
- [+] Running 9/9
- ⠿ Container nginx Stopped 0.3s
- ⠿ Container harbor-jobservice Stopped 0.3s
- ⠿ Container registryctl Stopped 10.2s
- ⠿ Container harbor-core Stopped 0.2s
- ⠿ Container harbor-portal Stopped 0.2s
- ⠿ Container redis Stopped 0.4s
- ⠿ Container registry Stopped 0.3s
- ⠿ Container harbor-db Stopped 0.3s
- ⠿ Container harbor-log Stopped
-
- ###启动harbor。
- root@hub:/data/harbor# docker-compose start
- [+] Running 9/9
- ⠿ Container harbor-log Started 0.5s
- ⠿ Container registry Started 2.0s
- ⠿ Container redis Started 1.9s
- ⠿ Container harbor-portal Started 1.7s
- ⠿ Container harbor-db Started 1.8s
- ⠿ Container registryctl Started 1.4s
- ⠿ Container harbor-core Started 0.7s
- ⠿ Container nginx Started 1.5s
- ⠿ Container harbor-jobservice Started
-
- ###也可以使用-f参数指定yml文件。
- root@hub:/# docker-compose -f /data/harbor/docker-compose.yml stop
- [+] Running 9/9
- ⠿ Container nginx Stopped 0.3s
- ⠿ Container registryctl Stopped 10.2s
- ⠿ Container harbor-jobservice Stopped 0.3s
- ⠿ Container harbor-portal Stopped 0.2s
- ⠿ Container harbor-core Stopped 3.2s
- ⠿ Container harbor-db Stopped 0.3s
- ⠿ Container registry Stopped 0.3s
- ⠿ Container redis Stopped 0.3s
- ⠿ Container harbor-log Stopped
5、Harbor的卸载
Harbor 的组件都是容器，卸载其实就是把容器停止并删掉。docker-compose down命令会停止并删掉正在运行的容器，后面加-v参数可以删除已经在compose文件中定义的和匿名的附在容器上的数据卷。
- ###进到harbor的安装包目录。
- root@hub:~# cd /data/harbor
-
- ###停止并删除配置文件中定义的所有容器。
- root@hub:/data/harbor# docker-compose down -v
- [+] Running 10/10
- ⠿ Container harbor-jobservice Removed 0.3s
- ⠿ Container registryctl Removed 10.2s
- ⠿ Container nginx Removed 0.3s
- ⠿ Container harbor-portal Removed 0.2s
- ⠿ Container harbor-core Removed 3.2s
- ⠿ Container redis Removed 0.3s
- ⠿ Container harbor-db Removed 0.3s
- ⠿ Container registry Removed 0.3s
- ⠿ Container harbor-log Removed 10.2s
- ⠿ Network harbor_harbor Removed
-
- ###删除定义好的harbor数据目录。
- root@hub:/data/harbor# rm -rf /data/harbordata/
-
- ###如果彻底不用了，还可以把拉取的镜像一并删掉。
- ###如需要重装，还是执行安装脚本即可。
总结
以上就是今天学习的内容，主要是两种私有镜像仓库的搭建，在生产中可以根据自己的具体需求来构建自己的镜像仓库。
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/码创造者/article/detail/797274
推荐阅读
相关标签
