做这个实验目标是往kubernetes集群里交付dubbo服务,采用jenkins+maven+gitee实现,jenkins使用的是v2.303.1,maven使用3.6.3+jre8u91。整个项目的拓扑图如下:
主机 | 作用 |
host11 | zk节点、为整个集群提供dns、nginx为k8s集群提供proxy |
host12 | zk节点、nginx为k8s集群提供proxy |
host21 | k8s-node、zk节点、etcd节点 |
host22 | k8s-node、etcd节点 |
host200 | harbor仓库节点、httpd为集群提供资源配置清单文件,nfs为pod提供持久化存储 |
01 在jenkins创建流水线
1)新建任务
2)名称为dubbo-demo,选择流水线,并确定
3)保留30份,并使用参数化构建过程
4)添加参数,这里我们一共需要添加10个参数
4.1 设置app_name 和 image_name
app_name: app名字
参数类型:字符参数变量名: app_name变量的默认值: 描述:dubbo-service
Cop
image_name: 镜像名字
参数类型:字符参数变量名: image_name 变量的默认值: 描述:app/dubbo-demo-service
4.2 设置变量 git_repo 和 git_version
git_repo: 代码的git地址,这里xxx需要替换为自己的用户名
参数类型:字符参数变量名: git_repo变量的默认值:描述:git@gitee.com:xxxxxx/dubbo-demo-service.git
git_ver: 拉取代码哪个分支
参数类型:字符参数变量名: git_ver变量的默认值: master描述:master
4.3 设置变量 add_tag 和 mvn_dir
add_tag: 打包代码后的镜像的tag,生成docker镜像时:image_name:add_tag
参数类型:字符参数变量名: add_tag变量的默认值: 描述:211003_1857 即年月日_时间
mvn_dir: maven打包时候的执行目录
参数类型:字符参数变量名: mvn_dir变量的默认值: 描述:./
4.4 设置变量 target_dir 和 mvn_cmd
target_dir: maven打包完成后生成的jar包所在的目录, 一般是在执行目录下生成target目录, jar包在target目录下
类型:字符参数变量名: target_dir变量的默认值: 描述:/dubbo-server/target
mvn_cmd: maven打包时候执行的命令
参数类型:字符参数变量名: mvn_cmd变量的默认值: mvn clean package -Dmaven.test.skip=true描述:mvn clean package -Dmaven.test.skip=true
4.5 设置变量 base_image 和 maven
base_image:构建镜像时候所用的底包是哪个镜像
参数:选项参数变量名: base_image变量的选项:base/jre8:8u112
maven: 构建镜像时候使用的maven版本,上次我们配置了java11和java8
这里的变量的选项要和在
host200:/data/jenkins/jenkins_home
下的maven的目录是一样的, 在打包时候直接是使用这个绝对路径来调用mvn
命令进行打包的
参数:选项参数变量名: maven变量的选项:3.6.3-8u2913.6.3-11.0.12
4.6 配置流水线(pipeline)
pipeline { agent any stages { stage('pull') { //get project code from repo steps { sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}" } } stage('build') { //exec mvn cmd steps { sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}" } } stage('package') { //move jar file into project_dir steps { sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir" } } stage('image') { //build image and push to registry steps { writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}ADD ${params.target_dir}/project_dir /opt/project_dir""" sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}" } } }}
stage('pull'):从gitee上拉取代码
stage('build'):进入maven,执行maven命令
stage('package'):编译为jar包
stage('image'):在生成镜像并上传到harbor
5)保存
02 配置自己的gitee仓库
1)登录gitee.com,fork一个dubbo-demo-service仓库,注意看dubbo-server/src/main/java/config.properties,zk地址是否为od.com
2)在管理中设置为私有仓库
3)添加公钥 -- 这个如果在安装jenkins篇中已经做过的就不用再做了。在我-设置-公钥,添加host200的公钥进去(/root/.ssh/id_rsa.pub)
4)登录jenkins测试,xxx要替换为自己的用户名
[root@host22 src]# kubectl exec -it pod/jenkins-7df8bf4ff9-qj85m /bin/bash -n infraroot@jenkins-7df8bf4ff9-qj85m:/# git clone git@gitee.com:xxxxx/dubbo-demo-service Cloning into 'dubbo-demo-service'...Warning: Permanently added 'gitee.com,212.64.62.183' (ECDSA) to the list of known hosts.remote: Enumerating objects: 97, done.remote: Counting objects: 100% (97/97), done.remote: Compressing objects: 100% (61/61), done.remote: Total 97 (delta 30), reused 72 (delta 18), pack-reused 0Receiving objects: 100% (97/97), 11.95 KiB | 1.99 MiB/s, done.Resolving deltas: 100% (30/30), done.root@jenkins-7df8bf4ff9-qj85m:/# lsbin boot dev dubbo-demo-service
03 配置harbor仓库
1)新建一个私有仓库app
04 在host200构建dubbo底包
1)下载jre:8u211容器
docker pull docker.io/stanleyws/jre8:8u112
2)添加tag
docker tag fa3a085d6ef1 harbor.od.com/public/jre:8u112docker push harbor.od.com/public/jre:8u112
3)基于底包构建新的jre镜像
mkdir -pv /data/dockerfile/jre8# 创建dockerfile文件cd /data/dockerfile/jre8cat DockerfileFROM harbor.od.com/public/jre:8u112RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo 'Asia/Shanghai' >/etc/timezoneADD config.yml /opt/prom/config.ymlADD jmx_javaagent-0.3.1.jar /opt/prom/WORKDIR /opt/project_dirADD entrypoint.sh /entrypoint.shCMD ["/entrypoint.sh"]# 创建entrypoint.sh文件cat entrypoint.sh#!/bin/shM_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"C_OPTS=${C_OPTS}JAR_BALL=${JAR_BALL}exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}# 创建config.yml文件cat config.yml ---rules: - pattern: '.*'# 下载jmx_javaagent-0.3.1.jar,这主要是后面给普罗米修斯用的wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar
4)构建新镜像,并上传到harbor,先在harbor中创建一个公开仓库base
docker build . -t harbor.od.com/base/jre:8u211docker push harbor.od.com/base/jre:8u211
05 执行流水线
1)打开blue ocean
2)选择dubbo-demo
3)点击运行
4)配置参数
app_name:dubbo-demo-serviceimage_name:app/dubbo-demo-servicegit_repo:git@gitee.com:xxx/dubbo-demo-servicegit_version:masteradd_tag:211004_0011mvn_dir:./target_dir:./dubbo-server/target/mvn_cmd:默认base_image:base/jre:8u112maven:3.6.3-8u291
5)run起来
6)去harbor查看是否有镜像了
06 在任意一台node配置kubernetes namespace
1)创建namespace
kubectl create namespace app
2)由于使用的是私有仓库,所以要配置一个secret
kubectl create secret docker-registry harbor --docker-username=admin --docker-password=xxx -n app
07 交付dubbo-server
1)dubbo-server资源配置清单,注意image的名字要和harbor中的对应上。
# 在host200上创建目录mkdir -pv /data/k8s-yaml/dubbo-server/vi /data/k8s-yaml/dubbo-server/dp.yaml kind: DeploymentapiVersion: extensions/v1beta1metadata: name: dubbo-demo-service namespace: app labels: name: dubbo-demo-servicespec: replicas: 1 selector: matchLabels: name: dubbo-demo-service template: metadata: labels: app: dubbo-demo-service name: dubbo-demo-service spec: containers: - name: dubbo-demo-service image: harbor.od.com/app/dubbo-demo-service:master_211004_0011 ports: - containerPort: 20880 protocol: TCP env: - name: JAR_BALL value: dubbo-server.jar imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600
2)在k8s node节点创建
kubectl apply -f http://k8s-yaml.od.com/dubbo-server/dp.yaml
3)查看app命名空间中的pod
[root@host22 src]# kubectl get all -n app NAME READY STATUS RESTARTS AGEpod/dubbo-demo-service-5f669d9f6f-mhptw 1/1 Running 0 15sNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/dubbo-demo-service 1/1 1 1 13hNAME DESIRED CURRENT READY AGEreplicaset.apps/dubbo-demo-service-5f669d9f6f 1 1 1 13h
4)登录zk查看是否有dubbo目录
/opt/zookeeper/bin/zkCli.sh -server localhost:2181ls /