做这个实验目标是往kubernetes集群里交付dubbo服务,采用jenkins+maven+gitee实现,jenkins使用的是v2.303.1,maven使用3.6.3+jre8u91。整个项目的拓扑图如下:

主机

作用

host11

zk节点、为整个集群提供dns、nginx为k8s集群提供proxy

host12

zk节点、nginx为k8s集群提供proxy

host21

k8s-node、zk节点、etcd节点

host22

k8s-node、etcd节点

host200

harbor仓库节点、httpd为集群提供资源配置清单文件,nfs为pod提供持久化存储

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker_02

​​01 在jenkins创建流水线​​

 1)新建任务

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_git_03

2)名称为dubbo-demo,选择流水线,并确定

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker_04

3)保留30份,并使用参数化构建过程

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_git_05

4)添加参数,这里我们一共需要添加10个参数

4.1 设置app_name 和 image_name

  • ​app_name: app名字​

参数类型:字符参数变量名: app_name变量的默认值: 描述:dubbo-service

Cop

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_maven_06

  • ​image_name: 镜像名字​

参数类型:字符参数变量名: image_name  变量的默认值: 描述:app/dubbo-demo-service

4.2 设置变量 git_repo 和 git_version

  • ​git_repo: 代码的git地址,这里xxx需要替换为自己的用户名​

参数类型:字符参数变量名: git_repo变量的默认值:描述:git@gitee.com:xxxxxx/dubbo-demo-service.git
  • ​git_ver: 拉取代码哪个分支​

参数类型:字符参数变量名: git_ver变量的默认值: master描述:master

4.3 设置变量 add_tag 和 mvn_dir

  • ​add_tag: 打包代码后的镜像的tag,生成docker镜像时:image_name:add_tag​

参数类型:字符参数变量名: add_tag变量的默认值: 描述:211003_1857 即年月日_时间
  • ​mvn_dir: maven打包时候的执行目录​

参数类型:字符参数变量名: mvn_dir变量的默认值: 描述:./

4.4 设置变量 target_dir 和 mvn_cmd

  • ​target_dir: maven打包完成后生成的jar包所在的目录, 一般是在执行目录下生成target目录, jar包在target目录下​

类型:字符参数变量名: target_dir变量的默认值: 描述:/dubbo-server/target
  • ​mvn_cmd: maven打包时候执行的命令​

参数类型:字符参数变量名: mvn_cmd变量的默认值: mvn clean package -Dmaven.test.skip=true描述:mvn clean package -Dmaven.test.skip=true

4.5 设置变量 base_image 和 maven

  • ​base_image:构建镜像时候所用的底包是哪个镜像​

参数:选项参数变量名: base_image变量的选项:base/jre8:8u112

  • ​maven: 构建镜像时候使用的maven版本,上次我们配置了java11和java8​

    ​这里的变量的选项要和在 ​​host200:/data/jenkins/jenkins_home​​​下的maven的目录是一样的, 在打包时候直接是使用这个绝对路径来调用​​mvn​​​命令进行打包的​

参数:选项参数变量名: maven变量的选项:3.6.3-8u2913.6.3-11.0.12

4.6 配置流水线(pipeline)

pipeline {  agent any     stages {      stage('pull') { //get project code from repo         steps {          sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"        }      }      stage('build') { //exec mvn cmd        steps {          sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"        }      }      stage('package') { //move jar file into project_dir        steps {          sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"        }      }      stage('image') { //build image and push to registry        steps {          writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}ADD ${params.target_dir}/project_dir /opt/project_dir"""          sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"        }      }    }}
  • ​stage('pull'):从gitee上拉取代码​

  • ​stage('build'):进入maven,执行maven命令​

  • ​stage('package'):编译为jar包​

  • ​stage('image'):在生成镜像并上传到harbor​

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_git_07

5)保存

​​02 配置自己的gitee仓库​​

1)登录gitee.com,fork一个dubbo-demo-service仓库,注意看dubbo-server/src/main/java/config.properties,zk地址是否为od.com

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_maven_08

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker_09

2)在管理中设置为私有仓库

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_maven_10

3)添加公钥 -- 这个如果在安装jenkins篇中已经做过的就不用再做了。在我-设置-公钥,添加host200的公钥进去(/root/.ssh/id_rsa.pub)

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_git_11

4)登录jenkins测试,xxx要替换为自己的用户名

[root@host22 src]# kubectl exec -it pod/jenkins-7df8bf4ff9-qj85m /bin/bash -n infraroot@jenkins-7df8bf4ff9-qj85m:/# git clone git@gitee.com:xxxxx/dubbo-demo-service Cloning into 'dubbo-demo-service'...Warning: Permanently added 'gitee.com,212.64.62.183' (ECDSA) to the list of known hosts.remote: Enumerating objects: 97, done.remote: Counting objects: 100% (97/97), done.remote: Compressing objects: 100% (61/61), done.remote: Total 97 (delta 30), reused 72 (delta 18), pack-reused 0Receiving objects: 100% (97/97), 11.95 KiB | 1.99 MiB/s, done.Resolving deltas: 100% (30/30), done.root@jenkins-7df8bf4ff9-qj85m:/# lsbin  boot  dev  dubbo-demo-service

​03 配置harbor仓库​

1)新建一个私有仓库app

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_maven_12

​04 在host200构建dubbo底包​

1)下载jre:8u211容器

docker pull docker.io/stanleyws/jre8:8u112

2)添加tag

docker tag fa3a085d6ef1 harbor.od.com/public/jre:8u112docker push harbor.od.com/public/jre:8u112

3)基于底包构建新的jre镜像

mkdir -pv /data/dockerfile/jre8# 创建dockerfile文件cd /data/dockerfile/jre8cat DockerfileFROM  harbor.od.com/public/jre:8u112RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\    echo 'Asia/Shanghai' >/etc/timezoneADD config.yml /opt/prom/config.ymlADD jmx_javaagent-0.3.1.jar /opt/prom/WORKDIR /opt/project_dirADD entrypoint.sh /entrypoint.shCMD ["/entrypoint.sh"]# 创建entrypoint.sh文件cat entrypoint.sh#!/bin/shM_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"C_OPTS=${C_OPTS}JAR_BALL=${JAR_BALL}exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}# 创建config.yml文件cat config.yml ---rules:  - pattern: '.*'# 下载jmx_javaagent-0.3.1.jar,这主要是后面给普罗米修斯用的wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar

4)构建新镜像,并上传到harbor,先在harbor中创建一个公开仓库base

docker build . -t harbor.od.com/base/jre:8u211docker push harbor.od.com/base/jre:8u211

​05 执行流水线​

1)打开blue ocean

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_git_13

2)选择dubbo-demo

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker_14

3)点击运行

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_maven_15

4)配置参数

app_name:dubbo-demo-serviceimage_name:app/dubbo-demo-servicegit_repo:git@gitee.com:xxx/dubbo-demo-servicegit_version:masteradd_tag:211004_0011mvn_dir:./target_dir:./dubbo-server/target/mvn_cmd:默认base_image:base/jre:8u112maven:3.6.3-8u291

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker_16

5)run起来

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_git_17

6)去harbor查看是否有镜像了

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker_18

​06 在任意一台node配置kubernetes namespace​

1)创建namespace

kubectl create namespace app

2)由于使用的是私有仓库,所以要配置一个secret

kubectl create secret docker-registry harbor --docker-username=admin --docker-password=xxx -n app

​07 交付dubbo-server​

1)dubbo-server资源配置清单,注意image的名字要和harbor中的对应上。

# 在host200上创建目录mkdir -pv /data/k8s-yaml/dubbo-server/vi /data/k8s-yaml/dubbo-server/dp.yaml kind: DeploymentapiVersion: extensions/v1beta1metadata:  name: dubbo-demo-service  namespace: app  labels:     name: dubbo-demo-servicespec:  replicas: 1  selector:    matchLabels:       name: dubbo-demo-service  template:    metadata:      labels:         app: dubbo-demo-service        name: dubbo-demo-service    spec:      containers:      - name: dubbo-demo-service        image: harbor.od.com/app/dubbo-demo-service:master_211004_0011        ports:        - containerPort: 20880          protocol: TCP        env:        - name: JAR_BALL          value: dubbo-server.jar        imagePullPolicy: IfNotPresent      imagePullSecrets:      - name: harbor      restartPolicy: Always      terminationGracePeriodSeconds: 30      securityContext:         runAsUser: 0      schedulerName: default-scheduler  strategy:    type: RollingUpdate    rollingUpdate:       maxUnavailable: 1      maxSurge: 1  revisionHistoryLimit: 7  progressDeadlineSeconds: 600

2)在k8s node节点创建

kubectl apply -f http://k8s-yaml.od.com/dubbo-server/dp.yaml

3)查看app命名空间中的pod

[root@host22 src]# kubectl get all -n app NAME                                      READY   STATUS    RESTARTS   AGEpod/dubbo-demo-service-5f669d9f6f-mhptw   1/1     Running   0          15sNAME                                 READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/dubbo-demo-service   1/1     1            1           13hNAME                                            DESIRED   CURRENT   READY   AGEreplicaset.apps/dubbo-demo-service-5f669d9f6f   1         1         1       13h

4)登录zk查看是否有dubbo目录

/opt/zookeeper/bin/zkCli.sh -server localhost:2181ls /

[kubernetes] 交付dubbo之jenkins持续交付dubbo-server_docker_19