docker部署dataX(基于xxl-job)_docker datax

docker部署dataX(基于xxl-job)

所需包可在以下链接下载

https://download.csdn.net/download/qq_41974343/88014392

创建所需目录

mkdir -p /opt/docker/dataX-xxl-job/config/xxl-job            ----xxl-job配置文件目录
mkdir -p /opt/docker/dataX-xxl-job/logs/xxl-job              ----xxl-job日志目录
mkdir -p /opt/docker/dataX-xxl-job/libs                      ----dataX包存放处
mkdir -p /opt/docker/dataX-xxl-job/list                      ----国内源
mkdir -p /opt/docker/dataX-xxl-job/jdk                      ----存放更新的jdk
mkdir -p /opt/docker/dataX-xxl-job/project                  ----默认执行器存放位置
1
2
3
4
5
6

Dockerfile文件

进入目录

cd /opt/docker/dataX-xxl-job
vim Dockerfile

FROM xuxueli/xxl-job-admin:2.3.0
RUN mkdir -p  /libs
RUN mv /etc/apt/sources.list /etc/apt/sources.list.back
RUN mkdir -p /project
COPY ./list/source.list /etc/apt/sources.list
COPY ./project/xxl-job-executor-sample-springboot-2.4.0-SNAPSHOT.jar /project/application.jar
ADD ./project/application.properties /project/application.properties
RUN mv /usr/local/openjdk-8 /usr/local/openjdk-8-bak
COPY ./jdk/openjdk-8 /usr/local/openjdk-8
EXPOSE 8081
EXPOSE 9999
1
2
3
4
5
6
7
8
9
10
11

Dockerfile描述

1.基于xxl-job-admin 的镜像
2.创建/libs文件夹用于存放datax文件
3.备份镜像中apt源文件
4.创建名project目录，用于存放默认执行器项目
5.上传准备好的国内apt源文件
6.上传默认执行器jar项目
7.上传默认执行器配置文件
8.将镜像的默认jdk备份
9.上传准备好的jdk
10.对外暴露端口8081（默认执行器端口）
11.对外暴露端口9999（默认执行器注册到xxl-job的端口）

更换容器apt源

上传源文件sources.list ，由于xuxueli/xxl-job-admin:2.3.0镜像版本是debian 10系统的，固修改以下内容

echo "deb http://mirrors.aliyun.com/debian/ buster main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian/ buster main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb http://mirrors.aliyun.com/debian-security buster/updates main" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian-security buster/updates main" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb http://mirrors.aliyun.com/debian/ buster-updates main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian/ buster-updates main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb http://mirrors.aliyun.com/debian/ buster-backports main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian/ buster-backports main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list
1
2
3
4
5
6
7
8

xxl-job安装

1. 准备一台mysql，新建数据库xxl_job，并执行脚本sql
2. 上传xxl-job配置文件到/opt/docker/dataX-xxl-job/config/xxl-job目录下

### web
server.port=8080
server.servlet.context-path=/xxl-job-admin

### actuator
management.server.servlet.context-path=/actuator
management.health.mail.enabled=false

### resources
spring.mvc.servlet.load-on-startup=0
spring.mvc.static-path-pattern=/static/**
spring.resources.static-locations=classpath:/static/

### freemarker
spring.freemarker.templateLoaderPath=classpath:/templates/
spring.freemarker.suffix=.ftl
spring.freemarker.charset=UTF-8
spring.freemarker.request-context-attribute=request
spring.freemarker.settings.number_format=0.##########

### mybatis
mybatis.mapper-locations=classpath:/mybatis-mapper/*Mapper.xml
#mybatis.type-aliases-package=com.xxl.job.admin.core.model
### xxl-job, datasource
spring.datasource.url=jdbc:mysql://数据地址:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai
spring.datasource.username=root
spring.datasource.password=密码
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

### datasource-pool
spring.datasource.type=com.zaxxer.hikari.HikariDataSource
spring.datasource.hikari.minimum-idle=10
spring.datasource.hikari.maximum-pool-size=30
spring.datasource.hikari.auto-commit=true
spring.datasource.hikari.idle-timeout=30000
spring.datasource.hikari.pool-name=HikariCP
spring.datasource.hikari.max-lifetime=900000
spring.datasource.hikari.connection-timeout=10000
spring.datasource.hikari.connection-test-query=SELECT 1
spring.datasource.hikari.validation-timeout=1000

### xxl-job, email
spring.mail.host=smtp.qq.com
spring.mail.port=25
spring.mail.username=xxx@qq.com
spring.mail.from=xxx@qq.com
spring.mail.password=xxx
spring.mail.properties.mail.smtp.auth=true
spring.mail.properties.mail.smtp.starttls.enable=true
spring.mail.properties.mail.smtp.starttls.required=true
spring.mail.properties.mail.smtp.socketFactory.class=javax.net.ssl.SSLSocketFactory

### xxl-job, access token
xxl.job.accessToken=

### xxl-job, i18n (default is zh_CN, and you can choose "zh_CN", "zh_TC" and "en")
xxl.job.i18n=zh_CN

## xxl-job, triggerpool max size
xxl.job.triggerpool.fast.max=200
xxl.job.triggerpool.slow.max=100
### xxl-job, log retention days
xxl.job.logretentiondays=30
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

上传默认执行器项目到project目录下,注：这里的项目由jdk11打包的，如需要其他版本可自行打包上传

上传以下执行器配置文件至 project目录下

# web port
server.port=8081
# no web
#spring.main.web-environment=false

# log config
logging.config=classpath:logback.xml


### xxl-job admin address list, such as "http://address" or "http://address01,http://address02"
xxl.job.admin.addresses=http://当前宿主机ip:8380/xxl-job-admin/

### xxl-job, access token
xxl.job.accessToken=

### xxl-job executor appname
xxl.job.executor.appname=xxl-job-executor-sample
### xxl-job executor registry-address: default use address to registry , otherwise use ip:port if address is null
xxl.job.executor.address=
### xxl-job executor server-info
xxl.job.executor.ip=当前宿主机ip
xxl.job.executor.port=9999
### xxl-job executor log-path
xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler
### xxl-job executor log-retention-days
xxl.job.executor.logretentiondays=30
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

上传datax压缩包到 libs目录下并解压缩
下载地址如下：https://github.com/alibaba/DataX

上传jdk至jdk目录下(这里需要openjdk)，注:如果默认执行器是jdk8打包的无需此步骤
下载地址：https://jdk.java.net/java-se-ri/11-MR2
也可下面地址下载：
将包解压缩并改名为openjdk-8

构建镜像及运行容器

cd /opt/docker/dataX-xxl-job
docker build -t xxl-job-datax .    ---构建容器
docker run -d --name xxl-job-dataX -p 8380:8080 -p 9999:9999 -p 8081:8081 \ 
-v /opt/docker/dataX-xxl-job/config/xxl-job/application.properties:/application.properties \ 
-v /opt/docker/dataX-xxl-job/logs/xxl-job:/data/applogs \ 
-v /opt/docker/dataX-xxl-job/libs:/libs \ 
-e PARAMS='--spring.config.location=/application.properties' \ 
xxl-job-datax
1
2
3
4
5
6
7
8

启动完成后，可通过http://ip地址:8380/xxl-job-admin/访问
账号：admin 密码：123456

看到页面后需进入容器

docker exec -it xxl-job-dataX /bin/bash    ---进入容器
apt-get update                             ---更新apt源
apt-get install -y python2                 ---安装python2
cd /project
java -jar -Duser.timezone=GMT+08   -Dspring.config.location=./application.properties application.jar  --启动默认执行器项目
1
2
3
4
5

再次访问地址http://ip地址:8380/xxl-job-admin/访问
账号：admin 密码：123456
运行完可在执行器列表找到该执行器

到此部署完毕，只需把dataX的脚本放到/opt/docker/dataX-xxl-job/libs/datax/job 下
便可通过python2 /libs/datax/bin/datax.py /libs/datax/job/脚本名 进行同步！