当前位置:   article > 正文

docker部署dataX(基于xxl-job)_docker datax

docker datax

docker部署dataX(基于xxl-job)

所需包可在以下链接下载

https://download.csdn.net/download/qq_41974343/88014392

创建所需目录
mkdir -p /opt/docker/dataX-xxl-job/config/xxl-job            ----xxl-job配置文件目录
mkdir -p /opt/docker/dataX-xxl-job/logs/xxl-job              ----xxl-job日志目录
mkdir -p /opt/docker/dataX-xxl-job/libs                      ----dataX包存放处
mkdir -p /opt/docker/dataX-xxl-job/list                      ----国内源
mkdir -p /opt/docker/dataX-xxl-job/jdk                      ----存放更新的jdk
mkdir -p /opt/docker/dataX-xxl-job/project                  ----默认执行器存放位置
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
Dockerfile文件
进入目录

cd /opt/docker/dataX-xxl-job
vim Dockerfile

FROM xuxueli/xxl-job-admin:2.3.0
RUN mkdir -p  /libs
RUN mv /etc/apt/sources.list /etc/apt/sources.list.back
RUN mkdir -p /project
COPY ./list/source.list /etc/apt/sources.list
COPY ./project/xxl-job-executor-sample-springboot-2.4.0-SNAPSHOT.jar /project/application.jar
ADD ./project/application.properties /project/application.properties
RUN mv /usr/local/openjdk-8 /usr/local/openjdk-8-bak
COPY ./jdk/openjdk-8 /usr/local/openjdk-8
EXPOSE 8081
EXPOSE 9999
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
Dockerfile描述

1.基于xxl-job-admin 的镜像
2.创建/libs文件夹用于存放datax文件
3.备份镜像中apt源文件
4.创建名project目录,用于存放默认执行器项目
5.上传准备好的国内apt源文件
6.上传默认执行器jar项目
7.上传默认执行器配置文件
8.将镜像的默认jdk备份
9.上传准备好的jdk
10.对外暴露端口8081(默认执行器端口)
11.对外暴露端口9999(默认执行器注册到xxl-job的端口)

更换容器apt源

上传源文件sources.list ,由于xuxueli/xxl-job-admin:2.3.0镜像版本是debian 10系统的,固修改以下内容

echo "deb http://mirrors.aliyun.com/debian/ buster main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian/ buster main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb http://mirrors.aliyun.com/debian-security buster/updates main" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian-security buster/updates main" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb http://mirrors.aliyun.com/debian/ buster-updates main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian/ buster-updates main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb http://mirrors.aliyun.com/debian/ buster-backports main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list&&
echo "deb-src http://mirrors.aliyun.com/debian/ buster-backports main non-free contrib" >> /opt/docker/dataX-xxl-job/list/source.list
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
xxl-job安装
  1. 准备一台mysql,新建数据库xxl_job,并执行脚本sql
  2. 上传xxl-job配置文件到/opt/docker/dataX-xxl-job/config/xxl-job目录下

### web
server.port=8080
server.servlet.context-path=/xxl-job-admin

### actuator
management.server.servlet.context-path=/actuator
management.health.mail.enabled=false

### resources
spring.mvc.servlet.load-on-startup=0
spring.mvc.static-path-pattern=/static/**
spring.resources.static-locations=classpath:/static/

### freemarker
spring.freemarker.templateLoaderPath=classpath:/templates/
spring.freemarker.suffix=.ftl
spring.freemarker.charset=UTF-8
spring.freemarker.request-context-attribute=request
spring.freemarker.settings.number_format=0.##########

### mybatis
mybatis.mapper-locations=classpath:/mybatis-mapper/*Mapper.xml
#mybatis.type-aliases-package=com.xxl.job.admin.core.model
### xxl-job, datasource
spring.datasource.url=jdbc:mysql://数据地址:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai
spring.datasource.username=root
spring.datasource.password=密码
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

### datasource-pool
spring.datasource.type=com.zaxxer.hikari.HikariDataSource
spring.datasource.hikari.minimum-idle=10
spring.datasource.hikari.maximum-pool-size=30
spring.datasource.hikari.auto-commit=true
spring.datasource.hikari.idle-timeout=30000
spring.datasource.hikari.pool-name=HikariCP
spring.datasource.hikari.max-lifetime=900000
spring.datasource.hikari.connection-timeout=10000
spring.datasource.hikari.connection-test-query=SELECT 1
spring.datasource.hikari.validation-timeout=1000

### xxl-job, email
spring.mail.host=smtp.qq.com
spring.mail.port=25
spring.mail.username=xxx@qq.com
spring.mail.from=xxx@qq.com
spring.mail.password=xxx
spring.mail.properties.mail.smtp.auth=true
spring.mail.properties.mail.smtp.starttls.enable=true
spring.mail.properties.mail.smtp.starttls.required=true
spring.mail.properties.mail.smtp.socketFactory.class=javax.net.ssl.SSLSocketFactory

### xxl-job, access token
xxl.job.accessToken=

### xxl-job, i18n (default is zh_CN, and you can choose "zh_CN", "zh_TC" and "en")
xxl.job.i18n=zh_CN

## xxl-job, triggerpool max size
xxl.job.triggerpool.fast.max=200
xxl.job.triggerpool.slow.max=100
### xxl-job, log retention days
xxl.job.logretentiondays=30
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64

上传默认执行器项目到project目录下,注:这里的项目由jdk11打包的,如需要其他版本可自行打包上传

上传以下执行器配置文件至 project目录下
# web port
server.port=8081
# no web
#spring.main.web-environment=false

# log config
logging.config=classpath:logback.xml


### xxl-job admin address list, such as "http://address" or "http://address01,http://address02"
xxl.job.admin.addresses=http://当前宿主机ip:8380/xxl-job-admin/

### xxl-job, access token
xxl.job.accessToken=

### xxl-job executor appname
xxl.job.executor.appname=xxl-job-executor-sample
### xxl-job executor registry-address: default use address to registry , otherwise use ip:port if address is null
xxl.job.executor.address=
### xxl-job executor server-info
xxl.job.executor.ip=当前宿主机ip
xxl.job.executor.port=9999
### xxl-job executor log-path
xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler
### xxl-job executor log-retention-days
xxl.job.executor.logretentiondays=30
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26

上传datax压缩包到 libs目录下并解压缩
下载地址如下:https://github.com/alibaba/DataX
在这里插入图片描述

上传jdk至jdk目录下(这里需要openjdk),注:如果默认执行器是jdk8打包的无需此步骤
下载地址:https://jdk.java.net/java-se-ri/11-MR2
也可下面地址下载:
将包解压缩并改名为openjdk-8

构建镜像及运行容器
cd /opt/docker/dataX-xxl-job
docker build -t xxl-job-datax .    ---构建容器
docker run -d --name xxl-job-dataX -p 8380:8080 -p 9999:9999 -p 8081:8081 \ 
-v /opt/docker/dataX-xxl-job/config/xxl-job/application.properties:/application.properties \ 
-v /opt/docker/dataX-xxl-job/logs/xxl-job:/data/applogs \ 
-v /opt/docker/dataX-xxl-job/libs:/libs \ 
-e PARAMS='--spring.config.location=/application.properties' \ 
xxl-job-datax
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8

启动完成后,可通过http://ip地址:8380/xxl-job-admin/访问
账号:admin 密码:123456

看到页面后需进入容器

docker exec -it xxl-job-dataX /bin/bash    ---进入容器
apt-get update                             ---更新apt源
apt-get install -y python2                 ---安装python2
cd /project
java -jar -Duser.timezone=GMT+08   -Dspring.config.location=./application.properties application.jar  --启动默认执行器项目
  • 1
  • 2
  • 3
  • 4
  • 5

再次访问地址http://ip地址:8380/xxl-job-admin/访问
账号:admin 密码:123456
运行完可在执行器列表找到该执行器
在这里插入图片描述

到此部署完毕,只需把dataX的脚本放到/opt/docker/dataX-xxl-job/libs/datax/job 下
便可通过python2 /libs/datax/bin/datax.py /libs/datax/job/脚本名 进行同步!
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

本文内容由网友自发贡献,转载请注明出处:【wpsshop博客】
推荐阅读
相关标签
  

闽ICP备14008679号