devbox
笔触狂放9
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

DataX-数据迁移Oracle到Mysql-ETL工具_datax实现orcel迁移mysql

作者：笔触狂放9 | 2024-04-17 22:11:01

赞

踩

datax实现orcel迁移mysql

一、安装

https://github.com/WeiYe-Jing/datax-web/blob/master/doc/datax-web/datax-web-deploy.md

  • 1、直接下载DataX工具包：DataX下载地址

    下载后解压至本地某个目录，进入bin目录，即可运行同步作业：

    $ cd  {YOUR_DATAX_HOME}/bin
$ python datax.py {YOUR_JOB.json}

    自检脚本：    python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json

  • 2、Python (2.x) (支持Python3需要修改替换datax/bin下面的三个python文件，替换文件在doc/datax-web/datax-python3下) 必选，主要用于调度执行底层DataX的启动脚本，默认的方式是以Java子进程方式执行DataX，用户可以选择以Python方式来做自定义的改造

二、测试脚本

  1. C:\DATAX\datax\bin>CHCP 65001
  2. Active code page: 65001
  3.  
  4. C:\DATAX\datax\bin>python datax.py C:\DATAX\datax\job\test.json
  5.  
  6. DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
  7. Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
  8.  
  9.  
  10. 2024-03-14 15:02:01.361 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
  11. 2024-03-14 15:02:01.363 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
  12. 2024-03-14 15:02:01.375 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
  13. 2024-03-14 15:02:01.381 [main] INFO  Engine - the machine info  =>
  14.  
  15.         osInfo: Windows 10 amd64 10.0
  16.         jvmInfo:        Oracle Corporation 1.8 25.341-b10
  17.         cpu num:        6
  18.  
  19.         totalPhysicalMemory:    -0.00G
  20.         freePhysicalMemory:     -0.00G
  21.         maxFileDescriptorCount: -1
  22.         currentOpenFileDescriptorCount: -1
  23.  
  24.         GC Names        [PS MarkSweep, PS Scavenge]
  25.  
  26.         MEMORY_NAME                    | allocation_size                | init_size
  27.         PS Eden Space                  | 256.00MB                       | 256.00MB
  28.         Code Cache                     | 240.00MB                       | 2.44MB
  29.         Compressed Class Space         | 1,024.00MB                     | 0.00MB
  30.         PS Survivor Space              | 42.50MB                        | 42.50MB
  31.         PS Old Gen                     | 683.00MB                       | 683.00MB
  32.         Metaspace                      | -0.00MB                        | 0.00MB
  33.  
  34.  
  35. 2024-03-14 15:02:01.392 [main] INFO  Engine -
  36. {
  37.         "setting":{
  38.                 "speed":{
  39.                         "channel":4
  40.                 }
  41.         },
  42.         "content":[
  43.                 {
  44.                         "reader":{
  45.                                 "name":"oraclereader",
  46.                                 "parameter":{
  47.                                         "username":"test",
  48.                                         "password":"******",
  49.                                         "column":[
  50.                                                 "*"
  51.                                         ],
  52.                                         "connection":[
  53.                                                 {
  54.                                                         "table":[
  55.                                                                 "CAR.S_U"
  56.                                                         ],
  57.                                                         "jdbcUrl":[
  58.                                                                 "jdbc:oracle:thin:@//10.33.51.231:1521/helowin"
  59.                                                         ]
  60.                                                 }
  61.                                         ]
  62.                                 }
  63.                         },
  64.                         "writer":{
  65.                                 "name":"mysqlwriter",
  66.                                 "parameter":{
  67.                                         "username":"car",
  68.                                         "password":"******",
  69.                                         "writeMode":"replace",
  70.                                         "column":[
  71.                                                 "*"
  72.                                         ],
  73.                                         "connection":[
  74.                                                 {
  75.                                                         "jdbcUrl":"jdbc:mysql://10.33.51.231:3306/car",
  76.                                                         "table":[
  77.                                                                 "S_U"
  78.                                                         ]
  79.                                                 }
  80.                                         ]
  81.                                 }
  82.                         }
  83.                 }
  84.         ]
  85. }
  86.  
  87. 2024-03-14 15:02:01.417 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false
  88. 2024-03-14 15:02:01.418 [main] INFO  JobContainer - DataX jobContainer starts job.
  89. 2024-03-14 15:02:01.420 [main] INFO  JobContainer - Set jobId = 0
  90. 2024-03-14 15:02:01.623 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:oracle:thin:@//10.33.51.231:1521/helowin.
  91. 2024-03-14 15:02:01.624 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置 读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
  92. Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
  93. 2024-03-14 15:02:07.748 [job-0] INFO  OriginalConfPretreatmentUtil - table:[S_BU] all columns:[
  94. M_ROW$$,ANNLRVW_ENDT_OFFST,ANNLRVW_STDT_OFFST,BU_FLG,CONFLICT_ID,CREATED,CREATED_BY,CURR_PRD_OBJ_NAME,CURR_PRD_RVW_NAME,DYN_HRCHY_ID,LAST_UPD,LAST_UPD_BY,MODIFICATION_NUM,NAME,PAR_ROW_ID,ROW_ID,X_LANG_ID
  95. ].
  96. 2024-03-14 15:02:07.748 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写 入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
  97. 2024-03-14 15:02:07.752 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
  98. replace INTO %s (M_ROW$$,ANNLRVW_ENDT_OFFST,ANNLRVW_STDT_OFFST,BU_FLG,CONFLICT_ID,CREATED,CREATED_BY,CURR_PRD_OBJ_NAME,CURR_PRD_RVW_NAME,DYN_HRCHY_ID,LAST_UPD,LAST_UPD_BY,MODIFICATION_NUM,NAME,PAR_ROW_ID,ROW_ID,X_LANG_ID) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
  99. ], which jdbcUrl like:[jdbc:mysql://10.33.51.231:3306/carmen?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&rewriteBatchedStatements=true&tinyInt1isBit=false]
  100. 2024-03-14 15:02:07.753 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
  101. 2024-03-14 15:02:07.753 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do prepare work .
  102. 2024-03-14 15:02:07.754 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
  103. 2024-03-14 15:02:07.755 [job-0] INFO  JobContainer - jobContainer starts to do split ...
  104. 2024-03-14 15:02:07.755 [job-0] INFO  JobContainer - Job set Channel-Number to 4 channels.
  105. 2024-03-14 15:02:07.757 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
  106. 2024-03-14 15:02:07.758 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
  107. 2024-03-14 15:02:07.778 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
  108. 2024-03-14 15:02:07.779 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
  109. 2024-03-14 15:02:07.781 [job-0] INFO  JobContainer - Running by standalone Mode.
  110. 2024-03-14 15:02:07.786 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
  111. 2024-03-14 15:02:07.790 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
  112. 2024-03-14 15:02:07.790 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
  113. 2024-03-14 15:02:07.796 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
  114. 2024-03-14 15:02:07.798 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from CARMEN.S_BU
  115. ] jdbcUrl:[jdbc:oracle:thin:@//10.33.51.231:1521/helowin].
  116. 2024-03-14 15:02:16.306 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from CARMEN.S_BU
  117. ] jdbcUrl:[jdbc:oracle:thin:@//10.33.51.231:1521/helowin].
  118. 2024-03-14 15:02:16.845 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[9050]ms
  119. 2024-03-14 15:02:16.847 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
  120. 2024-03-14 15:02:17.798 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 14644530 bytes | Speed 1.40MB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 6.239s |  All Task WaitReaderTime 1.988s | Percentage 100.00%
  121. 2024-03-14 15:02:17.800 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
  122. 2024-03-14 15:02:17.805 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
  123. 2024-03-14 15:02:17.805 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do post work.
  124. 2024-03-14 15:02:17.806 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
  125. 2024-03-14 15:02:17.807 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: C:\DATAX\datax\hook
  126. 2024-03-14 15:02:17.808 [job-0] INFO  JobContainer -
  127.          [total cpu info] =>
  128.                 averageCpu                     | maxDeltaCpu                    | minDeltaCpu
  129.                 -1.00%                         | -1.00%                         | -1.00%
  132.          [total gc info] =>
  133.                  NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime
  134.                  PS MarkSweep         | 1                  | 1                  | 1                  | 0.018s             | 0.018s             | 0.018s
  135.                  PS Scavenge          | 5                  | 5                  | 5                  | 0.028s             | 0.028s             | 0.028s
  137. 2024-03-14 15:02:17.808 [job-0] INFO  JobContainer - PerfTrace not enable!
  138. 2024-03-14 15:02:17.808 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 14644530 bytes | Speed 1.40MB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 6.239s |  All Task WaitReaderTime 1.988s | Percentage 100.00%
  139. 2024-03-14 15:02:17.810 [job-0] INFO  JobContainer -
  140. 任务启动时刻                    : 2024-03-14 15:02:01
  141. 任务结束时刻                    : 2024-03-14 15:02:17
  142. 任务总计耗时                    :                 16s
  143. 任务平均流量                    :            1.40MB/s
  144. 记录写入速度                    :          10000rec/s
  145. 读出记录总数                    :              100000
  146. 读写失败总数                    :                   0
  149. C:\DATAX\datax\bin>

三、datax-web: DataX管理调度GitHub - WeiYe-Jing/datax-web: DataX集成可视化页面，选择数据源即可一键生成数据同步任务，支持RDBMS、Hive、HBase、ClickHouse、MongoDB等数据源，批量创建RDBMS数据同步任务，集成开源调度系统，支持分布式、增量同步数据、实时查看运行日志、监控执行器资源、KILL运行进程、数据源信息加密等。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/笔触狂放9/article/detail/442514
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号