热门标签
热门文章
- 1Mastering Delphi7 阅读笔记(摘要) 续_delphi7 ipersiststreaminit 引用
- 2gpu 纹理_在android gpu检查器中使用纹理计数器
- 3openvpn组网技术原理及配置过程（centos服务器/安卓客户端/linux客户端）_openvpl 多客户端
- 4ElementUI中Loading特效的使用_elementui dialog loading
- 5VSCode无法用ctrl+鼠标滚轮调整字体大小了
- 6python爬虫requests返回网页内容为空_python爬虫使用requests请求无法获取网页元素时终极解决方案...
- 7动态规划：矩阵连乘问题
- 8启动eclipse时候报错Error：Could not create the Java Virtual Machine. Error:A Fatal exception has occurred_error: could not create the java virtual machine.
- 9微信小程序自定义弹窗组件 action-sheet_微信插件 action
- 10如何分辨Mac设备X86或ARM_macos x86和arm
当前位置: article > 正文
oracle使用ogg导出数据到kafka_时空无限 csdn
作者：凡人多烦事01 | 2024-03-08 17:26:00
赞
踩
时空无限 csdn
前提
环境
oracle 19c 10.0.2.15
kafka kafka_2.13-2.8.0 二进制安装 10.0.2.12
- 1
- 2
安装包
oracle源端
191004_fbo_ggs_Linux_x64_shiphome.zip
- 1
kafka目标端
OGG_BigData_Linux_x64_19.1.0.0.5.tar.gz
kafka_2.13-2.8.0.tgz
- 1
- 2
oracle源端操作
安装ogg
创建ogg目录，解压安装包，root用户操作
mkdir -p /ogg
任意目录下解压，这里解压到了/home/oracle/目录
unzip 191004_fbo_ggs_Linux_x64_shiphome.zip -d /home/oracle/
- 1
- 2
- 3
授权root用户操作
chown -R oracle.oinstall /home/oracle/fbo_ggs_Linux_x64_shiphome/
- 1
编辑安装应答文件
cat /home/oracle/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp #################################################################### ## Copyright(c) Oracle Corporation 2019. All rights reserved. ## ## ## ## Specify values for the variables listed below to customize ## ## your installation. ## ## ## ## Each variable is associated with a comment. The comment ## ## can help to populate the variables with the appropriate ## ## values. ## ## ## ## IMPORTANT NOTE: This file should be secured to have read ## ## permission only by the oracle user or an administrator who ## ## own this installation to protect any sensitive input values. ## ## ## #################################################################### #------------------------------------------------------------------------------- # Do not change the following system generated value. #------------------------------------------------------------------------------- oracle.install.responseFileVersion=/oracle/install/rspfmt_ogginstall_response_schema_v19_1_0 ################################################################################ ## ## ## Oracle GoldenGate installation option and details ## ## ## ################################################################################ #------------------------------------------------------------------------------- # Specify the installation option. # Specify ORA19c for installing Oracle GoldenGate for Oracle Database 19c or # ORA18c for installing Oracle GoldenGate for Oracle Database 18c or # ORA12c for installing Oracle GoldenGate for Oracle Database 12c or # ORA11g for installing Oracle GoldenGate for Oracle Database 11g #------------------------------------------------------------------------------- INSTALL_OPTION=ORA19c #------------------------------------------------------------------------------- # Specify a location to install Oracle GoldenGate #------------------------------------------------------------------------------- SOFTWARE_LOCATION=/ogg #------------------------------------------------------------------------------- # Specify true to start the manager after installation. #------------------------------------------------------------------------------- START_MANAGER=true #------------------------------------------------------------------------------- # Specify a free port within the valid range for the manager process. # Required only if START_MANAGER is true. #------------------------------------------------------------------------------- MANAGER_PORT=7809 #------------------------------------------------------------------------------- # Specify the location of the Oracle Database. # Required only if START_MANAGER is true. #------------------------------------------------------------------------------- DATABASE_LOCATION=/opt/oracle/product/19.3.0/db_1 ################################################################################ ## ## ## Specify details to Create inventory for Oracle installs ## ## Required only for the first Oracle product install on a system. ## ## ## ################################################################################ #------------------------------------------------------------------------------- # Specify the location which holds the install inventory files. # This is an optional parameter if installing on # Windows based Operating System. #------------------------------------------------------------------------------- INVENTORY_LOCATION=/opt/oraInventory #------------------------------------------------------------------------------- # Unix group to be set for the inventory directory. # This parameter is not applicable if installing on # Windows based Operating System. #------------------------------------------------------------------------------- UNIX_GROUP_NAME=oinstall
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
开始安装
[oracle@localhost Disk1]$ pwd
/home/oracle/fbo_ggs_Linux_x64_shiphome/Disk1
[oracle@localhost Disk1]$ ./runInstaller -silent -responseFile /home/oracle/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp
- 1
- 2
- 3
- 4
配置环境变量oracle用户操作
[root@localhost oracle]# su - oracle
Last login: Mon May 24 16:10:01 CST 2021 on pts/1
[oracle@localhost ~]$ cat ~/.bash_profile
export OGG_HOME=/ogg
export PATH=$OGG_HOME:$PATH
export LD_LIBRARY_PATH=$OGG_HOME:$LD_LIBRARY_PATH
- 1
- 2
- 3
- 4
- 5
- 6
确认是否安装成功
切换到ogg安装目录
ggsci
- 1
oracle设置日志归档模式
切换到oracle用户并登录数据库
su - oracle
sqlplus / as sysdba
- 1
- 2
查看是否为归档模式
SQL> archive log list
Database log mode Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 11
Next log sequence to archive 13
Current log sequence 13
- 1
- 2
- 3
- 4
- 5
- 6
- 7
设置数据库归档模式
conn / as sysdba (以DBA身份连接数据库)
SQL> shutdown immediate (立即关闭数据库)
SQL> startup mount (启动实例并加载数据库，但不打开)
SQL> alter database archivelog; (更改数据库为归档模式)
SQL> alter database open; (打开数据库)
SQL> alter system archive log start; (启用自动归档)
- 1
- 2
- 3
- 4
- 5
- 6
确认
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 11
Next log sequence to archive 13
Current log sequence 13
- 1
- 2
- 3
- 4
- 5
- 6
- 7
打开日志开关
查看是否开启日志
SQL> select force_logging, supplemental_log_data_min from v$database;
FORCE_LOGGING
--------------------------------------------------------------------------------
SUPPLEMENTAL_LOG_DATA_MI
------------------------
NO
NO
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
开启相关日志
SQL> ALTER DATABASE FORCE LOGGING;
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
SQL> alter system switch logfile;
SQL> alter system set enable_goldengate_replication=true scope=both;
- 1
- 2
- 3
- 4
- 5
- 6
- 7
确认
SQL> select force_logging, supplemental_log_data_min from v$database;
FORCE_LOGGING
--------------------------------------------------------------------------------
SUPPLEMENTAL_LOG_DATA_MI
------------------------
YES
YES
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
oracle创建复制用户并授权
创建表空间目录并授权
mkdir -p /u01/app/oracle/oggdata/orcl
chown -R oracle:oinstall /u01/app/oracle/oggdata/orcl
- 1
- 2
创建复制用户并授权
SQL> create tablespace oggtbs datafile '/u01/app/oracle/oggdata/orcl/oggtbs01.dbf' size 1000M autoextend on;
SQL> create user ogg identified by ogg default tablespace oggtbs;
SQL> grant dba to ogg;
- 1
- 2
- 3
- 4
- 5
ogg初始化
[oracle@localhost ogg]$ cd /ogg/
[oracle@localhost ogg]$ ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
GGSCI (localhost.localdomain) 1> create subdirs
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
oracle创建测试表
SQL> create user test_ogg identified by test_ogg default tablespace users;
SQL> grant dba to test_ogg;
SQL> conn test_ogg/test_ogg;
SQL> create table test_ogg(id int ,name varchar(20),primary key(id));
- 1
- 2
- 3
- 4
kafka目标端操作
安装ogg客户端
创建目录
mkdir -p /opt/ogg
- 1
解压安装包到安装目录
gzip OGG_BigData_Linux_x64_19.1.0.0.5.tar.gz
tar zxvf OGG_BigData_Linux_x64_19.1.0.0.5.tar.gz -C /opt/ogg/
- 1
- 2
授权
useradd oracle
chown -R oracle.oracle /opt/ogg
- 1
- 2
配置环境变量
PATH=$PATH:$HOME/bin
export OGG_HOME=/opt/ogg
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64/jre
export LD_LIBRARY_PATH=/opt/ogg/dirprm:/opt/kafka/libs:$JAVA_HOME/lib/amd64:$JAVA_HOME/lib/amd64/server:$JAVA_HOME/lib/amd64/libjsig.so:$JAVA_HOME/lib/amd64/server/libjvm.so:$OGG_HOME/lib:$OGG_HOME/dirprm
export PATH=$JAVA_HOME/bin:$OGG_HOME:$PATH
export CLASSPATH=/opt/ogg/dirprm:/opt/kafka/libs:/opt/ogg/ggjava/ggjava.jar
- 1
- 2
- 3
- 4
- 5
- 6
使环境变量生效
source /etc/profile
- 1
测试下ogg
ggsci
- 1
初始化ogg
[root@localhost opt]# cd /opt/ogg/ [root@localhost ogg]# ggsci Oracle GoldenGate for Big Data Version 19.1.0.0.5 (Build 007) Oracle GoldenGate Command Interpreter Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141 Linux, x64, 64bit (optimized), Generic on Jun 28 2020 23:01:58 Operating system character set identified as UTF-8. Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved. GGSCI (localhost.localdomain) 1> create subdirs
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
OGG源端操作
配置OGG的全局变量
su oracle [oracle@localhost ogg]$ cd /ogg/ [oracle@localhost ogg]$ ggsci Oracle GoldenGate Command Interpreter for Oracle Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29 Operating system character set identified as UTF-8. Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved. 登录数据库 GGSCI (localhost.localdomain) 1> dblogin userid ogg password ogg Successfully logged into database. GGSCI (localhost.localdomain as ogg@orcl) 2> edit param ./globals oggschema ogg
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
配置mgr进程
GGSCI (localhost.localdomain as ogg@orcl) 3> edit param mgr
PORT 7809
DYNAMICPORTLIST 7810-7909
AUTORESTART EXTRACT *,RETRIES 5,WAITMINUTES 3
PURGEOLDEXTRACTS ./dirdat/*,usecheckpoints, minkeepdays 3
- 1
- 2
- 3
- 4
- 5
- 6
- 7
说明：
PORT 即mgr的默认监听端口；
DYNAMICPORTLIST 动态端口列表，当指定的mgr端口不可用时，会在这个端口列表中选择一个，最大指定范围为256个；
AUTORESTART 重启参数设置表示重启所有EXTRACT进程，最多5次，每次间隔3分钟；
PURGEOLDEXTRACTS 即TRAIL文件的定期清理
- 1
- 2
- 3
- 4
GGSCI (localhost.localdomain as ogg@orcl) 4> add trandata test_ogg.test_ogg 2021-05-24 16:46:11 INFO OGG-15131 Logging of supplemental redo log data is already enabled for table TEST_OGG.TEST_OGG. 2021-05-24 16:46:11 INFO OGG-15135 TRANDATA for instantiation CSN has been added on table TEST_OGG.TEST_OGG. 2021-05-24 16:46:11 INFO OGG-10471 ***** Oracle Goldengate support information on table TEST_OGG.TEST_OGG ***** Oracle Goldengate support native capture on table TEST_OGG.TEST_OGG. Oracle Goldengate marked following column as key columns on table TEST_OGG.TEST_OGG: ID. GGSCI (localhost.localdomain as ogg@orcl) 5> info trandata test_ogg.test_ogg 2021-05-24 16:46:18 INFO OGG-10471 ***** Oracle Goldengate support information on table TEST_OGG.TEST_OGG ***** Oracle Goldengate support native capture on table TEST_OGG.TEST_OGG. Oracle Goldengate marked following column as key columns on table TEST_OGG.TEST_OGG: ID. Logging of supplemental redo log data is enabled for table TEST_OGG.TEST_OGG. Columns supplementally logged for table TEST_OGG.TEST_OGG: "ID". Prepared CSN for table TEST_OGG.TEST_OGG: 3094843
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
配置extract进程
GGSCI (localhost.localdomain as ogg@orcl) 6> edit param extkafka
extract extkafka
dynamicresolution
SETENV (ORACLE_SID = "orcl")
SETENV (NLS_LANG = "american_america.AL32UTF8")
userid ogg,password ogg
exttrail /ogg/dirdat/to
table test_ogg.test_ogg;
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
说明：
第一行指定extract进程名称；
dynamicresolution 动态解析；
SETENV 设置环境变量，这里分别设置了Oracle数据库以及字符集；
userid ggs,password ggs 即OGG连接Oracle数据库的帐号密码，这里使用2.5中特意创建的复制帐号；
exttrail 定义trail文件的保存位置以及文件名，配置的是源端的目录，注意这里文件名只能是2个字母，其余部分OGG会补齐；
table 即复制表的表名，支持*通配，必须以;结尾
- 1
- 2
- 3
- 4
- 5
- 6
添加extract进程：
GGSCI (localhost.localdomain as ogg@orcl) 7> add extract extkafka,tranlog,begin now
# 或者指定捕获时间点
# GGSCI>ADD EXTRACT extkafka,TRANLOG,BEGIN 2022-03-25 17:00
# 可选：删除Extract组：
# GGSCI >DELETE EXTRACT extkafka
- 1
- 2
- 3
- 4
- 5
- 6
添加trail文件的定义与extract进程绑定
GGSCI (localhost.localdomain as ogg@orcl) 8> add exttrail /opt/ogg/dirdat/to,extract extkafka
# 可选，删除Trail文件：
# GGSCI >DELETE EXTTRAIL /opt/ogg/dirdat/to
# rm /opt/ogg/dirdat/to*
- 1
- 2
- 3
- 4
注意： /opt/ogg/dirdat/是源端的目录
配置pump进程
pump进程负责把trail文件传送到目标端
Pump进程也是一种Extract进程，只因配置不同而功能不同。
GGSCI (localhost.localdomain as ogg@orcl) 9> edit param pukafka
extract pukafka
passthru
dynamicresolution
userid ogg,password ogg
rmthost 10.0.2.12 mgrport 7809
rmttrail /opt/ogg/dirdat/to
table test_ogg.test_ogg;
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
说明：
第一行指定extract进程名称；
passthru 即禁止OGG与Oracle交互，我们这里使用pump逻辑传输，故禁止即可；
dynamicresolution 动态解析；
userid ogg,password ogg 即OGG连接Oracle数据库的帐号密码
rmthost和mgrport 即目标端(kafka)OGG的mgr服务的地址以及监听端口；
rmttrail 即目标端trail文件存储位置以及名称。
- 1
- 2
- 3
- 4
- 5
- 6
分别将本地trail文件和目标端的trail文件绑定到extract进程：
GGSCI (localhost.localdomain as ogg@orcl) 10> add extract pukafka,exttrailsource /ogg/dirdat/to
# EXTTRAILSOURCE指前面extkafka的Trail文件
# 可选：删除Extract组：
# GGSCI >DELETE EXTRACT pukafka
- 1
- 2
- 3
- 4
Pump从本地Trail文件提取的变更，保存到远程Trail文件路径的位置，一个本地（源端）Trail文件可以分成多个远程Trail文件，在远程用多个Replicat同步，以增大速度。
GGSCI (localhost.localdomain as ogg@orcl) 11> add rmttrail /opt/ogg/dirdat/to,extract pukafka
# 可选，删除Data Pump组：DELETE EXTRACT pump201
# 删除远程Trail文件：
# DELETE RMTTRAIL /opt/ogg/dirdat/to
# rm /opt/ogg/dirdat/to*
- 1
- 2
- 3
- 4
- 5
注意：第一条命令exttrailsource后是源端的目录
第二条命令rmttrail后是目标端的目录
配置define文件
GGSCI (localhost.localdomain as ogg@orcl) 12> edit param test_ogg
defsfile /ogg/dirdef/test_ogg.test_ogg
userid ogg,password ogg
table test_ogg.test_ogg;
- 1
- 2
- 3
- 4
- 5
- 6
注意：/ogg/dirdef/为源端目录
table后是表名，oracle表名格式为 用户名.表名
[oracle@localhost ogg]$ cd /ogg/
[oracle@localhost ogg]$ ./defgen paramfile dirprm/test_ogg.prm
- 1
- 2
将生成的/opt/ogg/dirdef/test_ogg.test_ogg发送的目标端ogg目录下的dirdef里：
[oracle@localhost ogg]$ scp -r /ogg/dirdef/test_ogg.test_ogg root@10.0.2.12:/opt/ogg/dirdef/
- 1
OGG目标端配置
前提kafka服务开启
配置管理器mgr
[root@localhost ogg]# cd /opt/ogg/ [root@localhost ogg]# ggsci Oracle GoldenGate for Big Data Version 19.1.0.0.5 (Build 007) Oracle GoldenGate Command Interpreter Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141 Linux, x64, 64bit (optimized), Generic on Jun 28 2020 23:01:58 Operating system character set identified as UTF-8. Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved. GGSCI (localhost.localdomain) 1> edit param mgr PORT 7809 DYNAMICPORTLIST 7810-7909 AUTORESTART EXTRACT *,RETRIES 5,WAITMINUTES 3 PURGEOLDEXTRACTS ./dirdat/*,usecheckpoints, minkeepdays 3
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
配置checkpoint
checkpoint即复制可追溯的一个偏移量记录，在全局配置里添加checkpoint表即可。
GGSCI (localhost.localdomain) 2> edit param ./GLOBALS
CHECKPOINTTABLE test_ogg.checkpoint
- 1
- 2
- 3
- 4
配置replicate进程
GGSCI (localhost.localdomain) 3> edit param rekafka
REPLICAT rekafka
sourcedefs /opt/ogg/dirdef/test_ogg.test_ogg
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP test_ogg.test_ogg, TARGET test_ogg.test_ogg;
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
说明：
REPLICATE rekafka 定义 rep进程名称；
sourcedefs 即在4.6中在源服务器上做的表映射文件（如果该文件已经复制到了目标端，这里就是指目标端的路径及其文件）；
TARGETDB LIBFILE 即定义kafka一些适配性的库文件以及配置文件，配置文件位于OGG主目录下的dirprm/kafka.props；
REPORTCOUNT 即复制任务的报告生成频率；
GROUPTRANSOPS 为以事务传输时，事务合并的单位，减少IO操作；
MAP 即源端与目标端的映射关系
- 1
- 2
- 3
- 4
- 5
- 6
配置kafka.props
cd /opt/ogg/dirprm/
vim kafka.props
- 1
- 2
[root@localhost ogg]# cd /opt/ogg/dirprm/
[root@localhost dirprm]# cat kafka.props
gg.handlerlist=kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
gg.handler.kafkahandler.topicMappingTemplate=test_ogg
gg.handler.kafkahandler.format=json
gg.handler.kafkahandler.mode=op
gg.classpath=dirprm/:/opt/kafka_2.13-2.8.0/libs/*:/opt/ogg/:/opt/ogg/lib/*
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
[root@localhost ogg]# cd /opt/ogg/dirprm/
[root@localhost dirprm]# cat custom_kafka_producer.properties
bootstrap.servers=10.0.2.12:9092
acks=1
compression.type=gzip
reconnect.backoff.ms=1000
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
batch.size=102400
linger.ms=10000
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
其中需要将后面的注释去掉，ogg不识别注释，如果不去掉会报错
添加trail文件到replicate进程
[root@localhost dirprm]# cd /opt/ogg/ [root@localhost ogg]# ggsci Oracle GoldenGate for Big Data Version 19.1.0.0.5 (Build 007) Oracle GoldenGate Command Interpreter Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141 Linux, x64, 64bit (optimized), Generic on Jun 28 2020 23:01:58 Operating system character set identified as UTF-8. Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved. GGSCI (localhost.localdomain) 1> add replicat rekafka exttrail /opt/ogg/dirdat/to,checkpointtable test_ogg.checkpoint
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
说明：add replicat rekafka exttrail 后的目录为目标端的目录
测试
源端操作
启动进程
[oracle@localhost ogg]$ cd /ogg/
[oracle@localhost ogg]$ ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
GGSCI (localhost.localdomain) 1> start mgr
GGSCI (localhost.localdomain) 2> start extkafka
GGSCI (localhost.localdomain) 3> start pukafka
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
目标端操作
启动进程
[root@localhost ogg]# cd /opt/ogg/ [root@localhost ogg]# ggsci Oracle GoldenGate for Big Data Version 19.1.0.0.5 (Build 007) Oracle GoldenGate Command Interpreter Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141 Linux, x64, 64bit (optimized), Generic on Jun 28 2020 23:01:58 Operating system character set identified as UTF-8. Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved. GGSCI (localhost.localdomain) 1> start mgr GGSCI (localhost.localdomain) 2> start rekafka
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
查看源端进程是否启动成功
GGSCI (localhost.localdomain) 4> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING EXTKAFKA 00:00:00 00:00:09
EXTRACT RUNNING PUKAFKA 00:00:00 00:00:08
- 1
- 2
- 3
- 4
- 5
- 6
- 7
查看目标端进程是否启动成功
GGSCI (localhost.localdomain) 3> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING REKAFKA 00:00:00 00:00:03
- 1
- 2
- 3
- 4
- 5
- 6
排查错误方法
查看进程日志
登录ogg命令行以下执行命令
view report 进程名
GGSCI (localhost.localdomain) 4> view report rekafka *********************************************************************** Oracle GoldenGate for Big Data Version 19.1.0.0.5 (Build 007) Oracle GoldenGate Delivery Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141 Linux, x64, 64bit (optimized), Generic on Jun 28 2020 23:07:55 Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved. Starting at 2021-05-24 15:48:53 *********************************************************************** Operating System Version: Linux Version #1 SMP Fri Mar 6 11:36:42 UTC 2015, Release 3.10.0-229.el7.x86_64 Node: localhost.localdomain Machine: x86_64 soft limit hard limit Address Space Size : unlimited unlimited Heap Size : unlimited unlimited File Size : unlimited unlimited CPU Time : unlimited unlimited Process id: 3161 Description: *********************************************************************** ** Running with the following parameters ** *********************************************************************** 2021-05-24 15:48:53 INFO OGG-03059 Operating system character set identified as UTF-8. 2021-05-24 15:48:53 INFO OGG-02695 ANSI SQL parameter syntax is used for parameter parsing. 2021-05-24 15:48:53 INFO OGG-03528 The source database character set, as determined from the table definition file, is UTF-8.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
登录oracle数据库
[oracle@localhost ogg]$ sqlplus / as sysdba
- 1
更新数据
conn test_ogg/test_ogg
insert into test_ogg values(1,'test');
commit;
update test_ogg set name='zhangsan' where id=1;
commit;
delete test_ogg where id=1;
commit;
- 1
- 2
- 3
- 4
- 5
- 6
- 7
查看源端trail文件状态
[oracle@localhost ogg]$ ll /ogg/dirdat/
total 8
-rw-r----- 1 oracle oinstall 1285 May 24 15:57 to000000000
- 1
- 2
- 3
查看目标端trail文件状态
[root@localhost ogg]# ll /opt/ogg/dirdat/
total 4
-rw-r-----. 1 oracle oracle 2498 May 24 17:09 to000000000
- 1
- 2
- 3
查看kafka是否自动建立对应的主题
[root@localhost kafka_2.13-2.8.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
__consumer_offsets
test_ogg
- 1
- 2
- 3
- 4
通过消费者看是否有同步消息
[root@localhost kafka_2.13-2.8.0]# bin/kafka-console-consumer.sh --bootstrap-server 10.0.2.12:9092 --topic test_ogg --from-beginning
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
{"table":"TEST_OGG.TEST_OGG","op_type":"I","op_ts":"2021-05-24 15:59:41.000814","current_ts":"2021-05-24T15:59:48.395000","pos":"00000000000000001827","after":{"ID":2,"NAME":"test"}}
{"table":"TEST_OGG.TEST_OGG","op_type":"I","op_ts":"2021-05-24 15:59:49.001165","current_ts":"2021-05-24T15:59:55.782000","pos":"00000000000000001958","after":{"ID":3,"NAME":"test"}}
{"table":"TEST_OGG.TEST_OGG","op_type":"I","op_ts":"2021-05-24 15:59:53.002124","current_ts":"2021-05-24T15:59:57.815000","pos":"00000000000000002085","after":{"ID":4,"NAME":"test"}}
{"table":"TEST_OGG.TEST_OGG","op_type":"U","op_ts":"2021-05-24 17:09:39.002891","current_ts":"2021-05-24T17:09:46.082000","pos":"00000000000000002211","before":{"ID":1,"NAME":"test"},"after":{"ID":1,"NAME":"zhangsan"}}
{"table":"TEST_OGG.TEST_OGG","op_type":"D","op_ts":"2021-05-24 17:09:39.002891","current_ts":"2021-05-24T17:09:46.091000","pos":"00000000000000002366","before":{"ID":1,"NAME":"zhangsan"}}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
操作命令
#停止服务
GGSCI >stop extract ex201
- 1
#查看状态：
GGSCI >info ex201
GGSCI >stats ex201
GGSCI >status ex201
- 1
- 2
- 3
#如果成功运行，会在本地Trail路径下应该生成文件，如果出错，查看错误原因：
GGSCI >view report ex201
- 1
注意
如果是大数据量抽取，需要调整几个参数。
分别在kafka端和ogg目标端调整
kafka的配置文件
server.properties
- 1
socket.send.buffer.bytes=5024000
socket.receive.buffer.bytes=5024000
- 1
- 2
ogg目标端参数
custom_kafka_producer.properties
- 1
max.request.size=5024000
send.buffer.bytes=5024000
- 1
- 2
修改完重启kafka和目标端ogg进程。
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/凡人多烦事01/article/detail/211379
推荐阅读
相关标签
