暂无图片
暂无图片
4
暂无图片
暂无图片
暂无图片

张小白教你使用OGG实现Oracle 19C到MySQL 5.7的数据同步

原创 张辉 2022-05-30
4779

(零)简述

OGG是Oracle GoldenGate的简称,它可以完成Oracle到异构数据库的实时数据同步。


使用OGG完成Oracle到MySQL的同步,其原理如下所示:



故使用OGG完成Oracle到MySQL的数据同步需要以下几个步骤:

(一)在Oracle端安装OGG for Oracle

(二)源端OGG配置

(1)OGG源端相关的Oracle配置(创建c##ogg用户、建立测试表test_ogg等等)

(2)OGG源端表级补全日志trandata配置

(3)OGG源端管理进程MGR配置(侦听本地端口7809)

(4)OGG源端抽取进程extorcl配置(实现从Oracle数据库表抽取数据到数据文件)

(5)OGG源端传输进程pumysql配置(实现将抽取的数据文件发送到OGG目的端)

(6)OGG源端异构mapping文件defgen生成

(三)在MySQL端安装OGG for MySQL

(四)目的端OGG配置

(1)OGG目的端的MySQL配置

(2)OGG目的端管理进程MGR配置(侦听本地端口7810)

(3)OGG目的端检查点日志表checkpoint配置

(4)OGG目的端回放线程r_tab1配置(实现将抽取的数据文件写到MySQL数据库表)


(一)在Oracle端安装OGG for Oracle

打开 https://www.oracle.com/middleware/technologies/goldengate-downloads.html

搜索 Oracle GoldenGate 21.3.0.0.0 for Oracle on Linux x86-64

下载OGG for Oracle安装包 213000_fbo_ggs_Linux_x64_Oracle_shiphome.zip

以Oracle用户登录:

修改.bashrc,增加以下内容

export JAVA_HOME=$HOME/jdk1.8.0_141
export PATH=$JAVA_HOME/bin:$PATH
export TNS_ADMIN=$ORACLE_HOME/network/admin
export OGG_HOME=/opt/ogg/source_endpoint
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64:$JAVA_HOME/jre/lib/amd64/server:$JAVA_HOME/jre/lib/amd64/libjsig.so:$JAVA_HOME/jre/lib/amd64/server/libjvm.so:$OGG_HOME/lib:$LD_LIBRARY_PATH:$ORACLE_HOME/lib:/usr/lib
export PATH=$OGG_HOME:$PATH:$ORACLE_HOME/bin


切换到root用户

su - root

useradd ogg -g oinstall

passwd ogg

mkdir -p /opt/ogg

chown -R ogg /opt/ogg

mkdir -p /home/oracle/oradata/OGG/ORCLCDB

mkdir -p /home/oracle/oradata/OGG/tablespace

chown -R oracle:oinstall /home/oracle/oradata/OGG

切换到ogg用户:

将ogg安装包 213000_fbo_ggs_Linux_x64_Oracle_shiphome.zip 传到/home/ogg下:

unzip -d ~ 213000_fbo_ggs_Linux_x64_Oracle_shiphome.zip

cd fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/response/

编辑 oggcore.rsp

更新以下行:

INSTALL_OPTION=ORA19C
SOFTWARE_LOCATION=/opt/ogg/source_endpoint

cd ..

./runInstaller -silent -nowait -responseFile /home/ogg/fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/response/oggcore.rsp

报 /u01/app/oracle/oraInventory 没有权限写入。。。查了一下,确实没有,那就加上权限。

重来:

切换到ogg用户:

cd fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/

./runInstaller -silent -nowait -responseFile /home/ogg/fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/response/oggcore.rsp


(二)源端OGG配置

编辑~/.bashrc

增加以下内容:

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/19c/db_1
export ORACLE_SID=zhanghui
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

export OGG_HOME=/opt/ogg/source_endpoint
export PATH=$OGG_HOME:$PATH:$ORACLE_HOME/bin
export TNS_ADMIN=$ORACLE_HOME/network/admin

source ~/.bashrc使其生效:

建立软连接:

ln -s $ORACLE_HOME/lib/libnnz19.so /opt/ogg/source_endpoint/libnnz19.so

cd $OGG_HOME

ldd ggsci


(1)OGG源端相关的Oracle配置(创建c##ogg用户、建立测试表test_ogg等等)

切换到oracle用户

su - oracle

mkdir /home/oracle/archivelog

查看目前是否打开了归档日志:

sqlplus '/as sysdba'

archive log list

没打开。

那么就打开它。注意如果打开了归档日志,那么所有的DML操作都会记录到归档日志中,这个文件可能会非常大,直到让文件系统爆满,这会直接导致oracle无法提供服务。所以要务必小心跟踪归档日志所属文件系统的剩余空间情况。

alter system set log_archive_dest_1='location=/home/oracle/archivelog';

shutdown immediate;

startup mount;

alter database archivelog;

alter database open;

alter system archive log start;

alter system archive log current;

alter database force logging;

alter database add supplemental log data;

alter system set enable_goldengate_replication=true;

select name,open_mode,force_logging,supplemental_log_data_min from v$database;

archive log list

创建c##ogg所需要的表空间:

create tablespace oggtbs datafile '/home/oracle/oradata/OGG/tablespace/oggtbs01.dbf' size 1000M autoextend on;

并在上面创建c##ogg用户:

create user c##ogg identified by ogg default tablespace oggtbs;

赋予权限:

grant dba ,connect, resource, unlimited tablespace to c##ogg;

exec dbms_goldengate_auth.grant_admin_privilege('c##ogg','*',TRUE);

select username,common,con_id from cdb_users where username like '%OGG%';

commit;


查看pdb状态:show pdbs

如果发现pdb是MOUNTED状态,可以执行:

alter session set container=ORCLPDB;

alter pluggable database open;

将其改为READ/WRITE状态:

alter session set container=ORCLPDB;

show con_name;

确认现在是在ORCLPDB库中,

给c##ogg用户赋权限:

grant dba ,connect, resource, unlimited tablespace to c##ogg;


编辑$TNS_ADMIN下的 tnsnames.ora文件:

LISTENER_ZHANGHUI =
  (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))


ZHANGHUI =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = zhanghui)
    )
  )

ORCLPDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ORCLPDB )
    )
  )


增加ORCLPDB这个PDB数据库的配置。


创建ogg所需要的表空间:

create tablespace mytbs datafile '/home/oracle/oradata/OGG/tablespace/mytbs01.dbf' size 500M autoextend on;

并在上面创建ogg用户:

create user ogg identified by ogg default tablespace mytbs;

赋予权限:

grant dba ,connect, resource, unlimited tablespace to ogg;

exec dbms_goldengate_auth.grant_admin_privilege('ogg','*',TRUE);

select username,common,con_id from cdb_users where username like '%OGG%';

commit;

验证ogg可以正常登录ORCLPDB数据库:


切换到ogg用户:

cd $OGG_HOME

ggsci

create subdirs


切换到oracle用户,创建一个test_ogg用户:

create user test_ogg identified by ogg ;

grant dba ,connect, resource, unlimited tablespace to test_ogg;(实际上用户的权限不必这么大)

quit

切换到test_ogg用户登录ORCLPDB

sqlplus test_ogg/ogg@ORCLPDB

create table test_ogg(id int,name varchar(20),primary key(id));


(2)OGG源端表级补全日志trandata配置

切换到ogg用户:


cd $OGG_HOME

ggsci

edit param ./globals

填入以下内容:

oggschema ogg

dblogin userid ogg@ORCLPDB password ogg

add schematrandata ORCLPDB.test_ogg

add trandata test_ogg.*

capture tabledef test_ogg.*


(3)OGG源端管理进程MGR配置(侦听本地端口7809)

edit param mgr

编辑以下内容:

PORT 7809

保存退出


(4)OGG源端抽取进程extorcl配置(实现从Oracle数据库表抽取数据到数据文件)

edit param extorcl

extract extorcl

DDL INCLUDE ALL

SETENV (ORACLE_HOME = "/u01/app/oracle/product/19c/db_1")
SETENV (ORACLE_SID="zhanghui")
SETENV (NLS_LANG="american_america.AL32UTF8")
userid c##ogg@zhanghui , password ogg
exttrail /opt/ogg/source_endpoint/dirdat/to
table ORCLPDB.test_ogg.*;


注意:

1.如果使用ORCLPDB中的ogg用户,在启动extorcl的时候就会报以下错误:

所以必须配置使用c##ogg用户登录才行。

2.如果table忘记带了数据库名,写成这样:table test_ogg.*;,启动extorcl的时候就会报以下错误:


delete extract extorcl

add extract extorcl,INTEGRATED TRANLOG,BEGIN NOW

add exttrail /opt/ogg/source_endpoint/dirdat/to, extract extorcl

dblogin userid ogg@ORCLPDB, password ogg

register extract extorcl database container(ORCLPDB)

很奇怪,它不能在PDB里面注册,那我们切换都CDB环境试试:

dblogin userid c##ogg@zhanghui, password ogg

register extract extorcl database container(ORCLPDB)


(5)OGG源端传输进程pumysql配置(实现将抽取的数据文件发送到OGG目的端)

edit param pumysql

加入以下内容:

extract pumysql
userid c##ogg@zhanghui , password ogg
rmthost 127.0.0.1,mgrport 7810
rmttrail /opt/ogg/target_endpoint/dirdat/rt
table ORCLPDB.test_ogg.*;

注意:如果使用ORCLPDB中的ogg用户,在启动pumysql的时候就会报以下错误:

所以必须配置使用c##ogg用户登录才行。

delete extract pumysql

add extract pumysql, exttrailsource /opt/ogg/source_endpoint/dirdat/to

add rmttrail /opt/ogg/target_endpoint/dirdat/rt, extract pumysql, megabytes 100

info all

start mgr

start extorcl

start pumysql


(6)OGG源端异构mapping文件defgen生成

edit param tab1

dsdffile /opt/ogg/source_endpoint/dirdef/ogg_test.ogg_test
userid test_ogg@ORCLPDB, password ogg
table test_ogg.test_ogg;

quit

./defgen paramfile dirprm/tab1.prm


info all

插入一条test_ogg表的记录:

insert into test_ogg values(1,'Hello World');

commit;

在ggsci中查看extorcl的日志:

view report extorcl

在Run Time Messages后面就是从oracle抽取记录的日志。

在ggsci中查看pumysql的日志:

在Run Time Messages后面就是将上面抽取的表记录发送到mysql服务端MGR的日志。


(三)在MySQL端安装OGG for MySQL

卸载CentOS自带的mariadb:

rpm -qa | grep mariadb

rpm -e --nodeps mariadb-server-5.5.68-1.el7.x86_64 mariadb-5.5.68-1.el7.x86_64 mariadb-libs-5.5.68-1.el7.x86_64

安装MySQL:

cd /usr/local

wget https://cdn.mysql.com/Downloads/MySQL-5.7/mysql-5.7.37-el7-x86_64.tar.gz

解压:

tar -zxvf mysql-*.gz

mv mysql-5.7.37-el7-x86_64 mysql

设置权限:

chown -R mysql:mysql /usr/local/mysql

chmod -R 755 /usr/local/mysql

mkdir -p /data/mysql/data

mkdir -p /data/mysql/logs

mkdir -p /data/mysql/tmp

chown -R mysql:mysql /data/mysql

编辑 /etc/my.cnf 文件:

内容如下:

[client]
port = 3306
socket = /data/mysql/tmp/mysql.sock
[mysqld]
user = mysql
basedir = /usr/local/mysql 
datadir = /data/mysql/data 
port = 3306 
socket = /data/mysql/tmp/mysql.sock
pid-file = /data/mysql/tmp/mysqld.pid
tmpdir = /data/mysql/tmp 
skip_name_resolve = 1
symbolic-links=0
max_connections = 2000
group_concat_max_len = 1024000
sql_mode = NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
lower_case_table_names = 1
log_timestamps=SYSTEM
character-set-server = utf8
interactive_timeout = 1800 
wait_timeout = 1800
max_allowed_packet = 32M
binlog_cache_size = 4M
sort_buffer_size = 2M
read_buffer_size = 4M
join_buffer_size = 4M
tmp_table_size = 96M
max_heap_table_size = 96M
max_length_for_sort_data = 8096
#logs
server-id = 1003306
log-error = /data/mysql/logs/error.log
slow_query_log = 1
slow_query_log_file = /data/mysql/logs/slow.log
long_query_time = 3
log-bin = /data/mysql/logs/binlog
binlog_format = row
expire_logs_days = 15
log_bin_trust_function_creators = 1
relay-log = /data/mysql/logs/relay-bin
relay-log-recovery = 1 
relay_log_purge = 1 
#innodb 
innodb_file_per_table = 1
innodb_log_buffer_size = 16M
innodb_log_file_size = 256M
innodb_log_files_in_group = 2
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
innodb_flush_neighbors = 0
innodb_flush_method = O_DIRECT
innodb_autoinc_lock_mode = 2
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_buffer_pool_size = 2G

将mysql设置为linux服务:

cd /usr/local/mysql/support-files

cp mysql.server /etc/init.d/mysql

编辑 /etc/init.d/mysql 文件:

设置以下两行:

basedir=/usr/local/mysql
datadir=/data/mysql/data

chkconfig --add mysql

chkconfig --list

编辑环境变量:

echo "PATH=$PATH:/usr/local/mysql/bin " >> /etc/profile

source /etc/profile

rpm -qa|grep libaio

如果没有,则需要安装:yum install libaio-devel.x86_64

/usr/local/mysql/bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/data/mysql/data

more /data/mysql/logs/error.log | grep password

重启MySQL:

service stop mysql

service start mysql

得使用

systemctl restart mysql

systemctl status mysql

登录:

mysql -u root -p

输入前面的密码:BOc:CUr?S0aj

修改root密码:

alter user 'root'@'localhost' identified by 'zhanghui';

flush privileges;

quit退出命令行。

ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql

使用新密码zhanghui登录:

mysql -u root -p

create user 'ogg'@'%' identified by 'ogg';

grant all on *.* to 'ogg'@'%';


打开 https://www.oracle.com/middleware/technologies/goldengate-downloads.html

找到 Oracle GoldenGate 21.3.0.0.0 for MySQL-compatible Databases on Linux x86-64 下载:

使用root登录

创建目的端ogg用户:ogg2

useradd ogg2

passwd ogg2

su - ogg2

切换到ogg2用户:su - ogg2

将ogg for mysql安装包 传到/home/ogg2下:

unzip -d ~ ./213000_ggs_Linux_x64_MySQL_64bit.zip

切换到root用户

mkdir /opt/ogg/target_endpoint

chown -R ogg2 /opt/ogg/target_endpoint

切换回ogg2用户:

tar -xvf ggs_Linux_x64_MySQL_64bit.tar -C /opt/ogg/target_endpoint

编辑 ~/.bashrc文件:

export JAVA_HOME=$HOME/jdk1.8.0_141
export PATH=$JAVA_HOME/bin:$PATH

export OGG_HOME=/opt/ogg/target_endpoint
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64:$JAVA_HOME/jre/lib/amd64/server:$JAVA_HOME/jre/lib/amd64/libjsig.so:$JAVA_HOME/jre/lib/amd64/server/libjvm.so:$OGG_HOME/lib:$HOME/libs

export PATH=$OGG_HOME:$PATH

source ~/.bashrc使其生效。


(四)目的端OGG配置


(1)OGG目的端的MySQL配置

创建MySQL的ogg用户:

drop user ogg;

create user 'ogg'@'%' identified by 'ogg';

grant all on *.* to 'ogg'@'%';

创建checkpoint所需的ogg库:

create database ogg;

use ogg

create table test_ogg(id int,name varchar(20),primary key(id));


cd $OGG_HOME

ggsci

create subdirs


(2)OGG目的端管理进程MGR配置(侦听本地端口7810)

edit param mgr

PORT 7810

(这个端口号跟 pumysql配置中的 rmthost 127.0.0.1,mgrport 7810 相对应)

OGG目的端检查点日志表checkpoint配置

edit param ./GLOBALS


(3)OGG目的端检查点日志表checkpoint配置

CHECKPOINTTABLE ogg.checkpoint


将上次生成的 /opt/ogg/source_endpoint/dirdef/ogg_test.ogg_test 文件拷贝到 $OGG_HOME/dirdef下:

cp /opt/ogg/source_endpoint/dirdef/ogg_test.ogg_test /opt/ogg/target_endpoint/dirdef/ogg_test.ogg_test

如果没有权限,可以用root复制后修改权限:

进入ggsci命令行:

dblogin sourcedb ogg@127.0.0.1:3306 ,userid ogg, password ogg

add checkpointtable ogg.ggs_checkpoint


(4)OGG目的端回放线程r_tab1配置(实现将抽取的数据文件写到MySQL数据库表)

edit params r_tab1

replicat r_tab1
targetdb ogg@127.0.0.1:3306,userid ogg,password ogg
sourcedefs /opt/ogg/target_endpoint/dirdef/ogg_test.ogg_test
HANDLECOLLISIONS
MAP test_ogg.test_ogg,target ogg.test_ogg;

dblogin sourcedb ogg@127.0.0.1:3306 ,userid ogg, password ogg

delete replicat r_tab1

add replicat r_tab1,exttrail /opt/ogg/target_endpoint/dirdat/rt,checkpointtable ogg.ggs_checkpoint

info all

start mgr

start r_tab1


验证:

在Oracle表插入一条记录:

到MySQL看一下:

再插一条:

再看看MySQL:

可见同步成功!


TIPS:如果遇到 ERROR OGG-15161 Could not initialize the connection with MGR MGR (No route to host),一般是本机的IP在/etc/hosts中没定义,更新IP即可。

TIPS2:OGG不仅仅可以做Oracle到MySQL的实时数据同步,只要装上OGG for xx DB,就可以实现Oracle到其他各种数据库的实时数据同步,甚至实现从OGG到Kafka的数据同步。有兴趣的读者可以自行研究。


(全文完,谢谢阅读)

最后修改时间:2022-06-10 06:37:36
「喜欢这篇文章,您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容,转载时必须标注文章的来源(墨天轮),文章链接,文章作者等基本信息,否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容,欢迎发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

评论