(零)简述
OGG是Oracle GoldenGate的简称,它可以完成Oracle到异构数据库的实时数据同步。
使用OGG完成Oracle到MySQL的同步,其原理如下所示:
故使用OGG完成Oracle到MySQL的数据同步需要以下几个步骤:
(一)在Oracle端安装OGG for Oracle
(二)源端OGG配置
(1)OGG源端相关的Oracle配置(创建c##ogg用户、建立测试表test_ogg等等)
(2)OGG源端表级补全日志trandata配置
(3)OGG源端管理进程MGR配置(侦听本地端口7809)
(4)OGG源端抽取进程extorcl配置(实现从Oracle数据库表抽取数据到数据文件)
(5)OGG源端传输进程pumysql配置(实现将抽取的数据文件发送到OGG目的端)
(6)OGG源端异构mapping文件defgen生成
(三)在MySQL端安装OGG for MySQL
(四)目的端OGG配置
(1)OGG目的端的MySQL配置
(2)OGG目的端管理进程MGR配置(侦听本地端口7810)
(3)OGG目的端检查点日志表checkpoint配置
(4)OGG目的端回放线程r_tab1配置(实现将抽取的数据文件写到MySQL数据库表)
(一)在Oracle端安装OGG for Oracle
打开 https://www.oracle.com/middleware/technologies/goldengate-downloads.html
搜索 Oracle GoldenGate 21.3.0.0.0 for Oracle on Linux x86-64
下载OGG for Oracle安装包 213000_fbo_ggs_Linux_x64_Oracle_shiphome.zip

以Oracle用户登录:
修改.bashrc,增加以下内容
export JAVA_HOME=$HOME/jdk1.8.0_141
export PATH=$JAVA_HOME/bin:$PATH
export TNS_ADMIN=$ORACLE_HOME/network/admin
export OGG_HOME=/opt/ogg/source_endpoint
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64:$JAVA_HOME/jre/lib/amd64/server:$JAVA_HOME/jre/lib/amd64/libjsig.so:$JAVA_HOME/jre/lib/amd64/server/libjvm.so:$OGG_HOME/lib:$LD_LIBRARY_PATH:$ORACLE_HOME/lib:/usr/lib
export PATH=$OGG_HOME:$PATH:$ORACLE_HOME/bin
切换到root用户
su - root
useradd ogg -g oinstall
passwd ogg

mkdir -p /opt/ogg
chown -R ogg /opt/ogg
mkdir -p /home/oracle/oradata/OGG/ORCLCDB
mkdir -p /home/oracle/oradata/OGG/tablespace
chown -R oracle:oinstall /home/oracle/oradata/OGG

切换到ogg用户:
将ogg安装包 213000_fbo_ggs_Linux_x64_Oracle_shiphome.zip 传到/home/ogg下:

unzip -d ~ 213000_fbo_ggs_Linux_x64_Oracle_shiphome.zip

cd fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/response/
编辑 oggcore.rsp
更新以下行:
INSTALL_OPTION=ORA19C
SOFTWARE_LOCATION=/opt/ogg/source_endpoint

cd ..
./runInstaller -silent -nowait -responseFile /home/ogg/fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/response/oggcore.rsp

报 /u01/app/oracle/oraInventory 没有权限写入。。。查了一下,确实没有,那就加上权限。

重来:
切换到ogg用户:
cd fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/
./runInstaller -silent -nowait -responseFile /home/ogg/fbo_ggs_Linux_x64_Oracle_shiphome/Disk1/response/oggcore.rsp

(二)源端OGG配置
编辑~/.bashrc
增加以下内容:
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/19c/db_1
export ORACLE_SID=zhanghui
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
export OGG_HOME=/opt/ogg/source_endpoint
export PATH=$OGG_HOME:$PATH:$ORACLE_HOME/bin
export TNS_ADMIN=$ORACLE_HOME/network/admin
source ~/.bashrc使其生效:

建立软连接:
ln -s $ORACLE_HOME/lib/libnnz19.so /opt/ogg/source_endpoint/libnnz19.so
cd $OGG_HOME
ldd ggsci

(1)OGG源端相关的Oracle配置(创建c##ogg用户、建立测试表test_ogg等等)
切换到oracle用户
su - oracle
mkdir /home/oracle/archivelog
查看目前是否打开了归档日志:
sqlplus '/as sysdba'
archive log list

没打开。
那么就打开它。注意如果打开了归档日志,那么所有的DML操作都会记录到归档日志中,这个文件可能会非常大,直到让文件系统爆满,这会直接导致oracle无法提供服务。所以要务必小心跟踪归档日志所属文件系统的剩余空间情况。
alter system set log_archive_dest_1='location=/home/oracle/archivelog';
shutdown immediate;
startup mount;
alter database archivelog;
alter database open;
alter system archive log start;
alter system archive log current;
alter database force logging;
alter database add supplemental log data;

alter system set enable_goldengate_replication=true;
select name,open_mode,force_logging,supplemental_log_data_min from v$database;
archive log list

创建c##ogg所需要的表空间:
create tablespace oggtbs datafile '/home/oracle/oradata/OGG/tablespace/oggtbs01.dbf' size 1000M autoextend on;
并在上面创建c##ogg用户:
create user c##ogg identified by ogg default tablespace oggtbs;
赋予权限:
grant dba ,connect, resource, unlimited tablespace to c##ogg;
exec dbms_goldengate_auth.grant_admin_privilege('c##ogg','*',TRUE);
select username,common,con_id from cdb_users where username like '%OGG%';
commit;
查看pdb状态:show pdbs
如果发现pdb是MOUNTED状态,可以执行:
alter session set container=ORCLPDB;
alter pluggable database open;
将其改为READ/WRITE状态:

alter session set container=ORCLPDB;
show con_name;

确认现在是在ORCLPDB库中,
给c##ogg用户赋权限:
grant dba ,connect, resource, unlimited tablespace to c##ogg;

编辑$TNS_ADMIN下的 tnsnames.ora文件:
LISTENER_ZHANGHUI =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
ZHANGHUI =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = zhanghui)
)
)
ORCLPDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORCLPDB )
)
)
增加ORCLPDB这个PDB数据库的配置。
创建ogg所需要的表空间:
create tablespace mytbs datafile '/home/oracle/oradata/OGG/tablespace/mytbs01.dbf' size 500M autoextend on;
并在上面创建ogg用户:
create user ogg identified by ogg default tablespace mytbs;
赋予权限:
grant dba ,connect, resource, unlimited tablespace to ogg;


exec dbms_goldengate_auth.grant_admin_privilege('ogg','*',TRUE);
select username,common,con_id from cdb_users where username like '%OGG%';
commit;

验证ogg可以正常登录ORCLPDB数据库:

切换到ogg用户:
cd $OGG_HOME
ggsci
create subdirs

切换到oracle用户,创建一个test_ogg用户:
create user test_ogg identified by ogg ;
grant dba ,connect, resource, unlimited tablespace to test_ogg;(实际上用户的权限不必这么大)
quit
切换到test_ogg用户登录ORCLPDB
sqlplus test_ogg/ogg@ORCLPDB
create table test_ogg(id int,name varchar(20),primary key(id));
(2)OGG源端表级补全日志trandata配置
切换到ogg用户:
cd $OGG_HOME
ggsci
edit param ./globals

填入以下内容:
oggschema ogg

dblogin userid ogg@ORCLPDB password ogg

add schematrandata ORCLPDB.test_ogg

add trandata test_ogg.*

capture tabledef test_ogg.*

(3)OGG源端管理进程MGR配置(侦听本地端口7809)
edit param mgr
编辑以下内容:
PORT 7809
保存退出

(4)OGG源端抽取进程extorcl配置(实现从Oracle数据库表抽取数据到数据文件)
edit param extorcl
extract extorcl
DDL INCLUDE ALL
SETENV (ORACLE_HOME = "/u01/app/oracle/product/19c/db_1")
SETENV (ORACLE_SID="zhanghui")
SETENV (NLS_LANG="american_america.AL32UTF8")
userid c##ogg@zhanghui , password ogg
exttrail /opt/ogg/source_endpoint/dirdat/to
table ORCLPDB.test_ogg.*;
注意:
1.如果使用ORCLPDB中的ogg用户,在启动extorcl的时候就会报以下错误:

所以必须配置使用c##ogg用户登录才行。
2.如果table忘记带了数据库名,写成这样:table test_ogg.*;,启动extorcl的时候就会报以下错误:

delete extract extorcl
add extract extorcl,INTEGRATED TRANLOG,BEGIN NOW
add exttrail /opt/ogg/source_endpoint/dirdat/to, extract extorcl
dblogin userid ogg@ORCLPDB, password ogg

register extract extorcl database container(ORCLPDB)

很奇怪,它不能在PDB里面注册,那我们切换都CDB环境试试:
dblogin userid c##ogg@zhanghui, password ogg
register extract extorcl database container(ORCLPDB)

(5)OGG源端传输进程pumysql配置(实现将抽取的数据文件发送到OGG目的端)
edit param pumysql
加入以下内容:
extract pumysql
userid c##ogg@zhanghui , password ogg
rmthost 127.0.0.1,mgrport 7810
rmttrail /opt/ogg/target_endpoint/dirdat/rt
table ORCLPDB.test_ogg.*;
注意:如果使用ORCLPDB中的ogg用户,在启动pumysql的时候就会报以下错误:

所以必须配置使用c##ogg用户登录才行。
delete extract pumysql
add extract pumysql, exttrailsource /opt/ogg/source_endpoint/dirdat/to
add rmttrail /opt/ogg/target_endpoint/dirdat/rt, extract pumysql, megabytes 100
info all

start mgr

start extorcl

start pumysql

(6)OGG源端异构mapping文件defgen生成
edit param tab1
dsdffile /opt/ogg/source_endpoint/dirdef/ogg_test.ogg_test
userid test_ogg@ORCLPDB, password ogg
table test_ogg.test_ogg;
quit
./defgen paramfile dirprm/tab1.prm

info all

插入一条test_ogg表的记录:
insert into test_ogg values(1,'Hello World');
commit;

在ggsci中查看extorcl的日志:
view report extorcl

在Run Time Messages后面就是从oracle抽取记录的日志。
在ggsci中查看pumysql的日志:

在Run Time Messages后面就是将上面抽取的表记录发送到mysql服务端MGR的日志。
(三)在MySQL端安装OGG for MySQL
卸载CentOS自带的mariadb:
rpm -qa | grep mariadb

rpm -e --nodeps mariadb-server-5.5.68-1.el7.x86_64 mariadb-5.5.68-1.el7.x86_64 mariadb-libs-5.5.68-1.el7.x86_64

安装MySQL:
cd /usr/local
wget https://cdn.mysql.com/Downloads/MySQL-5.7/mysql-5.7.37-el7-x86_64.tar.gz

解压:
tar -zxvf mysql-*.gz

mv mysql-5.7.37-el7-x86_64 mysql
设置权限:
chown -R mysql:mysql /usr/local/mysql
chmod -R 755 /usr/local/mysql
mkdir -p /data/mysql/data
mkdir -p /data/mysql/logs
mkdir -p /data/mysql/tmp
chown -R mysql:mysql /data/mysql

编辑 /etc/my.cnf 文件:

内容如下:
[client]
port = 3306
socket = /data/mysql/tmp/mysql.sock
[mysqld]
user = mysql
basedir = /usr/local/mysql
datadir = /data/mysql/data
port = 3306
socket = /data/mysql/tmp/mysql.sock
pid-file = /data/mysql/tmp/mysqld.pid
tmpdir = /data/mysql/tmp
skip_name_resolve = 1
symbolic-links=0
max_connections = 2000
group_concat_max_len = 1024000
sql_mode = NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
lower_case_table_names = 1
log_timestamps=SYSTEM
character-set-server = utf8
interactive_timeout = 1800
wait_timeout = 1800
max_allowed_packet = 32M
binlog_cache_size = 4M
sort_buffer_size = 2M
read_buffer_size = 4M
join_buffer_size = 4M
tmp_table_size = 96M
max_heap_table_size = 96M
max_length_for_sort_data = 8096
#logs
server-id = 1003306
log-error = /data/mysql/logs/error.log
slow_query_log = 1
slow_query_log_file = /data/mysql/logs/slow.log
long_query_time = 3
log-bin = /data/mysql/logs/binlog
binlog_format = row
expire_logs_days = 15
log_bin_trust_function_creators = 1
relay-log = /data/mysql/logs/relay-bin
relay-log-recovery = 1
relay_log_purge = 1
#innodb
innodb_file_per_table = 1
innodb_log_buffer_size = 16M
innodb_log_file_size = 256M
innodb_log_files_in_group = 2
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
innodb_flush_neighbors = 0
innodb_flush_method = O_DIRECT
innodb_autoinc_lock_mode = 2
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_buffer_pool_size = 2G
将mysql设置为linux服务:
cd /usr/local/mysql/support-files
cp mysql.server /etc/init.d/mysql
编辑 /etc/init.d/mysql 文件:
设置以下两行:
basedir=/usr/local/mysql
datadir=/data/mysql/data

chkconfig --add mysql
chkconfig --list

编辑环境变量:
echo "PATH=$PATH:/usr/local/mysql/bin " >> /etc/profile
source /etc/profile

rpm -qa|grep libaio
如果没有,则需要安装:yum install libaio-devel.x86_64

/usr/local/mysql/bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/data/mysql/data
more /data/mysql/logs/error.log | grep password

重启MySQL:
service stop mysql
service start mysql
得使用
systemctl restart mysql
systemctl status mysql

登录:
mysql -u root -p
输入前面的密码:BOc:CUr?S0aj

修改root密码:
alter user 'root'@'localhost' identified by 'zhanghui';
flush privileges;
quit退出命令行。
ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql

使用新密码zhanghui登录:
mysql -u root -p
create user 'ogg'@'%' identified by 'ogg';
grant all on *.* to 'ogg'@'%';

打开 https://www.oracle.com/middleware/technologies/goldengate-downloads.html
找到 Oracle GoldenGate 21.3.0.0.0 for MySQL-compatible Databases on Linux x86-64 下载:

使用root登录
创建目的端ogg用户:ogg2
useradd ogg2
passwd ogg2
su - ogg2

切换到ogg2用户:su - ogg2
将ogg for mysql安装包 传到/home/ogg2下:

unzip -d ~ ./213000_ggs_Linux_x64_MySQL_64bit.zip

切换到root用户
mkdir /opt/ogg/target_endpoint
chown -R ogg2 /opt/ogg/target_endpoint

切换回ogg2用户:
tar -xvf ggs_Linux_x64_MySQL_64bit.tar -C /opt/ogg/target_endpoint

编辑 ~/.bashrc文件:
export JAVA_HOME=$HOME/jdk1.8.0_141
export PATH=$JAVA_HOME/bin:$PATH
export OGG_HOME=/opt/ogg/target_endpoint
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64:$JAVA_HOME/jre/lib/amd64/server:$JAVA_HOME/jre/lib/amd64/libjsig.so:$JAVA_HOME/jre/lib/amd64/server/libjvm.so:$OGG_HOME/lib:$HOME/libs
export PATH=$OGG_HOME:$PATH
source ~/.bashrc使其生效。

(四)目的端OGG配置
(1)OGG目的端的MySQL配置
创建MySQL的ogg用户:
drop user ogg;
create user 'ogg'@'%' identified by 'ogg';
grant all on *.* to 'ogg'@'%';
创建checkpoint所需的ogg库:
create database ogg;

use ogg
create table test_ogg(id int,name varchar(20),primary key(id));

cd $OGG_HOME
ggsci
create subdirs

(2)OGG目的端管理进程MGR配置(侦听本地端口7810)
edit param mgr

PORT 7810
(这个端口号跟 pumysql配置中的 rmthost 127.0.0.1,mgrport 7810 相对应)
OGG目的端检查点日志表checkpoint配置
edit param ./GLOBALS

(3)OGG目的端检查点日志表checkpoint配置
CHECKPOINTTABLE ogg.checkpoint

将上次生成的 /opt/ogg/source_endpoint/dirdef/ogg_test.ogg_test 文件拷贝到 $OGG_HOME/dirdef下:
cp /opt/ogg/source_endpoint/dirdef/ogg_test.ogg_test /opt/ogg/target_endpoint/dirdef/ogg_test.ogg_test
如果没有权限,可以用root复制后修改权限:

进入ggsci命令行:
dblogin sourcedb ogg@127.0.0.1:3306 ,userid ogg, password ogg
add checkpointtable ogg.ggs_checkpoint

(4)OGG目的端回放线程r_tab1配置(实现将抽取的数据文件写到MySQL数据库表)
edit params r_tab1
replicat r_tab1
targetdb ogg@127.0.0.1:3306,userid ogg,password ogg
sourcedefs /opt/ogg/target_endpoint/dirdef/ogg_test.ogg_test
HANDLECOLLISIONS
MAP test_ogg.test_ogg,target ogg.test_ogg;
dblogin sourcedb ogg@127.0.0.1:3306 ,userid ogg, password ogg
delete replicat r_tab1
add replicat r_tab1,exttrail /opt/ogg/target_endpoint/dirdat/rt,checkpointtable ogg.ggs_checkpoint

info all
start mgr
start r_tab1
验证:
在Oracle表插入一条记录:

到MySQL看一下:

再插一条:

再看看MySQL:

可见同步成功!
TIPS:如果遇到 ERROR OGG-15161 Could not initialize the connection with MGR MGR (No route to host),一般是本机的IP在/etc/hosts中没定义,更新IP即可。
TIPS2:OGG不仅仅可以做Oracle到MySQL的实时数据同步,只要装上OGG for xx DB,就可以实现Oracle到其他各种数据库的实时数据同步,甚至实现从OGG到Kafka的数据同步。有兴趣的读者可以自行研究。
(全文完,谢谢阅读)