运行了5-6年的ogg抽取进程突然abend异常

oracle11.2.0.4 rac+redhat6.5+ogg12.2.0.1

ogg抽取进程abend异常，报错信息：

view repoort zycq

terminate called after throwing an instance of 'ggs::gglib::ggcore::CException' 

检查dirdat的队列文件，可看到时间发生在27号的14:32左右队列文件不再增长，询问一圈没有异常数据库变更操作。

ogg的报告中没有看到任何其它错误信息，检查操作系统和oracle alert日志在故障的时间点也没有看到任何错误信息。

metalink唯一找到一篇报错信息匹配的文档说明，但我的环境并不是ogg的集成模式，而是传统经典模式。

如下可看到：传统经典模式的redo不带“Integrated”前缀

Integrated Extract abend: terminate called after throwing ‘ggs::gglib::ggcore::CException’ (Doc ID 2012919.1)

Oracle GoldenGate - Version 12.1.2.1.0 and later
Information in this document applies to any platform.
SYMPTOMS
Integrated extract abends with:

2015-04-08 09:08:16  INFO    OGG-03522  Setting session time zone to source database time zone '-05:00'. 
terminate called after throwing an instance of 'ggs::gglib::ggcore::CException' 


We tried to unregister, re-register and 'begin now' a few times.  It ran fine for a while before hitting the problem again. 

Following id gdb stacks:

(gdb) where 
#0  0x0000003ff8e32625 in raise () from /lib64/libc.so.6 
#1  0x0000003ff8e33e05 in abort () from /lib64/libc.so.6 
#2  0x0000003245ebea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6 
#3  0x0000003245ebcbd6 in ?? () from /usr/lib64/libstdc++.so.6 
#4  0x0000003245ebcc03 in std::terminate() () from /usr/lib64/libstdc++.so.6 
#5  0x0000003245ebcd22 in __cxa_throw () from /usr/lib64/libstdc++.so.6 
#6  0x0000000000928777 in ggs::gglib::ggparser::CParserUtil::throwParseError (this=<value optimized out>, 
    pParser=<value optimized out>, pTokenStream=<value optimized out>, pLexer=<value optimized out>, 
    pInputStream=<value optimized out>) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggparser/CParserUtil.cpp:213 
#7  0x000000000091def0 in ggs::gglib::ggapp::CDBObjNameParser::parseDBObjIdentifier (this=<value optimized out>, 
    sDBObjIdentifier=<value optimized out>, bIsQualified=false) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggapp/CDBObjNameParser.cpp:227 
#8  0x000000000054d130 in ggs::gglib::ggapp::CQualDBObjName<DBTABLE>::assignParameterName (this=0x7fffffff9dc0, sName= 
    ..., bSourceDB=true) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/include/ggapp/CDBObjName.h:1698 
#9  0x00000000006002ca in ddlStatement_t::findTableAfterKeyword (this=<value optimized out>, refKeyword=..., table=...) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggapp/ddlstmt.c:1958 
#10 0x000000000060eec1 in processObjectAfterKeyword (extractData=0x2219920, ddlPointer=0x7fffffffa030, refKeyword=..., 
    isMaster=false) at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggapp/ddlext.c:1511 
#11 0x0000000000608cbb in DDLEXT_process_extract (extractData=0x2219920, cc=<value optimized out>, 
    ggs_tkn_buf=0x1a4d4c0 "", ggs_tkn_len=0x1a51340, ddlRecord=0x7fffffffb930, ddlRecordLen=0x7fffffffb9c8, 
    clearCache=1, doEMI=1, tryLoginUser=1, altDDLString=0x0) 
---Type <return> to continue, or q <return> to quit--- 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggapp/ddlext.c:1869 
#12 0x00000000007d6c37 in processDDLtrans (lc=0x2218ca0, ggs_tkn_buf=0x1a4d4c0 "", ggs_tkn_len=0x1a51340, 
    com_hdr=0x7fffe2064010, mt=<value optimized out>, hdr=<value optimized out>, is_ddl=0x7fffffffbba0) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/redo/oracle/redooraix.c:5080 
#13 0x00000000007f36c8 in REDOORAIX_get_committed_lcr (mp=0x222bce0, xoutcom=0x2dd5680, mt=0x1ebc078, 
    rec=0x7fffeadfd010 "\240\003\370\001", rec_len=0x1a55d44, rec_ts=<value optimized out>, more_record=0x1ebc070, 
    ggs_tkn_buf=0x1a4d4c0 "", ggs_tkn_len=0x1a51340, commit_pos=..., errtext=0x1a49b80 "") 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/redo/oracle/redooraix.c:5709 
#14 0x000000000074ba59 in REDOORAIX_TransferLCRfromCOM (ctx=0x1ebc060, rec=0x7fffeadfd010 "\240\003\370\001", 
    rec_len=0x1a55d44, rec_ts=0xdffca8, ggs_tkn_buf=0x1a4d4c0 "", ggs_tkn_len=0x1a51340, errtext=0x1a49b80 "") 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/redo/oracle/redoora.c:11365 
#15 0x000000000074d4f9 in REDO_xout_read (mp=0x222bce0, checkpointing=1, rec=0x7fffeadfd010 "\240\003\370\001", 
    rec_len=0x1a55d44, rec_ts=0xdffca8, ggs_tkn_buf=0x1a4d4c0 "", ggs_tkn_len=0x1a51340, errtext=...) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/redo/oracle/redoora.c:11697 
#16 0x000000000074e523 in REDO_read (rec=0x7fffeadfd010 "\240\003\370\001", rec_len=0x1a55d44, rec_ts=0xdffca8, 
    ggs_tkn_buf=0x1a4d4c0 "", ggs_tkn_len=0x1a51340, checkpointing=1, errtext=...) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/redo/oracle/redoora.c:7228 
#17 0x0000000000535709 in READ_EXTRACT_RECORD (cp=0x2226410, data_source=<value optimized out>, 
    prec=0x7da0 <Address 0x7da0 out of bounds>, rec_size=4194304, hdr=0x206, return_seqno=0x0, return_rba=0x0, 
    last_ts=0x7fffffffcae0, header_supplied=0, ggs_tkn=0x8 <Address 0x8 out of bounds>, ggs_tkn_len=0x7ffff1ac3330, 
    user_tkn=0x1a55f00 "", user_tkn_len=0x1a59d80, defer_apply_ts=0x0, err_text=...) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/api.c:4986 
#18 0x00000000005b86a6 in process_extract_loop () 
---Type <return> to continue, or q <return> to quit--- 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/extract/processloop.cpp:290 
#19 0x00000000005b6aa8 in extract_main (argc=<value optimized out>, argv=<value optimized out>) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/extract/extract.cpp:1039 
#20 0x00000000006a1e2f in ggs::gglib::MultiThreading::MainThread::ExecMain (this=0x7fffffffe330) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggstd/MultiThreading.cpp:1212 
#21 0x00000000006a2334 in ggs::gglib::MultiThreading::Thread::RunThread(struct {...} *) ( 
    threadArgs=<value optimized out>) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggstd/MultiThreading.cpp:452 
#22 0x00000000006a26fb in ggs::gglib::MultiThreading::MainThread::Run (this=0x7fffffffe330, 
    argc=<value optimized out>, argv=0x7fffffffe518) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/gglib/ggstd/MultiThreading.cpp:1203 
#23 0x00000000005b643f in main (argc=3, argv=0x7fffffffe518) 
    at /scratch/aime1/adestore/views/aime1_adc4150384/oggcore/OpenSys/src/app/er/extract/extract.cpp:1059 
 

CAUSE
This was initially fixed via Bug 21329486. However the change was not complete and Bug 21548926 has the complete fix.

Integrated Extract core dumps during parsing of a DDL (create synonym) if the table name spec after keyword 'FOR' contains spaces after '.'. For example: the DDL 'create or replace synonym syn1 for scott. tiger;' will fail with core dump. 


If IE crashes during parsing of a synonym creation DDL, check to see if there are any spaces after '.' in table name spec, and if so, we are running into same issue.

SOLUTION
Request for a OGG build with fix in Bug 21548926

In addition, workaround is to use extract parameter to temporarily exclude the DDL, although this might cause data loss:

DDL exclude all

DDLOPTIONS IGNOREAPPLOPS, GETREPLICATES

不过从文档可猜测应该是遇到了bug

但我的版本是12.2.0.1.170919明显比12.2.0.1.170221更新了，bug依然，所以这个补丁并不满足要求，没法打。
只能考虑尝试忽略ddl错误的方式处理，虽然看不到有DDL错误。
当前并没有参考官方给的临时处理方式，而是增加如下参数跳过DDL错误，启动正常：DDLERROR RESTARTSKIP 100000 SKIPTRIGGERERROR 100000

抽取追平之后尝试去掉参数，尝试了半个多小时，再启动发现还是有问题，所以暂时只能保留参数继续。当然对于DDL的操作可能存在丢失的问题，这个需要进一步观察分析，目前没有发现数据丢失问题。
经过1天后，尝试去掉参数，抽取进程竟然恢复正常了，bug!!!