ORA-01089 select fail over database link on RAC ADG Standby

前段时间配的一套11203 RAC ADG on EXADATA Machine的环境，在ADG 的standby side的node2 通过dblink查询时提示ora-1089错误，但是在node1 查询正常，DG的recover 进程是在node1 上，后确认是个bug 简单的记录。

-- on standby node2 ,but on standby node1 was worked
select count(*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
ORA-01089: immediate shutdown in progress - no operations are permitted
Process ID: 771
Session ID: 1517 Serial number: 4211
-- diag
alter session set events '10046 trace name context forever,level 12:1089 trace name errorstack level 3';
select count(*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
--error
复制

Call Stacke in trace file

----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFFB26EA518 ? 000000001 ?
                                                   000000001 ? 000000002 ?
ksedst1()+98         call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFFB26EA518 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksedst()+34          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFFB26EA518 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbkedDefDump()+2741  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFFB26EA518 ? 000000001 ?
                                                   000000000 ? 000000002 ?
ksedmp()+36          call     dbkedDefDump()       000000003 ? 000000000 ?
                                                   7FFFB26EA518 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbkdaKsdActDriver()  call     ksedmp()             000000003 ? 000000000 ?
+1960                                              7FFFB26EA518 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgdaExecuteAction(  call     dbkdaKsdActDriver()  7F86B4312710 ? 7FFFB26F11A0 ?
)+1065                                             7FFFB26EA518 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbgdaRunAction()+81  call     dbgdaExecuteAction(  7F86B4312710 ? 00A1B8DA0 ?
0                             )                    0020C0003 ? 7FFFB26F11A0 ?
                                                   000000001 ? 000000002 ?
dbgdRunActions()+59  call     dbgdaRunAction()     7F86B4312710 ? 000000005 ?
                                                   0020C0003 ? 7FFFB26F11A0 ?
                                                   000000001 ? 000000002 ?
dbgdProcessEventAct  call     dbgdRunActions()     7F86B4312710 ? 000000005 ?
ions()+651                                         0020C0003 ? 7FFFB26F11A0 ?
                                                   000000001 ? 000000002 ?
dbgdChkEventKgErr()  call     dbgdProcessEventAct  7F86B4312710 ? 00BC1D9A0 ?
复制

在MOS中Bug 17162712相符，该bug 影响11.2.0.3/4 在12.2 (Future Release) 12.1.0.2 (Server Patch Set) 版本中才修改了bug;或在不升级到12C前的版本有提供ONE patch 小补丁修复; 或重启standby db 的方式绕过这个bug,如下

This bug is only relevant when using Real Application Clusters (RAC) and Database Link / Distributed
If the recovering instance of a RAC standby fails, one of the remaining instances will do special instance recovery.
After this recovery one of the shutdown flags is not cleared, triggering the ORA-1089:"immediate shutdown in progress"
when a database link is used.
During the recovery an ORA-1089 is normal, the problem is that the error is raised even after the recovery
STACK TRACE:
------------
skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp <- dbkdaKsdActDriver
<- dbgdaExecuteAction <- dbgdaRunAction <- dbgdRunActions <-
dbgdProcessEventActions <- dbgdChkEventKgErr <- dbkdChkEventRdbmsEr <- ksfpec
<- dbgePostErrorKGE <- 1129 <- dbkePostKGE_kgsf <- kgeselv <- ksesecl0 <-
k2gInsert <- k2lbeg <- k2sibg <- npibeg <- kpnpre <- upirtrc <- kpurcsc <-
kpuexec <- OCIStmtExecute <- OCIKGetDescInfo <- ddfnetCFull <- ddfnet2Normal
<- kkmfcbrm <- kkmpfcbk <- qcsprfro <- qcsprfro_tree <- qcsprfro_tree <-
qcspafq <- qcspqbDescendents <- qcspqb <- kkmdrv <- opiSem <- opiprs <-
kksParseChildCursor <- rpiswu2 <- kksLoadChild <- kxsGetRuntimeLock <- kksfbc
<- kkspsc0 <- kksParseCursor <- opiosq0 <- kpooprx <- kpoal8 <- opiodr <-
ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real <-
ssthrdmain <- main <- libc_start_main <- start

# 重启STANDBY INSTANCE
[oracle@qdexa1db01 (orarpt1)oracle]$ srvctl config database
rptstby
[oracle@qdexa1db01 (orarpt1)oracle]$ srvctl stop database -d rptstby
[oracle@qdexa1db01 (orarpt1)oracle]$ srvctl start database -d rptstby  
SQL> select name,open_mode,database_role from gv$database;
NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
ORARPT    MOUNTED              PHYSICAL STANDBY
ORARPT    MOUNTED              PHYSICAL STANDBY
# open  all standby instances 
SQL> alter database open;
SQL> select name,open_mode,database_role from gv$database;
NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
ORARPT    READ ONLY            PHYSICAL STANDBY
ORARPT    READ ONLY            PHYSICAL STANDBY
SQL>  alter database recover managed standby database PARALLEL 48 using current logfile disconnect from session;
Database altered.
#  in both instances run same queries over dblink not with ora-1089 error also Now.
SQL> select count(*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
  COUNT(*)
----------
       317
复制