某客户19C DG库crash处理

某客户生产环境Oracle 19c dg突然crash，报错信息如下：

alert

2020-06-04T09:36:34.523210+08:00
(3):Successfully added datafile 296 to media recovery
(3):Datafile #296: '/u01/oradata/HSXTDG3/88B13FD20A185CC6E0538302870A0457/datafile/o1_mf_fas_prod_hfjmdltb_.dbf'
2020-06-04T09:37:00.774787+08:00
TMON (PID:30260): Process (PID:6920) hung on an I/O after 268 seconds with threshold of 240 at [krsb.c:3579]
TMON (PID:30260): WARN: Terminating process hung on an operation (PID:6920)

2020-06-04T09:38:47.001513+08:00
opiodr aborting process unknown ospid (7511) as a result of ORA-609
2020-06-04T09:38:47.214040+08:00
Errors in file /u01/app/oracle/diag/rdbms/hsxtdg3/hsxtdg3/trace/hsxtdg3_ora_7177.trc:
ORA-01013: user requested cancel of current operation
2020-06-04T09:38:47.235712+08:00
Errors in file /u01/app/oracle/diag/rdbms/hsxtdg3/hsxtdg3/trace/hsxtdg3_ora_7177.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
2020-06-04T09:38:47.236231+08:00
Errors in file /u01/app/oracle/diag/rdbms/hsxtdg3/hsxtdg3/trace/hsxtdg3_ora_7177.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
2020-06-04T09:38:48.153592+08:00
 rfs (PID:7619): Primary database is in MAXIMUM PERFORMANCE mode
2020-06-04T09:38:50.250362+08:00
TMON (PID:30260): Process (PID:7072) hung on an I/O after 377 seconds with threshold of 240 at [krsb.c:3579]
2020-06-04T09:38:50.250373+08:00
 rfs (PID:7619): Killing 1 processes (PIDS:6920) (RFS live redo by DBID/thread for ASYNC) in order to receive ASYNC. Requested by OS process 7619
TMON (PID:30260): WARN: Terminating process hung on an operation (PID:7072)
2020-06-04T09:38:50.250681+08:00
Process termination requested for pid 6920 [source = rdbms], [info = 2] [request issued by pid: 7619, uid: 1102]
2020-06-04T09:40:44.715910+08:00
DIA0 Critical Database Process As Root: Hang ID 24 blocks 1 sessions
     Final blocker is session ID 2270 serial# 9773 OSPID 30189 on Instance 1
     No resolution will be attempted by Hang Manager


2020-06-04T09:40:44.816345+08:00
opiodr aborting process unknown ospid (7704) as a result of ORA-609
2020-06-04T09:40:44.829545+08:00
 rfs (PID:7619): Selected LNO:13 for T-2.S-38686 dbid 939117387 branch 1008102349
 rfs (PID:7619): Possible network disconnect with primary database
2020-06-04T09:40:45.015759+08:00
 rfs (PID:22409): Possible network disconnect with primary database
2020-06-04T09:40:47.065864+08:00
TMON (PID:30260): Killing 1 processes (PIDS:7072) (Process by index) in order to remove hung processes. Requested by OS process 30260
2020-06-04T09:40:47.066680+08:00
Process termination requested for pid 7072 [source = rdbms], [info = 2] [request issued by pid: 30260, uid: 1102]
复制

trace

[oracle@zbhsxt ~]$ more /u01/app/oracle/diag/rdbms/hsxtdg3/hsxtdg3/trace/hsxtdg3_ora_7177.trc
Trace file /picclife/app/oracle/diag/rdbms/hsxtdg3/hsxtdg3/trace/hsxtdg3_ora_7177.trc
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
Build label:    RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417
ORACLE_HOME:    /u01/app/oracle/product/19.3/db_1
System name:    Linux
Node name:      zbhsxt
Release:        3.10.0-957.10.1.el7.x86_64
Version:        #1 SMP Mon Mar 18 15:06:45 UTC 2019
Machine:        x86_64
Instance name: hsxtdg3
Redo thread mounted by this instance: 1
Oracle process number: 91
Unix process pid: 7177, image: oracle@zbhsxt


*** 2020-06-04T09:38:46.741610+08:00 (CDB$ROOT(1))
*** SESSION ID:(1151.51933) 2020-06-04T09:38:46.741682+08:00
*** CLIENT ID:() 2020-06-04T09:38:46.741699+08:00
*** SERVICE NAME:(hsxtdg3) 2020-06-04T09:38:46.741707+08:00
*** MODULE NAME:(JDBC Thin Client) 2020-06-04T09:38:46.741715+08:00
*** ACTION NAME:() 2020-06-04T09:38:46.741723+08:00
*** CLIENT DRIVER:(jdbcthin : 19.3.0.0.0) 2020-06-04T09:38:46.741730+08:00
*** CONTAINER ID:(1) 2020-06-04T09:38:46.741738+08:00
 
Log read is SYNCHRONOUS though disk_asynch_io is enabled!
krst_gims: Failed to retrieve SRL info for /u01/oradata/group_15.730.1011529109 LNO:0 T-2.S-38685 reason: invalid NAB error: 1013

*** 2020-06-04T09:38:47.213733+08:00 (CDB$ROOT(1))
<error barrier> at 0x7fffd1383bb8 placed krst.c@1756
ORA-01013: user requested cancel of current operation
<error barrier> at 0x7fffd1384340 placed krst.c@3226
ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
<error barrier> at 0x7fffd13846c0 placed krst.c@2576
ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
复制

处理如下：

cdb中执行：
–将备库的standby_file_management参数修改为手动管理模式

alter system set standby_file_management='MANUAL' scope=both;
复制

在相应pdb中执行：
– 把UNNAMED数据文件放到正确路径

alter database create datafile '/u01/app/oracle/product/19.3/db_1/dbs/UNNAMED00297' as '/u01/oradata/HSXTDG3/88B13FD20A185CC6E0538302870A0457/datafile/fas_prod_tbs297.dbf';
复制

cdb中执行：
– 将备库的standby_file_management参数修改回自动模式

ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO scope=both;
复制

开启mrp应用：

alter database recover managed standby database using current logfile disconnect from session;
复制