2021-03-26
Oracle 12.2.0.1 RAC数据库由于hang住导致所有实例被迫重启
10M在12.2.0.1的2节点rac数据库，几乎在同一时间数据库突然被hang住，告警日志中出现swap频繁交换、以及死锁，随后数据库后台进程被多次hang住之后，所有实例被迫重启，在近三周的时间内出现过3次这样的现象，急需排查根本原因。
以下是最近一次数据库宕机的告警日志
节点2：
3-25 10:35:01出现死锁，10:56节点2被迫重启
2021-03-25T09:35:21.511689+08:00
Archived Log entry 196171 added for T-2.S-13088 ID 0x8ef7a2a LAD:1
2021-03-25T09:50:21.065080+08:00
Thread 2 advanced to log sequence 13090 (LGWR switch)
Current log# 5 seq# 13090 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_5.293.1040551221
Current log# 5 seq# 13090 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_5.319.1040551223
2021-03-25T09:50:21.908176+08:00
TT04: Standby redo logfile selected for thread 2 sequence 13090 for destination LOG_ARCHIVE_DEST_2
2021-03-25T09:50:22.731306+08:00
Archived Log entry 196175 added for T-2.S-13089 ID 0x8ef7a2a LAD:1
2021-03-25T10:05:19.715736+08:00
Thread 2 advanced to log sequence 13091 (LGWR switch)
Current log# 6 seq# 13091 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_6.294.1040551365
Current log# 6 seq# 13091 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_6.356.1040551367
2021-03-25T10:05:20.588604+08:00
TT04: Standby redo logfile selected for thread 2 sequence 13091 for destination LOG_ARCHIVE_DEST_2
2021-03-25T10:05:21.252570+08:00
Archived Log entry 196179 added for T-2.S-13090 ID 0x8ef7a2a LAD:1
2021-03-25T10:20:21.074877+08:00
Thread 2 advanced to log sequence 13092 (LGWR switch)
Current log# 12 seq# 13092 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_12.335.1040550823
Current log# 12 seq# 13092 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_12.574.1040550825
2021-03-25T10:20:21.915435+08:00
TT04: Standby redo logfile selected for thread 2 sequence 13092 for destination LOG_ARCHIVE_DEST_2
2021-03-25T10:20:22.253918+08:00
Archived Log entry 196183 added for T-2.S-13091 ID 0x8ef7a2a LAD:1
2021-03-25T10:33:42.048764+08:00
LGWR (ospid: 25700) waits for event 'enq: CF - contention' for 72 secs.
2021-03-25T10:33:42.048922+08:00
LGWR (ospid: 25700) is hung in an acceptable location (inwait 0x201.00).
2021-03-25T10:34:54.768885+08:00
opiodr aborting process unknown ospid (2554) as a result of ORA-609
2021-03-25T10:35:01.052652+08:00
opiodr aborting process unknown ospid (2602) as a result of ORA-609
这里出现死锁
2021-03-25T10:35:01.592659+08:00
Global Enqueue Services Deadlock detected (DID = 12_1_3). More information in file
/oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_lmd0_25660.trc.
2021-03-25T10:36:01.201890+08:00
opiodr aborting process unknown ospid (3598) as a result of ORA-609
2021-03-25T10:37:53.500710+08:00
opiodr aborting process unknown ospid (3741) as a result of ORA-609
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T10:37:54.045081+08:00
opiodr aborting process unknown ospid (3732) as a result of ORA-609
2021-03-25T10:37:55.155683+08:00
opiodr aborting process unknown ospid (3720) as a result of ORA-609
2021-03-25T10:38:19.438892+08:00
WARNING: Waiting on CRSD for password file, for 20 seconds, check CRS and OS logs
2021-03-25T10:38:25.378438+08:00
QC with OSPID=16731 on instance 1 was signalled 12805 from process with OSPID=3168
2021-03-25T10:39:32.892850+08:00
WARNING: Heavy swapping observed on system in last 5 mins.
Heavy swapping can lead to timeouts, poor performance, and instance eviction.
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_dbrm_25628.trc (incident=1600169) (PDBNAME=CDB$ROOT):
ORA-00700: soft internal error, arguments: [kskvmstatact: excessive swapping observed], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/incident/incdir_1600169/dzzbcdb2_dbrm_25628_i1600169.trc
2021-03-25T10:39:40.893594+08:00
Dumping diagnostic data in directory=[cdmp_20210325103938], requested by (instance=2, osid=25628 (DBRM)), summary=[incident=1600169].
2021-03-25T10:39:44.542542+08:00
2021-03-25T10:42:57.620494+08:00
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
------------------------------------------------------------
TT04: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (16191)
TT04: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
2021-03-25T10:42:58.194087+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_tt04_3730.trc:
ORA-16191: Primary log shipping client not logged on standby
Error 16191 for archive log file 13 to 'DZZBCDBX'
2021-03-25T10:42:58.212673+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_tt04_3730.trc:
ORA-16191: Primary log shipping client not logged on standby
2021-03-25T10:42:58.548106+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_tt04_3730.trc:
ORA-16191: Primary log shipping client not logged on standby
2021-03-25T10:43:04.721823+08:00
2021-03-25T10:45:40.504865+08:00
WARNING: inbound connection timed out (ORA-3136)
2021-03-25T10:45:46.357905+08:00
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
------------------------------------------------------------
ARCH: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (16191)
ARCH: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
2021-03-25T10:45:47.318037+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_tt00_25880.trc:
ORA-16191: Primary log shipping client not logged on standby
krsg_check_connection: Error 16191 connecting to standby 'DZZBCDBX' (Process:TT00) (PID:25880)
2021-03-25T10:46:14.651442+08:00
IPC Send timeout detected. Sender: ospid 4929 [oracle@jtszdzzbvprd02 (PP9X)]
Receiver: inst 1 binc 10 ospid 12423
2021-03-25T10:46:14.669074+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_pp9x_4929.trc:
ORA-27508: IPC error sending a message
2021-03-25T10:47:05.246725+08:00
DBRM (ospid: 25628) waits for event 'latch: shared pool' for 23 secs.
DBRM (ospid: 25628) waits for latch 'shared pool' for 23 secs.
2021-03-25T10:47:05.247075+08:00
DBRM (ospid: 25628) is hung in an acceptable location (inwait|latch-get 0x5.ffff).
2021-03-25T10:47:12.958323+08:00
DBRM (ospid: 25628) waits for event 'latch: shared pool' for 39 secs.
DBRM (ospid: 25628) waits for latch 'shared pool' for 39 secs.
2021-03-25T10:47:44.189395+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_mmon_25756.trc (incident=1600417) (PDBNAME=CDB$ROOT):
ORA-00445: background process "m002" did not start after 120 seconds
Incident details in: /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/incident/incdir_1600417/dzzbcdb2_mmon_25756_i1600417.trc
2021-03-25T10:48:03.723402+08:00
IPC Send timeout detected. Sender: ospid 3168 [oracle@jtszdzzbvprd02 (PPA2)]
Receiver: inst 1 binc 10 ospid 17894
2021-03-25T10:48:03.735805+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_ppa2_3168.trc:
ORA-27508: IPC error sending a message
2021-03-25T10:48:13.055119+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/trace/dzzbcdb2_lmhb_25684.trc (incident=1600273) (PDBNAME=CDB$ROOT):
ORA-29771: process USER (OSID 8126) blocks DBRM (OSID 25628) for more than 70 seconds
Incident details in: /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb2/incident/incdir_1600273/dzzbcdb2_lmhb_25684_i1600273.trc
2021-03-25T10:48:19.354797+08:00
USER (ospid: 8126) is blocking DBRM (ospid: 25628) in a wait
LMHB (ospid: 25684) kills USER (ospid: 8126).
Please check LMHB trace file for more detail.
2021-03-25T10:48:44.351466+08:00
PMON failed to acquire latch, see PMON dump
2021-03-25T10:49:43.237884+08:00
PMON failed to acquire latch, see PMON dump
2021-03-25T10:50:24.440850+08:00
TNS-12537: TNS:connection closed
TNS-12537: TNS:connection closed
TNS-12537: TNS:connection closed
TNS-12537: TNS:connection closed
ns secondary err code: 12560
ns secondary err code: 12560
ns secondary err code: 12560
ns secondary err code: 12560
nt main err code: 0
nt main err code: 0
nt main err code: 0
nt secondary err code: 0
nt main err code: 0
nt secondary err code: 0
nt secondary err code: 0
nt OS err code: 0
nt secondary err code: 0
nt OS err code: 0
nt OS err code: 0
nt OS err code: 0
2021-03-25T10:53:31.321789+08:00
opiodr aborting process unknown ospid (13147) as a result of ORA-609
2021-03-25T10:53:31.321991+08:00
opiodr aborting process unknown ospid (12542) as a result of ORA-609
2021-03-25T10:53:31.322185+08:00
opiodr aborting process unknown ospid (12510) as a result of ORA-609
2021-03-25T10:53:31.322642+08:00
opiodr aborting process unknown ospid (12215) as a result of ORA-609
2021-03-25T10:53:33.115948+08:00
LGWR (ospid: 25700) waits for event 'enq: CF - contention' for 101 secs.
2021-03-25T10:53:33.116120+08:00
LGWR (ospid: 25700) is hung in an acceptable location (cfio 0x11.00).
2021-03-25T10:56:12.583520+08:00
Starting ORACLE instance (normal) (OS id: 5244)
2021-03-25T10:56:12.601354+08:00
CLI notifier numLatches:29 maxDescs:8289
2021-03-25T10:56:12.698158+08:00
**********************************************************************
2021-03-25T10:56:12.698289+08:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
2021-03-25T10:56:12.698509+08:00
Per process system memlock (soft) limit = UNLIMITED
2021-03-25T10:56:12.698629+08:00
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 35G
2021-03-25T10:56:12.698861+08:00
Available system pagesizes:
4K, 2048K
2021-03-25T10:56:12.699105+08:00
Supported system pagesize(s):
2021-03-25T10:56:12.699226+08:00
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2021-03-25T10:56:12.699346+08:00
4K Configured 6 9175046 NONE
2021-03-25T10:56:12.699577+08:00
2048K 0 17921 0 NONE
2021-03-25T10:56:12.699706+08:00
RECOMMENDATION:
2021-03-25T10:56:12.699847+08:00
1. For optimal performance, configure system with expected number
of pages for every supported system pagesize prior to the next
instance restart operation.
2021-03-25T10:56:12.779827+08:00
**********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 16
Number of processor cores in the system is 16
Number of processor sockets in the system is 4
2021-03-25T10:56:14.558881+08:00
Exafusion(Msgq-Auto) enabled
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
ARCH: Warning; less destinations available than specified
by LOG_ARCHIVE_MIN_SUCCEED_DEST init.ora parameter
2021-03-25T10:56:15.785197+08:00
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is enabled
2021-03-25T10:56:16.119376+08:00
NOTE: remote asm mode is remote (mode 0x2; from cluster type)
NOTE: Cluster configuration type = CLUSTER [4]
NUMA system with 4 nodes detected
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production.
ORACLE_HOME: /oracle/app/12.2.0/db_1
System name: Linux
Node name: jtszdzzbvprd02
Release: 3.10.0-1127.13.1.el7.x86_64
Version: #1 SMP Fri Jun 12 14:34:17 EDT 2020
Machine: x86_64
Using parameter settings in server-side spfile +DATAVG/DZZBCDB/PARAMETERFILE/spfile.272.960830541
System parameters with non-default values:
复制节点1：
10:44:23 节点1有swap交换告警 10:58:20被迫重启
2021-03-25T10:05:16.367810+08:00
Thread 1 advanced to log sequence 12883 (LGWR switch)
Current log# 3 seq# 12883 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_3.270.1040551003
Current log# 3 seq# 12883 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_3.601.1040551005
2021-03-25T10:05:17.142422+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12883 for destination LOG_ARCHIVE_DEST_2
2021-03-25T10:05:18.530878+08:00
Archived Log entry 196177 added for T-1.S-12882 ID 0x8ef7a2a LAD:1
2021-03-25T10:20:15.107680+08:00
Thread 1 advanced to log sequence 12884 (LGWR switch)
Current log# 7 seq# 12884 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_7.330.1040550719
Current log# 7 seq# 12884 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_7.549.1040550721
2021-03-25T10:20:16.008770+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12884 for destination LOG_ARCHIVE_DEST_2
2021-03-25T10:20:16.719214+08:00
Archived Log entry 196181 added for T-1.S-12883 ID 0x8ef7a2a LAD:1
2021-03-25T10:32:00.996251+08:00
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.31.24)(PORT=43538))
2021-03-25T10:34:01.186836+08:00
BIDPROD(4):WARNING: inbound connection timed out (ORA-3136)
2021-03-25T10:35:38.956322+08:00
Thread 1 advanced to log sequence 12885 (LGWR switch)
Current log# 8 seq# 12885 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_8.331.1040550787
Current log# 8 seq# 12885 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_8.562.1040550789
2021-03-25T10:37:22.962903+08:00
WARNING: Waiting on CRSD for password file, for 10 seconds, check CRS and OS logs
2021-03-25T10:37:24.682513+08:00
Archived Log entry 196185 added for T-1.S-12884 ID 0x8ef7a2a LAD:1
2021-03-25T10:38:02.456770+08:00
***********************************************************************
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 10:38:02
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.168.52)(PORT=18438))
2021-03-25T10:38:02.461110+08:00
WARNING: inbound connection timed out (ORA-3136)
2021-03-25T10:38:09.294139+08:00
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
------------------------------------------------------------
TT04: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (16191)
TT04: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
2021-03-25T10:38:09.300833+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_tt04_21135.trc:
ORA-16191: Primary log shipping client not logged on standby
Error 16191 for archive log file 8 to 'DZZBCDBX'
2021-03-25T10:38:09.422859+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_tt04_21135.trc:
ORA-16191: Primary log shipping client not logged on standby
2021-03-25T10:38:09.430656+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_tt04_21135.trc:
ORA-16191: Primary log shipping client not logged on standby
2021-03-25T10:38:18.222070+08:00
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
------------------------------------------------------------
ARCH: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (16191)
ARCH: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
2021-03-25T10:38:18.234115+08:00
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_tt00_27329.trc:
ORA-16191: Primary log shipping client not logged on standby
krsg_check_connection: Error 16191 connecting to standby 'DZZBCDBX' (Process:TT00) (PID:27329)
2021-03-25T10:39:43.749811+08:00
Dumping diagnostic data in directory=[cdmp_20210325103938], requested by (instance=2, osid=25628 (DBRM)), summary=[incident=1600169].
2021-03-25T10:39:49.736370+08:00
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 10:41:01
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.200.1.77)(PORT=59746))
2021-03-25T10:41:01.739858+08:00
SRMPROD(3):WARNING: inbound connection timed out (ORA-3136)
2021-03-25T10:43:55.399693+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12885 for destination LOG_ARCHIVE_DEST_2
2021-03-25T10:44:23.515748+08:00
WARNING: Heavy swapping observed on system in last 5 mins.
Heavy swapping can lead to timeouts, poor performance, and instance eviction.
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_dbrm_21300.trc (incident=2880145) (PDBNAME=CDB$ROOT):
ORA-00700: soft internal error, arguments: [kskvmstatact: excessive swapping observed], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/incident/incdir_2880145/dzzbcdb1_dbrm_21300_i2880145.trc
2021-03-25T10:44:38.674805+08:00
Dumping diagnostic data in directory=[cdmp_20210325104437], requested by (instance=1, osid=21300 (DBRM)), summary=[incident=2880145].
2021-03-25T10:46:40.522722+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 10:46:40
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T10:46:40.537285+08:00
opiodr aborting process unknown ospid (29487) as a result of ORA-609
2021-03-25T10:48:02.039191+08:00
Active Session History (ASH) performed an emergency flush. This may mean that ASH is undersized. If emergency flushes are a recurring issue, you may consider increasing ASH size by setting the value of _ASH_SIZE to a sufficiently large value. Currently, ASH size is 33554432 bytes. Both ASH size and the total number of emergency flushes since instance startup can be monitored by running the following query:
select total_size,awr_flush_emergency_count from v$ash_info;
2021-03-25T10:50:24.919619+08:00
LGWR (ospid: 21372) waits for event 'enq: CF - contention' for 74 secs.
2021-03-25T10:50:24.919800+08:00
LGWR (ospid: 21372) is hung in an acceptable location (cfio 0x11.00).
2021-03-25T10:50:31.162226+08:00
BIDPROD(4):Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_m005_20442.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-12751: cpu time or run time policy violation
2021-03-25T10:51:06.472077+08:00
opiodr aborting process unknown ospid (3965) as a result of ORA-609
2021-03-25T10:51:11.915306+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 69 secs.
2021-03-25T10:51:11.915445+08:00
CKPT (ospid: 21376) is hung in an acceptable location (inwait 0x1.ffff).
2021-03-25T10:51:30.374391+08:00
LGWR (ospid: 21372) has not called a wait for sub 0 secs.
2021-03-25T10:51:43.791733+08:00
GES: System Load is HIGH.
GES: Current load is 92.88 and high load threshold is 80.00
2021-03-25T10:52:02.062772+08:00
WARNING: Waiting on CRSD for password file, for 35 seconds, check CRS and OS logs
2021-03-25T10:53:17.611454+08:00
Thread 1 advanced to log sequence 12886 (LGWR switch)
Current log# 9 seq# 12886 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_9.332.1040550791
Current log# 9 seq# 12886 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_9.569.1040550795
2021-03-25T10:53:49.016281+08:00
Reconfiguration started (old inc 12, new inc 14)
List of instances (total 1) :
1
Dead instances (total 1) :
2
My inst 1
publish big name space - dead or down/up instance detected, invalidate domain 0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
* dead instance detected - domain 2 invalid = TRUE, need cdb-level instance recovery
* dead instance detected - domain 3 invalid = TRUE, need cdb-level instance recovery
* dead instance detected - domain 4 invalid = TRUE, need cdb-level instance recovery
Communication channels reestablished
2021-03-25T10:53:49.703597+08:00
IPC Send timeout to 2.2 inc 12 for msg type 65521 from opid 25
2021-03-25T10:53:49.703880+08:00
IPC Send timeout to 2.2 inc 12 for msg type 65521 from opid 25
2021-03-25T10:53:49.704073+08:00
IPC Send timeout to 2.2 inc 12 for msg type 65521 from opid 25
2021-03-25T10:53:49.704261+08:00
IPC Send timeout to 2.2 inc 12 for msg type 65521 from opid 25
2021-03-25T10:53:49.704457+08:00
IPC Send timeout to 2.2 inc 12 for msg type 65518 from opid 25
2021-03-25T10:53:49.704543+08:00
IPC Send timeout to 2.0 inc 12 for msg type 65521 from opid 23
2021-03-25T10:53:49.704750+08:00
IPC Send timeout to 2.2 inc 12 for msg type 65518 from opid 25
2021-03-25T10:53:49.704964+08:00
IPC Send timeout to 2.2 inc 12 for msg type 65518 from opid 25
2021-03-25T10:53:49.705143+08:00
IPC Send timeout to 2.2 inc 12 for msg type 44 from opid 25
2021-03-25T10:53:49.711324+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65521 from opid 24
2021-03-25T10:53:49.711566+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65521 from opid 24
2021-03-25T10:53:49.711747+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65521 from opid 24
2021-03-25T10:53:49.711928+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65521 from opid 24
2021-03-25T10:53:49.712109+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65521 from opid 24
2021-03-25T10:53:49.712293+08:00
IPC Send timeout to 2.1 inc 12 for msg type 36 from opid 24
2021-03-25T10:53:49.712466+08:00
IPC Send timeout to 2.1 inc 12 for msg type 36 from opid 24
2021-03-25T10:53:49.712659+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65518 from opid 24
2021-03-25T10:53:49.712830+08:00
IPC Send timeout to 2.1 inc 12 for msg type 44 from opid 24
2021-03-25T10:53:49.713006+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65518 from opid 24
2021-03-25T10:53:49.713176+08:00
IPC Send timeout to 2.1 inc 12 for msg type 44 from opid 24
2021-03-25T10:53:49.713356+08:00
IPC Send timeout to 2.1 inc 12 for msg type 65521 from opid 24
2021-03-25T10:53:49.813034+08:00
Master broadcasted resource hash value bitmaps
2021-03-25T10:53:57.269129+08:00
Non-local Process blocks cleaned out
2021-03-25T10:54:28.314026+08:00
WARNING: Waiting on CRSD for password file, for 57 seconds, check CRS and OS logs
2021-03-25T10:54:31.327808+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12886 for destination LOG_ARCHIVE_DEST_2
2021-03-25T10:54:36.742216+08:00
LMS 0: 65 GCS shadows cancelled, 8 closed, 0 Xw survived, skipped 0
2021-03-25T10:54:36.785177+08:00
LMS 1: 57 GCS shadows cancelled, 6 closed, 0 Xw survived, skipped 0
2021-03-25T10:54:44.887089+08:00
LGWR (ospid: 21372) waits for event 'enq: CF - contention' for 82 secs.
2021-03-25T10:54:44.887254+08:00
LGWR (ospid: 21372) is hung in an acceptable location (inwait 0x201.00).
2021-03-25T10:54:58.774781+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 77 secs.
2021-03-25T10:54:58.774909+08:00
CKPT (ospid: 21376) is hung in an acceptable location (inwait 0x1.ffff).
2021-03-25T10:55:06.595446+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 88 secs.
2021-03-25T10:55:15.593477+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 97 secs.
2021-03-25T10:55:23.947510+08:00
CLMN: clean deferred state objects - failed
2021-03-25T10:55:25.643364+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 107 secs.
2021-03-25T10:55:35.719441+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 117 secs.
2021-03-25T10:55:45.636338+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 127 secs.
2021-03-25T10:55:55.690446+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 137 secs.
2021-03-25T10:56:03.919053+08:00
LGWR (ospid: 21372) waits for event 'enq: CF - contention' for 161 secs.
2021-03-25T10:56:03.919179+08:00
LGWR (ospid: 21372) is hung in an acceptable location (inwait 0x201.00).
2021-03-25T10:56:05.751448+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 147 secs.
2021-03-25T10:56:15.660540+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 157 secs.
2021-03-25T10:56:20.968810+08:00
Reconfiguration started (old inc 14, new inc 16)
List of instances (total 2) :
1 2
New instances (total 1) :
2
My inst 1
Nested reconfiguration detected.
2021-03-25T10:56:25.727719+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 167 secs.
2021-03-25T10:56:35.653460+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 177 secs.
2021-03-25T10:56:45.660554+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 187 secs.
2021-03-25T10:56:55.674448+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 197 secs.
2021-03-25T10:57:05.757311+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 207 secs.
2021-03-25T10:57:14.006541+08:00
LGWR (ospid: 21372) waits for event 'enq: CF - contention' for 231 secs.
2021-03-25T10:57:14.006684+08:00
LGWR (ospid: 21372) is hung in an acceptable location (inwait 0x201.00).
2021-03-25T10:57:15.690314+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 217 secs.
2021-03-25T10:57:25.764313+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 227 secs.
2021-03-25T10:57:43.676018+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 245 secs.
2021-03-25T10:57:53.753238+08:00
CKPT (ospid: 21376) waits for event 'enq: XR - database force logging' for 255 secs.
2021-03-25T10:58:01.569845+08:00
LMON (ospid: 21328): terminating the instance due to error 484
2021-03-25T10:58:01.636175+08:00
System state dump requested by (instance=1, osid=21328 (LMON)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_diag_21288_20210325105801.trc
2021-03-25T10:58:10.146995+08:00
Instance terminated by LMON, pid = 21328
2021-03-25T10:58:10.827020+08:00
Warning: 2 processes are still attach to shmid 27:
(size: 24576 bytes, creator pid: 21117, last attach/detach pid: 15030)
2021-03-25T10:58:11.222864+08:00
License high water mark = 1308
2021-03-25T10:58:12.226096+08:00
USER (ospid: 15030): terminating the instance
2021-03-25T10:58:12.244313+08:00
Instance terminated by USER, pid = 15030
2021-03-25T10:58:20.799828+08:00
Starting ORACLE instance (normal) (OS id: 15488)
2021-03-25T10:58:20.818774+08:00
CLI notifier numLatches:29 maxDescs:8289
2021-03-25T10:58:20.879432+08:00
**********************************************************************
2021-03-25T10:58:20.879569+08:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
2021-03-25T10:58:20.879777+08:00
Per process system memlock (soft) limit = UNLIMITED
2021-03-25T10:58:20.879892+08:00
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 35G
2021-03-25T10:58:20.880110+08:00
Available system pagesizes:
4K, 2048K
2021-03-25T10:58:20.880354+08:00
Supported system pagesize(s):
2021-03-25T10:58:20.880469+08:00
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2021-03-25T10:58:20.880598+08:00
4K Configured 6 9175046 NONE
2021-03-25T10:58:20.880832+08:00
2048K 0 17921 0 NONE
2021-03-25T10:58:20.880944+08:00
RECOMMENDATION:
2021-03-25T10:58:20.881062+08:00
1. For optimal performance, configure system with expected number
of pages for every supported system pagesize prior to the next
instance restart operation.
2021-03-25T10:58:20.888047+08:00
**********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 16
Number of processor cores in the system is 16
Number of processor sockets in the system is 4
2021-03-25T10:58:22.208758+08:00
Exafusion(Msgq-Auto) enabled
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
ARCH: Warning; less destinations available than specified
by LOG_ARCHIVE_MIN_SUCCEED_DEST init.ora parameter
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is enabled
2021-03-25T10:58:23.314931+08:00
NOTE: remote asm mode is remote (mode 0x2; from cluster type)
NOTE: Cluster configuration type = CLUSTER [4]
NUMA system with 2 nodes detected
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production.
ORACLE_HOME: /oracle/app/12.2.0/db_1
System name: Linux
Node name: jtszdzzbvprd01
Release: 3.10.0-1127.13.1.el7.x86_64
Version: #1 SMP Fri Jun 12 14:34:17 EDT 2020
Machine: x86_64
Using parameter settings in server-side spfile +DATAVG/DZZBCDB/PARAMETERFILE/spfile.272.960830541
System parameters with non-default values:
processes = 5000
sga_max_size = 35G
nls_language = "AMERICAN"
nls_territory = "AMERICA"
sga_target = 35G
control_files = "+DATAVG/DZZBCDB/CONTROLFILE/new.ctl"
db_file_name_convert = "+DATADG/DZZBCDB/DATAFILE/"
db_file_name_convert = "+DATAVG/DZZBCDB/DATAFILE/"
db_file_name_convert = "+DATADG/DZZBCDB/TEMPFILE/"
db_file_name_convert = "+DATAVG/DZZBCDB/TEMPFILE/"
db_file_name_convert = "+DATADG/DZZBCDB/5EA3EA6F8C3E49F0E053241F000A2C70/DATAFILE/"
db_file_name_convert = "+DATAVG/DZZBCDB/4700A987085B3DFAE05387E5E50A8C7B/DATAFILE/"
db_file_name_convert = "+DATADG/DZZBCDB/5EA401811F483C9DE053251F000A1182/DATAFILE/"
db_file_name_convert = "+DATAVG/DZZBCDB/5EA401811F483C9DE053251F000A1182/DATAFILE/"
db_file_name_convert = "+DATADG/DZZBCDB/6171F59F822B6844E053241F000AB344/DATAFILE/"
db_file_name_convert = "+DATAVG/DZZBCDB/6171F59F822B6844E053241F000AB344/DATAFILE/"
db_file_name_convert = "+DATADG/DZZBCDB/6171F59F822B6844E053241F000AB344/DATAFILE/"
db_file_name_convert = "+DATAVG/DZZBCDB/6171F59F822B6844E053241F000AB344/TEMPFILE/"
log_file_name_convert = "+ARCHDG/DZZBCDBX/ONLINELOG/"
log_file_name_convert = "+DATAVG/DZZBCDB/ONLINELOG/"
log_file_name_convert = "+ARCHDG/DZZBCDBX/ONLINELOG/"
log_file_name_convert = "+ARCHDG/DZZBCDB/ONLINELOG/"
control_file_record_keep_time= 15
db_block_size = 8192
compatible = "12.2.0"
log_archive_dest_2 = "SERVICE=DZZBCDBX LGWR ASYNC REGISTER VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DZZBCDBX "
log_archive_dest_state_2 = "ENABLE"
fal_client = "DZZBCDB"
fal_server = "DZZBCDBX"
log_archive_config = "DG_CONFIG=(dzzbcdb,dzzbcdbx)"
log_buffer = 256M
archive_lag_target = 900
cluster_database = TRUE
db_create_file_dest = "+DATAVG"
db_recovery_file_dest = "+ARCHDG"
db_recovery_file_dest_size= 200000M
standby_file_management = "AUTO"
thread = 1
_allow_resetlogs_corruption= FALSE
undo_tablespace = "UNDOTBS1"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
dispatchers = "(PROTOCOL=TCP) (SERVICE=dzzbcdbXDB)"
local_listener = "(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.31.38)(PORT=1521))"
audit_file_dest = "/oracle/app/db/admin/dzzbcdb/adump"
audit_trail = "NONE"
db_name = "dzzbcdb"
open_cursors = 600
pga_aggregate_target = 10G
awr_pdb_autoflush_enabled= TRUE
awr_snapshot_time_offset = 1000000
_umf_remote_target_dblink= ""
_umf_remote_enabled = TRUE
_dbmsumf$$nn = "prim"
_dbmsumf$$p = "1"
_dbmsumf$$1x = "<UMF_SCHEMA>
<TOPOLOGY>
<TOPOLOGY_INST>
<TOPOLOGY_NAME>Topology_1</TOPOLOGY_NAME>
<TARGET_ID>2014871576</TARGET_ID>
<TOPOLOGY_VERSION>12</TOPOLOGY_VERSION>
<TOPOLOGY_STATE>0</TOPOLOGY_STATE>
</TOPOLOGY_INST>
</TOPOLOGY>"
_dbmsumf$$1x = "
<REGISTRATION>
<REGISTRATION_INST>
<TOPOLOGY_NAME>Topology_1</TOPOLOGY_NAME>
<NODE_NAME>prim</NODE_NAME>
<NODE_ID>2014871576</NODE_ID>
<NODE_TYPE>0</NODE_TYPE>
<AS_SOURCE>0</AS_SOURCE>
<AS_CANDIDATE_TARGET>0</AS_"
_dbmsumf$$1x = "CANDIDATE_TARGET>
<STATE>0</STATE>
</REGISTRATION_INST>
<REGISTRATION_INST>
<TOPOLOGY_NAME>Topology_1</TOPOLOGY_NAME>
<NODE_NAME>stby</NODE_NAME>
<NODE_ID>1730117407</NODE_ID>
<NODE_TYPE>0</NODE_TYPE>
<AS_SOURCE"
_dbmsumf$$1x = ">0</AS_SOURCE>
<AS_CANDIDATE_TARGET>0</AS_CANDIDATE_TARGET>
<STATE>0</STATE>
</REGISTRATION_INST>
</REGISTRATION>
<LINK>
<LINK_INST>
<TOPOLOGY_NAME>Topology_1</TOPOLOGY_NAME>
<FROM_NODE_ID>2014871576</FROM_NODE_ID>
"
_dbmsumf$$1x = " <TO_NODE_ID>1730117407</TO_NODE_ID>
<LINK_NAME>DBL_RACDB_TO_RACDBS</LINK_NAME>
</LINK_INST>
<LINK_INST>
<TOPOLOGY_NAME>Topology_1</TOPOLOGY_NAME>
<FROM_NODE_ID>1730117407</FROM_NODE_ID>
<TO_NODE_ID>2014871576</TO_NODE_ID>"
_dbmsumf$$1x = "
<LINK_NAME>DBL_RACDBS_TO_RACDB</LINK_NAME>
</LINK_INST>
</LINK>
<SERVICE>
<SERVICE_INST>
<TOPOLOGY_NAME>Topology_1</TOPOLOGY_NAME>
<NODE_ID>1730117407</NODE_ID>
<SERVICE_ID>1</SERVICE_ID>
</SERVICE_INST>
</SERVIC"
_dbmsumf$$1x = "E>
</UMF_SCHEMA>
"
diagnostic_dest = "/oracle/app/db"
enable_pluggable_database= TRUE
NOTE: remote asm mode is remote (mode 0x2; from cluster type)
2021-03-25T10:58:23.605992+08:00
Cluster Communication is configured to use IPs from: GPnP
IP: 169.254.161.7 Subnet: 169.254.0.0
KSIPC Loopback IP addresses(OSD):
127.0.0.1
KSIPC Available Transports: UDP:TCP
KSIPC: Client: KCL Transport: UDP
KSIPC: Client: DLM Transport: UDP
KSIPC CAPABILITIES :IPCLW:GRPAM:TOPO:DLL
KSXP: ksxpsg_ipclwtrans: 2 UDP
cluster interconnect IPC version: [IPCLW over UDP(mode 3) ]
IPC Vendor 1 proto 2
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 4.0
============================================================
NOTE: PatchLevel of this instance 0
============================================================
Starting background process PMON
2021-03-25T10:58:23.733743+08:00
PMON started with pid=2, OS id=15683
Starting background process CLMN
2021-03-25T10:58:23.832828+08:00
CLMN started with pid=3, OS id=15689
Starting background process PSP0
2021-03-25T10:58:23.864346+08:00
PSP0 started with pid=4, OS id=15700
Starting background process IPC0
2021-03-25T10:58:24.907697+08:00
IPC0 started with pid=5, OS id=15782
2021-03-25T10:58:24.907701+08:00
Starting background process VKTM
Starting background process GEN0
2021-03-25T10:58:24.946992+08:00
VKTM started with pid=6, OS id=15786 at elevated (RT) priority
2021-03-25T10:58:24.949717+08:00
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Starting background process MMAN
2021-03-25T10:58:24.975579+08:00
GEN0 started with pid=7, OS id=15793
2021-03-25T10:58:25.002756+08:00
MMAN started with pid=8, OS id=15797
Starting background process GEN1
Starting background process DIAG
2021-03-25T10:58:25.090055+08:00
GEN1 started with pid=10, OS id=15805_15809
Starting background process OFSD
2021-03-25T10:58:25.117354+08:00
DIAG started with pid=12, OS id=15813
Starting background process DBRM
2021-03-25T10:58:25.156922+08:00
OFSD started with pid=13, OS id=15817_15821
2021-03-25T10:58:25.158046+08:00
Oracle running with ofslib:'Oracle File Server Library' version=2
2021-03-25T10:58:25.187643+08:00
DBRM started with pid=15, OS id=15825
Starting background process VKRM
Starting background process PING
2021-03-25T10:58:25.215815+08:00
VKRM started with pid=16, OS id=15836
2021-03-25T10:58:25.284100+08:00
PING started with pid=17, OS id=15855
Starting background process SVCB
2021-03-25T10:58:25.311944+08:00
SVCB started with pid=18, OS id=15863
Starting background process ACMS
2021-03-25T10:58:25.342530+08:00
ACMS started with pid=19, OS id=15870
Starting background process PMAN
2021-03-25T10:58:25.371039+08:00
PMAN started with pid=20, OS id=15890
Starting background process DIA0
2021-03-25T10:58:25.400451+08:00
DIA0 started with pid=21, OS id=15894
Starting background process LMON
Starting background process LMD0
2021-03-25T10:58:25.429217+08:00
LMON started with pid=22, OS id=15900
2021-03-25T10:58:25.459923+08:00
LMD0 started with pid=23, OS id=15906
Starting background process LMD1
2021-03-25T10:58:25.462007+08:00
LMS0 started with pid=24, OS id=15908 at elevated (RT) priority
2021-03-25T10:58:25.462372+08:00
LMS1 started with pid=25, OS id=15910 at elevated (RT) priority
2021-03-25T10:58:25.462562+08:00
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [15360 - 20480]
Starting background process RMS0
2021-03-25T10:58:25.486479+08:00
LMD1 started with pid=26, OS id=15922
Starting background process LMHB
2021-03-25T10:58:25.515443+08:00
RMS0 started with pid=27, OS id=15926
Starting background process LCK1
2021-03-25T10:58:25.544580+08:00
LMHB started with pid=28, OS id=15931
Starting background process DBW0
2021-03-25T10:58:25.575792+08:00
LCK1 started with pid=29, OS id=15935
Starting background process DBW1
2021-03-25T10:58:25.607451+08:00
DBW0 started with pid=30, OS id=15939
2021-03-25T10:58:25.645335+08:00
DBW1 started with pid=31, OS id=15943
Starting background process LGWR
Starting background process CKPT
2021-03-25T10:58:25.678596+08:00
LGWR started with pid=32, OS id=15947
2021-03-25T10:58:25.718424+08:00
CKPT started with pid=33, OS id=15951
Starting background process SMON
2021-03-25T10:58:25.787646+08:00
LGWR slave LG00 created with pid=34, OS pid=15955
2021-03-25T10:58:25.879114+08:00
SMON started with pid=35, OS id=15960
LGWR slave LG01 created with pid=36, OS pid=15966
2021-03-25T10:58:25.910682+08:00
Starting background process SMCO
Starting background process RECO
2021-03-25T10:58:25.938741+08:00
SMCO started with pid=37, OS id=15978
2021-03-25T10:58:25.970189+08:00
RECO started with pid=38, OS id=15982
Starting background process LREG
2021-03-25T10:58:26.031957+08:00
LREG started with pid=40, OS id=15991
Starting background process PXMN
2021-03-25T10:58:26.095117+08:00
PXMN started with pid=42, OS id=15999
Starting background process RBAL
2021-03-25T10:58:26.124991+08:00
RBAL started with pid=43, OS id=16003
Starting background process ASMB
2021-03-25T10:58:26.158301+08:00
ASMB started with pid=44, OS id=16007
Starting background process FENC
Starting background process MMON
2021-03-25T10:58:26.188951+08:00
FENC started with pid=45, OS id=16011
Starting background process MMNL
2021-03-25T10:58:26.228379+08:00
MMON started with pid=46, OS id=16015
2021-03-25T10:58:26.297530+08:00
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
2021-03-25T10:58:26.297563+08:00
MMNL started with pid=47, OS id=16019
starting up 1 shared server(s) ...
2021-03-25T10:58:27.788912+08:00
lmon registered with NM - instance number 1 (internal mem no 0)
2021-03-25T10:58:28.202583+08:00
Using default pga_aggregate_limit of 20480 MB
2021-03-25T10:58:28.331887+08:00
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [15360 - 20480]
2021-03-25T10:58:28.419890+08:00
* allocate domain 0, valid ? 0
Attached to domain 0 (addr: 0x11d6ad640)
2021-03-25T10:58:28.426950+08:00
Reconfiguration started (old inc 0, new inc 20)
List of instances (total 2) :
1 2
My inst 1 (I'm a new instance)
Global Resource Directory frozen
* Setting GES domain 0
Switching DLM PDB isolation (kjidomena: 0 -> 5, mm: NONE -> NORM)
Communication channels reestablished
2021-03-25T10:58:28.731281+08:00
* domain 0 valid = 0 (flags x224, pdb flags x0) according to instance 2
2021-03-25T10:58:28.734794+08:00
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
2021-03-25T10:58:28.768358+08:00
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived, skipped 0
2021-03-25T10:58:28.768389+08:00
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived, skipped 0
Set master node info
2021-03-25T10:58:28.787812+08:00
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
2021-03-25T10:58:28.906965+08:00
Reconfiguration complete (total time 0.5 secs)
2021-03-25T10:58:28.908015+08:00
Starting background process LCK0
2021-03-25T10:58:28.936582+08:00
LCK0 started with pid=57, OS id=16100
2021-03-25T10:58:28.953372+08:00
NOTE: ASMB0 registering with ASM instance as Flex client 0xffffffffffffffff (reg:3413627517) (new connection)
2021-03-25T10:58:28.960004+08:00
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Starting background process RSMN
2021-03-25T10:58:29.241930+08:00
RSMN started with pid=58, OS id=16134
Starting background process TMON
2021-03-25T10:58:29.331625+08:00
TMON started with pid=60, OS id=16145
ORACLE_BASE from environment = /oracle/app/db
2021-03-25T10:58:29.733680+08:00
NOTE: ASMB0 (16007) connected to ASM instance +ASM1, osid: 16105 (Flex mode; client id 0x10003)
NOTE: initiating MARK startup
Starting background process MARK
2021-03-25T10:58:29.796140+08:00
MARK started with pid=59, OS id=16177
2021-03-25T10:58:29.850379+08:00
NOTE: MARK has subscribed
2021-03-25T10:58:30.212247+08:00
NOTE: ASMB mounting group 2 (DATAVG)
NOTE: ASM background process initiating disk discovery for grp 2 (reqid:0)
2021-03-25T10:58:30.320030+08:00
KSXPPING: KSXP selected for Ping
2021-03-25T10:58:30.325883+08:00
NOTE: Assigning number (2,6) to disk (/dev/oracleasm/disks/DATADISK9)
NOTE: Assigning number (2,7) to disk (/dev/oracleasm/disks/DATADISK8)
NOTE: Assigning number (2,4) to disk (/dev/oracleasm/disks/DATADISK5)
NOTE: Assigning number (2,2) to disk (/dev/oracleasm/disks/DATADISK3)
NOTE: Assigning number (2,3) to disk (/dev/oracleasm/disks/DATADISK4)
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/DATADISK2)
NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/DATADISK1)
NOTE: Assigning number (2,5) to disk (/dev/oracleasm/disks/DATADISK6)
SUCCESS: mounted group 2 (DATAVG)
NOTE: grp 2 disk 6: DATAVG_0006 path:/dev/oracleasm/disks/DATADISK9
NOTE: grp 2 disk 7: DATAVG_0007 path:/dev/oracleasm/disks/DATADISK8
NOTE: grp 2 disk 4: DATAVG_0004 path:/dev/oracleasm/disks/DATADISK5
NOTE: grp 2 disk 2: DATAVG_0002 path:/dev/oracleasm/disks/DATADISK3
NOTE: grp 2 disk 3: DATAVG_0003 path:/dev/oracleasm/disks/DATADISK4
NOTE: grp 2 disk 1: DATAVG_0001 path:/dev/oracleasm/disks/DATADISK2
NOTE: grp 2 disk 0: DATAVG_0000 path:/dev/oracleasm/disks/DATADISK1
NOTE: grp 2 disk 5: DATAVG_0005 path:/dev/oracleasm/disks/DATADISK6
2021-03-25T10:58:30.484836+08:00
NOTE: dependency between database dzzbcdb and diskgroup resource ora.DATAVG.dg is established
2021-03-25T10:58:31.086706+08:00
ALTER SYSTEM SET remote_listener=' jtszdzzbp-scan:1526' SCOPE=MEMORY SID='dzzbcdb1';
ALTER DATABASE MOUNT /* db agent *//* {0:13:991} */
2021-03-25T10:58:35.265603+08:00
Network throttle feature is disabled as mount time
2021-03-25T10:58:35.449349+08:00
Successful mount of redo thread 1, with mount id 160225477
2021-03-25T10:58:35.460639+08:00
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
Completed: ALTER DATABASE MOUNT /* db agent *//* {0:13:991} */
ALTER DATABASE OPEN /* db agent *//* {0:13:991} */
2021-03-25T10:58:47.857653+08:00
* validated domain 0, flags = 0x200
2021-03-25T10:58:49.404739+08:00
Ping without log force is disabled:
not an Exadata system.
Picked broadcast on commit scheme to generate SCNs
Endian type of dictionary set to little
2021-03-25T10:58:49.502950+08:00
LGWR: STARTING ARCH PROCESSES
Starting background process ARC0
2021-03-25T10:58:49.527699+08:00
TT00: Gap Manager starting (PID:16880)
2021-03-25T10:58:49.554250+08:00
ARC0 started with pid=66, OS id=16884
2021-03-25T10:58:49.564371+08:00
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
2021-03-25T10:58:49.564656+08:00
ARC0: Becoming a 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
2021-03-25T10:58:49.585498+08:00
ARCH: STARTING ARCH PROCESSES
2021-03-25T10:58:49.600162+08:00
NOTE: ASMB mounting group 1 (ARCHDG)
NOTE: ASM background process initiating disk discovery for grp 1 (reqid:0)
Starting background process ARC1
NOTE: Assigning number (1,0) to disk (/dev/oracleasm/disks/ARCHDISK)
SUCCESS: mounted group 1 (ARCHDG)
NOTE: grp 1 disk 0: ARCHDG_0000 path:/dev/oracleasm/disks/ARCHDISK
2021-03-25T10:58:49.637893+08:00
NOTE: Under CF enqueue, no dependency request for disk group ARCHDG
Starting background process ARC2
2021-03-25T10:58:49.643818+08:00
ARC1 started with pid=69, OS id=16896
Starting background process ARC3
2021-03-25T10:58:49.671460+08:00
ARC2 started with pid=70, OS id=16904
2021-03-25T10:58:49.698875+08:00
ARC1: Archival started
2021-03-25T10:58:49.698875+08:00
ARC3 started with pid=71, OS id=16908
ARC2: Archival started
ARC3: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
2021-03-25T10:58:49.726200+08:00
Thread 1 advanced to log sequence 12888 (thread open)
2021-03-25T10:58:49.745282+08:00
NOTE: dependency between database dzzbcdb and diskgroup resource ora.ARCHDG.dg is established
2021-03-25T10:58:50.029507+08:00
Thread 1 opened at log sequence 12888
Current log# 11 seq# 12888 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_11.334.1040550801
Current log# 11 seq# 12888 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_11.571.1040550803
Successful open of redo thread 1
2021-03-25T10:58:50.030736+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
2021-03-25T10:58:50.313170+08:00
Archived Log entry 196191 added for T-1.S-12886 ID 0x8ef7a2a LAD:1
2021-03-25T10:58:50.329794+08:00
Archived Log entry 196192 added for T-1.S-12887 ID 0x8ef7a2a LAD:1
2021-03-25T10:58:51.721392+08:00
[16176] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:1942170767 end:1942171426 diff:659 ms (0.7 seconds)
Verifying minimum file header compatibility for tablespace encryption for pdb 1..
Verifying file header compatibility for tablespace encryption completed for pdb 1
Database Characterset is AL32UTF8
No Resource Manager plan active
2021-03-25T10:58:52.713919+08:00
Starting background process GTX0
2021-03-25T10:58:52.742757+08:00
GTX0 started with pid=79, OS id=17001
Starting background process RCBG
2021-03-25T10:58:52.932895+08:00
RCBG started with pid=80, OS id=17012
2021-03-25T10:58:52.935465+08:00
replication_dependency_tracking turned off (no async multimaster replication found)
2021-03-25T10:58:53.122007+08:00
ARC1: Standby redo logfile selected for thread 1 sequence 12887 for destination LOG_ARCHIVE_DEST_2
Starting background process AQPC
2021-03-25T10:58:53.401803+08:00
AQPC started with pid=81, OS id=17022
2021-03-25T10:58:53.413947+08:00
ARC2: Standby redo logfile selected for thread 1 sequence 12886 for destination LOG_ARCHIVE_DEST_2
2021-03-25T10:58:53.688326+08:00
* allocate domain 2, valid ? 1
all enqueues go to domain 0
2021-03-25T10:58:54.269205+08:00
PDB$SEED(2):Autotune of undo retention is turned on.
2021-03-25T10:58:54.693280+08:00
PDB$SEED(2):attach called for domid 2 (domuid: 0x843797d6, options: 0x4, pid: 16176)
PDB$SEED(2):queued attach broadcast request 0x111799010
PDB$SEED(2):Endian type of dictionary set to little
2021-03-25T10:58:55.815218+08:00
PDB$SEED(2):Undo initialization finished serial:0 start:1942175503 end:1942175503 diff:0 ms (0.0 seconds)
PDB$SEED(2):Database Characterset for PDB$SEED is AL32UTF8
PDB$SEED(2):Opatch validation is skipped for PDB PDB$SEED (con_id=0)
2021-03-25T10:58:56.587060+08:00
* allocate domain 4, valid ? 1
all enqueues go to domain 0
2021-03-25T10:58:56.645074+08:00
PDB$SEED(2):Opening pdb with no Resource Manager plan active
2021-03-25T10:58:56.724824+08:00
* allocate domain 3, valid ? 1
all enqueues go to domain 0
2021-03-25T10:58:57.127272+08:00
BIDPROD(4):Autotune of undo retention is turned on.
2021-03-25T10:58:57.391393+08:00
SRMPROD(3):Autotune of undo retention is turned on.
2021-03-25T10:58:57.434583+08:00
BIDPROD(4):attach called for domid 4 (domuid: 0x487606da, options: 0x4, pid: 17045)
BIDPROD(4):queued attach broadcast request 0x111798fb8
2021-03-25T10:58:57.535737+08:00
SRMPROD(3):attach called for domid 3 (domuid: 0x62ff8823, options: 0x4, pid: 17041)
SRMPROD(3):queued attach broadcast request 0x111798f60
2021-03-25T10:58:57.744818+08:00
Unable to obtain current patch information due to error: 20003, ORA-20003: Configuring job Load_opatch_inventory_1on node and on instancefailed
ORA-06512: at "SYS.DBMS_QOPATCH", line 777
ORA-06512: at "SYS.DBMS_QOPATCH", line 479
ORA-06512: at "SYS.DBMS_QOPATCH", line 455
ORA-06512: at "SYS.DBMS_QOPATCH", line 574
ORA-06512: at "SYS.DBMS_QOPATCH", line 2247
===========================================================
Dumping current patch information
===========================================================
Unable to obtain current patch information due to error: 20003
===========================================================
2021-03-25T10:58:57.956311+08:00
BIDPROD(4):Endian type of dictionary set to little
2021-03-25T10:58:58.052283+08:00
SRMPROD(3):Endian type of dictionary set to little
2021-03-25T10:59:00.066060+08:00
SRMPROD(3):[17041] Successfully onlined Undo Tablespace 5.
2021-03-25T10:59:00.107149+08:00
BIDPROD(4):[17045] Successfully onlined Undo Tablespace 5.
2021-03-25T10:59:00.178367+08:00
SRMPROD(3):Undo initialization finished serial:0 start:1942178365 end:1942179866 diff:1501 ms (1.5 seconds)
2021-03-25T10:59:00.186154+08:00
BIDPROD(4):Undo initialization finished serial:0 start:1942178277 end:1942179874 diff:1597 ms (1.6 seconds)
2021-03-25T10:59:00.189635+08:00
SRMPROD(3):Database Characterset for SRMPROD is AL32UTF8
2021-03-25T10:59:00.199280+08:00
BIDPROD(4):Database Characterset for BIDPROD is AL32UTF8
2021-03-25T10:59:00.689746+08:00
SRMPROD(3):Opatch validation is skipped for PDB SRMPROD (con_id=0)
2021-03-25T10:59:00.698402+08:00
BIDPROD(4):Opatch validation is skipped for PDB BIDPROD (con_id=0)
2021-03-25T10:59:02.699779+08:00
BIDPROD(4):Opening pdb with no Resource Manager plan active
2021-03-25T10:59:02.907772+08:00
SRMPROD(3):Opening pdb with no Resource Manager plan active
2021-03-25T10:59:03.958699+08:00
Pluggable database BIDPROD opened read write
2021-03-25T10:59:05.003124+08:00
Pluggable database SRMPROD opened read write
2021-03-25T10:59:05.237706+08:00
alter pluggable database all open
Completed: alter pluggable database all open
Starting background process CJQ0
Completed: ALTER DATABASE OPEN /* db agent *//* {0:13:991} */
2021-03-25T10:59:05.304113+08:00
CJQ0 started with pid=167, OS id=18150
2021-03-25T10:59:09.736456+08:00
BIDPROD(4):Shared IO Pool defaulting to 512MB. Trying to get it from Buffer Cache for process 18984.
2021-03-25T10:59:14.982293+08:00
WARNING: too many parse errors, count=100 SQL hash=0x750004bb
PARSE ERROR: ospid=17811, error=933 for statement:
2021-03-25T10:59:14.982578+08:00
DELETE FROM wri$_adv_sqlt_rtn_planWHERE task_id = :tid AND exec_name = :execution_name
Additional information: hd=0x97bb24e8 phd=0x97c03468 flg=0x28 cisid=0 sid=0 ciuid=0 uid=0
2021-03-25T10:59:14.982875+08:00
----- PL/SQL Call Stack -----
object line object
handle number name
0xcfae0f38 259 type body SYS.WRI$_ADV_SQLTUNE.SUB_DELETE_EXECUTION
0x86acfaa0 2134 package body SYS.PRVT_ADVISOR.COMMON_DELETE_TASK
0x86acfaa0 7342 package body SYS.PRVT_ADVISOR.DELETE_EXPIRED_TASKS
0x96d88198 1 anonymous block
2021-03-25T10:59:17.876295+08:00
WARNING: too many parse errors, count=200 SQL hash=0x750004bb
PARSE ERROR: ospid=17811, error=933 for statement:
2021-03-25T10:59:17.876567+08:00
DELETE FROM wri$_adv_sqlt_rtn_planWHERE task_id = :tid AND exec_name = :execution_name
Additional information: hd=0x97bb24e8 phd=0x97c03468 flg=0x28 cisid=0 sid=0 ciuid=0 uid=0
2021-03-25T10:59:17.876829+08:00
----- PL/SQL Call Stack -----
object line object
handle number name
0xcfae0f38 259 type body SYS.WRI$_ADV_SQLTUNE.SUB_DELETE_EXECUTION
0x86acfaa0 2134 package body SYS.PRVT_ADVISOR.COMMON_DELETE_TASK
0x86acfaa0 7342 package body SYS.PRVT_ADVISOR.DELETE_EXPIRED_TASKS
0x96d88198 1 anonymous block
2021-03-25T10:59:20.520942+08:00
WARNING: too many parse errors, count=300 SQL hash=0x750004bb
PARSE ERROR: ospid=17811, error=933 for statement:
2021-03-25T10:59:20.521215+08:00
DELETE FROM wri$_adv_sqlt_rtn_planWHERE task_id = :tid AND exec_name = :execution_name
Additional information: hd=0x97bb24e8 phd=0x97c03468 flg=0x28 cisid=0 sid=0 ciuid=0 uid=0
2021-03-25T10:59:20.521849+08:00
----- PL/SQL Call Stack -----
object line object
handle number name
0xcfae0f38 259 type body SYS.WRI$_ADV_SQLTUNE.SUB_DELETE_EXECUTION
0x86acfaa0 2134 package body SYS.PRVT_ADVISOR.COMMON_DELETE_TASK
0x86acfaa0 7342 package body SYS.PRVT_ADVISOR.DELETE_EXPIRED_TASKS
0x96d88198 1 anonymous block
2021-03-25T11:00:05.921672+08:00
ALTER SYSTEM SET remote_listener=' jtszdzzbp-scan:1526' SCOPE=MEMORY SID='dzzbcdb1';
2021-03-25T11:00:05.930933+08:00
ALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='dzzbcdb1';
2021-03-25T11:00:17.749427+08:00
ALTER SYSTEM ARCHIVE LOG
2021-03-25T11:00:17.810070+08:00
Thread 1 advanced to log sequence 12889 (LGWR switch)
Current log# 1 seq# 12889 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_1.271.1040550995
Current log# 1 seq# 12889 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_1.604.1040550997
2021-03-25T11:00:18.324787+08:00
Archived Log entry 196197 added for T-1.S-12888 ID 0x8ef7a2a LAD:1
2021-03-25T11:00:18.747872+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12889 for destination LOG_ARCHIVE_DEST_2
2021-03-25T11:00:18.841690+08:00
ARC3: Standby redo logfile selected for thread 1 sequence 12888 for destination LOG_ARCHIVE_DEST_2
2021-03-25T11:01:38.836681+08:00
Starting control autobackup
2021-03-25T11:01:48.120113+08:00
Control autobackup written to SBT_TAPE device
comment 'API Version 2.0,MMS Version 5.0.0.0'
media '@aaaao'
handle 'c-52949962-20210325-0a'
2021-03-25T11:04:29.159193+08:00
ALTER SYSTEM ARCHIVE LOG
2021-03-25T11:04:29.396277+08:00
Thread 1 advanced to log sequence 12890 (LGWR switch)
Current log# 2 seq# 12890 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_2.263.1040551137
Current log# 2 seq# 12890 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_2.626.1040551139
2021-03-25T11:04:29.846954+08:00
Archived Log entry 196202 added for T-1.S-12889 ID 0x8ef7a2a LAD:1
2021-03-25T11:04:30.179335+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12890 for destination LOG_ARCHIVE_DEST_2
2021-03-25T11:04:31.891045+08:00
Archived Log entry 196203 added for T-2.S-13096 ID 0x8ef7a2a LAD:1
2021-03-25T11:04:44.208304+08:00
Starting control autobackup
2021-03-25T11:04:52.109734+08:00
Control autobackup written to SBT_TAPE device
comment 'API Version 2.0,MMS Version 5.0.0.0'
media '@aaaao'
handle 'c-52949962-20210325-0b'
2021-03-25T11:06:08.016664+08:00
SRMPROD(3):SMON: Parallel transaction recovery tried
2021-03-25T11:16:24.431473+08:00
Thread 1 advanced to log sequence 12891 (LGWR switch)
Current log# 3 seq# 12891 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_3.270.1040551003
Current log# 3 seq# 12891 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_3.601.1040551005
2021-03-25T11:16:26.594443+08:00
Archived Log entry 196206 added for T-1.S-12890 ID 0x8ef7a2a LAD:1
2021-03-25T11:16:30.138834+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12891 for destination LOG_ARCHIVE_DEST_2
2021-03-25T11:31:12.146091+08:00
Thread 1 advanced to log sequence 12892 (LGWR switch)
Current log# 7 seq# 12892 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_7.330.1040550719
Current log# 7 seq# 12892 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_7.549.1040550721
2021-03-25T11:31:14.209265+08:00
Archived Log entry 196210 added for T-1.S-12891 ID 0x8ef7a2a LAD:1
2021-03-25T11:31:21.071192+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12892 for destination LOG_ARCHIVE_DEST_2
2021-03-25T11:46:05.316834+08:00
Thread 1 advanced to log sequence 12893 (LGWR switch)
Current log# 8 seq# 12893 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_8.331.1040550787
Current log# 8 seq# 12893 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_8.562.1040550789
2021-03-25T11:46:06.484654+08:00
Archived Log entry 196214 added for T-1.S-12892 ID 0x8ef7a2a LAD:1
2021-03-25T11:46:07.034031+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12893 for destination LOG_ARCHIVE_DEST_2
2021-03-25T12:01:06.826426+08:00
Thread 1 advanced to log sequence 12894 (LGWR switch)
Current log# 9 seq# 12894 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_9.332.1040550791
Current log# 9 seq# 12894 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_9.569.1040550795
2021-03-25T12:01:07.674148+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12894 for destination LOG_ARCHIVE_DEST_2
2021-03-25T12:01:07.699807+08:00
Archived Log entry 196218 added for T-1.S-12893 ID 0x8ef7a2a LAD:1
2021-03-25T12:16:05.615655+08:00
Thread 1 advanced to log sequence 12895 (LGWR switch)
Current log# 10 seq# 12895 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_10.333.1040550797
Current log# 10 seq# 12895 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_10.566.1040550799
2021-03-25T12:16:06.388717+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12895 for destination LOG_ARCHIVE_DEST_2
2021-03-25T12:16:06.584766+08:00
Archived Log entry 196222 added for T-1.S-12894 ID 0x8ef7a2a LAD:1
2021-03-25T12:31:07.401758+08:00
Thread 1 advanced to log sequence 12896 (LGWR switch)
Current log# 11 seq# 12896 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_11.334.1040550801
Current log# 11 seq# 12896 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_11.571.1040550803
2021-03-25T12:31:08.167551+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12896 for destination LOG_ARCHIVE_DEST_2
2021-03-25T12:31:08.803177+08:00
Archived Log entry 196226 added for T-1.S-12895 ID 0x8ef7a2a LAD:1
2021-03-25T12:46:05.745596+08:00
Thread 1 advanced to log sequence 12897 (LGWR switch)
Current log# 1 seq# 12897 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_1.271.1040550995
Current log# 1 seq# 12897 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_1.604.1040550997
2021-03-25T12:46:06.477008+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12897 for destination LOG_ARCHIVE_DEST_2
2021-03-25T12:46:07.371670+08:00
Archived Log entry 196230 added for T-1.S-12896 ID 0x8ef7a2a LAD:1
2021-03-25T13:00:17.154890+08:00
ALTER SYSTEM ARCHIVE LOG
2021-03-25T13:00:19.163063+08:00
Thread 1 advanced to log sequence 12898 (LGWR switch)
Current log# 2 seq# 12898 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_2.263.1040551137
Current log# 2 seq# 12898 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_2.626.1040551139
2021-03-25T13:00:20.035794+08:00
Archived Log entry 196235 added for T-2.S-13104 ID 0x8ef7a2a LAD:1
2021-03-25T13:00:20.502998+08:00
Archived Log entry 196236 added for T-1.S-12897 ID 0x8ef7a2a LAD:1
2021-03-25T13:00:24.959419+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12898 for destination LOG_ARCHIVE_DEST_2
2021-03-25T13:02:27.065421+08:00
Starting control autobackup
2021-03-25T13:02:43.730033+08:00
Control autobackup written to SBT_TAPE device
comment 'API Version 2.0,MMS Version 5.0.0.0'
media '@aaaao'
handle 'c-52949962-20210325-0c'
2021-03-25T13:05:28.284537+08:00
ALTER SYSTEM ARCHIVE LOG
2021-03-25T13:05:28.323778+08:00
Thread 1 advanced to log sequence 12899 (LGWR switch)
Current log# 3 seq# 12899 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_3.270.1040551003
Current log# 3 seq# 12899 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_3.601.1040551005
2021-03-25T13:05:28.877033+08:00
Archived Log entry 196238 added for T-1.S-12898 ID 0x8ef7a2a LAD:1
2021-03-25T13:05:29.061984+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12899 for destination LOG_ARCHIVE_DEST_2
2021-03-25T13:05:43.798372+08:00
Starting control autobackup
2021-03-25T13:05:51.918760+08:00
Control autobackup written to SBT_TAPE device
comment 'API Version 2.0,MMS Version 5.0.0.0'
media '@aaaao'
handle 'c-52949962-20210325-0d'
2021-03-25T13:20:23.304146+08:00
Thread 1 advanced to log sequence 12900 (LGWR switch)
Current log# 7 seq# 12900 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_7.330.1040550719
Current log# 7 seq# 12900 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_7.549.1040550721
2021-03-25T13:20:24.163364+08:00
Archived Log entry 196242 added for T-1.S-12899 ID 0x8ef7a2a LAD:1
2021-03-25T13:20:24.405254+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12900 for destination LOG_ARCHIVE_DEST_2
2021-03-25T13:35:24.712785+08:00
Thread 1 advanced to log sequence 12901 (LGWR switch)
Current log# 8 seq# 12901 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_8.331.1040550787
Current log# 8 seq# 12901 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_8.562.1040550789
2021-03-25T13:35:25.902085+08:00
Archived Log entry 196246 added for T-1.S-12900 ID 0x8ef7a2a LAD:1
2021-03-25T13:35:26.768304+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12901 for destination LOG_ARCHIVE_DEST_2
2021-03-25T13:50:25.899220+08:00
Thread 1 advanced to log sequence 12902 (LGWR switch)
Current log# 9 seq# 12902 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_9.332.1040550791
Current log# 9 seq# 12902 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_9.569.1040550795
2021-03-25T13:50:27.915222+08:00
Archived Log entry 196250 added for T-1.S-12901 ID 0x8ef7a2a LAD:1
2021-03-25T13:50:28.596472+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12902 for destination LOG_ARCHIVE_DEST_2
2021-03-25T14:05:24.457844+08:00
Thread 1 advanced to log sequence 12903 (LGWR switch)
Current log# 10 seq# 12903 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_10.333.1040550797
Current log# 10 seq# 12903 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_10.566.1040550799
2021-03-25T14:05:25.213018+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12903 for destination LOG_ARCHIVE_DEST_2
2021-03-25T14:05:27.857140+08:00
Archived Log entry 196254 added for T-1.S-12902 ID 0x8ef7a2a LAD:1
2021-03-25T14:20:25.737713+08:00
Thread 1 advanced to log sequence 12904 (LGWR switch)
Current log# 11 seq# 12904 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_11.334.1040550801
Current log# 11 seq# 12904 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_11.571.1040550803
2021-03-25T14:20:26.578998+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12904 for destination LOG_ARCHIVE_DEST_2
2021-03-25T14:20:29.348141+08:00
Archived Log entry 196260 added for T-1.S-12903 ID 0x8ef7a2a LAD:1
2021-03-25T14:29:22.455189+08:00
***********************************************************************
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:29:22
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505
TNS-00505: Operation timed out
nt secondary err code: 110
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.168.35)(PORT=33366))
2021-03-25T14:35:24.012351+08:00
Thread 1 advanced to log sequence 12905 (LGWR switch)
Current log# 1 seq# 12905 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_1.271.1040550995
Current log# 1 seq# 12905 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_1.604.1040550997
2021-03-25T14:35:25.091513+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12905 for destination LOG_ARCHIVE_DEST_2
2021-03-25T14:35:26.221639+08:00
Archived Log entry 196264 added for T-1.S-12904 ID 0x8ef7a2a LAD:1
2021-03-25T14:47:39.921734+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:47:39
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:47:39.937872+08:00
opiodr aborting process unknown ospid (20534) as a result of ORA-609
2021-03-25T14:47:56.017216+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:47:56
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:47:56.039515+08:00
opiodr aborting process unknown ospid (20835) as a result of ORA-609
2021-03-25T14:48:07.017829+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:48:07
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:48:07.025390+08:00
opiodr aborting process unknown ospid (20822) as a result of ORA-609
2021-03-25T14:49:31.346080+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:49:31
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:49:31.353981+08:00
opiodr aborting process unknown ospid (21893) as a result of ORA-609
2021-03-25T14:49:51.378547+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
2021-03-25T14:49:51.589188+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
2021-03-25T14:49:52.630875+08:00
Time: 25-MAR-2021 14:49:51
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:49:54.286908+08:00
Time: 25-MAR-2021 14:49:51
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:49:55.940859+08:00
opiodr aborting process unknown ospid (22002) as a result of ORA-609
2021-03-25T14:49:56.122594+08:00
opiodr aborting process unknown ospid (22008) as a result of ORA-609
2021-03-25T14:50:35.803686+08:00
Thread 1 advanced to log sequence 12906 (LGWR switch)
Current log# 2 seq# 12906 mem# 0: +DATAVG/DZZBCDB/ONLINELOG/group_2.263.1040551137
Current log# 2 seq# 12906 mem# 1: +ARCHDG/DZZBCDB/ONLINELOG/group_2.626.1040551139
2021-03-25T14:51:35.582503+08:00
***********************************************************************
2021-03-25T14:51:35.582499+08:00
***********************************************************************
2021-03-25T14:51:35.582506+08:00
***********************************************************************
2021-03-25T14:51:35.582501+08:00
***********************************************************************
2021-03-25T14:51:35.582742+08:00
***********************************************************************
2021-03-25T14:51:35.582751+08:00
***********************************************************************
2021-03-25T14:51:35.582753+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
2021-03-25T14:51:35.582955+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
2021-03-25T14:51:35.584321+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:35
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
2021-03-25T14:51:36.017954+08:00
opiodr aborting process unknown ospid (22819) as a result of ORA-609
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:36.018959+08:00
opiodr aborting process unknown ospid (22847) as a result of ORA-609
2021-03-25T14:51:36.021741+08:00
opiodr aborting process unknown ospid (22845) as a result of ORA-609
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:36.035096+08:00
opiodr aborting process unknown ospid (22849) as a result of ORA-609
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:36.051110+08:00
opiodr aborting process unknown ospid (22808) as a result of ORA-609
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:36.075937+08:00
opiodr aborting process unknown ospid (22806) as a result of ORA-609
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:36.089733+08:00
opiodr aborting process unknown ospid (22817) as a result of ORA-609
2021-03-25T14:51:36.115547+08:00
opiodr aborting process unknown ospid (22991) as a result of ORA-609
2021-03-25T14:51:36.130303+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:36
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:36.143590+08:00
opiodr aborting process unknown ospid (22811) as a result of ORA-609
2021-03-25T14:51:36.619383+08:00
opiodr aborting process unknown ospid (22851) as a result of ORA-609
2021-03-25T14:51:39.728247+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:39
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:39.732040+08:00
opiodr aborting process unknown ospid (22815) as a result of ORA-609
2021-03-25T14:51:45.844115+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:45
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
2021-03-25T14:51:45.845505+08:00
***********************************************************************
ns secondary err code: 12560
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
nt main err code: 0
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
nt secondary err code: 0
nt OS err code: 0
Time: 25-MAR-2021 14:51:45
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
2021-03-25T14:51:45.847250+08:00
opiodr aborting process unknown ospid (23143) as a result of ORA-609
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:45.849023+08:00
opiodr aborting process unknown ospid (23038) as a result of ORA-609
2021-03-25T14:51:48.831197+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:48
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
2021-03-25T14:51:48.832629+08:00
***********************************************************************
ns secondary err code: 12560
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
nt main err code: 0
nt secondary err code: 0
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
nt OS err code: 0
Time: 25-MAR-2021 14:51:48
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:51:48.835119+08:00
opiodr aborting process unknown ospid (23166) as a result of ORA-609
2021-03-25T14:51:48.835765+08:00
opiodr aborting process unknown ospid (23170) as a result of ORA-609
2021-03-25T14:51:59.271565+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:51:59
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:52:00.099778+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:52:00
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:52:00.103163+08:00
opiodr aborting process unknown ospid (23276) as a result of ORA-609
2021-03-25T14:52:00.256169+08:00
opiodr aborting process unknown ospid (23247) as a result of ORA-609
2021-03-25T14:53:00.032650+08:00
TT04: Standby redo logfile selected for thread 1 sequence 12906 for destination LOG_ARCHIVE_DEST_2
2021-03-25T14:53:07.978951+08:00
Archived Log entry 196268 added for T-1.S-12905 ID 0x8ef7a2a LAD:1
2021-03-25T14:53:25.129355+08:00
WARNING: Waiting on CRSD for password file, for 16 seconds, check CRS and OS logs
2021-03-25T14:53:31.024252+08:00
WARNING: Heavy swapping observed on system in last 5 mins.
Heavy swapping can lead to timeouts, poor performance, and instance eviction.
Errors in file /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/trace/dzzbcdb1_dbrm_15825.trc (incident=2920145) (PDBNAME=CDB$ROOT):
ORA-00700: soft internal error, arguments: [kskvmstatact: excessive swapping observed], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/db/diag/rdbms/dzzbcdb/dzzbcdb1/incident/incdir_2920145/dzzbcdb1_dbrm_15825_i2920145.trc
2021-03-25T14:54:32.783174+08:00
Dumping diagnostic data in directory=[cdmp_20210325145430], requested by (instance=1, osid=15825 (DBRM)), summary=[incident=2920145].
2021-03-25T14:58:28.780150+08:00
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 25-MAR-2021 14:58:28
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
2021-03-25T14:58:30.840384+08:00
opiodr aborting process unknown ospid (28144) as a result of ORA-609
2021-03-25T14:58:34.994147+08:00
复制 我来答
添加附件
收藏
分享
问题补充
4条回答
默认
最新
1.需要排查OS监控，查看各项指标故障前后趋势，尤其内存、OS磁盘以及ASM磁盘I/O，CPU，网络是否有异常。
2.需要排查dba_hist_active_sess_history，看故障前是否有捕获到一些异常等待事件以及PGA_ALLOCATED列是否有异常大的分配之类的。
3.如果有oswatcher之类的，oswatcher有ps监控，可以看看故障前后ps统计的LOCAL=NO准确判断故障前后是否有明显会话数增长。
4.再结合GRID、ORACLE日志判断吧。
如果没有监控之类，先从日志跟dba_hist_active_sess_history查起来吧。
评论
有用 1 打赏 0
我遇到的也是这个版本12.2.0.1，查看集群日志就是：Network communication with node appnode2(2) missing for 50% of timeout interval........; DB日志是有些IPC send timeout detected......，然后节点被剔除集群的信息
查看oswatch下的vmstate，就在异常时段内，bi值突然过高外，其他没什么；继续查看aswiostat,只是发现对几个磁盘的读相对很高，但最高的util%也不会超过80%，持续2-3分钟，后续就没有了
查看oswtop日志，是在异常时间段之前，有些oracle进程相对使用CPU资源较高
后面不知道怎么分析下去，如果大佬知道，请指明下分析方向。或者可能的原因。
评论 有用 0 打赏 0
回答交流
Markdown
请输入正文
提交
问题信息
请登录之后查看
附件列表
请登录之后查看
邀请回答
暂无人订阅该标签，敬请期待~~
墨值悬赏