1
某客户数据库运行时突然重启，排查如下：
#数据库相关环境信息
数据库是一套两节点 RAC+两 DG 的架构，相关信息如下：
主库操作系统：CentOS release 6.10
备库一操作系统： CentOS Linux release 7.4.1708
备库二操作系统： CentOS Linux release 7.6.1810
数据库版本： Release 11.2.0.3.0 Production
一、问题诊断
在 10 点 30 分开始已经报错:
Sun Feb 23 10:30:37 2020
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x1100000023]
[PC:0x9379D18, kglic0()+1086] [flags: 0x0, count: 1]
Errors in file
/picclife/app/oracle/diag/rdbms/dbsms/dbsms1/trace/dbsms1_m000_5759.trc
(incident=1200391):
ORA-07445: exception encountered: core dump [kglic0()+1086] [SIGSEGV]
[ADDR:0x1100000023] [PC:0x9379D18] [Address not mapped to object] []
Incident details in:
/picclife/app/oracle/diag/rdbms/dbsms/dbsms1/incident/incdir_1200391/dbsms1_m000
_5759_i1200391.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
直到 12 点仍然存在:
Sun Feb 23 12:00:10 2020
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x333F]
[PC:0x9379D18, kglic0()+1086] [flags: 0x0, count: 1]
Errors in file
/picclife/app/oracle/diag/rdbms/dbsms/dbsms1/trace/dbsms1_m000_12514.trc
(incident=1200623):
ORA-07445: exception encountered: core dump [kglic0()+1086] [SIGSEGV]
[ADDR:0x333F] [PC:0x9379D18] [Address not mapped to object] []
Incident details in:
/picclife/app/oracle/diag/rdbms/dbsms/dbsms1/incident/incdir_1200623/dbsms1_m000
_12514_i1200623.trc
最终，在重启系统前的半小时，已经由于 bug 导致多个进程没有响应，且 pmon 无法获取到 latch 需要重启
数据库实例解决。
Sun Feb 23 12:58:55 2020
PMON failed to acquire latch, see PMON dump
Sun Feb 23 12:59:27 2020
Errors in file
/picclife/app/oracle/diag/rdbms/dbsms/dbsms1/trace/dbsms1_qmnc_5370.trc
(incident=1200375):
ORA-00445: background process "q001" did not start after 120 seconds
Sun Feb 23 13:02:30 2020
Starting ORACLE instance (normal)
。。。。。。
经 MOS 匹配，与 bug 吻合:
Database Hangs with ORA-7445 [kglic0()+1172], PMON Failed to Acquire Latch, ORA-
29771 (Doc ID 2128960.1)
进一步定位到 Bug 14538018 : INSTANCE CRASHED WITH ORA-7445 [KGLIC0+1086]
现象基本匹配:
1.影响版本基本匹配，当前环境为 11.2.0.3 且无任何补丁:
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Database 11g 11.2.0.3.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
Local node = n1smsdb1
Remote node = n1smsdb2
--------------------------------------------------------------------------------
2.cursor_sharing 设置为 EXACT
3.optimizer 参数相同
show parameter optimizer;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
optimizer_capture_sql_plan_baselines boolean FALSE
optimizer_dynamic_sampling integer 2
4.同
时/picclife/app/oracle/diag/rdbms/dbsms/dbsms1/incident/incdir_1200623/dbsms1_m0
00_12514_i1200623.trc 中堆栈信息与 incident 的 trace 文件匹配:
STACK TRACE:
------------
skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp
<- ssexhd <- sighandler <- kglic0 <- kksIterCursorStat <-
kewrrtsq_rank_topsq
<- kewrbtsq_build_tops <- kewrftsq_flush_tops <- kewrft_flush_table
<- kewrftec_flush_tabl <- e_ehdlcx
<- kewrfat_flush_all_t <- ables <- kewrfsr_flush_snaps <- hot_r <-
kewrrfs_remote_flus
<- h_slave <- kebm_slave_main <- ksvrdp <- opirip <- opidrv
<- sou2o <- opimai_real <- ssthrdmain <- main <- libc_start_main
<- start
在 10 点 30 分已经开始出现问题，影响到 mmon 进程调度 AWR 的快照自动生成:
实例 1 AWR 从 10 点半开始就无记录:
Instance DB Name Snap Id Snap Started Level
------------ ------------ --------- ------------------ -----
dbsms1 SMS 111561 23 Feb 2020 00:00 1
111581 23 Feb 2020 10:00 1
111588 23 Feb 2020 13:30 1
111589 23 Feb 2020 14:00 1
该 bug 会导致业务会话阻塞 LCK0 进程(LCK 进程主要在 RAC 环境上处理 library 和 row cache 的请求)
获得 shared pool 的 latch，同时造成业务会话大量堆积，杀业务会话无效果，直到实例重启。
在 11 点 54 分时候已经出现过一次阻塞，后业务会话被 LMHB 进程 kill 掉而释放:
Sun Feb 23 11:54:57 2020
LCK0 (ospid: 5113) waits for latch 'shared pool' for 93 secs.
Errors in file
of 15
10墨值下载
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文档的来源（墨天轮），文档链接，文档作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
下载排行榜
评论