Oracle 共享内存段未释放导致数据库无法启动

Oracle 数据库异常关闭,共享内存未释放,导致无法启动。
以下内容为测试环境复原故障现场:

流程:
尝试启动数据库出现报错信息:

[oracle@server1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Thu Jan 21 00:18:04 2021
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup;
ORA-00845: MEMORY_TARGET not supported on this system
复制

警告日志:

Starting ORACLE instance (normal)
WARNING: You are trying to use the MEMORY_TARGET feature. 
This feature requires the /dev/shm file system to be mounted for at least 834666496 bytes. 
/dev/shm is either not mounted or is mounted with available space less than this size. 
Please fix this so that MEMORY_TARGET can work as expected. 
Current available is 804745216 and used is 238903296 bytes.
Ensure that the mount point is /dev/shm for this directory.
memory_target needs larger /dev/shm
复制

排查发现Oracle共享内存段未释放:

[oracle@server1 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       50G   15G   33G  32% /
tmpfs                 996M  228M  768M  23% /dev/shm
/dev/sda1             477M   40M  412M   9% /boot
/dev/mapper/VolGroup-lv_home
                      144G   60M  136G   1% /home


[oracle@server1 ~]$ ipcs 
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 2359299    oracle     640        4096       0                       
0xba1697c4 2424837    oracle     640        4096       0                       

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x4d96aff0 753666     oracle     640        154       

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages  
复制

使用ipcrm命令进行手工释放共享内存段:

[oracle@server1 ~]$ ipcrm -m 2359299
[oracle@server1 ~]$ ipcrm -m 2424837
[oracle@server1 ~]$ ipcrm -s 753666
[oracle@server1 ~]$ ipcs
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    
复制

伴随着如上操作:
警告日志中出现如下内容:

Thu Jan 21 00:22:30 2021
Thu Jan 21 00:22:30 2021
Errors in file /oracle/app/diag/rdbms/prod/prod/trace/prod_gen0_3022.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Errors in file /oracle/app/diag/rdbms/prod/prod/trace/prod_mman_3030.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 21 00:22:30 2021
Errors in file /oracle/app/diag/rdbms/prod/prod/trace/prod_psp0_3016.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 21 00:22:30 2021
Errors in file /oracle/app/diag/rdbms/prod/prod/trace/prod_smon_3038.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 21 00:22:30 2021
Errors in file /oracle/app/diag/rdbms/prod/prod/trace/prod_dbrm_3026.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 21 00:22:30 2021
Errors in file /oracle/app/diag/rdbms/prod/prod/trace/prod_q000_3116.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
MMAN (ospid: 3030): terminating the instance due to error 27157
Instance terminated by MMAN, pid = 3030
Errors in file /oracle/app/diag/rdbms/prod/prod/trace/prod_mman_3030.trc:
ORA-27300: OS system dependent operation:semctl failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgpwrm1
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 21 00:22:31 2021
Instance termination got error 27120 from SGA destruction.
  Error cleared. Process exiting.
复制

尝试再次启动:

SQL> startup;
ORA-00845: MEMORY_TARGET not supported on this system
复制

依旧不行

进而查看/dev/shm空间是否释放:

[oracle@server1 shm]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       50G   15G   33G  32% /
tmpfs                 996M  228M  768M  23% /dev/shm
/dev/sda1             477M   40M  412M   9% /boot
/dev/mapper/VolGroup-lv_home
                      144G   60M  136G   1% /home
复制

进入/dev/shm一探究竟:

[oracle@server1 shm]$ ls
ora_prod_2359299_0    ora_prod_2392068_119  ora_prod_2392068_142  ora_prod_2392068_166  ora_prod_2392068_19   ora_prod_2392068_37  ora_prod_2392068_60  ora_prod_2392068_84
ora_prod_2359299_1    ora_prod_2392068_12   ora_prod_2392068_143  ora_prod_2392068_167  ora_prod_2392068_190  ora_prod_2392068_38  ora_prod_2392068_61  ora_prod_2392068_85
ora_prod_2359299_2    ora_prod_2392068_120  ora_prod_2392068_144  ora_prod_2392068_168  ora_prod_2392068_191  ora_prod_2392068_39  ora_prod_2392068_62  ora_prod_2392068_86
ora_prod_2392068_0    ora_prod_2392068_121  ora_prod_2392068_145  ora_prod_2392068_169  ora_prod_2392068_192  ora_prod_2392068_4   ora_prod_2392068_63  ora_prod_2392068_87
ora_prod_2392068_1    ora_prod_2392068_122  ora_prod_2392068_146  ora_prod_2392068_17   ora_prod_2392068_193  ora_prod_2392068_40  ora_prod_2392068_64  ora_prod_2392068_88
ora_prod_2392068_10   ora_prod_2392068_123  ora_prod_2392068_147  ora_prod_2392068_170  ora_prod_2392068_194  ora_prod_2392068_41  ora_prod_2392068_65  ora_prod_2392068_89
ora_prod_2392068_100  ora_prod_2392068_124  ora_prod_2392068_148  ora_prod_2392068_171  ora_prod_2392068_195  ora_prod_2392068_42  ora_prod_2392068_66  ora_prod_2392068_9
ora_prod_2392068_101  ora_prod_2392068_125  ora_prod_2392068_149  ora_prod_2392068_172  ora_prod_2392068_2    ora_prod_2392068_43  ora_prod_2392068_67  ora_prod_2392068_90
ora_prod_2392068_102  ora_prod_2392068_126  ora_prod_2392068_15   ora_prod_2392068_173  ora_prod_2392068_20   ora_prod_2392068_44  ora_prod_2392068_68  ora_prod_2392068_91
ora_prod_2392068_103  ora_prod_2392068_127  ora_prod_2392068_150  ora_prod_2392068_174  ora_prod_2392068_21   ora_prod_2392068_45  ora_prod_2392068_69  ora_prod_2392068_92
ora_prod_2392068_104  ora_prod_2392068_128  ora_prod_2392068_151  ora_prod_2392068_175  ora_prod_2392068_22   ora_prod_2392068_46  ora_prod_2392068_7   ora_prod_2392068_93
ora_prod_2392068_105  ora_prod_2392068_129  ora_prod_2392068_152  ora_prod_2392068_176  ora_prod_2392068_23   ora_prod_2392068_47  ora_prod_2392068_70  ora_prod_2392068_94
ora_prod_2392068_106  ora_prod_2392068_13   ora_prod_2392068_153  ora_prod_2392068_177  ora_prod_2392068_24   ora_prod_2392068_48  ora_prod_2392068_71  ora_prod_2392068_95
ora_prod_2392068_107  ora_prod_2392068_130  ora_prod_2392068_154  ora_prod_2392068_178  ora_prod_2392068_25   ora_prod_2392068_49  ora_prod_2392068_72  ora_prod_2392068_96
ora_prod_2392068_108  ora_prod_2392068_131  ora_prod_2392068_155  ora_prod_2392068_179  ora_prod_2392068_26   ora_prod_2392068_5   ora_prod_2392068_73  ora_prod_2392068_97
ora_prod_2392068_109  ora_prod_2392068_132  ora_prod_2392068_156  ora_prod_2392068_18   ora_prod_2392068_27   ora_prod_2392068_50  ora_prod_2392068_74  ora_prod_2392068_98
ora_prod_2392068_11   ora_prod_2392068_133  ora_prod_2392068_157  ora_prod_2392068_180  ora_prod_2392068_28   ora_prod_2392068_51  ora_prod_2392068_75  ora_prod_2392068_99
ora_prod_2392068_110  ora_prod_2392068_134  ora_prod_2392068_158  ora_prod_2392068_181  ora_prod_2392068_29   ora_prod_2392068_52  ora_prod_2392068_76  ora_prod_2424837_0
ora_prod_2392068_111  ora_prod_2392068_135  ora_prod_2392068_159  ora_prod_2392068_182  ora_prod_2392068_3    ora_prod_2392068_53  ora_prod_2392068_77  pulse-shm-3740554943
ora_prod_2392068_112  ora_prod_2392068_136  ora_prod_2392068_16   ora_prod_2392068_183  ora_prod_2392068_30   ora_prod_2392068_54  ora_prod_2392068_78  pulse-shm-613326773
ora_prod_2392068_113  ora_prod_2392068_137  ora_prod_2392068_160  ora_prod_2392068_184  ora_prod_2392068_31   ora_prod_2392068_55  ora_prod_2392068_79
ora_prod_2392068_114  ora_prod_2392068_138  ora_prod_2392068_161  ora_prod_2392068_185  ora_prod_2392068_32   ora_prod_2392068_56  ora_prod_2392068_8
ora_prod_2392068_115  ora_prod_2392068_139  ora_prod_2392068_162  ora_prod_2392068_186  ora_prod_2392068_33   ora_prod_2392068_57  ora_prod_2392068_80
ora_prod_2392068_116  ora_prod_2392068_14   ora_prod_2392068_163  ora_prod_2392068_187  ora_prod_2392068_34   ora_prod_2392068_58  ora_prod_2392068_81
ora_prod_2392068_117  ora_prod_2392068_140  ora_prod_2392068_164  ora_prod_2392068_188  ora_prod_2392068_35   ora_prod_2392068_59  ora_prod_2392068_82
ora_prod_2392068_118  ora_prod_2392068_141  ora_prod_2392068_165  ora_prod_2392068_189  ora_prod_2392068_36   ora_prod_2392068_6   ora_prod_2392068_83
复制

使用fuser命令判断是否有进程占用:

[oracle@server1 shm]$ fuser ora_prod_2359299_0
复制

进行手动清理:

[oracle@server1 shm]$ rm -rf ora_prod_*
[oracle@server1 shm]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       50G   15G   33G  32% /
tmpfs                 996M   72K  996M   1% /dev/shm
/dev/sda1             477M   40M  412M   9% /boot
/dev/mapper/VolGroup-lv_home
                      144G   60M  136G   1% /home
复制

再次尝试启动:

SQL> startup;
ORACLE instance started.

Total System Global Area  830930944 bytes
Fixed Size                  2257800 bytes
Variable Size             524291192 bytes
Database Buffers          297795584 bytes
Redo Buffers                6586368 bytes
Database mounted.
Database opened.
复制

正常~
20210120 记录!