10M还是之前的问题ORACLE12.1.0.2 不定时宕库，查看系统top会有ksawp0进程占用高负载
系统top显示信息：
alter.log信息:
Instance Critical Process (pid: 3, ospid: 2557, PSP0) died unexpectedly
PMON (ospid: 2553): terminating the instance due to error 490
Mon Mar 30 00:32:15 2020
trace文件信息：
Trace file /u01/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_pmon_2553.trc
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
ORACLE_HOME = /u01/Oracle12c/product/12.1.0/db_1
System name: Linux
Node name: zhhy
Release: 2.6.32-431.23.3.el6.x86_64
Version: #1 SMP Thu Jul 31 17:20:51 UTC 2014
Machine: x86_64
Instance name: orcl
Redo thread mounted by this instance: 1
Oracle process number: 2
Unix process pid: 2553, image: oracle@zhhy (PMON)
*** 2020-03-30 00:32:15.861
*** SESSION ID:(61.54861) 2020-03-30 00:32:15.861
*** CLIENT ID:() 2020-03-30 00:32:15.861
*** SERVICE NAME:(SYS$BACKGROUND) 2020-03-30 00:32:15.861
*** MODULE NAME:() 2020-03-30 00:32:15.861
*** CLIENT DRIVER:() 2020-03-30 00:32:15.861
*** ACTION NAME:() 2020-03-30 00:32:15.861
Instance Critical Process (pid: 3, ospid: 2557, PSP0) died unexpectedly
Background process PSP0 found dead
Oracle pid = 3
OS pid (from detached process) = 2557
OS pid (from process state) = 2557
dtp = 0x60013888, proc = 0x8504fc650
Dump of memory from 0x0000000060013888 to 0x00000000600138E0
060013880 0000000C 00000000 […]
060013890 504FC650 00000008 00000000 30505350 [P.OP…PSP0]
0600138A0 00000000 00000000 000009FD 00000000 […]
0600138B0 13427378 00000000 3580E260 00007F7F [xsB…..5....] 0600138C0 000009FD 00000000 13427378 00000000 [........xsB.....] 0600138D0 00000001 2CD9D908 00040081 00000000 [.......,........] Dump of memory from 0x00000008504FC650 to 0x00000008504FD178 8504FC650 00000301 00000000 00000302 00000000 [................] 8504FC660 00000000 00000000 504FC650 00000008 [........P.OP....] 8504FC670 504FC670 00000008 504FC670 00000008 [p.OP....p.OP....] 8504FC680 00000000 00000000 00000000 00000000 [................] Repeat 2 times 8504FC6B0 00000006 00000000 50890E00 00000008 [...........P....] 8504FC6C0 50A2A9E0 00000008 5062B798 00000008 [...P......bP....] 8504FC6D0 5CACE340 00000008 00000000 00000000 [@..\............] 8504FC6E0 5CACE5E0 00000008 5CACE5E0 00000008 [...\.......\....] 8504FC6F0 50A2A9C0 00000008 00000601 00000010 [...P............] 8504FC700 546DFB78 00000008 5062B798 00000008 [x.mT......bP....] 8504FC710 00000003 00000000 00000000 00000003 [................] 8504FC720 00000001 C0827A50 5BFBE2A8 00000008 [....Pz.....[....] 8504FC730 00000000 00000000 60013888 00000000 [.........8.…]
8504FC740 00000000 00000000 00000000 00000000 […]
Repeat 2 times
8504FC770 50890C58 00000008 50890E60 00000008 [X…P…..P....] 8504FC780 00000000 00000000 00000000 00000000 [................] Repeat 3 times 8504FC7C0 00000000 00000000 00030000 00000000 [................] Repeat 1 times 8504FC7E0 00000000 00000000 00000000 00000000 [................] 8504FC7F0 00000000 00000000 00040000 00000000 [................] 8504FC800 00000000 00000000 00030000 00000000 [................] Repeat 1 times 8504FC820 00000000 00000000 00040000 00000000 [................] 8504FC830 00000000 00000000 00050000 00000000 [................] 8504FC840 00000000 00000000 00030000 00000000 [................] 8504FC850 00000000 00000000 00000000 00000000 [................] Repeat 4 times 8504FC8A0 00000000 00000000 00040000 00000000 [................] 8504FC8B0 00000000 00000000 00000000 00000000 [................] Repeat 5 times 8504FC910 00000000 00000000 00000064 00000000 [........d.......] 8504FC920 00000000 00000000 00000000 00000000 [................] 8504FC930 504FC930 00000008 504FC930 00000008 [0.OP....0.OP....] 8504FC940 00000000 00000000 00000000 00000000 [................] Repeat 1 times 8504FC960 00000001 00000000 504FC968 00000008 [........h.OP....] 8504FC970 504FC968 00000008 00000000 00000000 [h.OP............] 8504FC980 00000000 00000000 00000000 00000000 [................] Repeat 2 times 8504FC9B0 00000000 00000000 0000001A 00000049 [............I...] 8504FC9C0 00000001 0000165E 54666FD8 00000008 [....^....ofT....] 8504FC9D0 5052FFE0 00000008 0000273A 00000000 [..RP....:'......] 8504FC9E0 00010003 00000000 00000000 00000000 [................] 8504FC9F0 00000000 00000000 00000000 00000000 [................] Repeat 1 times 8504FCA10 00000000 00000000 000009FD 00000000 [................] 8504FCA20 00000000 00000000 00000000 00000000 [................] Repeat 2 times 8504FCA50 00000000 00000000 504FC650 00000008 [........P.OP....] 8504FCA60 00000000 00000000 00000000 00000000 [................] Repeat 11 times 8504FCB20 00000000 00000000 00000000 5E80CD8B [...............^] 8504FCB30 00000000 00000000 00000000 00000000 [................] Repeat 1 times 8504FCB50 00000000 00000000 00000000 00418002 [..............A.] 8504FCB60 00000007 00000000 000009FD 00000000 [................] 8504FCB70 13427378 00000000 3580E260 00007F7F [xsB.....…5…]
8504FCB80 000009FD 00000000 13427378 00000000 […xsB…]
8504FCB90 00000000 00000000 504FCBA0 00000008 […OP…]
8504FCBA0 00000000 00000000 00000000 00000000 […]
Repeat 2 times
8504FCBD0 00000000 00000000 00000003 FFFFFFFF […]
8504FCBE0 00000000 00000000 00000000 00000000 […]
Repeat 13 times
8504FCCC0 6361726F 0000656C 00000000 00000000 [oracle…]
8504FCCD0 00000000 00000000 00000000 00000000 […]
8504FCCE0 00000006 00000000 7968687A 00000000 […zhhy…]
8504FCCF0 00000000 00000000 00000000 00000000 […]
Repeat 2 times
8504FCD20 00000000 00000000 00000004 00000000 […]
8504FCD30 00000000 4E4B4E55 004E574F 00000000 […UNKNOWN…]
8504FCD40 00000000 00000000 00000000 00000000 […]
8504FCD50 00000000 00000000 00000007 00000000 […]
8504FCD60 37353532 00000000 00000000 00000000 [2557…]
8504FCD70 00000000 00000000 00000004 00000000 […]
8504FCD80 37353532 00000000 00000000 00000000 [2557…]
8504FCD90 00000000 00000000 00000004 00000000 […]
8504FCDA0 37353532 00000000 00000000 00000000 [2557…]
8504FCDB0 00000000 00000000 00000004 00000000 […]
8504FCDC0 6361726F 7A40656C 20796868 50535028 [oracle@zhhy (PSP]
8504FCDD0 00002930 00000000 00000000 00000000 [0)…]
8504FCDE0 00000000 00000000 00000000 00000000 […]
8504FCDF0 00000012 00000000 30505350 00000000 […PSP0…]
8504FCE00 00000002 00000000 00000000 00000000 […]
8504FCE10 00000000 00000000 00000000 00000000 […]
Repeat 8 times
8504FCEA0 00000000 00000000 00000000 00000200 […]
8504FCEB0 00000000 00000000 00000000 00000000 […]
Repeat 2 times
8504FCEE0 54697CE8 00000008 5C601658 00000008 [.|iT…X.`…]
8504FCEF0 546640A8 00000008 587AE2E4 00000008 [.@fT…zX…]
8504FCF00 00000000 00000000 00000000 00000000 […]
Repeat 1 times
8504FCF20 504FCF20 00000008 504FCF20 00000008 [ .OP… .OP…]
8504FCF30 00060000 00000000 00000000 00000000 […]
8504FCF40 00000000 00000000 00087DD4 00000000 […}…]
8504FCF50 000BD5AC 00000000 00000000 00000000 […]
8504FCF60 000E7DD4 00000000 000E7DD4 00000000 [.}…}…]
8504FCF70 5C334280 00000008 00000000 00000000 [.B3…]
8504FCF80 00000000 00000000 00000000 00000000 […]
8504FCF90 00000001 00000000 00000000 00000000 […]
8504FCFA0 00000000 00000000 00000000 00000000 […]
8504FCFB0 000007B8 00000000 000000E0 00000000 […]
8504FCFC0 000007B8 00000000 00000000 00000000 […]
8504FCFD0 00000000 00000000 00000000 00000000 […]
Repeat 2 times
8504FD000 00000000 00000000 03000100 27A5A000 […’]
8504FD010 00000000 01000000 00000000 00000000 […]
8504FD020 00000000 00000000 00000000 00000000 […]
Repeat 1 times
8504FD040 00000000 00000001 00000000 00000000 […]
8504FD050 00000000 00000000 00000000 00000000 […]
Repeat 10 times
8504FD100 504FD100 00000008 504FD100 00000008 […OP…OP…]
8504FD110 00000000 00000000 504FD118 00000008 […OP…]
8504FD120 504FD118 00000008 00000000 00000000 […OP…]
8504FD130 00000000 00000000 00000000 00000000 […]
Repeat 3 times
8504FD170 00000000 00000000 […]
2020-03-30 00:32:15.890775 :kjzduptcctx(): Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+244<-kjzdssdmp()+321<-kjzduptcctx()+692<-kjzdicrshnfy()+992<-ksuitm()+5857<-ksucln_fatalproc_death()+296<-ksulhdcb()+391<-ksucln()+17253<-ksbrdp()+1068<-op
irip()+1488<-opidrv()+616<-sou2o()+145<-opimai_real()+270<-ssthrdmain()+412<-main()+236<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----
*** 2020-03-30 00:32:15.959
PMON (ospid: 2553): terminating the instance due to error 490
ksuitm: waiting up to [5] seconds before killing DIAG(2619)
操作系统信息：
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final
messge日志：只有宕库前的，宕库后的日志信息没有记录，好像就是只要宕库就不记录信息
Mar 30 00:04:45 xxxx sshd[20871]: Connection closed by 175.126.165.151 port 59866 [preauth]
Mar 30 00:05:25 xxxx sshd[20899]: Accepted publickey for root from 70.178.158.59 port 32888 ssh2: RSA SHA256:MkYY9qiVsFGBC5WkjoClCkwEFW5iSjcGQF7m4n4H7Cw
Mar 30 00:13:25 xxxx sshd[21200]: Connection closed by 218.234.16.140 port 53322 [preauth]
Mar 30 00:14:52 xxxx sshd[21248]: Connection closed by 111.33.30.41 port 51260 [preauth]
Mar 30 00:14:57 xxxx sshd[21252]: Accepted publickey for root from 192.241.211.94 port 59222 ssh2: RSA SHA256:MkYY9qiVsFGBC5WkjoClCkwEFW5iSjcGQF7m4n4H7Cw
Mar 30 00:16:38 xxxx sshd[21396]: Connection closed by 175.126.165.151 port 36116 [preauth]
Mar 30 00:17:16 xxxx sshd[21428]: Connection closed by 159.8.183.11 port 58152 [preauth]
Mar 30 00:25:14 xxxx sshd[21708]: Connection closed by 218.234.16.140 port 34064 [preauth]
Mar 30 00:26:17 xxxx sshd[21748]: Connection closed by 111.33.30.41 port 51852 [preauth]
Mar 30 00:28:28 xxxx sshd[21820]: Connection closed by 175.126.165.151 port 40599 [preauth]
评论
有用 0kswapd是linux用于页面回收的内核进程，定期扫描内存使用情况，根据/proc/sys/vm/min_free_kbytes阈值来触发回收内存动作。
（1）是不是开了numa了？# numactl --hardware
（2）是不是swappiness设置的太大了？# sysctl -a|grep swap
（3）是不是/proc/sys/vm/min_free_kbytes设置过大了？
如果开了numa，建议关闭numa，如果swappiness设置过大，建议调低，以降低swap积极性。
参考资料，来自墨天轮Moone大师文档，《从内存故障到CPU过高诊断案例(上).pdf》,可以从文档模块下载。
评论 有用 0sysctl如下：
vm.swappiness = 0
numactl信息如下：
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 65535 MB
node 0 free: 2230 MB
node distances:
node 0
0: 10
评论 有用 0