ORA-603 错误解析

m000进程，不过根据metalink上的描述，这个进行应该是MMON进程启动的从属平行进程，属于轻量级的进程，对系统没有多大的影响

关于告警日志里面的错误信息，我们看出m000进程创建失败，PMON进程无法启动该进程。一般情况下，PMON无法启动进程原因有下面一些：
1、Oracle连接数超过进程数限制。（正是由于Oracle达到了进程数限制，进而PMON无法创建m000进程）
2、进程死锁。


事后我检查了一下v$resource_limit,发现会话连接数、进程数并没有超。那么完全可以排除这个因素，那么现在就有可能是进程死锁或bug造成的

同事在检查过程中发现Physic memory资源严重不足，引起了Swap频繁读写。继续检查SGA参数发现sga_max_size、sga_target设置过大（这台测试服务器是虚拟机做的克隆，生产环境的RAM为64G，SGA也设置较大，克隆过后ORACLE实例启动不了，调整了SGA_TARGET、SGA_MAX_SIZE等参数后才启动成功，但是不知为什么sga_max_size设置了成了11264M（11G）,有可能是当时要设置为1G多，因为物理内存才3G多，但是不知是手抖了还是搞晕了，当然也不排除后面被人改掉，居然设置成了11264M大小，汗颜啊。居然运行了这么久直到最近才出现问题，测试数据库基本不会做巡检）

经查MOS(Oracle Linux: ORA-27301:OS Failure Message: No Buffer Space Available (文档 ID 2041723.1)，发现可能是因为网卡的MUT参数设置过高导致网卡的缓存不足导致的。


Fri Nov 24 09:11:42 2017
skgxpvfynet: mtype: 61 process 11799 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_11799.trc (incident=123381):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_123381/orcl2_ora_11799_i123381.trc
Fri Nov 24 09:11:42 2017
skgxpvfynet: mtype: 61 process 11801 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)
opiodr aborting process unknown ospid (11743) as a result of ORA-603


This happens due to less space available for network buffer reservation.

SOLUTION

1. On servers with High Physical Memory, the parameter vm.min_free_kbytes should be set in the order of 0.4% of total
Physical Memory. This helps in keeping a larger range of defragmented memory pages available for network buffers
reducing the probability of a low-buffer-space conditions.

*** For example, on a server which is having 256GB RAM, the parameter vm.min_free_kbytes should be set to 1048576 ***

vm.min_free_kbytes=0.4%*total Physical Memory

On NUMA Enabled Systems, the value of vm.min_free_kbytes should be multiplied by the number of NUMA nodes since the value
is to be split across all the nodes.


On NUMA Enabled Systems, the value of vm.min_free_kbytes = n * 0.4% of total Physical Memory. Here 'n' is the
number of NUMA nodes.


2. Additionally, the MTU value should be modified as below

#ifconfig lo mtu 16436

To make the change persistent over reboot add the following line in the file /etc/sysconfig/network-scripts/ifcfg-lo :

MTU=16436
Save the file and restart the network service to load the changes

#service network restart

当前服务器网卡本地回环的MTU默认设置是65536，按照MOS文档的方法修改这个设置。

ifconfig lo mtu 16436

但是重启后将恢复默认值，如果保证重启也生效，就需要修改网卡的配置文件。

vi /etc/sysconfig/network-scripts/ifcfg-lo

DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
MTU=16436