记录Oracle10.2.0.5RAC节点服务器重启后-CRS无法启动故障排查

原创 DonACE All China Database Union 2023-08-29

355

RAC运行环境

aix6.1

Oracle10.2.0.5RAC

crsctl start crs

集群状态检测

crs_stat:
CRS_0184: Cannot communicate with the CRS daemon

crsctl check crs:
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

ocrcheck和crsctl query css votedisk状态也是正常的

--crs启动检查

crsctl check boot

Failure 3 in CLSC use global init

查看日志没有最新启动状态的日志

$ORACLE_CRS_HOME/log/dbnode1/alertdbnode1.log

$ORACLE_CRS_HOME/log/dbnode1/crsd

$ORACLE_CRS_HOME/log/dbnode1/cssd

检查crsd.log及ocssd.log，没有发现有用的信息

而在$ORACLE_CRS_HOME/log/dbnode/client中发现

cat clsc52.log

NS user global alloc failed

查看tmp

tail -f /tmp/crsctl.3146230

Failure 3 in CLSC use global init

一时无解，无意测试了一下网络发现服务器无法ping通自己

ping 127.0.0.1

ping 10.8.8.21 --public IP

ping 172.16.8.21 --private IP

2块网卡的ip地址都ping不通

使用ifconfig -a 查看状态

lo0: flags=e08084b,c0<BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>

发现网卡回环状态不对，尝试启用回环

ifconfig lo0 inet 127.0.0.1 up

启用后正常状态

lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255

然后手动重启CRS,可以正常启动，故障处理到此结束！