4月22日早上8点收到业务人员消息，数据库连不上，查看集群状态，发现2个节点的crs都处于停止状态，采取了重新启动节点2集群crsctl start crs，重启了节点1服务器，之后集群正常，业务恢复
现在的问题是找不到原因，看了日志判断不出是心跳线的问题还是存储的问题，还麻烦大师帮忙分析下，感谢感谢
其中节点2系统时间比节点1系统时间差50s
从日志上来看应该是IO出了问题
2019-04-22 01:08:45.109: [ CSSD][1744828160](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 2132880192 ms for voting file /dev/mapper/mpathg)
2019-04-22 01:08:45.109: [ CSSD][2762966784](:CSSNM00005:)clssnmvDiskKillCheck: Aborting, evicted by node hxrac01, number 1, sync 427157773, stamp 2723370914
2019-04-22 01:08:45.109: [ CSSD][1744828160]clssnmvDiskAvailabilityChange: voting file /dev/mapper/mpathg now offline
2019-04-22 01:08:45.109: [ CSSD][2762966784]clssnmRemoveNodeInTerm: node 2, hxrac02 terminated due to Normal Shutdown. Removing from member and connected bitmaps
2019-04-22 01:08:45.109: [ CSSD][1744828160](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 3 configured voting disks available, need 2
2019-04-22 01:08:45.109: [ CSSD][2772481792](:CSSNM00005:)clssnmvDiskKillCheck: Aborting, evicted by node hxrac01, number 1, sync 427157773, stamp 2723370914
2019-04-22 01:08:45.109: [ CSSD][2772481792]clssnmRemoveNodeInTerm: node 2, hxrac02 terminated due to Normal Shutdown. Removing from member and connected bitmaps
2019-04-22 01:08:45.109: [ CSSD][1744828160]clssnmRemoveNodeInTerm: node 2, hxrac02 terminated due to Normal Shutdown. Removing from member and connected bitmaps
2019-04-22 01:08:45.109: [ CSSD][2762966784]clssscExit: abort already set 1
2019-04-22 01:08:45.109: [ CSSD][2772481792]clssscExit: abort already set 1
2019-04-22 01:08:45.109: [ SKGFD][2761389824]Lib :UFS:: closing handle 0x7f135c0406f0 for disk :/dev/mapper/mpathf:
评论
有用 0winkey大师，您好！
我还有一些疑问想要咨询下：
您是怎么从众多信息中确认是IO的问题，我想要知道这个思路，因为我自己在分析的时候会被各种信息扰乱，找不到关键问题
我看到最开始22号晚告警信息最先出现在节点2的数据库日志里
Mon Apr 22 01:06:17 2019
IPC Send timeout detected. Sender: ospid 77020 [oracle@hxrac02 (J000)]
报的是检测超时
另外，节点1集群日志里有如下信息显示网络故障：
2019-04-22 01:08:30.150:
[cssd(25369)]CRS-1612:Network communication with node hxrac02 (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.530 seconds
2019-04-22 01:08:38.151:
[cssd(25369)]CRS-1611:Network communication with node hxrac02 (2) missing for 75% of timeout interval. Removal of this node from cluster in 6.530 seconds
2019-04-22 01:08:42.152:
[cssd(25369)]CRS-1610:Network communication with node hxrac02 (2) missing for 90% of timeout interval. Removal of this node from cluster in 2.530 seconds
ocssd1.l01里也有关于网卡的信息
2019-04-22 01:08:42.152: [ CSSD][4229900032]clssnmvDHBValidateNcopy: node 2, hxrac02, has a disk HB, but no network HB
所以，这应该是一种什么样的逻辑关系呢？集群宕掉的整个过程是什么样的呢？
这个IO问题是存储的IO问题吗？今天让同事看了存储上的报错信息，反馈22号没有记录，只有23号的4条记录，该如何确认存储IO问题
评论 有用 0
