暂无图片
12C RAC故障分析
我来答
返回问答列表
分享
Uncopyrightable
2021-08-26
12C RAC故障分析
暂无图片 10M
oracle racoracle

从RAC的crs日志看到vip漂移失败，导致数据库集群节点重启

2021-08-26 09:26:41.272 [CRSD(16816)]CRS-2771: Maximum restart attempts reached for resource ‘ora.node2.vip’; will not restart.
2021-08-26 09:26:42.147 [CRSD(16816)]CRS-2878: Failed to restart resource ‘ora.net1.network’
2021-08-26 09:26:42.148 [CRSD(16816)]CRS-2769: Unable to failover resource ‘ora.net1.network’.
2021-08-26 09:26:43.478 [CRSD(16816)]CRS-2769: Unable to failover resource ‘ora.net1.network’.
2021-08-26 09:47:51.642 [ORAROOTAGENT(16321)]CRS-5818: Aborted command ‘check’ for resource ‘ora.drivers.acfs’. Details at (:CRSAGF00113:) {0:0:231} in /u01/app/grid/diag/crs/node1/crs/trace/ohasd_orarootagent_root.trc.
2021-08-26 09:48:06.560 [ORAROOTAGENT(16321)]CRS-5014: Agent “ORAROOTAGENT” timed out starting process “/u01/app/12c/grid/bin/acfsload” for action “check”: details at “(:CLSN00009:)” in “/u01/app/grid/diag/crs/node1/crs/trace/ohasd_orarootagent_root.trc”

2021-08-26 10:06:59.351 [ORAAGENT(17118)]CRS-5014: Agent “ORAAGENT” timed out starting process “/u01/app/12c/grid/bin/lsnrctl” for action “check”: details at “(:CLSN00009:)” in “/u01/app/grid/diag/crs/node1/crs/trace/crsd_oraagent_grid.trc”
2021-08-26 10:10:32.503 [OCSSD(16630)]CRS-1608: This node was evicted by node 2, node2; details at (:CSSNM00005:) in /u01/app/grid/diag/crs/node1/crs/trace/ocssd.trc.
2021-08-26 10:10:35.078 [OCSSD(16630)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/node1/crs/trace/ocssd.trc
2021-08-26 10:10:40.531 [OCSSD(16630)]CRS-1652: Starting clean up of CRSD resources.
2021-08-26 10:10:50.327 [CSSDMONITOR(16598)]CRS-1661: The CSS daemon is not responding. Reboot will occur in 14030 milliseconds; Details at (:CLSN00111:) in /u01/app/grid/diag/crs/node1/crs/trace/ohasd_cssdmonitor_root.trc
2021-08-26 10:10:50.327 [CSSDAGENT(16615)]CRS-1661: The CSS daemon is not responding. Reboot will occur in 13990 milliseconds; Details at (:CLSN00111:) in /u01/app/grid/diag/crs/node1/crs/trace/ohasd_cssdagent_root.trc
2021-08-26 10:13:49.388 [OHASD(3182)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 3182
2021-08-26 10:13:49.485 [OHASD(3182)]CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
2021-08-26 10:13:49.524 [OHASD(3182)]CRS-2112: The OLR service started on node node1.
2021-08-26 10:13:49.580 [OHASD(3182)]CRS-1301: Oracle High Availability Service started on node node1.

从/u01/app/grid/diag/crs/node1/crs/trace/ohasd_orarootagent_root.trc上看日志如下
2021-08-26 09:47:52.116 : USRTHRD:3358193408: {0:5:3} HAIP: assigned ip ‘169.254.8.94’
2021-08-26 09:47:52.116 : USRTHRD:3358193408: {0:5:3} HAIP: check ip ‘169.254.8.94’
2021-08-26 09:47:52.116 : USRTHRD:3358193408: {0:5:3} Start: 1 HAIP assignment, 1, 1, 1, 1
2021-08-26 09:47:52.116 : USRTHRD:3358193408: {0:5:3} to verify wt, 1-1-1
2021-08-26 09:47:52.116 : USRTHRD:3358193408: {0:5:3} to verify inf event
2021-08-26 09:47:52.170 : AGENT:3382523648: {0:0:231} {0:0:231} Created alert : (:CRSAGF00113:) : Aborting the command: check for resource: ora.drivers.acfs 1 1
2021-08-26 09:47:52.170 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] (:CLSN00110:) clsn_agent::abort {
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] abort {
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Agent::doStateDump Default Agent Dump
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Agent::doStateDump last call info:
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:44:14.124 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck check clsagfw_res_status:0
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:44:14.124 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck exit clsagfw_res_status:0 }
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:45:12.978 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck entry {
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:45:12.978 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck notifyObservers
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:45:12.978 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck call check
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:45:12.990 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:DaemonAgent::check 100 entry
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:45:13.772 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:DaemonAgent::check 199 exit clsagfw_res_Status:0
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:45:13.772 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck check clsagfw_res_status:0
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:45:13.772 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck exit clsagfw_res_status:0 }
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:46:12.980 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck entry {
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:46:12.980 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck notifyObservers
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:46:12.980 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck call check
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:46:13.34 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:DaemonAgent::check 100 entry
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:46:13.675 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:DaemonAgent::check 199 exit clsagfw_res_Status:0
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:46:13.675 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck check clsagfw_res_status:0
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:46:13.675 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck exit clsagfw_res_status:0 }
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:47:12.971 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck entry {
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:47:12.971 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck notifyObservers
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:47:12.971 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:Agent::commonCheck call check
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] Time:08/26/2021 09:47:14.431 Tint:{0:0:231} action:104 resname:ora.drivers.acfs lastCall:DaemonAgent::check 100 entry
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] abort command: check
2021-08-26 09:47:52.171 :CLSDYNAM:3382523648: [ora.drivers.acfs]{0:0:231} [check] tryActionLock {
2021-08-26 09:47:52.224 :CLSDYNAM:3365713664: [ora.drivers.acfs]{0:0:231} [check] (:CLSN00009:)Utils:execCmd abort request: killing /u01/app/12c/grid/bin/acfsload
2021-08-26 09:47:54.040 : CLSDMC:2330683136: command 0 failed with status 16908801
CLSDMC:3363612416: Timeout [16000 sec] to receive ack from connection [ipc://node1_CTSSD]
2021-08-26 09:48:04.387 :CLSDYNAM:2330683136: [ora.crsd]{0:5:3} [check] DaemonAgent::check returned 0

我来答
添加附件
收藏
分享
问题补充
5条回答
默认
最新
Uncopyrightable
上传附件：alert.log
暂无图片 评论
暂无图片 有用 0
打赏 0
暂无图片
Uncopyrightable
上传附件：ohasd_orarootagent_root.trc
暂无图片 评论
暂无图片 有用 0
打赏 0
Uncopyrightable
上传附件：ohasd_orarootagent_root.trc
暂无图片 评论
暂无图片 有用 0
打赏 0
Uncopyrightable

正常理解，如果集群模式下，单节点异常宕机或者网卡异常不会影响整个数据库集群对外提供服务的，vip会漂移到另一个节点上，本次故障是单节点超时为什么会重启节点集群服务呐？

https://www.cnblogs.com/shenfeng/p/Oracle_12c_RAC_service_died.html

暂无图片 评论
暂无图片 有用 0
打赏 0
Uncopyrightable
升级问题到: 紧急故障
暂无图片 评论
暂无图片 有用 0
打赏 0
回答交流
Markdown


请输入正文
提交
相关推荐
怎么让oracle实现连续登陆失败5次，锁定账号5分钟
回答 5
最粗暴的做法，每分钟清空一次登录失败次数，当然我们假设用户不会在这个时间节点前后执行操作，就算刚好碰上了，也不过是1分钟变2分钟而已。
Oracle数据库记录用户最后登录时间的问题
回答 3
已采纳
selectt1.username,t1.logontimelastlogontime,t2.accountstatus,created账号创建时间from(selectusername,max(ti
Oracle12C RAC+DG单机 ORA-19660: some files in the backup set could not be verified
回答 5
1、oracle可执行文件权限是否正确；2、asm磁盘组状态是否正常3、oracle数组是否正确。参考mos:2545858.14、是否有rebalance未完成。
怎么从另外一张表调用对应的字段，用到update里的where条件？
回答 1
你如果能看懂下面这个sql的写法,估计就不会问这个问题了。。。updatetjcesiasetnumcount(selectcount()fromcgltjcesibwhereb.texta.text
SQLLDR 导入CSV，行末无逗号怎么处理？
回答 2
已采纳
fieldsterminatedby',' 其实就是把逗号两边的字段分开，就代表在有逗号的地方切一刀，行末是否有逗号或其他符号对于当前行其实没有什么影响，如果行末有其他符号配置文件中应该还需
报错OGG-01028
回答 6
意思是我这边OGG的版本低了吗😃
Oracle表空间加密，忘记wallet密码如何解密？
回答 2
基本无解。可以尝试暴力破解，如果隐约觉得自己的密码不是特别复杂。
Oracle角色如何继承？
回答 1
已采纳
一个角色可以继承其他角色的权限集合。例如角色MYROLE语句具备对表fruits的增加删除权限，此时创建一个新角色MYROLE01，该角色继承角色MYROLE的权限，具体的语句如下：GRANTMYRO
多文本行转列（pivot）的实现
回答 5
有几百万个ID，写在IN里确实不现实，但是我想问的是，为啥有这么个需求呢？是需求不合理还是结构设计不合理
oracle rac 12.2.0.1 ora-04031
回答 7
上传附件：175rac1awr2019102905.html.gz
问题信息
请登录之后查看
附件列表
请登录之后查看
邀请回答
暂无人订阅该标签，敬请期待~~
暂无图片墨值悬赏
50M
ORACLE 11.2.0 触发器 获取 空间属性的wkt格式，使用 sde.st_astext 函数，数据错乱
25M
求Cylance杀毒软件安装包测试数据库安全
25M
PostgreSQL ERROR: found xmin 2146512304 from before relfrozenxid 2146512317
10M
clickhouse 查询视图需要给基表的权限，有什么办法能场查视图不能查表呢？
10M
oracle apex是否可以使用nfc功能，以及录像保存功能
10M
视图v$sql_cs_selectivity 同一个RANGE_ID之下出现了同一个微词的重复记录。
10M
一个update 语句行锁升级 触发bug导致死锁myql宕机？请大佬帮忙分析下更具体原因
10M
时区升级
10M
Goldengate源端PDB，被克隆替代后，重新配置麻烦，有无其它方便，高效方法，恢复Goldengate？
10M
OGG使用COLMAP部分字段不同步问题
热门文章
从Oracle迁移到PostgreSQL的可行性研究
ORACLE如何生成SELECT表中所有字段的SQL语句
数据库管理-第137期 Oracle的数据生命周期管理（20240123）
Oracle 21C 新特性：Unicode 12。1支持
快讯：Oracle 23c 免费开发者版发布，快来下载！
Oracle 12C Data Guard安装及原理
Linux8利用RPM包一键部署19c环境
2020年了，还有必要学习Oracle数据库技术么？
小麦苗数据库巡检脚本，支持Oracle、MySQL、SQL Server和PG等数据库
Oracle 基础数据库服务新增 AMD VM 标准 E5 Flex 配置