暂无图片
rac开机后其中一个节点无法自动正常启动，请高手帮忙解答一下。
我来答
返回问答列表
分享
暂无图片 匿名用户
rac开机后其中一个节点无法自动正常启动，请高手帮忙解答一下。
oracle racoracle 19coracle数据库oracle

数据库版本：oracle 19.3 rac

操作系统：centos7.8

各种尝试后依然无法解决，请各位大神帮忙解决一下。


### 1.rac开机后其中一个节点无法自动正常启动

```bash
[grid@racdb1 ~]$ crsctl check cluster -all
**************************************************************
racdb1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racdb2:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
```

### 2.手动启动asm，可以启动。但是会报错OCR无法mount

```bash
SQL> startup
ASM instance started

Total System Global Area 1137173320 bytes
Fixed Size 8905544 bytes
Variable Size 1103101952 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15017: diskgroup "OCR" cannot be mounted
ORA-15040: diskgroup is incomplete
[grid@racdb2 trace]$ crsctl check cluster -all
**************************************************************
racdb1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racdb2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[grid@racdb2 trace]$ srvctl status database -d orcl
Instance orcl1 is running on node racdb1
Instance orcl2 is running on node racdb2
```
### 3.修改asm_diskgroups仍然报错
查看asm_diskgroups参数发现修改asm磁盘组，创建spfile的时候没有删除原来的磁盘组OCR，这里直接修改asm_diskgroups。

```bash
SQL> show parameter asm_diskgroups;
VALUE
------------------------------
asm_diskgroups string
ARCH, DATA, OCRVOTE

SQL> alter system set asm_diskgroups='DATA','ARCH','OCRNEW'
```
再次重启节点依然无法开机自动启动，手动重启还是报刚才的错误。

### 4.查看日志
查看crs日志
/u01/app/grid/diag/crs/racdb2/crs/trace/alert.log
OCR 磁盘OCRVOTE无法mount
```bash
2022-09-14 09:49:31.017 [OHASD(2381)]CRS-2771: Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2022-09-14 09:52:48.515 [ORAROOTAGENT(2510)]CRS-5019: All OCR locations are on ASM disk groups [OCRVOTE], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/racdb2/crs/trace/ohasd_orarootagent_root.trc".
```
查看
/u01/app/grid/diag/crs/racdb2/crs/trace/ohasd_orarootagent_root.trc
这里看到关键错误：ORA-01017: invalid username/password
```bash
2022-09-14 09:44:38.140 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] 9348 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS

2022-09-14 09:44:38.148 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] 9348 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS

2022-09-14 09:44:38.217 : CLSCRED:4202641152: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.05101c2993facfe9bf726dfa23bbcd4b.root not found
2022-09-14 09:44:38.217 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] 9066 Error 4 opening dom root in 0x7fb0bc178b80

2022-09-14 09:44:38.275 : USRTHRD:4213147392: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
2022-09-14 09:44:39.076 : USRTHRD:4213147392: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
2022-09-14 09:44:39.505 : default:4202641152: clsCredDomClose: Credctx deleted 0x7fb0bc255470
2022-09-14 09:44:39.505 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] -- trace dump on error exit --

2022-09-14 09:44:39.505 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] Error [kgfoAl06] in [kgfokge] at kgfo.c:3169

2022-09-14 09:44:39.505 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] ORA-01017: invalid username/password; logon denied
```
再次重启节点依然无法开机自动启动，手动重启还是报刚才的错误。
### 5.修改asm密码文件，依然报错

[grid@racdb1 ~]$ asmcmd lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE TRUE
ASMSNMP TRUE FALSE FALSE
CRSUSER__ASM_001 TRUE FALSE TRUE

将CRSUSER__ASM_001改为CRSUSER__ASM_002也是一样结果

我来答
添加附件
收藏
分享
问题补充
7条回答
默认
最新
刘贵宾
2022-09-14
检查检查文件权限
暂无图片 评论
暂无图片 有用 2
打赏 0
愤怒的蜗牛
2022-09-15
学习
许
2022-09-15

[grid@racdb2 ~]$ ll /dev/asm*
brw-rw---- 1 grid asmadmin 253, 7 Sep 15 09:17 /dev/asm-disk3
brw-rw---- 1 grid asmadmin 253, 5 Sep 15 09:15 /dev/asm-disk6
brw-rw---- 1 grid asmadmin 253, 6 Sep 15 09:15 /dev/asm-disk7


主要是两问题

1.手工启动其中一个节点asm实例，为什么会报ORC无法mount

ORC已经被卸载

2.无法自动重启

报权限错误的问题

暂无图片 评论
暂无图片 有用 0
打赏 0
Root__Liu
2022-09-15

参考下：How to recreate shared ASM password file in 12c GI cluster (Doc ID 1929673.1)

暂无图片 评论
暂无图片 有用 0
打赏 0
许
2022-09-15
[grid@racdb1 ~]$ asmcmd lspwusr Username sysdba sysoper sysasm SYS TRUE TRUE TRUE ASMSNMP TRUE FALSE FALSE CRSUSER__ASM_001 TRUE FALSE TRUE 将CRSUSER__ASM_001改为CRSUSER__ASM_002也是一样结果 按照那些操作过。
刘贵宾
2022-09-15

你在故障节点执行下这两个命令截图看一下

crsctl stat res -t

crsctl stat res -t -init

暂无图片 评论
暂无图片 有用 1
打赏 0
许
2022-09-15
后面评论发出来了。
cqiwen
2022-09-15
asm磁盘的权限感觉不对呀～把你的udev配置文件发出来看看呢
暂无图片 评论
暂无图片 有用 0
打赏 0
许
2022-09-15
[grid@racdb2 ~]$ cat /etc/udev/rules.d/99-oracle-asmdevices.rules #KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c4552384f546f46792d666c476c2d4e574d39", RUN+="/bin/sh -c 'mknod /dev/asm-disk2 b $major $minor; chown grid:asmadmin /dev/asm-disk2; chmod 0660 /dev/asm-disk2'" KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45526c7a6c46784d2d427633512d79636866", RUN+="/bin/sh -c 'mknod /dev/asm-disk3 b $major $minor; chown grid:asmadmin /dev/asm-disk3; chmod 0660 /dev/asm-disk3'" #KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c455277653366667a2d676576792d6a574934", RUN+="/bin/sh -c 'mknod /dev/asm-disk4 b $major $minor; chown grid:asmadmin /dev/asm-disk4; chmod 0660 /dev/asm-disk4'" #KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45523741436b64482d3175424c2d54725944", RUN+="/bin/sh -c 'mknod /dev/asm-disk5 b $major $minor; chown grid:asmadmin /dev/asm-disk5; chmod 0660 /dev/asm-disk5'" KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c455239387a4954382d507758342d74304553", RUN+="/bin/sh -c 'mknod /dev/asm-disk6 b $major $minor; chown grid:asmadmin /dev/asm-disk6; chmod 0660 /dev/asm-disk6'" KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c4552447a5637447a2d4b5563352d554c7064", RUN+="/bin/sh -c 'mknod /dev/asm-disk7 b $major $minor; chown grid:asmadmin /dev/asm-disk7; chmod 0660 /dev/asm-disk7'" [grid@racdb2 ~]$ ls -l /dev/asm* brw-rw---- 1 grid asmadmin 253, 3 Sep 15 12:11 /dev/asm-disk3 brw-rw---- 1 grid asmadmin 253, 7 Sep 15 11:32 /dev/asm-disk6 brw-rw---- 1 grid asmadmin 253, 4 Sep 15 11:32 /dev/asm-disk7
许
2022-09-15

重启节点后结果

[grid@racdb2 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racdb2 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE racdb2 STABLE
ora.crf
1 ONLINE ONLINE racdb2 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE racdb2 STABLE
ora.cssdmonitor
1 ONLINE ONLINE racdb2 STABLE
ora.ctssd
1 ONLINE ONLINE racdb2 ACTIVE:0,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE racdb2 STABLE
ora.gipcd
1 ONLINE ONLINE racdb2 STABLE
ora.gpnpd
1 ONLINE ONLINE racdb2 STABLE
ora.mdnsd
1 ONLINE ONLINE racdb2 STABLE
ora.storage
1 ONLINE ONLINE racdb2 STABLE
--------------------------------------------------------------------------------
[grid@racdb2 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[grid@racdb2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Sep 15 12:10:20 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.

Connected to an idle instance.

SQL>


开机后手动startup asm实列后的结果

[grid@racdb1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racdb1 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE racdb1 STABLE
ora.crf
1 ONLINE ONLINE racdb1 STABLE
ora.crsd
1 ONLINE ONLINE racdb1 STABLE
ora.cssd
1 ONLINE ONLINE racdb1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE racdb1 STABLE
ora.ctssd
1 ONLINE ONLINE racdb1 ACTIVE:0,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE racdb1 STABLE
ora.gipcd
1 ONLINE ONLINE racdb1 STABLE
ora.gpnpd
1 ONLINE ONLINE racdb1 STABLE
ora.mdnsd
1 ONLINE ONLINE racdb1 STABLE
ora.storage
1 ONLINE ONLINE racdb1 STABLE
--------------------------------------------------------------------------------
[grid@racdb1 ~]$
[grid@racdb1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
ora.chad
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
ora.net1.network
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
ora.ons
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ARCH.dg(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE racdb1 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE racdb1 Started,STABLE
2 ONLINE ONLINE racdb2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE racdb1 STABLE
ora.orcl.db
1 ONLINE ONLINE racdb1 Open,HOME=/u01/app/o
racle/product/19.3.0
/db_1,STABLE
2 ONLINE ONLINE racdb2 Open,HOME=/u01/app/o
racle/product/19.3.0
/db_1,STABLE
ora.qosmserver
1 ONLINE ONLINE racdb1 STABLE
ora.racdb1.vip
1 ONLINE ONLINE racdb1 STABLE
ora.racdb2.vip
1 ONLINE ONLINE racdb2 STABLE
ora.scan1.vip
1 ONLINE ONLINE racdb1 STABLE
--------------------------------------------------------------------------------

暂无图片 评论
暂无图片 有用 0
打赏 0
许
2022-09-20

问题未解决有没有人看一下。

暂无图片 评论
暂无图片 有用 0
打赏 0
回答交流
Markdown


请输入正文
提交
相关推荐
oracle12g存入的表数据存在乱码情况，各项编码配置都已是utf-8？
回答 3
原则上，和DB端字符集设置无关。一句话：NLSLANG里的NLSCHARCTERSET的设置，可以和DB端的NLSCHARCTERSET不一致（因为与DB端交互时，系统会自动做字符集转换），但必须和O
Oracle中的数据库和实例是怎么的关系？
回答 3
已采纳
在Oracle领域，“数据库”和“实例”容易混淆，作为Oracle术语，这两个词的定义如下：数据库（database)：操作系统文件或磁盘的集合。实例（instance）：一组Oracle后台进程/线
设置oracle数据库用户密码必须使用大小写+数字+特殊字符，如何解决校验问题？
回答 1
已采纳
oracle里有这几个函数,可以进行密码相关的一些校验ORA12CVERIFYFUNCTIONORA12CSTRONGVERIFYFUNCTIONORA12CSTIGVERIFYFUNCTIONora
sql优化
回答 2
执行计划看着很乱，给个sqlhc报告。另外gamsassetcard过滤之后的数据量是多少？temp、tempYe子查询返回数据量多少？
Oracle 11.2.0.4DG，备库强制开启为主库后，原来的主库用什么方法可以最快恢复成备库？
回答 3
已采纳
新主库备份个standby控制文件到旧主库进行恢复
oracle的XTTS迁移之后，表的ROWID是否会发生变化？
回答 1
已采纳
传输表空间，是直接复制块文件，我认为rowid是不会变化的。
RAC会崩溃吗？
回答 1
已采纳
当然会，糟糕的设计或选择会让它崩溃。除了数据库本身之外，还有许多组件参与提供数据库服务。RAC可能准备就绪并处在运行状态，但客户却无法访问；客户端和服务器之间的中间网络组件可能存在故障；破坏所有硬件的
含BLOB字段的800G的非分区大表如何清理历史数据
回答 3
忘了有adg，开了附加日志，没法不记日志
OGG MONITOR 告警邮件外发配置问题
回答 5
这是官网信息，请参考https://docs.oracle.com/goldengate/m1221/ggmonitor/GMNCH/alerts2apps.htmGMNCH195
ORA-04030 alert日志显示内存不足，但是从数据库重启之前的awr里看不到pga的激增，为什么
回答 3
检查一下：pgamaxsize这个参数的值看是否是这个值过大导致的，如果是这个值可以适当调整这个值Bug9506362ORA04030:OUTOFPROCESSMEMORY(QERHJHASHJOI,
问题信息
请登录之后查看
邀请回答
暂无人订阅该标签，敬请期待~~
暂无图片墨值悬赏
25M
PostgreSQL ERROR: found xmin 2146512304 from before relfrozenxid 2146512317
25M
oracle12c有主键的表做了闪回数据归档但某段时间闪回查询出现重复记录，为什么？
10M
clickhouse 查询视图需要给基表的权限，有什么办法能场查视图不能查表呢？
10M
Goldengate源端PDB，被克隆替代后，重新配置麻烦，有无其它方便，高效方法，恢复Goldengate？
10M
oracle apex是否可以使用nfc功能，以及录像保存功能
10M
一个update 语句行锁升级 触发bug导致死锁myql宕机？请大佬帮忙分析下更具体原因
10M
时区升级
10M
oracle pdb refresh 迁移
10M
视图v$sql_cs_selectivity 同一个RANGE_ID之下出现了同一个微词的重复记录。
10M
OGG使用COLMAP部分字段不同步问题