基础环境

操作系统：redhat 7.9
数据库：oracle 11.2.0.4
架构：oracle rac

问题描述

远程连接服务器对大数据库集群补丁，但是由于中间时间稍微长点超过了ssh允许的空闲时间，导致连接断开，但是集群补丁此时还没有完成，再次启动集群以及数据库失败

问题过程

打补丁

root@rac01 bin]# cd ../OPatch
[root@rac01 OPatch]# ./opatch auto /u01/app/grid/opatch0211/32495145
Executing /u01/app/grid/product/11.2.0/grid/perl/bin/perl /u01/app/grid/product/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /u01/app/grid/opatch0211 -patchn 32495145 -paramfile /u01/app/grid/product/11.2.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/grid/product/11.2.0/grid/cfgtoollogs/opatchauto2024-04-15_22-53-20.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/grid/product/11.2.0/grid/cfgtoollogs/opatchauto2024-04-15_22-53-20.report.log

2024-04-15 22:53:20: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/grid/product/11.2.0/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.0/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/db_1 successfully

patch /u01/app/grid/opatch0211/32495145/32328626  apply successful for home  /u01/app/oracle/product/11.2.0/db_1 

Stopping CRS...
The Oracle Grid Infrastructure stack failed to stop.
Resetting the status, starting CRS...

超时断开

打补丁的时间超过了shell默认的时间，导致连接中断，补丁没有完成。

尝试启动集群

 root@rac01 bin]#  ./crsctl start crs
sql
部分报错信息如下：
```sql

ORA-15025: could not open disk "/dev/asm-DATA"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
ORA-15025: could not open disk "/dev/asm-FRA"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
SUCCESS: diskgroup FRA was dismounted
ERROR: diskgroup FRA was not mounted
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+FRA/sxdb/controlfile/current.256.967224223'
ORA-17503: ksfdopn:2 Failed to open file +FRA/sxdb/controlfile/current.256.967224223
ORA-15001: diskgroup "FRA" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/sxdb/controlfile/current.260.967224223'
ORA-17503: ksfdopn:2 Failed to open file +DATA/sxdb/controlfile/current.260.967224223
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-205 signalled during: alter database mount...

问题处理

1、ASM磁盘组权限排查

出现了ORA-15025和ORA-27041错误。这些错误通常表示Oracle无法打开指定的磁盘或文件，并且出现了权限被拒绝的问题

[root@rac01 ~]# multipath -ll
asm_ocr2 (36000c296003109934736cb97d5c93a0a) dm-3 VMware, ,VMware Virtual S
size=30G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 33:0:1:0 sdd 8:48 active ready running
asm_ocr1 (36000c298f954dd10f0acabab9a7fe7e2) dm-2 VMware, ,VMware Virtual S
size=30G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 33:0:0:0 sdc 8:32 active ready running
asm_data (36000c298eeaecd1c1536043cc0662d50) dm-4 VMware, ,VMware Virtual S
size=100G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 33:0:3:0 sdf 8:80 active ready running
asm_ocr3 (36000c29f932fbc0baa1d764302c836db) dm-5 VMware, ,VMware Virtual S
size=30G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 33:0:2:0 sde 8:64 active ready running
[root@swp5 ~]# ll /dev/dm-*
brw-rw---- 1 root disk     253, 0 Apr 15 10:48 /dev/dm-0
brw-rw---- 1 root disk     253, 1 Apr 15 10:48 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Apr 15 11:41 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Apr 15 11:41 /dev/dm-3
brw-rw---- 1 grid asmadmin 253, 4 Apr 15 11:41 /dev/dm-4
brw-rw---- 1 grid asmadmin 253, 5 Apr 15 11:41 /dev/dm-5
brw-rw---- 1 root disk     253, 6 Apr 15 10:48 /dev/dm-6

发现都是正常的

2、oracle文件权限

   如下发现oracle文件权限发生更改了

[root@rac1 ~]# su - oracle
Last login: Mon Apr 15 11:39:33 CST 2024
[oracle@rac1 ~]$ cd $ORACLE_HOME/bin
[oracle@rac1 bin]$ ll oracle
-rw-r--x 1 root asmadmin 457067896 Sep  6  2020 oracle

[grid@rac1 ~]$ cd $ORACLE_HOME/bin
[grid@rac1 bin]$ ll oracle
-rw-r--x 1 root asmadmin 457067896 Sep  6  2020 oracle

3、调整权限

-rwsr-s--x 1 oracle asmadmin 457067896 Sep  6 2020 oracle
-rwsr-s--x 1 grid oinstall 456023808 Sep  6  2020 oracle

启动集群

启动集群服务，查看状态已经正常了

CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

打补丁过程中异常中断，导致之前修改后的文件权限等没有更改过来，文件无法访问，导致集群启动失败。