问题描述：安装oracle 12c rac节点2在打 33610989 补丁时出现"cannot move libasmclntsh12.so"的告警，如下所示：

数据库：oracle 12.2.0.1(2节点rac)

系统：rhel 7.9 64位

1、异常重现

[root@cbdps02 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/33610989 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Fri Feb 28 17:37:56 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-28_05-38-00PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-28_05-38-27PM.log

The id for this session is FBY7

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps02/crsconfig/crspatch_cbdps02_2025-02-28_05-38-41PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Failed while applying binary patches on home /u01/app/12.2.0.1/grid

 

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : cbdps02->/u01/app/12.2.0.1/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/12.2.0.1/grid, host: cbdps02.

Command failed:  /u01/app/12.2.0.1/grid/OPatch/opatchauto  apply /u01/setup/RU/33583921/33610989 -oh /u01/app/12.2.0.1/grid -target_type cluster -binary -invPtrLoc /u01/app/12.2.0.1/grid//oraInst.loc -jre /u01/app/12.2.0.1/grid/OPatch/jre -persistresult /u01/app/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_cbdps02_crs_1.ser -analyzedresult /u01/app/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_cbdps02_crs_1.ser

Command failure output:

==Following patches FAILED in apply:

 

Patch: /u01/setup/RU/33583921/33610989

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-28_17-38-44PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

make: *** [update_patchrep] Error 1

'

 

After fixing the cause of failure Run opatchauto resume

 

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

 

OPatchauto session completed at Fri Feb 28 17:39:00 2025

Time taken to complete the session 1 minute, 4 seconds

 

 opatchauto failed with error code 42

 

Doc ID 31493961.8

 

2、解决方案

-- 此时像节点1一样执行以下两步.

[root@cbdps02 RU]# /u01/app/12.2.0.1/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_PREPATCH -state START

[root@cbdps02 RU]# /u01/app/12.2.0.1/grid/crs/install/rootcrs.sh -prepatch

Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/cbdps02/crsconfig/crspatch_cbdps02_2025-03-02_06-24-39PM.log

Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [3721916208].

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cbdps02'

CRS-2673: Attempting to stop 'ora.crsd' on 'cbdps02'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'cbdps02'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.chad' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'cbdps02'

CRS-2677: Stop of 'ora.DATA.dg' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.OCR.dg' on 'cbdps02'

CRS-2677: Stop of 'ora.OCR.dg' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'cbdps02'

CRS-2677: Stop of 'ora.asm' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'cbdps02'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'cbdps02'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.cbdps02.vip' on 'cbdps02'

CRS-2677: Stop of 'ora.chad' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.cbdps02.vip' on 'cbdps02' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'cbdps01'

CRS-2672: Attempting to start 'ora.cbdps02.vip' on 'cbdps01'

CRS-2676: Start of 'ora.cbdps02.vip' on 'cbdps01' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'cbdps01' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'cbdps01'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'cbdps02'

CRS-2677: Stop of 'ora.ons' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'cbdps02'

CRS-2677: Stop of 'ora.net1.network' on 'cbdps02' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'cbdps02' has completed

CRS-2677: Stop of 'ora.crsd' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.storage' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.crf' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'cbdps02'

CRS-2677: Stop of 'ora.drivers.acfs' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.crf' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.storage' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'cbdps02'

CRS-2677: Stop of 'ora.mdnsd' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.asm' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'cbdps02'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.evmd' on 'cbdps02'

CRS-2677: Stop of 'ora.evmd' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'cbdps02'

CRS-2677: Stop of 'ora.cssd' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'cbdps02'

CRS-2677: Stop of 'ora.gipcd' on 'cbdps02' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cbdps02' has completed

CRS-4133: Oracle High Availability Services has been stopped.

2025/03/02 18:25:22 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.

2025/03/02 18:25:23 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.

2025/03/02 18:25:24 CLSRSC-347: Successfully unlock /u01/app/12.2.0.1/grid

 

n  在此apply补丁时却出现"Grid is not running on the local host"的报警.

[root@cbdps02 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/33610989 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Sun Mar  2 18:25:42 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-03-02_06-25-46PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-03-02_06-25-50PM.log

The id for this session is H8KJ

OPATCHAUTO-72132: Grid is not running on the local host.

OPATCHAUTO-72132: Cannot start a new apply or rollback session when the local grid is not running.

OPATCHAUTO-72132: Please start grid service on the local host to start patching.

OPatchAuto failed.

 

OPatchauto session completed at Sun Mar  2 18:25:56 2025

Time taken to complete the session 0 minute, 15 seconds

 

 opatchauto failed with error code 42

 

-- lock资源后再次拉起集群 

[root@cbdps02 RU]# /u01/app/12.2.0.1/grid/crs/install/rootcrs.sh -lock 

Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/cbdps02/crsconfig/crslock_cbdps02_2025-03-02_06-26-50PM.log

2025/03/02 18:26:53 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

[root@cbdps02 RU]# crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

 

 

[root@cbdps02 ~]# /u01/app/12.2.0.1/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PREPATCH -status

SUCCESS

[root@cbdps02 ~]# /u01/app/12.2.0.1/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_PREPATCH -state START

 

3、打33610989和26839277补丁

[root@cbdps02 ~]# cd /tmp

[root@cbdps02 tmp]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/33610989 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Tue Mar  4 12:54:44 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-03-04_12-54-47PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-03-04_12-55-11PM.log

The id for this session is INXC

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps02/crsconfig/crspatch_cbdps02_2025-03-04_12-55-24AM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Binary patch applied successfully on home /u01/app/12.2.0.1/grid

 

 

Running rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

Successfully executed rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

 

 

Performing postpatch operations on CRS - starting CRS service on home /u01/app/12.2.0.1/grid

Postpatch operation log file location: /u01/app/grid/crsdata/cbdps02/crsconfig/crspatch_cbdps02_2025-03-04_12-56-28AM.log

CRS service started successfully on home /u01/app/12.2.0.1/grid

 

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Patching is completed successfully. Please find the summary as follows:

 

Host:cbdps02

CRS Home:/u01/app/12.2.0.1/grid

Version:12.2.0.1.0

Summary:

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /u01/setup/RU/33583921/33610989

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-03-04_12-56-16PM_1.log

 

 

 

OPatchauto session completed at Tue Mar  4 12:59:14 2025

Time taken to complete the session 4 minutes, 30 seconds

[root@cbdps02 tmp]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/26839277 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Tue Mar  4 12:59:50 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-03-04_12-59-53PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-03-04_01-00-22PM.log

The id for this session is 41DR

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps02/crsconfig/crspatch_cbdps02_2025-03-04_01-00-37PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Binary patch applied successfully on home /u01/app/12.2.0.1/grid

 

 

Running rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

Successfully executed rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

 

 

Performing postpatch operations on CRS - starting CRS service on home /u01/app/12.2.0.1/grid

Postpatch operation log file location: /u01/app/grid/crsdata/cbdps02/crsconfig/crspatch_cbdps02_2025-03-04_01-03-09PM.log

CRS service started successfully on home /u01/app/12.2.0.1/grid

 

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Patching is completed successfully. Please find the summary as follows:

 

Host:cbdps02

CRS Home:/u01/app/12.2.0.1/grid

Version:12.2.0.1.0

Summary:

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /u01/setup/RU/33583921/26839277

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-03-04_13-03-00PM_1.log

 

 

 

OPatchauto session completed at Tue Mar  4 13:06:23 2025

Time taken to complete the session 6 minutes, 33 seconds

 

说明：如上所示，成功安装补丁.

 

参考文档：Doc ID 27554103.8