问题描述：安装oracle 12c rac节点1在打33583921 RU时出现"Failed to bring down CRS service on home"告警，如下所示：

数据库：oracle 12.2.0.1(2节点rac)

系统：rhel 7.9 64位

1、问题重现

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921 -analyze

 

OPatchauto session is initiated at Wed Feb 26 10:32:08 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_10-32-10AM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_10-32-32AM.log

The id for this session is 3XY6

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Analysis for applying patches has completed successfully:

 

Host:cbdps01

CRS Home:/u01/app/12.2.0.1/grid

Version:12.2.0.1.0

 

 

==Following patches were SKIPPED:

 

Patch: /u01/setup/RU/33583921/33116894

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_10-32-48AM_1.log

Reason: /u01/setup/RU/33583921/33116894 is not required to be applied to oracle home /u01/app/12.2.0.1/grid

 

 

==Following patches were SUCCESSFULLY analyzed to be applied:

 

Patch: /u01/setup/RU/33583921/33678030

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_10-32-48AM_1.log

 

Patch: /u01/setup/RU/33583921/26839277

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_10-32-48AM_1.log

 

Patch: /u01/setup/RU/33583921/33610989

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_10-32-48AM_1.log

 

Patch: /u01/setup/RU/33583921/33587128

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_10-32-48AM_1.log

 

 

 

OPatchauto session completed at Wed Feb 26 10:32:59 2025

Time taken to complete the session 0 minute, 52 seconds

 

 

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921

 

OPatchauto session is initiated at Wed Feb 26 10:33:21 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_10-33-23AM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_10-33-44AM.log

The id for this session is E9ZG

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_10-34-13AM.log

Failed to bring down CRS service on home /u01/app/12.2.0.1/grid

 

Execution of [GIShutDownAction] patch action failed, check log for more details. Failures:

Patch Target : cbdps01->/u01/app/12.2.0.1/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/12.2.0.1/grid, host: cbdps01.

Command failed:  bash -c "LD_LIBRARY_PATH=/u01/app/12.2.0.1/grid/lib:/u01/app/12.2.0.1/grid/srvm/lib:/u01/setup/RU/33583921/33678030/files/lib:/u01/app/12.2.0.1/grid/lib:/u01/app/12.2.0.1/grid/lib:/u01/app/12.2.0.1/grid/srvm/lib /u01/app/12.2.0.1/grid/perl/bin/perl -I/u01/app/12.2.0.1/grid/perl/lib -I/u01/app/12.2.0.1/grid/opatchautocfg/db/dbtmp/bootstrap_cbdps01/patchwork/crs/install /u01/app/12.2.0.1/grid/opatchautocfg/db/dbtmp/bootstrap_cbdps01/patchwork/crs/install/rootcrs.pl -prepatch"

Command failure output:

Using configuration parameter file: /u01/app/12.2.0.1/grid/opatchautocfg/db/dbtmp/bootstrap_cbdps01/patchwork/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_10-34-13AM.log

Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3721916208].

ORA-15158: rolling upgrade prevented by One or more modules not ready for RM

CRS-4698: Error code 2 in retrieving the patch levelCRS-1154: There was an error setting Oracle ASM to rolling patch mode.

CRS-4000: Command Start failed, or completed with errors.

2025/02/26 10:35:25 CLSRSC-430: Failed to start rolling patch mode

 

After fixing the cause of failure Run opatchauto resume

 

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

 

OPatchauto session completed at Wed Feb 26 10:35:27 2025

Time taken to complete the session 2 minutes, 7 seconds

 

 opatchauto failed with error code 42

 

# more /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_10-34-13AM.log

......

2025-02-26 10:34:22: Executing cmd: /u01/app/12.2.0.1/grid/bin/crsctl start rollingpatch

2025-02-26 10:35:25: Command output:

>  ORA-15158: rolling upgrade prevented by One or more modules not ready for RM

>  CRS-4698: Error code 2 in retrieving the patch levelCRS-1154: There was an error setting Oracle ASM to rolling patch mode.

>  CRS-4000: Command Start failed, or completed with errors.

>End Command output

2025-02-26 10:35:25: Executing cmd: /u01/app/12.2.0.1/grid/bin/clsecho -p has -f clsrsc -m 430

2025-02-26 10:35:25: Command output:

>  CLSRSC-430: Failed to start rolling patch mode

>End Command output

2025-02-26 10:35:25: CLSRSC-430: Failed to start rolling patch mode

2025-02-26 10:35:25: ###### Begin DIE Stack Trace ######

2025-02-26 10:35:25:     Package         File                 Line Calling  

2025-02-26 10:35:25:     --------------- -------------------- ---- ----------

2025-02-26 10:35:25:  1: main            rootcrs.pl            287 crsutils::dietrap

2025-02-26 10:35:25:  2: crspatch        crspatch.pm          1121 main::__ANON__

2025-02-26 10:35:25:  3: crspatch        crspatch.pm          1017 crspatch::performPrePatch

2025-02-26 10:35:25:  4: crspatch        crspatch.pm           258 crspatch::crsPrePatch

2025-02-26 10:35:25:  5: main            rootcrs.pl            296 crspatch::new

2025-02-26 10:35:25: ####### End DIE Stack Trace #######

 

2025-02-26 10:35:25: ROOTCRS_PREPATCH checkpoint has failed

2025-02-26 10:35:25:      ckpt: -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PREPATCH

2025-02-26 10:35:25: Invoking "/u01/app/12.2.0.1/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PREPATCH"

2025-02-26 10:35:25: trace file=/u01/app/grid/crsdata/cbdps01/crsconfig/cluutil8.log

2025-02-26 10:35:25: Running as user grid: /u01/app/12.2.0.1/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PREPATCH

2025-02-26 10:35:25: s_run_as_user2: Running /bin/su grid -c ' echo CLSRSC_START; /u01/app/12.2.0.1/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PREPATCH

 

说明：如上所示，关闭CRS服务失败，告警信息为"Failed to bring down CRS service on home /u01/app/12.2.0.1/grid".

 

2、解决方案

分别打33583921下的补丁，因为在安装GI时 33116894 补丁已提前打上，所以此处忽略该补丁.

2.1、安装33587128补丁

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/33587128 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Wed Feb 26 17:33:13 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_05-33-14PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_05-33-36PM.log

The id for this session is C3JM

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_05-33-55PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Binary patch applied successfully on home /u01/app/12.2.0.1/grid

 

 

Running rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

Successfully executed rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

 

 

Performing postpatch operations on CRS - starting CRS service on home /u01/app/12.2.0.1/grid

 

Postpatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_05-38-42PM.log

CRS service started successfully on home /u01/app/12.2.0.1/grid

 

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Patching is completed successfully. Please find the summary as follows:

 

Host:cbdps01

CRS Home:/u01/app/12.2.0.1/grid

Version:12.2.0.1.0

Summary:

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /u01/setup/RU/33583921/33587128

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_17-34-58PM_1.log

 

 

 

OPatchauto session completed at Wed Feb 26 17:51:34 2025

Time taken to complete the session 18 minutes, 22 seconds

 

说明：安装该补丁无异常发生.

 

2.2、安装33678030补丁

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/33678030 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Wed Feb 26 17:53:05 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_05-53-08PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_05-53-32PM.log

The id for this session is 17IM

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_05-53-50PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Binary patch applied successfully on home /u01/app/12.2.0.1/grid

 

 

Running rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

Successfully executed rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

 

 

Performing postpatch operations on CRS - starting CRS service on home /u01/app/12.2.0.1/grid

Postpatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_05-58-00PM.log

CRS service started successfully on home /u01/app/12.2.0.1/grid

 

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Patching is completed successfully. Please find the summary as follows:

 

Host:cbdps01

CRS Home:/u01/app/12.2.0.1/grid

Version:12.2.0.1.0

Summary:

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /u01/setup/RU/33583921/33678030

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_17-56-18PM_1.log

 

 

 

OPatchauto session completed at Wed Feb 26 18:03:55 2025

Time taken to complete the session 10 minutes, 50 seconds

 

说明：安装该补丁无异常发生.

 

2.3、安装33610989补丁

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/33610989

 

OPatchauto session is initiated at Wed Feb 26 18:04:08 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_06-04-10PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_06-04-52PM.log

The id for this session is FCDY

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_06-05-08PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Failed while applying binary patches on home /u01/app/12.2.0.1/grid

 

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : cbdps01->/u01/app/12.2.0.1/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/12.2.0.1/grid, host: cbdps01.

Command failed:  /u01/app/12.2.0.1/grid/OPatch/opatchauto  apply /u01/setup/RU/33583921/33610989 -oh /u01/app/12.2.0.1/grid -target_type cluster -binary -invPtrLoc /u01/app/12.2.0.1/grid/oraInst.loc -jre /u01/app/12.2.0.1/grid/OPatch/jre -persistresult /u01/app/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_cbdps01_crs_1.ser -analyzedresult /u01/app/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_cbdps01_crs_1.ser

Command failure output:

==Following patches FAILED in apply:

 

Patch: /u01/setup/RU/33583921/33610989

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_18-05-12PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

make: *** [update_patchrep] Error 1

'

 

After fixing the cause of failure Run opatchauto resume

 

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

 

OPatchauto session completed at Wed Feb 26 18:05:52 2025

Time taken to complete the session 1 minute, 44 seconds

 

 opatchauto failed with error code 42

 

[root@cbdps01 RU]# cat /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_18-05-12PM_1.log | grep -i error -C5

[Feb 26, 2025 6:05:33 PM] [INFO]    Copying file to "/u01/app/12.2.0.1/grid/tomcat/lib/tomcat-util.jar"

[Feb 26, 2025 6:05:33 PM] [INFO]    [OPSR-TIME] Finished modifying the system for patch 33610989

[Feb 26, 2025 6:05:33 PM] [INFO]    Start invoking 'patchgen' at Wed Feb 26 18:05:33 CST 2025

[Feb 26, 2025 6:05:33 PM] [INFO]    Command call is : /u01/app/12.2.0.1/grid/bin/patchgen commit -pi 33610989

[Feb 26, 2025 6:05:34 PM] [INFO]    mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

[Feb 26, 2025 6:05:34 PM] [WARNING] OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

                                    '

[Feb 26, 2025 6:05:34 PM] [INFO]    Restoring "/u01/app/12.2.0.1/grid" to the state prior to running NApply...

[Feb 26, 2025 6:05:34 PM] [INFO]    Restoring files: copy recurse from /u01/app/12.2.0.1/grid/.patch_storage/NApply/2025-02-26_18-05-12PM/backup to /u01/app/12.2.0.1/grid

[Feb 26, 2025 6:05:48 PM] [INFO]    Removed patch "33610989" with UPI + "24523071" from OUI inventory memory..

[Feb 26, 2025 6:05:48 PM] [INFO]    Restoring files: delete new file in /u01/app/12.2.0.1/grid/install/acls/oracle.tomcat.crs.acl

--

                                      - files are restored properly.

                                      - binaries are re-linked correctly.

                                    (use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under

                                    "/u01/app/12.2.0.1/grid/.patch_storage/NApply/2025-02-26_18-05-12PM"

[Feb 26, 2025 6:05:49 PM] [SEVERE]  OUI-67073:UtilSession failed: ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

                                    '

[Feb 26, 2025 6:05:49 PM] [INFO]    --------------------------------------------------------------------------------

[Feb 26, 2025 6:05:49 PM] [INFO]    The following warnings have occurred during OPatch execution:

[Feb 26, 2025 6:05:49 PM] [INFO]    1) OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

                                    '

[Feb 26, 2025 6:05:49 PM] [INFO]    2) OUI-67124:

                                    NApply restored the home. Please check your ORACLE_HOME to make sure:

                                      - files are restored properly.

                                      - binaries are re-linked correctly.

 

说明：安装33610989补丁时出现 "cannot move 'GI_HOME/lib/libasmclntsh12.so' to 'GI_HOME/lib/libasmclntsh12.so.bak'的异常".

 

[grid@cbdps01 ~]$ ls -tlr /u01/app/12.2.0.1/grid/lib/libasmclntsh12.so

-rw-r--r-- 1 root oinstall 2058624 Feb 26 17:57 /u01/app/12.2.0.1/grid/lib/libasmclntsh12.so

 

说明：此处根据报错显示libasmclntsh12.so文件的权限不足导致，尝试将其属主改为grid:oinstall或者改为777的权限均不行，同时在节点2打该补丁时也遇到此问题.

 

2.4、安装33610989补丁

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/26839277

 

OPatchauto session is initiated at Wed Feb 26 18:08:15 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_06-08-18PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_06-08-46PM.log

The id for this session is 7PES

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_06-09-01PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Failed while applying binary patches on home /u01/app/12.2.0.1/grid

 

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:

Patch Target : cbdps01->/u01/app/12.2.0.1/grid Type[crs]

Details: [

---------------------------Patching Failed---------------------------------

Command execution failed during patching in home: /u01/app/12.2.0.1/grid, host: cbdps01.

Command failed:  /u01/app/12.2.0.1/grid/OPatch/opatchauto  apply /u01/setup/RU/33583921/26839277 -oh /u01/app/12.2.0.1/grid -target_type cluster -binary -invPtrLoc /u01/app/12.2.0.1/grid/oraInst.loc -jre /u01/app/12.2.0.1/grid/OPatch/jre -persistresult /u01/app/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_cbdps01_crs_1.ser -analyzedresult /u01/app/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_cbdps01_crs_1.ser

Command failure output:

==Following patches FAILED in apply:

 

Patch: /u01/setup/RU/33583921/26839277

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_18-09-05PM_1.log

Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

make: *** [update_patchrep] Error 1

'

 

After fixing the cause of failure Run opatchauto resume

 

]

OPATCHAUTO-68061: The orchestration engine failed.

OPATCHAUTO-68061: The orchestration engine failed with return code 1

OPATCHAUTO-68061: Check the log for more details.

OPatchAuto failed.

 

OPatchauto session completed at Wed Feb 26 18:10:11 2025

Time taken to complete the session 1 minute, 56 seconds

 

 opatchauto failed with error code 42

 

[root@cbdps01 RU]# cat /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_18-09-05PM_1.log | grep -i error -C5

                                    the file in incoming patch "/u01/setup/RU/33583921/26839277/files/wlm/mesg/wlmvus.msg"

[Feb 26, 2025 6:09:24 PM] [INFO]    [OPSR-TIME] Finished modifying the system for patch 26839277

[Feb 26, 2025 6:09:24 PM] [INFO]    Start invoking 'patchgen' at Wed Feb 26 18:09:24 CST 2025

[Feb 26, 2025 6:09:24 PM] [INFO]    Command call is : /u01/app/12.2.0.1/grid/bin/patchgen commit -pi 26839277

[Feb 26, 2025 6:09:25 PM] [INFO]    mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

[Feb 26, 2025 6:09:25 PM] [WARNING] OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

                                    '

[Feb 26, 2025 6:09:25 PM] [INFO]    Restoring "/u01/app/12.2.0.1/grid" to the state prior to running NApply...

[Feb 26, 2025 6:09:25 PM] [INFO]    Restoring files: copy recurse from /u01/app/12.2.0.1/grid/.patch_storage/NApply/2025-02-26_18-09-05PM/backup to /u01/app/12.2.0.1/grid

[Feb 26, 2025 6:10:07 PM] [INFO]    Removed patch "26839277" with UPI + "21578760" from OUI inventory memory..

[Feb 26, 2025 6:10:08 PM] [INFO]    Checking if OPatch needs to invoke 'make' to restore some binaries...

--

                                      - files are restored properly.

                                      - binaries are re-linked correctly.

                                    (use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under

                                    "/u01/app/12.2.0.1/grid/.patch_storage/NApply/2025-02-26_18-09-05PM"

[Feb 26, 2025 6:10:09 PM] [SEVERE]  OUI-67073:UtilSession failed: ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

                                    '

[Feb 26, 2025 6:10:09 PM] [INFO]    --------------------------------------------------------------------------------

[Feb 26, 2025 6:10:09 PM] [INFO]    The following warnings have occurred during OPatch execution:

[Feb 26, 2025 6:10:09 PM] [INFO]    1) OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: mv: cannot move ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so’ to ‘/u01/app/12.2.0.1/grid/lib/libasmclntsh12.so.bak’: Permission denied

                                    make: *** [update_patchrep] Error 1

                                    '

[Feb 26, 2025 6:10:09 PM] [INFO]    2) OUI-67124:

                                    NApply restored the home. Please check your ORACLE_HOME to make sure:

                                      - files are restored properly.

                                      - binaries are re-linked correctly.

 

 

说明：安装33610989补丁时出现 "cannot move 'GI_HOME/lib/libasmclntsh12.so' to 'GI_HOME/lib/libasmclntsh12.so.bak'的异常".

 

3、处理 cannot move libasmclntsh12.so的异常

--此处记录处理节点1 cannot move libasmclntsh12.so的异常                             

[root@cbdps01 ~]# /u01/app/12.2.0.1/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_PREPATCH -state START

[root@cbdps01 ~]# /u01/app/12.2.0.1/grid/crs/install/rootcrs.sh -prepatch

Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_11-29-45PM.log

Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [3721916208].

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cbdps01'

CRS-2673: Attempting to stop 'ora.crsd' on 'cbdps01'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'cbdps01'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.chad' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.chad' on 'cbdps02'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.OCR.dg' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.cvu' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.qosmserver' on 'cbdps01'

CRS-2677: Stop of 'ora.DATA.dg' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.OCR.dg' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'cbdps01'

CRS-2677: Stop of 'ora.cvu' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'cbdps01'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.scan3.vip' on 'cbdps01'

CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.scan2.vip' on 'cbdps01'

CRS-2677: Stop of 'ora.asm' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'cbdps01'

CRS-2677: Stop of 'ora.chad' on 'cbdps02' succeeded

CRS-2677: Stop of 'ora.chad' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.mgmtdb' on 'cbdps01'

CRS-2677: Stop of 'ora.scan1.vip' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.qosmserver' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.scan2.vip' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.mgmtdb' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'cbdps01'

CRS-2677: Stop of 'ora.scan3.vip' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.MGMTLSNR' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.cbdps01.vip' on 'cbdps01'

CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'cbdps02'

CRS-2677: Stop of 'ora.cbdps01.vip' on 'cbdps01' succeeded

CRS-2676: Start of 'ora.MGMTLSNR' on 'cbdps02' succeeded

CRS-2672: Attempting to start 'ora.qosmserver' on 'cbdps02'

CRS-2672: Attempting to start 'ora.scan1.vip' on 'cbdps02'

CRS-2672: Attempting to start 'ora.scan2.vip' on 'cbdps02'

CRS-2672: Attempting to start 'ora.scan3.vip' on 'cbdps02'

CRS-2672: Attempting to start 'ora.cbdps01.vip' on 'cbdps02'

CRS-2672: Attempting to start 'ora.cvu' on 'cbdps02'

CRS-2676: Start of 'ora.cvu' on 'cbdps02' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'cbdps02' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'cbdps02'

CRS-2676: Start of 'ora.scan2.vip' on 'cbdps02' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'cbdps02'

CRS-2676: Start of 'ora.scan3.vip' on 'cbdps02' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'cbdps02'

CRS-2676: Start of 'ora.cbdps01.vip' on 'cbdps02' succeeded

CRS-2676: Start of 'ora.qosmserver' on 'cbdps02' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'cbdps02' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'cbdps02' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'cbdps02' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'cbdps01'

CRS-2677: Stop of 'ora.ons' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'cbdps01'

CRS-2677: Stop of 'ora.net1.network' on 'cbdps01' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'cbdps01' has completed

CRS-2677: Stop of 'ora.crsd' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.crf' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'cbdps01'

CRS-2677: Stop of 'ora.drivers.acfs' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.crf' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.asm' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'cbdps01'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'cbdps01'

CRS-2673: Attempting to stop 'ora.evmd' on 'cbdps01'

CRS-2677: Stop of 'ora.ctssd' on 'cbdps01' succeeded

CRS-2677: Stop of 'ora.evmd' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'cbdps01'

CRS-2677: Stop of 'ora.cssd' on 'cbdps01' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'cbdps01'

CRS-2677: Stop of 'ora.gipcd' on 'cbdps01' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cbdps01' has completed

CRS-4133: Oracle High Availability Services has been stopped.

2025/02/26 23:30:58 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.

2025/02/26 23:32:29 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.

2025/02/26 23:32:30 CLSRSC-347: Successfully unlock /u01/app/12.2.0.1/grid

 

4、打33610989和26839277补丁

[root@cbdps01 ~]# cd /u01/setup/RU

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/33610989 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Wed Feb 26 23:44:45 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_11-44-49PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_11-44-54PM.log

The id for this session is X3VR

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_11-45-05PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Binary patch applied successfully on home /u01/app/12.2.0.1/grid

 

 

Running rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

Successfully executed rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

 

 

Performing postpatch operations on CRS - starting CRS service on home /u01/app/12.2.0.1/grid

Postpatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_11-45-34PM.log

CRS service started successfully on home /u01/app/12.2.0.1/grid

 

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Patching is completed successfully. Please find the summary as follows:

 

Host:cbdps01

CRS Home:/u01/app/12.2.0.1/grid

Version:12.2.0.1.0

Summary:

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /u01/setup/RU/33583921/33610989

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_23-45-09PM_1.log

 

 

 

OPatchauto session completed at Wed Feb 26 23:51:26 2025

Time taken to complete the session 6 minutes, 42 seconds

[root@cbdps01 RU]# /u01/app/12.2.0.1/grid/OPatch/opatchauto apply /u01/setup/RU/33583921/26839277 -oh /u01/app/12.2.0.1/grid/

 

OPatchauto session is initiated at Wed Feb 26 23:52:51 2025

 

System initialization log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-26_11-52-56PM.log.

 

Session log file is /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2025-02-26_11-53-02PM.log

The id for this session is ZVC8

 

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0.1/grid

Patch applicability verified successfully on home /u01/app/12.2.0.1/grid

 

 

Executing patch validation checks on home /u01/app/12.2.0.1/grid

Patch validation checks successfully completed on home /u01/app/12.2.0.1/grid

 

 

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/12.2.0.1/grid

Prepatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_11-53-13PM.log

CRS service brought down successfully on home /u01/app/12.2.0.1/grid

 

 

Start applying binary patch on home /u01/app/12.2.0.1/grid

Binary patch applied successfully on home /u01/app/12.2.0.1/grid

 

 

Running rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

Successfully executed rootadd_rdbms.sh on home /u01/app/12.2.0.1/grid

 

 

Performing postpatch operations on CRS - starting CRS service on home /u01/app/12.2.0.1/grid

Postpatch operation log file location: /u01/app/grid/crsdata/cbdps01/crsconfig/crspatch_cbdps01_2025-02-26_11-56-12PM.log

CRS service started successfully on home /u01/app/12.2.0.1/grid

 

OPatchAuto successful.

 

--------------------------------Summary--------------------------------

 

Patching is completed successfully. Please find the summary as follows:

 

Host:cbdps01

CRS Home:/u01/app/12.2.0.1/grid

Version:12.2.0.1.0

Summary:

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /u01/setup/RU/33583921/26839277

Log: /u01/app/12.2.0.1/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-26_23-55-58PM_1.log

 

 

 

OPatchauto session completed at Thu Feb 27 00:01:37 2025

Time taken to complete the session 8 minutes, 46 seconds

 

[grid@cbdps01 ~]$ opatch lspatches

26839277;DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277)

33610989;TOMCAT RELEASE UPDATE 12.2.0.1.0(ID:RELEASE) (33610989)

33678030;OCW JAN 2022 RELEASE UPDATE 12.2.0.1.220118 (33678030)

33587128;Database Jan 2022 Release Update : 12.2.0.1.220118 (33587128)

33116894;ACFS JUL 2021 RELEASE UPDATE 12.2.0.1.210720 (33116894)

 

OPatch succeeded.  

 

说明：如上所示，补丁成功安装.                                

 

参考文档：Doc ID 27554103.8