本次搭建Oracle19C的数据库是以该平台：VMware虚拟机、RHEL7.4、19.3.0.0 、PSU 30557433的双节点的RAC。当第一个节点的GI、DB顺利打完补丁后，我就照着节点1的步骤对节点2进行打补丁。在进行节点2的DB补丁安装时，在快完成的时候报错了，从报错信息可以很容易的看出是/u01/app/oraInventory/ContentsXML/oui-patch.xml这个文件的权限导致的。
查看该文件的权限：
# ls -lrt /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-r--r-- 1 grid oinstall 174 Mar 6 12:50 /u01/app/oraInventory/ContentsXML/oui-patch.xml
对于oracle用户只有读的权限，所以导致IO异常，从opatch的日志也可以看出：
[OPSR-TIME] Finished modifying the system for patch 30557433
ApplySession adding interim patch '30557433' to inventory
[OPSR-TIME] Saving patch 30557433 to inventory
OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: **java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)**'
原厂建议需进行手动修改该权限，改成660或者666：
# chmod 660 oui-patch.xm
手动改成666之后进行重跑opatch：
# ls -lrt
total 16
-rw-rw---- 1 grid oinstall 549 Mar 6 10:51 inventory.xml
-rw-rw-rw- 1 grid oinstall 174 Mar 6 13:03 oui-patch.xml
-rw-rw---- 1 grid oinstall 300 Mar 6 13:03 comps.xml
-rw-rw---- 1 grid oinstall 292 Mar 6 13:03 libs.xml
# /u01/app/oracle/product/19c/db/OPatch/opatch apply -oh /u01/app/oracle/product/19c/db -local /u01/software/30557433
然而，这么操作并不行，从opatch log可以看到ORACLE_HOME/inventory/oneoffs/目录下一些文件不存在。
opatch-external.jar is in /u01/app/oracle/product/19c/db/OPatch/jlib/opatch-external.jar
Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/30557433 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19c/db/inventory/oneoffs/30557433/etc/config/actions" or "/u01/app/oracle/product/19c/db/inventory/oneoffs/30557433/etc/config/inventory" does not exist.
OUI-67073:UtilSession failed: NApply was not able to get the list of patches to apply.
Finishing UtilSession at Fri Mar 06 13:00:15 CST 2020
进了 ORACLE_HOME/inventory/oneoffs/目录进行查看，发现该目录下没有30557433：
#cd $ORACLE_HOME
# cd inventory/oneoffs/
# ls
29517242 29585399
重新对DB执行打补丁命令失败了，所以，个人认为对oui-patch.xm文件权限的修改一定要及时。既然不能重新打补丁，只能选择回滚了。
# /u01/app/oracle/product/19c/db/OPatch/opatch rollback -id 30557433
然而这次回滚也失败了：
The details are:
Exception occured : Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/30557433 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19c/db/inventory/oneoffs/30557433/etc/config/actions" or "/u01/app/oracle/product/19c/db/inventory/oneoffs/30557433/etc/config/inventory" does not exist.
也是同样的问题：ORACLE_HOME/inventory/oneoffs/目录下没有30557433相关的文件。于是去节点1查看，日志中的actions跟inventory这两个文件记录了啥内容。actions文件记录了patch过程中的所有动作，包括打包、备份、删除和拷贝相关文件，而inventory则记录了相关bug描述、操作系统平台等信息。
不能重新patch也不能rollback。于是突发奇想，既然rollback需要使用ORACLE_HOME/inventory/oneoffs/30557433目录下相关的文件，那就把节点1的文件拷过来，然后再进行rollback，最后在再重新打补丁。
# su - oracle
# scp -r node1:$ORACLE_HOME/inventory/oneoffs/30557433 $ORACLE_HOME/inventory/oneoffs/
# /u01/app/oracle/product/19c/db/OPatch/opatch rollback -id 30557433
[INFO] Patches will be rolled back in the following order:
30557433
.
.
.
[INFO] [OPSR-TIME] Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.
[INFO] UtilSession: N-Rollback done.
从log上看确实删掉了好多30557433的相关内容。rollback结束后，先查看下目前的补丁情况：
# /u01/app/oracle/product/19c/db/OPatch/opatch lsinv
# /u01/app/oracle/product/19c/db/OPatch/opatch version
查看补丁情况失败，再看看opatch版本，发现opatch被玩坏了。于是把节点2的整个Opatch文件夹删掉，然后从节点1拷贝新的OPatch过来。拷贝过来后先查看opatch版本和目前的补丁情况。
# rm -rf $ORACLE_HOME/OPatch/
# scp -r node1:/u01/app/oracle/product/19c/db/OPatch /u01/app/oracle/product/19c/db/
# /u01/app/oracle/product/19c/db/OPatch/opatch version
# /u01/app/oracle/product/19c/db/OPatch/opatch lsinv
.
.
.
Patch description: "Database Release Update : 19.3.0.0.190416 (29517242)"
Created on 17 Apr 2019, 23:27:10 hrs PST8PDT
目前补丁是更新到19.3.0.0.190416 。opatch可以正常使用了，那就可以继续打补丁了。
# ls -lrt
total 16
-rw-rw---- 1 grid oinstall 549 Mar 6 10:51 inventory.xml
-rw-rw-rw- 1 grid oinstall 174 Mar 6 13:03 oui-patch.xml
-rw-rw---- 1 grid oinstall 300 Mar 6 13:03 comps.xml
-rw-rw---- 1 grid oinstall 292 Mar 6 13:03 libs.xml
# /u01/app/oracle/product/19c/db/OPatch/opatch apply -oh /u01/app/oracle/product/19c/db -local /u01/software/30557433
# /u01/app/oracle/product/19c/db/OPatch/opatch lsinv
执行opatch apply之前先查看下oui-patch.xml 这个文件的权限，别再采坑了。重新opatch apply可以正常进行，结束后再查看下补丁情况，发现补丁已经更新到19.6.0.0.200114 (30557433)。之后就可以继续进行打完补丁的后续步骤了。
以后再对节点2的DB进行PSU升级时，一定要先查看oui-patch.xml 这个文件的权限。想查看下该文件是啥时候创建的，无奈xfs文件系统不支持。把查看文件时间属性命令也贴上，方便以后查看。
# cd /u01/app/oraInventory/ContentsXML/
# stat oui-patch.xml
File: ‘oui-patch.xml’
Size: 174 Blocks: 8 IO Block: 4096 regular file
Device: fd0ah/64778d Inode: 134550981 Links: 1
Access: (0660/-rw-rw----) Uid: ( 1000/ grid) Gid: ( 1001/oinstall)
Access: 2020-03-06 14:33:33.375372450 +0800
Modify: 2020-03-06 14:25:38.877693174 +0800
Change: 2020-03-06 14:25:38.877693174 +0800
Birth: -
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv 10G 69M 10G 1% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 6.0G 893M 5.2G 15% /dev/shm
tmpfs 7.8G 17M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/mapper/rootvg-usrlv 5.0G 1.5G 3.6G 29% /usr
/dev/sda2 197M 115M 82M 59% /boot
/dev/sda1 200M 9.8M 191M 5% /boot/efi
/dev/mapper/rootvg-optlv 2.0G 36M 2.0G 2% /opt
/dev/mapper/rootvg-tmplv 5.0G 113M 4.9G 3% /tmp
/dev/mapper/rootvg-homelv 5.0G 33M 5.0G 1% /home
/dev/mapper/rootvg-varlv 5.0G 183M 4.9G 4% /var
/dev/mapper/appvg-oralv 99G 27G 73G 27% /u01
tmpfs 1.6G 0 1.6G 0% /run/user/0
tmpfs 1.6G 0 1.6G 0% /run/user/1000
# debugfs -h
debugfs 1.42.9 (28-Dec-2013)
debugfs: invalid option -- 'h'
debugfs: Usage: debugfs [-b blocksize] [-s superblock] [-f cmd_file] [-R request] [-V] [[-w] [-c] device]
# debugfs -R 'stat 134551068' /dev/mapper/appvg-oralv
debugfs 1.42.9 (28-Dec-2013)
/dev/mapper/appvg-oralv: Bad magic number in super-block while opening filesystem
stat: Filesystem not open
# statx oui-patch.xml
-bash: statx: command not found
