Oracle RAC Release Update 19.24 on RHEL8.10 Issues

 

Whille applying the RU on base 19.3 Oracle RAC version on RHEL 8.10 we faced the below issue

Error#1
——–

[root@oraclehost bin]# opatchauto apply /u01/soft/36582629 -oh /u01/app/19.0.0/grid

PATCHAUTO-72030: Cannot execute in rolling mode, as CRS home is shared.
OPATCHAUTO-72030: Execute in non-rolling mode.
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.AbstractTopologyBuilder.buildTopology(AbstractTopologyBuilder.java:130)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.CRSTopologyBuilder.buildTopology(CRSTopologyBuilder.java:117)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport.loadTopology(DBBaseProductSupport.java:211)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBProductSupport.loadTopology(DBProductSupport.java:69)
at com.oracle.glcm.patch.auto.OPatchAuto.loadTopology(OPatchAuto.java:1735)
at com.oracle.glcm.patch.auto.OPatchAuto.prepareOrchestration(OPatchAuto.java:733)
at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:397)
at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:344)
at com.oracle.glcm.patch.auto.OPatchAuto.main(OPatchAuto.java:212)
2024-12-16 15:19:37,867 INFO [1] com.oracle.cie.common.util.reporting.CommonReporter – Reporting console output : Message{id=’null’, message=’OPATCHAUTO-72030: Execution mode invalid.
OPATCHAUTO-72030: Cannot execute in rolling mode, as CRS home is shared.
OPATCHAUTO-72030: Execute in non-rolling mode.’}
2024-12-16 15:19:37,867 INFO [1] com.oracle.cie.common.util.reporting.CommonReporter – Reporting console output : Message{id=’null’, message=’OPatchAuto failed.’}

For the above issue we implemented the below solution

Solution
———-
export CV_ASSUME_DISTID=OEL7.8
export ORACLE_SRVM_REMOTECOPY=/u01/tmp/scp
[root@oraclehost bin]# cat /u01/tmp/scp
#!/bin/sh

/usr/bin/scp -T $*

Re-run the Patch

After applying the solution the above error was resolved however we started encountering new error related to ACFS.

Error#2
———–
End Command output
2024-12-16 16:12:16: acfs is supported
2024-12-16 16:12:16: Stopping ora.drivers.acfs if it exists, so that it doesn’t race.
2024-12-16 16:12:16: isACFSSupported: 1
2024-12-16 16:12:16: Executing cmd: /u01/app/19.0.0/grid/bin/crsctl stat res ora.drivers.acfs -init
2024-12-16 16:12:16: Command output:
> CRS-4639: Could not contact Oracle High Availability Services
> CRS-4000: Command Status failed, or completed with errors.
>End Command output
2024-12-16 16:12:16: ODMDIR=
2024-12-16 16:12:16: Executing ‘/u01/app/19.0.0/grid/bin/acfsroot install -t2’
2024-12-16 16:12:16: Executing cmd: /u01/app/19.0.0/grid/bin/acfsroot install -t2
2024-12-16 16:12:38: Command output:
> ACFS-9176: Entering ‘ga admin name’
> ACFS-9176: Entering ‘va admin group’
> ACFS-9178: Return code = 0
> ACFS-9177: Return from ‘va admin group’
> ACFS-9178: Return code = asmadmin
> ACFS-9177: Return from ‘ga admin name’
> ACFS-9176: Entering ‘uncompress’
> ACFS-9578: Directory /u01/app/19.0.0/grid/usm/install/Oracle/EL8/x86_64/4.18.0-513.5.1/4.18.0-513.5.1-x86_64/bin has write and read permissions.
> Current ‘oracleacfs.ko’ matches ‘oracleacfs.zip’
> Current ‘oracleadvm.ko’ matches ‘oracleadvm.zip’
> Current ‘oracleafd.ko’ matches ‘oracleafd.zip’
> Current ‘oracleoks.ko’ matches ‘oracleoks.zip’
> ACFS-9177: Return from ‘uncompress’
> ACFS-9176: Entering ‘kabi compatible’
> ACFS-9176: Entering ‘lib_set_configuration_rpm’
> ACFS-9177: Return from ‘lib_set_configuration_rpm’
> Running fast method
> ACFS-9176: Entering ‘kabi compatible fast’
> ACFS-9176: Entering ‘get symbols’
> ACFS-9177: Return from ‘get symbols’
> ACFS-9176: Entering ‘is rpm compatible’
> ACFS-9176: Entering ‘get symbols’
> ACFS-9177: Return from ‘get symbols’
> ACFS-9176: Entering ‘symbols check’
> ACFS-9177: Return from ‘symbols check’
> ACFS-9463: The /u01/app/19.0.0/grid/usm/install/Oracle/EL8/x86_64/4.18.0-513.5.1/4.18.0-513.5.1-x86_64/bin/oracleacfs-4.18.0-513.5.1.el8_9.x86_64.rpm RPM is compatible with kernel kernel-core-4.18.0-553.27.1.el8_10.x86_64.
> ACFS-9177: Return from ‘is rpm compatible’
> ACFS-9177: Return from ‘kabi compatible fast = <yes>, rc=1’
> ACFS-9577: Kabi Method: RPM
> ACFS-9177: Return from ‘kabi compatible rc=1’
> Setting install_files_loc to default for main
> ACFS-9180: Sub-command is ‘install’
> ACFS-9505: Using acfsutil executable from location: ‘/u01/app/19.0.0/grid/usm/install/cmds/bin/acfsutil’
> ACFS-9176: Entering ‘install’
> ACFS-9300: ADVM/ACFS distribution files found.
> ACFS-9176: Entering ‘uld usm drvs’
> ACFS-9178: Return code = USM_SUCCESS
> ACFS-9177: Return from ‘uld usm drvs’
> ACFS-9176: Entering ‘fail mode’
> ACFS-9178: Return code = 0
> ACFS-9177: Return from ‘fail mode’
> ACFS-9176: Entering ‘uninstall’
> ACFS-9176: Entering ‘uld usm drvs’
> ACFS-9178: Return code = USM_SUCCESS
> ACFS-9177: Return from ‘uld usm drvs’
> ACFS-9314: Removing previous ADVM/ACFS installation.
> ACFS-9176: Entering ‘getparm’
> ACFS-9178: Return code = soa-stg
> ACFS-9177: Return from ‘getparm’
> ACFS-9176: Entering ‘depmod uninstall’
> ACFS-9177: Return from ‘depmod uninstall’
> ACFS-9315: Previous ADVM/ACFS components successfully removed.
> ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
> ACFS-9178: Return code = USM_SUCCESS
> ACFS-9177: Return from ‘uninstall’
> ACFS-9307: Installing requested ADVM/ACFS software.

Depmod output ”
> ACFS-9177: Return from ‘mpr’
> ACFS-9109: oracleoks.ko driver failed to load.
> ACFS-9178: Return code = USM_FAIL
> ACFS-9177: Return from ‘ld usm drvs’
> ACFS-9428: Failed to load ADVM/ACFS drivers. A system reboot is recommended.
> ACFS-9310: ADVM/ACFS installation failed.
> ACFS-9178: Return code = USM_REBOOT_RECOMMENDED
> ACFS-9177: Return from ‘install’
> ACFS-9176: Entering ‘acroot ex’
> ACFS-9178: Return code = 3
> ACFS-9177: Return from ‘acroot ex’
>End Command output
2024-12-16 16:12:38: /u01/app/19.0.0/grid/bin/crsctl disable crs … disabling CRS in preparation for reboot.
2024-12-16 16:12:38: Executing cmd: /u01/app/19.0.0/grid/bin/crsctl disable crs
2024-12-16 16:12:39: Command output:
> CRS-4621: Oracle High Availability Services autostart is disabled.
>End Command output
2024-12-16 16:12:39: /u01/app/19.0.0/grid/bin/crsctl disable crs … CRS disabled, ready for reboot.
2024-12-16 16:12:39: USM driver install status is 3
2024-12-16 16:12:39: ACFS drivers cannot be installed, and reboot may resolve this
2024-12-16 16:12:39: Executing cmd: /u01/app/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 400
2024-12-16 16:12:39: Executing cmd: /u01/app/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 400
2024-12-16 16:12:39: Command output:
> CLSRSC-400: A system reboot is required to continue installing.
>End Command output
2024-12-16 16:12:39: CLSRSC-400: A system reboot is required to continue installing.

2024-12-16 16:12:39: ROOTCRS_POSTPATCH_ACFSINST checkpoint has failed
2024-12-16 16:12:39: ckpt: -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_POSTPATCH_ACFSINST
2024-12-16 16:12:39: Invoking “/u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_POSTPATCH_ACFSINST”
2024-12-16 16:12:39: trace file=/u01/app/grid/crsdata/oraclehost/crsconfig/cluutil1.log
2024-12-16 16:12:39: Running as user grid: /u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_POSTPATCH_ACFSINST
2024-12-16 16:12:39: Removing file /tmp/kfTWCykH63
2024-12-16 16:12:39: Successfully removed file: /tmp/kfTWCykH63
2024-12-16 16:12:39: pipe exit code: 0
2024-12-16 16:12:39: /bin/su successfully executed

2024-12-16 16:12:39: TRUE

2024-12-16 16:12:39: Invoking “/u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_POSTPATCH_ACFSINST -status”
2024-12-16 16:12:39: trace file=/u01/app/grid/crsdata/oraclehost/crsconfig/cluutil2.log
2024-12-16 16:12:39: Running as user grid: /u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_POSTPATCH_ACFSINST -status
2024-12-16 16:12:39: Removing file /tmp/K97le8jyuW
2024-12-16 16:12:39: Successfully removed file: /tmp/K97le8jyuW
2024-12-16 16:12:39: pipe exit code: 0
2024-12-16 16:12:39: /bin/su successfully executed

2024-12-16 16:12:39: FAIL

2024-12-16 16:12:39: The ‘ROOTCRS_POSTPATCH_ACFSINST’ status is FAILED
2024-12-16 16:12:39: ROOTCRS_POSTPATCH_ACFSINST state is FAIL
2024-12-16 16:12:39: Invoking “/u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH_ACFSINST -state FAIL”
2024-12-16 16:12:39: trace file=/u01/app/grid/crsdata/oraclehost/crsconfig/cluutil3.log
2024-12-16 16:12:39: Running as user grid: /u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_POSTPATCH_ACFSINST -state FAIL
2024-12-16 16:12:39: Removing file /tmp/RsoNxf1PJx
2024-12-16 16:12:39: Successfully removed file: /tmp/RsoNxf1PJx
2024-12-16 16:12:39: pipe exit code: 0
2024-12-16 16:12:39: /bin/su successfully executed

2024-12-16 16:12:39:
2024-12-16 16:12:39: Succeeded in writing the checkpoint:’ROOTCRS_POSTPATCH_ACFSINST’ with status:FAIL
2024-12-16 16:12:39: Invoking “/u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_STACK -state FAIL”
2024-12-16 16:12:39: trace file=/u01/app/grid/crsdata/oraclehost/crsconfig/cluutil4.log
2024-12-16 16:12:39: Running as user grid: /u01/app/19.0.0/grid/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_STACK -state FAIL
2024-12-16 16:12:40: Removing file /tmp/5hwqQ6FuTZ
2024-12-16 16:12:40: Successfully removed file: /tmp/5hwqQ6FuTZ
2024-12-16 16:12:40: pipe exit code: 0
2024-12-16 16:12:40: /bin/su successfully executed

2024-12-16 16:12:40:
2024-12-16 16:12:40: Succeeded in writing the checkpoint:’ROOTCRS_STACK’ with status:FAIL

We implemented the below solution to resolve the ACFS issue

Solution
==========
Disabling Secure Boot

Check Secure Boot is enabled or disabled (Doc ID 2528738.1)
mokutil –sb-state
Enabled

If Enabled and using Virtualization follow the below steps

https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-898217D4-689D-4EB5-866C-888353FE241C.html

1. Browse to the virtual machine in the vSphere Client inventory.
2. Right-click the virtual machine and select Edit Settings.
3. Click the VM Options tab, and expand Boot Options.
4. Under Boot Options, ensure that firmware is set to EFI.
5. Select your task.
6. Select the Secure Boot check box to activate secure boot.
7. Deselect the Secure Boot check box to deactivate secure boot.
8. Click OK.

Re-run the patch using below command

opatchauto resume

Now check if the patch is applied successfully

NEVER RUN ROLLBACK IF THE PATCH IS FAILED IN APPLY MODE AT POSTPATCH STAGE.

If the patch fails in the post patch stage apply the following solution. PLease note only follow the below when the patch is applied successfully on the GRID HOME local node.

Use the below command to check.

opatch lspatches

and

crsctl query crs activeversion -f

Solution
============
[root@oraclehost bin]# opatchauto apply /u01/soft/36582629 -oh /u01/app/19.0.0/grid

OPatchauto session is initiated at Thu Dec 19 08:54:19 2024

System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2024-12-19_08-54-30AM.log.

Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2024-12-19_08-54-34AM.log
The id for this session is UKNB

Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid
Patch applicability verified successfully on home /u01/app/19.0.0/grid

OPatchAuto successful.

——————————–Summary——————————–

Patching is completed successfully. Please find the summary as follows:

Host:oraclehost
CRS Home:/u01/app/19.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/soft/36582629/36587798
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-12-19_08-54-50AM_1.log
Reason: /u01/soft/36582629/36587798 is not required to be applied to oracle home /u01/app/19.0.0/grid

Patch: /u01/soft/36582629/36590554
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-12-19_08-54-50AM_1.log
Reason: /u01/soft/36582629/36590554 is not required to be applied to oracle home /u01/app/19.0.0/grid

Patch: /u01/soft/36582629/36758186
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-12-19_08-54-50AM_1.log
Reason: /u01/soft/36582629/36758186 is not required to be applied to oracle home /u01/app/19.0.0/grid

Patch: /u01/soft/36582629/36648174
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-12-19_08-54-50AM_1.log
Reason: /u01/soft/36582629/36648174 is not required to be applied to oracle home /u01/app/19.0.0/grid

Patch: /u01/soft/36582629/36582781
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-12-19_08-54-50AM_1.log
Reason: /u01/soft/36582629/36582781 is not required to be applied to oracle home /u01/app/19.0.0/grid

 

OPatchauto session completed at Thu Dec 19 08:57:08 2024
Time taken to complete the session 2 minutes, 38 seconds
[root@oraclehost bin]#
[root@oraclehost bin]# ./crsctl check cluster -all
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Check failed, or completed with errors.
[root@oraclehost bin]#
[root@oraclehost bin]#
[root@oraclehost bin]# ./crsctl check cluster -all
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Check failed, or completed with errors.
[root@oraclehost bin]#
[root@oraclehost bin]# su – grid
[grid@oraclehost ~]$ . grid.env
[grid@oraclehost ~]$ cd $GRID_HOME
[grid@oraclehost grid]$ cd OPatch
[grid@oraclehost OPatch]$ ./opatch lspatches
36758186;DBWLM RELEASE UPDATE 19.0.0.0.0 (36758186)
36648174;TOMCAT RELEASE UPDATE 19.0.0.0.0 (36648174)
36590554;ACFS RELEASE UPDATE 19.24.0.0.0 (36590554)
36587798;OCW RELEASE UPDATE 19.24.0.0.0 (36587798)
36582781;Database Release Update : 19.24.0.0.240716 (36582781)

OPatch succeeded.
[grid@oraclehost OPatch]$ exit
logout
[root@oraclehost bin]# pwd
/u01/app/19.0.0/grid/bin
[root@oraclehost bin]# ps -ef|grep java
root 602324 587240 0 08:59 pts/1 00:00:00 grep –color=auto java
[root@oraclehost bin]# ps -ef|grep d.bin
root 602394 587240 0 09:00 pts/1 00:00:00 grep –color=auto d.bin
[root@oraclehost bin]#
[root@oraclehost bin]#
[root@oraclehost bin]#
[root@oraclehost bin]# ./crsctl get resource use
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Get failed, or completed with errors.
[root@oraclehost bin]# ./crsctl start crs
CRS-6706: Oracle Clusterware Release patch level (‘3669234883’) does not match Software patch level (‘724960844’). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
[root@oraclehost bin]# cd ..
[root@oraclehost grid]# cd crs
[root@oraclehost crs]# cd install/
[root@oraclehost install]# ls -l rootcrs.*
-rwxr-xr-x 1 root oinstall 25041 Jul 13 23:20 rootcrs.pl
-rwxr-x— 1 root oinstall 8046 Dec 16 12:59 rootcrs.sh
[root@oraclehost install]# sh rootcrs.sh -unlock
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oraclehost/crsconfig/crsunlock_oraclehost_2024-12-19_09-02-55AM.log
2024/12/19 09:02:56 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2024/12/19 09:03:17 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2024/12/19 09:03:17 CLSRSC-347: Successfully unlock /u01/app/19.0.0/grid
[root@oraclehost install]# sh rootcrs.sh -lock
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oraclehost/crsconfig/crslock_oraclehost_2024-12-19_09-03-39AM.log
2024/12/19 09:03:43 CLSRSC-329: Replacing Clusterware entries in file ‘oracle-ohasd.service’
[root@oraclehost install]# cd ..
[root@oraclehost crs]# cd ..
[root@oraclehost grid]# cd bin/
[root@oraclehost bin]# clscfg
clscfg clscfg.bin
[root@oraclehost bin]# clscfg -localpatch
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
[root@oraclehost bin]# ps -ef|grep d.bin
root 607318 587240 0 09:05 pts/1 00:00:00 grep –color=auto d.bin
[root@oraclehost bin]# ./crsctl query crs activeversion -f
[root@oraclehost bin]# ./crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [19.0.0.0.0][root@oraclehost bin]# clscfg -patch
clscfg: -patch mode specified
Unable to open cssctx for node information 3.
[root@oraclehost bin]# cd ..
[root@oraclehost grid]# cd crs/install/
[root@oraclehost install]# ./rootcrs.sh -unlock
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oraclehost/crsconfig/crsunlock_oraclehost_2024-12-19_09-13-21AM.log
2024/12/19 09:13:21 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2024/12/19 09:13:39 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2024/12/19 09:13:39 CLSRSC-347: Successfully unlock /u01/app/19.0.0/grid
[root@oraclehost install]# cd ..
[root@oraclehost crs]# cd ..
[root@oraclehost grid]# cd bin
[root@oraclehost bin]# clscfg -localpatch
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
[root@oraclehost bin]# clscfg -patch
clscfg: -patch mode specified
Unable to open cssctx for node information 3.
[root@oraclehost bin]# cd ..
[root@oraclehost grid]# cd crs/install/
[root@oraclehost install]# ./rootcrs.sh -lock
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oraclehost/crsconfig/crslock_oraclehost_2024-12-19_09-15-01AM.log
2024/12/19 09:15:05 CLSRSC-329: Replacing Clusterware entries in file ‘oracle-ohasd.service’
[root@oraclehost install]#
[root@oraclehost install]# cd ..
[root@oraclehost crs]# cd ..
[root@oraclehost grid]# cd bin/
[root@oraclehost bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

 

[root@oraclehost bin]# ./crsctl query crs releasepatch
Oracle Clusterware release patch level is [3669234883] and the complete list of patches [36582781 36587798 36590554 36648174 36758186 ] have been applied on the local node. The release patch string is [19.24.0.0.0].