Introduction
Out-of-place patching builds a brand new Oracle home that already contains the Release Update, then moves the cluster or the database over to it. The old home stays untouched on disk, so the running environment is never modified and going back is a matter of pointing at the old path again. This guide patches a two-node Oracle 19c RAC from the base 19.3 release to 19.33, covering both the Grid Infrastructure home and the Database home, with each node switched one at a time so the database stays available on the other node.
The sequence is: capture a baseline, install the new Grid home with the RU applied, switch Grid on Node 1, install the new Database home, switch the database on Node 1, switch Grid on Node 2, run datapatch once, and verify.
Prerequisites
Check the following before starting.
- A working two-node Oracle 19c RAC at the base 19.3 release, with the Grid home at
/u01/app/grid/19c/grid_1and the database home at/u01/app/oracle/19c/db_1. - The
oracleOS user owning both homes, plus root access on both nodes. - Passwordless SSH for
oraclebetween the nodes, which the installer uses to copy a new home to the other node. - Free space on
/u01on both nodes for one new Grid home and one new Database home, plus a staging directory (/stagein this guide) for the downloaded files. About 15 GB of extra staging space was added on both nodes for this run. - The database open in READ WRITE mode and every cluster resource online, which Phase A confirms.
Files to Download
These five files were staged before the work started. The base images are the standard 19.3 Linux zips, the two RU zips are the 19.33 patches, and the OPatch zip is needed because the OPatch inside a fresh 19.3 home is too old to apply the RU.
| File | Purpose |
|---|---|
LINUX.X64_193000_grid_home.zip | Base 19.3 Grid Infrastructure image (2.7 GB) |
LINUX.X64_193000_db_home.zip | Base 19.3 Database image (2.9 GB) |
p39922757_190000_Linux-x86-64.zip | Grid Infrastructure Release Update 19.33 (2.7 GB) |
p39915792_190000_Linux-x86-64.zip | Database Release Update 19.33 (2.3 GB) |
p6880880_190000_Linux-x86-64.zip | Latest OPatch for 19c (129 MB) |
Download links: 19.33 Release Update (My Oracle Support), OPatch (patch 6880880), and 19c base binaries.
Scenario
The cluster runs the RACPRDB database across two nodes with three services: racprdb_svc on both instances, racprdb_batch_svc preferring Node 1, and racprdb_rman_svc preferring Node 2. Instead of patching the existing homes, a new Grid home (grid_2) and Database home (db_2) are installed alongside the old homes, with the RU applied during installation. The nodes are switched in sequence. Node 1’s instance restarts twice, once during the Grid home switch and again during the Database home switch. Node 2’s instance restarts once because its Grid home switch also moves it to the new Database home. During each switch, services can run on the surviving node, subject to service configuration and available capacity.
Environment Used in this Guide
| Cluster Name | racprd |
|---|---|
| Nodes | racprd1.oraeasy.com, racprd2.oraeasy.com |
| Database / Instances | RACPRDB (racprdb1 on Node 1, racprdb2 on Node 2) |
| Services | racprdb_svc, racprdb_batch_svc, racprdb_rman_svc |
| Source Release | 19.3 (Database RU 19.3.0.0.190416) |
| Target Release | 19.33 (Database RU 19.33.0.0.261020) |
| Old Grid Home | /u01/app/grid/19c/grid_1 |
| New Grid Home | /u01/app/grid/19c/grid_2 |
| Old Database Home | /u01/app/oracle/19c/db_1 |
| New Database Home | /u01/app/oracle/19c/db_2 |
| Grid RU Patch | 39922757 |
| Database RU Patch | 39915792 |
| OPatch Version Used | 12.2.0.1.54 |
| Grid/Database OS User | oracle |
| Staging Directory | /stage |
Summary of Activities
At the top level, the patching is done in the phases below.
- Pre-Patch Checks.
- New Grid Home Installation with RU.
- Grid Home Switch on Node 1.
- New Database Home Installation with RU.
- Database Home Switch on Node 1.
- Grid Home Switch on Node 2.
- Datapatch.
- Node Reboots.
- Post-Patch Verification.
Now let's proceed with the patching step by step.
Pre-Patch Checks
Before touching any binaries, record the state of the database, the listeners, the clusterware, and both patch inventories. This baseline proves the cluster is healthy enough to patch, and it is what the final verification is compared against.
Step 1: Check Database Status and Version
The first block confirms the role and open mode, the running release, the SQL patch history, the component registry, and the count of invalid objects, then the service placement from srvctl:
[oracle@racprd1 ~]$ . .db_env
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ env|grep ORA
ORACLE_SID=racprdb1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/19c/db_1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sat Oct 23 08:03:49 2026
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> set lines 200 pages 1000
SQL> col open_mode for a15
SQL> col HOST_NAME for a15
SQL> select distinct NAME,DATABASE_ROLE,OPEN_MODE,INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,VERSION,STARTUP_TIME,STATUS from gv$database,gv$instance;
NAME DATABASE_ROLE OPEN_MODE INSTANCE_NUMBER INSTANCE_NAME HOST_NAME VERSION STARTUP_T STATUS
--------- ---------------- ----------- --------------- -------------- ------------------- ----------- --------- -------
RACPRDB PRIMARY READ WRITE 1 racprdb1 racprd1.oraeasy.com 19.0.0.0.0 23-OCT-26 OPEN
RACPRDB PRIMARY READ WRITE 2 racprdb2 racprd2.oraeasy.com 19.0.0.0.0 23-OCT-26 OPEN
SQL> SELECT banner_full FROM v$version;
BANNER_FULL
--------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> col ACTION_TIME for a20
SQL> col ACTION for a10
SQL> col STATUS for a10
SQL> select patch_uid,patch_id,SOURCE_VERSION, TARGET_VERSION, action, status,action_time,description from dba_registry_sqlpatch;
PATCH_UID PATCH_ID SOURCE_VERSION TARGET_VERSION ACTION STATUS ACTION_TIME DESCRIPTION
---------- ---------- --------------- --------------- ---------- ---------- -------------------- ------------------------------------------------------
22862832 29517242 19.1.0.0.0 19.3.0.0.0 APPLY SUCCESS 27-SEP-26 10.11.51.9 Database Release Update : 19.3.0.0.190416 (29517242)
51770 PM
SQL> col comp_id for a10
SQL> col version for a11
SQL> col status for a10
SQL> col comp_name for a37
SQL> select comp_id,comp_name,version,status from dba_registry;
COMP_ID COMP_NAME VERSION STATUS
---------- ------------------------------------- ----------- ----------
CATALOG Oracle Database Catalog Views 19.0.0.0.0 VALID
CATPROC Oracle Database Packages and Types 19.0.0.0.0 VALID
RAC Oracle Real Application Clusters 19.0.0.0.0 VALID
JAVAVM JServer JAVA Virtual Machine 19.0.0.0.0 VALID
XML Oracle XDK 19.0.0.0.0 VALID
CATJAVA Oracle Database Java Packages 19.0.0.0.0 VALID
APS OLAP Analytic Workspace 19.0.0.0.0 VALID
XDB Oracle XML Database 19.0.0.0.0 VALID
OWM Oracle Workspace Manager 19.0.0.0.0 VALID
CONTEXT Oracle Text 19.0.0.0.0 VALID
ORDIM Oracle Multimedia 19.0.0.0.0 VALID
SDO Spatial 19.0.0.0.0 VALID
XOQ Oracle OLAP API 19.0.0.0.0 VALID
OLS Oracle Label Security 19.0.0.0.0 VALID
DV Oracle Database Vault 19.0.0.0.0 VALID
15 rows selected.
SQL> SELECT COUNT(*) FROM dba_objects WHERE status = 'INVALID';
COUNT(*)
----------
0
SQL> exit
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
Both instances are open on 19.3.0.0.0, dba_registry_sqlpatch holds only the base RU 29517242, all 15 registry components are VALID, and there are zero invalid objects. That last number matters: if invalid objects appear after patching, a clean baseline shows they came from the patch and not from existing code.
Step 2: Check Listener and SCAN Status
[oracle@racprd1 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node racprd2
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node racprd1
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node racprd1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): racprd2,racprd1
[oracle@racprd1 ~]$
The three SCAN VIPs are spread over both nodes (one on racprd2, two on racprd1) and the local listener runs on both. Note where each SCAN lives, because the SCAN VIPs move while a node's stack is restarted and are checked again at the end.
Step 3: Check Cluster Resources
The cluster check is run as root from the old Grid home, since that is still the active home at this stage:
[root@racprd1 ~]# cd /u01/app/grid/19c/grid_1/bin
[root@racprd1 bin]#
[root@racprd1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@racprd1 bin]#
[root@racprd1 bin]# ./crsctl check has
CRS-4638: Oracle High Availability Services is online
[root@racprd1 bin]#
[root@racprd1 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.chad
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.net1.network
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.ons
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.proxy_advm
OFFLINE OFFLINE racprd1 STABLE
OFFLINE OFFLINE racprd2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 ONLINE OFFLINE STABLE
ora.CRS.dg(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.FRA.dg(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE racprd2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE racprd1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE racprd1 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE racprd1 Started,STABLE
2 ONLINE ONLINE racprd2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE racprd1 STABLE
ora.qosmserver
1 ONLINE ONLINE racprd1 STABLE
ora.racprd1.vip
1 ONLINE ONLINE racprd1 STABLE
ora.racprd2.vip
1 ONLINE ONLINE racprd2 STABLE
ora.racprdb.db
1 ONLINE ONLINE racprd1 Open,HOME=/u01/app/o
racle/19c/db_1,STABL
E
2 ONLINE ONLINE racprd2 Open,HOME=/u01/app/o
racle/19c/db_1,STABL
E
ora.racprdb.racprdb_batch_svc.svc
2 ONLINE ONLINE racprd1 STABLE
ora.racprdb.racprdb_rman_svc.svc
1 ONLINE ONLINE racprd2 STABLE
ora.racprdb.racprdb_svc.svc
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
ora.scan1.vip
1 ONLINE ONLINE racprd2 STABLE
ora.scan2.vip
1 ONLINE ONLINE racprd1 STABLE
ora.scan3.vip
1 ONLINE ONLINE racprd1 STABLE
--------------------------------------------------------------------------------
[root@racprd1 bin]#
Every resource is online. ora.proxy_advm is offline because no ADVM is configured, and the third offline slot on the ASM resources is the Flex ASM default on a two-node cluster. Also note that ora.racprdb.db reports HOME=/u01/app/oracle/19c/db_1 on both nodes. This detail is compared with the final check.
Step 4: Check the Existing Patch Inventory
opatch lspatches and opatch lsinventory are run in both homes on both nodes, so there is a record of exactly what each home contains before the switch:
Node 1:
[oracle@racprd1 ~]$ env|grep ORA
ORACLE_SID=racprdb1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/19c/db_1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ cd /u01/app/oracle/19c/db_1/OPatch/
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
OPatch succeeded.
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 29585399 : applied on Thu Apr 18 12:51:33 IST 2019
Patch 29517242 : applied on Thu Apr 18 12:51:17 IST 2019
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ cd
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ . .grid_env
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ env|grep ORA
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/grid
ORACLE_HOME=/u01/app/grid/19c/grid_1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ cd /u01/app/grid/19c/grid_1/OPatch/
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)
OPatch succeeded.
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 29585399 : applied on Thu Apr 18 13:06:24 IST 2019
Patch 29517247 : applied on Thu Apr 18 13:06:02 IST 2019
Patch 29517242 : applied on Thu Apr 18 13:05:39 IST 2019
Patch 29401763 : applied on Thu Apr 18 13:05:16 IST 2019
[oracle@racprd1 OPatch]$
Node 2:
[oracle@racprd2 ~]$ . .db_env
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ env|grep ORA
ORACLE_SID=racprdb2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/19c/db_1
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ cd /u01/app/oracle/19c/db_1/OPatch/
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
OPatch succeeded.
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 29585399 : applied on Thu Apr 18 12:51:33 IST 2019
Patch 29517242 : applied on Thu Apr 18 12:51:17 IST 2019
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ cd
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ . .grid_env
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ env|grep ORA
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/grid
ORACLE_HOME=/u01/app/grid/19c/grid_1
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ cd /u01/app/grid/19c/grid_1/OPatch/
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)
OPatch succeeded.
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 29585399 : applied on Thu Apr 18 13:06:24 IST 2019
Patch 29517247 : applied on Thu Apr 18 13:06:02 IST 2019
Patch 29517242 : applied on Thu Apr 18 13:05:39 IST 2019
Patch 29401763 : applied on Thu Apr 18 13:05:16 IST 2019
[oracle@racprd2 OPatch]$
Both nodes match. The database home holds two base patches and the Grid home holds four (OCW, ACFS, Database RU, and Tomcat), all from April 2019. Matching inventories across nodes is what makes a node-by-node switch safe, because any difference would have to be understood first.
Pre-patch checks are complete, so let's install the new Grid home.
New Grid Home Installation with RU
The new Grid home is a software-only install. The base image is unzipped into grid_2, OPatch is upgraded inside it, and gridSetup.sh applies the RU while installing, so the home is already at 19.33 before the cluster ever uses it. The running cluster is not touched in this phase.
Step 1: Create the New Grid Home Directory
Node 1:
[root@racprd1 ~]# mkdir -p /u01/app/grid/19c/grid_2
[root@racprd1 ~]# chown -R oracle:oinstall /u01/app/grid/19c/grid_2
[root@racprd1 ~]# chmod -R 775 /u01/app/grid/19c/grid_2
Node 2:
[root@racprd2 ~]# mkdir -p /u01/app/grid/19c/grid_2
[root@racprd2 ~]# chown -R oracle:oinstall /u01/app/grid/19c/grid_2
[root@racprd2 ~]# chmod -R 775 /u01/app/grid/19c/grid_2
The directory is created on both nodes, owned by oracle:oinstall. Creating it on Node 2 as well gives the installer an empty, writable target when it copies the home across.
Step 2: Unzip the Software and the RU
[oracle@racprd1 grid_home]$ pwd
/stage/grid_home
[oracle@racprd1 grid_home]$ ls -lrth
total 5.4G
-rw-r--r--. 1 oracle oinstall 2.7G Oct 22 21:08 LINUX.X64_193000_grid_home.zip
-rw-r--r--. 1 oracle oinstall 2.7G Oct 22 21:17 p39922757_190000_Linux-x86-64.zip
[oracle@racprd1 grid_home]$ unzip LINUX.X64_193000_grid_home.zip -d /u01/app/grid/19c/grid_2
[oracle@racprd1 grid_home]$ pwd
/stage/grid_home
[oracle@racprd1 grid_home]$ ls -lrth
total 5.4G
-rw-r--r--. 1 oracle oinstall 2.7G Oct 22 21:08 LINUX.X64_193000_grid_home.zip
-rw-r--r--. 1 oracle oinstall 2.7G Oct 22 21:17 p39922757_190000_Linux-x86-64.zip
[oracle@racprd1 grid_home]$ unzip p39922757_190000_Linux-x86-64.zip
[oracle@racprd1 grid_home]$ ls -lrth
total 5.4G
drwxr-x---. 8 oracle oinstall 4.0K Oct 16 09:24 39922757
-rw-rw-r--. 1 oracle oinstall 3.3M Oct 20 19:42 PatchSearch.xml
-rw-r--r--. 1 oracle oinstall 2.7G Oct 22 21:08 LINUX.X64_193000_grid_home.zip
-rw-r--r--. 1 oracle oinstall 2.7G Oct 22 21:17 p39922757_190000_Linux-x86-64.zip
[oracle@racprd1 grid_home]$
The base image is extracted directly into the new home, while the RU is extracted into /stage and stays outside the home. The extra PatchSearch.xml in the listing comes from the patch zip and is not used. Both steps run on Node 1 only, because the installer copies the home to Node 2.
Step 3: Upgrade OPatch in the New Home
[oracle@racprd1 grid_home]$ cd /u01/app/grid/19c/grid_2
[oracle@racprd1 grid_2]$ cd OPatch/
[oracle@racprd1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.17
OPatch succeeded.
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ cd ..
[oracle@racprd1 grid_2]$
[oracle@racprd1 grid_2]$ mv OPatch/ OPatch_bkp
[oracle@racprd1 grid_2]$ cd /stage/opatch/
[oracle@racprd1 opatch]$ ls -lrth
total 129M
-rw-r--r--. 1 oracle oinstall 129M Oct 22 21:18 p6880880_190000_Linux-x86-64.zip
[oracle@racprd1 opatch]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/grid/19c/grid_2/
[oracle@racprd1 opatch]$ cd /u01/app/grid/19c/grid_2/OPatch
[oracle@racprd1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.54
OPatch succeeded.
[oracle@racprd1 OPatch]$
A fresh 19.3 image ships OPatch 12.2.0.1.17, which is too old to apply this RU. The old directory is kept as OPatch_bkp, and version 12.2.0.1.54 from patch 6880880 replaces it. This must happen before -applyRU, since the installer calls OPatch internally.
Step 4: Prepare the Response File
[oracle@racprd1 OPatch]$ cd /u01/app/grid/19c/grid_2/
[oracle@racprd1 grid_2]$ cd install/response/
[oracle@racprd1 response]$ ls -lrth
total 40K
-rw-r-----. 1 oracle oinstall 1.6K May 21 2016 sample.ccf
-rw-r-----. 1 oracle oinstall 36K Jan 19 2019 gridsetup.rsp
[oracle@racprd1 response]$ mv gridsetup.rsp gridsetup.rsp_bkp
[oracle@racprd1 response]$ vi gridsetup.rsp
[oracle@racprd1 response]$ cat gridsetup.rsp
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0
INVENTORY_LOCATION=/u01/app/oraInventory
ORACLE_BASE=/u01/grid
SELECTED_LANGUAGES=en,en_GB
oracle.install.option=CRS_SWONLY
oracle.install.asm.OSDBA=asmdba
oracle.install.asm.OSOPER=asmoper
oracle.install.asm.OSASM=asmadmin
oracle.install.crs.config.ClusterConfiguration=STANDALONE
oracle.install.crs.config.configureAsExtendedCluster=false
oracle.install.crs.config.clusterName=racprd
oracle.install.crs.config.gpnp.configureGNS=false
oracle.install.crs.config.autoConfigureClusterNodeVIP=false
oracle.install.crs.config.clusterNodes=racprd1,racprd2
[oracle@racprd1 response]$
oracle.install.option=CRS_SWONLY makes this a software-only install, so no cluster configuration is attempted. ORACLE_BASE=/u01/grid matches the existing Grid base, and clusterNodes=racprd1,racprd2 tells the installer to copy the home to both nodes. No VIPs or disk groups are listed because the existing cluster is reused.
Step 5: Launch gridSetup.sh with the RU
[oracle@racprd1 response]$ cd /u01/app/grid/19c/grid_2/
[oracle@racprd1 grid_2]$ ls -lrth gridSetup.sh
-rwxr-x---. 1 oracle oinstall 3.3K Mar 8 2017 gridSetup.sh
[oracle@racprd1 grid_2]$
[oracle@racprd1 grid_2]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent -applyRU /stage/grid_home/39922757 -responseFile /u01/app/grid/19c/grid_2/install/response/gridsetup.rsp
Preparing the home to patch...
Applying the patch /stage/grid_home/39922757...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2026-10-23_08-26-25PM/installerPatchActions_2026-10-23_08-26-25PM.log
Launching Oracle Grid Infrastructure Setup Wizard...
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2026-10-23_08-26-25PM/gridSetupActions2026-10-23_08-26-25PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2026-10-23_08-26-25PM/gridSetupActions2026-10-23_08-26-25PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
/u01/app/grid/19c/grid_2/install/response/grid_2026-10-23_08-26-25PM.rsp
You can find the log of this install session at:
/u01/app/oraInventory/logs/GridSetupActions2026-10-23_08-26-25PM/gridSetupActions2026-10-23_08-26-25PM.log
As a root user, execute the following script(s):
1. /u01/app/grid/19c/grid_2/root.sh
Execute /u01/app/grid/19c/grid_2/root.sh on the following nodes:
[racprd1, racprd2]
Successfully Setup Software with warning(s).
[oracle@racprd1 grid_2]$
-applyRU patches the home first, which is why the log reports Successfully applied the patch before the setup wizard even launches. -ignorePrereq lets the run continue past the optional-requirement warning INS-13014; read the log path it prints before ignoring it, and fix the checks properly on a production system. Only root.sh is requested, and there is no -executeConfigTools step, because nothing is being configured.
Step 6: Run root.sh on Both Nodes
Node 1:
[root@racprd1 ~]# /u01/app/grid/19c/grid_2/root.sh
Check /u01/app/grid/19c/grid_2/install/root_racprd1.oraeasy.com_2026-10-23_21-04-35-672811183.log for the output of root script
[root@racprd1 ~]#
[root@racprd1 ~]# cat /u01/app/grid/19c/grid_2/install/root_racprd1.oraeasy.com_2026-10-23_21-04-35-672811183.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/grid/19c/grid_2
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
To configure Grid Infrastructure for a Cluster execute the following command as oracle user:
/u01/app/grid/19c/grid_2/gridSetup.sh
This command launches the Grid Infrastructure Setup Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.
[root@racprd1 ~]#
Node 2:
[root@racprd2 ~]# /u01/app/grid/19c/grid_2/root.sh
Check /u01/app/grid/19c/grid_2/install/root_racprd2.oraeasy.com_2026-10-23_21-04-50-201864921.log for the output of root script
[root@racprd2 ~]#
[root@racprd2 ~]# cat /u01/app/grid/19c/grid_2/install/root_racprd2.oraeasy.com_2026-10-23_21-04-50-201864921.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/grid/19c/grid_2
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
To configure Grid Infrastructure for a Cluster execute the following command as oracle user:
/u01/app/grid/19c/grid_2/gridSetup.sh
This command launches the Grid Infrastructure Setup Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.
[root@racprd2 ~]#
In a software-only install, root.sh only copies dbhome, oraenv, and coraenv to /usr/local/bin. The message suggesting gridSetup.sh to configure a cluster does not apply here, since the existing cluster is reused and this home is activated later by the switch.
Step 7: Verify the Patches in the New Grid Home
Node 1:
[oracle@racprd1 grid_2]$ cd OPatch
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lspatches
39922802;ACFS RELEASE UPDATE 19.33.0.0.0 (39922802)
39922783;OCW RELEASE UPDATE 19.33.0.0.0 (39922783)
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
39107855;TOMCAT RELEASE UPDATE 19.0.0.0.0 (39107855)
39107825;DBWLM RELEASE UPDATE 19.0.0.0.0 (39107825)
OPatch succeeded.
[oracle@racprd1 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 39922802 : applied on Fri Oct 23 20:20:46 IST 2026
Patch 39922783 : applied on Fri Oct 23 20:18:51 IST 2026
Patch 39915792 : applied on Fri Oct 23 19:57:30 IST 2026
Patch 39107855 : applied on Fri Oct 23 19:45:35 IST 2026
Patch 39107825 : applied on Fri Oct 23 19:45:11 IST 2026
[oracle@racprd1 OPatch]$
Node 2:
[oracle@racprd2 ~]$ cd /u01/app/grid/19c/grid_2/OPatch
[oracle@racprd2 grid_2]$
[oracle@racprd2 grid_2]$
[oracle@racprd2 grid_2]$ cd OPatch
[oracle@racprd2 OPatch]$ ./opatch lspatches
39922802;ACFS RELEASE UPDATE 19.33.0.0.0 (39922802)
39922783;OCW RELEASE UPDATE 19.33.0.0.0 (39922783)
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
39107855;TOMCAT RELEASE UPDATE 19.0.0.0.0 (39107855)
39107825;DBWLM RELEASE UPDATE 19.0.0.0.0 (39107825)
OPatch succeeded.
[oracle@racprd2 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 39922802 : applied on Fri Oct 23 20:20:46 IST 2026
Patch 39922783 : applied on Fri Oct 23 20:18:51 IST 2026
Patch 39915792 : applied on Fri Oct 23 19:57:30 IST 2026
Patch 39107855 : applied on Fri Oct 23 19:45:35 IST 2026
Patch 39107825 : applied on Fri Oct 23 19:45:11 IST 2026
[oracle@racprd2 OPatch]$
Both nodes show the full 19.33 set: ACFS 39922802, OCW 39922783, Database RU 39915792, Tomcat 39107855, and DBWLM 39107825. Identical lists on both nodes confirm the installer copied the patched home intact. The cluster is still running from grid_1 at this point.
The new Grid home is ready, so let's switch Node 1 to it.
Grid Home Switch on Node 1
The switch is rolling: only Node 1's clusterware stack restarts, and Node 2 keeps serving from grid_1. gridSetup.sh -switchGridHome registers the new home for that node, and root.sh then stops the stack from the old home, runs the patch steps, and restarts it from the new one.
Step 1: Switch the Grid Home
The environment is empty and every clusterware daemon and PMON process still runs from grid_1, which is the starting state worth capturing. The switch command lists only racprd1 in clusterNodes, so only this node changes:
[oracle@racprd1 ~]$ env|grep ORA
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ ps -ef|grep d.bin
root 1913 1 1 06:52 ? 00:03:19 /u01/app/grid/19c/grid_1/bin/ohasd.bin reboot
oracle 2294 1 0 06:52 ? 00:00:44 /u01/app/grid/19c/grid_1/bin/mdnsd.bin
oracle 2296 1 0 06:52 ? 00:01:51 /u01/app/grid/19c/grid_1/bin/evmd.bin
oracle 2376 1 0 06:52 ? 00:00:51 /u01/app/grid/19c/grid_1/bin/gpnpd.bin
oracle 2475 1 0 06:52 ? 00:01:56 /u01/app/grid/19c/grid_1/bin/gipcd.bin
root 3108 1 4 06:53 ? 00:09:03 /u01/app/grid/19c/grid_1/bin/osysmond.bin
oracle 3155 1 2 06:53 ? 00:05:51 /u01/app/grid/19c/grid_1/bin/ocssd.bin
root 3298 1 0 06:54 ? 00:01:57 /u01/app/grid/19c/grid_1/bin/octssd.bin reboot
root 3650 1 1 06:55 ? 00:04:00 /u01/app/grid/19c/grid_1/bin/crsd.bin reboot
oracle 4600 4501 0 10:31 pts/1 00:00:00 grep --color=auto d.bin
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ ps -ef|grep pmon
oracle 3516 1 0 06:54 ? 00:00:01 asm_pmon_+ASM1
oracle 4607 4501 0 10:31 pts/1 00:00:00 grep --color=auto pmon
oracle 4656 1 0 06:56 ? 00:00:01 ora_pmon_racprdb1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ cd /u01/app/grid/19c/grid_2
[oracle@racprd1 grid_2]$
[oracle@racprd1 grid_2]$ ./gridSetup.sh \
> -silent -switchGridHome \
> oracle.install.option=CRS_SWONLY \
> ORACLE_HOME=/u01/app/grid/19c/grid_2 \
> oracle.install.crs.config.clusterNodes=racprd1 \
> oracle.install.crs.rootconfig.executeRootScript=false
Launching Oracle Grid Infrastructure Setup Wizard...
As a root user, execute the following script(s):
1. /u01/app/grid/19c/grid_2/root.sh
Execute /u01/app/grid/19c/grid_2/root.sh on the following nodes:
[racprd1]
You can find the log of this install session at:
/u01/app/oraInventory/logs/UpdateNodeList2026-10-24_10-32-06AM.log
You can find the log of this install session at:
/u01/app/oraInventory/logs/UpdateNodeList2026-10-24_10-32-06AM.log
Successfully Setup Software.
[oracle@racprd1 grid_2]$
executeRootScript=false leaves the root script for a manual run, and the output asks for root.sh on racprd1 only.
Step 2: Run root.sh on Node 1
[root@racprd1 ~]# /u01/app/grid/19c/grid_2/root.sh
Check /u01/app/grid/19c/grid_2/install/root_racprd1.oraeasy.com_2026-10-24_10-37-27-937246478.log for the output of root script
[root@racprd1 ~]#
[root@racprd1 ~]# cat /u01/app/grid/19c/grid_2/install/root_racprd1.oraeasy.com_2026-10-24_10-37-27-937246478.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/grid/19c/grid_2
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/grid/19c/grid_1/lib:/u01/app/grid/19c/grid_2/lib:'
Using configuration parameter file: /u01/app/grid/19c/grid_2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/grid/crsdata/racprd1/crsconfig/rootcrs_racprd1_2026-10-24_10-46-38AM.log
Using configuration parameter file: /u01/app/grid/19c/grid_2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/grid/crsdata/racprd1/crsconfig/crs_prepatch_apply_oop_racprd1_2026-10-24_10-46-46AM.log
2026/10/24 10:48:31 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/grid/19c/grid_2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/grid/crsdata/racprd1/crsconfig/crs_postpatch_apply_oop_racprd1_2026-10-24_10-48-36AM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [724960844].
2026/10/24 10:53:01 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
NAME=ora.qosmserver
TYPE=ora.qosmserver.type
TARGET=ONLINE
STATE=ONLINE on racprd2
Redirecting to /bin/systemctl restart rsyslog.service
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [724960844].
2026/10/24 11:04:02 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2026/10/24 11:04:15 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
[root@racprd1 ~]#
The log tells the story of the switch. CLSRSC-671 and CLSRSC-672 bracket the pre-patch and post-patch steps, and CLSRSC-329 rewrites the oracle-ohasd.service entries so the stack starts from grid_2 after a reboot. The cluster upgrade state moves to ROLLING PATCH because Node 2 is still on the old home, and it only returns to NORMAL once both nodes are switched.
Two other lines are easy to misread. Relinking oracle with rac_on option means the script rebuilds the oracle binary in the new home with RAC enabled, which is a normal part of the switch. The LD_LIBRARY_PATH line lists both grid_1 and grid_2 because the old stack is still running while the new home's scripts execute, so it is not a sign of a wrong path.
Step 3: Watch the Stack While It Switches
[oracle@racprd1 ~]$ ps -ef|grep d.bin
root 1913 1 1 06:52 ? 00:03:33 /u01/app/grid/19c/grid_1/bin/ohasd.bin reboot
oracle 2296 1 0 06:52 ? 00:02:01 /u01/app/grid/19c/grid_1/bin/evmd.bin
oracle 2376 1 0 06:52 ? 00:00:55 /u01/app/grid/19c/grid_1/bin/gpnpd.bin
oracle 2475 1 0 06:52 ? 00:02:06 /u01/app/grid/19c/grid_1/bin/gipcd.bin
oracle 3155 1 2 06:53 ? 00:06:24 /u01/app/grid/19c/grid_1/bin/ocssd.bin
root 3298 1 0 06:54 ? 00:02:07 /u01/app/grid/19c/grid_1/bin/octssd.bin reboot
oracle 9617 3306 0 10:52 pts/0 00:00:00 grep --color=auto d.bin
[oracle@racprd1 ~]$ ps -ef|grep pmon
oracle 3516 1 0 06:54 ? 00:00:01 asm_pmon_+ASM1
oracle 9620 3306 0 10:52 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd1 ~]$ ps -ef|grep d.bin
oracle 10105 3306 0 10:53 pts/0 00:00:00 grep --color=auto d.bin
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ ps -ef|grep pmon
oracle 9641 3306 0 10:52 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ ps -ef|grep d.bin
oracle 2508 3306 0 11:08 pts/0 00:00:00 grep --color=auto d.bin
root 17508 1 2 10:54 ? 00:00:17 /u01/app/grid/19c/grid_2/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
oracle 18044 1 0 10:54 ? 00:00:07 /u01/app/grid/19c/grid_2/bin/evmd.bin
oracle 18047 1 0 10:54 ? 00:00:03 /u01/app/grid/19c/grid_2/bin/mdnsd.bin
oracle 18125 1 0 10:54 ? 00:00:03 /u01/app/grid/19c/grid_2/bin/gpnpd.bin
oracle 18273 1 1 10:54 ? 00:00:14 /u01/app/grid/19c/grid_2/bin/gipcd.bin
root 18817 1 2 10:55 ? 00:00:23 /u01/app/grid/19c/grid_2/bin/osysmond.bin
oracle 18968 1 4 10:55 ? 00:00:34 /u01/app/grid/19c/grid_2/bin/ocssd.bin
root 19475 1 0 10:55 ? 00:00:07 /u01/app/grid/19c/grid_2/bin/octssd.bin reboot
root 19805 1 2 10:56 ? 00:00:16 /u01/app/grid/19c/grid_2/bin/crsd.bin reboot
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ ps -ef|grep pmon
oracle 2541 3306 0 11:09 pts/0 00:00:00 grep --color=auto pmon
oracle 21442 1 0 10:57 ? 00:00:00 asm_pmon_+ASM1
oracle 22691 1 0 10:57 ? 00:00:00 ora_pmon_racprdb1
[oracle@racprd1 ~]$
The outputs are taken minutes apart. First the grid_1 daemons and both PMON processes disappear, which means ASM and the racprdb1 instance went down with the stack. Then the daemons return under grid_2 and the PMON processes come back at 10:57. Expect an instance outage on this node of about five minutes in this lab, during which Node 2 serves all connections.
Step 4: Verify Grid and Database on Node 1
The Grid side now reports 19.33, while the database home has not been switched yet and still reports 19.3:
Grid:
[oracle@racprd1 ~]$ sqlplus -v
SQL*Plus: Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
[oracle@racprd1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Sat Oct 24 11:24:47 2026
Version 19.33.0.0.0
Copyright (c) 1982, 2026, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
SQL> def
DEFINE _DATE = "24-OCT-26" (CHAR)
DEFINE _CONNECT_IDENTIFIER = "+ASM1" (CHAR)
DEFINE _USER = "SYS" (CHAR)
DEFINE _PRIVILEGE = "AS SYSASM" (CHAR)
DEFINE _SQLPLUS_RELEASE = "1933000000" (CHAR)
DEFINE _EDITOR = "vi" (CHAR)
DEFINE _O_VERSION = "Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0" (CHAR)
DEFINE _O_RELEASE = "1933000000" (CHAR)
SQL>
Database:
[oracle@racprd1 ~]$ . .db_env
[oracle@racprd1 ~]$ env|grep ORA
ORACLE_SID=racprdb1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/19c/db_1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ sqlplus -v
SQL*Plus: Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@racprd1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sat Oct 24 11:25:29 2026
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> alter session set nls_date_format= 'DD-MON-YYYY HH24:MI:SS';
Session altered.
SQL> set lines 200 pages 1000
SQL> col open_mode for a15
SQL> col HOST_NAME for a15
SQL> select distinct NAME,DATABASE_ROLE,OPEN_MODE,INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,VERSION,STARTUP_TIME,STATUS from gv$database,gv$instance;
NAME DATABASE_ROLE OPEN_MODE INSTANCE_NUMBER INSTANCE_NAME HOST_NAME VERSION STARTUP_TIME STATUS
--------- ---------------- --------------- --------------- ---------------- ------------------- ------------- -------------------- ------------
RACPRDB PRIMARY READ WRITE 1 racprdb1 racprd1.oraeasy.com 19.0.0.0.0 24-OCT-2026 10:57:38 OPEN
RACPRDB PRIMARY READ WRITE 2 racprdb2 racprd2.oraeasy.com 19.0.0.0.0 24-OCT-2026 07:22:50 OPEN
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@racprd1 ~]$
sqlplus from the Grid environment shows Version 19.33.0.0.0, but the database instance still runs from db_1 and prints 19.3.0.0.0. The startup times confirm only racprdb1 restarted (10:57), and racprdb2 kept its 07:22 start.
Step 5: Move the Services Back
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl relocate service -s racprdb_batch_svc -d racprdb -i racprdb2 -t racprdb1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
While the stack was down, the services whose preferred instance was on Node 1 moved to racprdb2, so Node 2 now carries racprdb_batch_svc too. The relocate command moves the batch service back to racprdb1, where -i names the instance the service leaves and -t names the instance it moves to. Clusterware does not fail services back on its own after a restart, which is why this step is needed.
Step 6: Confirm Node 2 Is Untouched
[oracle@racprd2 ~]$ ps -ef|grep d.bin
root 1951 1 1 07:18 ? 00:04:09 /u01/app/grid/19c/grid_1/bin/ohasd.bin reboot
oracle 2366 1 1 07:19 ? 00:02:39 /u01/app/grid/19c/grid_1/bin/evmd.bin
oracle 2367 1 0 07:19 ? 00:01:00 /u01/app/grid/19c/grid_1/bin/mdnsd.bin
oracle 2411 1 0 07:19 ? 00:01:09 /u01/app/grid/19c/grid_1/bin/gpnpd.bin
oracle 2512 1 1 07:19 ? 00:02:45 /u01/app/grid/19c/grid_1/bin/gipcd.bin
oracle 3537 1 3 07:20 ? 00:08:56 /u01/app/grid/19c/grid_1/bin/ocssd.bin
root 3723 1 1 07:20 ? 00:02:41 /u01/app/grid/19c/grid_1/bin/octssd.bin reboot
root 3846 1 1 07:20 ? 00:03:55 /u01/app/grid/19c/grid_1/bin/crsd.bin reboot
oracle 24668 506 0 11:39 pts/0 00:00:00 grep --color=auto d.bin
root 27362 1 14 11:03 ? 00:05:16 /u01/app/grid/19c/grid_1/bin/osysmond.bin
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ ps -ef|grep pmon
oracle 4556 1 0 07:22 ? 00:00:01 asm_pmon_+ASM2
oracle 5092 1 0 07:22 ? 00:00:02 ora_pmon_racprdb2
oracle 24729 506 0 11:39 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd2 ~]$
Every daemon on Node 2 still runs from grid_1, and ora_pmon_racprdb2 is up, so the cluster ran with one node on each Grid home during this window. That mixed state is expected in a rolling switch and is why the work continues node by node.
Grid on Node 1 is done, so let's install the new Database home.
New Database Home Installation with RU
The Database home follows the same pattern as the Grid home: software-only install into a new directory, OPatch upgraded first, and the RU applied by the installer. The running instances are not affected.
Step 1: Create the New Database Home Directory
Node 1:
[root@racprd1 ~]# mkdir -p /u01/app/oracle/19c/db_2
[root@racprd1 ~]# chown -R oracle:oinstall /u01/app/oracle/19c/db_2
[root@racprd1 ~]# chmod -R 775 /u01/app/oracle/19c/db_2
Node 2:
[root@racprd2 ~]# mkdir -p /u01/app/oracle/19c/db_2
[root@racprd2 ~]# chown -R oracle:oinstall /u01/app/oracle/19c/db_2
[root@racprd2 ~]# chmod -R 775 /u01/app/oracle/19c/db_2
[root@racprd2 ~]#
As with Grid, the empty directory is created on both nodes with oracle:oinstall ownership so the installer can populate Node 2 remotely.
Step 2: Unzip the Software and the RU
[oracle@racprd1 db_home]$ ls -lrth
total 5.1G
-rw-r--r--. 1 oracle oinstall 2.9G Oct 24 12:59 LINUX.X64_193000_db_home.zip
-rw-r--r--. 1 oracle oinstall 2.3G Oct 24 13:07 p39915792_190000_Linux-x86-64.zip
[oracle@racprd1 db_home]$
[oracle@racprd1 db_home]$ unzip LINUX.X64_193000_db_home.zip -d /u01/app/oracle/19c/db_2
[oracle@racprd1 db_home]$ ls -lrth
total 5.1G
-rw-r--r--. 1 oracle oinstall 2.9G Oct 24 12:59 LINUX.X64_193000_db_home.zip
-rw-r--r--. 1 oracle oinstall 2.3G Oct 24 13:07 p39915792_190000_Linux-x86-64.zip
[oracle@racprd1 db_home]$
[oracle@racprd1 db_home]$ unzip p39915792_190000_Linux-x86-64.zip
The base image goes straight into db_2, and the RU is extracted in the staging area. Only Node 1 needs the files.
Step 3: Upgrade OPatch in the New Home
[oracle@racprd1 ~]$ cd /u01/app/oracle/19c/db_2/OPatch/
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.17
OPatch succeeded.
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ cd ..
[oracle@racprd1 db_2]$ mv OPatch/ OPatch_bkp
[oracle@racprd1 db_2]$ cd /stage/opatch/
[oracle@racprd1 opatch]$
[oracle@racprd1 opatch]$ ls -lrth
total 129M
-rw-r--r--. 1 oracle oinstall 129M Oct 22 21:18 p6880880_190000_Linux-x86-64.zip
[oracle@racprd1 opatch]$
[oracle@racprd1 opatch]$ unzip p6880880_190000_Linux-x86-64.zip -d /u01/app/oracle/19c/db_2
[oracle@racprd1 opatch]$ cd /u01/app/oracle/19c/db_2/OPatch/
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.54
OPatch succeeded.
[oracle@racprd1 OPatch]$
The same OPatch upgrade is repeated for the database home, from 12.2.0.1.17 to 12.2.0.1.54, because this home carries its own copy of OPatch.
Step 4: Prepare the Response File
[oracle@racprd1 ~]$ cd /u01/app/oracle/19c/db_2/install/response/
[oracle@racprd1 response]$ ls -lrth
total 20K
-rw-r--r--. 1 oracle oinstall 20K Feb 6 2019 db_install.rsp
[oracle@racprd1 response]$ mv db_install.rsp db_install.rsp_bkp
[oracle@racprd1 response]$
[oracle@racprd1 response]$ vi db_install.rsp
[oracle@racprd1 response]$ cat db_install.rsp
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0
oracle.install.option=INSTALL_DB_SWONLY
UNIX_GROUP_NAME=oinstall
INVENTORY_LOCATION=/u01/app/oraInventory
ORACLE_HOME=/u01/app/oracle/19c/db_2
ORACLE_BASE=/u01/app/oracle
oracle.install.db.InstallEdition=EE
oracle.install.db.OSDBA_GROUP=dba
oracle.install.db.OSOPER_GROUP=oper
oracle.install.db.OSBACKUPDBA_GROUP=backupdba
oracle.install.db.OSDGDBA_GROUP=dgdba
oracle.install.db.OSKMDBA_GROUP=kmdba
oracle.install.db.OSRACDBA_GROUP=racdba
oracle.install.db.CLUSTER_NODES=racprd1,racprd2
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
DECLINE_SECURITY_UPDATES=true
oracle.install.db.rootconfig.executeRootScript=false
[oracle@racprd1 response]$
ORACLE_HOME points at db_2, and the OS group parameters match the groups the oracle user already belongs to. CLUSTER_NODES makes the installer copy the home to Node 2, and DECLINE_SECURITY_UPDATES=true skips the My Oracle Support prompt so the run stays non-interactive.
Step 5: Launch runInstaller with the RU
[oracle@racprd1 response]$ cd /u01/app/oracle/19c/db_2
[oracle@racprd1 db_2]$
[oracle@racprd1 db_2]$ ls -lrth runInstaller
-rwxr-x---. 1 oracle oinstall 1.8K Mar 8 2017 runInstaller
[oracle@racprd1 db_2]$
[oracle@racprd1 db_2]$ ./runInstaller \
> -silent \
> -ignorePrereqFailure \
> -waitforcompletion \
> -applyRU /stage/db_home/39915792 \
> -responseFile /u01/app/oracle/19c/db_2/install/response/db_install.rsp
Preparing the home to patch...
Applying the patch /stage/db_home/39915792...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/InstallActions2026-10-25_08-54-23AM/installerPatchActions_2026-10-25_08-54-23AM.log
Launching Oracle Database Setup Wizard...
[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/InstallActions2026-10-25_08-54-23AM/installActions2026-25_08-54-23AM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/InstallActions2026-10-25_08-54-23AM/installActions2026-10-25_04-23AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
/u01/app/oracle/19c/db_2/install/response/db_2026-10-25_08-54-23AM.rsp
You can find the log of this install session at:
/u01/app/oraInventory/logs/InstallActions2026-10-25_08-54-23AM/installActions2026-10-25_08-54-23AM.log
As a root user, execute the following script(s):
1. /u01/app/oracle/19c/db_2/root.sh
Execute /u01/app/oracle/19c/db_2/root.sh on the following nodes:
[racprd1, racprd2]
Successfully Setup Software with warning(s).
[oracle@racprd1 db_2]$
-applyRU patches the home before installation, as it did for Grid. The INS-13013 warning is a mandatory-prerequisite warning that -ignorePrereqFailure lets the run pass; open the log it names to see which check failed, and treat it as a blocker on a production system. As before, only root.sh remains.
Step 6: Run root.sh on Both Nodes
[root@racprd1 ~]# /u01/app/oracle/19c/db_2/root.sh
Check /u01/app/oracle/19c/db_2/install/root_racprd1.oraeasy.com_2026-10-25_10-00-41-091909787.log for the output of root script
[root@racprd1 ~]#
[root@racprd1 ~]# cat /u01/app/oracle/19c/db_2/install/root_racprd1.oraeasy.com_2026-10-25_10-00-41-091909787.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/19c/db_2
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@racprd1 ~]#
Root Node 2:
[root@racprd2 ~]# /u01/app/oracle/19c/db_2/root.sh
Check /u01/app/oracle/19c/db_2/install/root_racprd2.oraeasy.com_2026-10-25_10-02-20-746354973.log for the output of root script
[root@racprd2 ~]# cat /u01/app/oracle/19c/db_2/install/root_racprd2.oraeasy.com_2026-10-25_10-02-20-746354973.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/19c/db_2
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@racprd2 ~]#
The Database home root.sh only copies helper files to /usr/local/bin. It does not touch the cluster, so it can run on the two nodes in any order.
Step 7: Verify the Patches in the New Database Home
Node 1:
[oracle@racprd1 db_2]$ cd OPatch
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lspatches
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 39915792 : applied on Sun Oct 25 09:24:31 IST 2026
Patch 29585399 : applied on Thu Apr 18 12:51:33 IST 2019
[oracle@racprd1 OPatch]$
Node 2:
[oracle@racprd2 ~]$ cd /u01/app/oracle/19c/db_2/OPatch
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lspatches
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 39915792 : applied on Sun Oct 25 09:24:31 IST 2026
Patch 29585399 : applied on Thu Apr 18 12:51:33 IST 2019
[oracle@racprd2 OPatch]$
The new home lists Database RU 39915792 (19.33.0.0.261020) and the original OCW patch 29585399. That OCW entry is carried over from the 19.3 base image and is expected. Both nodes match.
The new Database home is ready, so let's switch the database on Node 1.
Database Home Switch on Node 1
Switching the database home has two parts: changing the home registered in the cluster, and restarting each instance so it starts from the new home. The registration is cluster-wide, so it is done once. Only the restart is node by node.
Step 1: Check the Existing Configuration
The configuration shows which home the database is registered with, and the shell's ORACLE_HOME is then pointed at the new home so the next srvctl command runs from it:
[oracle@racprd1 ~]$ ps -ef|grep pmon
oracle 6497 1 0 12:33 ? 00:00:00 asm_pmon_+ASM1
oracle 8889 1 0 12:34 ? 00:00:00 ora_pmon_racprdb1
oracle 10981 5284 0 12:36 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ . .db_env
[oracle@racprd1 ~]$ srvctl config
racprdb
[oracle@racprd1 ~]$ srvctl config database -d racprdb
Database unique name: racprdb
Database name: racprdb
Oracle home: /u01/app/oracle/19c/db_1
Oracle user: oracle
Spfile: +DATA/RACPRDB/PARAMETERFILE/spfile.268.1245104519
Password file: +DATA/RACPRDB/PASSWORD/pwdracprdb.256.1245102217
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services: racprdb_batch_svc,racprdb_rman_svc,racprdb_svc
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: racprdb1,racprdb2
Configured nodes: racprd1,racprd2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ export ORACLE_HOME=/u01/app/oracle/19c/db_2
[oracle@racprd1 ~]$ echo $ORACLE_HOME
/u01/app/oracle/19c/db_2
[oracle@racprd1 ~]$
Oracle home: /u01/app/oracle/19c/db_1 is the value being replaced, and the three services are listed as attached to the database. Exporting ORACLE_HOME to db_2 ensures the modify command uses the new home's own srvctl.
Step 2: Modify the Database Home in the Cluster Registry
[oracle@racprd1 ~]$ $ORACLE_HOME/bin/srvctl modify database -d racprdb -oraclehome /u01/app/oracle/19c/db_2
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ $ORACLE_HOME/bin/srvctl config database -d racprdb | grep -i "oracle home"
Oracle home: /u01/app/oracle/19c/db_2
[oracle@racprd1 ~]$
This single command updates the registered home for the whole database, both instances included. Running instances are not restarted by it, so nothing changes yet, and that is also why no second modify is needed for Node 2.
Step 3: Stop the Instance on Node 1
The batch service is relocated to Node 2 first and racprdb_svc is stopped on this instance, so sessions are directed away before the instance shuts down:
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl relocate service -s racprdb_batch_svc -d racprdb -i racprdb1 -t racprdb2
[oracle@racprd1 ~]$ srvctl stop service -s racprdb_svc -d racprdb -i racprdb1
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl stop instance -d racprdb -i racprdb1 -o immediate
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is not running on node racprd1
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open,Running from Old Oracle Home.
[oracle@racprd1 ~]$
After the stop, racprdb2 reports Open,Running from Old Oracle Home. That status is the clusterware telling you the registered home has changed but this instance has not restarted from it yet, which is exactly the expected state until Node 2 is switched.
Step 4: Start the Instance on Node 1
[oracle@racprd1 ~]$ srvctl start instance -d racprdb -i racprdb1
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open,Running from Old Oracle Home.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ ps -ef|grep pmon
oracle 6497 1 0 12:33 ? 00:00:00 asm_pmon_+ASM1
oracle 11563 1 0 13:04 ? 00:00:00 ora_pmon_racprdb1
oracle 14325 5284 0 13:07 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd1 ~]$
The start now launches racprdb1 from db_2. racprdb_svc comes back with it, while the batch service stays on racprdb2 until it is relocated at the end.
Step 5: Update the Environment File and Check the Version
[oracle@racprd1 ~]$ vi .db_env
[oracle@racprd1 ~]$ cat .db_env
# Environment variables for Database - DB Home
ORACLE_SID=racprdb1; export ORACLE_SID
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/19c/db_2; export ORACLE_HOME
JAVA_HOME=/usr/bin/java; export JAVA_HOME
PATH=$ORACLE_HOME/bin:$JAVA_HOME/bin:$PATH:.
export PATH
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ . .db_env
[oracle@racprd1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Oct 25 13:13:17 2026
Version 19.33.0.0.0
Copyright (c) 1982, 2026, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
SQL> SELECT banner_full FROM v$version;
BANNER_FULL
--------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
SQL>
Editing ORACLE_HOME in .db_env makes new sessions use the new home. sqlplus and v$version now report 19.33.0.0.0. This is the binary version. The SQL patch level inside the database is updated later by datapatch.
Step 6: Confirm Node 2 Is Untouched
[oracle@racprd2 ~]$ . .db_env
[oracle@racprd2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Oct 25 13:15:31 2026
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> SELECT banner_full FROM v$version;
BANNER_FULL
--------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
racprdb2
SQL>
Node 2 still reports 19.3.0.0.0 and racprdb2 is running from the old home, so the database is serving from two different home versions for now. Both are compatible with the same datafiles, which is what makes the rolling approach work.
Node 1 is now on the new Grid and Database homes, so let's move Node 2.
Grid Home Switch on Node 2
The Grid switch on Node 2 repeats what was done on Node 1, with one useful side effect: restarting the stack also restarts the racprdb2 instance, and because the database home was already changed in the registry, it comes back on the new Database home automatically.
Step 1: Switch the Grid Home
[oracle@racprd2 ~]$ ps -ef|grep d.bin
root 3085 1 1 12:43 ? 00:02:15 /u01/app/grid/19c/grid_1/bin/ohasd.bin reboot
oracle 3756 1 0 12:44 ? 00:00:25 /u01/app/grid/19c/grid_1/bin/mdnsd.bin
oracle 3757 1 0 12:44 ? 00:01:07 /u01/app/grid/19c/grid_1/bin/evmd.bin
oracle 3814 1 0 12:44 ? 00:00:29 /u01/app/grid/19c/grid_1/bin/gpnpd.bin
oracle 3882 1 0 12:44 ? 00:01:09 /u01/app/grid/19c/grid_1/bin/gipcd.bin
oracle 5704 1 4 12:45 ? 00:05:12 /u01/app/grid/19c/grid_1/bin/ocssd.bin
oracle 5938 27690 0 14:45 pts/1 00:00:00 grep --color=auto d.bin
root 6068 1 0 12:45 ? 00:01:08 /u01/app/grid/19c/grid_1/bin/octssd.bin reboot
root 6748 1 1 12:45 ? 00:01:37 /u01/app/grid/19c/grid_1/bin/crsd.bin reboot
root 30625 1 23 14:30 ? 00:03:32 /u01/app/grid/19c/grid_1/bin/osysmond.bin
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ cd /u01/app/grid/19c/grid_2
[oracle@racprd2 grid_2]$
[oracle@racprd2 grid_2]$ ./gridSetup.sh \
> -silent -switchGridHome \
> oracle.install.option=CRS_SWONLY \
> ORACLE_HOME=/u01/app/grid/19c/grid_2 \
> oracle.install.crs.config.clusterNodes=racprd2 \
> oracle.install.crs.rootconfig.executeRootScript=false
Launching Oracle Grid Infrastructure Setup Wizard...
As a root user, execute the following script(s):
1. /u01/app/grid/19c/grid_2/root.sh
Execute /u01/app/grid/19c/grid_2/root.sh on the following nodes:
[racprd2]
You can find the log of this install session at:
/u01/app/oraInventory/logs/UpdateNodeList2026-10-25_02-46-38PM.log
You can find the log of this install session at:
/u01/app/oraInventory/logs/UpdateNodeList2026-10-25_02-46-38PM.log
Successfully Setup Software.
[oracle@racprd2 grid_2]$
The command is the same as on Node 1 with clusterNodes=racprd2. All daemons are still running from grid_1 before the switch.
Step 2: Run root.sh on Node 2
[root@racprd2 ~]# /u01/app/grid/19c/grid_2/root.sh
Check /u01/app/grid/19c/grid_2/install/root_racprd2.oraeasy.com_2026-10-25_14-50-25-741062449.log for the output of root script
[root@racprd2 ~]#
[root@racprd2 ~]# cat /u01/app/grid/19c/grid_2/install/root_racprd2.oraeasy.com_2026-10-25_14-50-25-741062449.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/grid/19c/grid_2
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/grid/19c/grid_1/lib:/u01/app/grid/19c/grid_2/lib:'
Using configuration parameter file: /u01/app/grid/19c/grid_2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/grid/crsdata/racprd2/crsconfig/rootcrs_racprd2_2026-10-25_03-04-51PM.log
Using configuration parameter file: /u01/app/grid/19c/grid_2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/grid/crsdata/racprd2/crsconfig/crs_prepatch_apply_oop_racprd2_2026-10-25_03-04-58PM.log
2026/10/25 15:08:01 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/grid/19c/grid_2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/grid/crsdata/racprd2/crsconfig/crs_postpatch_apply_oop_racprd2_2026-10-25_03-08-07PM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [724960844].
2026/10/25 15:12:38 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
CRS-2613: Could not find resource 'ora.qosmserver'.
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [234314449].
2026/10/25 15:30:39 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2026/10/25 15:31:32 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
[root@racprd2 ~]#
This time the state starts as ROLLING PATCH and ends as NORMAL, and the cluster active patch level changes from 724960844 to 234314449. The patch level only advances once every node runs the new home. The line CRS-2613: Could not find resource 'ora.qosmserver' is also worth noting, and it is revisited in the final checks.
Step 3: Watch the Stack While It Switches
[oracle@racprd2 grid_2]$ ps -ef|grep d.bin
root 3085 1 1 12:43 ? 00:02:43 /u01/app/grid/19c/grid_1/bin/ohasd.bin reboot
oracle 3756 1 0 12:44 ? 00:00:30 /u01/app/grid/19c/grid_1/bin/mdnsd.bin
oracle 3757 1 0 12:44 ? 00:01:22 /u01/app/grid/19c/grid_1/bin/evmd.bin
oracle 3814 1 0 12:44 ? 00:00:35 /u01/app/grid/19c/grid_1/bin/gpnpd.bin
oracle 3882 1 0 12:44 ? 00:01:24 /u01/app/grid/19c/grid_1/bin/gipcd.bin
oracle 5704 1 4 12:45 ? 00:06:14 /u01/app/grid/19c/grid_1/bin/ocssd.bin
root 6068 1 0 12:45 ? 00:01:23 /u01/app/grid/19c/grid_1/bin/octssd.bin reboot
root 6748 1 1 12:45 ? 00:01:58 /u01/app/grid/19c/grid_1/bin/crsd.bin reboot
oracle 17066 27690 0 15:06 pts/1 00:00:00 grep --color=auto d.bin
root 30625 1 22 14:30 ? 00:08:15 /u01/app/grid/19c/grid_1/bin/osysmond.bin
[oracle@racprd2 grid_2]$
[oracle@racprd2 grid_2]$ ps -ef|grep pmon
oracle 8004 1 0 12:47 ? 00:00:01 asm_pmon_+ASM2
oracle 8568 1 0 12:47 ? 00:00:01 ora_pmon_racprdb2
oracle 20564 27690 0 15:08 pts/1 00:00:00 grep --color=auto pmon
[oracle@racprd2 grid_2]$ ps -ef|grep pmon
oracle 23034 27690 0 15:12 pts/1 00:00:00 grep --color=auto pmon
[oracle@racprd2 grid_2]$
[oracle@racprd2 grid_2]$ ps -ef|grep d.bin
oracle 23060 27690 0 15:12 pts/1 00:00:00 grep --color=auto d.bin
[oracle@racprd2 grid_2]$
[oracle@racprd2 grid_2]$ ps -ef|grep d.bin
oracle 1553 27690 0 15:18 pts/1 00:00:00 grep --color=auto d.bin
root 28607 1 5 15:14 ? 00:00:13 /u01/app/grid/19c/grid_2/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
oracle 28864 1 1 15:14 ? 00:00:04 /u01/app/grid/19c/grid_2/bin/evmd.bin
oracle 28868 1 0 15:14 ? 00:00:01 /u01/app/grid/19c/grid_2/bin/mdnsd.bin
oracle 28916 1 0 15:14 ? 00:00:01 /u01/app/grid/19c/grid_2/bin/gpnpd.bin
oracle 28963 1 2 15:14 ? 00:00:05 /u01/app/grid/19c/grid_2/bin/gipcd.bin
root 29111 1 3 15:14 ? 00:00:09 /u01/app/grid/19c/grid_2/bin/osysmond.bin
oracle 29281 1 5 15:14 ? 00:00:13 /u01/app/grid/19c/grid_2/bin/ocssd.bin
root 29676 1 1 15:14 ? 00:00:04 /u01/app/grid/19c/grid_2/bin/octssd.bin reboot
root 30047 1 6 15:15 ? 00:00:13 /u01/app/grid/19c/grid_2/bin/crsd.bin reboot
[oracle@racprd2 grid_2]$
[oracle@racprd2 grid_2]$ ps -ef|grep pmon
oracle 1597 27690 0 15:18 pts/1 00:00:00 grep --color=auto pmon
oracle 32101 1 0 15:16 ? 00:00:00 asm_pmon_+ASM2
oracle 32726 1 0 15:17 ? 00:00:00 ora_pmon_racprdb2
[oracle@racprd2 grid_2]$
As on Node 1, the old daemons and both PMON processes stop, and the grid_2 daemons start at 15:14. The ora_pmon_racprdb2 process returns at 15:17, which is the instance restarting from the new Database home without any manual action.
Step 4: Update the Grid Environment and Verify
[oracle@racprd2 ~]$ vi .grid_env
[oracle@racprd2 ~]$ cat .grid_env
# Environment variables for ASM - Grid Home
ORACLE_SID=+ASM2; export ORACLE_SID
ORACLE_BASE=/u01/grid; export ORACLE_BASE
ORACLE_HOME=/u01/app/grid/19c/grid_2; export ORACLE_HOME
JAVA_HOME=/usr/bin/java; export JAVA_HOME
PATH=$ORACLE_HOME/bin:$JAVA_HOME/bin:$PATH:.
export PATH
[oracle@racprd2 ~]$ . .grid_env
[oracle@racprd2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Oct 25 17:15:24 2026
Version 19.33.0.0.0
Copyright (c) 1982, 2026, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
[oracle@racprd2 ~]$ sqlplus -v
SQL*Plus: Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
[oracle@racprd2 ~]$
.grid_env now points at grid_2, and sqlplus reports 19.33.0.0.0 for the ASM connection. Both nodes are now on the new Grid home.
Step 5: Verify the Database on Node 2
No separate database switch is needed on Node 2, because the registry was already changed and the instance restarted during the Grid switch. The checks below only confirm it:
[oracle@racprd2 ~]$ vi .db_env
[oracle@racprd2 ~]$ cat .db_env
# Environment variables for Database - DB Home
ORACLE_SID=racprdb2; export ORACLE_SID
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/19c/db_2; export ORACLE_HOME
JAVA_HOME=/usr/bin/java; export JAVA_HOME
PATH=$ORACLE_HOME/bin:$JAVA_HOME/bin:$PATH:.
export PATH
[oracle@racprd2 ~]$ . .db_env
[oracle@racprd2 ~]$ env|grep ORA
ORACLE_SID=racprdb2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/19c/db_2
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ srvctl config database -d racprdb
Database unique name: racprdb
Database name: racprdb
Oracle home: /u01/app/oracle/19c/db_2
Oracle user: oracle
Spfile: +DATA/RACPRDB/PARAMETERFILE/spfile.268.1245104519
Password file: +DATA/RACPRDB/PASSWORD/pwdracprdb.256.1245102217
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: FRA,DATA
Mount point paths:
Services: racprdb_batch_svc,racprdb_rman_svc,racprdb_svc
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: racprdb1,racprdb2
Configured nodes: racprd1,racprd2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Oct 25 17:10:49 2026
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_svc. Instance status: Open.
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ srvctl relocate service -s racprdb_rman_svc -d racprdb -i racprdb1 -t racprdb2
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd2 ~]$
srvctl config database shows db_2 and the instance reports 19.33. The SQL*Plus banner line still says 19.3 while the Connected to section says 19.33, which suggests the client binary came from the old path while the instance is on the new home; check with which sqlplus if you need to confirm. When Node 2 restarted, every service landed on racprdb1, so racprdb_rman_svc is relocated back to its preferred instance at the end.
Both nodes are on the new homes, so the SQL patch can now be applied.
Datapatch
The new binaries are in place, but the database itself still records only the 19.3 SQL patch level. datapatch applies the SQL part of the RU to the data dictionary.
Step 1: Run datapatch
[oracle@racprd1 ~]$ cd $ORACLE_HOME/OPatch
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./datapatch -verbose
SQL Patching tool version 19.33.0.0.0 Production on Sun Oct 25 17:22:41 2026
Copyright (c) 2012, 2026, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_18910_2026_10_25_17_22_41/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Bootstrapping registry and package to current versions...done
Determining current state... done
Current state of interim SQL patches:
No interim patches found
Current state of release update SQL patches:
Binary registry:
19.33.0.0.0 Release_Update 261009085306: Installed
SQL registry:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 27-SEP-26 10.11.51.951770 PM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
No interim patches need to be rolled back
Patch 39915792 (Database Release Update : 19.33.0.0.261020 (39915792)):
Apply from 19.3.0.0.0 Release_Update 190410122720 to 19.33.0.0.0 Release_Update 261009085306
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 1
Validating logfiles...done
Patch 39915792 apply: SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/39915792/29514650/39915792_apply_RACPRDB_2026Oct25_17_33_00.log (no errors)
SQL Patching tool complete on Sun Oct 25 19:42:14 2026
[oracle@racprd1 OPatch]$
datapatch runs once, from one node, because the SQL changes are made inside the single shared database. It runs from the new home and only after every instance runs from it, as is the case here. The tool version in the first line, 19.33.0.0.0, confirms the shell is using the new db_2 copy, and that matters because the old home's datapatch does not know about this patch. The summary shows the binary registry at 19.33 and the SQL registry at 19.3, so it queues patch 39915792 and finishes with apply: SUCCESS and no errors in the log. The run started at 17:22 and ended at 19:42, so plan for a long window on a small lab system.
Node Reboots
An OS restart is not required for the patch itself, but restarting each node in turn proves the cluster and the database start on their own from the new homes, which is the real test of the switch.
Step 1: Reboot Node 1
[oracle@racprd1 ~]$ su -
Password:
Last login: Sun Oct 25 19:39:08 IST 2026
[root@racprd1 ~]#
[root@racprd1 ~]# init 6
Remote side unexpectedly closed network connection
─────────────────────────────────────────────────────────────────────────────────────────────────────
Session stopped
- Press <Return> to exit tab
- Press R to restart session
- Press S to save terminal output to file
┌──────────────────────────────────────────────────────────────────────┐
│ • MobaXterm Personal Edition v23.2 • │
│ (SSH client, X server and network tools) │
│ │
│ ⮞ SSH session to oracle@192.168.1.61 │
│ • Direct SSH : ✓ │
│ • SSH compression : ✓ │
│ • SSH-browser : ✓ │
│ • X11-forwarding : ✓ (remote display is forwarded through SSH) │
│ │
│ ⮞ For more info, ctrl+click on help or visit our website. │
└──────────────────────────────────────────────────────────────────────┘
Last login: Sun Oct 25 20:13:38 2026
[oracle@racprd1 ~]$ uptime -p
up 5 minutes
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ ps -ef|grep pmon
oracle 7286 1 0 20:14 ? 00:00:00 asm_pmon_+ASM1
oracle 7950 1 0 20:14 ? 00:00:00 ora_pmon_racprdb1
oracle 9276 6412 0 20:17 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ . .db_env
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status database -d racprdb
Instance racprdb1 is running on node racprd1
Instance racprdb2 is running on node racprd2
[oracle@racprd1 ~]$
The SSH session drops when init 6 runs, which is expected. uptime -p reports five minutes, which confirms the node really restarted, and the PMON start times of 20:14 show ASM and racprdb1 came up by themselves right after boot. srvctl then shows both instances up, so the stack started from grid_2 without help.
Step 2: Reboot Node 2
[oracle@racprd2 ~]$ su -
Password:
Last login: Sun Oct 25 20:19:30 IST 2026
[root@racprd2 ~]#
[root@racprd2 ~]# ps -ef|grep pmon
root 9589 8857 0 20:20 pts/0 00:00:00 grep --color=auto pmon
oracle 32101 1 0 15:16 ? 00:00:02 asm_pmon_+ASM2
oracle 32726 1 0 15:17 ? 00:00:02 ora_pmon_racprdb2
[root@racprd2 ~]#
[root@racprd2 ~]# init 6
Remote side unexpectedly closed network connection
─────────────────────────────────────────────────────────────────────────────────────────────────────
Session stopped
- Press <Return> to exit tab
- Press R to restart session
- Press S to save terminal output to file
┌──────────────────────────────────────────────────────────────────────┐
│ • MobaXterm Personal Edition v23.2 • │
│ (SSH client, X server and network tools) │
│ │
│ ⮞ SSH session to oracle@192.168.1.62 │
│ • Direct SSH : ✓ │
│ • SSH compression : ✓ │
│ • SSH-browser : ✓ │
│ • X11-forwarding : ✓ (remote display is forwarded through SSH) │
│ │
│ ⮞ For more info, ctrl+click on help or visit our website. │
└──────────────────────────────────────────────────────────────────────┘
Last login: Sun Oct 25 20:33:41 2026
[oracle@racprd2 ~]$ uptime -p
up 6 minutes
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ ps -ef|grep pmon
oracle 8939 1 0 20:31 ? 00:00:00 asm_pmon_+ASM2
oracle 10283 1 0 20:32 ? 00:00:00 ora_pmon_racprdb2
oracle 13075 12419 0 20:35 pts/1 00:00:00 grep --color=auto pmon
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ . .db_env
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ srvctl status database -d racprdb
Instance racprdb1 is running on node racprd1
Instance racprdb2 is not running on node racprd2
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ srvctl status database -d racprdb
Instance racprdb1 is running on node racprd1
Instance racprdb2 is running on node racprd2
[oracle@racprd2 ~]$
The first srvctl status right after login shows racprdb2 not running, and the second shows it running. The instance simply was not started yet, since the clusterware brings the database up a little after the OS is reachable. Run the status twice a minute apart before assuming a problem.
Both nodes have restarted cleanly, so let's do the final verification.
Post-Patch Verification
The checks below mirror the baseline from Phase A in the same order, so each result can be compared line by line with what was recorded before patching. The database and SQL patch registry come first, then the services, listeners, cluster resources, and finally both patch inventories.
Step 1: Verify the Database and SQL Patch Registry
[oracle@racprd1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Oct 25 20:45:39 2026
Version 19.33.0.0.0
Copyright (c) 1982, 2026, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
SQL> set lines 200 pages 1000
SQL> col open_mode for a15
SQL> col HOST_NAME for a15
SQL> select distinct NAME,DATABASE_ROLE,OPEN_MODE,INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,VERSION,STARTUP_TIME,STATUS from gv$database,gv$instance;
NAME DATABASE_ROLE OPEN_MODE INSTANCE_NUMBER INSTANCE_NAME HOST_NAME VERSION STARTUP_T STATUS
--------- ---------------- ----------- --------------- -------------- ------------------- ----------- --------- -------
RACPRDB PRIMARY READ WRITE 1 racprdb1 racprd1.oraeasy.com 19.0.0.0.0 25-OCT-26 OPEN
RACPRDB PRIMARY READ WRITE 2 racprdb2 racprd2.oraeasy.com 19.0.0.0.0 25-OCT-26 OPEN
SQL> SELECT banner_full FROM v$version;
BANNER_FULL
-----------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
SQL> col ACTION_TIME for a20
SQL> col ACTION for a10
SQL> col STATUS for a10
SQL> select patch_uid,patch_id,SOURCE_VERSION, TARGET_VERSION, action, status,action_time,description from dba_registry_sqlpatch;
PATCH_UID PATCH_ID SOURCE_VERSION TARGET_VERSION ACTION STATUS ACTION_TIME DESCRIPTION
---------- ---------- --------------- --------------- ---------- ---------- -------------------- -----------------------------------------------------------
22862832 29517242 19.1.0.0.0 19.3.0.0.0 APPLY SUCCESS 27-SEP-26 10.11.51.9 Database Release Update : 19.3.0.0.190416 (29517242)
51770 PM
29514650 39915792 19.3.0.0.0 19.33.0.0.0 APPLY SUCCESS 25-OCT-26 07.37.46.4 Database Release Update : 19.33.0.0.261020 (39915792)
02464 PM
SQL> col comp_id for a10
SQL> col version for a11
SQL> col status for a10
SQL> col comp_name for a37
SQL> select comp_id,comp_name,version,status from dba_registry;
COMP_ID COMP_NAME VERSION STATUS
---------- ------------------------------------- ----------- ----------
CATALOG Oracle Database Catalog Views 19.0.0.0.0 VALID
CATPROC Oracle Database Packages and Types 19.0.0.0.0 VALID
RAC Oracle Real Application Clusters 19.0.0.0.0 VALID
JAVAVM JServer JAVA Virtual Machine 19.0.0.0.0 VALID
XML Oracle XDK 19.0.0.0.0 VALID
CATJAVA Oracle Database Java Packages 19.0.0.0.0 VALID
APS OLAP Analytic Workspace 19.0.0.0.0 VALID
XDB Oracle XML Database 19.0.0.0.0 VALID
OWM Oracle Workspace Manager 19.0.0.0.0 VALID
CONTEXT Oracle Text 19.0.0.0.0 VALID
ORDIM Oracle Multimedia 19.0.0.0.0 VALID
SDO Spatial 19.0.0.0.0 LOADING
XOQ Oracle OLAP API 19.0.0.0.0 VALID
OLS Oracle Label Security 19.0.0.0.0 VALID
DV Oracle Database Vault 19.0.0.0.0 VALID
15 rows selected.
The banner reports 19.33.0.0.0 and both instances are open after the restart, and dba_registry_sqlpatch now lists the base RU and the new 39915792 entry as APPLY SUCCESS, which is the authoritative proof the SQL patch is in. One line needs attention: Spatial (SDO) shows LOADING instead of VALID.
Step 2: Recompile Objects and Recheck the Registry
SQL> @?/rdbms/admin/utlrp.sql
Session altered.
TIMESTAMP
--------------------------------------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN 2026-10-25 20:47:14
DOC> The following PL/SQL block invokes UTL_RECOMP to recompile invalid
DOC> objects in the database. Recompilation time is proportional to the
DOC> number of invalid objects in the database, so this command may take
DOC> a long time to execute on a database with a large number of invalid
DOC> objects.
DOC>
DOC> Use the following queries to track recompilation progress:
DOC>
DOC> 1. Query returning the number of invalid objects remaining. This
DOC> number should decrease with time.
DOC> SELECT COUNT(*) FROM obj$ WHERE status IN (4, 5, 6);
DOC>
DOC> 2. Query returning the number of objects compiled so far. This number
DOC> should increase with time.
DOC> SELECT COUNT(*) FROM UTL_RECOMP_COMPILED;
DOC>
DOC> This script automatically chooses serial or parallel recompilation
DOC> based on the number of CPUs available (parameter cpu_count) multiplied
DOC> by the number of threads per CPU (parameter parallel_threads_per_cpu).
DOC> On RAC, this number is added across all RAC nodes.
DOC>
DOC> UTL_RECOMP uses DBMS_SCHEDULER to create jobs for parallel
DOC> recompilation. Jobs are created without instance affinity so that they
DOC> can migrate across RAC nodes. Use the following queries to verify
DOC> whether UTL_RECOMP jobs are being created and run correctly:
DOC>
DOC> 1. Query showing jobs created by UTL_RECOMP
DOC> SELECT job_name FROM dba_scheduler_jobs
DOC> WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>
DOC> 2. Query showing UTL_RECOMP jobs that are running
DOC> SELECT job_name FROM dba_scheduler_running_jobs
DOC> WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>#
PL/SQL procedure successfully completed.
TIMESTAMP
--------------------------------------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_END 2026-10-25 20:47:35
DOC> The following query reports the number of invalid objects.
DOC>
DOC> If the number is higher than expected, please examine the error
DOC> messages reported with each object (using SHOW ERRORS) to see if they
DOC> point to system misconfiguration or resource constraints that must be
DOC> fixed before attempting to recompile these objects.
DOC>#
OBJECTS WITH ERRORS
-------------------
0
DOC> The following query reports the number of exceptions caught during
DOC> recompilation. If this number is non-zero, please query the error
DOC> messages in the table UTL_RECOMP_ERRORS to see if any of these errors
DOC> are due to misconfiguration or resource constraints that must be
DOC> fixed before objects can compile successfully.
DOC> Note: Typical compilation errors (due to coding errors) are not
DOC> logged into this table: they go into DBA_ERRORS instead.
DOC>#
ERRORS DURING RECOMPILATION
---------------------------
0
Function created.
PL/SQL procedure successfully completed.
Function dropped.
PL/SQL procedure successfully completed.
SQL>
SQL> select comp_id,comp_name,version,status from dba_registry;
COMP_ID COMP_NAME VERSION STATUS
---------- ------------------------------------- ----------- ----------
CATALOG Oracle Database Catalog Views 19.0.0.0.0 VALID
CATPROC Oracle Database Packages and Types 19.0.0.0.0 VALID
RAC Oracle Real Application Clusters 19.0.0.0.0 VALID
JAVAVM JServer JAVA Virtual Machine 19.0.0.0.0 VALID
XML Oracle XDK 19.0.0.0.0 VALID
CATJAVA Oracle Database Java Packages 19.0.0.0.0 VALID
APS OLAP Analytic Workspace 19.0.0.0.0 VALID
XDB Oracle XML Database 19.0.0.0.0 VALID
OWM Oracle Workspace Manager 19.0.0.0.0 VALID
CONTEXT Oracle Text 19.0.0.0.0 VALID
ORDIM Oracle Multimedia 19.0.0.0.0 VALID
SDO Spatial 19.0.0.0.0 VALID
XOQ Oracle OLAP API 19.0.0.0.0 VALID
OLS Oracle Label Security 19.0.0.0.0 VALID
DV Oracle Database Vault 19.0.0.0.0 VALID
15 rows selected.
SQL> SELECT COUNT(*) FROM dba_objects WHERE status = 'INVALID';
COUNT(*)
----------
0
SQL>
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
A component stuck at LOADING right after patching usually means post-patch work is still pending. Running utlrp.sql recompiled everything with 0 objects in error and 0 errors during recompilation, after which all 15 components are VALID and the invalid-object count is zero, matching the baseline from Phase A.
Step 3: Restore the Service Layout
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl relocate service -s racprdb_rman_svc -d racprdb -i racprdb1 -t racprdb2
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
The reboots moved the rman service to Node 1. Relocating it back puts every service on its preferred instance again, the same layout that existed before patching.
Step 4: Check Listener and SCAN
[oracle@racprd1 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node racprd2
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node racprd1
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node racprd1
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): racprd2,racprd1
[oracle@racprd1 ~]$
The output is identical to the baseline: three SCAN VIPs across both nodes and the listener on both.
Step 5: Check Cluster Resources
[root@racprd1 ~]# cd /u01/app/grid/19c/grid_2/bin/
[root@racprd1 bin]#
[root@racprd1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@racprd1 bin]#
[root@racprd1 bin]# ./crsctl check has
CRS-4638: Oracle High Availability Services is online
[root@racprd1 bin]#
[root@racprd1 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.chad
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.net1.network
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.ons
ONLINE ONLINE racprd1 STABLE
ONLINE ONLINE racprd2 STABLE
ora.proxy_advm
OFFLINE OFFLINE racprd1 STABLE
OFFLINE OFFLINE racprd2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 ONLINE OFFLINE STABLE
ora.CRS.dg(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.FRA.dg(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE racprd2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE racprd1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE racprd1 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE racprd1 Started,STABLE
2 ONLINE ONLINE racprd2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE racprd1 STABLE
ora.racprd1.vip
1 ONLINE ONLINE racprd1 STABLE
ora.racprd2.vip
1 ONLINE ONLINE racprd2 STABLE
ora.racprdb.db
1 ONLINE ONLINE racprd1 Open,HOME=/u01/app/o
racle/19c/db_2,STABL
E
2 ONLINE ONLINE racprd2 Open,HOME=/u01/app/o
racle/19c/db_2,STABL
E
ora.racprdb.racprdb_batch_svc.svc
2 ONLINE ONLINE racprd1 STABLE
ora.racprdb.racprdb_rman_svc.svc
2 ONLINE ONLINE racprd2 STABLE
ora.racprdb.racprdb_svc.svc
1 ONLINE ONLINE racprd1 STABLE
2 ONLINE ONLINE racprd2 STABLE
ora.scan1.vip
1 ONLINE ONLINE racprd2 STABLE
ora.scan2.vip
1 ONLINE ONLINE racprd1 STABLE
ora.scan3.vip
1 ONLINE ONLINE racprd1 STABLE
--------------------------------------------------------------------------------
[root@racprd1 bin]#
Everything is online, and ora.racprdb.db now shows HOME=/u01/app/oracle/19c/db_2 on both nodes, which is the clearest single confirmation that the database runs from the new home. Compared with the baseline, ora.qosmserver no longer appears in the list, which matches the CRS-2613 message seen in the Node 2 root script. Check this against the My Oracle Support patch notes if you rely on QoS Management.
Step 6: Verify the Final Patch Inventory
Node 1:
[oracle@racprd1 ~]$ env|grep ORA
ORACLE_SID=racprdb1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/19c/db_2
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ cd /u01/app/oracle/19c/db_2/OPatch
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lspatches
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 39915792 : applied on Sun Oct 25 09:24:31 IST 2026
Patch 29585399 : applied on Thu Apr 18 12:51:33 IST 2019
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ cd
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ . .grid_env
[oracle@racprd1 ~]$ env|grep ORA
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/grid
ORACLE_HOME=/u01/app/grid/19c/grid_2
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ cd /u01/app/grid/19c/grid_2/OPatch
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lspatches
39922802;ACFS RELEASE UPDATE 19.33.0.0.0 (39922802)
39922783;OCW RELEASE UPDATE 19.33.0.0.0 (39922783)
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
39107855;TOMCAT RELEASE UPDATE 19.0.0.0.0 (39107855)
39107825;DBWLM RELEASE UPDATE 19.0.0.0.0 (39107825)
OPatch succeeded.
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 39922802 : applied on Fri Oct 23 20:20:46 IST 2026
Patch 39922783 : applied on Fri Oct 23 20:18:51 IST 2026
Patch 39915792 : applied on Fri Oct 23 19:57:30 IST 2026
Patch 39107855 : applied on Fri Oct 23 19:45:35 IST 2026
Patch 39107825 : applied on Fri Oct 23 19:45:11 IST 2026
[oracle@racprd1 OPatch]$
[oracle@racprd1 OPatch]$
Node 2:
[oracle@racprd2 ~]$ env|grep ORA
ORACLE_SID=racprdb2
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/19c/db_2
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ cd /u01/app/oracle/19c/db_2/OPatch
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lspatches
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
OPatch succeeded.
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lsinventory|grep -i applied
Patch 39915792 : applied on Sun Oct 25 09:24:31 IST 2026
Patch 29585399 : applied on Thu Apr 18 12:51:33 IST 2019
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ cd
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ . .grid_env
[oracle@racprd2 ~]$ env|grep ORA
ORACLE_SID=+ASM2
ORACLE_BASE=/u01/grid
ORACLE_HOME=/u01/app/grid/19c/grid_2
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ cd /u01/app/grid/19c/grid_2/OPatch
[oracle@racprd2 OPatch]$
[oracle@racprd2 OPatch]$ ./opatch lspatches
39922802;ACFS RELEASE UPDATE 19.33.0.0.0 (39922802)
39922783;OCW RELEASE UPDATE 19.33.0.0.0 (39922783)
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
39107855;TOMCAT RELEASE UPDATE 19.0.0.0.0 (39107855)
39107825;DBWLM RELEASE UPDATE 19.0.0.0.0 (39107825)
OPatch succeeded.
[oracle@racprd2 OPatch]$
On both nodes, the Database home db_2 and the Grid home grid_2 show the 19.33 patches, and the environment variables for both homes point at the new paths. The old homes grid_1 and db_1 are still on disk, unchanged.
Troubleshooting: Database Home Installation Fails While Linking
During the database home installation (Phase D, Step 5), the first attempt stopped with a fatal error from the linking stage. The installer was building the Pro*C precompiler tools in the new home and could not finish linking them:
[FATAL] Error in invoking target 'links proc gen_pcscfg procob' of makefile '/u01/app/oracle/19c/db_2/precomp/lib/ins_precomp.mk'. See '/u01/app/oraInventory/logs/InstallActions2026-10-25_06-59-09AM/installActions2026-10-25_06-59-09AM.log' for details.
The message names the makefile that failed (ins_precomp.mk) and the installer log that holds the actual linker errors, so that log is the first place to look. Here the fix was to add the missing -lnnz19 library, Oracle's network security library, to the link lines in two makefiles.
Step 1: Clean Up the Home and Upgrade OPatch Again
A failed install leaves a partly built home, so remove the contents of db_2 and extract the base image again. Extracting the base image brings back the old OPatch 12.2.0.1.17, so upgrade it again as in Phase D, Step 3, before the installer applies the RU.
Step 2: Add the Missing Library to the Makefiles
Each makefile is backed up with a _bkp copy first, so the original can be restored. The first change puts -lnnz19 at the start of PROCLDLIBS, the library list used to link the precompiler tools:
[oracle@racprd1 db_2]$ cd /u01/app/oracle/19c/db_2/precomp/lib
[oracle@racprd1 lib]$ ls ins_precomp*
ins_precomp.mk
[oracle@racprd1 lib]$ cp ins_precomp.mk ins_precomp.mk_bkp
[oracle@racprd1 lib]$ sed -i 's/PROCLDLIBS=/PROCLDLIBS=-lnnz19 /g' ins_precomp.mk
[oracle@racprd1 lib]$
The second change appends -lnnz19 to LINKTTLIBS in env_rdbms.mk, the library list used by the link rules in that file:
[oracle@racprd1 lib]$ cd /u01/app/oracle/19c/db_2/rdbms/lib
[oracle@racprd1 lib]$
[oracle@racprd1 lib]$ ls env_rdbms*
env_rdbms.mk
[oracle@racprd1 lib]$
[oracle@racprd1 lib]$ cp env_rdbms.mk env_rdbms.mk_bkp
[oracle@racprd1 lib]$ sed -i 's/LINKTTLIBS=$(LLIBCLNTSH) $(ORACLETTLIBS) $(LINKLDLIBS)/LINKTTLIBS=$(LLIBCLNTSH) $(ORACLETTLIBS) $(LINKLDLIBS) -lnnz19/g' env_rdbms.mk
[oracle@racprd1 lib]$
Step 3: Re-run the Installer
Run the same runInstaller command from Phase D, Step 5. With the library on the link lines, the link step completes and the installation continues to the root.sh stage as before. These edits change generated makefiles, so treat them as a workaround for this build and check My Oracle Support for your platform before using them on a production system.
Rollback to the Old Homes
If any error occurs during testing and you want to roll back, use the steps below. Because grid_1 and db_1 are never modified, rolling back means switching to them again, one node at a time.
Database Home:
[oracle@racprd1 ~]$ $ORACLE_HOME/bin/srvctl modify database -d racprdb -oraclehome /u01/app/oracle/19c/db_1
Grid Home (run on each node, with clusterNodes set to that node):
[oracle@racprd1 grid_1]$ ./gridSetup.sh -silent -switchGridHome oracle.install.option=CRS_SWONLY ORACLE_HOME=/u01/app/grid/19c/grid_1 oracle.install.crs.config.clusterNodes=racprd1 oracle.install.crs.rootconfig.executeRootScript=false
[root@racprd1 ~]# /u01/app/grid/19c/grid_1/root.sh
Run the database command with ORACLE_HOME set to db_1, then restart each instance so it starts from the old home. The Grid switch on a node restarts its stack and its instance. If datapatch was already run, check the SQL patch registry against the older binaries with datapatch -verbose from db_1 after both nodes are back, and confirm the exact procedure in the My Oracle Support patch notes.
Closing Thoughts
The main advantage of this method is that the running homes are never modified. All installation and patching work happens in new directories while the cluster serves traffic, and the only outage per node is the one restart during its switch. Keep grid_1 and db_1 in place until the system has run cleanly on the new homes for a while, because they are the fallback. The two things that need attention afterward are the registry components after datapatch and the service placement after each restart.
Service Commands Used in This Post
The srvctl service commands were used throughout this post to relocate and stop services around each restart, while the instance restarts brought the services back up with their instances. A detailed article on managing services with srvctl, together with the RAC architecture behind them, will be published in the next post.
Thank you for reading!
I hope this content has been helpful to you. Your feedback and suggestions are always welcome, so feel free to leave a comment or reach out with any queries.
Abhishek Shrivastava
Oracle DBA with hands-on experience managing production Data Guard, RAC, GoldenGate, and APEX environments. I write practical, tested installation and troubleshooting guides based on real deployment work.
Comments
Post a Comment