Skip to main content

Oracle RAC Architecture, Processes and Services Guide

Oracle RAC Architecture, Processes and Services Guide

Introduction

Oracle Real Application Clusters lets more than one database instance mount and open the same physical database at the same time, with each instance running on its own server node. The instances share the underlying datafiles, control files, and redo threads through shared storage, while each node runs its own memory structures and background processes. This post lays out the architecture, the key differences from a standalone database, the processes that keep a RAC cluster running, what is shared versus what is node-specific, and the listener and service layer that routes client connections, including the practical srvctl commands for managing services day to day.

Quick Glossary

A few acronyms come up repeatedly below. Worth skimming once before the rest of the post, especially if RAC is new territory.

TermMeaning
GIGrid Infrastructure, the combined Clusterware and ASM software stack underneath the database
ASMAutomatic Storage Management, the shared storage layer every node reads and writes through
OCROracle Cluster Registry, stores cluster resource configuration
CRSCluster Ready Services, the Clusterware component that starts, stops, and monitors cluster resources
VIPVirtual IP, a node-bound address that relocates on node failure so clients fail fast
SCANSingle Client Access Name, the cluster-wide DNS name clients connect to

Standalone Database vs RAC Database

A standalone database runs a single instance against a single set of datafiles on one server. RAC changes this relationship from one-to-one to many-to-one: multiple instances, multiple servers, one database.

AspectStandaloneRAC
InstancesOne instance, one serverTwo or more instances, one per node, against one database
StorageLocal or single-server attached storageShared storage (ASM) visible to every node
Memory (SGA/PGA)One SGA on one serverIndependent SGA and PGA per instance, not shared across nodes
InterconnectNot applicablePrivate network between nodes for cache fusion traffic
AvailabilityInstance failure takes the database downSurviving instances keep the database available
Client connectionsConnect to one fixed hostRouted through SCAN and services across available instances
Software stackOracle Database home onlyGrid Infrastructure (Clusterware plus ASM) underneath the database home

The practical effect of this shift is that RAC trades the simplicity of one instance for horizontal scalability and instance-level fault tolerance, at the cost of needing a cluster-aware layer underneath the database: Grid Infrastructure.

How This Differs from Data Guard

RAC and Data Guard solve different problems and are often confused because both involve more than one server. RAC is multiple instances mounting the same database on shared storage, usually within one data center, protecting against instance or node failure. Data Guard is a separate copy of the database, usually on independent storage and often in a different data center, protecting against storage failure, site loss, or disasters that take the whole primary environment down. The two are complementary rather than substitutes: a production environment commonly runs RAC at the primary site for local availability and Data Guard to a standby site for disaster recovery.

RAC Architecture Overview

A RAC cluster is built in layers. At the bottom sits Grid Infrastructure, which bundles Clusterware and ASM. Clusterware manages cluster membership, node monitoring, and resource start and stop. ASM presents shared storage to every node as diskgroups, so a datafile written from one instance is immediately visible to the others.

Above Grid Infrastructure, each node runs its own RDBMS instance against the shared database. Instances communicate over a private interconnect using Cache Fusion, which lets one instance request a data block directly from another instance's buffer cache instead of reading it from disk again. This is what allows multiple instances to behave as one logical database rather than independent copies.

On top of the instance layer sits the connection layer: SCAN listeners, local listeners, VIPs, and services, which together decide which instance a given client session lands on and keep that routing current as instances start, stop, or move.

Layered Oracle RAC architecture with Grid Infrastructure, database instances, private interconnect, and shared storage

Figure 1: Oracle RAC architecture showing Grid Infrastructure and shared ASM storage, the database instance on each node, the private interconnect between nodes, and the client connection layer.

Admin-Managed vs Policy-Managed Databases

Grid Infrastructure supports two ways of deciding where instances run. An admin-managed database has each instance explicitly pinned to a named node, set at creation time and changed manually afterward, which is the simpler and far more common setup for small, fixed clusters like a 2-node environment. A policy-managed database instead assigns instances to a server pool, and Clusterware decides which physical nodes run instances for that pool based on pool size and node availability, which suits larger, more dynamic clusters where capacity is added or removed over time. The -policyManaged flag seen in dbca response files controls this choice, and it defaults to admin-managed unless set otherwise.

Important Processes in RAC

Alongside the usual single-instance background processes (PMON, SMON, DBWn, LGWR, CKPT), RAC adds processes whose entire purpose is coordinating state across instances.

Process Role
LMONGlobal Enqueue Service Monitor, manages cluster reconfiguration when instances join or leave
LMD0Global Enqueue Service Daemon, handles lock requests for global resources
LMSGlobal Cache Service process, ships data blocks between instances for Cache Fusion
LCK0Lock process, manages instance-level resource requests not handled by LMS
DIAGDiagnosability daemon, captures diagnostic data for the instance
ACMSAtomic Controlfile to Memory Service, ensures distributed SGA memory updates either complete on every instance or none
LMHBGlobal Cache Service Heartbeat Monitor, watches LMON, LMD, and LMS for hangs and triggers recovery if one stops responding
RMSnRAC Management Server, carries out cluster resource management tasks such as server pool changes on behalf of Clusterware
RSMNRemote Slave Monitor, manages resources used by parallel execution slaves running across instances
GTX0-jGlobal Transaction Process, coordinates distributed and XA transactions that span more than one instance
PINGResponse Time Agent, measures interconnect latency between instances and feeds that data to monitoring views
ASMBConnects the database instance to its local ASM instance to request space and receive extent maps

The common thread across these is coordination: every one of them exists to keep multiple independent instances behaving as a single consistent database, which is the core engineering problem RAC solves.

What Is Shared and What Is Individual per Node

Understanding RAC comes down to knowing which pieces are cluster-wide and which belong to a single node. Getting this distinction wrong is a common source of confusion when troubleshooting.

Shared across the clusterIndividual per node
Datafiles, control files, and redo log files (via ASM)SGA and PGA memory for that node's instance
ASM diskgroupsBackground process set (LMON, LMS, PMON, and so on) for that instance
OCR (Oracle Cluster Registry) and voting filesInstance-specific spfile parameters (the sid.parameter entries)
Diskgroup compatibility attributes (compatible.asm, compatible.rdbms)Local listener and node VIP
Cluster-wide spfile parameters (the *.parameter entries)Online redo threads (each instance owns its own thread, though the files live on shared storage)
SCAN name and SCAN listeners (float across nodes, not tied to one)Archived redo generated by that instance (unless archiving to shared storage)

The pattern worth remembering is that anything written to disk through ASM is cluster-wide by nature, because the storage itself is shared, while anything living in a server's RAM, such as the SGA, or tied to a specific network identity, such as a local listener or VIP, is necessarily scoped to that one node.

Listeners in RAC

RAC introduces more than one kind of listener, and distinguishing them is essential for diagnosing connection problems.

Local Listener

Each node runs its own local listener, bound to that node's VIP, listening for connections directed specifically at that instance. It is the direct equivalent of the single listener on a standalone database, just scoped to one node.

SCAN Listener

The Single Client Access Name listener is cluster-wide rather than node-specific. Typically three SCAN listeners run across the cluster, resolved through a SCAN name that maps to multiple IP addresses in DNS. Clients connect to the SCAN name, not to individual node hostnames, and Clusterware routes the connection to an instance that is actually available. Because the SCAN listener is not pinned to one node, it can relocate automatically if the node hosting it fails.

The reason three IPs are used, even on a 2-node cluster, is round-robin DNS resolution: each client connection attempt gets handed a different IP from the set, which spreads initial connection load across the available SCAN listeners before the session is even routed to an instance. A DNS lookup against the SCAN name shows this directly.

racprd1:
nslookup racprd-scan.oraeasy.com

Name:   racprd-scan.oraeasy.com
Address: 192.168.80.81
Address: 192.168.80.82
Address: 192.168.80.83

Three separate addresses for one name is expected and correct, not a misconfiguration. If a lookup returns only one address, or the addresses do not round-robin across repeated queries, that usually points to a DNS setup issue rather than a Clusterware one.

Node VIP

The Virtual IP is bound to a node but is not the same as the node's physical hostname IP. If a node goes down, its VIP relocates to a surviving node so clients get a fast connection refusal instead of a long TCP timeout, which is what drives faster failover at the connection layer.

In short: SCAN listeners provide a stable cluster-wide entry point, local listeners handle the final handoff to a specific instance, and VIPs make node failure visible to clients quickly instead of silently.

Oracle RAC listener architecture with SCAN listeners, local listeners, and node VIPs routing client connections to instances

Figure 2: Oracle RAC listener architecture showing how SCAN listeners, local listeners, and node VIPs route client connections to database instances.

Services in RAC

A service is a logical layer on top of the physical instances. Instead of clients connecting to an instance by name, applications connect to a service, and RAC decides which instance or instances actually service that connection. This indirection is what makes planned maintenance and load distribution possible without changing application connection strings.

Each service has a list of preferred instances, which normally run it, and optionally available instances, which take over the service if a preferred instance is down. Services can also be configured with load balancing goals so that Clusterware distributes new connections toward the least-loaded qualifying instance rather than always favoring the first one listed.

Connect-Time vs Runtime Load Balancing

These two terms get mixed up often, but they act at different moments. Connect-time load balancing decides which instance a brand new connection lands on, driven by the SCAN listener spreading new sessions across available instances for a service. Runtime load balancing happens afterward, adjusting how work is distributed among sessions that are already connected, which matters most for connection pools that hold a fixed set of long-lived sessions rather than opening a fresh connection per request. A service's load balancing goal setting controls the runtime side; the SCAN listener and the service's preferred and available instance list control the connect-time side.

What Happens When a Node Fails

The architecture and listener pieces above come together most clearly during an actual node failure, which is worth walking through as a single scenario rather than leaving as separate facts.

Three things happen in quick succession once Clusterware confirms the node is actually down rather than just slow: the failed node's VIP relocates to a surviving node so new connection attempts fail fast instead of hanging on a timeout, any service whose preferred instance was on the failed node fails over to its configured available instance, and the SCAN listener simply stops offering the dead node as a destination for new connections. Existing sessions that were mid-transaction on the failed instance are lost and must reconnect, but the database itself stays open throughout on the surviving instance, which is the entire point of running RAC instead of a standalone database.

Service Management: Practical Steps

The examples below use a two-node 19c cluster, with every command run as the oracle user on one of the nodes, and the environment is summarized here so the output in each step is easy to follow.

Cluster nameracprd
Nodesracprd1, racprd2
Database nameracprdb
Instancesracprdb1 (racprd1), racprdb2 (racprd2)
Oracle version19c

Step 1: Verify the Current Service Configuration

First, view the current service configuration using srvctl, then check which services the listener has registered.


[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1. Instance status: Open.
Instance racprdb2 is running on node racprd2. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status service -db racprdb
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl config service -db racprdb
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ lsnrctl status LISTENER

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 29-SEP-2026 19:04:12

Copyright (c) 1991, 2019, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                29-SEP-2026 18:32:13
Uptime                    0 days 0 hr. 31 min. 59 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/grid/19c/grid_1/network/admin/listener.ora
Listener Log File         /u01/grid/diag/tnslsnr/racprd1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.80.61)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.80.71)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_CRS" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_FRA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "racprdb" has 1 instance(s).
  Instance "racprdb1", status READY, has 1 handler(s) for this service...
Service "racprdbXDB" has 1 instance(s).
  Instance "racprdb1", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@racprd1 ~]$

The empty output from srvctl status service and srvctl config service confirms that no user-defined service exists yet. The only entries in the listener status are the automatic ones: racprdb and racprdbXDB from the database, and the +ASM services from the local ASM instance. This LISTENER runs from the Grid home, as the parameter file path shows, and it lists only the instances registered with Node 1.

Step 2: Create a Service

Create a service named racprdb_svc that should run on both nodes.


[oracle@racprd1 ~]$ srvctl add service -db racprdb -service racprdb_svc -preferred racprdb1,racprdb2
[oracle@racprd1 ~]$ srvctl config service -db racprdb -s racprdb_svc
Service name: racprdb_svc
Server pool:
Cardinality: 2
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: racprdb1,racprdb2
Available instances:
CSS critical: no
Service uses Java: false
[oracle@racprd1 ~]$
  

This only registers the service in the cluster registry; it does not start it. Both instances are preferred, so the cardinality is 2 and there is no available instance to fail over to. Two defaults are worth noting: Connection Load Balancing Goal: LONG suits long-lived connections, and TAF policy specification: NONE means no transparent failover is configured, so sessions on a failed instance must reconnect themselves.

Step 3: Start and Verify the Service


[oracle@racprd1 ~]$ srvctl start service -db racprdb -service racprdb_svc
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_svc. Instance status: Open.

The -v status now lists racprdb_svc as an online service on both instances, which confirms the start worked. Because the service has two preferred instances, one start command brought it up on both.

Step 4: Create Service racprdb_batch_svc with Node 1 as Preferred


[oracle@racprd1 ~]$ srvctl add service -db racprdb -service racprdb_batch_svc -preferred racprdb1 -available racprdb2
[oracle@racprd1 ~]$ srvctl config service -db racprdb -s racprdb_batch_svc
Service name: racprdb_batch_svc
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: racprdb1
Available instances: racprdb2
CSS critical: no
Service uses Java: false
[oracle@racprd1 ~]$

[oracle@racprd1 ~]$ srvctl start service -db racprdb -service racprdb_batch_svc
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_svc. Instance status: Open.

This output shows racprdb_batch_svc online only on Node 1, while racprdb_svc stays on both nodes. With one preferred and one available instance, the cardinality is 1, so the service runs only on racprdb1 and racprdb2 is held in reserve as the failover target. This keeps batch workload on one node during normal operation without giving up availability.

Step 5: Create Service racprdb_rman_svc with Node 2 as Preferred


[oracle@racprd1 ~]$ srvctl add service -db racprdb -service racprdb_rman_svc -preferred racprdb2 -available racprdb1
[oracle@racprd1 ~]$ srvctl config service -db racprdb -s racprdb_rman_svc
Service name: racprdb_rman_svc
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: racprdb2
Available instances: racprdb1
CSS critical: no
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl start service -db racprdb -service racprdb_rman_svc
[oracle@racprd1 ~]$ srvctl status service -db racprdb -service racprdb_rman_svc
Service racprdb_rman_svc is running on instance(s) racprdb2
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.

This is the mirror-image layout: racprdb_rman_svc runs on Node 2 with Node 1 as the failover target, so backup connections stay away from the node running batch work. Pointing the RMAN connect string at this service instead of a node hostname also means the backup still connects if Node 2 is down.

Step 6: View Service Running Status via SQL*Plus


SQL> SELECT INST_ID,
       NAME,
       NETWORK_NAME
     FROM GV$ACTIVE_SERVICES
     ORDER BY INST_ID, NAME;

 INST_ID NAME                           NETWORK_NAME
---------- ------------------------------ ------------------------------
         1 SYS$BACKGROUND
         1 SYS$USERS
         1 racprdb                        racprdb
         1 racprdbXDB                     racprdbXDB
         1 racprdb_batch_svc              racprdb_batch_svc
         1 racprdb_svc                    racprdb_svc
         2 SYS$BACKGROUND
         2 SYS$USERS
         2 racprdb                        racprdb
         2 racprdbXDB                     racprdbXDB
         2 racprdb_rman_svc               racprdb_rman_svc
         2 racprdb_svc                    racprdb_svc

12 rows selected.

The global view confirms the layout from inside the database: racprdb_svc is active on both instances, racprdb_batch_svc only on instance 1, and racprdb_rman_svc only on instance 2. SYS$BACKGROUND and SYS$USERS are internal default services that exist on every instance and have no network name, while racprdb and racprdbXDB are the automatic services already seen in the listener output.

Step 7: Relocate a Service


[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$ srvctl relocate service -db racprdb -service racprdb_batch_svc -oldinst racprdb1 -newinst racprdb2
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.

The relocate moves racprdb_batch_svc from Node 1 to Node 2, which is why Node 2 now lists three services and Node 1 only one. It changes where the service runs now, not its preferred and available configuration, so srvctl config service would still show racprdb1 as preferred. By default, sessions that are already connected are left alone; adding -force or a drain timeout changes that.

Step 8: Stop and Start a Service


[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$

At Instance level:

[oracle@racprd1 ~]$ srvctl stop service -s racprdb_svc -d racprdb -i racprdb1
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.

[oracle@racprd1 ~]$ srvctl start service -s racprdb_svc -d racprdb -i racprdb1
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.

At Database level:

[oracle@racprd1 ~]$ srvctl stop service -s racprdb_batch_svc -d racprdb
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl start service -s racprdb_batch_svc -d racprdb
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$

Adding -i racprdb1 stops racprdb_svc on that instance only, so it keeps running on Node 2 while the instance and the batch service stay untouched. Without -i, as in the racprdb_batch_svc stop, the service stops/starts on instance(s) as per service configuration rule. In both cases the instances stay open, which makes this a practical way to drain a service before maintenance.

Step 9: If a Database Instance Crashes on Any Node


[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ ps -ef|grep pmon
oracle    3929     1  0 17:27 ?        00:00:00 asm_pmon_+ASM2
oracle    5801     1  0 17:28 ?        00:00:00 ora_pmon_racprdb2
oracle   16765 10200  0 17:39 pts/0    00:00:00 grep --color=auto pmon
[oracle@racprd2 ~]$

Kill the PMON PID:

[oracle@racprd2 ~]$ kill -9 5801
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ ps -ef|grep pmon
oracle    3929     1  0 17:27 ?        00:00:00 asm_pmon_+ASM2
oracle   17024 10200  0 17:39 pts/0    00:00:00 grep --color=auto pmon
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is not running on node racprd2
[oracle@racprd2 ~]$

Killing the PMON process crashes the instance, since an instance cannot run without it, so use this only on a test cluster. The status check right afterward shows racprdb2 down and racprdb_rman_svc already running on Node 1, its available instance, while racprdb_svc keeps serving from the surviving instance. Clusterware normally restarts the failed instance on its own, so re-running the status command a minute later should show it open again.

Step 10: If a Node Crashes


[root@racprd1 ~]# init 0
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.

Remote side unexpectedly closed network connection

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Session stopped
    - Press  to exit tab
    - Press R to restart session
    - Press S to save terminal output to file


[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is not running on node racprd1
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.

After Node 1 Start:

[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.

init 0 shuts the node down cleanly, which is a gentler test than a real crash such as powering off the VM, but the cluster response is the same: Node 2 now carries all three services. After Node 1 returns, only racprdb_svc comes back to it, while racprdb_batch_svc stays on Node 2 even though Node 1 is its preferred instance. Services do not fail back on their own, so use srvctl relocate service to move it back.

srvctl Cheat Sheet

Services are not the only resource srvctl manages day to day. The commands below follow the same status and start and stop pattern for the database, listener, and ASM layers, so once the service commands above are familiar, these extend naturally.

Resource Status Start Stop
Databasesrvctl status database -db racprdbsrvctl start database -db racprdbsrvctl stop database -db racprdb
Instancesrvctl status instance -db racprdb -instance racprdb1srvctl start instance -db racprdb -instance racprdb1srvctl stop instance -db racprdb -instance racprdb1
Listenersrvctl status listenersrvctl start listenersrvctl stop listener
SCAN listenersrvctl status scan_listenersrvctl start scan_listenersrvctl stop scan_listener
ASMsrvctl status asmsrvctl start asmsrvctl stop asm
VIPsrvctl status vip -node racprd1Managed automatically by Clusterware, not normally started or stopped by hand

Main srvctl Operations

Operation Purpose
addRegister a resource
configDisplay resource configuration
statusCheck resource status
startStart a resource
stopStop a resource
modifyChange resource configuration
removeRemove resource configuration
enableEnable Clusterware management
disableDisable Clusterware management
relocateRelocate a supported resource
updateUpdate supported resource settings
getenvDisplay configured environment variables where supported
setenvSet environment variables where supported
unsetenvRemove configured environment variables where supported

Common srvctl Command Options

Option Meaning Example
-db / -dDatabase unique name-db racprdb
-instance / -iInstance name-instance racprdb1
-node / -nCluster node name-node racprd1
-serviceService name-service appsvc
-oraclehomeOracle Database home path-oraclehome /u01/app/oracle/19c/db_1
-spfileSPFILE path-spfile +DATA/.../spfile.ora
-roleDatabase or service role-role PRIMARY
-startoptionDatabase startup option-startoption MOUNT
-stopoptionDatabase shutdown option-stopoption IMMEDIATE
-policyResource management policy-policy AUTOMATIC
-preferredPreferred service instances-preferred racprdb1
-availableAvailable service instances-available racprdb2
-forceForce an operation where supported-force
-verboseDisplay detailed output where supported-verbose
-helpDisplay command helpsrvctl modify database -help

Frequently Asked Questions

Does RAC require shared storage?

Yes. ASM, or a certified cluster file system, is what makes the same datafiles visible and writable from every node at once, which is the foundation the rest of RAC depends on.

Can RAC nodes run different OS patch levels?

Briefly during a rolling patch window, yes, but it should not be left that way. Clusterware supports mixed patch levels temporarily so one node can be patched while the other stays available, with the expectation that both converge to the same level once the rolling operation finishes.

How many SCAN IPs does a cluster need?

Three, regardless of how many nodes are in the cluster, including a 2-node setup. This is a fixed Oracle recommendation, not something that scales with node count.

Does stopping a service stop the instance?

No. A service is a logical entry point layered on top of an instance. Stopping it closes that entry point only; the instance and any other services running on it continue unaffected.

What is the difference between a local listener and the SCAN listener?

The local listener is bound to one node and handles the final connection to that node's instance. The SCAN listener is cluster-wide, is the address clients actually connect to first, and hands the connection off to a local listener afterward.

Closing Thoughts

RAC architecture is ultimately about separating what must be shared, the database itself, from what must stay independent, each node's memory and processes, and then layering listeners and services on top so client connections can be routed and relocated without touching the application. Once that shared versus individual distinction is clear, the rest of RAC administration, including service management, becomes a matter of applying a small, consistent set of srvctl commands.



Thank you for reading!

I hope this content has been helpful to you. Your feedback and suggestions are always welcome, so feel free to leave a comment or reach out with any queries.

Abhishek Shrivastava

Oracle DBA with hands-on experience managing production Data Guard, RAC, GoldenGate, and APEX environments. I write practical, tested installation and troubleshooting guides based on real deployment work.

📧 Email: oraeasyy@gmail.com
🌐 Website: www.oraeasy.com

Follow Us

Comments

↑