Introduction
Oracle Real Application Clusters lets more than one database instance mount and open the same physical database at the same time, with each instance running on its own server node. The instances share the underlying datafiles, control files, and redo threads through shared storage, while each node runs its own memory structures and background processes. This post lays out the architecture, the key differences from a standalone database, the processes that keep a RAC cluster running, what is shared versus what is node-specific, and the listener and service layer that routes client connections, including the practical srvctl commands for managing services day to day.
Quick Glossary
A few acronyms come up repeatedly below. Worth skimming once before the rest of the post, especially if RAC is new territory.
| Term | Meaning |
|---|---|
| GI | Grid Infrastructure, the combined Clusterware and ASM software stack underneath the database |
| ASM | Automatic Storage Management, the shared storage layer every node reads and writes through |
| OCR | Oracle Cluster Registry, stores cluster resource configuration |
| CRS | Cluster Ready Services, the Clusterware component that starts, stops, and monitors cluster resources |
| VIP | Virtual IP, a node-bound address that relocates on node failure so clients fail fast |
| SCAN | Single Client Access Name, the cluster-wide DNS name clients connect to |
Standalone Database vs RAC Database
A standalone database runs a single instance against a single set of datafiles on one server. RAC changes this relationship from one-to-one to many-to-one: multiple instances, multiple servers, one database.
| Aspect | Standalone | RAC |
|---|---|---|
| Instances | One instance, one server | Two or more instances, one per node, against one database |
| Storage | Local or single-server attached storage | Shared storage (ASM) visible to every node |
| Memory (SGA/PGA) | One SGA on one server | Independent SGA and PGA per instance, not shared across nodes |
| Interconnect | Not applicable | Private network between nodes for cache fusion traffic |
| Availability | Instance failure takes the database down | Surviving instances keep the database available |
| Client connections | Connect to one fixed host | Routed through SCAN and services across available instances |
| Software stack | Oracle Database home only | Grid Infrastructure (Clusterware plus ASM) underneath the database home |
The practical effect of this shift is that RAC trades the simplicity of one instance for horizontal scalability and instance-level fault tolerance, at the cost of needing a cluster-aware layer underneath the database: Grid Infrastructure.
How This Differs from Data Guard
RAC and Data Guard solve different problems and are often confused because both involve more than one server. RAC is multiple instances mounting the same database on shared storage, usually within one data center, protecting against instance or node failure. Data Guard is a separate copy of the database, usually on independent storage and often in a different data center, protecting against storage failure, site loss, or disasters that take the whole primary environment down. The two are complementary rather than substitutes: a production environment commonly runs RAC at the primary site for local availability and Data Guard to a standby site for disaster recovery.
RAC Architecture Overview
A RAC cluster is built in layers. At the bottom sits Grid Infrastructure, which bundles Clusterware and ASM. Clusterware manages cluster membership, node monitoring, and resource start and stop. ASM presents shared storage to every node as diskgroups, so a datafile written from one instance is immediately visible to the others.
Above Grid Infrastructure, each node runs its own RDBMS instance against the shared database. Instances communicate over a private interconnect using Cache Fusion, which lets one instance request a data block directly from another instance's buffer cache instead of reading it from disk again. This is what allows multiple instances to behave as one logical database rather than independent copies.
On top of the instance layer sits the connection layer: SCAN listeners, local listeners, VIPs, and services, which together decide which instance a given client session lands on and keep that routing current as instances start, stop, or move.
Figure 1: Oracle RAC architecture showing Grid Infrastructure and shared ASM storage, the database instance on each node, the private interconnect between nodes, and the client connection layer.
Admin-Managed vs Policy-Managed Databases
Grid Infrastructure supports two ways of deciding where instances run. An admin-managed database has each instance explicitly pinned to a named node, set at creation time and changed manually afterward, which is the simpler and far more common setup for small, fixed clusters like a 2-node environment. A policy-managed database instead assigns instances to a server pool, and Clusterware decides which physical nodes run instances for that pool based on pool size and node availability, which suits larger, more dynamic clusters where capacity is added or removed over time. The -policyManaged flag seen in dbca response files controls this choice, and it defaults to admin-managed unless set otherwise.
Important Processes in RAC
Alongside the usual single-instance background processes (PMON, SMON, DBWn, LGWR, CKPT), RAC adds processes whose entire purpose is coordinating state across instances.
| Process | Role |
|---|---|
LMON | Global Enqueue Service Monitor, manages cluster reconfiguration when instances join or leave |
LMD0 | Global Enqueue Service Daemon, handles lock requests for global resources |
LMS | Global Cache Service process, ships data blocks between instances for Cache Fusion |
LCK0 | Lock process, manages instance-level resource requests not handled by LMS |
DIAG | Diagnosability daemon, captures diagnostic data for the instance |
ACMS | Atomic Controlfile to Memory Service, ensures distributed SGA memory updates either complete on every instance or none |
LMHB | Global Cache Service Heartbeat Monitor, watches LMON, LMD, and LMS for hangs and triggers recovery if one stops responding |
RMSn | RAC Management Server, carries out cluster resource management tasks such as server pool changes on behalf of Clusterware |
RSMN | Remote Slave Monitor, manages resources used by parallel execution slaves running across instances |
GTX0-j | Global Transaction Process, coordinates distributed and XA transactions that span more than one instance |
PING | Response Time Agent, measures interconnect latency between instances and feeds that data to monitoring views |
ASMB | Connects the database instance to its local ASM instance to request space and receive extent maps |
The common thread across these is coordination: every one of them exists to keep multiple independent instances behaving as a single consistent database, which is the core engineering problem RAC solves.
What Is Shared and What Is Individual per Node
Understanding RAC comes down to knowing which pieces are cluster-wide and which belong to a single node. Getting this distinction wrong is a common source of confusion when troubleshooting.
| Shared across the cluster | Individual per node |
|---|---|
| Datafiles, control files, and redo log files (via ASM) | SGA and PGA memory for that node's instance |
| ASM diskgroups | Background process set (LMON, LMS, PMON, and so on) for that instance |
| OCR (Oracle Cluster Registry) and voting files | Instance-specific spfile parameters (the sid.parameter entries) |
Diskgroup compatibility attributes (compatible.asm, compatible.rdbms) | Local listener and node VIP |
Cluster-wide spfile parameters (the *.parameter entries) | Online redo threads (each instance owns its own thread, though the files live on shared storage) |
| SCAN name and SCAN listeners (float across nodes, not tied to one) | Archived redo generated by that instance (unless archiving to shared storage) |
The pattern worth remembering is that anything written to disk through ASM is cluster-wide by nature, because the storage itself is shared, while anything living in a server's RAM, such as the SGA, or tied to a specific network identity, such as a local listener or VIP, is necessarily scoped to that one node.
Listeners in RAC
RAC introduces more than one kind of listener, and distinguishing them is essential for diagnosing connection problems.
Local Listener
Each node runs its own local listener, bound to that node's VIP, listening for connections directed specifically at that instance. It is the direct equivalent of the single listener on a standalone database, just scoped to one node.
SCAN Listener
The Single Client Access Name listener is cluster-wide rather than node-specific. Typically three SCAN listeners run across the cluster, resolved through a SCAN name that maps to multiple IP addresses in DNS. Clients connect to the SCAN name, not to individual node hostnames, and Clusterware routes the connection to an instance that is actually available. Because the SCAN listener is not pinned to one node, it can relocate automatically if the node hosting it fails.
The reason three IPs are used, even on a 2-node cluster, is round-robin DNS resolution: each client connection attempt gets handed a different IP from the set, which spreads initial connection load across the available SCAN listeners before the session is even routed to an instance. A DNS lookup against the SCAN name shows this directly.
racprd1:
nslookup racprd-scan.oraeasy.com
Name: racprd-scan.oraeasy.com
Address: 192.168.80.81
Address: 192.168.80.82
Address: 192.168.80.83
Three separate addresses for one name is expected and correct, not a misconfiguration. If a lookup returns only one address, or the addresses do not round-robin across repeated queries, that usually points to a DNS setup issue rather than a Clusterware one.
Node VIP
The Virtual IP is bound to a node but is not the same as the node's physical hostname IP. If a node goes down, its VIP relocates to a surviving node so clients get a fast connection refusal instead of a long TCP timeout, which is what drives faster failover at the connection layer.
In short: SCAN listeners provide a stable cluster-wide entry point, local listeners handle the final handoff to a specific instance, and VIPs make node failure visible to clients quickly instead of silently.
Figure 2: Oracle RAC listener architecture showing how SCAN listeners, local listeners, and node VIPs route client connections to database instances.
Services in RAC
A service is a logical layer on top of the physical instances. Instead of clients connecting to an instance by name, applications connect to a service, and RAC decides which instance or instances actually service that connection. This indirection is what makes planned maintenance and load distribution possible without changing application connection strings.
Each service has a list of preferred instances, which normally run it, and optionally available instances, which take over the service if a preferred instance is down. Services can also be configured with load balancing goals so that Clusterware distributes new connections toward the least-loaded qualifying instance rather than always favoring the first one listed.
Connect-Time vs Runtime Load Balancing
These two terms get mixed up often, but they act at different moments. Connect-time load balancing decides which instance a brand new connection lands on, driven by the SCAN listener spreading new sessions across available instances for a service. Runtime load balancing happens afterward, adjusting how work is distributed among sessions that are already connected, which matters most for connection pools that hold a fixed set of long-lived sessions rather than opening a fresh connection per request. A service's load balancing goal setting controls the runtime side; the SCAN listener and the service's preferred and available instance list control the connect-time side.
What Happens When a Node Fails
The architecture and listener pieces above come together most clearly during an actual node failure, which is worth walking through as a single scenario rather than leaving as separate facts.
Three things happen in quick succession once Clusterware confirms the node is actually down rather than just slow: the failed node's VIP relocates to a surviving node so new connection attempts fail fast instead of hanging on a timeout, any service whose preferred instance was on the failed node fails over to its configured available instance, and the SCAN listener simply stops offering the dead node as a destination for new connections. Existing sessions that were mid-transaction on the failed instance are lost and must reconnect, but the database itself stays open throughout on the surviving instance, which is the entire point of running RAC instead of a standalone database.
Service Management: Practical Steps
The examples below use a two-node 19c cluster, with every command run as the oracle user on one of the nodes, and the environment is summarized here so the output in each step is easy to follow.
| Cluster name | racprd |
|---|---|
| Nodes | racprd1, racprd2 |
| Database name | racprdb |
| Instances | racprdb1 (racprd1), racprdb2 (racprd2) |
| Oracle version | 19c |
Step 1: Verify the Current Service Configuration
First, view the current service configuration using srvctl, then check which services the listener has registered.
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1. Instance status: Open.
Instance racprdb2 is running on node racprd2. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status service -db racprdb
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl config service -db racprdb
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ lsnrctl status LISTENER
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 29-SEP-2026 19:04:12
Copyright (c) 1991, 2019, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date 29-SEP-2026 18:32:13
Uptime 0 days 0 hr. 31 min. 59 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/grid/19c/grid_1/network/admin/listener.ora
Listener Log File /u01/grid/diag/tnslsnr/racprd1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.80.61)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.80.71)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_CRS" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_FRA" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "racprdb" has 1 instance(s).
Instance "racprdb1", status READY, has 1 handler(s) for this service...
Service "racprdbXDB" has 1 instance(s).
Instance "racprdb1", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@racprd1 ~]$
The empty output from srvctl status service and srvctl config service confirms that no user-defined service exists yet. The only entries in the listener status are the automatic ones: racprdb and racprdbXDB from the database, and the +ASM services from the local ASM instance. This LISTENER runs from the Grid home, as the parameter file path shows, and it lists only the instances registered with Node 1.
Step 2: Create a Service
Create a service named racprdb_svc that should run on both nodes.
[oracle@racprd1 ~]$ srvctl add service -db racprdb -service racprdb_svc -preferred racprdb1,racprdb2
[oracle@racprd1 ~]$ srvctl config service -db racprdb -s racprdb_svc
Service name: racprdb_svc
Server pool:
Cardinality: 2
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: racprdb1,racprdb2
Available instances:
CSS critical: no
Service uses Java: false
[oracle@racprd1 ~]$
This only registers the service in the cluster registry; it does not start it. Both instances are preferred, so the cardinality is 2 and there is no available instance to fail over to. Two defaults are worth noting: Connection Load Balancing Goal: LONG suits long-lived connections, and TAF policy specification: NONE means no transparent failover is configured, so sessions on a failed instance must reconnect themselves.
Step 3: Start and Verify the Service
[oracle@racprd1 ~]$ srvctl start service -db racprdb -service racprdb_svc
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_svc. Instance status: Open.
The -v status now lists racprdb_svc as an online service on both instances, which confirms the start worked. Because the service has two preferred instances, one start command brought it up on both.
Step 4: Create Service racprdb_batch_svc with Node 1 as Preferred
[oracle@racprd1 ~]$ srvctl add service -db racprdb -service racprdb_batch_svc -preferred racprdb1 -available racprdb2
[oracle@racprd1 ~]$ srvctl config service -db racprdb -s racprdb_batch_svc
Service name: racprdb_batch_svc
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: racprdb1
Available instances: racprdb2
CSS critical: no
Service uses Java: false
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl start service -db racprdb -service racprdb_batch_svc
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_svc. Instance status: Open.
This output shows racprdb_batch_svc online only on Node 1, while racprdb_svc stays on both nodes. With one preferred and one available instance, the cardinality is 1, so the service runs only on racprdb1 and racprdb2 is held in reserve as the failover target. This keeps batch workload on one node during normal operation without giving up availability.
Step 5: Create Service racprdb_rman_svc with Node 2 as Preferred
[oracle@racprd1 ~]$ srvctl add service -db racprdb -service racprdb_rman_svc -preferred racprdb2 -available racprdb1
[oracle@racprd1 ~]$ srvctl config service -db racprdb -s racprdb_rman_svc
Service name: racprdb_rman_svc
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: racprdb2
Available instances: racprdb1
CSS critical: no
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl start service -db racprdb -service racprdb_rman_svc
[oracle@racprd1 ~]$ srvctl status service -db racprdb -service racprdb_rman_svc
Service racprdb_rman_svc is running on instance(s) racprdb2
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
This is the mirror-image layout: racprdb_rman_svc runs on Node 2 with Node 1 as the failover target, so backup connections stay away from the node running batch work. Pointing the RMAN connect string at this service instead of a node hostname also means the backup still connects if Node 2 is down.
Step 6: View Service Running Status via SQL*Plus
SQL> SELECT INST_ID,
NAME,
NETWORK_NAME
FROM GV$ACTIVE_SERVICES
ORDER BY INST_ID, NAME;
INST_ID NAME NETWORK_NAME
---------- ------------------------------ ------------------------------
1 SYS$BACKGROUND
1 SYS$USERS
1 racprdb racprdb
1 racprdbXDB racprdbXDB
1 racprdb_batch_svc racprdb_batch_svc
1 racprdb_svc racprdb_svc
2 SYS$BACKGROUND
2 SYS$USERS
2 racprdb racprdb
2 racprdbXDB racprdbXDB
2 racprdb_rman_svc racprdb_rman_svc
2 racprdb_svc racprdb_svc
12 rows selected.
The global view confirms the layout from inside the database: racprdb_svc is active on both instances, racprdb_batch_svc only on instance 1, and racprdb_rman_svc only on instance 2. SYS$BACKGROUND and SYS$USERS are internal default services that exist on every instance and have no network name, while racprdb and racprdbXDB are the automatic services already seen in the listener output.
Step 7: Relocate a Service
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$ srvctl relocate service -db racprdb -service racprdb_batch_svc -oldinst racprdb1 -newinst racprdb2
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
The relocate moves racprdb_batch_svc from Node 1 to Node 2, which is why Node 2 now lists three services and Node 1 only one. It changes where the service runs now, not its preferred and available configuration, so srvctl config service would still show racprdb1 as preferred. By default, sessions that are already connected are left alone; adding -force or a drain timeout changes that.
Step 8: Stop and Start a Service
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
At Instance level:
[oracle@racprd1 ~]$ srvctl stop service -s racprdb_svc -d racprdb -i racprdb1
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$ srvctl start service -s racprdb_svc -d racprdb -i racprdb1
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
At Database level:
[oracle@racprd1 ~]$ srvctl stop service -s racprdb_batch_svc -d racprdb
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
[oracle@racprd1 ~]$ srvctl start service -s racprdb_batch_svc -d racprdb
[oracle@racprd1 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd1 ~]$
Adding -i racprdb1 stops racprdb_svc on that instance only, so it keeps running on Node 2 while the instance and the batch service stay untouched. Without -i, as in the racprdb_batch_svc stop, the service stops/starts on instance(s) as per service configuration rule. In both cases the instances stay open, which makes this a practical way to drain a service before maintenance.
Step 9: If a Database Instance Crashes on Any Node
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_rman_svc,racprdb_svc. Instance status: Open.
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ ps -ef|grep pmon
oracle 3929 1 0 17:27 ? 00:00:00 asm_pmon_+ASM2
oracle 5801 1 0 17:28 ? 00:00:00 ora_pmon_racprdb2
oracle 16765 10200 0 17:39 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd2 ~]$
Kill the PMON PID:
[oracle@racprd2 ~]$ kill -9 5801
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ ps -ef|grep pmon
oracle 3929 1 0 17:27 ? 00:00:00 asm_pmon_+ASM2
oracle 17024 10200 0 17:39 pts/0 00:00:00 grep --color=auto pmon
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
Instance racprdb2 is not running on node racprd2
[oracle@racprd2 ~]$
Killing the PMON process crashes the instance, since an instance cannot run without it, so use this only on a test cluster. The status check right afterward shows racprdb2 down and racprdb_rman_svc already running on Node 1, its available instance, while racprdb_svc keeps serving from the surviving instance. Clusterware normally restarts the failed instance on its own, so re-running the status command a minute later should show it open again.
Step 10: If a Node Crashes
[root@racprd1 ~]# init 0
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.
Remote side unexpectedly closed network connection
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Session stopped
- Press to exit tab
- Press R to restart session
- Press S to save terminal output to file
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is not running on node racprd1
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
After Node 1 Start:
[oracle@racprd2 ~]$
[oracle@racprd2 ~]$ srvctl status database -d racprdb -v
Instance racprdb1 is running on node racprd1 with online services racprdb_svc. Instance status: Open.
Instance racprdb2 is running on node racprd2 with online services racprdb_batch_svc,racprdb_rman_svc,racprdb_svc. Instance status: Open.
init 0 shuts the node down cleanly, which is a gentler test than a real crash such as powering off the VM, but the cluster response is the same: Node 2 now carries all three services. After Node 1 returns, only racprdb_svc comes back to it, while racprdb_batch_svc stays on Node 2 even though Node 1 is its preferred instance. Services do not fail back on their own, so use srvctl relocate service to move it back.
srvctl Cheat Sheet
Services are not the only resource srvctl manages day to day. The commands below follow the same status and start and stop pattern for the database, listener, and ASM layers, so once the service commands above are familiar, these extend naturally.
| Resource | Status | Start | Stop |
|---|---|---|---|
| Database | srvctl status database -db racprdb | srvctl start database -db racprdb | srvctl stop database -db racprdb |
| Instance | srvctl status instance -db racprdb -instance racprdb1 | srvctl start instance -db racprdb -instance racprdb1 | srvctl stop instance -db racprdb -instance racprdb1 |
| Listener | srvctl status listener | srvctl start listener | srvctl stop listener |
| SCAN listener | srvctl status scan_listener | srvctl start scan_listener | srvctl stop scan_listener |
| ASM | srvctl status asm | srvctl start asm | srvctl stop asm |
| VIP | srvctl status vip -node racprd1 | Managed automatically by Clusterware, not normally started or stopped by hand | |
Main srvctl Operations
| Operation | Purpose |
|---|---|
add | Register a resource |
config | Display resource configuration |
status | Check resource status |
start | Start a resource |
stop | Stop a resource |
modify | Change resource configuration |
remove | Remove resource configuration |
enable | Enable Clusterware management |
disable | Disable Clusterware management |
relocate | Relocate a supported resource |
update | Update supported resource settings |
getenv | Display configured environment variables where supported |
setenv | Set environment variables where supported |
unsetenv | Remove configured environment variables where supported |
Common srvctl Command Options
| Option | Meaning | Example |
|---|---|---|
-db / -d | Database unique name | -db racprdb |
-instance / -i | Instance name | -instance racprdb1 |
-node / -n | Cluster node name | -node racprd1 |
-service | Service name | -service appsvc |
-oraclehome | Oracle Database home path | -oraclehome /u01/app/oracle/19c/db_1 |
-spfile | SPFILE path | -spfile +DATA/.../spfile.ora |
-role | Database or service role | -role PRIMARY |
-startoption | Database startup option | -startoption MOUNT |
-stopoption | Database shutdown option | -stopoption IMMEDIATE |
-policy | Resource management policy | -policy AUTOMATIC |
-preferred | Preferred service instances | -preferred racprdb1 |
-available | Available service instances | -available racprdb2 |
-force | Force an operation where supported | -force |
-verbose | Display detailed output where supported | -verbose |
-help | Display command help | srvctl modify database -help |
Frequently Asked Questions
Does RAC require shared storage?
Yes. ASM, or a certified cluster file system, is what makes the same datafiles visible and writable from every node at once, which is the foundation the rest of RAC depends on.
Can RAC nodes run different OS patch levels?
Briefly during a rolling patch window, yes, but it should not be left that way. Clusterware supports mixed patch levels temporarily so one node can be patched while the other stays available, with the expectation that both converge to the same level once the rolling operation finishes.
How many SCAN IPs does a cluster need?
Three, regardless of how many nodes are in the cluster, including a 2-node setup. This is a fixed Oracle recommendation, not something that scales with node count.
Does stopping a service stop the instance?
No. A service is a logical entry point layered on top of an instance. Stopping it closes that entry point only; the instance and any other services running on it continue unaffected.
What is the difference between a local listener and the SCAN listener?
The local listener is bound to one node and handles the final connection to that node's instance. The SCAN listener is cluster-wide, is the address clients actually connect to first, and hands the connection off to a local listener afterward.
Closing Thoughts
RAC architecture is ultimately about separating what must be shared, the database itself, from what must stay independent, each node's memory and processes, and then layering listeners and services on top so client connections can be routed and relocated without touching the application. Once that shared versus individual distinction is clear, the rest of RAC administration, including service management, becomes a matter of applying a small, consistent set of srvctl commands.
Thank you for reading!
I hope this content has been helpful to you. Your feedback and suggestions are always welcome, so feel free to leave a comment or reach out with any queries.
Abhishek Shrivastava
Oracle DBA with hands-on experience managing production Data Guard, RAC, GoldenGate, and APEX environments. I write practical, tested installation and troubleshooting guides based on real deployment work.



Comments
Post a Comment