*How to Shutdown Oracle Real Application Clusters Database ?

Oracle RAC

Here, We will start with stopping resources from Enterprise Manager Grid Control to Oracle Clusterware or
CRS process and will end at stopping CRS services. Here I am taking an example of Two node RAC database to perform all operations.
Steps followed are as:

  1. Shutdown Oracle Home process accessing database.
  2. Shutdown RAC Database Instances on all nodes.
  3. Shutdown All ASM instances from all nodes.
  4. Shutdown Node applications running on nodes.
  5. Shut down the Oracle Cluster ware or CRS.
  6. Shutdown Oracle Home process accessing database:

There could be oracle and non Oracle application which access database, So first step is to stop all the applications or
DBA should inform application owner and he should stop all applications accessing Database.
DBA should also stop Oracle application like Enterprise Manager Grid Control and Database Listener which access
database for monitoring and database connections.

[[email protected] ~]$ emctl stop dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
https://database.example.com:5500/em/console/aboutApplication
Stopping Oracle Enterprise Manager 11g Database Control …
… Stopped.

[[email protected] bin]$ srvctl stop listener -n node1

[[email protected] bin]$ srvctl status listener -n node1
Listener LISTENER is enabled on node(s): node1
Listener LISTENER is not running on node(s): node1

  1. Shutdown RAC Database Instances on all nodes: Suppose DBA has two node RAC database,
    So he has to stop all instances from all DB nodes. Here, I am taking an example of two node RAC.
    First, I am checking on which server database is running and then stopping and verify for the same.

Syntax: srvctl stop database -d {databasename}

[[email protected] ~]$ srvctl status database -d oradb
Instance oradb1 is running on node node1
Instance oradb2 is running on node node2
[[email protected] ~]$ srvctl stop database -d oradb

[[email protected] ~]$ srvctl status database -d oradb
Instance oradb1 is not running on node node1
Instance oradb2 is not running on node node2

We just need to execute one command from any one of the server having database and it will stop all database instances on all servers.
If you have more than one database configured on Nodes, then Database Administrator has to execute this command for each database.

  1. Shutdown All ASM instances from all nodes: Next DBA has to shut down an ASM instance which are used to acess database,
    enter the following command, where node is the name of the node where the ASM instance is running

Syntax: srvctl stop asm -n {node}

[[email protected] oracle]# srvctl stop asm -n node1 -f

[[email protected] oracle]# srvctl stop asm -n node2 -f

[[email protected] oracle]# srvctl status asm -n node1
ASM is not running on node1

[[email protected] oracle]# srvctl status asm -n node2
ASM is not running on node2

Sometimes, Database administrator face some issues in stopping ASM instance, In that case use “-f” option to forcefully shutdown ASM instances.

  1. Shutdown Node applications running on nodes: To stop node applications running on a node, enter the following command,
    where node is the name of the node where the applications are running

[[email protected] oracle]# srvctl stop nodeapps -n node1 -f

[[email protected] oracle]# srvctl status nodeapps -n node1
VIP node1-vip is enabled
VIP node1-vip is running on node: node1
Network is enabled
Network is running on node: node1
GSD is disabled
GSD is not running on node: node1
ONS is enabled
ONS daemon is running on node: node1

Repeat same command for all nodes one by one. If you face any issue in stopping node applications use “-f” as force option to stop applications.

  1. Shut down the Oracle Clusterware or CRS: In this command all CRS related process will be stopped.
    This is the only command which needs to be executed by “root” user on all database nodes.

[[email protected] bin]# crsctl check cluster -all


node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


[[email protected] bin]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘node1’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘node1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘node1’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘node1’

CRS-2673: Attempting to stop ‘ora.node2.vip’ on ‘node1’

————————————————-

CRS-2677: Stop of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘node1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘node1’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘node1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.

[[email protected] bin]# crsctl check cluster -all

CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Check failed, or completed with errors.

DBA can see Now RAC is completely down. Now you can move ahead with your patching, maintenance activity.
Next let’s see how to start Oracle RAC cluster database.

[[email protected] check check has
[[email protected] crsctl stop has
[[email protected] check check has
[[email protected] crsctl stop has

[[email protected] ps -ef|grep d.bin
[[email protected] ps -ef|grep d.bin


How to Start Oracle Real Application Clusters Database ?


In Starting Real Application Clusters Database, We will also follow a set of sequence.
The order will be just opposite to Shutting down the Real Application Clusters Database.

  1. Start Oracle Clusterware or CRS.
  2. Start Node applications running on nodes.
  3. Start All ASM instances from all nodes.
  4. Start RAC Database Instances on all nodes.
  5. Start Oracle Home process accessing database.
  6. Start Oracle Clusterware or CRS: CRS starts automatically when you start or restart Server, but Here DBA has manually shutdown the CRS So,
    he has to start it manually. This is the only command which needs to be executed by “root” user.
    Database Administrator should execute this command on all nodes.
    [[email protected] check check has
    [[email protected] crsctl start has
    [[email protected] check check has
    [[email protected] crsctl start has

[[email protected] ps -ef|grep d.bin
[r[email protected] ps -ef|grep d.bin

[[email protected] bin]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started
[[email protected] bin]# crsctl check cluster -all


node1:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


Here, DBA can see “CRS-4639: Could not contact Oracle High Availability Services” or
“CRS-4535: Cannot communicate with Cluster Ready Services” messages. Wait 5 minutes and then again check with “crsctl check cluster -all” command.
This time Database administrator will get “CRS-4537: Cluster Ready Services is online”. If still same issue DBA can start ora.crsd process to resolve this issue.
Below is the command

[[email protected] bin]# crsctl start res ora.crsd -init

[[email protected] bin]# crsctl check cluster -all


node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


Now, Cluster is up and running. DBA can also use “ps -ef | grep d.bin” to check cluster status. OR ‘crs_stat -t’ as root user from crs_home/bin

  1. Start Node applications running on nodes: To start node applications running on a node, enter the following command,
    where node is the name of the node where the applications are running

[[email protected] bin]$ srvctl start nodeapps -n node1

[[email protected] bin]$ srvctl status nodeapps -n node1
VIP node1-vip is enabled
VIP node1-vip is running on node: node1
Network is enabled
Network is running on node: node1
GSD is disabled
GSD is not running on node: node1
ONS is enabled
ONS daemon is running on node: node1

DBA has to execute this command for each node to start Real Application Clusters Cluster database.

  1. Start All ASM instances from all nodes: Next DBA has to start all ASM instances which are used to access database, enter the following command,
    where node is the name of the node where the ASM instance has to start.

[[email protected] bin]$ srvctl start asm -n node1

[[email protected] bin]$ srvctl status asm -n node1
ASM is running on node1

DBA has to start ASM instance on all database nodes.

  1. Start RAC Database Instances on all nodes: Now, We will start database instances on database nodes to access data.

[[email protected] bin]$ srvctl start database -d oradb

[[email protected] bin]$ srvctl status database -d oradb
Instance oradb1 is running on node node1
Instance oradb2 is running on node node2

Now database is up and running on both DB nodes.

  1. Start Oracle Home process accessing database: let’s start listener and OEM to access database and allow users to connect to the database.

[[email protected] bin]$ srvctl start listener -n node1

[[email protected] bin]$ srvctl status listener -n node1
Listener LISTENER is enabled on node(s): node1
Listener LISTENER is running on node(s): node1

[[email protected] ~]$ emctl start dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
https://database.example.com:5500/em/console/aboutApplication
Starting Oracle Enterprise Manager 11g Database Control ……………………..

. started.

Logs are generated in directory /etc/oracle/oracle/database.example.com_orcl/sysman/log

Now Database is ready to user. Since, shutting down and starting up RAC database is a series of steps.