How to remove a RAC node from and Cluster

The following steps the are required to successfully remove a Cluster Node from an existing Cluster:

1) Before starting the node removal process it is always recommended to take backup of OCR and databases.
-Login as root on racattack15u and verify the last backup of OCR
#$GRID_HOME/bin/ocrconfig -showbackup
-If required take a manual backup of OCR
#$GRID_HOME/bin/ocrconfig –manualbackup

2) Unpin the node to be removed if it is pinned
Run the below command from one of the node to check if the node to be removed is pinned or unpinned.
$ olsnodes -t -s 
racattack15u   Active  Unpinned
racattack18u   Active  Unpinned
(NOTE: Verified they are already unpinned)
-Run the below command if racattack18u is pinned
$crsctl unpin css -n racattack18u

3) Remove the database instance using DBCA
Note: This can also be done using a AIX GUI interface like CYGWIN.
Remove the database instance using DBCA:
-Invoke DBCA from racattack15u as oracle user
$$ORACLE_HOME/bin/dbca
Select the following in each screen to remove all the database instances  running in racattack18u
RAC database-->Instance management-->Delete Instance
Repeat to remove all the 3 Instances.

4) Verify the instance has been removed with the following commands on racattack18u.
$srvctl config database -d racdb1
$srvctl config database -d racdb2
$srvctl config database -d racdb3
$ps –ef|grep pmon
$crsctl stat res -t
NOTE: At this point database instances and its related services are removed from cluster. ASM and other clusterware compontents will be removed as part of GRID infrastructure removal.

5) Disable and Stop Listener on the Node to be removed.
oracle@racattack18u::/u01 $ ps -ef|grep tns (To get the running listener_alias name)oracle@racattack18u::/u01 $ lsnrctl status <Listener Alias>oracle@racattack18u::/u01 $ srvctl config listener -a
-Use the below commands to disable the listener(s) which are running from ORACLE_HOMEoracle@racattack18u::/u01 $ srvctl disable listener -l <Listener Name> -n racattack18uoracle@racattack18u::/u01 $ srvctl stop listener -l <Listener Name> -n racattack18u

6) Update Inventory on the Node to be deleted
-Verify the inventory in racattack18u before updating it.
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
-Run the following command on racattack18u to update the inventory
$ /u01/products/rdbms_11203/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/products/rdbms_11203 "CLUSTER_NODES={racattack18u}" -local
-Verify the inventory in racattack18u it should not have information about racattack15u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

7.) Remove the  RAC Database Software
The next step is to remove the RAC Database Software from the node to be deleted (racattack18u).
-Run below command as oracle user from racattack18u
$ cd $ORACLE_HOME/deinstall
$ ./deinstall -local
(-local option is to remoce ORACLE_HOME from local node)

8.) Update Inventory on the Remaining Nodes
-Verify the inventory in racattack15u before updating it.
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
-Run the following command on racattack15u to update the inventory
$ cd /u01/products/rdbms_11203/oui/bin
$ runInstaller -updateNodeList ORACLE_HOME=/u01/products/rdbms_11203 "CLUSTER_NODES={racattack15u}" -local
-Verify the inventory in racattack15u it should not have information about racattack18u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

9.) Remove Clusterware
The next step is to remove the clusterware from the node you want to delete.
Note: This must do as root.

-As root on racattack18u run the below command to deconfig clusterware
# cd /u01/products/grid_11203
# pwd
/u01/products/grid_11203
# cd crs/install
# pwd
/u01/products/grid_11203/crs/install
# hostname
racattack18u
# ./rootcrs.pl -deconfig -force
(At this point All the clusterware services including ASM are stopped)

10.) Delete node from Clusterware configuration
The next step is to remove the desired node from the clusterware configuration.  This is done with the following commands.
# cd /u01/products/grid_11203/bin
# olsnodes –t –s (For verification)
-As root on racattack15u run the below command to delete racattack18u from cluster
# crsctl delete node -n racattack18u
(racattack18u will be removed from clusterware and registry)
-Run the below command from racattack15u to verify the existing nodes
# olsnodes –t –s (For verification)
(Should only show racattack15u)

11.) Update Inventory on node to be removed as "Grid"
The next step is to update the inventory on the desired node to be removed as the "Grid" software owner (Oracle for this environment).
Note:  make sure that is done as the "Grid Owner" which is "Oracle" in this environment.
Verify the inventory in racattack18u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
-Run the following command on racattack18u to update the inventory
$ cd /u01/products/grid_11203/oui/bin
$ runInstaller -updateNodeList ORACLE_HOME=/u01/products/grid_11203 "CLUSTER_NODES={racattack18u}" -local
-Verify the inventory in racattack18u it should not have information about racattack15u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

12.) Remove the Clusterware Software from node to be removed.
The next step is to remove the clusterware software from the desired node (racattack18u). 
Note:  make sure that is done as the "Grid Owner". which is "Oracle" in this environment.
-Run below command as oracle user from racattack18u
$ cd /u01/products/grid_11203/deinstall
$ ./deinstall -local
(-local option is to remoce ORACLE_HOME from local node)
-In the end it asks us to run a script from a different session as root, run the script

13.) Update Inventory on remaining nodes for "Grid"
The next step is to update inventory on the remaining nodes (racattack15u) for "Grid". 
Note:  make sure that is done as the "Grid Owner"
Verify the inventory in racattack15u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
- Run the following command on racattack15u to update the inventory
$ cd /u01/products/grid_11203/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/products/grid_11203 "CLUSTER_NODES={racattack15u}"
-Verify the inventory in racattack15u it should not have information about racattack18u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

14.) Confirm the node has been successfully removed.
The next step is to verify that the node has been successfully removed.  To do this, execute the following command.
-Run the below command from racattack15u as oracle useroracle@racattack15u:racdb1:/u01 $ cluvfy stage -post nodedel -n racattack18uoracle@racattack15u:racdb1:/u01 $ crsctl stat res -t
NOTE: At this Stage racattack18u is clean without any Oracle software installations. racattack15u still remains as a single instance cluster database.