Monday, September 21, 2015

How to Upgrade from 11g to 12c on AIX

--*****************************
-- Stage Installation Files
--*****************************

--Verify Disk Space Requirements for Oracle Software on IBM AIX on POWER Systems (64-Bit)
Installation Type  Disk Space for Software Files
Enterprise Edition 9.2 GB

--Transfer files from Local Windows to Remote AIX
Open up PSFTP:
Start > All Programs > PuTTY > PSFTP

psftp> open oracledbserver
login as:
oracleoracle@oracledbserver's password:
Remote working directory is /u01

psftp> cd /johhenning/Oracle12cStage
Remote directory is now /johhenning/Oracle12cStage

psftp> pwd
Remote directory is /johhenning/Oracle12cStage

psftp> ls
Listing directory /johhenning/Oracle12cStage
drwxr-xr-x    8 oracle   oinstall       4096 Nov  8 02:55 database
drwxr-xr-x    8 oracle   oinstall       4096 Nov  8 03:00 grid
-rw-r--r--    1 oracle   oinstall 2062053595 Apr 16 13:26 aix.ppc64_12102_database_1of2.zip
-rw-r--r--    1 oracle   oinstall 1035130518 Apr 16 13:28 aix.ppc64_12102_database_2of2.zip
-rw-r--r--    1 oracle   oinstall 1481484377 Apr 16 13:32 aix.ppc64_12102_grid_1of2.zip
-rw-r--r--    1 oracle   oinstall 2040828798 Apr 16 13:36 aix.ppc64_12102_grid_2of2.zip

psftp> lpwd
Current local directory is C:\Program Files\PuTTY
psftp>

psftp> lcd C:\Users\johhenning\Downloads
New local directory is C:\Users\johhenning\Downloads
psftp>

psftp> mput aix.ppc64_12102_database_1of2.zip aix.ppc64_12102_database_2of2.zip aix.ppc64_12102_grid_1of2.zip aix.ppc64_12102_grid_2of2.zip
$ unzip aix.ppc64_12102_database_1of2.zip
$ unzip aix.ppc64_12102_database_2of2.zip
$ unzip aix.ppc64_12102_grid_1of2.zip
$ unzip aix.ppc64_12102_grid_2of2.zip


--******************************************************
-- 1. Checking Server Hardware and Memory Configuration
--******************************************************

1. To determine the physical RAM size, enter the following command:
oracle@henndue14:rman:/u01 $ /usr/sbin/lsattr -E -l sys0 -a realmem
realmem 4194304 Amount of usable physical memory in Kbytes False

2. To determine the available RAM and swap space, enter the following command:

# /usr/sbin/lsps -a
Page Space      Physical Volume   Volume Group  Size     %Used Active Auto  Type Chksum
hd6             hdisk0            rootvg        6144MB     6     yes    yes   lv   0

RAM    Swap Space
Between 1 GB and 2 GB  1.5 times the size of RAM
Between 2 GB and 16 GB  Equal to the size of RAM
More than 16 GB   16 GB


--*****************************************
-- 2. Server Storage Minimum Requirements
--*****************************************

1.  Verify Disk Space Requirements for Oracle Software on IBM AIX on POWER Systems (64-Bit)
Installation Type  Disk Space for Software Files
Enterprise Edition 9.2 GB


2.  Add the following mountpoint to henndue14. This mountpoint contains all of the installation software.

Filesystem       GB blocks  Free   %Used   Iused     %Iused  Mounted on
nim:/sw/sw61/ucms     120.00     39.15   68%    38117     1%      /sw61/ucms


3.  Disk space requirements

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/u01lv        20.00      6.43   68%    60891     4% /u01
/dev/u02lv        20.00     17.08   15%       21     1% /u02
/dev/u03lv        20.00     19.76    2%       21     1% /u03


4.  Sizing estimates for the oracle homes:

oracle@henndue14:rman:/u01 $ du -sg ./* | sort -rn | head -5
12 GB   ./OraHome_2
12 GB   ./OraHome_2_bk
10 GB   ./OraHome_12102

tar czf OraHome_2.tar.gz rdbms_11201/

5.  To determine the amount of space available in the /tmp directory, enter the following command (5 GB Recommended):

# df -m /tmp
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3        6144.00   6122.70    1%      148     1% /tmp


--***********************************
-- 3. Operating System Requirements
--***********************************

--*****************************************
-- 3.1 Identifying the Installed Packages
--*****************************************

--Check the maintenance level using the following command:

# oslevel -s

7100-03-04-1441
--To determine the supported kernel mode, enter a command similar to the following:
# getconf KERNEL_BITMODE
64
IBM AIX 7.1 Oerating System Filesets
bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools
xlC.aix61.rte.11.1.0.4 or later
xlC.rte.11.1.0.4 or later

--Verify that the required packages are installed.
oracle@henndue14:rman:/u01 $ lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libperfstat bos.perf.perfstat bos.perf.proctools xlC.aix61.rte xlC.rte
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.adt.base              7.1.3.15  COMMITTED  Base Application Development
                                                 Toolkit
  bos.adt.lib               7.1.2.15  COMMITTED  Base Application Development
                                                 Libraries
  bos.adt.libm               7.1.3.0  COMMITTED  Base Application Development
                                                 Math Library
  bos.perf.libperfstat      7.1.3.15  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat         7.1.3.15  COMMITTED  Performance Statistics
                                                 Interface
  bos.perf.proctools        7.1.3.15  COMMITTED  Proc Filesystem Tools
  xlC.aix61.rte             12.1.0.1  COMMITTED  IBM XL C++ Runtime for AIX 6.1
                                                 and 7.1
  xlC.rte                   12.1.0.1  COMMITTED  IBM XL C++ Runtime for AIX

Path: /etc/objrepos
  bos.adt.base              7.1.3.15  COMMITTED  Base Application Development
                                                 Toolkit
  bos.perf.libperfstat      7.1.3.15  COMMITTED  Performance Statistics Library
                                                 Interface
  bos.perf.perfstat         7.1.3.15  COMMITTED  Performance Statistics
                                                 Interface


--*********************************
-- 3.1 Check if APAR is installed
--*********************************

--To determine if an APAR is installed, enter a command similar to the following:
# /usr/sbin/instfix -i -k "IV16716 IV20880 IV21128 IV28319"
--If an APAR is not installed, then download it from the following website and install it:
http://www-933.ibm.com/support/fixcentral/
# /usr/sbin/instfix -i -k "IV16737 IV21116 IV21116 IV21235 IV28925 IV34869 IV35057 IV37790 IV39136 IV41380 IV41415 IV45072 IV45073"

--**********************************************************
-- 3.2 Creating Required Operating System Groups and Users
--**********************************************************

oracle@henndue14:ratcq:/u01/products $ id oracle
uid=501(oracle) gid=267(oinstall) groups=1(staff),264(dba)

--****************************************************************
-- 3.3 Configure Shell Limits and System Configuration Parameters
--****************************************************************

Shell Limit (As Shown in smit)  Recommended Value
Soft File Descriptors           at least 1024 KB
Hard File Descriptors           at least 65536 KB
Soft Maxproc                    at least 2047 KB
Hard Maxproc                    at least 16384 KB
Soft STACK size                 at least 10240 KB
Hard STACK size                 at least 10240 KB; at most 32768 KB
Soft FILE size                  -1 (Unlimited)
Soft CPU time                   -1 (Unlimited)
Soft DATA segment               -1 (Unlimited)
Soft Real Memory size           -1 (Unlimited)
Processes (per user)            -1 (Unlimited)

To view the current value specified for these shell limits, and to change them if necessary:

1. Enter the following command:

# smit chuser

2. In the User NAME field, enter the user name of the Oracle software owner, for example oracle.

3. Scroll down the list and verify that the value shown for the soft limits listed in the previous table is -1.

   If necessary, edit the existing value.

4. When you have finished making changes, press F10 to exit.


--************************************************
-- 3.4 Configure System Configuration Parameters
--************************************************

Parameter Recommended Value
maxuprocs 2048
ncargs  128

To verify that the maximum number of processes allowed per user is set to 2048 or greater, use the following steps:

1. Enter the following command:

# smit chgsys

2. Verify that the value shown for Maximum number of PROCESSES allowed per user is greater than or equal to 2048.

   If necessary, edit the existing value.

3. When you have finished making changes, press F10 to exit.


To verify that long commands can be executed from shell, use the following steps:

1. Enter the following command:

# smit chgsys

2. Verify that the value shown for ARG/ENV list size in 4K byte blocks is greater than or equal to 128.

   If necessary, edit the existing value.

3. When you have finished making changes, press F10 to exit.



--***************************************************
-- 3.5 Checking Asynchronous Input Output Processes
--***************************************************

Confirm the aio_maxreqs value using the following procedure:
# ioo -o aio_maxreqs
aio_maxreqs = 65536 (Recommended Value)

--Current output on henndue14
# ioo -o aio_maxreqs
aio_maxreqs = 131072



****************************************************************************************************
-- 3.6 Enable the "iocp" option, To enable IOCP, set IOCP to Available using the following procedure:
--****************************************************************************************************

Log in as root and run the following command:
# smitty iocp

Select Change / Show Characteristics of I/O Completion Ports.
Change configured state at system restart from Defined to Available.
Run the lsdev command to confirm the IOCP status is set to Available:

$ lsdev | grep iocp
iocp0 Available I/O Completion Ports


--************************************
-- 4. Install the 12c Oracle Software
--************************************


--**************************************************
-- 4.1 Creating Required Directories
--**************************************************

--Oracle Home
$ mkdir -p /u01/products/rdbms_12102


--**************************************************
-- 4.2 Running the rootpre.sh Script
--**************************************************

Switch user to root:
$ sudo /sbin/ksh
password:
#

Complete one of the following steps, depending on the location of the installation files:
# cd /sw61/ucms/Oracle/Oracle12cStage/database/
# ./rootpre.sh

./rootpre.sh output will be logged in /tmp/rootpre.out_15-05-27.09:14:25
Checking if group services should be configured....
Nothing to configure.

Exit from the root account:
# exit


--**************************************************
-- 4.3 Configuring Oracle Software Owner Environment
--**************************************************

--Start CYGWIN
startxwin
xhost +
export DISPLAY=10.10.10.10:0.0 (DISPLAY = The IP Address of your local PC not the server you're running the scirpt.)


--**************************************************
-- 4.4 Database Software Only Installation :
--**************************************************

--Install 12.1.0.2 version of 12c database binaries using the runInstaller command from the UNIX command line. This will populate the Oracle 12c home directory.  The existing 11.2.0.3 home directory is preserved.
$  cd /Oracle12cStage/database
$  ./runInstaller

(At the end of the installation it will prompt to run root.sh script under the New ORACLE_HOME directory as root user. Run it as root to finish the installation.
# ./root.sh
Performing root user operation.

The following environment variables are set as:
      ORACLE_OWNER= oracle
      ORACLE_HOME=  /u01/products/rdbms_12102

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: y
    Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: y
    Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

** Note at this change there are no modifications to the existing (11G) database software

--***********************************
-- Pre Database Upgrade Tasks
--***********************************

--***********************************************************
-- Using the Pre-Upgrade Information Tool (preupgrd.sql)
--***********************************************************

To run the Pre-Upgrade Information Tool on the source database:
1. Copy preupgrd.sql and utluppkg.sql from the rdbms/admin directory of the new Oracle home where you installed Oracle Database 12c to a directory that is accessible when you connect to your source database, which is the database to be upgraded. Preferably, this should be a temp directory.
   $ mkdir -p /u01/OraHome1_11203/rdbms/temp
   $ cd /u01/OraHome1_11203/rdbms/temp
   $ cp /u01/products/rdbms_12102/rdbms/admin/preupgrd.sql  .
   $ cp /u01/products/rdbms_12102/rdbms/admin/utluppkg.sql .

2. Log in to the system as the owner of the environment of the database being upgraded.


3. Start SQL*Plus and connect to the database to be upgraded using an account with DBA privileges:

   SQL> CONNECT / AS SYSDBA

4. (Recommended.) Run the Pre-Upgrade Information Tool (preupgrd.sql) for the following scenarios:

   a. Before you upgrade a non-CDB in the source Oracle Home.
   SQL> @$ORACLE_HOME/rdbms/temp/preupgrd.sql

5. View and read through the resulting generated fixup scripts and log file, which are located in $ORACLE_BASE/cfgtoollogs/db_unique_name/preupgrade if ORACLE_BASE is defined, or $ORACLE_HOME/cfgtoollogs/db_unique_name/preupgrade if ORACLE_BASE is not defined.

6. Review output and determine course of action before proceeding upgrading the database.

7. Gather Dictionary Statistics 24 hrs before running the DBUA as recommended by preupgrade script:
  
   $ sqlplus "/as sysdba"
   sql> EXEC DBMS_STATS.GATHER_DICTIONARY_STATS


--*********************************************
-- Upgrade the Database
--*********************************************

1. Take full RMAN backup and verify backup is successful. Take full system export of the RMAN database.
   For RMAN backup, use a script scheduled in the crontab for level0 backup.
   For Export use the below commands,
  
   $ export ORACLE_SID=rman
   $ . oraenv

   $ expdp system/xxxxx full=y directory=<DATA_PUMP_DIR> dumpfile=exp_rman_full_<date>.dmp logfile= exp_rman_full_<date>.log

2. Run the Pre-Upgrade Information Tool (preupgrd.sql) for the following scenarios:

   Before you upgrade a non-CDB in the source Oracle Home.
     
   $sqlplus ‘/as sysdba’
   SQL> @$ORACLE_HOME/rdbms/temp/preupgrd.sql


3. View and read through the resulting generated fixup scripts and log file, which are located in $ORACLE_BASE/cfgtoollogs/db_unique_name/preupgrade if ORACLE_BASE is defined, or $ORACLE_HOME/cfgtoollogs/db_unique_name/preupgrade if ORACLE_BASE is not defined.


4. After you have reviewed the scripts, Oracle recommends that you execute preupgrade_fixups.sql manually on the source database. The preupgrade_fixups.sql script will attempt to resolve issues reported by the preupgrade process.


5. Fix the flagged issues that require you to complete manual procedures. See "Pre-Upgrade Information Tool Warnings and Recommendations for Oracle Database" for more details about actions to take.


6. Run the Pre-Upgrade Information Tool as many times as needed for you to work through and resolve warnings.


7. Manually Remove the DB Control with emremove.sql script located in 12c ORACLE_HOME/rdbms/admin directory as recommended by preupgrade script.

   $ emctl stop dbconsole
   $ cd /u01/OraHome1_11203/rdbms/temp
   $ cp /u01/products/rdbms_12102/rdbms/admin/emremove.sql  .
   $ sqlplus ‘/as sysdba’

   SQL>SET ECHO ON;
   SQL>SET SERVEROUTPUT ON;
   SQL>@emremove.sql
   SQL>exit

   After emremove.sql completes, you must manually remove ORACLE_HOME/HOSTNAME_SID and ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_HOSTNAME_SID directories from your file system.
   $ rm -rf /u01/OraHome1_11203/henndue14_rman
   $ rm -rf /u01/OraHome1_11203/oc4j/j2ee/OC4J_DBConsole_henndue14_rman


8. Verify and if not being used than manually remove APEX.

$ sqlplus '/as sysdba'
  
Sql> select workspace, view_date, seconds_ago from apex_workspace_activity_log;
No rows returned
  
Sql>select workspace, workspace_id from apex_workspaces;
No rows with exponential values.
  
Sql>@$ORACLE_HOME/apex/apxremov.sql
APEX is removed with the above script.

Look for below users if any exists, remove them manually:
Sql> slect username from dba_users where username like '%APEX%';
Sql> slect username from dba_users where username like '%FLOWS%';

Now look for the below objects in the database and remove them manually if exists.
Sql> select owner, OBJECT_NAME, OBJECT_TYPE, status from dba_objects where object_name='HTMLDB_SYSTEM';
Sql> select owner, OBJECT_NAME, OBJECT_TYPE, status from dba_objects where object_name like 'WWV_%';


9. As recommended by preupgrade script, remove OLAP AMD component which is depreciated in 12c from the database.
   $ cd $ORACLE_HOME/olap/admin
   $ sqlplus '/as sysdba'
   Sql> @catnoamd.sql


10. Check for INVALID database components and objects in the Source database

Ensure that there are NO INVALID database components/objects in the source database prior to starting the upgrade
You can execute the following query to check the invalid database components/objects in the source database:
set pagesize500
set linesize 100
select substr(comp_name,1,40) comp_name, status, substr(version,1,10) version from dba_registry order by comp_name;

select substr(object_name,1,40) object_name,substr(owner,1,15) owner,object_type from dba_objects where status='INVALID' order by owner,object_type;
select owner,object_type,count(*) from dba_objects where status='INVALID' group by owner,object_type order by owner,object_type ;
If you find invalid objects and/or database components then try to VALIDATE the invalid objects and/or database components by executing the following steps:
Run $ORACLE_HOME/rdbms/admin/utlrp.sql to validate the invalid objects in the database. You can execute the utlrp.sql scripts multiple times to validate the invalid objects.

$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus "/ as sysdba"
SQL> @utlrp.sql


11. Verify That Materialized View Refreshes Have Completed Before Upgrading
    To determine if there are any materialized view refreshes still in progress:

    Run the following SQL query:

    SQL> SELECT o.name FROM sys.obj$ o, sys.user$ u, sys.sum$ s WHERE o.type# = 42 AND bitand(s.mflags, 8) =8;

12. Ensure That No Files Need Media Recovery Before Upgrading
    To get a list of files that require media recovery:
    Run the following statement:
    SQL> SELECT * FROM v$recover_file;

13. Ensure That No Files Are in Backup Mode Before Upgrading

    To get a list of files in backup mode:
    Run the following statement:
    SQL> SELECT * FROM v$backup WHERE status != 'NOT ACTIVE';

14. Resolve Outstanding Distributed Transactions Before Upgrading

    To resolve outstanding distributed transactions:
    1. Run the following statement:
    SQL> SELECT * FROM dba_2pc_pending;
    2. If the query in the previous step returns any rows, then run the following statements:
    SQL> SELECT local_tran_id FROM dba_2pc_pending;
    SQL> EXECUTE dbms_transaction.purge_lost_db_entry('');
    SQL> COMMIT;


15. Purge the Database Recycle Bin Before Upgrading

    To empty the database recycle bin, run the following command:
    SQL> PURGE DBA_RECYCLEBIN

16. Purge Audit records or preprocess them to decrease the amount of downtime during the upgrade


    SQL> truncate table sys.aud$;
    SQL> truncate table sys.fga_log$;

17. Perserve registered RMAN Datbases


Query the RMAN catalog to determine all registered databases.
SQL>
set pagesize 500
set linesize 500
column  RESETLOGS_CHANGE# format a19
select * from rman.rc_database
order by NAME;
SQL>

Or at the RMAN prompt enter the following command.
$ rman catalog rman/xxxxx
RMAN> list incarnation;


--************************************************
-- 6.1 Using DBUA to Upgrade the Database on UNIX
--************************************************

Run the upgrade assistant (DBUA) to upgrade the database to 12c.  At this point the database will be pointing to the new Oracle Home.

--************************************************
-- 7 Post-Upgrade Tasks for Oracle Database
--************************************************

All the post upgrade steps are performed by DBUA. In case DBUA displays the final component status in the Upgrade Results page.
Please check the final component status on the Upgrade Result page. Invalid components are marked with an X and the option to recompile is provided.

Run utlrp.sql to recompile the invalid objects as follows:

sql> connect / as sysdba

sql> @?/rdbms/admin/utlrp.sql

--*************************************************************************************************
-- 7.1 Set oratab and Scripts to Point to the New Oracle Location After Upgrading Oracle Database
--*************************************************************************************************

After you upgrade Oracle Database to the new release, you must ensure that your oratab file and any client scripts that set the value of ORACLE_HOME point to the new Oracle home that is created for the new Oracle Database 12c release. Although DBUA automatically points oratab to the new Oracle home, client scripts must be checked no matter which method you use to upgrade.

--************************************************************************
-- 7.2 Upgrade the Recovery Catalog After Upgrading Oracle Database
--************************************************************************

If you use a version of the recovery catalog schema that is older than that required by the RMAN client, then you must upgrade it. For complete information about upgrading the recovery catalog and the UPGRADE CATALOG command, see Oracle Database Backup and Recovery User's Guide.
1. Use SQL*Plus to connect to the recovery catalog database as the SYS user with SYSDBA privilege.

2. Run the dbmsrmansys.sql script to grant additional privileges that are required for the RECOVERY_CATALOG_OWNER role.

   SQL> @$ORACLE_HOME/rdbms/admin/dbmsrmansys.sql

3. Connect RMAN to the base recovery catalog, upgrade the base recovery catalog, and then exit RMAN.

   $ rman CATALOG
my_catalog_owner@catdb
   recovery catalog database Password:
   RMAN> UPGRADE CATALOG;
   RMAN> UPGRADE CATALOG;
   RMAN> EXIT;


--*******************************************
-- 7.3 Run the postupgrade_fixups.sql Script
--*******************************************

Although DBUA runs the postupgrade_fixups.sql script as part of completing the upgrade process, you can run it any time after upgrading. The postupgrade_fixups.sql script generates three categories of information for your upgraded database: general warnings, errors, and informational recommendations.

Run this script any time after completing an upgrade with DBUA or manually. If Oracle_Base is defined, then the generated scripts and log files are created in Oracle_Base/cfgtoollogs/ of the original database from which you ran the upgrade. If Oracle_Base is not defined, then the generated scripts and log files are created in ORACLE_HOME/cfgtoollogs/ of the database from which you ran the upgrade.

Set the system to spool results to a log file so you can read the output. Do not, however, spool to the admin directory. Run the script from the location of the database from which you ran the upgrade (not the new upgraded location):

SQL> SPOOL postupgrade.log

Turn off the spooling of script results to the log file.

SQL> SPOOL OFF

--******************************************************
-- 7.4 Gather Fixed Objects Statistics with DBMS_STATS
--******************************************************

A few days after upgrading Oracle Database, a best practice is to gather fixed objects statistics with the DBMS_STATS.GATHER_FIXED_OBJECTS_STATS PL/SQL procedure. This can have a positive impact on overall database performance. DBMS_STATS.GATHER_FIXED_OBJECTS_STATS also displays a recommendation to remove all hidden or underscore parameters and events from init.ora/spfile.

Because of the transient nature of the x$ tables, it is important that you gather fixed objects statistics when there is a representative workload on the system. This may not always be feasible on large systems due to additional resources needed to gather the statistics. If you cannot do this during peak load, then you should do it after the system has warmed up and the key types of fixed object tables have been populated.

To gather statistics for fixed objects, run the following PL/SQL procedure:

SQL> execute dbms_stats.gather_fixed_objects_stats;

--****************************************************************
-- 7.5 Take a full system export and backup of the RMAN database 
--****************************************************************

Make sure you perform a full backup of the production database.


--*******************************************************
-- 7.6 Configuring tnsnames.ora and Listener Parameters 
--*******************************************************

After performing a manual upgrade, you must adjust local_listener and remote_listener parameter references if they must be resolved in tnsnames.ora. DBUA handles changes to network naming and listeners during automatic upgrades, but during a manual upgrade, tnsnames.ora is not changed, nor are the listeners.

--*********************
-- 7.7 Smoke Test 
--*********************

Perform smoke test

--******************************
-- 7.8 11G Compatibility Mode 
--******************************

Set the COMPATIBLE initialization parameter to 12.1.0.0, enter the following statement:
SQL> ALTER SYSTEM SET COMPATIBLE = '12.1.0.0' SCOPE=SPFILE;
Shut down and restart the instance.


--**************************************************
-- Useful links and metalink articles.
--**************************************************
--Database Quick Installation Guide
http://docs.oracle.com/database/121/AXDQI/toc.htm
--Preparing to Upgrade Oracle Databasehttp://docs.oracle.com/database/121/UPGRD/preup.htm#BABHHIBG
--Using Flashback Database and Restore Pointshttp://docs.oracle.com/cd/E11882_01/backup.112/e10642/flashdb.htm
--Database Upgrade Guidehttp://docs.oracle.com/database/121/UPGRD/intro.htm#i1008722
--How to Upgrade to Oracle Grid Infrastructure 12c Release 1http://docs.oracle.com/database/121/CWLIN/procstop.htm#CEGJHHFJ
--Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure http://asanga-pradeep.blogspot.co.uk/2013/07/upgrading-from-11gr2-11203-to-12c-12101.html
--Oracle Support Document 1516557.1 (Complete Checklist for Upgrading to Oracle Database 12c Release 1 using DBUA)https://support.oracle.com/epmos/faces/DocumentDisplay?id=1516557.1
--Database Upgrade From 11202 to 12101 Using DBUA http://www.askmlabs.com/2014/02/database-upgrade-from-11202-to-12101.html
--Rac 3node upgrade from 11202 to 12101 http://www.askmlabs.com/2014/03/rac-3node-upgrade-from-11202-to-12101.html
--Database Upgrade From 11204 to 12101 - Manual Stepshttp://www.askmlabs.com/2014/03/database-upgrade-from-11204-to-12101.html

Monday, August 31, 2015

How to Automate a Datapump Export Process

#################################
# Create datapump directory
#################################

CREATE DIRECTORY datapump as '/u01/datapump/datapump_exports';
GRANT READ,WRITE ON DIRECTORY datapump TO SYSTEM;

###############################################
# Create shell script to pass the scp password
###############################################

#!/usr/bin/expect -f
# connect via scp
 spawn scp /u01/datapump/datapump_exports/exp_*.dmp* "oracle@testserver:/u01/datapump/"
 expect {
 -re ".*es.*o.*" {
 exp_send "yes\r"
 exp_continue
 }
 -re ".*sword.*" {
 exp_send "Password\r"
 }
 }
 interact

#############################################################
# Create shell script for the entire datapump export process
#############################################################

#!/bin/ksh
# First, we must set the environment . . . .
ORACLE_SID=$1
export ORACLE_SID
ORACLE_HOME=`cat /etc/oratab|grep ^$ORACLE_SID:|cut -f2 -d':'`
export ORACLE_HOME
PATH=$PATH:/u01/datapump/datapump_exports
DIR=/u01/datapump/datapump_exports
LOGDIR=/u01/datapump/datapump_exports
EXPLOGDIR=/u01/datapump/datapump_exports
export PATH DIR LOGDIR EXPLOGDIR
. oraenv

ADMIN=`cat /u01/datapump/datapump_exports/bkup_mail_list.lis`
EM_LIST='henndue14@gmail.com'
today=`date '+%m/%d/%y %H:%M:%S'`
HOSTNAME=`hostname`
expdate=`date '+%d%m%Y'`

user=`cat /u01/datapump/datapump_exports/.pwdapp |grep ^$ORACLE_SID:|cut -f2 -d':'`
LOGIN=`cat /u01/datapump/datapump_exports/.pwdapp |grep ^$ORACLE_SID:|cut -f3 -d':'`
TEMP_FILE=$LOGDIR/expdp_log.txt

#################################
# Removing 7 days old dump files
#################################

echo "Start dmp file cleanup $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

find $EXPLOGDIR/exp_*.dmp* -mtime +7 -exec rm {} \;

echo "Finished dmp file cleanup $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

#################################
# Alter undo_retention parameter
#################################

echo "Start UNDO_RETUENTION Update $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

$ORACLE_HOME/bin/sqlplus -s <<!
/ as sysdba

whenever sqlerror exit 2;

set echo off
set feedback off
spool $LOGDIR/undo.err

alter system set undo_retention=19169 scope=both;
commit;

spool off
!
undo_exit_status=$?

if [ $undo_exit_status -eq 0 ]
then
  echo "undo_retention successfully updated"
else
  mailx -s "Update undo_retention on Database $ORACLE_SID Failed!!! "henndue14@gmail.com < $LOGDIR/undo.err
  exit 1  
fi

echo "Finish UNDO_RETUENTION Update $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

#################################
# Export Schema
#################################

echo "Start datapump Export $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

expdp $LOGIN PARALLEL=6 dumpfile=exp_$(date +%Y%m%d)_%U directory=datapump logfile=exp_$(date +%Y%m%d).log schemas=hennco EXCLUDE=TABLE:\"IN \(\'TEST_TBL\'\)\" flashback_time=systimestamp

exp_exit_status=$?

if [ $exp_exit_status -eq 0 ]
then
  echo "Export finished Successfully"
else
  cat $EXPLOGDIR/exp_$(date +%Y%m%d).log | egrep -i "ORA-|Failing|error" > $LOGDIR/expdp.err
  mailx -s "Export on Database $ORACLE_SID Failed!!! " henndue14@gmail.com < $LOGDIR/expdp.err
  exit 1  
fi

echo "Finish datapump Export $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

#################################
# Alter undo_retention parameter
#################################

echo "Start UNDO_RETUENTION Update $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

$ORACLE_HOME/bin/sqlplus -s <<!
/ as sysdba

whenever sqlerror exit 2;

set echo off
set feedback off
spool $LOGDIR/undo.err

alter system set undo_retention=1800 scope=both;
commit;

spool off
!
undo_exit_status=$?

if [ $undo_exit_status -eq 0 ]
then
  echo "undo_retention successfully updated"
else
  mailx -s "Update undo_retention on Database $ORACLE_SID Failed!!! " henndue14@gmail.com < $LOGDIR/undo.err
  exit 1  
fi

echo "Finish UNDO_RETUENTION Update $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

#################################
# Grant permissions for dmp files
#################################

echo "Start dmp file permission Update $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

chmod -R 775 $EXPLOGDIR/exp_*.dmp

echo "Finish dmp file permission Update $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

#################################
# Compressing files
#################################

echo "Start compressing dmp files $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

gzip $EXPLOGDIR/exp_*.dmp

echo "Finish compressing dmp files $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

#################################
# SCP dmp files to lihbg000tpcdu
#################################

echo "Start dmp file scp $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

/u01/datapump/datapump_exports/scp_dump.sh

echo "Finish dmp file scp $(date)" >> $TEMP_FILE
echo "\n" >> $TEMP_FILE

#################################
# Email export process output
#################################

mailx -s "Database Export process has successfully completed" henndue14@gmail.com < $TEMP_FILE

#################################
# Log file cleanup
#################################

rm /u01/datapump/datapump_exports/expdp_log.txt
rm /u01/datapump/datapump_exports/undo.err
rm /u01/datapump/datapump_exports/expdp.err
exit;

Friday, July 17, 2015

How To Configure SSH for a RAC Installation (Doc ID 300548.1)

To configure SSH you need to perform the following steps on each node in the cluster.

$ cd $HOME
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ ssh-keygen -t rsa


Now accept the default location for the key file
Enter and confirm a passphrase. (you can also press enter twice).


$ ssh-keygen -t dsa

Now accept the default location for the key file
Enter and confirm a passphrase. (you can also press enter twice).


$ cat *.pub >> authorized_keys.<nodeX> (nodeX could be the nodename to differentiate files later)

Now do the same steps on the other nodes in the cluster.
When all those steps are done on the other nodes, start to copy the authorized_keys.<nodeX> to all the nodes into $HOME/.ssh/


For example if you have 4 nodes you will have after the copy in the .ssh 4 files with the name authorized_keys.<nodeX>

Then on EACH node continue the configuration of SSH by doing the following:

$ cd $HOME/.ssh
$ cat *.node* >> authorized_keys
$ chmod 600 authorized_keys


NOTE: ALL public keys must appear in ALL authorized_keys files, INCLUDING the LOCAL public key for each node.

To test that everything is working correct now execute the commands

$ ssh <hostnameX> date

So on example in a 4 node environment:
$ ssh node1 date
$ ssh node2 date
$ ssh node3 date
$ ssh node4 date


Repeat this 4 times on each node, including ssh back to the node itself. The nodeX is the hostname of the node.

The first time you will be asked to add the node to a file called 'known_hosts' this is correct and answer the question with 'yes'. After that when correctly configured you must be able to get the date returned and you will not be prompted for a password.

Note: the above will work if during RSA and DSA configuration no password was provided. If you provide a password then you need to do 2 addition steps.

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add


These statements will inform the ssh agent to add the keys to the shell used. So when a new shell is started you need to repeat the last to statements to make sure ssh is working properly.

Please note, ssh will not allow passwordless access if permissions on the home directory of the account you are using allow write access for everyone.

You will also see permission denied error when the permissions on $HOME are 777 or 775. $HOME should have permission 700 or 755.

How To Download And Install The Latest OPatch Version (Doc ID 274526.1)

1) Please download the latest OPatch version from My Oracle Support (MOS) using the next reference:

Patch 6880880

Or using the next URL:
https://updates.oracle.com/download/6880880.html

2) In the "Platform" field select the relevant platform that corresponds to the Oracle product.

3) Click the Download button


2) Upload the opatch.zip file to your OS host and unzip it:
Example:

% mv p6880880_101000_SOLARIS64.zip $ORACLE_HOME
% cd $ORACLE_HOME
(If a former OPatch directory already exist, then rename it)
% mv Opatch OPatch_100057_OLD
% unzip p6880880_101000_SOLARIS64.zip
Archive: p6880880_101000_SOLARIS64.zip
creating: OPatch/
creating: OPatch/docs/
inflating: OPatch/docs/FAQ
inflating: OPatch/docs/README.txt
inflating: OPatch/docs/Users_Guide.txt
inflating: OPatch/docs/bt1.txt
inflating: OPatch/docs/bt2.txt
inflating: OPatch/docs/tmp
inflating: OPatch/emdpatch.pl
creating: OPatch/jlib/
inflating: OPatch/jlib/opatch.jar
inflating: OPatch/opatch
inflating: OPatch/opatch.bat
inflating: OPatch/opatch.pl
creating: OPatch/perl_modules/
inflating: OPatch/perl_modules/Apply.pm
inflating: OPatch/perl_modules/AttachHome.pm
inflating: OPatch/perl_modules/Command.pm
inflating: OPatch/perl_modules/LsInventory.pm
inflating: OPatch/perl_modules/Query.pm
inflating: OPatch/perl_modules/RollBack.pm
inflating: OPatch/perl_modules/Version.pm
inflating: OPatch/perl_modules/XML.pm
inflating: OPatch/perl_modules/opatchIO.pm
inflating: OPatch/README.txt
creating: OPatch/ocm/
extracting: OPatch/ocm/ocm.zip
creating: OPatch/ocm/doc/
inflating: OPatch/ocm/doc/license.txt
creating: OPatch/ocm/lib/
inflating: OPatch/ocm/lib/emocmutl.jar
creating: OPatch/ocm/bin/
inflating: OPatch/ocm/bin/emocmrsp

% cd OPatch
% opatch version

Oracle Interim Patch Installer version 1.0.0.0.58
Copyright (c) 2007 Oracle Corporation. All Rights Reserved..

We recommend you refer to the OPatch documentation under
OPatch/docs for usage reference. We also recommend using
the latest OPatch version. For the latest OPatch version
and other support related issues, please refer to document
293369.1 which is viewable from metalink.oracle.com

OPatch Version: 1.0.0.0.58

3)You can launch OPatch tool by using the absolute path :

ex.:

$ORACLE_HOME/OPatch/opatch version

Or you can set the OPatch directory in your PATH variable so you can execute the OPatch command (opatch <args>) from anywhere without using the absolute path.
ex.:

For Korn / Bourne shell:

% export PATH=$PATH:$ORACLE_HOME/OPatch

For C Shell:
% setenv PATH $PATH:$ORACLE_HOME/OPatch

How to Add a RAC Node to an Oracle Cluster

A. Verify Prequisites:

Check System Hardware requirements; make sure it is matching with existing nodes (Recommended).

RAM size (Min 2.5 GB, to match with other nodes 36 GB):
/usr/sbin/lsattr -E -l sys0 -a realmem


Swap Space (Min 16 GB, to match with other nodes 24 GB):
/usr/sbin/lsps –s


Temp directory space (Min 1GB, to match with other nodes 4GB):
/usr/bin/df -k /tmp


Compare the size of /u01 mount point with existing nodes to match the size of software installation location
df –g

1. Create Groups and users:
Make sure the following User and Groups exists.
User: Oracle Groups: Oinstall, DBA


Run the below command in racattack18u:
$id oracle
If they don’t exist than create them.
$mkgroup id=’267’ adms='root' oinstall
$ mkgroup id=’264’ adms='root' dba
$ mkuser id=’501’  pgrp='oinstall' groups='dba' home='/u01' oracle


Id’s matching with current production id’s.

2. Create required Directories
ORACLE_BASE
mkdir –p /u01/app/oracle
chown oracle:oinstall /u01/app/oracle
chmod –R 775 /u01/app/oracle

ORACLE_HOME
mkdir –p /u01/products/rdbms_11203
chown oracle:oinstall /u01/products/rdbms_11203
chmod –R 775 /u01/products/rdbms_11203

GRID_HOME
mkdir –p /u01/products/grid_11203
chown root:oinstall /u01/products/grid_11203
chmod –R 775 /u01/products/grid_11203


3. UDP and TCP Kernal Parameters:
Contact AIX team and request them to verify if the kernel parameters are setup as recommended by Oracle.

http://docs.oracle.com/cd/E11882_01/install.112/e48294/preaix.htm#CWAIX418

4. Check the network requirements
Contact Network team and request to setup the network as recommended by Oracle for cluster environment.

http://docs.oracle.com/cd/E11882_01/install.112/e48294/preaix.htm#CWAIX196

A Oracle database cluster node must have the below ip addresses:
Public ip address
Private ip address
Virtual ip address
Single client access name


Current configuration in racattack16u and racattack17u (Can be found in /etc/hosts file):

Public ip address:
10.10.10.148     racattack16u
10.10.10.150     racattack17u


Private id address:
10.10.10.71     racattack16u
10.10.10.72     racattack17u


Virtual ip address:
10.10.10.149  racora3.gmail.com   racora3 # ORA Virt. Address A
10.10.10.151  racora4.gmail.com   racora4 # ORA Virt. Address B


Scan IP address:
SCAN IP Addresses for RAC Cluster (addresses selected round robin in DNS):
# 10.10.10.147 \
# 10.10.10.54  } rac-scan
# 10.10.10.155 /


For Current production cluster setup the Single client access name is defined as “rac-scan”  , new node racattack18u should be included in it.

Once the configuration is done, run the below command to confirm.
$nslookup rac-scan
Output should be similar in all the 3 nodes.


Current output:oracle@racattack17u:rac2:/u01 $  nslookup rac-scan
Server:         10.10.10.37
Address:       10.10.10.53


Name:   rac-scan.li-sec.state.pa.us
Address: 10.10.10.155
Name:   rac-scan.li-sec.state.pa.us
Address: 10.10.10.154
Name:   rac-scan.li-sec.state.pa.us
Address: 10.10.10.147


5. Make sure that all the ASM disks can be accessed by the Oracle user on new node.

Contact Storage team and request them to present all the storage devices currently used by ASM on production cluster database to racattack18u, and set their ownership to oracle user.

Once done, DBA can verify the same using below steps:

On racattack16u:
$export ORACLE_SID=ASM
$ . oraenv
$sqlplus ‘/as sysasm’


Sql>select path from v$asm_disk;
/dev/asm_128G_P0_1
/dev/asm_128G_P0_2
/dev/asm_128G_P0_3
/dev/asm_128G_P10_1
/dev/asm_128G_P1_1
/dev/asm_128G_P1_2
/dev/asm_128G_P1_3
Total 39 disks available right now…..


Exit out of sqlplus,
$ls –ltr /dev/asm_*


Now connect to racattack18u and run the below command:
$ls –ltr /dev/asm_*


Make sure the result is same for “ls” command in racattack16u and racattack18u.

6. Configure ssh and set user equivalency, Make sure you are able to connect to oracle user of racattack16u, racattack17u from racattack18u using ssh without providing password.

As Oracle user Run the Below commands on racattack18u:
$ cd /u01/.ssh
$mv authorized_keys2 authorized_keys2.old


Now As Oracle user run the below Commands on racattack16u
$cd /u01/.ssh
$scp authorized_keys2
oracle@racattack18u: /u01/.ssh/
$ssh racattack18u 
(You should be able to connect without prompting a password)


NOTE: In case something is missed in prerequisites, it can be detected automatically when we run “cluvfy” utility in below steps. Which can be fixed subsequently.

B. Verify using Cluster verification utility for Node add procedure:

1. From an existing node(racattack16u) perform post hardware and OS check
$ cluvfy stage -post hwos -n racattack18u


2. From an existing node run the below command to Compare one of the existing node with new node
$ cluvfy comp peer -refnode racattack16u -n racattack18u -orainv oinstall -osdba dba –verbose


3. Run the below command to Verify the integrity of the cluster and the new node
$cluvfy stage –pre nodeadd –n racattack18u –verbose


We may receive PRVF-5449 Error, it’s an Oracle BUG, workaround is to set the below environment variable.

export IGNORE_PREADDNODE_CHECKS=Y

C. Add Grid Home to the new Node:

Run the below addnode.sh command from racattack16u to add new node to the cluster, this will run Oracle universal installer in silent mode and copy the Oracle Grid_Infrastructure binaries from existing node to new node.


$GRID_HOME/oui/bin/addnode.sh “CLUSTER_NEW_NODES=
{ racattack18u }” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={ racora5}”


At the end of the script, it will prompt to run scripts as root user (in the new node racattack18u), connect to a new session with root user and run the scripts suggested. Once the root scripts are executed successfully, hit enter on the current screen to complete the GRID_HOME installation in new node.

If this is successful than we can see the new node as part of cluster and along with this it starts clusterware deamons, ASM instance, Listener, VIP, Scan services in the new node.

Run the below commands to verify,

$ps –ef|grep grid|grep –v grep (In New Node racattack18u)
$ps –ef|grep d.bin (In New Node racattack18u)
$crsctl check crs (In New Node racattack18u)
$crsctl stat res –t (From any node)
$olsnodes (From any node)


D. Add Oracle Home to the new Node:

Run the below addnode.sh command from racattack16u to add new node to the cluster, this will run Oracle universal installer in silent mode and copy the Oracle rdbms binaries from existing node to new node.

$ORACLE_HOME/oui/bin/addnode.sh “CLUSTER_NEW_NODES=
{ racattack18u }”


At the end of the script, it will prompt to run scripts as root user (in the new node racattack18u), connect to a new session with root user and run the scripts suggested. Hit enter on the current screen to complete the ORACLE_HOME installation in new node.

E. Add instance to the database for newly added node

Invoke DBCA from one of the old node and add new instance to the database.
RAC database-->Instance management-->Add Instance


Do this process for all the databases running under the cluster. 

F. Verification of Node addition:

Run the below commands from one of the node racattack16u to confirm the new node and instance are added successfully.

$ cluvfy stage –post nodeadd –n racattack18u -verbose
$ olsnodes
$ crsctl stat res –t
        
Connect to the new node racattack18u and run the below commands to check the details of newly added node.

$ crsctl check crs  
$ ps –ef|grep d.bin
$ ps –ef|grep pmon

Go through the directories of ORACLE_HOME, ORACLE_BASE and GRID_HOME.

Connect to Oracle database instance using sqlplus.
Connect to the ASM instance using asmcmd and sqlplus.


 

How to remove a RAC node from and Cluster

The following steps the are required to successfully remove a Cluster Node from an existing Cluster:

1) Before starting the node removal process it is always recommended to take backup of OCR and databases.
-Login as root on racattack15u and verify the last backup of OCR
#$GRID_HOME/bin/ocrconfig -showbackup
-If required take a manual backup of OCR
#$GRID_HOME/bin/ocrconfig –manualbackup


2) Unpin the node to be removed if it is pinned
Run the below command from one of the node to check if the node to be removed is pinned or unpinned.
$ olsnodes -t -s 
racattack15u   Active  Unpinned
racattack18u   Active  Unpinned
(NOTE: Verified they are already unpinned)

-Run the below command if racattack18u is pinned
$crsctl unpin css -n racattack18u


3) Remove the database instance using DBCA
Note: This can also be done using a AIX GUI interface like CYGWIN.
Remove the database instance using DBCA:
-Invoke DBCA from racattack15u as oracle user
$$ORACLE_HOME/bin/dbca
Select the following in each screen to remove all the database instances  running in racattack18u
RAC database-->Instance management-->Delete Instance

Repeat to remove all the 3 Instances.

4) Verify the instance has been removed with the following commands on racattack18u.
$srvctl config database -d racdb1
$srvctl config database -d racdb2
$srvctl config database -d racdb3
$ps –ef|grep pmon
$crsctl stat res -t

NOTE: At this point database instances and its related services are removed from cluster. ASM and other clusterware compontents will be removed as part of GRID infrastructure removal.

5) Disable and Stop Listener on the Node to be removed.
oracle@racattack18u::/u01 $ ps -ef|grep tns (To get the running listener_alias name)oracle@racattack18u::/u01 $ lsnrctl status <Listener Alias>oracle@racattack18u::/u01 $ srvctl config listener -a
-Use the below commands to disable the listener(s) which are running from ORACLE_HOME
oracle@racattack18u::/u01 $ srvctl disable listener -l <Listener Name> -n racattack18uoracle@racattack18u::/u01 $ srvctl stop listener -l <Listener Name> -n racattack18u

6) Update Inventory on the Node to be deleted
-Verify the inventory in racattack18u before updating it.
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

-Run the following command on racattack18u to update the inventory
$ /u01/products/rdbms_11203/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/products/rdbms_11203 "CLUSTER_NODES={racattack18u}" -local

-Verify the inventory in racattack18u it should not have information about racattack15u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml


7.) Remove the  RAC Database Software
The next step is to remove the RAC Database Software from the node to be deleted (racattack18u).
-Run below command as oracle user from racattack18u
$ cd $ORACLE_HOME/deinstall
$ ./deinstall -local
(-local option is to remoce ORACLE_HOME from local node)


8.) Update Inventory on the Remaining Nodes
-Verify the inventory in racattack15u before updating it.
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

-Run the following command on racattack15u to update the inventory
$ cd /u01/products/rdbms_11203/oui/bin
$ runInstaller -updateNodeList ORACLE_HOME=/u01/products/rdbms_11203 "CLUSTER_NODES={racattack15u}" -local

-Verify the inventory in racattack15u it should not have information about racattack18u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml


9.) Remove Clusterware
The next step is to remove the clusterware from the node you want to delete.
Note: This must do as root.

-As root on racattack18u run the below command to deconfig clusterware
# cd /u01/products/grid_11203
# pwd
/u01/products/grid_11203
# cd crs/install
# pwd
/u01/products/grid_11203/crs/install
# hostname
racattack18u
# ./rootcrs.pl -deconfig -force

(At this point All the clusterware services including ASM are stopped)

10.) Delete node from Clusterware configuration
The next step is to remove the desired node from the clusterware configuration.  This is done with the following commands.
# cd /u01/products/grid_11203/bin
# olsnodes –t –s (For verification)
-As root on racattack15u run the below command to delete racattack18u from cluster
# crsctl delete node -n racattack18u
(racattack18u will be removed from clusterware and registry)
-Run the below command from racattack15u to verify the existing nodes
# olsnodes –t –s (For verification)
(Should only show racattack15u)


11.) Update Inventory on node to be removed as "Grid"
The next step is to update the inventory on the desired node to be removed as the "Grid" software owner (Oracle for this environment).
Note:  make sure that is done as the "Grid Owner" which is "Oracle" in this environment.

Verify the inventory in racattack18u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

-Run the following command on racattack18u to update the inventory
$ cd /u01/products/grid_11203/oui/bin
$ runInstaller -updateNodeList ORACLE_HOME=/u01/products/grid_11203 "CLUSTER_NODES={racattack18u}" -local

-Verify the inventory in racattack18u it should not have information about racattack15u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml


12.) Remove the Clusterware Software from node to be removed.
The next step is to remove the clusterware software from the desired node (racattack18u). 
Note:  make sure that is done as the "Grid Owner". which is "Oracle" in this environment.

-Run below command as oracle user from racattack18u
$ cd /u01/products/grid_11203/deinstall
$ ./deinstall -local
(-local option is to remoce ORACLE_HOME from local node)
-In the end it asks us to run a script from a different session as root, run the script


13.) Update Inventory on remaining nodes for "Grid"
The next step is to update inventory on the remaining nodes (racattack15u) for "Grid". 
Note:  make sure that is done as the "Grid Owner"

Verify the inventory in racattack15u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml

- Run the following command on racattack15u to update the inventory
$ cd /u01/products/grid_11203/oui/bin
$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/products/grid_11203 "CLUSTER_NODES={racattack15u}"

-Verify the inventory in racattack15u it should not have information about racattack18u
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml


14.) Confirm the node has been successfully removed.
The next step is to verify that the node has been successfully removed.  To do this, execute the following command.
-Run the below command from racattack15u as oracle useroracle@racattack15u:racdb1:/u01 $ cluvfy stage -post nodedel -n racattack18uoracle@racattack15u:racdb1:/u01 $ crsctl stat res -t
NOTE: At this Stage racattack18u is clean without any Oracle software installations. racattack15u still remains as a single instance cluster database.