Adding A Node To Oracle RAC 11g Release 2 - DatabaseJournal

Top White Papers and Webcasts
5 Reasons Your SQL Server is Slow

This white paper is a guide to the
5 most common things that cause
your SQL Server to run more
slowly than it should. These are
issues that, once addressed, will
allow your SQL Server instances
to run faster. Best of all, very little work is required to fix
them.
Internet of Things: Choose an Intelligent
Database
Live Event Date: June 5, 2014 @
1:00 p.m. ET / 10:00 a.m. PT The
"Internet of Things" refers to the
growing number of devices and
sensors that communicate and
interact via the Internet, offering
businesses new customers and revenue opportunities.
Harnessing data from billions of connected devices lies
in the ability to store, access, and query SQL and
NoSQL data together, seamlessly. Check out this
upcoming eSeminar to learn how to bridge the gap
between new computing technologies designed for big
data,

MS SQL Oracle DB2 Access MySQL PostgreSQL Sybase PHP SQL Etc SQL Scripts & Samples Links Database Forum
Database Journal
Home
Database Articles
Database Tutorials
MS SQL
Oracle
DB2
MS Access
MySQL
RESOURCES
Database Tools
SQL Scripts &
Samples
Links
Database Forum
Sitemap
Free Newsletters:
DatabaseDaily
News Via RSS Feed
Database Journal |DBA Support |SQLCourse |SQLCourse2
Featured Database Articles
Oracle
Posted Mar 30, 2011
Adding a Node to Oracle RAC 11g Release 2
By Vincent Chan
Oracle Real Application Clusters (Oracle RAC) delivers the ability to horizontally scale out your database system to meet increasing business demands by utilizing inexpensive commodity server architectures. Nodes can be added to an
existing pool of RAC nodes to balance computing resources and to achieve Service Level Agreements (SLAs).
In this guide, we will demonstrate a step-by-step procedure of adding a third node to our existing two-node Oracle RAC 11g.
Downloads for this guide are available at My Oracle Support:
Oracle Database 11g Release 2 (11.2.0.2.0) for Linux x86-64 (p10098816_112020_Linux-x86-64_1of7 and p10098816_112020_Linux-x86-64_2of7)
Oracle Database 11g Release 2 Grid Infrastructure (11.2.0.2.0) for Linux x86-64 (p10098816_112020_Linux-x86-64_3of7)
Overview
Our RAC environment set up comprises of two Intel x86 servers running on Oracle Enterprise Linux 5. The Operating System has been installed and patched on both nodes and each has access to a shared storage and connectivity to the public
and private network.
This guide is structured into the following steps:
Preliminary Installation 1.
Install the Oracle Database 11g Release 2 Grid Infrastructure software on the third node 2.
Install the Oracle Database 11g Release 2 software on the third node 3.
Add a third database instance on the third node 4.
An overview of the two-node Oracle RAC environment:
Software owner accounts and locations:
Shared File storage:
IP Addresses:
And an overview of the third node:
Software owner accounts and locations:
Shared File storage:
IP Addresses:

Adding a Node to Oracle RAC 11g Release 2 DatabaseJournal.com http://www.databasejournal.com/features/oracle/article.php/3929556/Ad...
1 of 10 5/21/2014 12:33 PM
For educational purposes, we will use the same oracle Unix account and group to install the Grid Infrastructure and database software. Oracle recommends using different Unix users and groups for separation of duties.
1. Preliminary Installation
This section lists the setup necessary before starting the Grid Infrastructure software installation.
1.1 Software and Hardware requirements
The required OS patches have been applied to the third node and connectivity to the database shared storage, public and private network have been configured. Review the software and hardware requirements listed here.
1.2 Configure GNS (Grid Naming Service) Virtual IP Address
The Grid Naming Service (GNS) is a part of the Grid Plug and Play feature (GPnP) which allows the Oracle Clusterware to dynamically allocate virtual IP address for each node through DHCP service running on the public network. All name
resolution requests for the cluster within a subdomain delegated by the DNS are handed off to GNS using multicast Domain Name Service (mDNS) included within the Oracle Clusterware. Using GNS eliminates the need for managing IP
addresses and name resolution and is especially advantageous in a dynamic cluster environment where nodes are often added or removed.
Prior to the Grid Infrastructure software installation, we have configured a static IP address for the GNS virtual IP and defined ayu.vcconsult.com as the subdomain to delegate in DNS.
Refer to the Oracle Grid Infrastructure Installation Guide for additional information about GNS.
1.3 SCAN (Single Client Access Name)
SCAN simplifies client management by providing a single point of access for clients to connect to any database in the cluster. In the event when nodes were added or removed from the cluster, clients that use SCAN to access the cluster do not
have to make any explicit changes to the client settings.
SCAN is resolvable by either of the following methods:
Domain Name Service (DNS): At least one IP address has to be defined in DNS for the SCAN. The recommended approach is to create a round robin SCAN resolvable to a maximum of three different addresses for high availability and
scalability.
1.
Grid Naming Service (GNS): Three IP addresses are automatically acquired from the DHCP service and the GNS provides name resolution for the SCAN. 2.
SCAN addresses, virtual IP addresses, and public IP addresses must all be on the same subnet.
Refer to the following whitepaper for additional information on SCAN.
In this installment, we will use GNS to resolve the SCAN addresses.
oracle@ayu1-> dig ayu-cluster-scan.ayu.vcconsult.com
; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> ayu-cluster-scan.ayu.vcconsult.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37381
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;ayu-cluster-scan.ayu.vcconsult.com. IN A
;; ANSWER SECTION:
ayu-cluster-scan.ayu.vcconsult.com. 3600 IN A 192.168.1.235
;; AUTHORITY SECTION:
ayu.vcconsult.com. 86400 IN NS gns.ayu.vcconsult.com.
;; Query time: 1 msec
;; SERVER: 192.168.1.13#53(192.168.1.13)
;; WHEN: Sun Dec 12 14:18:44 2010
;; MSG SIZE rcvd: 126
1.4 Configure Oracle ASM (Automatic Storage Management) shared storage
In our environment, we have created four ASMdisks; three of which were dedicated for storing the OCR, Voting disk, datafiles and online redo logs and one for the Fast Recovery Area. Verify that the ASM disks are visible on the new node.
[root@ayu1 ~]# oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
VOL1
VOL2
VOL3
VOL4
VOL1
VOL2
VOL3
VOL4
Oracle ASMstorage configuration is described here.
1.5 CTSS (Cluster Time Synchronization Service)
CTSS is configured automatically as a part of the Grid Infrastructure installation to synchronize the clocks on all the nodes. As an alternative, you can manually configure NTP (Network Time Protocol) on all the nodes. We will synchronize
the time across the cluster nodes using CTSS.
1.6 Modify /etc/hosts on all nodes
Add the public and private IP addresses of the third node in /etc/hosts. Verify that the /etc/hosts file on each node has the same entries.
oracle@ayu1-> more /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.1.121 ayu1.vcconsult.com ayu1
10.10.10.121 ayu1-priv.vcconsult.com ayu1-priv
1.7 Establish user equivalence with SSH
oracle@ayu3-> pwd
/home/oracle/.ssh
oracle@ayu3-> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
f1:64:97:c2:ae:d2:12:e8:82:3a:07:34:a0:ab:71:01 oracle@ayu3
oracle@ayu3-> ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
33:e7:43:9b:b8:76:f0:9e:7a:e3:f0:43:c8:04:af:53 oracle@ayu3
On ayu1:
oracle@ayu1-> pwd
/home/oracle/.ssh
oracle@ayu1-> ssh ayu3 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'ayu3 (192.168.1.123)' can't be established.
RSA key fingerprint is 24:04:fc:51:ed:cb:03:44:2d:90:8f:91:4a:7b:7e:4f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ayu3,192.168.1.123' (RSA) to the list of known hosts.
oracle@ayu3's password:
oracle@ayu1-> scp ~/.ssh/authorized_keys ayu2:~/.ssh/authorized_keys
authorized_keys 100% 3760 3.7KB/s 00:00
oracle@ayu1-> scp ~/.ssh/authorized_keys ayu3:~/.ssh/authorized_keys
oracle@ayu3's password:
authorized_keys 100% 3760 3.7KB/s 00:00
Test the connections on each node. Verify that you are not prompted for password.
1.8 Verify the integrity of the cluster and the new node
2 of 10 5/21/2014 12:33 PM
oracle@ayu1-> more .profile_grid
export EDITOR=vi
export ORACLE_UNQNAME=qdb
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0.2/grid
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:/bin:/usr/bin:/usr/sbin:/usr/local/bin
umask 022
oracle@ayu1-> . ./.profile_grid
oracle@ayu1-> which cluvfy
/u01/app/11.2.0.2/grid/bin/cluvfy
oracle@ayu1-> cluvfy stage -pre nodeadd -n ayu3 -verbose
Performing pre-checks for node addition
Checking node reachability...
Check: Node reachability from node "ayu1"
Destination Node Reachable?
------------------------------------ ------------------------
ayu3 yes
Result: Node reachability check passed from node "ayu1"
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
ayu3 passed
Result: User equivalence check passed for user "oracle"
Checking node connectivity...
Checking hosts config file...
Node Name Status Comment
------------ ------------------------ ------------------------
ayu3 passed
ayu2 passed
ayu1 passed
Verification of the hosts config file successful
Code
<< snipped for clarity >>
Checking OCR integrity...
OCR integrity check passed
Checking Oracle Cluster Voting Disk configuration...
ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:VOL1(ORCL:VOL1)" failed on the following nodes:
ayu3:No such file or directory
ERROR:
ERROR:
PRVF-5431 : Oracle Cluster Voting Disk configuration check failed
Check: Time zone consistency
Result: Time zone consistency check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Checking to make sure user "oracle" is not in "root" group
------------ ------------------------ ------------------------
ayu3 does not exist passed
ayu1 does not exist passed
Result: User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "vcconsult.com" as found on node "ayu1"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
Node Name Status
------------------------------------ ------------------------
ayu1 passed
ayu3 passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "ayu.vcconsult.com" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "192.168.1.200" resolves to a valid IP address
Checking the status of GNS VIP...
PRVF-5229 : GNS VIP is active before Clusterware installation
Checking if FDQN names for domain "ayu.vcconsult.com" are reachable
GNS integrity check passed
Pre-check for node addition was unsuccessful on all the nodes.
You can ignore the PRVF-5449 message if you have already configured and verified that the ASMshared disks are visible on the third node. The error was a result of having the voting disks stored in Oracle ASMstorage, a new feature of
Oracle 11g Release 2.
2. Install the Oracle Database 11g Release 2 Grid Infrastructure software on the third node
You will execute the addNode.sh script to install the Oracle Grid Infrastructure software on the new node. The script cannot proceed unless all the pre-requisites checks are successful. In our case, since we are getting the PRVF-5449
messages, we will set the IGNORE_PREADDNODE_CHECKS variable to bypass the node addition pre-check.
Execute the addNode.sh script on ayu1 as oracle:
oracle@ayu1-> export IGNORE_PREADDNODE_CHECKS=Y
oracle@ayu1-> cd $ORACLE_HOME/oui/bin
oracle@ayu1-> pwd
/u01/app/11.2.0.2/grid/oui/bin
oracle@ayu1-> ./addNode.sh "CLUSTER_NEW_NODES={ayu3}"
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 1154 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes ayu2,ayu3 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/11.2.0.2/grid
New Nodes
Space Requirements
New Nodes
ayu3
/u01: Required 3.55GB : Available 16.25GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.3
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Server) 11.2.0.2.0
Installation Plugin Files 11.2.0.2.0
Universal Storage Manager Files 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Automatic Storage Management Assistant 11.2.0.2.0
3 of 10 5/21/2014 12:33 PM
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.2.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Oracle Net Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Enterprise Manager plugin Common Files 11.2.0.2.0
Platform Required Support Files 11.2.0.2.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.2.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Cluster Verification Utility Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle LDAP administration 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
PL/SQL 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Cluster Ready Services Files 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Sunday, December 12, 2010 11:48:21 AM EST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Sunday, December 12, 2010 11:48:30 AM EST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Sunday, December 12, 2010 11:53:16 AM EST)
. 100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'ayu3'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oraInventory/orainstRoot.sh #On nodes ayu3
/u01/app/11.2.0.2/grid/root.sh #On nodes ayu3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/11.2.0.2/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
Execute the orainstRoot.sh script on the third node as root:
[root@ayu3 ~]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
Execute the root.sh script on the third node as root:
[root@ayu3 ~]# /u01/app/11.2.0.2/grid/root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0.2/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig _params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node ayu2, number 2, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Perform post-installation checks
oracle@ayu3-> pwd
/u01/app/11.2.0.2/grid/bin
oracle@ayu3-> ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE ayu1
ora....ER.lsnr ora....er.type ONLINE ONLINE ayu1
ora....N1.lsnr ora....er.type ONLINE ONLINE ayu2
ora.asm ora.asm.type ONLINE ONLINE ayu1
ora....SM1.asm application ONLINE ONLINE ayu1
ora....U1.lsnr application ONLINE ONLINE ayu1
ora.ayu1.gsd application OFFLINE OFFLINE
ora.ayu1.ons application ONLINE ONLINE ayu1
ora.ayu1.vip ora....t1.type ONLINE ONLINE ayu1
4 of 10 5/21/2014 12:33 PM
ora.cvu ora.cvu.type ONLINE ONLINE ayu1
ora.gns ora.gns.type ONLINE ONLINE ayu1
ora.gns.vip ora....ip.type ONLINE ONLINE ayu1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE ayu1
ora.oc4j ora.oc4j.type ONLINE ONLINE ayu1
ora.ons ora.ons.type ONLINE ONLINE ayu1
ora.qdb.db ora....se.type ONLINE ONLINE ayu1
ora....ry.acfs ora....fs.type ONLINE ONLINE ayu1
ora.scan1.vip ora....ip.type ONLINE ONLINE ayu2
oracle@ayu3-> ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.LISTENER.lsnr
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.asm
ONLINE ONLINE ayu1 Started
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.gsd
OFFLINE OFFLINE ayu1
ora.net1.network
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.ons
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.registry.acfs
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ayu2
ora.ayu1.vip
ora.ayu2.vip
ora.ayu3.vip
ora.cvu
ora.gns
ora.gns.vip
ora.oc4j
ora.qdb.db
1 ONLINE ONLINE ayu1 Open
ora.scan1.vip
ora.scan2.vip
ora.scan3.vip

oracle@ayu1-> srvctl status nodeapps
VIP 192.168.1.232 is enabled
VIP 192.168.1.232 is running on node: ayu1
Network is enabled
Network is running on node: ayu1
GSD is disabled
GSD is not running on node: ayu1
ONS is enabled
ONS daemon is running on node: ayu1
oracle@ayu1-> srvctl config nodeapps
Network exists: 1/192.168.1.0/255.255.255.0/eth0, type dhcp
VIP exists: /192.168.1.232/192.168.1.232/192.168.1.0/255.255.255.0/eth0, hosting node ayu1
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
oracle@ayu3-> crsctl check cluster -all
**************************************************************
ayu1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ayu2:
**************************************************************
ayu3:
**************************************************************
oracle@ayu3-> srvctl config gns
GNS is enabled.
oracle@ayu3->
oracle@ayu3-> srvctl status gns
GNS is running on node ayu1.
GNS is enabled on node ayu1.
oracle@ayu3-> srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node ayu2
oracle@ayu3-> srvctl config scan
SCAN name: ayu-cluster-scan.ayu.vcconsult.com, Network: 1/192.168.1.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /192.168.1.233/192.168.1.233
oracle@ayu3-> srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node ayu2
oracle@ayu3-> srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
oracle@ayu1-> srvctl status database -d qdb
Instance qdb1 is running on node ayu1
oracle@ayu1-> srvctl config database -d qdb
Database unique name: qdb
Database name: qdb
Oracle home: /u01/app/oracle/product/11.2.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATA/qdb/spfileqdb.ora
Domain: vcconsult.com
Start options: open
Stop options: immediate
Database role: PRIMARY
5 of 10 5/21/2014 12:33 PM
Management policy: AUTOMATIC
Server pools: qdb
Database instances: qdb1,qdb2
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed
You have completed the installation of the Oracle Grid Infrastructure on the new node!
3. Install the Oracle Database 11g Release 2 software on the third node
Similarly, you will set the IGNORE_PREADDNODE_CHECKS and execute the addNode.sh script to install the Oracle Database software on the new node.
Execute the addNode.sh script on ayu1 as oracle:
oracle@ayu1-> more .profile
export EDITOR=vi
export ORACLE_UNQNAME=qdb
export ORACLE_SID=qdb1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.2/dbhome_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:/bin:/usr/bin:/usr/sbin:/usr/local/bin
umask 022
oracle@ayu1-> export IGNORE_PREADDNODE_CHECKS=Y
oracle@ayu1-> . ./.profile
oracle@ayu1-> cd $ORACLE_HOME/oui/bin
oracle@ayu1-> pwd
/u01/app/oracle/product/11.2.0.2/dbhome_1/oui/bin
oracle@ayu1-> ./addNode.sh "CLUSTER_NEW_NODES={ayu3}"
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 1175 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes ayu2,ayu3 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/oracle/product/11.2.0.2/dbhome_1
New Nodes
Space Requirements
New Nodes
ayu3
/u01: Required 3.96GB : Available 13.45GB
Installed Products
Product Names
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.3.1.1
Oracle ODBC Driverfor Instant Client 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
SSL Required Support Files for InstantClient 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.2.0
Oracle Database Vault J2EE Application 11.2.0.2.0
Oracle Label Security 11.2.0.2.0
Oracle Data Mining RDBMS Files 11.2.0.2.0
Oracle OLAP RDBMS Files 11.2.0.2.0
Oracle OLAP API 11.2.0.2.0
Platform Required Support Files 11.2.0.2.0
Oracle Database Vault option 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.2.0
Oracle SQL Developer 11.2.0.2.0
Oracle Application Express 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
SQLJ Runtime 11.2.0.2.0
Database Workspace Manager 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.2.0
Enterprise Manager Repository Core Files 10.2.0.4.3
Enterprise Manager Database Plugin -- Agent Support 11.2.0.2.0
Enterprise Manager Grid Control Core Files 10.2.0.4.3
Enterprise Manager Common Core Files 10.2.0.4.3
Enterprise Manager Agent Core Files 10.2.0.4.3
RDBMS Required Support Files 11.2.0.2.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Parser Generator Required Support Files 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Multimedia Annotator 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Sample Schema Data 11.2.0.2.0
Oracle Starter Database 11.2.0.2.0
Oracle Message Gateway Common Files 11.2.0.2.0
Oracle XML Query 11.2.0.2.0
XML Parser for Oracle JVM 11.2.0.2.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.2.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.2.0
Oracle Containers for Java 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
Oracle Net Required Support Files 11.2.0.2.0
Secure Socket Layer 11.2.0.2.0
Oracle Universal Connection Pool 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Precompiler Common Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
PL/SQL 11.2.0.2.0
Generic Connectivity Common Files 11.2.0.2.0
Oracle Database Gateway for ODBC 11.2.0.2.0
Oracle Programmer 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Call Interface (OCI) 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle XML Development Kit 11.2.0.2.0
Database Configuration and Upgrade Assistants 11.2.0.2.0
Oracle JVM 11.2.0.2.0
6 of 10 5/21/2014 12:33 PM
Oracle Advanced Security 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Enterprise Manager Console DB 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Oracle Text 11.2.0.2.0
Oracle Net Services 11.2.0.2.0
Oracle OLAP 11.2.0.2.0
Oracle Spatial 11.2.0.2.0
Oracle Partitioning 11.2.0.2.0
Enterprise Edition Options 11.2.0.2.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Sunday, December 12, 2010 12:27:57 PM EST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Sunday, December 12, 2010 12:28:04 PM EST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Sunday, December 12, 2010 12:38:21 PM EST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oracle/product/11.2.0.2/dbhome_1/root.sh #On nodes ayu3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/oracle/product/11.2.0.2/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
Execute the root.sh script on the third node as root:
[root@ayu3 ~]# /u01/app/oracle/product/11.2.0.2/dbhome_1/root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0.2/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
Execute the rootcrs.pl script on the third node as root:
[root@ayu3 ~]# /u01/app/11.2.0.2/grid/crs/install/rootcrs.pl
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Perform post-installation checks
oracle@ayu3-> which cluvfy
/u01/app/11.2.0.2/grid/bin/cluvfy
oracle@ayu3-> cluvfy stage -post nodeadd -n ayu3 -verbose
Performing post-checks for node addition
Checking node reachability...
Check: Node reachability from node "ayu3"
Destination Node Reachable?
------------------------------------ ------------------------
ayu3 yes
Result: Node reachability check passed from node "ayu3"
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
ayu3 passed
Result: User equivalence check passed for user "oracle"
Checking node connectivity...
Checking hosts config file...
------------ ------------------------ ------------------------
ayu3 passed
Verification of the hosts config file successful
<< snipped for clarity >>
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
ayu3 Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
Node Name Time Offset Status
------------ ------------------------ ------------------------
ayu3 0.0 passed
Time offset is within the specified limits on the following set of nodes:
"[ayu3]"
Result: Check of clock time offsets passed
Oracle Cluster Time Synchronization Services check passed
Post-check for node addition was successful.
You have just completed the database software installation on the new node. At this stage, you should take a back up of your voting disk and Oracle Cluster Registry file. In the next section, we will create a third database instance on the new
node.
4. Add a third database instance on the third node
We will use the Database Configuration Assistant (DBCA) to add the third instance, qdb3.
Execute dbca on ayu3 as oracle:
oracle@ayu1-> . ./.profile
oracle@ayu3-> which dbca
/u01/app/oracle/product/11.2.0.2/dbhome_1/bin/dbca
oracle@ayu3-> dbca
Welcome: Select Oracle Real Application Clusters (RAC) database as the database type that you would like to create or administer. 1.
Operations: Select Instance Management. 2.
Instance Management: Select Add an instance. 3.
List of cluster databases: Enter the sys username and password. 4.
List of cluster database instances: Verify the existing database instances. 5.
Instance naming and node selection: Verify the instance name (qdb3) to be added and the node (ayu3) on which to add the instance. 6.
Instance Storage: Verify the storage configuration. 7.
Summary: Verify the set up and click on OK. 8.
7 of 10 5/21/2014 12:33 PM
1. Database Configuration Assistant: Click on No.
oracle@ayu3-> ps -ef|grep dbw
oracle 13721 1 0 12:09 ? 00:00:00 asm_dbw0_+ASM3
oracle 21113 1 0 13:03 ? 00:00:00 ora_dbw0_qdb3
oracle 23785 2580 0 13:13 pts/0 00:00:00 grep dbw
oracle@ayu3-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE ayu1
ora....ER.lsnr ora....er.type ONLINE ONLINE ayu1
ora.asm ora.asm.type ONLINE ONLINE ayu1
ora.cvu ora.cvu.type ONLINE ONLINE ayu1
ora.gns ora.gns.type ONLINE ONLINE ayu1
ora.gns.vip ora....ip.type ONLINE ONLINE ayu1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE ayu1
ora.oc4j ora.oc4j.type ONLINE ONLINE ayu1
ora.ons ora.ons.type ONLINE ONLINE ayu1
ora.qdb.db ora....se.type ONLINE ONLINE ayu1
ora....ry.acfs ora....fs.type ONLINE ONLINE ayu1
oracle@ayu3-> crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.LISTENER.lsnr
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.asm
ONLINE ONLINE ayu1 Started
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.gsd
ora.net1.network
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.ons
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
ora.registry.acfs
ONLINE ONLINE ayu1
ONLINE ONLINE ayu2
ONLINE ONLINE ayu3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ayu1.vip
8 of 10 5/21/2014 12:33 PM
Reply to this comment
ora.ayu2.vip
ora.ayu3.vip
ora.cvu
ora.gns
ora.gns.vip
ora.oc4j
ora.qdb.db
ora.scan1.vip
ora.scan2.vip
ora.scan3.vip

oracle@ayu3-> srvctl status database -d qdb
oracle@ayu3-> srvctl config database -d qdb
Database unique name: qdb
Database name: qdb
Oracle home: /u01/app/oracle/product/11.2.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATA/qdb/spfileqdb.ora
Domain: vcconsult.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: qdb
Database instances: qdb1,qdb2,qdb3
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed
oracle@ayu3-> srvctl status asm
ASM is running on ayu3,ayu1,ayu2
oracle@ayu3->
oracle@ayu3-> srvctl config asm
ASM home: /u01/app/11.2.0.2/grid
ASM listener: LISTENER
SQL> select thread#,instance,enabled from v$thread;
THREAD# INSTANCE ENABLED
---------- ---------- --------
1 qdb1 PUBLIC
2 qdb2 PUBLIC
3 qdb3 PUBLIC
SQL> select * from v$logfile order by 1;
GROUP# STATUS TYPE MEMBER IS_
---------- ------- ------- --------------------------------------------- ---
1 ONLINE +DATA/qdb/onlinelog/group_1.261.737743115 NO
6 rows selected.
SQL> select file_name from dba_data_files where tablespace_name='UNDOTBS3';
FILE_NAME
--------------------------------------------------------
+DATA/qdb/datafile/undotbs3.269.737989267
Congratulations, you have successfully added a third database instance!
Conclusion
With a few simple steps, a node can be easily added to an existing Oracle RAC 11g Release 2 cluster. Hopefully, this guide has provided a clear and concise method of performing a cluster node addition.
For more information about adding a RAC node, please refer to Oracle Real Application Clusters Administration and Deployment Guide.
Vincent Chan (vkchan99@yahoo.com) is a Senior Computer Systems Architect at Lockheed Martin Corporation. He is an Oracle Certified Master DBA with more than 15 years of experience architecting and implementing Oracle solutions
for various clients.
Oracle Archives
1 Comments (click to add your comment)
By Steve October 26 2011 05:33 PDT
What's the purpose of running the rootcrs.pl command after installing grid and before installing oracle? I can't find a reference to running that command anywhere in any Oracle documentation.
Comment and Contribute

Your name/nickname
Your email
Subject
(Maximum characters: 1200). You have characters left.

Latest Forum Threads
Oracle Forum
Topic By Replies Updated
Oracle Data Mining: Classification jan.hasller 0 July 5th, 07:19 AM
in! !u"licates # $ni%ue &Ds 'a(a 5 July )n!, 0*:+0 AM
no ,atching uni%ue or "ri,ary -ey rcanter 1 A"ril )5th, 1):+) .M
$"!ate (alues of one ta/le /ase! on con!ition of (alues in other ta/le using 0rigger 1la!iator + e/ruary )9th, 02:01 .M
9 of 10 5/21/2014 12:33 PM
Property of Quinstreet Enterprise.
Terms of Service | Licensing & Reprints | About Us | Privacy Policy | Advertise
Copyright 2014 QuinStreet Inc. All Rights Reserved.
10 of 10 5/21/2014 12:33 PM

Adding A Node To Oracle RAC 11g Release 2 - DatabaseJournal

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Adding A Node To Oracle RAC 11g Release 2 - DatabaseJournal

Uploaded by

Copyright:

Available Formats

Top White Papers and Webcasts

5 Reasons Your SQL Server is Slow

You might also like