Extending Virtualized

Oracle RAC Across

Data Centers:
True Active-Active Availability Over
Distance
Sam Lucido, EMC
Kannan Mani, VMware Inc.,

Copyright 2012 EMC Corporation. All rights reserved.

Agenda

Copyright 2012 EMC Corporation. All rights reserved.

Customer Challenges

What is Oracle Real Application

Clusters (RAC)

What happens when a server

fails using Oracle RAC

Transparent Application Failure

What is the single point of

failure with Oracle RAC

Customer Challenges
1. Application and database availability Many mission
critical systems demand 5 nines (99.999%) of uptime:
Per year no more than 6 minutes of unplanned downtime
2. Revenue Impact of unplanned downtime
Independent auditing of the revenue impact related to
unplanned downtime can range from 10s of thousands to
millions of dollars per 24 hours of outage
3. Broad application availability availability of the
entire application stack and not just the database
4. Rapid Provisioning and Greater Uptime New RAC
nodes are provisioned in minutes and vSphere HA along
with RAC provides greater Uptime
Copyright 2012 EMC Corporation. All rights reserved.

What Is Oracle Real Application Cluster?

Fault Tolerance
Scalability
Manageability
Rolling Upgrades

Most customers use

Oracle RAC for fault
tolerance and scalability

SAN Network
Physical RAC Nodes
Database Listener

RAC Interconnect

Real Application Clusters

(RAC)Is the federation of
servers to provide the
following benefits:

Database Listener

Database Listener

10 GbE Switch
SAN Storage Array

Copyright 2012 EMC Corporation. All rights reserved.

Stretched Oracle RAC Recovery Types

If A Server Fails..

A service called, Clusterware

reports a change in clusters
membership pool
In-flight transactions are
restarted on surviving nodes
Database scalability and
performance could temporarily be
reduced do to the loss of a server
in the cluster
Recovery involves reconfiguring
the RAC node back into the
cluster

SAN Network
Physical RAC Nodes
Database Listener

RAC Interconnect

Database Listener

Database Listener

10 GbE Switch
b

Copyright 2012 EMC Corporation. All rights reserved.

Stretched Oracle RAC and Fault Tolerance

SELECT: recovery of an database

transaction to a surviving node

SESSION: failover of a session

to a surviving node

While TAF can recover or failover

there might be an associated
delay
Applications must be designed to
use the OCI API to use TAF

Copyright 2012 EMC Corporation. All rights reserved.

SAN Network
Physical RAC Nodes
Database Listener

RAC Interconnect

Oracle RAC Increases Uptime

Each node in a RAC grid actively
services database sessions
Transparent Application Failover
(TAF) is an Oracle Call Interface
(OCI) that provides these failover
types:

Database Listener

Database Listener

10 GbE Switch
SAN Storage Array

Stretched Oracle RAC:

Redundancy At Most Every Layer

Layer of redundancy:

Physical

SAN Network

Federation of servers into

RAC nodes

Use of NIC bonding and

teaming
Software

Duplication of RAC services

across all nodes

Physical RAC Nodes

NIC
NIC

What is the Single Point of Failure?

However unlikely a storage array
failure represents a total
application outage
*Exception: ASM stretch mirroring

Copyright 2012 EMC Corporation. All rights reserved.

RAC Interconnect

NIC
NIC
NIC
NIC

10 GbE Switch
SAN Storage Array

How Does EMC VPLEX Address These

Challenges?
A Hardware And Software Solution

Both within and between datacenters

Even across two physically separate sites

Simplify Oracle RAC deployment

Solution is transparent to RAC and other applications

Turn off ASM protection, keep voting disk on virtual volume

Certified by Oracle

EMC worked with Oracle to certify this mutual solution

Enables Dynamic Workload Availability

Active / Active database and application access across datacenters

With virtualization the capability to non-disruptively move virtual machines

Copyright 2012 EMC Corporation. All rights reserved.

The Oracle Voting Disk

Oracle RAC Voting Disk on VPLEX

Copyright 2012 EMC Corporation. All rights reserved.

EMC VPLEX Stretched Oracle RAC

Metro Resiliency With Synchronous Access Anywhere

Oracle RAC manages
nodes across
interconnect
Virtual volume is
synchronously
replicated over VPLEX
interconnect
VPLEX Witness provides
failure resiliency via
storage monitoring
Is a required
component for
extended Oracle
RAC on VPLEX

Copyright 2012 EMC Corporation. All rights reserved.

Site A

RAC
Interconnect

LAN Backbone

Site B
LAN Backbone

Same Virtual
Volume
VPLEX
Interconnect

VPLEX Witness
3rd Domain

10

Stretched Oracle RAC Without EMC VPLEX

Third site for voting disk
using NFS or iSCSI

A Voting Disk is a shared

disk device or file used
to determine node
availability (establishes
quorum) and resolve
split-brain scenarios:
All nodes write to the
voting disk (check in)
Must be stored in a raw
partition or regular file
in a clustered file
system like ASM

Prevents splitbrain syndrome

Site A

Copyright 2012 EMC Corporation. All rights reserved.

LAN Backbone

RAC
Interconnect

Site B
LAN Backbone

11

Stretched Oracle RAC With EMC VPLEX

With VPLEX place the voting disk on
the virtual volume

Site A

RAC
Interconnect

LAN Backbone

LAN Backbone

VPLEX Witness
prevents split
brain syndrome
Simplified
Database
Architecture

Site B

Same Virtual
Volume
VPLEX
Interconnect

VPLEX Witness

Copyright 2012 EMC Corporation. All rights reserved.

12

Oracle Voting Disks: Recommendations

Oracle Clusterware
voting disks with
VPLEX Witness

Required
Should reside on
VPLEX virtual volumes
and not in an
independent 3rd
domain
VPLEX Witness
provides split-brain
protection

Copyright 2012 EMC Corporation. All rights reserved.

Oracle Clusterware
deployment

Create a unique disk

group for Oracle
Clusterware files
(+GRID)
The +GRID disk group
will benefit from using
Normal or High
redundancy
Relatively small if no
other database files

All other Oracle files

Use external
redundancy (array
based RAID) as this
should be adequate
protection for ASM
members

13

VMware Virtualization
Oracle RAC On VPLEX

Copyright 2012 EMC Corporation. All rights reserved.

14

Oracle Real Applications Clusters (RAC) VMware VMs

15

Oracle RAC on vSphere Deployment Process Chart

Create RAC
Database

Install Oracle
RAC

Install Oracle Grid

Infrastructure

Create Oracle RAC

Virtual Machine

Oracle RAC Deployment Process on VMware vSphere - VMFS

16

Install and Configure

VMware ESXi 5.0

Add VMFS VMDK OS and

Oracle Binary

Configure NTP client,

Storage and Networking
on all ESX servers

Create First Oracle RAC VM

(VMORARAC1)

Configure NTP Service

on GOS

Add two NIC Public and

Interconnect and set to
VMXNET3

Change Host Name and

assign new IP address

Add Shared Disks

following VMware KB 1034165

NOW
Save the VMORARAC1 VM
as Template

Create Three Nodes

Clone from Template

Change SCSI type to

Paravirtual. Add and
configure CRS/ Voting
Disks and format disks
using fdisk

Create and Add DAT,

REDO VMDKs Disks to
all RAC VMs. Change
SCSI type to Paravirtual

Format added VMDKs

using fdisk on Node 1

Complete Oracle RAC

Deployment on VMware

Create Custom Oracle RAC

Database

Create ASM Disk groups

for DATA and REDO

Install GOS, VMware

Tools and configure
Device Persistent

Install and Configure

Oracle Grid Infrastructure

Install Oracle RAC

Binares

Verify Grid Infrastructure

Run Cluster commands

RAC Node on vSphere Features vMotion for Maintenance

17

RAC Node on vSphere After Maintenance

18

Benefits of Oracle Databases on VMware

Performance

Server Consolidation

Rapid Provisioning

Workload Management

High Availability

Business Continuity

I/O is not an issue

Scale up and out

Fully utilize hardware

Maintain application isolation
Streamline activation, deployment, and validation of servers
Avoid manual configuration errors
Zero downtime maintenance
Migrate live databases
vMotion, HA, DRS
Without clustering or RAC
vCenter Site Recovery Manager
Hardware reduction at failover site

Oracle Redo Log

Latency
Oracle RAC On VPLEX

Copyright 2012 EMC Corporation. All rights reserved.

20

Oracle Database: Writing to the Redo Logs

Database
Transactions

Database

LGWR
(Log writer)

Write
Acknowledgement

Online
Redo Logs
ARCn
(Archive writer)

Copyright 2012 EMC Corporation. All rights reserved.

Database: Log writer latency

The Oracle database will not
commit transaction (insert,
update or delete) until write
acknowledgement has been
received from the online redo
logs
Archive
Archive
logfile
Archive
logfile
logfile

21

Oracle Online Redo Log Latency

Database
Transactions

Database

LOCAL Storage

REMOTE Storage

LGWR
(Log writer)
Online
Redo Logs

Write
Acknowledgement
EMC VPLEX Metro
*Redo log write latency = longest amount of time it takes to
acknowledge a write in remote storage
Copyright 2012 EMC Corporation. All rights reserved.

22

Round Trip Time

The Differentiators

Ideal For Oracle

Stretch RAC

1 2ms
Round trip time
(RTT)

Possible For Databases With

Low Transactional volume

3ms
Round trip time
(RTT)

4 - 5ms
Round trip time
(RTT)

All EMC proven

solutions use a
network latency of
2ms or less
The recommendation for VPLEX Metro is no more than 5ms RTT
latency over the wide area network. VPLEX Metro is certified for 5ms
RTT with Extended Oracle RAC.

Copyright 2012 EMC Corporation. All rights reserved.

23

VPLEX Metro With Stretched Oracle RAC

Transaction Rate

25000
20000
15000
10000

5000
0
1

Number of RAC Nodes

Copyright 2012 EMC Corporation. All rights reserved.

Tests Overview
OLTP workload
70% reads / 30% writes
2 RAC nodes local
Distance: 0 km
2 RAC nodes remote
(baseline)
VPLEX and RAC
Distance: 100
interconnect affected by
km (1ms RTT)
distance
Increase workload with
added nodes (16, 32, 48
& 64 drivers)
*EMC VPLEX Metro is RAC
Certified

24

Importance Of Write Order Fidelity

Consistency Groups

Write order fidelity and deterministic I/O continuation

All database LUNs (ie, Oracle data, control and log files) require preserving
write-order fidelity to maintain data integrity, and therefore should always be
placed in a single consistency group together.

Copyright 2012 EMC Corporation. All rights reserved.

25

VPLEX Resiliency
Oracle RAC On VPLEX

Copyright 2012 EMC Corporation. All rights reserved.

26

Failure Scenarios: VPLEX Resiliency

Solution

Physical RAC

Virtual RAC

Physical Single
Instance

Virtual Single
Instance

Loss of a server
(unplanned)

TAF

Manual restart or
automated if using
HA

Dependent on Boot
File System
replication

Manual restart or
automated if
using HA

Loss of a server (planned)

Manual restart at
surviving site

Live Migration

Dependent on Boot
File System
replication

Live Migration

Loss of VPLEX Witness

Loss of storage array

IO continues and Witness transparently rejoins

TAF

TAF + Cold
Migration or
automated if using
HA

Dependent on Boot
File System
replication

Manual restart or
automated if
using HA

No loss of database access in these scenarios. Possible impact on scalability and performance until system or storage recovery.

Copyright 2012 EMC Corporation. All rights reserved.

27

Oracle Support For VPLEX On Linux

Copyright 2012 EMC Corporation. All rights reserved.

28

Oracle Support for VPLEX on Unix

Copyright 2012 EMC Corporation. All rights reserved.

29

RAC on vSphere - History

Milestones
Oracle support statement extension 11.2.0.2 Fall 2010
MyOracleSupport.com Note 249212.1
For Oracle RAC, Oracle will only accept
Service Requests as described in this note on
Oracle RAC 11.2.0.2 and later releases.

Release of the Oracle Databases on VMware - RAC Deployment

Guide Jan 2011

30
Copyright 2012 EMC Corporation. All rights reserved.

30

EMC Proven Solutions For Oracle

Oracle Runs Faster And More Efficiently On EMC and VMware

Transformation

Cloud

Tested, Proven Results

Up to 3X
Performance
at Less Cost

40% More
Oracle CPU
Per Server

Copyright 2012 EMC Corporation. All rights reserved.

80% Faster
Provisioning

Backup With
30x Less
Storage

Replicate
And Recover
In Minutes

31

Visit Online Oracle at EMC Community

Search here!

Participate in our
discussions

Find EMC Experts, Tradeshows and

learn more about virtualizing Oracle

Blogs, Polls, Events and Videos: One place for all EMC Oracle solutions!

http://www.emc.com/everythingoracle

Copyright 2012 EMC Corporation. All rights reserved.

32