Under

the Hood of
the Next GeneraEon of
Oracle Real ApplicaEon Clusters

Markus Michalewicz Markus.Michalewicz@oracle.com

Senior Director of @OracleRACpm
Product Management, hRp://www.linkedin.com/in/markusmichalewicz
Oracle RAC Development hRp://www.slideshare.net/MarkusMichalewicz
November 12th, 2016

Copyright 2016, Oracle and/or its aliates. All rights reserved. |

Under the Hood of
Oracle Real ApplicaEon Clusters
(RAC) 12c Release 2

Markus Michalewicz Markus.Michalewicz@oracle.com

Senior Director of @OracleRACpm
Product Management, hRp://www.linkedin.com/in/markusmichalewicz
Oracle RAC Development hRp://www.slideshare.net/MarkusMichalewicz
November 12th, 2016

Copyright 2016, Oracle and/or its aliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direcEon. It is intended for
informaEon purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or funcEonality, and should not be relied upon
in making purchasing decisions. The development, release, and Eming of any features or
funcEonality described for Oracles products remains at the sole discreEon of Oracle.

Copyright 2016, Oracle and/or its aliates. All rights reserved. | 4

Oracle Database 12c Rel. 2 Real ApplicaEon Clusters (RAC)
Its all about scalability, availability and ecient management

BeRer availability
BeRer scalability Ecient management
(due to reduced
(for singleton services) for large scale deployments
reconguraEon Emes)

Copyright 2014 Oracle and/or its aliates. All rights reserved. | 5

Oracle Database 12c Rel. 2 Real ApplicaEon Clusters (RAC)
Its all about scalability, availability and ecient management

BeRer availability
BeRer scalability Ecient management
(due to reduced
(for singleton services) for large scale deployments
reconguraEon Emes)

Copyright 2014 Oracle and/or its aliates. All rights reserved. | 6

Oracle RAC Scalability A Brief Review

Oracle RAC scalability
is independent of the Oracle RAC scales
number of nodes most of the
does not require enterprise soluEons
applicaEon changes used today
(unlike sharding)

Oracle RAC scales

Oracle RAC scales
Oracle Database
Oracle MulEtenant
In-Memory

Copyright 2014 Oracle and/or its aliates. All rights reserved. | 7

Oracle RAC Scalability More InformaEon
hRp://www.slideshare.net/MarkusMichalewicz/

hRp://www.slideshare.net/MarkusMichalewicz/
oracle-rac-customer-proven-scalalbility
oracle-rac-internals-the-cache-fusion-ediEon
NEW:

hRp://www.slideshare.net/MarkusMichalewicz/
hRp://www.slideshare.net/MarkusMichalewicz/
applicaEon-development-best-pracEces-for-oracle-
paper-oracle-rac-internals-the-cache-fusion-ediEon
real-applicaEon-clusters-rac

www.slideshare.net/MarkusMichalewicz/oracle- hRp://www.slideshare.net/MarkusMichalewicz/
mulEtenant-meets-oracle-rac-ioug-2014-version oracle-database-inmemory-meets-oracle-rac

h'p://www.slideshare.net/MarkusMichalewicz
Copyright 2014 Oracle and/or its aliates. All rights reserved. | 8
Oracle RAC 12c Release 2 Scaling in Two Dimensions

Improved scaling for all-HUB, Standalone Clusters Flex Cluster-based Scaling

Copyright 2014 Oracle and/or its aliates. All rights reserved. | 9

Oracle RAC 12c Release 2 Scaling in Two Dimensions

Improved scaling for all-HUB, Standalone Clusters Flex Cluster-based Scaling

Copyright 2014 Oracle and/or its aliates. All rights reserved. | 10

OpEmized Singleton Workload Scaling

Service-oriented Buer Cache Access Pluggable Database and Service IsolaNon

determines the data (on database object level) accessed by the improves performance by reducing DLM operaEons for PDBs /
service and masters this data on the node on which the (singleton) Services not oered in all instances and opEmizing block
service is oered, which improves data access performance. operaEons based on in-memory block separaEon.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 11

Oracle RAC 12c Release 2 Scaling in Two Dimensions

Improved scaling for all-HUB, Standalone Clusters Flex Cluster-based Scaling

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 12

Oracle RAC 12c Release 2 Scaling in Two Dimensions

Improved scaling for all-HUB, Standalone Clusters Flex Cluster-based Scaling

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 13

Oracle Flex Cluster - A Brief Review

Introduced during OOW 2013: Recommended during OOW 2014:

hRp://www.slideshare.net/ The standard going forward
MarkusMichalewicz/ hRp://www.slideshare.net/ (every Oracle 12c Rel. 2 cluster
understanding-oracle-rac-12c- MarkusMichalewicz/oracle- is a Flex Cluster by default.)
internals-oow13-con8806 rac-12102-operaEonal-best-pracEces

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 14

Under the Hood: Any New Install Ends Up in a Flex Cluster

[GRID]> crsctl get cluster name

CRS-6724: Current cluster name is 'SolarCluster'

[GRID]> crsctl get cluster class
CRS-41008: Cluster class is 'Standalone Cluster'

[GRID]> crsctl get cluster type
CRS-6539: The cluster type is 'ex'.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 15

Under the Hood: Any New Cluster is a Flex Cluster

gridSetup.sh can be
used to add nodes!

ADVM proxy only

online when needed.

Aier the installaEon, any Oracle RAC 12.2 Cluster A GNS (just an IP, no domain-delegaEon) is required
is an all-HUB Flex Cluster, using for Leaf nodes to nd HUBs in a Flex Cluster. If Leaf
Flex ASM with count=3 (count = all aier upgrade). nodes are added later, a GNS must be added rst.
This setup compares to the pre-12.2 standard cluster.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 16

Oracle Flex Cluster the Scalable Architecture
** A Leaf node
failure does not
impact any
other node.

* Read-only WL
on Leaf-instances
will scale.

Use Case 1: Massive Parallel Query RAC Use Case 2: RAC Reader Nodes
Overlay your Hadoop Cluster (HDFS) with an Oracle Flex Use Read-Only workload (WL) on read-mostly Leaf node
instances for adoc data analysis scaled across hundreds of
Cluster to access data in Hadoop via SQL and perform nodes with no delay in accessing updated data, without any
cross-data (adhoc) analysis using standard interfaces. impact on OLTP performance* and with beRer HA**

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 17

Run a Database Instance on a Leaf Node PreparaEon

Connect Leaf nodes to storage Install Oracle Database Home Extend public network to Leaf(s)
Leaf nodes for applicaEons do not on all nodes and as needed. For RAC Reader Nodes use case only,
require direct storage access; If you ever want to run a database enable a public network connecEon
running database instances on Leaf instance on a Leaf node, it needs a on Mars by extending the network
nodes does. database home as any other node. and listener resources to the leaf.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 18

Run a Database Instance on a Leaf Node DB CreaEon

Serverpool OLTP was

Policy management pre-created using
allows for an easy re- the oracle user.
assignment of a Leaf
nodes to other tasks.

Create a Policy-Managed RAC DB For Massive Parallel Query RAC, For RAC Reader Nodes,
RAC Reader Nodes as well as Massive create new server pools along Create database on HUB nodes
Parallel Query RAC require a Policy- with the database. the addiEon of database instances
Managed database. Admin-managed Make sure to create a on Leaf nodes is dynamic and
DBs cannot be extended to Leafs. Parallel Query Server Pool. managed via command line.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 19

Run RAC Reader Nodes FinalizaEon

(Re-)starEng the
OLTPWL Service
nalizes the DWHWL
service setup.
For RAC Reader Nodes, Summary Connect

add a Reader Farm (RF) pool Note that if a Leaf node is used for
to the system using the add Massive Parallel Query RAC, it should
service command (dynamic). not allow for direct connecEons to
the Leaf node instance.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 20

Run Oracle Database In-Memory on Leaf Node Instances

select INST_ID, pool, alloc_bytes, alloc_bytes,

used_bytes from GV$INMEMORY_AREA;

The IMDB Colum Column Stores

A min. 100MB
Store will be need to be equally
Column Store size
acEvated aier sized across all
is required.
instance restart. instances.
Its easy to run Oracle Database
In-Memory on Leaf Nodes: Emphasizing Leaf Node Usage
Oracle Database In-Memory
ideal for RAC Reader Nodes alter system set by using instance-specic
inmemory_size=100M
scope=spfile sid=*; sesngs is work in progress

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 21

Oracle Database 12c Rel. 2 Real ApplicaEon Clusters (RAC)
Its all about scalability, availability and ecient management

BeRer availability
BeRer scalability Ecient management
(due to reduced
(for singleton services) for large scale deployments
reconguraEon Emes)

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 22

Oracle RAC 12c Rel. 2 Three Dimensions of Availability

Availability due to
Improved availability for all-HUB, Flex Cluster-based availability
Autonomous Health Framework
Standalone Clusters here: Node WeighNng
conNnuously working for you

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 23

Oracle RAC 12c Rel. 2 Three Dimensions of Availability

Availability due to
Improved availability for all-HUB, Flex Cluster-based availability
Autonomous Health Framework
Standalone Clusters here: Node WeighNng
conNnuously working for you

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 24

OpEmized (Singleton) ReconguraEon Time

4x
faster
Pluggable Database and Service IsolaNon Near Zero DownNme Recong. via Buddy Instances
improves availability by ensuring that instance failures of which track modied data blocks on other nodes to quickly
instances only hosEng singleton PDBs will not impact other idenEfy blocks requiring recovery, which allows for rapid
instances of the same RAC-based CDB. processing of new transacEons in case recovery is needed.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 25

Oracle RAC 12c Rel. 2 Three Dimensions of Availability

Availability due to
Improved availability for all-HUB, Flex Cluster-based availability
Autonomous Health Framework
Standalone Clusters here: Node WeighNng
conNnuously working for you

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 26

Node EvicEon Basics
h'p://www.slideshare.net/MarkusMichalewicz/oracle-clusterware-node-management-and-voNng-disks

Pre-12.2, node evicEon follows

a rather ignorant paRern

Example in a 2-node cluster: The node

with the lowest node number survives.

1 2 Customers must not base their

applicaEon logic on which node
survives the split brain.
As this may(!) change in future releases

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 27

Node WeighEng in Oracle RAC 12c Release 2
Idea: Everything equal, let the majority of work survive

Node WeighEng is a new feature that considers

the workload hosted in the cluster during fencing

1 2
The idea is to let the majority of work survive,
if everything else is equal
Example: In a 2-node cluster, the node hosEng the
majority of services (at fencing Eme) is meant to survive

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 28

Lets Dene Equal
Public network
Conict.
card failure.

A three node cluster
will benet from Node WeighEng, Secondary failure consideraNon
if three equally sized sub-clusters are A fallback scheme
can inuence which node survives.
built as s result of the failure, since is applied if consideraEons do not
two dierently sized sub-clusters are Secondary failure consideraEon
lead to an acEonable outcome.
not equal. will be enhanced successively.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 29

CSS_CRITICAL Fencing with Manual Override
Node evicEon
despite WL; WL Conict.
will failover.

srvctl modify database -help
|grep criEcal

-css_criEcal {YES | NO}
Dene whether the database
or service is CSS criEcal

crsctl set server

css_criNcal {YES|NO}
+ server restart

CSS_CRITICAL CSS_CRITICAL will be honored

can be set on various levels / if no other technical reason A fallback scheme is applied if
components to mark them as prohibits survival of the node CSS_CRITICAL sesngs do not lead
criEcal so that the cluster will try to which has at least one criEcal to an acEonable outcome.
preserve them in case of a failure. component at the Eme of failure.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 30

Last but Not Least Leaf Node Failover

Leaf nodes require at least one Hub node

in the cluster to which they can connect.
Oracle RAC
Oracle GI | Leaf
Mars If a Hub node fails, all Leaf nodes connected to
the failed Hub node re-connect to another Hub.
Failover is transparent
on cluster level.
on the Leaf nodes.
Oracle RAC Oracle RAC
for instances running on the Leaf nodes.
Oracle GI | HUB Oracle GI | HUB
Venus Earth

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 31

Oracle Database 12c Rel. 2 Real ApplicaEon Clusters (RAC)
Its all about scalability, availability and ecient management

BeRer availability
BeRer scalability Ecient management
(due to reduced
(for singleton services) for large scale deployments
reconguraEon Emes)

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 32

BeRer Management Thanks to Your Feedback

Congure ASM on NFS

$ORACLE_HOME/gridSetup.sh

Using these and other technologies, an average (2-4 nodes) cluster

can be installed in less than an hour with proper preparaEon.

Separate Diskgroup for Grid

gridSetup and zip-based install ASM Management for Infrastructure Management
allow for a simple unzip to install NFS-based Clusterware les Repository (GIMR)
the Grid Home and node for easier management and allows for more exibility during
management (addNode) thereaier thereby beRer availability. Grid Infrastructure InstallaEon

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 33

What IfYou have Hundreds of Environments
And what if
Soiware InstallaEon
Storage conguraEon
DiagnosEcs setup

would have to be performed only once
and can then be re-used mulNple Nmes?

allowing you to save many hours
performing these Ering tasks?

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 34

SimplicaEon and Eciency through CentralizaEon
Only some of the benets of the new Cluster Domain-based management
FacilitaEng
Rapid Home Provisioning (RHP)
AutomaEc Storage Management (ASM)
Autonomous Health Framework (AHF)

hosted on a dedicated cluster

the Domain Services Cluster (DSC)

RHP AHF all three management tasks can be
centralized and diagnosEcs can be opEmized

Cluster for Member Clusters in the Cluster Domain.
Domain Domain Services Cluster



Copyright 2015, Oracle and/or its aliates. All rights reserved. | 35
 Cluster Domain
Database ApplicaNon Database Database
Member Cluster Member Cluster Member Cluster Member Cluster

Uses IO & ASM Uses ASM

Private Uses local ASM GI only Service of DSC Service
Network

SAN

NAS Domain Services Cluster
Mgmt Trace File Rapid Home
Repository Analyzer AddiNonal
Provisioning OpNonal ASM
(GIMR) (TFA) (RHP) Service IO Service
Service Services
Service Service

Shared ASM

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 36

How to Create a Cluster Domain

Congure an Oracle
Domain Services Cluster

The Member Cluster

will use only services
for which it has
credenEals.

Congure an Oracle Create a credenNal le for each Run gridSetup on the server on
Domain Services Cluster (DSC) Member Cluster you want to which you want to run the
as part of the gridSetup-based deploy and make it accessible to Member Cluster install and
install. A DSC install follows the the server on which you will run provide access to the credenNal
Standalone Cluster install. the Member Cluster install. le when requested. Then follow
the instrucEons on the screen.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 37
Proven Features Even More Benecial on the DSC

Autonomous Health Framework Oracle ASM 12c Rel. 2 based

The DSC is the ideal hosEng storage consolidaNon is best
works more eciently for you on
environment for Rapid Home performed on the DSC, as it
the DSC, as conEnuous analysis is
Provisioning (RHP). enables numerous addiEonal
taken o the producEon cluster.
features and use cases.

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 38

Summary
Oracle RAC 12c Rel. 2 provides
BeRer Scalability
Inherently as part of the database
Via making Flex Cluster the standard

BeRer Availability
Inherently as part of the database
On cluster-level via Node WeighEng

Ecient Management for large scale deployments
Inherently as part of the the Installer and RHP
Via Cluster Domain-based Management

Copyright 2015, Oracle and/or its aliates. All rights reserved. | 39