Solutions for High Availability and Disaster Recovery with MySQL

Detlef Ulherr Sun Microsystems

USE

IMPROVE

EVANGELIZ E

USE

IMPROVE

EVANGELIZ E

Why Is High Availability Important?

USE

IMPROVE

Why is HA Important?

EVANGELIZ E

Downtime is costly to the business Components or systems failures are real possibilities Hardware, Software, Human Error, Disaster Automated recovery is the goal With a single physical system, single points of failure have greater impacts Network card dies, CPU misbehaves, Disk drive crashes ... HA Clusters and Solaris automate the recovery process from inevitable

USE

IMPROVE

EVANGELIZ E

HA Architectures for MySQL

USE

IMPROVE

HA Architectures for MySQL

Shared nothing MySQL replication MySQL Cluster Solaris Cluster / Open HA Cluster (planned) Shared disk Solaris Cluster / Open HA Cluster VCS, HACMP, MC Serviceguard and similar.

EVANGELIZ E

USE

IMPROVE

HA Architectures for MySQL (cont.)

Replicated Disk Linux + DRBD

EVANGELIZ E

Solaris Cluster/Open HA cluster + AVS (planned) Solaris Cluster/Open HA cluster + SRDF/Truecopy

USE

IMPROVE

EVANGELIZ E

MySQL and Business Continuity

USE

IMPROVE

MySQL and Business Continuity

Building Blocks

EVANGELIZ E

Local HA Infrastructure Replication Networks Nameservice DR Framework

USE

IMPROVE

MySQL and Business Continuity

Building Blocks

EVANGELIZ E

Most important: People and Processes Training Management Periodic DR tests

USE

IMPROVE

EVANGELIZ E

MySQL's Availability Features

10

USE

IMPROVE

MySQL's Availability Features

Replication
Asynchronous Read mostly scale out

EVANGELIZ E

MySQL Cluster
Synchronous High write profile (Commit latency)

11

USE

IMPROVE

When to use MySQL Cluster

EVANGELIZ E

Your database design is good for horizontal partitioning. The cluster should provide HA for MySQL only. You can tolerate downtimes while adding data nodes. The memory is large enough for at least twice all the indexes.

12

USE

IMPROVE

EVANGELIZ E

MySQL and Solaris Cluster / Open HA Cluster

13

USE

IMPROVE

EVANGELIZ E

Open HA Cluster Benefits

A shared disk HA is simpler than replication or MySQL Cluster Provides automated failover Can support larger databases than MySQL Cluster Can be extended with the rest of the application stack in the same Cluster. MySQL Cluster has no framework to integrate applications
14

USE

IMPROVE

EVANGELIZ E

Open HA Cluster Benefits cont.

The Open HA Cluster MySQL agent leverages all the rich infrastructure features of Open HA Cluster (strong membership, dependencies, ...) Can support Campus and Metro Clusters up to 700 km Supports MySQL replication in all its flavors

15

USE

IMPROVE

Open HA Cluster Stack

EVANGELIZ E

Applications
Heartbeats Membership

Agents Cluster Infrastructure Operating System

16

USE

IMPROVE

EVANGELIZ E

Failover Service

HA Failover Service

Application failover Within nodes or container Containers failover Failover of volumes/filesystems Failover IP address

17

USE

IMPROVE

EVANGELIZ E

Scalable Service

Global Network Servi

Provides Global IP addre with failure protection

Software Load Balancing

Scalable Service

18

USE

IMPROVE

Example: Apache and MySQL

EVANGELIZ E

Global Network Servi

Provides Global IP addre with failure protection

MySQL (Failover)

Apache (Scalable)

19

USE

IMPROVE

Solaris Cluster Architecture

Software Load Balancing

EVANGELIZ E

Scalable Service

Global Network Servi

Quorum Server

Provides Global IP addre with failure protection

Monitoring

Heartbeats Membership Quorum Disk Fencing Global File Service Failover File

HA Failover Service

Application failover Within nodes or container Containers failover Failover IP address

Resource Group Manager

Resource (application) dependencies Inter RG dependencies RG affinities

20

USE

IMPROVE

Open HA Cluster and Virtualisation

Cluster control is in the global zone Failover xVM Failover Container Zone Nodes

EVANGELIZ E

Cluster control is in the virtualization entity Cluster in LDOMS Zone Cluster

21

USE

IMPROVE

EVANGELIZ E

Cluster Agents

22

USE

IMPROVE

Open HA Cluster Stack

EVANGELIZ E

Applications Agents Cluster Infrastructure Operating System

23

USE

IMPROVE

Cluster Agents (Data Services)

EVANGELIZ E

Applications run on cluster unmodified (off-the-shelf if it complies to some rules) Cluster Agents are the glue layer between applications and cluster infrastructure Application cannot break into the high availability business on Solaris Cluster without an agent!

24

USE

IMPROVE

EVANGELIZ E

MySQL (Shared Disk)

Client MySQL Storage Log. Host Node1(:Zone 1) Node2(:Zone 2)

25

USE

IMPROVE

EVANGELIZ E

MySQL (Replicated Disk) Global Zone

MySQL Storage Log. Host Node1 Node2

Storage based replication Truecopy SRDF 26

USE

IMPROVE

EVANGELIZ E

MySQL (Shared Disk) Master - Slave

Client

MySQL1 MySQL2 Mys Repl Storage Log. Host Storage Log. Host Node1 : Zone1 Node1 : Zone3 Node2 : Zone2 Node2 : Zone4

27

USE

IMPROVE

EVANGELIZ E

MySQL (Shared Disk) Failover Zone

MySQL Zone1 Storage Log. Host Node1 Node2

28

USE

IMPROVE

EVANGELIZ E

MySQL (Shared Disk) Zone Cluster

MySQL Storage Log. Host Node1:Zon e-cl Node2:Zon e-cl

29

USE

IMPROVE

EVANGELIZ E

MySQL (Shared Disk) Multiple Clusters

Cluster 1
MySQL

Cluster 2 MySQL Replication MySQL

Storage Log. Host

Storage Log. Host

Node1 (Zone) Node2 (Zone) Node3 (Zone) Node4 (Zone)

30

MySQL (Shared Disk) replication outside a cluster

Cluster 1
MySQL

USE

IMPROVE

EVANGELIZ E

Node3 (Zone) MySQL Replication

Storage Log. Host

Node1 (Zone) Node2 (Zone)

31

Geographic Clustering
Admin. Client

USE

IMPROVE

EVANGELIZ E

Multi-Cluster and Multi-Site capability

User Client

Site 1
Cluster Replication Net
Optional storage network

Site 2
IP Net Inter/Int ra Cluster Replication Net

Storage
Optional heartbeat networks

Dial, Sat Net, etc.

Storage

32

USE

IMPROVE

EVANGELIZ E

MySQL and Open HA Cluster Future Configurations

33

USE

IMPROVE

Can give the option to use MySQL replication as a replacement for shared storage Can be extended to a full DR solution using MySQL replication as the replication protocol Can do loadbalancing between scalable slaves

Solaris Cluster / OHAC Future Benefits

EVANGELIZ E

34

USE

IMPROVE

Can be extended to integrate the rest of the MySQL features like MySQL Cluster Can leverage the Shared Address resource type to do load balancing for SQL nodes

Solaris Cluster / OHAC Future Benefits cont.

EVANGELIZ E

35

Geographic Clustering
Admin. Client

USE

IMPROVE

EVANGELIZ E

Multi-Cluster and Multi-Site capability

User Client

Site 1
Cluster Replication Net
Optional storage network

Site 2
IP Net Inter/Int ra Cluster Replication Net

Storage
Optional heartbeat networks

MySQL replication is available soon Dial,

Sat Net, etc.

Storage

36

USE

IMPROVE

EVANGELIZ E

MySQL Scalable Slaves

Client Shared Addr. Loadbalancer
MySQL-Slave

MySQL-Master

MySQL replication

Storage

Storage Log. Host

MySQL-Slave

Storage

37

USE

IMPROVE

EVANGELIZ E

MySQL Scalable Slaves

MySQL replication
MySQL-Slave

MySQL-Master

Shared Addr.

Storage Log. Host

MySQL-Slave

Storage

Storage

Node1(:zone1 )

Node2(:zon e2)

38

USE

IMPROVE

EVANGELIZ E

MySQL Shared Nothing (future)

Rolechanger Log. Host Quorum Server

MySQL-P Node1 : Zone 1 Master

MySQL replication

MySQL-S Node2 : Zone2 Slave

39

MySQL Cluster (Shared Nothing) Local/Global Zone (future) Shared Addr

USE

IMPROVE

EVANGELIZ E

Node 6,7 Mys mgt

Node5,6,7 Node1 logh hsp Mys query Mys data Node5,6,7 Node2 Mys query logh hsp Mys data Node3 Node5,6,7 hsp Mys query Mys data logh hsp Node4 hsp Mys data logh hsp hsp SQL-Node Data-Node hsp SQL-Node Data-Node SQL-Node Data-Node Data-Node ...

40

USE

IMPROVE

References Cluster References

http://www.opensolaris.org/os/community/ha-clusters/ohac/ http://www.opensolaris.org/os/project/ha-mysql/ http://docs.sun.com/app/docs/prod/sun.cluster32#hic Blogs
http://blogs.sun.com/sc/ http://blogs.sun.com/krishs/entry/deployment_and_failover_study_of Blueprints

EVANGELIZ E

http://wikis.sun.com/display/BluePrints/High+Availability+MySQL+Database+Replication+with+Solaris+Zone+Cluster

41

USE

IMPROVE

EVANGELIZ E

Thank you!
Detlef Ulherr detlef.ulherr@sun.com

open artwork and icons by chandan: http://blogs.sun.com/chandan