Setting up Multi-Source Replication

in MariaDB 10.0
November 3, 2014

Derek Downey
MySQL Principal Consultant

2014 Pythian
Who am I?
Web Developer and Sysadmin background
MySQL DBA for 10+ years
MySQL Principal Consultant at Pythian

2014 Pythian
About Pythian

10,000
Pythian currently manages more than 10,000
systems.

300
Pythian currently employs more than 300
200+ leading brands trust us to keep their systems fast,
reliable, and secure
Elite DBA & SysAdmin workforce: 7 Oracle ACEs, 2 Oracle
ACE Directors, 5 Microsoft MVPs, 1 Cloudera Champion of
Big Data More than any other company, regardless of head
people in 25 countries worldwide. count
Oracle, Microsoft, MySQL, Hadoop, Cassandra, MongoDB,
and more.

1997
Infrastructure, Cloud, DevOps, and application expertise
Big data practice includes architects, R&D, data scientists,
and operations capabilities
Zero lock-in, utility billing model, easily blended into existing
teams.
Pythian was founded in 1997

2014 Pythian
Overview
What is multi-source replication?
Why would we use multi-source replication?
How multi-source replication is implemented in MariaDB
Introduction to Vagrant / AWS environment
Set up mirrored environment
Work through some issues, conflicts and failures
Set up a fan-in environment
Limitations, issues, and missing features
Q&A

2014 Pythian
Hands-on access
Vagrant installation
requires VirtualBox and Vagrant 1.5+
git: https://github.com/dtest/pluk14-mariadb

2014 Pythian
What is Multi-source Replication

2014 Pythian
Multi-source Replication

2014 Pythian
Why Multi-source Replication

2014 Pythian
MySQL Topology
Fragile
Doesnt scale

Stand Alone
But pity anyone who falls and
has no one to help them up
- Ecclesiastes 4:10

2014 Pythian
Concept #1 - binary log
Contains DDL/DML executed (successfully?) on master
Required for replication
Point-in-time recovery

2014 Pythian
MySQL Topology
Slave roles
Failover
Read scaling

Master/Slave
2014 Pythian
Concept #2 - asynchronous
Changes on master dont wait for acknowledgement from
slave
Slave lag: unexpected results from read slaves
At best: semi-synchronous (5.5+)

2014 Pythian
MySQL Topology
Can affect performance of
master
Complexity of failover

Master/Slave
2014 Pythian
MySQL Topology

Master/Slave
2014 Pythian
MySQL Topology

Active Master/Passive Master

2014 Pythian
MySQL Topology

Active Master/Passive Master

2014 Pythian
MySQL Topology

Active Master/Active Master

2014 Pythian
MySQL Topology

Disaster Recovery
2014 Pythian
MySQL Topology

Mirrored
2014 Pythian
Concept #3
MySQL slaves can only have a single master

2014 Pythian
MySQL Topology

Ring
2014 Pythian
MySQL Topology

Shards
2014 Pythian
MySQL Topology

Fan-in

2014 Pythian
MariaDB Multi-source Replication
Implementation

2014 Pythian
MariaDB Multi-Source
Replication
Still requires binary log
Still asynchronous
Works with either binlog coordinates or MariaDB GTID
Removes single master constraint
No conflict resolution
Replication filters can be different per connection
Relay log and info files are generated per connection

2014 Pythian
MariaDB New Syntax
CHANGE MASTER [connec2on_name] ...
FLUSH RELAY LOGS ['connec4on_name']
RESET SLAVE ['connec2on_name']
SHOW RELAYLOG ['connec4on_name'] EVENTS
SHOW SLAVE ['connec2on_name'] STATUS
SHOW ALL SLAVES STATUS
START SLAVE ['connec2on_name'...]]
START ALL SLAVES
STOP SLAVE ['connec2on_name']
STOP ALL SLAVES ...

2014 Pythian
MariaDB Variables
default_master_connection:
connection_name to use if an explicit name is not
provided
is dynamic
log-basename: use to avoid missing files (error, binlog,
relaylog, pid, socket, etc)
replication filters:
prefix connection: main_connection.replicate_do_db
if no prefix, variable is used for all connections that
dont have variable set.

2014 Pythian
MariaDB New Files
multi-master-info-file: contains all master connections
in use
master--(connection).info: Current master position for
the connection that has been applied in the slave
relay-log-(connection).xxxxx: relay logs for a given
connection
relay-log-index-(connection).index: All relay logs for a
given connection
relay-log-(connection).info: Current master position for
the relay log (not applied)

2014 Pythian
MariaDB New Status Variables
Com_start_all_slaves
Com_start_slave: replaces Com_slave_start
Com_stop_slave: replaces Com_slave_stop
Com_stop_all_slaves

2014 Pythian
GTID Replication
3 numbers separated by dashes: eg 1-1-10
first number: domain id
second number: server id
third number: sequence number; increment per event
Domain IDs:
must be unique on multi-masters to guarantee binlog
order of each stream
slaves keep track of binlog position for each
domainID

2014 Pythian
GTID Replication
Can switch from normal coordinates to GTID after slave
first connects to a master that tracks GTIDs
MASTER_USE_GTID:
slave_pos: last replicated gtid (for the domain)
current_pos: includes events in master binlog
no: not using gtid

2014 Pythian
GTID: slave_pos vs current_pos
slave_pos: SELECT @@g)d_slave_pos;
last replicated gtid
Use if changes on slave should not affect GTID
replication position
current_pos: SELECT @@g)d_current_pos;
includes events in binlog not replicated; server was
previously a master
no need to know previous role of server

2014 Pythian
Vagrant Instances

Hands-on

2014 Pythian
Logging In
Vagrant

# star4ng/stopping instances
$ cd pluk-mariadb
$ vagrant up
$ vagrant ssh pluk
$ sudo su - pluk

2014 Pythian
Environment Overview
mysqld_mul2: 4 instances
cong le: /etc/my.cnf
datadir: /opt/mysql/(instance)
master1: port 3306
slave1: port 3307
master2: port 3308
slave2: port 3309

2014 Pythian
Interacting with MariaDB
# star4ng/stopping instances
$ sudo service mysqld_mul4 {start|stop|restart|report} [1-4]

# connec4on bash aliases
$ mysql_master1
$ mysql_slave1
$ mysql_master2
$ mysql_slave2

2014 Pythian
Setting up a Mirrored Environment

Hands-on

2014 Pythian
MySQL Topology

Mirrored
2014 Pythian
Script Repository
pluk$ cd /home/pluk/scripts/mirrored

2014 Pythian
slave1 - slave of master1
pluk$ cat 1_slave1.sql
pluk$ mysql_slave1


(root@slave1.sock) [(none)]> CHANGE MASTER 'master1' TO
MASTER_HOST='127.0.0.1', -- cant use unix socket
MASTER_PORT = 3306, -- master1
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

(root@slave1.sock) [(none)]>START SLAVE 'master1';

2014 Pythian
slave1 - slave of master1
(root@slave1.sock) [(none)]>START SLAVE 'master1';

(root@slave1.sock) [(none)]>SHOW SLAVE STATUS \G
Empty set (0.00 sec)

2014 Pythian
slave1 -
default_master_connection
(root@slave1.sock) [(none)]>SELECT @@default_master_connec4on;
+--------------------------------------------+
| @@default_master_connec4on |
+--------------------------------------------+
| |
+--------------------------------------------+
1 row in set (0.00 sec)

(root@slave1.sock) [(none)]>SET @@default_master_connec4on='master1';
Query OK, 0 rows aected (0.00 sec)

(root@slave1.sock) [(none)]>SHOW SLAVE STATUS \G

2014 Pythian
slave1 - multiple files
(root@slave1.sock) [(none)]>\! ls -l /opt/mysql/slave1/*info
-rw-rw---- 1 mysql mysql 110 Oct 22 00:42 /opt/mysql/slave1/master-master1.info
-rw-rw---- 1 mysql mysql 8 Oct 22 00:26 /opt/mysql/slave1/mul4-master.info
-rw-rw---- 1 mysql mysql 61 Oct 22 00:42 /opt/mysql/slave1/relay-log-master1.inf

2014 Pythian
master1 - slave of slave1
pluk$ cat 2_master1_s1.sql
pluk$ mysql_master1


(root@master1.sock)
[(none)]> CHANGE MASTER 'slave1' TO
MASTER_HOST='127.0.0.1',
MASTER_PORT = 3307, -- slave1
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

(root@master1.sock) [(none)]>START SLAVE 'slave1';

(root@master1.sock) [(none)]>SHOW SLAVE 'slave1' STATUS \G

2014 Pythian
slave2 - slave of master2
pluk$ cat 3_slave2.sql
pluk$ mysql_slave2


(root@slave2.sock) [(none)]> CHANGE MASTER 'master2' TO
MASTER_HOST='127.0.0.1',
MASTER_PORT = 3308, -- master2
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

(root@slave2.sock) [(none)]>START SLAVE 'master2';

(root@slave2.sock) [(none)]>SHOW SLAVE 'master2' STATUS \G

2014 Pythian
master2 - slave of slave2
pluk$ cat 4_master2_s2.sql
pluk$ mysql_master2


(root@master2.sock)
[(none)]> CHANGE MASTER 'slave2' TO
MASTER_HOST='127.0.0.1',
MASTER_PORT = 3309, -- slave2
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

(root@master2.sock) [(none)]>START SLAVE 'slave2';

(root@master2.sock) [(none)]>SHOW SLAVE 'slave2' STATUS \G

2014 Pythian
MySQL Topology

Pairs
2014 Pythian
MySQL Topology

Mirrored
2014 Pythian
master2 - slave of master1
pluk$ cat 5_master2_m1.sql
pluk$ mysql_master2


(root@master2.sock) [(none)]> CHANGE MASTER 'master1' TO
MASTER_HOST='127.0.0.1',
MASTER_PORT = 3306, -- master1
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

(root@master2.sock) [(none)]>START SLAVE 'master1';

(root@master2.sock) [(none)]>SHOW SLAVE 'master1' STATUS \G

2014 Pythian
master1 - slave of master2
pluk$ cat 6_master1_m2.sql
pluk$ mysql_master1


(root@master1.sock) [(none)]> CHANGE MASTER 'master2' TO

MASTER_HOST='127.0.0.1',
MASTER_PORT = 3308, -- master2
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

(root@master1.sock) [(none)]>START SLAVE 'master2';

(root@master1.sock) [(none)]>SHOW SLAVE 'master2' STATUS \G

2014 Pythian
MySQL Topology

Mirrored
2014 Pythian
Handling Failures

Hands-on

2014 Pythian
MySQL Topology

Mirrored
2014 Pythian
Multi-master Conflict
STOP SLAVE;
SET GTID_NEXT="7d72f9b4-8577-11e2-a3d7-080027635ef5:5";
BEGIN; COMMIT;
SET GTID_NEXT="AUTOMATIC";
START SLAVE;
[...]
Retrieved_Gtid_Set: 7d72f9b4-8577-11e2-a3d7-080027635ef5:1-5
Executed_Gtid_Set: 7d72f9b4-8577-11e2-a3d7-080027635ef5:1-5

Source:
http://www.percona.com/blog/2013/03/26/repair-mysql-5-6-gtid-replication-by-injecting-empty-
transactions/

2014 Pythian
Multi-master Conflict
SET @@default_master_connection=%connection%;
SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
START SLAVE;

2014 Pythian
master1 failure
pluk$ sudo service mysqld_mul4 stop 1
pluk$ sudo service mysqld_mul4 report


(root@master2.sock)
[(none)]> SET @@default_master_connec4on=master1
(root@master2.sock) [(none)]> STOP SLAVE;
(root@master2.sock) [(none)]> CHANGE MASTER TO MASTER_PORT=3307;
(root@master2.sock) [(none)]> START SLAVE;



(root@slave1.sock) [(none)]> SET @@default_master_connec4on=master1
(root@slave1.sock) [(none)]> STOP SLAVE;
(root@slave1.sock) [(none)]> CHANGE MASTER TO MASTER_PORT=3308;
(root@slave1.sock) [(none)]> START SLAVE;



2014 Pythian
Setting up a Fan-in Environment

Hands-on

2014 Pythian
MySQL Topology

Fan-in

2014 Pythian
Fan-in Reset environment
pluk$ cd /home/pluk/scripts/fan-in
pluk$ bash reset_environment.sh

2014 Pythian
slave1 - slave of master1
(root@slave1.sock) [(none)]>CHANGE MASTER 'master1' TO
MASTER_HOST='127.0.0.1',
MASTER_PORT = 3306, -- master1
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

2014 Pythian
slave1 - slave of master2
(root@slave1.sock) [(none)]> CHANGE MASTER 'master2' TO
MASTER_HOST='127.0.0.1',
MASTER_PORT = 3308, -- master2
MASTER_USER = 'repl',
MASTER_PASSWORD = 'mul4_repl',
MASTER_USE_GTID = current_pos;

2014 Pythian
Limitations and Missing Features

2014 Pythian
Limitations
Currently only 64 masters
slave_net_timeout affects all connections, so doesnt
check against slave_heartbeat_period

https://mariadb.com/kb/en/mariadb/documentation/
replication/standard-replication/multi-source-replication/
#limitations

2014 Pythian
Importable Tablespaces
Ability to importing innodb file_per_table tablespaces
from a different server
Useful for extremely large datasets where logical restore
would take too long
Downside of feature: doesnt support partitions.

(root@slave1.sock) [foo]>ALTER TABLE e DISCARD TABLESPACE;

ERROR 1031 (HY000): Storage engine InnoDB of the table `foo`.`e` doesn't have
this op4on

2014 Pythian
Semi-sync Replication
Master waits for acknowledgement of receipt from a
semi-sync slave before returning to client
Gain: reduced chance of lost transactions
Lose: performance
Currently slated 10.2:
https://mariadb.atlassian.net/browse/MDEV-162

2014 Pythian
 Complexity
Monitoring multiple connections
Connection naming for failover purposes
Careful with duplicated events; gtid_ignore_duplicates=1
Skipping events wont progress gtid sequence on slaves

2014 Pythian
Resources
MariaDB multi-source docs:
https://mariadb.com/kb/en/mariadb/documentation/
replication/standard-replication/multi-source-replication
MariaDB GTID docs:
https://mariadb.com/kb/en/mariadb/documentation/
replication/standard-replication/global-transaction-id
MariaDB bug database:
https://mariadb.atlassian.net/browse/ODBC-15?jql=
MySQL multi-source worklog (for MySQL 5.7):
http://dev.mysql.com/worklog/task/?id=1697

2014 Pythian
Resources
Kristian Nielsen blogs on replication streams and GTID
http://kristiannielsen.livejournal.com/16826.html

2014 Pythian
 Q&A

email: downey@pythian.com
twitter: https://twitter.com/derek_downey

2014 Pythian