Mysqltechdayparispsandsys 140523063324 Phpapp02

!
1 Copyright
Copyright 2014,
2014, Oracle
Oracle and/or
and/or its affiliates.
its affiliates. All rights
All rights reserved.Insert Information Protection Policy Classification from Slide 12
reserved.
MySQL Tech Day, Paris
Performance Schema & MySQL sys
Mark Leith
Senior Software Development Manager
MySQL @ Oracle
!2
Copyright 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12
Program Agenda
What is Performance Schema anyway?

Performance Schema Configuration
Profiling General Instance Activity
Profiling Statement Activity
Improvements made to date in MySQL 5.7
The MySQL SYS Schema
Things still to come
!3 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Know Your Audience
Using 5.1 (yes, still)?

Using 5.5 (but Ill upgrade soon, honest)?
Using 5.6?
Using 5.7?
Used Performance Schema?
#monitoringlove
#monitoringhate


Introduced in MySQL 5.5

A storage engine, built for recording instrumentation
PERFORMANCE_SCHEMA
Manages fixed buffers in memory with lock free hashes
Real engine, unlike INFORMATION_SCHEMA
A database schema to expose the instrumentation
performance_schema
A set of interfaces in the MySQL Server wrapping calls to trace
i.e. pthread_mutex_lock() -> mysql_mutex_lock()

Records latency of events that happen within the server

All latency exposed to picosecond (a trillionth of a second) precision
Though may be tracked at different precisions internally
Also tracks other information as appropriate
Bytes, source position, object metadata, etc.

Performance Schema in MySQL 5.5
17 Tables
222 Instruments
Instrument Event Class

File IO wait/io/file/%
Mutexes wait/synch/mutex/%
Read/Write wait/synch/rwlock/%
Locks
Conditions wait/synch/cond/%


Laid out the framework for instrumentation

Focused early on low level instruments
Prove it could handle load, build from ground up
Not turned on by default, had some overhead issues
Wait Events
mysql> select * from events_waits_history_long where event_name like 'wait/io/table/%'\G!
*************************** 1. row ***************************!
THREAD_ID: 137746!
EVENT_ID: 3604! Event Context
END_EVENT_ID: 3604!
EVENT_NAME: wait/io/table/sql/handler!
SOURCE: handler.cc:2568! Event type and origin
TIMER_START: 17101711846046490!
TIMER_END: 17101711857643010! Timing (~11.5 microseconds)
TIMER_WAIT: 11596520!
SPINS: NULL!
OBJECT_SCHEMA: mem__events!
OBJECT_NAME: events!
Database object info
INDEX_NAME: PRIMARY!
OBJECT_TYPE: TABLE! Object type
OBJECT_INSTANCE_BEGIN: 140246283168288!
NESTING_EVENT_ID: 3603! Whether the event was nested
NESTING_EVENT_TYPE: STAGE!
OPERATION: fetch!
NUMBER_OF_BYTES: NULL! Further info as appropriate
FLAGS: NULL
Fixed lots of performance issues

Focused on bringing the instrumentation to the DBA / Developer
Statements / Statement Digests
Execution Stages
Object tracking (table / index IO, table locks)
Network IO
Turned on by default, with a subset of instrumentation disabled
52 Tables (+35)
545 Instruments (+323)
Instrument Event Class

Type
Statements statement/%
Stages stage/%
Table IO wait/io/table/%
Table Locks wait/lock/table/%
Network IO wait/io/socket/%
Idle Timing idle
I really think Performance Schema overhead is reasonable for most
workloads.
On my old server I got some 20.2K QPS with Performance Schema
Disabled and 19.4 QPS with Performance Schema enabled which is
overhead of less than 5%.
For most workloads paying 5% to have insight about what is
happening with the system is a very fair trade.
Peter Zaitsev, CEO, Percona
http://www.mysqlperformanceblog.com/2014/02/11/performance_schema-vs-slow-query-log/
5 Copyright
!1Copyright 2014,
2014, Oracle
Oracle and/or
reserved.
Statement & Stage Events (5.6)
!
Statements have two types of event
SQL Statements statement/sql/%
Protocol Commands statement/com/%
Stages are the thread states. Like SHOW PROFILE output, but
available across connections
Mostly one form stage/sql/%
Apart from .. stage/mysys/Waiting for table level lock
Stage Events
mysql> select * from events_stages_history limit 2\G!
*************************** 1. row ***************************!
THREAD_ID: 23!
EVENT_ID: 18898!
END_EVENT_ID: 18898!
EVENT_NAME: stage/sql/Master has sent all binlog to slave; waiting for more updates!
SOURCE: rpl_binlog_sender.cc:420!
TIMER_START: 87465041640516000!
TIMER_END: 87465045562629000!
TIMER_WAIT: 3922113000!
NESTING_EVENT_ID: 10!
NESTING_EVENT_TYPE: STATEMENT!
*************************** 2. row ***************************!
THREAD_ID: 23!
EVENT_ID: 18899!
EVENT_NAME: stage/sql/Sending binlog event to slave!
SOURCE: rpl_binlog_sender.cc:432!
TIMER_START: 87465045562629000!
TIMER_END: 87465045593812000!
NESTING_EVENT_ID: 10!
NESTING_EVENT_TYPE: STATEMENT
Statement Events
mysql> select * from events_statements_history_long limit 1\G!
*************************** 1. row ***************************!
THREAD_ID: 138683!
EVENT_ID: 11268!
EVENT_NAME: statement/sql/update!
SOURCE: socket_connection.cc:94!
TIMER_START: 87644080265364000!
TIMER_END: 87644080504810000!
LOCK_TIME: 63000000!
SQL_TEXT: /* mem dbpool.default */ update `mem__inventory`.Àgent` set
`hasHostname`=1400665852927, `hasReachable`=1400665852927, `timestamp`=1400665852927,
`hasVersion`=1400665852927 where hid=x'FA8FDC4C1BC344A0899DAB320757CDF2'!
DIGEST: cc389abfcb093ae95cacfe42ed085191!
DIGEST_TEXT: UPDATE `mem__inventory` . Àgent` SET `hasHostname` = ? , `hasReachable` = ? ,
`timestamp` = ? , `hasVersion` = ? WHERE `hid` = ?!
CURRENT_SCHEMA: mem!
OBJECT_TYPE: NULL!
OBJECT_SCHEMA: NULL!
OBJECT_NAME: NULL!
OBJECT_INSTANCE_BEGIN: NULL!

Statement Events (cont.)
mysql> select * from events_statements_history_long limit 1\G!
*************************** 1. row ***************************!
!
MYSQL_ERRNO: 0!
RETURNED_SQLSTATE: 00000!
MESSAGE_TEXT: Rows matched: 1 Changed: 1 Warnings: 0!
ERRORS: 0!
WARNINGS: 0!
ROWS_AFFECTED: 1!
ROWS_SENT: 0!
ROWS_EXAMINED: 1!
CREATED_TMP_DISK_TABLES: 0!
CREATED_TMP_TABLES: 0! NO_INDEX_USED: 0!
SELECT_FULL_JOIN: 0! NO_GOOD_INDEX_USED: 0!
SELECT_FULL_RANGE_JOIN: 0! NESTING_EVENT_ID: 11248!
SELECT_RANGE: 0! NESTING_EVENT_TYPE: TRANSACTION!
SELECT_RANGE_CHECK: 0! NESTING_EVENT_LEVEL: 0
SELECT_SCAN: 0!
SORT_MERGE_PASSES: 0!
SORT_RANGE: 0!
SORT_ROWS: 0!
SORT_SCAN: 0
Performance Schema Table Types
Setup Tables mysql> select table_name!

-> from information_schema.tables!
-> where table_schema like 'perf%'!
-> and table_name like 'setup%';!
+-------------------+!
Used to define certain | table_name |!
configuration dynamically +-------------------+!
| setup_actors |!
Can perform DML against these | setup_consumers |!
tables | setup_instruments |!
| setup_objects |!
| setup_timers |!
+-------------------+
+-------------------------------------------+!
| table_name |!
+-------------------------------------------+!
Raw Data Tables

| accounts |!
| cond_instances |!
| events_stages_current |!
| events_stages_history |!
| events_stages_history_long |!
| events_statements_current |!
| events_statements_history |!
Expose events, objects, or | events_statements_history_long
| events_waits_current
|!
|!
instances of instruments in a raw | events_waits_history
| events_waits_history_long
|!
|!
manner | file_instances
| host_cache
|!
|!
| hosts |!
Allow seeing a (brief) history of | mutex_instances
| performance_timers
|!
|!
raw event metrics as well | rwlock_instances
| session_account_connect_attrs
|!
|!
| session_connect_attrs |!
| socket_instances |!
| threads |!
| users |!
+-------------------------------------------+
+------------------------------------------------------+!
| table_name |!
+------------------------------------------------------+!
| events_stages_summary_by_account_by_event_name |!
Summary Tables | events_stages_summary_by_host_by_event_name

| events_stages_summary_by_thread_by_event_name
| events_stages_summary_by_user_by_event_name
|!
|!
|!
| events_stages_summary_global_by_event_name |!
| events_statements_summary_by_account_by_event_name |!
| events_statements_summary_by_digest |!
| events_statements_summary_by_host_by_event_name |!
Summarise event information | events_statements_summary_by_program

| events_statements_summary_by_thread_by_event_name
|!
|!
over multiple dimensions

| events_statements_summary_by_user_by_event_name |!
| events_statements_summary_global_by_event_name |!
| events_waits_summary_by_account_by_event_name |!
Useful for longer term monitoring

| events_waits_summary_by_host_by_event_name |!
| events_waits_summary_by_instance |!
| events_waits_summary_by_thread_by_event_name |!
of activity | events_waits_summary_by_user_by_event_name
| events_waits_summary_global_by_event_name
|!
|!
| file_summary_by_event_name |!
| file_summary_by_instance |!
| objects_summary_global_by_type |!
| socket_summary_by_event_name |!
| socket_summary_by_instance |!
| table_io_waits_summary_by_index_usage |!
| table_io_waits_summary_by_table |!
| table_lock_waits_summary_by_table |!
+------------------------------------------------------+
There are three distinct types of configuration

A way to configure how much memory to allocate for instruments
Set with various system variables in options files, not dynamic
A way to enable/disable instrumentation at startup
Again set with options files, only available as of 5.6
A way to enable/disable instrumentation dynamically
Set by updating the various setup_% tables dynamically
Performance Configuration
mysql> select variable_name,!
-> variable_value!
History / Summary Size

-> from information_schema.global_variables!
-> where variable_name like 'perf%'!
-> and variable_name not like '%instances'!
-> and variable_name not like '%classes'!
-> order by variable_name;!
+--------------------------------------------------------+----------------
| variable_name | variable_value
Define total rows per type +--------------------------------------------------------+----------------
| PERFORMANCE_SCHEMA | ON
| PERFORMANCE_SCHEMA_ACCOUNTS_SIZE | 100
The %_history_size are per-thread | PERFORMANCE_SCHEMA_DIGESTS_SIZE
| PERFORMANCE_SCHEMA_EVENTS_STAGES_HISTORY_LONG_SIZE
| 5000
| 1000
| PERFORMANCE_SCHEMA_EVENTS_STAGES_HISTORY_SIZE | 10
The %_history_long_size are total | PERFORMANCE_SCHEMA_EVENTS_STATEMENTS_HISTORY_LONG_SIZE | 1000
| PERFORMANCE_SCHEMA_EVENTS_STATEMENTS_HISTORY_SIZE | 10
rows | PERFORMANCE_SCHEMA_EVENTS_WAITS_HISTORY_LONG_SIZE
| PERFORMANCE_SCHEMA_EVENTS_WAITS_HISTORY_SIZE
| 1000
| 10
| PERFORMANCE_SCHEMA_HOSTS_SIZE | 100
Only settable in my.[cnf|ini] | PERFORMANCE_SCHEMA_MAX_FILE_HANDLES
| PERFORMANCE_SCHEMA_MAX_TABLE_HANDLES
| 32768
| 732
| PERFORMANCE_SCHEMA_SESSION_CONNECT_ATTRS_SIZE | 512
| PERFORMANCE_SCHEMA_SETUP_ACTORS_SIZE | 100
| PERFORMANCE_SCHEMA_SETUP_OBJECTS_SIZE | 100
| PERFORMANCE_SCHEMA_USERS_SIZE | 100
+--------------------------------------------------------+----------------
mysql> select variable_name, variable_value!
Max Classes / Instances -> from information_schema.global_variables!

-> where variable_name like 'perf%classes'!
-> or variable_name like 'perf%instances'!
+------------------------------------------------------+----------------+!
| variable_name | variable_value |!
Classes count instrument +------------------------------------------------------+----------------+!
| PERFORMANCE_SCHEMA_MAX_COND_CLASSES | 80 |!
implementations, i.e: | PERFORMANCE_SCHEMA_MAX_COND_INSTANCES

| PERFORMANCE_SCHEMA_MAX_FILE_CLASSES
| 946
| 50
|!
|!
| PERFORMANCE_SCHEMA_MAX_FILE_INSTANCES | 2170 |!
wait/io/file/sql/binlog | PERFORMANCE_SCHEMA_MAX_MUTEX_CLASSES
| PERFORMANCE_SCHEMA_MAX_MUTEX_INSTANCES
| 200
| 4586
|!
|!
| PERFORMANCE_SCHEMA_MAX_RWLOCK_CLASSES | 40 |!
Instances are the different runtime | PERFORMANCE_SCHEMA_MAX_RWLOCK_INSTANCES
| PERFORMANCE_SCHEMA_MAX_SOCKET_CLASSES
| 3283
| 10
|!
|!
instances of those things, i.e: | PERFORMANCE_SCHEMA_MAX_SOCKET_INSTANCES

| PERFORMANCE_SCHEMA_MAX_STAGE_CLASSES
| 86
| 150
|!
|!
| PERFORMANCE_SCHEMA_MAX_STATEMENT_CLASSES | 189 |!
/data/logs/binlog.000123 | PERFORMANCE_SCHEMA_MAX_TABLE_INSTANCES
| PERFORMANCE_SCHEMA_MAX_THREAD_CLASSES
| 569
| 50
|!
|!
| PERFORMANCE_SCHEMA_MAX_THREAD_INSTANCES | 143 |!
+------------------------------------------------------+----------------+
mysql> select variable_name, variable_value!
-> from information_schema.global_status!
Max Classes / Instances

-> where variable_name like 'perf%classes_lost'!
-> or variable_name like 'perf%instances_lost'!
+-------------------------------------------+----------------+!
| variable_name | variable_value |!
+-------------------------------------------+----------------+!
Monitor the %_lost status variables

| PERFORMANCE_SCHEMA_COND_CLASSES_LOST | 0 |!
| PERFORMANCE_SCHEMA_COND_INSTANCES_LOST | 0 |!
opposite to see whether these

| PERFORMANCE_SCHEMA_FILE_CLASSES_LOST | 0 |!
| PERFORMANCE_SCHEMA_FILE_INSTANCES_LOST | 0 |!
| PERFORMANCE_SCHEMA_MUTEX_CLASSES_LOST | 0 |!
need tweaking | PERFORMANCE_SCHEMA_MUTEX_INSTANCES_LOST | 0 |!
| PERFORMANCE_SCHEMA_RWLOCK_CLASSES_LOST | 0 |!
You will generally not need to | PERFORMANCE_SCHEMA_RWLOCK_INSTANCES_LOST | 0
| PERFORMANCE_SCHEMA_SOCKET_CLASSES_LOST | 0
|!
|!
modify classes, only when loading | PERFORMANCE_SCHEMA_SOCKET_INSTANCES_LOST | 0

| PERFORMANCE_SCHEMA_STAGE_CLASSES_LOST | 0
|!
|!
new plugins | PERFORMANCE_SCHEMA_STATEMENT_CLASSES_LOST | 0

| PERFORMANCE_SCHEMA_TABLE_INSTANCES_LOST | 0
|!
|!
| PERFORMANCE_SCHEMA_THREAD_CLASSES_LOST | 0 |!
| PERFORMANCE_SCHEMA_THREAD_INSTANCES_LOST | 0 |!
+-------------------------------------------+----------------+
mysql> select * from setup_instruments limit 5;!
+-------------------------------------------------------+---------+-------
| NAME | ENABLED | TIMED
setup_instruments
+-------------------------------------------------------+---------+-------
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_tc | NO | NO
| wait/synch/mutex/sql/LOCK_des_key_file | NO | NO
| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit | NO | NO
| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit_queue | NO | NO
| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_done | NO | NO
+-------------------------------------------------------+---------+-------
Turn on/off individual instruments, !

5 rows in set (0.00 sec)!
or whether to just count and not

mysql> update setup_instruments!
-> set enabled = 'yes', timed = 'yes'!
-> where name like '%MYSQL_BIN_LOG%';!
time events Query OK, 16 rows affected (0.00 sec)!
Use UPDATE to modify

!
Rows matched: 16 Changed: 16 Warnings: 0!
mysql> select * from setup_instruments limit 5;!

+-------------------------------------------------------+---------+-------
dynamically | NAME | ENABLED | TIMED
+-------------------------------------------------------+---------+-------
| wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_tc | NO | NO
| wait/synch/mutex/sql/LOCK_des_key_file | NO | NO
| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit | YES | YES
| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_commit_queue | YES | YES
| wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_done | YES | YES
+-------------------------------------------------------+---------+-------
mysql> select * from setup_consumers;!
+----------------------------------+---------+!
setup_consumers | NAME | ENABLED |!

+----------------------------------+---------+!
| events_stages_current | NO |!
| events_stages_history | NO |!
| events_stages_history_long | NO |!
Define how much to record, either | events_statements_current | YES |!
| events_statements_history | NO |!
in history, or summaries (by | events_statements_history_long | NO |!
enabling the %_current for the | events_transactions_current
| events_transactions_history
| NO
| NO
|!
|!
class of event) | events_transactions_history_long | NO |!
| events_waits_current | NO |!
| events_waits_history | NO |!
| events_waits_history_long | NO |!
| global_instrumentation | YES |!
| thread_instrumentation | YES |!
| statements_digest | YES |!
+----------------------------------+---------+
Performance Schema Consumer Hierarchy
mysql> select * from setup_objects;!
+-------------+--------------------+-------------+---------+-------+!
setup_objects (5.6) | OBJECT_TYPE | OBJECT_SCHEMA
| mysql
| OBJECT_NAME | ENABLED | TIMED |!
+-------------+--------------------+-------------+---------+-------+!
| TABLE | % | NO | NO |!
| TABLE | performance_schema | % | NO | NO |!
| TABLE | information_schema | % | NO | NO |!
| TABLE | % | % | YES | YES |!
Define exactly which database

+-------------+--------------------+-------------+---------+-------+!
!
objects to monitor mysql> insert into setup_objects!

-> values ('TABLE', 'foo', 'bar', 'no', 'no');!
Filters standard databases by !

Query OK, 1 row affected (0.01 sec)!
mysql> select * from setup_objects;!

default +-------------+--------------------+-------------+---------+-------+!
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |!
+-------------+--------------------+-------------+---------+-------+!
Allows explicit enable/disable, | TABLE
| TABLE
| mysql | %
| performance_schema | %
| NO
| NO
| NO
| NO
|!
|!
matches on most specific | TABLE

| TABLE
| information_schema | %
| % | %
| NO
| YES
| NO
| YES
|!
|!
| TABLE | foo | bar | NO | NO |!
+-------------+--------------------+-------------+---------+-------+
mysql> select * from setup_actors;!
+------+------+------+!
| HOST | USER | ROLE |!
setup_actors (5.6) +------+------+------+!
| % | % | % |!
+------+------+------+!
1 row in set (0.00 sec)!
Define exactly which users to

!
mysql> delete from setup_actors;!
monitor Query OK, 1 row affected (0.00 sec)!
!
mysql> insert into setup_actors!
By default monitors all connections -> values ('%', 'mark', '%');!
(% is wildcard) Query OK, 1 row affected (0.00 sec)!
!
mysql> select * from setup_actors;!
Modifications only apply to new +------+------+------+!
connections | HOST | USER | ROLE |!
+------+------+------+!
| % | mark | % |!
+------+------+------+!
1 row in set (0.00 sec)
Performance Schema Options Files Config
As of 5.6, allows altering configuration at start up

Enabling / disabling instrumentation
performance_schema_instrument = instrument_name=value
Value = [on|1|true] | [off|0|false] | counted
instrument_name can have wildcards (wait/synch/mutex/%=off)
Enabling / disabling consumers
performance_schema_consumer_consumer_name=value
Value = [on|1|true] | [off|0|false]
Profiling General Instance Activity
Profiling Considerations
It is easy to just enable everything, but there are overhead concerns

On busy systems mutexes can be locked millions of times per second
It is best to just pick higher latency event types
Statements, Stages, Table IO, File IO, maybe Network IO
It is good to enable all %_current consumers, and statement history
For concurrency/profiling debugging toggle other instruments on an
as needed basis
Profiling Types
Once youve narrowed down what youre interested in, there are two
ways to start monitoring
View raw data in the summary views
Gives you an overall picture of usage on the instance
Snapshot data, and compute deltas over time
Gives you an idea of the rates of change for the events
Analyzing Global Waits
Some waits can include other waits

Table IO latency may also include some mutex and file IO latency
Wait times include concurrency across all threads
Do not assume you can sum all events in global tables and compare
to wall clock times
Top Waits By Latency
mysql> select event_name,!
-> count_star as count,!
-> sys.format_time(sum_timer_wait) as total_latency,!
-> sys.format_time(avg_timer_wait) as avg_latency,!
-> sys.format_time(max_timer_wait) as max_latency!
-> from events_waits_summary_global_by_event_name!
-> where event_name != 'idle'!
-> order by sum_timer_wait desc limit 5;!
+--------------------------------------+----------+---------------+-------------+-------------+!
| event_name | count | total_latency | avg_latency | max_latency |!
+--------------------------------------+----------+---------------+-------------+-------------+!
| wait/io/table/sql/handler | 21888502 | 1.27h | 208.06 us | 2.21 s |!
| wait/io/file/innodb/innodb_data_file | 4276800 | 00:48:12.49 | 676.32 us | 1.49 s |!
| wait/io/file/innodb/innodb_log_file | 1948199 | 00:25:24.36 | 782.45 us | 1.30 s |!
| wait/io/file/myisam/kfile | 4566406 | 00:13:45.92 | 180.87 us | 1.17 s |!
| wait/io/file/myisam/dfile | 1277589 | 00:05:46.23 | 271.01 us | 1.18 s |!
+--------------------------------------+----------+---------------+-------------+-------------+
Some mutex events that can affect global concurrency (if high in list):
wait/synch/mutex/innodb/buf_pool_mutex
Increase innodb_buffer_pool_instances
wait/synch/mutex/sql/Query_cache::structure_guard_mutex
Look in to disabling the Query Cache
wait/synch/mutex/myisam/MYISAM_SHARE::intern_lock
Use Innodb
Some File IO events to watch for (if high in list):

wait/io/file/sql/FRM
Tune table_open_cache / table_definition_cache
wait/io/file/sql/file_parser (view definition parsing)
If high on 5.5, upgrade to 5.6, (which can cache these like tables)
wait/io/file/sql/query_log and wait/io/file/sql/slow_log
Disable the general
Disable or tune what is logged to the slow log (decent long_query_time)
Top Files By Total IO
mysql> select sys.format_path(file_name) as file,!
-> count_read,!
-> sys.format_bytes(sum_number_of_bytes_read) as total_read,!
-> sys.format_bytes(IFNULL(sum_number_of_bytes_read / count_read, 0)) as avg_read,!
-> count_write,!
-> sys.format_bytes(sum_number_of_bytes_write) as total_written,!
-> sys.format_bytes(IFNULL(sum_number_of_bytes_write / count_write, 0)) as avg_write,!
-> sys.format_bytes(sum_number_of_bytes_read + sum_number_of_bytes_write) as total,!
-> IFNULL(ROUND(100-((sum_number_of_bytes_read/(sum_number_of_bytes_read + sum_number_of_bytes_write))*100), 2), 0.00) as write_pct!
-> from file_summary_by_instance!
-> order by (sum_number_of_bytes_read + sum_number_of_bytes_write) desc limit 5;!
+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+
| file | count_read | total_read | avg_read | count_write | total_written | avg_write | total | write_pct |
+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+
| @@datadir/ibdata1 | 888 | 15.84 MiB | 18.27 KiB | 1089824 | 61.99 GiB | 59.64 KiB | 62.00 GiB | 99.98 |
| @@datadir/mem__events/events.ibd | 114 | 1.80 MiB | 16.14 KiB | 117370 | 2.14 GiB | 19.14 KiB | 2.14 GiB | 99.92 |
| @@datadir/cerberus-bin.000010 | 296103 | 1.01 GiB | 3.59 KiB | 362852 | 1.00 GiB | 2.89 KiB | 2.01 GiB | 49.63 |
| @@datadir/ib_logfile0 | 6 | 68.00 KiB | 11.33 KiB | 506837 | 1.46 GiB | 3.03 KiB | 1.46 GiB | 100.00 |
| @@datadir/ib_logfile1 | 0 | 0 bytes | 0 bytes | 476961 | 1.45 GiB | 3.19 KiB | 1.45 GiB | 100.00 |
+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+
High IO on InnoDB per-tablespace tables can

show candidates for a separate mountpoint/disk
using DATA DIRECTORY in CREATE TABLE
Analyzing User Activity
All event_% summaries are exposed with a number of dimensions

To analyze connection activity you can do this in 4 ways
By User
By Host
By Account (User@Host)
By Thread
The follow examples are by user, but could be replaced with the host,
account or thread summary views
Top Users By Statement Latency
mysql> select user,!

-> sum(count_star) as statements,!
-> sys.format_time(sum(sum_timer_wait)) as total_latency,!
-> sys.format_time(sum(sum_timer_wait) / sum(count_star)) as avg_latency!
-> from events_statements_summary_by_user_by_event_name!
-> where user is not null!
-> group by user!
-> order by sum(sum_timer_wait) desc;!
+------+------------+---------------+-------------+!
| user | statements | total_latency | avg_latency |!
+------+------------+---------------+-------------+!
| root | 7229032 | 15.17h | 7.55 ms |!
| mark | 3072 | 1.77 s | 575.29 us |!
+------+------------+---------------+-------------+!
Top Users By IO Latency
mysql> select user, sum(count_star) as count,!

-> sys.format_time(sum(sum_timer_wait)) as total_latency!
-> from events_waits_summary_by_user_by_event_name!
-> where event_name like 'wait/io/file/%'!
-> and user is not null!
-> group by user!
-> order by sum(sum_timer_wait) desc;!
+------+----------+---------------+!
| user | count | total_latency |! Replace with some other
+------+----------+---------------+!
| root | 10892980 | 00:46:19.67 |!
pattern here for Top user by
| mark | 20043 | 346.79 ms |! wait class
+------+----------+---------------+!
Top Users By Connections
mysql> select * from users where user is not null order by current_connections desc;!
+------+---------------------+-------------------+!
| USER | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |!
+------+---------------------+-------------------+!
| root | 8 | 151655 |!
| mark | 1 | 1 |!
+------+---------------------+-------------------+!
!
mysql> select * from accounts where user is not null order by current_connections desc;!
+------+-----------+---------------------+-------------------+!
| USER | HOST | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |!
+------+-----------+---------------------+-------------------+!
| root | localhost | 39 | 151698 |!
| mark | localhost | 1 | 1 |!
+------+-----------+---------------------+-------------------+!
2 rows in set (0.00 sec)
User Statement Activity
Summarized by
mysql> select *!
-> from events_statements_summary_by_user_by_event_name!
statement type, e.g:
-> where user is not null! !
-> order by user, sum_timer_wait desc limit 20\G!
*************************** 1. row ***************************! statement/sql/select
USER: mark!
EVENT_NAME: statement/sql/select!
statement/sql/update
COUNT_STAR: 4!
SUM_TIMER_WAIT: 1692883853000!
MIN_TIMER_WAIT: 88330000! SUM_SELECT_FULL_JOIN: 0!
AVG_TIMER_WAIT: 423220963000! SUM_SELECT_FULL_RANGE_JOIN: 0!
MAX_TIMER_WAIT: 1692583043000! SUM_SELECT_RANGE: 0!
SUM_LOCK_TIME: 408000000! SUM_SELECT_RANGE_CHECK: 0!
SUM_ERRORS: 0! SUM_SELECT_SCAN: 2!
SUM_WARNINGS: 0! SUM_SORT_MERGE_PASSES: 6!
SUM_ROWS_AFFECTED: 0! SUM_SORT_RANGE: 0!
SUM_ROWS_SENT: 212! SUM_SORT_ROWS: 6458!
SUM_ROWS_EXAMINED: 15067! SUM_SORT_SCAN: 2!
SUM_CREATED_TMP_DISK_TABLES: 3! SUM_NO_INDEX_USED: 1!
SUM_CREATED_TMP_TABLES: 4 SUM_NO_GOOD_INDEX_USED: 0
User Top Stages
mysql> select user, event_name as stage,!
-> count_star as total,!
-> sys.format_time(avg_timer_wait) as avg_latency,!
-> from events_stages_summary_by_user_by_event_name!
-> where sum_timer_wait > 0!
-> order by user, sum_timer_wait desc;!
+------+--------------------------------+--------+---------------+-------------+-------------+!
| user | stage | total | total_latency | avg_latency | max_latency |!
+------+--------------------------------+--------+---------------+-------------+-------------+!
| root | stage/sql/starting | 591184 | 00:47:52.47 | 4.86 ms | 3.96 s |!
| root | stage/sql/update | 193940 | 00:15:00.84 | 4.64 ms | 4.02 s |!
| root | stage/sql/updating | 50334 | 00:01:41.38 | 2.01 ms | 2.35 s |!
| root | stage/sql/statistics | 73869 | 00:01:41.38 | 1.37 ms | 4.54 s |!
| root | stage/sql/Sending data | 73211 | 00:01:38.31 | 1.34 ms | 1.04 s |!
| root | stage/sql/Opening tables | 404230 | 00:01:23.48 | 206.51 us | 473.72 ms |!
| root | stage/sql/closing tables | 567351 | 00:01:13.40 | 129.38 us | 79.52 ms |!
| root | stage/sql/removing tmp table | 11603 | 56.87 s | 4.90 ms | 964.17 ms |!
| root | stage/sql/freeing items | 567520 | 19.75 s | 34.80 us | 219.19 ms |!
| root | stage/sql/init | 316805 | 8.26 s | 26.06 us | 217.54 ms |!
Analyzing Table Activity
3 summary views have been added to 5.6 on top of Table IO

objects_summary_global_by_type
A high level aggregate view of database object latency
table_io_waits_summary_by_table
A detailed aggregate by table, breaking down reads, writes, etc.
table_io_waits_summary_by_index_usage
Further breaking down usage per index
Top Tables By Latency
mysql> select object_schema,!
-> object_name,!
-> count_star,!
-> sys.format_time(sum_timer_wait / count_star) as avg_latency,!
-> from objects_summary_global_by_type!
+------------------+-------------------+------------+---------------+-------------+-------------+!
| object_schema | object_name | count_star | total_latency | avg_latency | max_latency |!
+------------------+-------------------+------------+---------------+-------------+-------------+!
| mem__inventory | mysqlserver | 1049496 | 00:04:48.32 | 274.72 us | 1.50 s |!
| mem__events | events | 227939 | 00:01:00.50 | 265.41 us | 2.15 s |!
| mem__events | event_chain_heads | 328664 | 50.80 s | 154.57 us | 2.21 s |!
| mem__inventory | agent | 442751 | 44.82 s | 101.23 us | 1.69 s |!
| mem__instruments | fsstatistics | 2233484 | 16.53 s | 7.40 us | 269.27 ms |!
+------------------+-------------------+------------+---------------+-------------+-------------+
Less events, yet higher latency, is

a sign that there could be
concurrency issues
Table Usage Detailed Breakdown
!
mysql> select object_schema, object_name,!
-> count_fetch as selects, sys.format_time(sum_timer_fetch) as select_latency,!
-> count_insert as inserts, sys.format_time(sum_timer_insert) as insert_latency,!
-> count_update as updates, sys.format_time(sum_timer_update) as update_latency,!
-> count_delete as deletes, sys.format_time(sum_timer_delete) as delete_latency!
-> from table_io_waits_summary_by_table!
+------------------+-------------------+---------+----------------+---------+----------------+---------+----------------+---------+----------------+!
| object_schema | object_name | selects | select_latency | inserts | insert_latency | updates | update_latency | deletes | delete_latency |!
+------------------+-------------------+---------+----------------+---------+----------------+---------+----------------+---------+----------------+!
| mem__inventory | mysqlserver | 418518 | 00:04:32.21 | 0 | 0 ps | 48534 | 15.51 s | 0 | 0 ps |!
| mem__events | events | 58512 | 5.54 s | 2701 | 4.01 s | 52414 | 50.83 s | 0 | 0 ps |!
| mem__events | event_chain_heads | 156194 | 6.93 s | 78 | 59.75 ms | 52366 | 43.60 s | 0 | 0 ps |!
| mem__inventory | agent | 188113 | 40.06 s | 0 | 0 ps | 6998 | 4.54 s | 0 | 0 ps |!
| mem__instruments | fsstatistics | 2222058 | 15.12 s | 1517 | 1.37 s | 1 | 72.69 us | 0 | 0 ps |!
| mem__instruments | qrtidata | 2771546 | 2.57 s | 3030 | 12.52 s | 0 | 0 ps | 0 | 0 ps |!
| mem__inventory | network | 91187 | 2.24 s | 0 | 0 ps | 11864 | 8.42 s | 0 | 0 ps |!
| mem__inventory | os | 79736 | 6.08 s | 0 | 0 ps | 8827 | 4.07 s | 0 | 0 ps |!
| mem__inventory | networking | 99114 | 4.24 s | 0 | 0 ps | 5928 | 5.54 s | 0 | 0 ps |!
| mem__inventory | environment | 100445 | 5.29 s | 0 | 0 ps | 6540 | 2.53 s | 0 | 0 ps |!
+------------------+-------------------+---------+----------------+---------+----------------+---------+----------------+---------+----------------+
mysql> select concat(object_schema, '.', object_name) as object,!
Tables with full scans ->

->
->
count_read as rows_scanned,!
sys.format_time(sum_timer_wait) as latency!
from table_io_waits_summary_by_index_usage!
-> where index_name is null!
-> and count_read > 0!
Search in +---------------------------------------+--------------+-----------
| object | rows_scanned | latency
table_io_waits_summary_by_index_usage +---------------------------------------+--------------+-----------
| mem__instruments.qrtidata | 2735434 | 14.56 s
where index_name is null | mem__instruments.databaseactivitydata |
| mem__instruments.diskiototaldata |
2735633 | 4.68 s
31813 | 3.26 s
This is a catch all row for rows that

| mem__instruments.connectionsdata | 2530816 | 3.17 s
| mem__inventory.agent | 117544 | 2.89 s
are read without indexes

| mem__instruments.fsstatistics | 738626 | 2.69 s
| mem__instruments.diskioopstotaldata | 25955 | 2.43 s
| mem__inventory.environment | 42549 | 1.55 s
| mem__enterprise.whats_new_entries | 1453 | 1.16 s
| mem__inventory.networking | 42549 | 559.14 ms
+---------------------------------------+--------------+-----------
mysql> select object_schema,!

Unused Indexes ->
->
object_name,!
index_name!
-> from table_io_waits_summary_by_index_usage!
-> where index_name is not null!
-> and count_star = 0!
Search in -> order by object_schema, object_name limit 10;!
+-------------------+---------------------+------------+!
| object_schema | object_name | index_name |!
table_io_waits_summary_by_index_usage +-------------------+---------------------+------------+!
where count_star = 0
| mem__advisors | advisor_initialized | PRIMARY |!
| mem__advisors | advisor_schedules | PRIMARY |!
| mem__advisors | app_identity_path | PRIMARY |!
You should ensure you have | mem__advisor_text | template_meta
| mem__bean_config | plists
| PRIMARY
| path
|!
|!
representative time frame before | mem__bean_config | plists
| mem__bean_config | plist_name_values
| PRIMARY
| PRIMARY
|!
|!
taking any actions! | mem__config

| mem__config
| asset_notes
| group_selections
| PRIMARY
| PRIMARY
|!
|!
| mem__config | group_selections | name |!
+-------------------+---------------------+------------+
Host Cache Usage and Errors
New host_cache view within 5.6, exposing hosts in the host cache
Counts errors, by the type of error that can occur
Can show when errors started happening
Compare the sum_connect_errors counter to the
max_connect_errors system variable
host_cache
mysql> desc host_cache;!
+--------------------------------------------+------------------+------+-----+---------------------+-------+!
| Field | Type | Null | Key | Default | Extra |!
+--------------------------------------------+------------------+------+-----+---------------------+-------+!
| IP | varchar(64) | NO | | NULL | |!
| HOST | varchar(255) | YES | | NULL | |!
| HOST_VALIDATED | enum('YES','NO') | NO | | NULL | |!
| SUM_CONNECT_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_HOST_BLOCKED_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_NAMEINFO_TRANSIENT_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_NAMEINFO_PERMANENT_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_FORMAT_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_ADDRINFO_TRANSIENT_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_ADDRINFO_PERMANENT_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_FCRDNS_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_HOST_ACL_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_NO_AUTH_PLUGIN_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_AUTH_PLUGIN_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_HANDSHAKE_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_PROXY_USER_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_PROXY_USER_ACL_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_AUTHENTICATION_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_SSL_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_MAX_USER_CONNECTIONS_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_DEFAULT_DATABASE_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_INIT_CONNECT_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_LOCAL_ERRORS | bigint(20) | NO | | NULL | |!
| COUNT_UNKNOWN_ERRORS | bigint(20) | NO | | NULL | |!
| FIRST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |!
| LAST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |!
| FIRST_ERROR_SEEN | timestamp | YES | | 0000-00-00 00:00:00 | |!
| LAST_ERROR_SEEN | timestamp | YES | | 0000-00-00 00:00:00 | |
Computing Rates Of Change
To look at how things change over time, you have a couple of options
Record the events you are interested in within persistent history tables
Get current stats from performance_schema, the last rows of history,
compute the time delta and event count deltas, store new deltas along
with the raw values
Or use some tool like MySQL Enterprise Monitor
Read the tables regularly, and run TRUNCATE TABLE on them
immediately afterwards
Some users have done this by dumping local traces to files on
disk, and resetting stats every 15 minutes, for example
For an example, the following event tracks the time spent waiting in
the replication SQL thread for new events to be written to the relay log
by the IO thread
wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond
This is essentially the SQL thread idle time, for a single SQL thread, if
we snapshot this over time, we can find overall busy time
Slave SQL Load Average
An example of monitoring this event over time
!
http://www.markleith.co.uk/2012/07/24/a-mysql-replication-load-average-with-performance-schema/
Slave SQL Load Average
http://www.markleith.co.uk/2012/07/24/a-mysql-replication-load-average-with-performance-schema/
If interested in calculating percentages, copy the approach shown in

the replication load average you have to take in to account waiting
100% on other events, or just this event, etc.
Note, this method can not be used generally, you have to be able to
correlate it to specific thread(s)
As latency is per thread, and you have to compare to a known
time delta, summary views do not give this to you (you dont get
thread counts per interval)
Should still compute the deltas for global events however, for
event throughput / latency statistics over time
!
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
!59
Profiling Statement Activity
Statement Profiling Options
The new instrumentation in 5.6 gives many options

See statements running currently with events_statements_current
Summary views by user, host, account
Statement histories within events_statements_history% tables
A normalized view within events_statements_summary_by_digest
Currently Executing Statements
mysql> select * from events_statements_current where timer_end is null\G!
*************************** 8. row ***************************!
THREAD_ID: 156945!
EVENT_ID: 15312!
END_EVENT_ID: NULL! All counters are live, and
EVENT_NAME: statement/sql/select!
SOURCE: socket_connection.cc:94! increment whilst the statements
TIMER_START: 99343994725205000!
TIMER_END: NULL! execute
TIMER_WAIT: NULL!
LOCK_TIME: 145000000!
SQL_TEXT: /* mem dbpool.ui */ select normalized0_.round_robin_bin as round1_1256_, /* ..snip .. */!
DIGEST: NULL!
DIGEST_TEXT: NULL!
CURRENT_SCHEMA: mem!
OBJECT_TYPE: NULL! SELECT_FULL_JOIN: 0!
OBJECT_SCHEMA: NULL! SELECT_FULL_RANGE_JOIN: 0!
OBJECT_NAME: NULL! SELECT_RANGE: 1!
OBJECT_INSTANCE_BEGIN: NULL! SELECT_RANGE_CHECK: 0!
MYSQL_ERRNO: 0! SELECT_SCAN: 0!
RETURNED_SQLSTATE: NULL! SORT_MERGE_PASSES: 0!
MESSAGE_TEXT: NULL! SORT_RANGE: 0!
ERRORS: 0! SORT_ROWS: 0!
WARNINGS: 0! SORT_SCAN: 0!
ROWS_AFFECTED: 0! NO_INDEX_USED: 0!
ROWS_SENT: 27! NO_GOOD_INDEX_USED: 0!
ROWS_EXAMINED: 0! NESTING_EVENT_ID: NULL!
CREATED_TMP_DISK_TABLES: 0! NESTING_EVENT_TYPE: NULL!
CREATED_TMP_TABLES: 0 NESTING_EVENT_LEVEL: 0
Per Thread Statement History
mysql> select thread_id,!
-> sys.format_statement(sql_text) as stmt,! Also has same columns as
->
->
sys.format_time(timer_wait) as latency!
from events_statements_history!
events_statements_current
-> order by thread_id, event_id;!
+-----------+-------------------------------------------------------------------+-----------+!
| thread_id | stmt | latency |!
+-----------+-------------------------------------------------------------------+-----------+!
...!
| 158162 | /* mem dbpool.default */ commit | 5.39 ms |!
| 158162 | /* mem dbpool.default */ inser ... currentlyOpen), currentlyOpen) | 1.25 ms |!
| 158162 | /* mem dbpool.default */ inser ... , VALUES(lastSeen)), lastSeen) | 259.75 us |!
| 158162 | /* mem dbpool.default */ inser ... (bytesTotal), bytesTotal), col | 383.05 us |!
| 158162 | /* mem dbpool.default */ inser ... ullScan), rowsReadviaFullScan) | 147.04 ms |!
| 158162 | /* mem dbpool.default */ inser ... ALUES(indexUsage), indexUsage) | 1.10 ms |!
...!
+-----------+-------------------------------------------------------------------+-----------+!
Statement Digests
A new summary view introduced in 5.6

Aggregates statistics based on normalised statements
Can be used to drill in to your problem statements instead of using the
slow query log (with less contention, and no need for post-processing)
Each statement gets an MD5 DIGEST, that is also available to link in
to events_statements_history% etc. for raw per-query statistics
events_statements_summary_by_digest
mysql> desc events_statements_summary_by_digest;!
+-----------------------------+---------------------+------+-----+---------------------+-------+!
+-----------------------------+---------------------+------+-----+---------------------+-------+!
| SCHEMA_NAME | varchar(64) | YES | | NULL | |!
| DIGEST | varchar(32) | YES | | NULL | |!
| DIGEST_TEXT | longtext | YES | | NULL | |!
| COUNT_STAR | bigint(20) unsigned | NO | | NULL | |!
| SUM_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!
| MIN_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!
| AVG_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!
| MAX_TIMER_WAIT | bigint(20) unsigned | NO | | NULL | |!
| SUM_LOCK_TIME | bigint(20) unsigned | NO | | NULL | |!
| SUM_ERRORS | bigint(20) unsigned | NO | | NULL | |!
| SUM_WARNINGS | bigint(20) unsigned | NO | | NULL | |!
| SUM_ROWS_AFFECTED | bigint(20) unsigned | NO | | NULL | |!
| SUM_ROWS_SENT | bigint(20) unsigned | NO | | NULL | |!
| SUM_ROWS_EXAMINED | bigint(20) unsigned | NO | | NULL | |!
| SUM_CREATED_TMP_DISK_TABLES | bigint(20) unsigned | NO | | NULL | |!
| SUM_CREATED_TMP_TABLES | bigint(20) unsigned | NO | | NULL | |!
| SUM_SELECT_FULL_JOIN | bigint(20) unsigned | NO | | NULL | |!
| SUM_SELECT_FULL_RANGE_JOIN | bigint(20) unsigned | NO | | NULL | |!
| SUM_SELECT_RANGE | bigint(20) unsigned | NO | | NULL | |!
| SUM_SELECT_RANGE_CHECK | bigint(20) unsigned | NO | | NULL | |!
| SUM_SELECT_SCAN | bigint(20) unsigned | NO | | NULL | |!
| SUM_SORT_MERGE_PASSES | bigint(20) unsigned | NO | | NULL | |!
| SUM_SORT_RANGE | bigint(20) unsigned | NO | | NULL | |!
| SUM_SORT_ROWS | bigint(20) unsigned | NO | | NULL | |!
| SUM_SORT_SCAN | bigint(20) unsigned | NO | | NULL | |!
| SUM_NO_INDEX_USED | bigint(20) unsigned | NO | | NULL | |!
| SUM_NO_GOOD_INDEX_USED | bigint(20) unsigned | NO | | NULL | |!
| FIRST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |!
| LAST_SEEN | timestamp | NO | | 0000-00-00 00:00:00 | |
Statement Normalization
Normalization folds certain constructs of statements

Strip whitespace / comments
Replace literals with ?
WHERE foo = 1 becomes WHERE foo = ?
Fold lists of things
IN (1,2,3) becomes IN ()
Fold multi-row inserts
VALUES (1), (2), (3) becomes VALUES (?) /*, */
!
Statement Digest Output
mysql> select * from events_statements_summary_by_digest order by sum_timer_wait desc limit 5\G!
...!
*************************** 4. row ***************************!
SCHEMA_NAME: mem!
DIGEST: 5f41a0036bae4fa1c79339c1c7da3c9e!
DIGEST_TEXT: SELECT DISTINCTROW àgent0_` . `hid` AS `hid1239_` ...!
COUNT_STAR: 64136!
SUM_TIMER_WAIT: 1835031591954000!
SUM_SELECT_FULL_JOIN: 0!
MIN_TIMER_WAIT: 209280000!
SUM_SELECT_FULL_RANGE_JOIN: 0!
AVG_TIMER_WAIT: 28611569000!
SUM_SELECT_RANGE: 0!
MAX_TIMER_WAIT: 4050621865000!
SUM_SELECT_RANGE_CHECK: 0!
SUM_LOCK_TIME: 31388434000000!
SUM_SELECT_SCAN: 59625!
SUM_ERRORS: 0!
SUM_SORT_MERGE_PASSES: 0!
SUM_WARNINGS: 0!
SUM_SORT_RANGE: 0!
SUM_ROWS_AFFECTED: 0!
SUM_SORT_ROWS: 0!
SUM_ROWS_SENT: 64133!
SUM_SORT_SCAN: 0!
SUM_ROWS_EXAMINED: 187363!
SUM_NO_INDEX_USED: 59625!
SUM_CREATED_TMP_DISK_TABLES: 59624!
SUM_NO_GOOD_INDEX_USED: 0!
SUM_CREATED_TMP_TABLES: 59624!
FIRST_SEEN: 2014-05-20 11:26:54!
LAST_SEEN: 2014-05-21 14:50:54
Statements With Temporary Tables
mysql> select sys.format_statement(digest_text) as stmt,!
-> sum_created_tmp_tables as in_memory,!
-> sum_created_tmp_disk_tables as on_disk,!
-> round(sum_created_tmp_tables / count_star) as avg_per_stmt,!
-> round((sum_created_tmp_disk_tables/sum_created_tmp_tables) * 100) as to_disk_pct!
-> from events_statements_summary_by_digest!
-> where sum_created_tmp_tables > 0!
-> order by sum_created_tmp_disk_tables desc, sum_created_tmp_tables desc limit 5;!
+-------------------------------------------------------------------+-------+-----------+---------+--------------+-------------+!
| stmt | total | in_memory | on_disk | avg_per_stmt | to_disk_pct |!
+-------------------------------------------------------------------+-------+-----------+---------+--------------+-------------+!
| SELECT DISTINCTROW àgent0_` . ... gent` àgent0_` INNER JOIN ... | 64668 | 60116 | 60116 | 1 | 100 |!
| SELECT DISTINCTROW `mysqlconne ... conne0_` . `socketPath` AS ... | 5605 | 5604 | 5604 | 1 | 100 |!
| SELECT * FROM ( SELECT digest ... ed AS `noIndexUsedCount` , ... | 1431 | 10017 | 4293 | 7 | 43 |!
| SELECT `s` . ìdentityId` , `s ... `subject_id` = `s` . ìd` ... | 1406 | 4218 | 2812 | 3 | 67 |!
| SELECT plugin_name FROM inform ... atus = ? ORDER BY plugin_name | 1726 | 1726 | 1726 | 1 | 100 |!
+-------------------------------------------------------------------+-------+-----------+---------+--------------+-------------+!
Statements With Full Table Scans
mysql> select sys.format_statement(digest_text) as stmt,!

-> sum_no_index_used as scan_count,!
-> round((sum_no_index_used/count_star) * 100) as scan_pct,!
-> sum_rows_examined as rows_scanned!
-> from events_statements_summary_by_digest!
-> where digest_text like 'select%'!
-> and (sum_no_index_used > 0 OR sum_no_good_index_used > 0)!
-> order by rows_scanned desc limit 5;!
+-------------------------------------------------------------------+-------+------------+----------+--------------+!
| stmt | total | scan_count | scan_pct | rows_scanned |!
+-------------------------------------------------------------------+-------+------------+----------+--------------+!
| SELECT `hid` , TIMESTAMP , èn ... DIV ? AS `slice` , `hid` , ... | 4258 | 4257 | 100 | 31644768 |!
| SELECT TIMESTAMP , SUM ( `sele ... by_bucket` GROUP BY TIMESTAMP | 807 | 807 | 100 | 4657959 |!
| SELECT TIMESTAMP , SUM ( òpti ... by_bucket` GROUP BY TIMESTAMP | 807 | 807 | 100 | 4656726 |!
| SELECT TIMESTAMP , SUM ( `tota ... by_bucket` GROUP BY TIMESTAMP | 807 | 807 | 100 | 4454821 |!
| SELECT * FROM ( SELECT digest ... ed AS `noIndexUsedCount` , ... | 1434 | 1434 | 100 | 3874018 |!
+-------------------------------------------------------------------+-------+------------+----------+--------------+
Tracing Individual Statement Activity
By taking all available data within the %_history_long tables, we can

build a complete picture of where latency lines within statements
We can link the histories of Statements, Stages and Waits using the
event_id and nesting_event_id columns, which define hierarchy
!
!
This means we can graph their relationships!
This is only really effective on development / test systems, production

systems with high concurrency / throughput age history very quickly,
and turning the %_history_long tables on with all instrumentation on is
not recommend generally
Good to disable all other threads, or all actors, and just enable a
single thread in development / test environments, to get a full trace
Procedure to dump the graph data is available within the MySQL
SYS schema
!
!
Memory Usage
Metadata Locking
Replication Configuration & Status Insert Picture Here
Prepared Statements
Transactions
Stored Programs
6 Copyright
!7Copyright 2014,
2014, Oracle
Oracle and/or
reserved.
75 Tables (+23)
784 Instruments (+239)
Instrument Type Event Class
Transactions transaction
Memory memory/%
Metadata Locks wait/lock/metadata/%
WL#3249!
PERFORMANCE SCHEMA, Instrument memory usage
Understanding where MySQL can allocate memory can help us to find
the cause in most cases. It is not as straightforward as it should be and
Im very hopeful future releases of MySQL, MariaDB or Drizzle bring
improvements in this space allowing us to see directly for what
purpose memory is allocated and so detect all kinds of memory usage
problems easier.
Peter Zaitsev, CEO, Percona
http://www.mysqlperformanceblog.com/2012/03/21/troubleshooting-mysql-memory-usage/
0 Copyright
!8Copyright 2014,
2014, Oracle
Oracle and/or
reserved.
Instrument memory usage
Added 212 different memory instrumentation types so far

Records current, high and low water marks of allocations
Does not track latency of memory allocation
5 new tables
memory_summary_by_account_by_event_name
memory_summary_by_host_by_event_name
memory_summary_by_thread_by_event_name
memory_summary_by_user_by_event_name
memory_summary_global_by_event_name
Instrument memory usage - global summary
mysql> SELECT * FROM sys.memory_global_by_current_allocated\G!
*************************** 1. row ***************************!
event_name: memory/performance_schema/internal_buffers!
current_count: 60!
current_alloc: 497.00 MiB!
current_avg_alloc: 8.28 MiB!
high_count: 60!
high_alloc: 497.00 MiB!
high_avg_alloc: 8.28 MiB!
*************************** 2. row ***************************!
event_name: memory/mysys/KEY_CACHE!
current_count: 3!
current_avg_alloc: 2.67 MiB!
high_count: 3!
high_alloc: 8.00 MiB!
high_avg_alloc: 2.67 MiB
Instrument memory usage - thread summaries
mysql> select * from memory_by_thread_by_current_allocated\G!
*************************** 1. row ***************************!
user: sql/main!
current_count: 2407!
current_avg_alloc: 4.63 KiB!
current_max_alloc: 8.00 MiB!
total_allocated: 30.55 MiB!
thread_id: 1!
*************************** 2. row ***************************!
user: mem@localhost!
current_count: 1914!
current_avg_alloc: 824 bytes!
current_max_alloc: 816.67 KiB!
total_allocated: 9.25 GiB!
thread_id: 4336
Instrument memory usage - thread details
mysql> SELECT event_name,!
-> sys.format_bytes(current_number_of_bytes_used) AS current_used!
-> FROM performance_schema.memory_summary_by_thread_by_event_name!
-> WHERE thread_id = 24!
-> ORDER BY current_number_of_bytes_used DESC;!
+-----------------------------------------------------+--------------+!
| event_name | current_used |!
+-----------------------------------------------------+--------------+!
| memory/sql/Filesort_buffer::sort_keys | 255.94 KiB |!
| memory/sql/sp_head::main_mem_root | 103.64 KiB |!
| memory/mysys/IO_CACHE | 64.05 KiB |!
| memory/mysys/lf_dynarray | 46.17 KiB |!
| memory/mysys/array_buffer | 24.20 KiB |!
| memory/sql/thd::main_mem_root | 23.95 KiB |!
| memory/sql/String::value | 16.13 KiB |!
| memory/sql/TABLE | 9.44 KiB |!
| memory/sql/TABLE_SHARE::mem_root | 8.70 KiB |!
| memory/myisam/MI_INFO | 7.07 KiB |!
| memory/sql/THD::transactions::mem_root | 4.02 KiB |!
| memory/myisam/MYISAM_SHARE | 3.29 KiB |!

Instrument memory usage - other details
!
Disabled by default
!
UPDATE setup_instruments
! SET enabled = YES
! WHERE name LIKE memory/%;
!
Still requires InnoDB instrumentation
WL#5879!
PERFORMANCE SCHEMA, MDL lock instrumentation
Unfortunately, its unlikely that Ill be able to create a reproducible test
case, because theres no way to actually see what is happening. I
hope that a future version of MySQL will include a more
comprehensive set of tables for inspecting locks, requests, and waits
at all layers of the server.
Baron Schwartz, CEO, VividCortex
http://www.xaprb.com/blog/2012/08/28/debugging-metadata-locking-in-mysql-5-5/
7 Copyright
!8Copyright 2014,
2014, Oracle
Oracle and/or
reserved.
MDL Lock Instrumentation
Added the wait/lock/metadata/sql/mdl instrument

2 new tables
metadata_locks
table_handles
Metadata Locks Table Structure
+-----------------------+---------------------+------+-----+---------+-------+!
+-----------------------+---------------------+------+-----+---------+-------+!
| OBJECT_TYPE | varchar(64) | NO | | NULL | |!
| OBJECT_SCHEMA | varchar(64) | YES | | NULL | |!
| OBJECT_NAME | varchar(64) | YES | | NULL | |!
| OBJECT_INSTANCE_BEGIN | bigint(20) unsigned | NO | | NULL | |!
| LOCK_TYPE | varchar(32) | NO | | NULL | |!
| LOCK_DURATION | varchar(32) | NO | | NULL | |!
| LOCK_STATUS | varchar(32) | NO | | NULL | |!
| SOURCE | varchar(64) | YES | | NULL | |!
| OWNER_THREAD_ID | bigint(20) unsigned | YES | | NULL | |!
| OWNER_EVENT_ID | bigint(20) unsigned | YES | | NULL | |!
+-----------------------+---------------------+------+-----+---------+-------+
MDL Lock Instrumentation
OBJECT_TYPE
GLOBAL, SCHEMA, TABLE, FUNCTION, PROCEDURE, TRIGGER, EVENT,
COMMIT
LOCK_TYPE
INTENTION_EXCLUSIVE, SHARED, SHARED_HIGH_PRIO,
SHARED_READ, SHARED_WRITE, SHARED_UPGRADABLE,
SHARED_NO_WRITE, SHARED_NO_READ_WRITE, EXCLUSIVE
LOCK_DURATION
STATEMENT, TRANSACTION, EXPLICIT
LOCK_STATUS
PENDING, GRANTED, VICTIM, TIMEOUT, KILLED
Metadata Locks Table Structure
mysql> select object_type as scope, object_schema, object_name, lock_type, lock_duration, lock_status!
-> from metadata_locks!
-> order by object_type = 'global' desc, object_type = 'schema' desc,!
-> object_type = 'table' desc, object_type = 'commit' desc;!
+--------+--------------------+----------------+---------------------+---------------+-------------+!
| scope | object_schema | object_name | lock_type | lock_duration | lock_status |!
+--------+--------------------+----------------+---------------------+---------------+-------------+!
| GLOBAL | NULL | NULL | SHARED | EXPLICIT | GRANTED |!
| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | STATEMENT | PENDING |!
|!
| TABLE | mem__events | action_logs | SHARED_READ | TRANSACTION | GRANTED |!
| TABLE | mem__events | events | SHARED_READ | TRANSACTION | GRANTED |!
| TABLE | performance_schema | metadata_locks | SHARED_READ | TRANSACTION | GRANTED |!
| COMMIT | NULL | NULL | SHARED | EXPLICIT | GRANTED |!
+--------+--------------------+----------------+---------------------+---------------+-------------+
WL#3656!
PERFORMANCE SCHEMA table for!
SHOW SLAVE STATUS
Replication Instrumentation
6 new tables
replication_connection_configuration
replication_connection_status
replication_execute_configuration
replication_execute_status
replication_execute_status_by_coordinator
replication_execute_status_by_worker
Replication Instrumentation
Replication Connection Configuration
mysql> select * from performance_schema.replication_connection_configuration\G
*************************** 1. row ***************************!
HOST: 127.0.0.1!
PORT: 3306!
USER: rpl!
NETWORK_INTERFACE:!
AUTO_POSITION: 1!
SSL_ALLOWED: YES!
SSL_CA_FILE:!
SSL_CA_PATH:!
SSL_CERTIFICATE:!
SSL_CIPHER:!
SSL_KEY:!
SSL_VERIFY_SERVER_CERTIFICATE: YES!
SSL_CRL_FILE:!
SSL_CRL_PATH:!
CONNECTION_RETRY_INTERVAL: 40!
CONNECTION_RETRY_COUNT: 10
Replication Connection Status
mysql> select * from replication_connection_status\G!

*************************** 1. row ***************************!
SOURCE_UUID:!
THREAD_ID: NULL!
SERVICE_STATE: CONNECTING!
RECEIVED_TRANSACTION_SET:!
LAST_ERROR_NUMBER: 1045!
LAST_ERROR_MESSAGE: error connecting to master 'repl@localhost:
5613' - retry-time: 60 retries: 1!
LAST_ERROR_TIMESTAMP: 2014-04-02 05:41:20
Replication Execute Status
Very high level overview of status
mysql> select * from replication_execute_status;!

+---------------+-----------------+!
| SERVICE_STATE | REMAINING_DELAY |!
+---------------+-----------------+!
| ON | NULL |!
+---------------+-----------------+
Replication Coordinator Status
Status of Coordinator thread in multi-threaded replication
mysql> select * from replication_execute_status_by_coordinator\G!

*************************** 1. row ***************************!
THREAD_ID: 1281!
SERVICE_STATE: ON!
LAST_ERROR_NUMBER: 0!
LAST_ERROR_MESSAGE:!
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
Replication Worker Status
Status of worker threads within multi-threaded replication
mysql> select * from replication_execute_status_by_worker;!

+-----------+-----------+---------------+-----------------------------------------+-------------------+--------------------+----------------------+!
| WORKER_ID | THREAD_ID | SERVICE_STATE | LAST_SEEN_TRANSACTION | LAST_ERROR_NUMBER | LAST_ERROR_MESSAGE | LAST_ERROR_TIMESTAMP |!
+-----------+-----------+---------------+-----------------------------------------+-------------------+--------------------+----------------------+!
| 1 | 1282 | ON | | 0 | | 0000-00-00 00:00:00 |!
| 2 | 1283 | ON | | 0 | | 0000-00-00 00:00:00 |!
| 3 | 1284 | ON | | 0 | | 0000-00-00 00:00:00 |!
| 4 | 1285 | ON | | 0 | | 0000-00-00 00:00:00 |!
| 5 | 1286 | ON | | 0 | | 0000-00-00 00:00:00 |!
| 6 | 1287 | ON | | 0 | | 0000-00-00 00:00:00 |!
| 7 | 1288 | ON | | 0 | | 0000-00-00 00:00:00 |!
| 8 | 1289 | ON | 181f2b36-a0b4-11e3-9ac7-1025863574a7:16 | 0 | | 0000-00-00 00:00:00 |!
+-----------+-----------+---------------+-----------------------------------------+-------------------+--------------------+----------------------+
WL#5768!
PERFORMANCE SCHEMA, prepared statements
instrumentation
Prepared Statement Instrumentation
Instrumentation depends on already existing statement instruments

statement/com/prepare, statement/com/execute
statement/sql/prepare_sql, statement/sql/execute_sql
1 new table
prepared_statements_instances
Prepared Statement Instances
mysql> select * from performance_schema.prepared_statements_instances\G!
*************************** 1. row ***************************!
OBJECT_INSTANCE_BEGIN: 140198306602144!
SUM_LOCK_TIME: 0!
STATEMENT_ID: 1! SUM_ERRORS: 0!
STATEMENT_NAME: stmt1! SUM_WARNINGS: 0!
SQL_TEXT: select * from test.t1! SUM_ROWS_AFFECTED: 0!
SUM_ROWS_SENT: 0!
OWNER_THREAD_ID: 54003! SUM_ROWS_EXAMINED: 0!
OWNER_EVENT_ID: 935! SUM_CREATED_TMP_DISK_TABLES: 0!
OWNER_OBJECT_TYPE: NULL! SUM_CREATED_TMP_TABLES: 0!
OWNER_OBJECT_SCHEMA: NULL! SUM_SELECT_FULL_JOIN: 0!
SUM_SELECT_FULL_RANGE_JOIN: 0!
OWNER_OBJECT_NAME: NULL! SUM_SELECT_RANGE: 0!
TIMER_PREPARE: 15386274000! SUM_SELECT_RANGE_CHECK: 0!
COUNT_REPREPARE: 0! SUM_SELECT_SCAN: 0!
SUM_SORT_MERGE_PASSES: 0!
COUNT_EXECUTE: 1! SUM_SORT_RANGE: 0!
SUM_TIMER_EXECUTE: 217699000! SUM_SORT_ROWS: 0!
MIN_TIMER_EXECUTE: 217699000! SUM_SORT_SCAN: 0!
AVG_TIMER_EXECUTE: 217699000! SUM_NO_INDEX_USED: 0!
SUM_NO_GOOD_INDEX_USED: 0
MAX_TIMER_EXECUTE: 217699000
WL#5864!
PERFORMANCE SCHEMA, instrument
TRANSACTIONS
Transaction Instrumentation
Added the transaction instrument
8 new tables
events_transactions_current
events_transactions_history
events_transactions_history_long
events_transactions_summary_by_account_by_event_name
events_transactions_summary_by_host_by_event_name
events_transactions_summary_by_thread_by_event_name
events_transactions_summary_by_user_by_event_name
events_transactions_summary_global_by_event_name
Transaction Instrumentation
Exposes details of transactions in raw or summary views

Show details such as
Transaction latency
Isolation levels
Auto commit
Savepoint info
GTID or transaction IDs
Link transactions to the statements within the transactions
Current Transaction Details
mysql> select * from events_transactions_current\G!
*************************** 1. row ***************************!
THREAD_ID: 1!
EVENT_ID: 23733!
EVENT_NAME: transaction! ACCESS_MODE: READ WRITE!
STATE: COMMITTED! ISOLATION_LEVEL: REPEATABLE READ!
TRX_ID: 281479898269256! AUTOCOMMIT: YES!
GTID: NULL! NUMBER_OF_SAVEPOINTS: 0!
NUMBER_OF_ROLLBACK_TO_SAVEPOINT: 0!
XID: NULL!
NUMBER_OF_RELEASE_SAVEPOINT: 0!
XA_STATE: NULL! OBJECT_INSTANCE_BEGIN: NULL!
SOURCE: handler.cc:1246! NESTING_EVENT_ID: NULL!
TIMER_START: 31140612726000! NESTING_EVENT_TYPE: NULL
TIMER_END: 31140647445000!
TIMER_WAIT: 34719000
Transaction Summary - per user
mysql> select * from events_transactions_summary_by_user_by_event_name\G!
*************************** 1. row ***************************!
USER: mem!
EVENT_NAME: transaction!
COUNT_STAR: 400044!
SUM_TIMER_WAIT: 21208048458267000!
MIN_TIMER_WAIT: 160744000!
AVG_TIMER_WAIT: 53014289000!
MAX_TIMER_WAIT: 24028520397000!
COUNT_READ_WRITE: 400044!
SUM_TIMER_READ_WRITE: 21208048458267000!
MIN_TIMER_READ_WRITE: 160744000!
AVG_TIMER_READ_WRITE: 53014289000!
MAX_TIMER_READ_WRITE: 24028520397000!
COUNT_READ_ONLY: 0!
SUM_TIMER_READ_ONLY: 0!
MIN_TIMER_READ_ONLY: 0!
AVG_TIMER_READ_ONLY: 0!
MAX_TIMER_READ_ONLY: 0
WL#5766!
PERFORMANCE SCHEMA, stored programs
instrumentation
Stored Program Instrumentation
Tracks Stored Procedures, Stored Functions, Triggers and Events
Added 16 new statement/sp/% instruments
Expose the different work flows that stored programs use, such as
cursor operations, workflow controls etc.
Integrated in to the normal statement instrumentation
1 new tables
events_statements_summary_by_program
Stored Program Summary
mysql> select * from events_statements_summary_by_program\G!
*************************** 1. row ***************************!
OBJECT_TYPE: PROCEDURE!
OBJECT_SCHEMA: ps_demo!
OBJECT_NAME: ps_demo_proc! SUM_ROWS_EXAMINED: 1!
COUNT_STAR: 1! SUM_CREATED_TMP_DISK_TABLES: 0!
SUM_TIMER_WAIT: 6970931000! SUM_CREATED_TMP_TABLES: 0!
MIN_TIMER_WAIT: 6970931000! SUM_SELECT_FULL_JOIN: 0!
AVG_TIMER_WAIT: 6970931000! SUM_SELECT_FULL_RANGE_JOIN: 0!
MAX_TIMER_WAIT: 6970931000! SUM_SELECT_RANGE: 0!
COUNT_STATEMENTS: 5! SUM_SELECT_RANGE_CHECK: 0!
SUM_STATEMENTS_WAIT: 6802181000! SUM_SELECT_SCAN: 0!
MIN_STATEMENTS_WAIT: 16372000! SUM_SORT_MERGE_PASSES: 0!
AVG_STATEMENTS_WAIT: 1360436000! SUM_SORT_RANGE: 0!
MAX_STATEMENTS_WAIT: 6484366000! SUM_SORT_ROWS: 0!
SUM_LOCK_TIME: 176401000000! SUM_SORT_SCAN: 0!
SUM_ERRORS: 0! SUM_NO_INDEX_USED: 0!
SUM_WARNINGS: 0! SUM_NO_GOOD_INDEX_USED: 0
SUM_ROWS_AFFECTED: 1!
SUM_ROWS_SENT: 0
Stored Program Instrumentation
Like Transaction instrumentation, stored programs link to the

statements within them, or even link to transactions within them,
which in turn link to their statements
Full Hierarchy of New Instrumentation
!113
!114
!115
!116
!117
!118
The MySQL SYS Schema
!119
MySQL SYS Schema Overview
Originally called ps_helper

Started as a collection of views, procedures and functions, designed
to make reading raw Performance Schema data easier
Implements many of the common DBA and Developer use cases,
including many of those shown already
Now bundled within MySQL Workbench 6.1
Available on GitHub
https://github.com/MarkLeith/mysql-sys
Loading MySQL SYS
$ git clone https://github.com/MarkLeith/mysql-sys.git /tmp/mysql-sys

$ cd /tmp/mysql-sys
$ mysql -u user -p < sys_<version>.sql
!
<version> can be 56 or 57, for 5.6 and 5.7 respectively
Loading MySQL SYS
Once loaded creates a new sys schema containing all objects
mysql> select object_type, count!

-> from sys.schema_object_overview!
-> where db = 'sys';!
+-------------+-------+!
| object_type | count |!
+-------------+-------+!
| PROCEDURE | 16 |!
| FUNCTION | 8 |!
| VIEW | 71 |!
+-------------+-------+
MySQL SYS Functions
Make the raw data more readable to a human

format_time() / format_bytes()
Compress data for CLI output
format_statement() / format_path()
Extract some data to assist with JOIN in certain cases
extract_[schema|table]_from_file_name()
Other functions
ps_is_account_enabled() / ps_thread_stack()
MySQL SYS Views
Reference set of views solving various admin use cases

Build upon both Performance Schema and INFORMATION_SCHEMA
Both formatted and raw views are available
All raw views are prefixed with x$
Allows tooling to poll raw views, but humans to use normal ones
User Summary Views
mysql> show tables like 'user%';!

+-----------------------------------+!
| Tables_in_sys (user%) |!
High level user overview +-----------------------------------+!
| user_summary |!
Breakdowns of IO usage
| user_summary_by_file_io |!
Drill in to stages per user | user_summary_by_file_io_type |!
| user_summary_by_stages |!
Drill in to statement details per
| user_summary_by_statement_latency |!
user | user_summary_by_statement_type |!
+-----------------------------------+
User Summary (5.7)
mysql> select * from user_summary\G!
*************************** 1. row ***************************!
user: mark!
statements: 3072!
statement_latency: 1.77 s!
statement_avg_latency: 575.29 us!
table_scans: 7!
file_ios: 20043!
file_io_latency: 346.79 ms!
current_connections: 1!
total_connections: 1!
unique_hosts: 1!
current_memory: 515.81 KiB!
total_memory_allocated: 30.69 MiB
User File IO Summary
mysql> select * from user_summary_by_file_io_type where user != 'background';!
+------+--------------------------------------+---------+-------------+-------------+!
| user | event_name | total | latency | max_latency |!
+------+--------------------------------------+---------+-------------+-------------+!
| mark | wait/io/file/myisam/dfile | 19540 | 215.20 ms | 120.79 ms |!
| mark | wait/io/file/myisam/kfile | 501 | 131.58 ms | 57.74 ms |!
| mark | wait/io/file/sql/dbopt | 2 | 17.29 us | 9.17 us |!
| root | wait/io/file/innodb/innodb_log_file | 2079432 | 00:28:27.62 | 1.30 s |!
| root | wait/io/file/myisam/kfile | 5195927 | 00:17:07.33 | 1.17 s |!
| root | wait/io/file/myisam/dfile | 1533813 | 00:07:27.59 | 1.18 s |!
| root | wait/io/file/sql/binlog | 3517008 | 00:03:07.34 | 1.42 s |!
| root | wait/io/file/innodb/innodb_data_file | 20720 | 00:02:37.81 | 492.23 ms |!
| root | wait/io/file/sql/FRM | 41412 | 3.17 s | 80.43 ms |!
| root | wait/io/file/sql/dbopt | 110993 | 1.23 s | 63.27 ms |!
| root | wait/io/file/sql/binlog_index | 80 | 1.16 s | 439.79 ms |!
| root | wait/io/file/csv/metadata | 16 | 175.97 ms | 79.53 ms |!
| root | wait/io/file/archive/data | 3345 | 18.35 ms | 5.05 ms |!
| root | wait/io/file/sql/file_parser | 142 | 5.80 ms | 331.82 us |!
| root | wait/io/file/sql/misc | 99 | 760.67 us | 25.19 us |!
| root | wait/io/file/csv/data | 6 | 596.99 us | 348.11 us |!
+------+--------------------------------------+---------+-------------+-------------+
User Statement Latency Summary
mysql> select * from user_summary_by_statement_latency where user != 'background';!

+------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+
| user | total | total_latency | max_latency | lock_latency | rows_sent | rows_examined | rows_affected | full_scans |
+------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+
| root | 8322401 | 19.37h | 00:07:08.11 | 00:40:26.25 | 10110692 | 68391445 | 4198113 | 189531 |
| mark | 3072 | 1.77 s | 1.73 s | 1.96 ms | 534 | 15389 | 0 | 7 |
+------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+
User Statement Type Summary
mysql> select * from user_summary_by_statement_type where user != 'background';!
+------+-----------------------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+!
| user | statement | total | total_latency | max_latency | lock_latency | rows_sent | rows_examined | rows_affected | full_scans |!
+------+-----------------------+---------+---------------+-------------+--------------+-----------+---------------+---------------+------------+!
| mark | select | 4 | 1.69 s | 1.69 s | 408.00 us | 212 | 15067 | 0 | 1 |!
| mark | Field List | 146 | 67.91 ms | 40.02 ms | 1.11 ms | 0 | 0 | 0 | 0 |!
| mark | jump_if_not | 2496 | 2.68 ms | 7.07 us | 0 ps | 0 | 0 | 0 | 0 |!
| mark | show_tables | 4 | 1.70 ms | 545.41 us | 296.00 us | 292 | 292 | 0 | 4 |!
| mark | freturn | 418 | 1.15 ms | 9.29 us | 0 ps | 0 | 0 | 0 | 0 |!
| mark | show_databases | 2 | 849.40 us | 431.05 us | 146.00 us | 30 | 30 | 0 | 2 |!
| mark | Init DB | 2 | 114.95 us | 59.58 us | 0 ps | 0 | 0 | 0 | 0 |!
| root | commit | 2463190 | 13.33h | 12.18 s | 0 ps | 0 | 0 | 0 | 0 |!
| root | insert | 2501839 | 3.78h | 6.64 s | 00:31:18.63 | 192 | 0 | 3674716 | 0 |!
| root | select | 1119999 | 1.74h | 16.34 s | 00:07:42.99 | 6187336 | 63933992 | 0 | 163913 |!
| root | update | 494217 | 00:15:23.09 | 4.83 s | 00:01:05.05 | 126 | 496604 | 495334 | 0 |!
| root | Binlog Dump GTID | 1 | 00:06:12.69 | 00:06:12.69 | 0 ps | 0 | 0 | 0 | 0 |!
| root | delete | 206176 | 00:01:59.54 | 2.22 s | 6.69 s | 12 | 20201 | 20203 | 0 |!
| root | set_option | 1068360 | 00:01:10.73 | 45.37 ms | 0 ps | 0 | 0 | 0 | 0 |!
| root | show_engine_status | 2222 | 46.62 s | 4.96 s | 0 ps | 0 | 0 | 0 | 0 |!
| root | rollback | 64527 | 26.21 s | 407.15 ms | 0 ps | 0 | 0 | 0 | 0 |!
| root | show_variables | 11219 | 12.28 s | 255.00 ms | 1.19 s | 2542172 | 2542172 | 0 | 11219 |!
| root | show_binlogs | 1850 | 8.96 s | 1.64 s | 0 ps | 0 | 0 | 0 | 0 |!
| root | show_tables | 4250 | 6.86 s | 123.34 ms | 182.80 ms | 5277 | 5277 | 0 | 4250 |!
| root | show_status | 5485 | 6.50 s | 242.60 ms | 581.02 ms | 1331264 | 1331264 | 0 | 5485 |!
IO Summary Views
mysql> show tables like 'io_%';!

+------------------------------+!
| Tables_in_sys (io_%) |!
IO breakdown by thread +------------------------------+!
| io_by_thread_by_latency |!
Global summaries by file and wait
| io_global_by_file_by_bytes |!
class, by both bytes and latency | io_global_by_file_by_latency |!
Stream of last raw file IO stats | io_global_by_wait_by_bytes |!
| io_global_by_wait_by_latency |!
| latest_file_io |!
+------------------------------+
IO Per Thread Summary
mysql> select * from io_by_thread_by_latency;!
+---------------------+---------+---------------+-------------+-------------+-------------+-----------+----------------+!
| user | total | total_latency | min_latency | avg_latency | max_latency | thread_id | processlist_id |!
+---------------------+---------+---------------+-------------+-------------+-------------+-----------+----------------+!
| io_write_thread | 1270599 | 00:28:51.42 | 1.46 us | 1.36 ms | 1.39 s | 9 | NULL |!
| page_cleaner_thread | 3356577 | 00:21:01.53 | 410.93 ns | 2.08 ms | 1.49 s | 19 | NULL |!
| io_write_thread | 338216 | 00:14:31.16 | 1.67 us | 2.58 ms | 1.17 s | 12 | NULL |!
| io_log_thread | 47023 | 00:04:06.10 | 3.65 us | 5.23 ms | 538.77 ms | 4 | NULL |!
| io_write_thread | 290066 | 00:02:20.99 | 2.78 us | 486.07 us | 703.20 ms | 10 | NULL |!
| io_write_thread | 221417 | 00:02:12.64 | 1.85 us | 599.05 us | 773.64 ms | 11 | NULL |!
| root@localhost | 587531 | 00:02:03.29 | 444.86 ns | 200.24 us | 372.09 ms | 2515 | 2495 |!
| io_read_thread | 5102 | 00:01:09.66 | 12.24 us | 13.65 ms | 281.13 ms | 5 | NULL |!
| srv_master_thread | 55199 | 58.11 s | 693.68 ns | 1.40 ms | 504.03 ms | 17 | NULL |!
| root@localhost | 1622373 | 41.20 s | 486.33 ns | 25.40 us | 359.49 ms | 23 | 3 |!
| io_read_thread | 2401 | 37.27 s | 11.86 us | 15.52 ms | 258.97 ms | 6 | NULL |!
| srv_purge_thread | 3874 | 12.16 s | 433.55 ns | 3.14 ms | 246.79 ms | 18 | NULL |!
| main | 11083 | 3.12 s | 588.12 ns | 547.21 us | 234.02 ms | 1 | NULL |!
| root@localhost | 5985 | 1.48 s | 625.82 ns | 1.06 ms | 239.17 ms | 137759 | 137739 |!
| root@localhost | 157 | 434.38 ms | 497.64 ns | 8.09 ms | 205.84 ms | 178521 | 178501 |!
| mark@localhost | 20043 | 346.79 ms | 418.47 ns | 94.10 us | 120.79 ms | 150606 | 150586 |!
| root@localhost | 138 | 204.72 ms | 486.33 ns | 2.68 ms | 86.54 ms | 178524 | 178504 |!
| root@localhost | 415 | 126.43 ms | 471.25 ns | 195.15 us | 37.21 ms | 178269 | 178249 |!
| root@localhost | 9 | 106.21 ms | 6.40 us | 8.85 ms | 103.88 ms | 178522 | 178502 |!
+---------------------+---------+---------------+-------------+-------------+-------------+-----------+----------------+!
IO Per File Summaries
mysql> select * from io_global_by_file_by_latency limit 1\G!
*************************** 1. row ***************************!
file: @@datadir/ibdata1!
total: 1395714!
total_latency: 00:28:53.06!
count_read: 1006!
read_latency: 4.51 s!
count_write: 1326529!
write_latency: 33.39 s!
count_misc: 68179!
misc_latency: 00:28:15.16!
!
mysql> select * from io_global_by_file_by_bytes limit 1\G!
*************************** 1. row ***************************!
file: @@datadir/ibdata1!
count_read: 1006!
total_read: 17.69 MiB!
avg_read: 18.00 KiB!
count_write: 1327221!
total_written: 77.82 GiB!
avg_write: 61.48 KiB!
total: 77.84 GiB!
write_pct: 99.98
Latest File IO
mysql> select * from latest_file_io;!
+-----------------------------+----------------------------------+-----------+-----------+-----------+!
| thread | file | latency | operation | requested |!
+-----------------------------+----------------------------------+-----------+-----------+-----------+!
| root@localhost:63153:179371 | @@datadir/ib_logfile1 | 7.16 us | lock | NULL |!
| root@localhost:63153:179371 | @@datadir/ib_logfile1 | 6.16 us | write | 1.50 KiB |!
| root@localhost:63153:179371 | @@datadir/ib_logfile1 | 14.36 ms | sync | NULL |!
| root@localhost:63153:179371 | @@datadir/cerberus-bin.000012 | 28.25 us | write | 905 bytes |!
| root@localhost:57487:3 | @@datadir/cerberus-bin.000012 | 32.12 us | read | 905 bytes |!
| root@localhost:63177:179394 | @@datadir/ib_logfile1 | 213.32 us | sync | NULL |!
Schema Analysis Views
mysql> show tables like 'schema%';!

+-------------------------------------+!
| Tables_in_sys (schema%) |!
Object overview +-------------------------------------+!
| schema_index_statistics |!
Table usage stats | schema_object_overview |!
Index usage stats | schema_table_statistics |!
| schema_table_statistics_with_buffer |!
| schema_tables_with_full_table_scans |!
| schema_unused_indexes |!
+-------------------------------------+
Schema Object Overview
mysql> select * from schema_object_overview;!
+-------------------------+---------------+-------+!
| db | object_type | count |!
+-------------------------+---------------+-------+!
| information_schema | SYSTEM VIEW | 60 |!
| mem__advisors | BASE TABLE | 3 |!
| mem__advisors | INDEX (BTREE) | 5 |!
| mem__advisor_text | BASE TABLE | 2 |!
| mem__advisor_text | INDEX (BTREE) | 5 |!
| mem__bean_config | BASE TABLE | 4 |!
| mem__bean_config | INDEX (BTREE) | 6 |!
| mem__config | BASE TABLE | 12 |!
| mem__config | INDEX (BTREE) | 21 |!
| mem__enterprise | BASE TABLE | 2 |!
| mem__enterprise | INDEX (BTREE) | 3 |!
| mem__events | BASE TABLE | 32 |!
| mem__events | INDEX (BTREE) | 69 |!
| mem__instruments | BASE TABLE | 117 |!
| mem__instruments | INDEX (BTREE) | 582 |
Table statistics
mysql> select * from schema_table_statistics limit 1\G!
*************************** 1. row ***************************!
table_schema: mem__quan!
table_name: normalized_statements_by_server_by_schema_data!
rows_fetched: 239685!
fetch_latency: 17.10 s!
rows_inserted: 1102895!
insert_latency: 00:52:18.10!
rows_updated: 55910!
update_latency: 00:02:28.62!
rows_deleted: 17899!
delete_latency: 3.98 s!
io_read_requests: 20639!
io_read: 324.94 MiB!
io_read_latency: 00:03:04.10!
io_write_requests: 1965973!
io_write: 40.36 GiB!
io_write_latency: 25.46 s!
io_misc_requests: 18217!
io_misc_latency: 00:14:06.02
Index statistics
mysql> select * from schema_index_statistics limit 1\G!

*************************** 1. row ***************************!
table_schema: mem__inventory!
table_name: mysqlconnectionconfiguration!
index_name: PRIMARY!
rows_selected: 131399!
select_latency: 00:07:05.03!
rows_inserted: 0!
insert_latency: 0 ps!
rows_updated: 41608!
update_latency: 21.96 s!
rows_deleted: 0!
delete_latency: 0 ps
InnoDB Buffer Usage Views
mysql> show tables like 'innodb%';!

Buffer usage by schema +-------------------------------+!
| Tables_in_sys (innodb%) |!
Buffer usage by table +-------------------------------+!
| innodb_buffer_stats_by_schema |!
| innodb_buffer_stats_by_table |!
+-------------------------------+
Buffer Usage By Schema
mysql> select * from innodb_buffer_stats_by_schema;!

+------------------+------------+------------+-------+--------------+-----------+-------------+!
| object_schema | allocated | data | pages | pages_hashed | pages_old | rows_cached |!
+------------------+------------+------------+-------+--------------+-----------+-------------+!
| mem__quan | 69.94 MiB | 45.39 MiB | 4476 | 4476 | 4476 | 123226 |!
| mem__instruments | 20.70 MiB | 10.75 MiB | 1325 | 1325 | 1325 | 82812 |!
| mem__events | 3.08 MiB | 2.09 MiB | 197 | 197 | 197 | 3411 |!
| mem__inventory | 1.34 MiB | 52.49 KiB | 86 | 86 | 86 | 18 |!
| mysql | 432.00 KiB | 251.34 KiB | 27 | 27 | 27 | 2252 |!
| mem__config | 144.00 KiB | 3.86 KiB | 9 | 9 | 9 | 29 |!
+------------------+------------+------------+-------+--------------+-----------+-------------+
Buffer Usage By Table
mysql> select * from innodb_buffer_stats_by_table limit 20;!
+------------------+------------------------------------------------+------------+------------+-------+--------------+-----------+-------------+!
| object_schema | object_name | allocated | data | pages | pages_hashed | pages_old | rows_cached |!
+------------------+------------------------------------------------+------------+------------+-------+--------------+-----------+-------------+!
| mem__quan | normalized_statements_by_server_by_schema_data | 49.97 MiB | 33.12 MiB | 3198 | 3198 | 3198 | 238845 |!
| mem__quan | example_statements | 15.95 MiB | 9.71 MiB | 1021 | 1021 | 1021 | 32435 |!
| mem__events | events | 2.75 MiB | 1.97 MiB | 176 | 176 | 176 | 5044 |!
| mem__quan | normalized_statements | 2.19 MiB | 1.42 MiB | 140 | 140 | 140 | 2297 |!
| mem__quan | normalized_statements_by_server_by_schema | 1.06 MiB | 693.58 KiB | 68 | 68 | 68 | 3840 |!
| mem__instruments | innodbundologsdata | 1.05 MiB | 754.07 KiB | 67 | 67 | 67 | 5529 |!
| mem__instruments | fsstatistics | 736.00 KiB | 603.64 KiB | 46 | 46 | 46 | 2092 |!
| mem__instruments | networktrafficadvisor_networktraffic | 496.00 KiB | 291.38 KiB | 31 | 31 | 31 | 1160 |!
| mysql | innodb_index_stats | 384.00 KiB | 230.37 KiB | 24 | 24 | 24 | 1986 |!
| mem__quan | explains | 368.00 KiB | 274.42 KiB | 23 | 23 | 23 | 333 |!
| mem__instruments | innodbchangebufferusagedata | 272.00 KiB | 126.94 KiB | 17 | 17 | 17 | 1104 |!
| mem__instruments | rowscanratiodata | 272.00 KiB | 141.18 KiB | 17 | 17 | 17 | 1207 |!
| mem__instruments | connectionsmaxdata | 256.00 KiB | 150.98 KiB | 16 | 16 | 16 | 1262 |!
| mem__instruments | innodbopenfilesdata | 256.00 KiB | 145.30 KiB | 16 | 16 | 16 | 1170 |!
| mem__instruments | myisamkeybufferusagedata | 256.00 KiB | 155.60 KiB | 16 | 16 | 16 | 1192 |!
| mem__instruments | connectionsdata | 240.00 KiB | 129.12 KiB | 15 | 15 | 15 | 1106 |!
| mem__instruments | diskioopstotaldata | 240.00 KiB | 143.94 KiB | 15 | 15 | 15 | 533 |!
| mem__instruments | diskiototaldata | 240.00 KiB | 143.94 KiB | 15 | 15 | 15 | 533 |!
| mem__instruments | innodbchangebufferactivitydata | 240.00 KiB | 124.43 KiB | 15 | 15 | 15 | 1003 |!
| mem__instruments | innodbcheckpointagedata | 240.00 KiB | 122.65 KiB | 15 | 15 | 15 | 1052 |!
+------------------+------------------------------------------------+------------+------------+-------+--------------+-----------+-------------+
Statement Analysis Views
mysql> show tables like 'statement%';!

+---------------------------------------------+!
| Tables_in_sys (statement%) |!
Statement overview +---------------------------------------------+!
| statement_analysis |!
Find statements by errors, full | statements_with_errors_or_warnings |!
| statements_with_full_table_scans |!
scans, sorting, temporary tables | statements_with_runtimes_in_95th_percentile |!
| statements_with_sorting |!
Find the statements with the | statements_with_temp_tables |!
longest runtimes +---------------------------------------------+
Statement Analysis
mysql> select * from statement_analysis limit 11\G!
*************************** 1. row ***************************!
query: INSERT INTO `mem__quan` . `nor ... nDuration` = IF ( VALUES ( ...!
db: mem!
full_scan:!
exec_count: 1110067!
err_count: 0!
warn_count: 0!
total_latency: 1.93h!
max_latency: 5.03 s!
avg_latency: 6.27 ms!
lock_latency: 00:18:29.18!
rows_sent: 0!
rows_sent_avg: 0!
rows_examined: 0!
rows_examined_avg: 0!
tmp_tables: 0!
tmp_disk_tables: 0!
rows_sorted: 0!
sort_merge_passes: 0!
digest: d48316a218e95b1b8b72db5e6b177788!
first_seen: 2014-05-20 10:42:17
Statement With Errors And Warnings
mysql> select * from statements_with_errors_or_warnings limit 1\G!

*************************** 1. row ***************************!
query: CREATE TEMPORARY TABLE IF NOT ... _logs` ( ìd` INT8 NOT NULL )!
db: mem!
exec_count: 1725!
errors: 1725!
error_pct: 100.0000!
warnings: 0!
warning_pct: 0.0000!
first_seen: 2014-05-20 10:42:32!
last_seen: 2014-05-21 18:39:22!
digest: 51fb979dbc3910a6294c7cdabd7839c3
Statement With Sorting
mysql> select * from statements_with_sorting limit 1\G!

*************************** 1. row ***************************!
query: SELECT `s` . ìdentityId` , `s ... `subject_id` = `s` . ìd` ...!
db: mem!
exec_count: 1406!
sort_merge_passes: 0!
avg_sort_merges: 0!
sorts_using_scans: 1406!
sort_using_range: 1406!
rows_sorted: 39865!
avg_rows_sorted: 28!
first_seen: 2014-05-20 10:42:08!
last_seen: 2014-05-21 14:04:02!
digest: 9972d1d2cc12b92fe938ec5bd74d2f1d
Wait Analysis Views
mysql> show tables like 'wait%';!

+------------------------------------+!
Wait summaries by class and per | Tables_in_sys (wait%) |!
+------------------------------------+!
instrument globally
| wait_classes_global_by_avg_latency |!
Wait details per user | wait_classes_global_by_latency |!
| waits_by_user_by_latency |!
! | waits_global_by_latency |!
+------------------------------------+
Global Wait Summary
mysql> select * from waits_global_by_latency;!
+------------------------------------------------------+----------+---------------+-------------+-------------+!
| events | total | total_latency | avg_latency | max_latency |!
+------------------------------------------------------+----------+---------------+-------------+-------------+!
| wait/io/table/sql/handler | 24196659 | 1.75h | 259.72 us | 2.29 s |!
| wait/io/file/innodb/innodb_data_file | 5964255 | 1.36h | 822.01 us | 1.49 s |!
| wait/io/file/innodb/innodb_log_file | 2272647 | 00:36:41.37 | 968.64 us | 1.30 s |!
| wait/io/file/myisam/kfile | 5463587 | 00:18:14.16 | 200.26 us | 1.17 s |!
| wait/io/file/myisam/dfile | 1715187 | 00:08:00.18 | 279.96 us | 1.18 s |!
| wait/io/file/sql/binlog | 3722079 | 00:03:23.71 | 54.73 us | 1.42 s |!
| wait/lock/table/sql/handler | 10074274 | 00:01:56.63 | 11.58 us | 186.90 ms |!
| wait/io/socket/sql/client_connection | 3879827 | 25.36 s | 6.54 us | 219.25 ms |!
| wait/io/file/sql/FRM | 43994 | 3.27 s | 74.35 us | 80.43 ms |!
| wait/io/file/sql/dbopt | 116765 | 1.29 s | 11.03 us | 63.27 ms |!
| wait/io/file/sql/binlog_index | 95 | 1.20 s | 12.67 ms | 439.79 ms |!
| wait/io/file/csv/metadata | 32 | 577.39 ms | 18.04 ms | 245.23 ms |!
| wait/synch/mutex/sql/THD::LOCK_thd_data | 10300455 | 221.84 ms | 21.49 ns | 12.62 ms |!
| wait/synch/mutex/sql/THD::LOCK_query_plan | 5656929 | 168.50 ms | 29.78 ns | 11.89 ms |!
| wait/io/file/archive/data | 3533 | 19.11 ms | 5.41 us | 5.05 ms |!
| wait/io/file/mysys/cnf | 5 | 3.75 ms | 750.59 us | 3.60 ms |!
| wait/synch/mutex/myisam/MYISAM_SHARE::intern_lock | 88257 | 2.20 ms | 24.88 ns | 361.49 us |!
| wait/io/file/sql/ERRMSG | 5 | 1.14 ms | 227.99 us | 502.64 us |!
| wait/io/file/csv/data | 12 | 1.03 ms | 85.56 us | 348.11 us |!
User Wait Summary
mysql> select * from waits_by_user_by_latency;!
+------+------------------------------------------------------+----------+---------------+-------------+-------------+!
| user | event | total | total_latency | avg_latency | max_latency |!
+------+------------------------------------------------------+----------+---------------+-------------+-------------+!
| mark | wait/io/file/myisam/dfile | 19540 | 215.20 ms | 11.01 us | 120.79 ms |!
| mark | wait/io/file/myisam/kfile | 501 | 131.58 ms | 262.63 us | 57.74 ms |!
| mark | wait/io/file/sql/dbopt | 2 | 17.29 us | 8.64 us | 9.17 us |!
| root | wait/io/table/sql/handler | 24189319 | 1.74h | 258.77 us | 2.29 s |!
| root | wait/io/file/innodb/innodb_log_file | 2162650 | 00:30:52.87 | 856.76 us | 1.30 s |!
| root | wait/io/file/myisam/kfile | 5460992 | 00:18:12.83 | 200.11 us | 1.17 s |!
| root | wait/io/file/myisam/dfile | 1694624 | 00:07:59.50 | 282.95 us | 1.18 s |!
| root | wait/io/file/sql/binlog | 3711637 | 00:03:23.16 | 54.74 us | 1.42 s |!
| root | wait/io/file/innodb/innodb_data_file | 22577 | 00:03:05.73 | 8.23 ms | 492.23 ms |!
| root | wait/lock/table/sql/handler | 10069000 | 00:01:56.59 | 11.58 us | 186.90 ms |!
| root | wait/io/socket/sql/client_connection | 3858476 | 25.36 s | 6.57 us | 219.25 ms |!
| root | wait/io/file/sql/FRM | 42701 | 3.18 s | 74.52 us | 80.43 ms |!
| root | wait/io/file/sql/dbopt | 116693 | 1.29 s | 11.03 us | 63.27 ms |!
| root | wait/io/file/sql/binlog_index | 80 | 1.16 s | 14.47 ms | 439.79 ms |!
| root | wait/io/file/csv/metadata | 32 | 577.39 ms | 18.04 ms | 245.23 ms |!
| root | wait/synch/mutex/sql/THD::LOCK_thd_data | 10280287 | 221.84 ms | 21.49 ns | 12.62 ms |!
| root | wait/synch/mutex/sql/THD::LOCK_query_plan | 5642052 | 168.50 ms | 29.78 ns | 11.89 ms |!
| root | wait/io/file/archive/data | 3529 | 19.09 ms | 5.41 us | 5.05 ms |!
MySQL SYS Procedures - P_S Analysis Helpers
Dump the data to create a graph dot file for a thread trace
ps_trace_thread()
Look in the statement history table trying to capture more info
ps_trace_statement_digest()
Reset all summary data
ps_truncate_all_tables()
Tracing Statement Digests
ps_trace_statement_digest() analyzes live traffic looking for certain

statement digest
Captures statistics on each matching statement it finds
Returns a report of the statistics
An overall summary
A break down for the longest running example
An EXPLAIN (if the statement is not truncated)
Tracing Statement Digests
ps_trace_statement_digest() parameters
in_digest The statement digest to analyze
in_runtime How long to run analysis for
in_interval How often to snapshot for data
in_start_fresh Whether to truncate p_s tables first
in_auto_enable Whether to auto enable required config
Tracing Statement Digests Example
mysql> call analyze_statement_digest('6134e9d6f25eb8e6cddf11f6938f202a', 60, 1, true, true);
+--------------------+
| SUMMARY STATISTICS |
+--------------------+
| SUMMARY STATISTICS |
+--------------------+
!
+------------+-----------+-----------+-----------+---------------+------------+------------+
| executions | exec_time | lock_time | rows_sent | rows_examined | tmp_tables | full_scans |
+------------+-----------+-----------+-----------+---------------+------------+------------+
| 360 | 1.41 s | 138.39 ms | 720 | 0 | 0 | 0 |
+------------+-----------+-----------+-----------+---------------+------------+------------+
!
+--------------------------------+-------+-----------+
| event_name | count | latency |
+--------------------------------+-------+-----------+
| stage/sql/Sending data | 14 | 1.10 s |
| stage/sql/update | 198 | 667.34 ms |
| stage/sql/removing tmp table | 9 | 240.48 ms |
...
!
+---------------------------+
| LONGEST RUNNING STATEMENT |
+---------------------------+
| LONGEST RUNNING STATEMENT |
+---------------------------+
!
+-----------+-----------+-----------+-----------+---------------+------------+-----------+
| thread_id | exec_time | lock_time | rows_sent | rows_examined | tmp_tables | full_scan |
+-----------+-----------+-----------+-----------+---------------+------------+-----------+
| 23277 | 77.64 ms | 97.00 us | 2 | 0 | 0 | 0 |
+-----------+-----------+-----------+-----------+---------------+------------+-----------+
!
+-------------------------------------------------------------------------------------------+
| sql_text |
+-------------------------------------------------------------------------------------------+
| /* mem dbpool.default */ insert into `mem__quan`.`normalized_statements_by_server_by_schema`
(firstSeen, lastSeen, normalized_statement_id, `schema`, server, id) values (1378983745000,
1378998600000, x'808ACEB88FC4B45BC47C4FE9D86C7F26', 'mem', '753c939e-1b99-11e3-b6d4-
bc761a1f4f2f', x'7CFE81F98B1E3FE0895D1AA2C39B326D') ON DUPLICATE KEY UPDATE ...
+-------------------------------------------------------------------------------------------+
...
!
+--------------------------------+----------+
| event_name | latency |
+--------------------------------+----------+
| stage/sql/init | 10.41 ms |
| stage/sql/checking permissions | 2.35 us |
| stage/sql/Opening tables | 10.32 us |
| stage/sql/init | 13.53 us |
| stage/sql/System lock | 3.66 us |
| stage/sql/update | 61.74 ms |
| stage/sql/end | 1.11 us |
| stage/sql/query end | 1.31 us |
| stage/sql/closing tables | 5.44 ms |
| stage/sql/freeing items | 23.31 us |
+--------------------------------+----------+
!
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | No tables used |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
!
...
Tracing Threads
ps_trace_thread() monitors a specific thread for a period

Captures as much information on the thread activity as possible
Returns a dot formatted file, that can graph the event hierarchy as
previously seen
http://en.wikipedia.org/wiki/DOT_(graph_description_language)
Tracing Threads
ps_trace_thread() parameters
in_thread_id The thread to analyze
in_outfile The file to dump the data to (INTO OUTFILE)
in_max_runtime How long to run analysis for
in_interval How often to snapshot for data
in_start_fresh Whether to truncate p_s tables first
in_auto_setup Whether to auto enable required config
in_debug Whether to also print debug info (source info)
Tracing Threads Example
mysql> call dump_thread_stack(27768, '/tmp/stack_27768.dot', 60, 0.1, true, true, true);
+------------------------------------------------+
| Info |
+------------------------------------------------+
| Data collection starting for THREAD_ID = 27768 |
+------------------------------------------------+
!
+---------------------------------------------+
| Info |
+---------------------------------------------+
| Stack trace written to /tmp/stack_27768.dot |
+---------------------------------------------+
!
+--------------------------------------------------------+
| Convert to PDF |
+--------------------------------------------------------+
| dot -Tpdf -o /tmp/stack_27768.pdf /tmp/stack_27768.dot |
+--------------------------------------------------------+
!
+--------------------------------------------------------+
| Convert to PNG |
+--------------------------------------------------------+
| dot -Tpng -o /tmp/stack_27768.png /tmp/stack_27768.dot |
+--------------------------------------------------------+
!!
1 row in set (60.90 sec)+
Tracing Threads Dot Format
cerberus:~ mark$ more Documents/statement_graphs/deletes.dot!
digraph events {!
graph [rankdir=LR, nodesep="0.10"];!
node [label="\N"];!
graph [bb="0,0,1860,17476"];!
1 [label="(0.20) mysys/THR_LOCK_threads\n", style=filled, color=lightskyblue, pos="209,18", width="3.19", height="0.50"];!
2 [label="(0.18) sql/THD::LOCK_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"];!
3 [label="(0.24) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height=0.50"];!
4 [label="(0.31) sql/LOGGER::LOCK_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22",
height="0.50"];!
5 [label="(0.08) sql/LOG::LOCK_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"];!
6 [label="(12.97) wait/io/file/sql/query_log - write 49 bytes\n/Users/mark/mysql/lp-mysql-trunk/mysql-test/var/mysqld.1/
mysqld.log\n", style=filled, color=red, shape=box, pos="209,235", width="5.78", height="0.56"];!
7 [label="(176.14) sql/select\nselect @@version_comment limit 1\nerrors: 0\nwarnings: 0\nlock time: 0.00\nrows affected:
0\nrows sent:\!
209,18", width="3.19", height="0.50"];mp disk tables: 0\nselect scan: 0\nselect full join: :!
2 [label="(0.18) sql/THD::LOCK_thd_data\n", style=filled, color=lightskyblue, pos="209,61", width="3.06", height="0.50"];!
3 [label="(0.24) sql/hash_filo::lock\n", style=filled, color=lightskyblue, pos="209,104", width="2.42", height="0.50"];!
4 [label="(0.31) sql/LOGGER::LOCK_logger - read_lock\n", style=filled, color=orchid, pos="209,147", width="4.22",
height="0.50"];!
5 [label="(0.08) sql/LOG::LOCK_log\n", style=filled, color=lightskyblue, pos="209,190", width="2.64", height="0.50"];!
!
Tracing Statement Output
MySQL SYS Procedures - P_S Setup Helpers
ps_setup_show_disabled() / ps_setup_show_enabled()
ps_setup_disable_thread() / ps_setup_enable_thread()
ps_setup_disable_background_threads() /
ps_setup_enable_background_threads()
ps_setup_disable_instrument() / ps_setup_enable_instrument()
ps_setup_disable_consumer() / ps_setup_enable_consumer()
ps_setup_save() / ps_setup_reload_saved()
ps_setup_reset_to_default()
ps_truncate_all_tables()
Things still to come
Status / Variable Instrumentation
Deprecate I_S.[GLOBAL|SESSION]_[STATUS|VARIABLES]
Replace with tables that show true state / aggregates
Per thread, user, host, account
See session statistics or configuration individually
Improve SHOW commands to only show appropriate variables
SHOW SESSION STATUS to only show session specific items
Backwards compatibility switch available
Less internal mutex contention for polling status
Further Replication Statistics
Redesigned lag monitoring

Consistently check lag across a replication chain
Further statistics on load of the replication threads
Queue lengths
Event counting
Busy times
Stage Progress Tracking
View the progress of longer stages of execution

Extends stage tables with estimate and completed columns
Bounded (estimated) or unbounded (only completed filled in)
Units depend on stage
stage/sql/copy to tmp table - 130 (rows) completed
Continued focus
On performance!
On filling missing instrumentation points!
On consolidating instrumentation within Performance schema!
The preceding is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract.
It is not a commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or functionality
described for Oracles products remains at the sole discretion of
Oracle.
!166
!167

Mysqltechdayparispsandsys 140523063324 Phpapp02

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mysqltechdayparispsandsys 140523063324 Phpapp02

Uploaded by

Copyright:

Available Formats

!

What is Performance Schema anyway?

!3 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Using 5.1 (yes, still)?

!4 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

!5 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Introduced in MySQL 5.5

!6 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Records latency of events that happen within the server

!7 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Instrument Event Class

!8 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

!9 Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Laid out the framework for instrumentation

Fixed lots of performance issues

Instrument Event Class

Setup Tables mysql> select table_name!

Raw Data Tables

Summary Tables | events_stages_summary_by_host_by_event_name

Summarise event information | events_statements_summary_by_program

over multiple dimensions

Useful for longer term monitoring

There are three distinct types of configuration

History / Summary Size

Max Classes / Instances -> from information_schema.global_variables!

implementations, i.e: | PERFORMANCE_SCHEMA_MAX_COND_INSTANCES

instances of those things, i.e: | PERFORMANCE_SCHEMA_MAX_SOCKET_INSTANCES

Max Classes / Instances

Monitor the %_lost status variables

opposite to see whether these

modify classes, only when loading | PERFORMANCE_SCHEMA_SOCKET_INSTANCES_LOST | 0

new plugins | PERFORMANCE_SCHEMA_STATEMENT_CLASSES_LOST | 0

Turn on/off individual instruments, !

or whether to just count and not

Use UPDATE to modify

mysql> select * from setup_instruments limit 5;!

setup_consumers | NAME | ENABLED |!

setup_objects (5.6) | OBJECT_TYPE | OBJECT_SCHEMA

Define exactly which database

objects to monitor mysql> insert into setup_objects!

Filters standard databases by !

mysql> select * from setup_objects;!

matches on most specific | TABLE

Define exactly which users to

As of 5.6, allows altering configuration at start up

It is easy to just enable everything, but there are overhead concerns

Some waits can include other waits

Some File IO events to watch for (if high in list):

High IO on InnoDB per-tablespace tables can

All event_% summaries are exposed with a number of dimensions

mysql> select user,!

mysql> select user, sum(count_star) as count,!

3 summary views have been added to 5.6 on top of Table IO

Less events, yet higher latency, is

Tables with full scans ->

This is a catch all row for rows that

are read without indexes

mysql> select object_schema,!

taking any actions! | mem__config

If interested in calculating percentages, copy the approach shown in

The new instrumentation in 5.6 gives many options

A new summary view introduced in 5.6

Normalization folds certain constructs of statements