Database admission control

Summary:
Add two global sys vars max_running_queries and max_waiting_queries.
They control maximum number of running queries on a database and maximum
waiting queries when max_running_queries limit is crossed on that database.
A value of 0 implies no limits are applied for the queries. If max_waiting_queries
limit is crossed, then new queries will simply fail.

For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'.
Only dagtabase set at the session level is considered in these checks. Admission checks
are by-passed in the following cases
1. Query is run by super user.
2. Query is run by replication threads.
3. No database is set for the session.
4. max_running_queries is 0.

Test Plan:
Added mtr tests. Ran performance tests.

Performance testing

1. start MySQL using mtr
> mtr --start --mysqld=--default_storage_engine=InnoDB

2. Connect to mysql using
> mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root

3. Set innodb_flush_log_at_trx_commit=2.
mysql> set @@global.innodb_flush_log_at_trx_commit=2

4. Check rows inserted using
> mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted"

5. Run mysqlslap using
> mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock

== Innodb_rows_inserted with concurrency=100 ==
38106 (max_running_queries=0, max_waiting_queries=0)
38733 (max_running_queries=1000, max_waiting_queries=0)
38365 (max_running_queries=100, max_waiting_queries=0)
40308 (max_running_queries=70, max_waiting_queries=0)
43790 (max_running_queries=50, max_waiting_queries=0)
47321 (max_running_queries=20, max_waiting_queries=0)
49473 (max_running_queries=10, max_waiting_queries=0)
54851 (max_running_queries=5, max_waiting_queries=0)

== Innodb_rows_inserted with concurrency=100 ==
26516 (max_running_queries=0, max_waiting_queries=0)
44494 (max_running_queries=10, max_waiting_queries=0)
51193 (max_running_queries=5, max_waiting_queries=0)

The above numbers show the improvement in quality of service.

== Innodb_rows_inserted with concurrency=20 ==
47228 (max_running_queries=0, max_waiting_queries=0)
47107 (max_running_queries=100, max_waiting_queries=0)
47082 (max_running_queries=20, max_waiting_queries=0)
47515 (max_running_queries=10, max_waiting_queries=0)

* There isn't much difference under low concurrency workloads.

* There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value.

Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar

Subscribers: webscalesql-eng, ebergen, domas

Differential Revision: https://reviews.facebook.net/D48405
Differential Revision: https://reviews.facebook.net/D52155
Differential Revision: https://reviews.facebook.net/D52185

mysql-test/r/admission_control.result

@@ -0,0 +1,40 @@
+create database test_db;
+create user test_user@localhost;
+grant all on test_db to test_user@localhost;
+grant all on test to test_user@localhost;
+use test_db;
+set @start_max_running_queries= @@global.max_running_queries;
+set @start_max_waiting_queries= @@global.max_waiting_queries;
+set @@global.max_running_queries=10;
+set @@global.max_waiting_queries=5;
+create table t1(a int) engine=InnoDB;
+lock table t1 write;
+Threads waiting for admission will have appropriate state set in processlist.
+Super user is exempted from admission control checks.
+select * from t1;
+a
+Maximum waiting queries reached. So this would hit an error.
+select * from t1;
+ERROR HY000: Maximum waiting queries 5 reached for database `test_db`
+Filled up queues on one db doesn't affect queries on other db.
+use test;
+set @@global.max_waiting_queries=6;
+Kill a thread that is waiting for admission.
+select count(*) from t1;
+kill ID;
+use test_db;
+unlock tables;
+Verify the waiting queries received wakeup signal.
+select count(*) from t1;
+count(*)
+15
+Run parallel load and drop the database.
+set @@global.max_waiting_queries=0;
+Cleanup.
+Verify there are no waiting threads.
+select count(*) from information_schema.processlist where state='waiting for admission';
+count(*)
+0
+set @@global.max_running_queries=@start_max_running_queries;
+set @@global.max_waiting_queries=@start_max_waiting_queries;
+drop user test_user@localhost;

mysql-test/r/admission_control_stress.result

@@ -0,0 +1,5 @@
+Test setup.
+Generate load. Toggle max_running_queries and randomly kill a query.
+Cleanup
+set global max_connections = @start_max_connections;
+set global max_running_queries = @start_max_running_queries;

mysql-test/r/mysqld--help-notwin-profiling.result

@@ -515,6 +515,9 @@ The following options may be given as the first argument:
  If non-zero: relay log will be rotated automatically when
  the size exceeds this value; if zero: when the size
  exceeds max_binlog_size
+ --max-running-queries=# 
+ The maximum number of running queries allowed for a
+ database. If this value is 0, no such limits are applied.
  --max-seeks-for-key=# 
  Limit assumed max number of seeks when looking up rows
  based on a key
@@ -528,6 +531,9 @@ The following options may be given as the first argument:
  --max-user-connections=# 
  The maximum number of active connections for a single
  user (0 = no limit)
+ --max-waiting-queries=# 
+ The maximum number of waiting queries allowed for a
+ database.If this value is 0, no such limits are applied.
  --max-write-lock-count=# 
  After this many write locks, allow some read locks to run
  in between
@@ -1564,11 +1570,13 @@ max-join-size 18446744073709551615
 max-length-for-sort-data 1024
 max-prepared-stmt-count 16382
 max-relay-log-size 0
+max-running-queries 0
 max-seeks-for-key 18446744073709551615
 max-sort-length 1024
 max-sp-recursion-depth 0
 max-tmp-tables 32
 max-user-connections 0
+max-waiting-queries 0
 max-write-lock-count 18446744073709551615
 memlock FALSE
 metadata-locks-cache-size 1024

mysql-test/r/mysqld--help-notwin.result

@@ -515,6 +515,9 @@ The following options may be given as the first argument:
  If non-zero: relay log will be rotated automatically when
  the size exceeds this value; if zero: when the size
  exceeds max_binlog_size
+ --max-running-queries=# 
+ The maximum number of running queries allowed for a
+ database. If this value is 0, no such limits are applied.
  --max-seeks-for-key=# 
  Limit assumed max number of seeks when looking up rows
  based on a key
@@ -528,6 +531,9 @@ The following options may be given as the first argument:
  --max-user-connections=# 
  The maximum number of active connections for a single
  user (0 = no limit)
+ --max-waiting-queries=# 
+ The maximum number of waiting queries allowed for a
+ database.If this value is 0, no such limits are applied.
  --max-write-lock-count=# 
  After this many write locks, allow some read locks to run
  in between
@@ -1562,11 +1568,13 @@ max-join-size 18446744073709551615
 max-length-for-sort-data 1024
 max-prepared-stmt-count 16382
 max-relay-log-size 0
+max-running-queries 0
 max-seeks-for-key 18446744073709551615
 max-sort-length 1024
 max-sp-recursion-depth 0
 max-tmp-tables 32
 max-user-connections 0
+max-waiting-queries 0
 max-write-lock-count 18446744073709551615
 memlock FALSE
 metadata-locks-cache-size 1024

mysql-test/suite/perfschema/r/dml_setup_instruments.result

@@ -19,6 +19,7 @@ where name like 'Wait/Synch/Rwlock/sql/%'
   and name not in ('wait/synch/rwlock/sql/CRYPTO_dynlock_value::lock')
 order by name limit 10;
 NAME	ENABLED	TIMED
+wait/synch/rwlock/sql/AC::rwlock	YES	YES
 wait/synch/rwlock/sql/Binlog_relay_IO_delegate::lock	YES	YES
 wait/synch/rwlock/sql/Binlog_storage_delegate::lock	YES	YES
 wait/synch/rwlock/sql/Binlog_transmit_delegate::lock	YES	YES
@@ -28,7 +29,6 @@ wait/synch/rwlock/sql/LOCK_grant	YES	YES
 wait/synch/rwlock/sql/LOCK_system_variables_hash	YES	YES
 wait/synch/rwlock/sql/LOCK_sys_init_connect	YES	YES
 wait/synch/rwlock/sql/LOCK_sys_init_slave	YES	YES
-wait/synch/rwlock/sql/LOCK_use_ssl	YES	YES
 select * from performance_schema.setup_instruments
 where name like 'Wait/Synch/Cond/sql/%'
   and name not in (

mysql-test/suite/sys_vars/r/max_running_queries_basic.result

@@ -0,0 +1,130 @@
+SET @start_value = @@global.max_running_queries;
+SELECT @start_value;
+@start_value
+0
+'#--------------------FN_DYNVARS_074_01------------------------#'
+SET @@global.max_running_queries = 5000;
+SET @@global.max_running_queries = DEFAULT;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+0
+'#---------------------FN_DYNVARS_074_02-------------------------#'
+SET @@global.max_running_queries = @start_value;
+SELECT @@global.max_running_queries = 151;
+@@global.max_running_queries = 151
+0
+'#--------------------FN_DYNVARS_074_03------------------------#'
+SET @@global.max_running_queries = 100000;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+100000
+SET @@global.max_running_queries = 99999;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+99999
+SET @@global.max_running_queries = 65536;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+65536
+SET @@global.max_running_queries = 1;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+1
+SET @@global.max_running_queries = 2;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+2
+'#--------------------FN_DYNVARS_074_04-------------------------#'
+SET @@global.max_running_queries = -1;
+Warnings:
+Warning	1292	Truncated incorrect max_running_queries value: '-1'
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+0
+SET @@global.max_running_queries = 100000000000;
+Warnings:
+Warning	1292	Truncated incorrect max_running_queries value: '100000000000'
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+100000
+SET @@global.max_running_queries = 10000.01;
+ERROR 42000: Incorrect argument type to variable 'max_running_queries'
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+100000
+SET @@global.max_running_queries = -1024;
+Warnings:
+Warning	1292	Truncated incorrect max_running_queries value: '-1024'
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+0
+SET @@global.max_running_queries = 0;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+0
+SET @@global.max_running_queries = 100001;
+Warnings:
+Warning	1292	Truncated incorrect max_running_queries value: '100001'
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+100000
+SET @@global.max_running_queries = ON;
+ERROR 42000: Incorrect argument type to variable 'max_running_queries'
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+100000
+SET @@global.max_running_queries = 'test';
+ERROR 42000: Incorrect argument type to variable 'max_running_queries'
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+100000
+'#-------------------FN_DYNVARS_074_05----------------------------#'
+SET @@session.max_running_queries = 4096;
+ERROR HY000: Variable 'max_running_queries' is a GLOBAL variable and should be set with SET GLOBAL
+SELECT @@session.max_running_queries;
+ERROR HY000: Variable 'max_running_queries' is a GLOBAL variable
+'#----------------------FN_DYNVARS_074_06------------------------#'
+SELECT @@global.max_running_queries = VARIABLE_VALUE 
+FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES 
+WHERE VARIABLE_NAME='max_running_queries';
+@@global.max_running_queries = VARIABLE_VALUE
+1
+SELECT @@max_running_queries = VARIABLE_VALUE 
+FROM INFORMATION_SCHEMA.SESSION_VARIABLES 
+WHERE VARIABLE_NAME='max_running_queries';
+@@max_running_queries = VARIABLE_VALUE
+1
+'#---------------------FN_DYNVARS_074_07----------------------#'
+SET @@global.max_running_queries = TRUE;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+1
+SET @@global.max_running_queries = FALSE;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+0
+'#---------------------FN_DYNVARS_074_08----------------------#'
+SET @@global.max_running_queries = 5000;
+SELECT @@max_running_queries = @@global.max_running_queries;
+@@max_running_queries = @@global.max_running_queries
+1
+'#---------------------FN_DYNVARS_074_09----------------------#'
+SET max_running_queries = 6000;
+ERROR HY000: Variable 'max_running_queries' is a GLOBAL variable and should be set with SET GLOBAL
+SELECT @@max_running_queries;
+@@max_running_queries
+5000
+SET local.max_running_queries = 7000;
+ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'max_running_queries = 7000' at line 1
+SELECT local.max_running_queries;
+ERROR 42S02: Unknown table 'local' in field list
+SET global.max_running_queries = 8000;
+ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'max_running_queries = 8000' at line 1
+SELECT global.max_running_queries;
+ERROR 42S02: Unknown table 'global' in field list
+SELECT max_running_queries = @@session.max_running_queries;
+ERROR 42S22: Unknown column 'max_running_queries' in 'field list'
+SET @@global.max_running_queries = @start_value;
+SELECT @@global.max_running_queries;
+@@global.max_running_queries
+0

mysql-test/suite/sys_vars/r/max_waiting_queries_basic.result

@@ -0,0 +1,130 @@
+SET @start_value = @@global.max_waiting_queries;
+SELECT @start_value;
+@start_value
+0
+'#--------------------FN_DYNVARS_074_01------------------------#'
+SET @@global.max_waiting_queries = 5000;
+SET @@global.max_waiting_queries = DEFAULT;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+0
+'#---------------------FN_DYNVARS_074_02-------------------------#'
+SET @@global.max_waiting_queries = @start_value;
+SELECT @@global.max_waiting_queries = 151;
+@@global.max_waiting_queries = 151
+0
+'#--------------------FN_DYNVARS_074_03------------------------#'
+SET @@global.max_waiting_queries = 100000;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+100000
+SET @@global.max_waiting_queries = 99999;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+99999
+SET @@global.max_waiting_queries = 65536;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+65536
+SET @@global.max_waiting_queries = 1;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+1
+SET @@global.max_waiting_queries = 2;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+2
+'#--------------------FN_DYNVARS_074_04-------------------------#'
+SET @@global.max_waiting_queries = -1;
+Warnings:
+Warning	1292	Truncated incorrect max_waiting_queries value: '-1'
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+0
+SET @@global.max_waiting_queries = 100000000000;
+Warnings:
+Warning	1292	Truncated incorrect max_waiting_queries value: '100000000000'
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+100000
+SET @@global.max_waiting_queries = 10000.01;
+ERROR 42000: Incorrect argument type to variable 'max_waiting_queries'
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+100000
+SET @@global.max_waiting_queries = -1024;
+Warnings:
+Warning	1292	Truncated incorrect max_waiting_queries value: '-1024'
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+0
+SET @@global.max_waiting_queries = 0;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+0
+SET @@global.max_waiting_queries = 100001;
+Warnings:
+Warning	1292	Truncated incorrect max_waiting_queries value: '100001'
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+100000
+SET @@global.max_waiting_queries = ON;
+ERROR 42000: Incorrect argument type to variable 'max_waiting_queries'
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+100000
+SET @@global.max_waiting_queries = 'test';
+ERROR 42000: Incorrect argument type to variable 'max_waiting_queries'
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+100000
+'#-------------------FN_DYNVARS_074_05----------------------------#'
+SET @@session.max_waiting_queries = 4096;
+ERROR HY000: Variable 'max_waiting_queries' is a GLOBAL variable and should be set with SET GLOBAL
+SELECT @@session.max_waiting_queries;
+ERROR HY000: Variable 'max_waiting_queries' is a GLOBAL variable
+'#----------------------FN_DYNVARS_074_06------------------------#'
+SELECT @@global.max_waiting_queries = VARIABLE_VALUE 
+FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES 
+WHERE VARIABLE_NAME='max_waiting_queries';
+@@global.max_waiting_queries = VARIABLE_VALUE
+1
+SELECT @@max_waiting_queries = VARIABLE_VALUE 
+FROM INFORMATION_SCHEMA.SESSION_VARIABLES 
+WHERE VARIABLE_NAME='max_waiting_queries';
+@@max_waiting_queries = VARIABLE_VALUE
+1
+'#---------------------FN_DYNVARS_074_07----------------------#'
+SET @@global.max_waiting_queries = TRUE;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+1
+SET @@global.max_waiting_queries = FALSE;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+0
+'#---------------------FN_DYNVARS_074_08----------------------#'
+SET @@global.max_waiting_queries = 5000;
+SELECT @@max_waiting_queries = @@global.max_waiting_queries;
+@@max_waiting_queries = @@global.max_waiting_queries
+1
+'#---------------------FN_DYNVARS_074_09----------------------#'
+SET max_waiting_queries = 6000;
+ERROR HY000: Variable 'max_waiting_queries' is a GLOBAL variable and should be set with SET GLOBAL
+SELECT @@max_waiting_queries;
+@@max_waiting_queries
+5000
+SET local.max_waiting_queries = 7000;
+ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'max_waiting_queries = 7000' at line 1
+SELECT local.max_waiting_queries;
+ERROR 42S02: Unknown table 'local' in field list
+SET global.max_waiting_queries = 8000;
+ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'max_waiting_queries = 8000' at line 1
+SELECT global.max_waiting_queries;
+ERROR 42S02: Unknown table 'global' in field list
+SELECT max_waiting_queries = @@session.max_waiting_queries;
+ERROR 42S22: Unknown column 'max_waiting_queries' in 'field list'
+SET @@global.max_waiting_queries = @start_value;
+SELECT @@global.max_waiting_queries;
+@@global.max_waiting_queries
+0