Debug assertion "last_val > 0" when auto_inc_inc > 1 and ullong_max underflow

[george-lorch]

The following. simple test case illustrates the issue:

DROP DATABASE IF EXISTS test; CREATE DATABASE test; USE test;
CREATE TABLE t1(c1 BIGINT UNSIGNED AUTO_INCREMENT,KEY (c1)) engine=RocksDB;
INSERT INTO t1 VALUES(18446744073709551613);
SET @@session.auto_increment_increment=2;
INSERT INTO t1 VALUES();
INSERT INTO t1 VALUES();

The first insert after the "SET @@session.auto_increment_increment=2;" will get this expected error

ERROR 1467 (HY000): Failed to read auto-increment value from storage engine

The next insert hits the assertion with the following call stack:

mysqld: /home/glorch/dev/ps/facebook-mysql/storage/rocksdb/ha_rocksdb.cc:11474: virtual void myrocks::ha_rocksdb::get_auto_increment(ulonglong, ulonglong, ulonglong, ulonglong*, ulonglong*): Assertion `last_val > 0' failed.

Thread 11 "my-oneconnectio" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe87f8700 (LWP 2302)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff5915801 in __GI_abort () at abort.c:79
#2 0x00007ffff590539a in __assert_fail_base (fmt=0x7ffff5a8c7d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555557438fe6 "last_val > 0",
file=file@entry=0x555557433348 "/home/glorch/dev/ps/facebook-mysql/storage/rocksdb/ha_rocksdb.cc", line=line@entry=11474,
function=function@entry=0x55555743fa00 <myrocks::ha_rocksdb::get_auto_increment(unsigned long long, unsigned long long, unsigned long long, unsigned long long*, unsigned long long*)::PRETTY_FUNCTION> "virtual void myrocks::ha_rocksdb::get_auto_increment(ulonglong, ulonglong, ulonglong, ulonglong*, ulonglong*)") at assert.c:92
#3 0x00007ffff5905412 in __GI___assert_fail (assertion=0x555557438fe6 "last_val > 0", file=0x555557433348 "/home/glorch/dev/ps/facebook-mysql/storage/rocksdb/ha_rocksdb.cc", line=11474,
function=0x55555743fa00 <myrocks::ha_rocksdb::get_auto_increment(unsigned long long, unsigned long long, unsigned long long, unsigned long long*, unsigned long long*)::PRETTY_FUNCTION> "virtual void myrocks::ha_rocksdb::get_auto_increment(ulonglong, ulonglong, ulonglong, ulonglong*, ulonglong*)") at assert.c:101
#4 0x0000555556c02f89 in myrocks::ha_rocksdb::get_auto_increment (this=0x7fffbc00b710, off=1, inc=2, nb_desired_values=1, first_value=0x7fffe87f4fe0, nb_reserved_values=0x7fffe87f4fe8) at /home/glorch/dev/ps/facebook-mysql/storage/rocksdb/ha_rocksdb.cc:11474
#5 0x000055555629b713 in handler::update_auto_increment (this=0x7fffbc00b710) at /home/glorch/dev/ps/facebook-mysql/sql/handler.cc:3718
#6 0x0000555556bfa612 in myrocks::ha_rocksdb::write_row (this=0x7fffbc00b710, buf=0x7fffbc016100 "\377") at /home/glorch/dev/ps/facebook-mysql/storage/rocksdb/ha_rocksdb.cc:9107
#7 0x00005555562a4fb6 in handler::ha_write_row (this=0x7fffbc00b710, buf=0x7fffbc016100 "\377") at /home/glorch/dev/ps/facebook-mysql/sql/handler.cc:7926
#8 0x0000555556495324 in write_record (thd=0x555558468480, table=0x7fffbc00adb0, info=0x7fffe87f53b0, update=0x7fffe87f5430) at /home/glorch/dev/ps/facebook-mysql/sql/sql_insert.cc:1889
#9 0x0000555556492f81 in mysql_insert (thd=0x555558468480, table_list=0x7fffbc007a10, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /home/glorch/dev/ps/facebook-mysql/sql/sql_insert.cc:1030
#10 0x00005555564be4a3 in mysql_execute_command (thd=0x555558468480, statement_start_time=0x7fffe87f6718, post_parse=0x7fffe87f6878) at /home/glorch/dev/ps/facebook-mysql/sql/sql_parse.cc:4575
#11 0x00005555564c78b4 in mysql_parse (thd=0x555558468480, rawbuf=0x7fffbc007930 "INSERT INTO t1 VALUES()", length=23, parser_state=0x7fffe87f7170, last_timer=0x7fffe87f6878, async_commit=0x7fffe87f6843 "")
at /home/glorch/dev/ps/facebook-mysql/sql/sql_parse.cc:7896
#12 0x00005555564b742e in dispatch_command (command=COM_QUERY, thd=0x555558468480, packet=0x55555851d3d1 "INSERT INTO t1 VALUES()", packet_length=23) at /home/glorch/dev/ps/facebook-mysql/sql/sql_parse.cc:1919
#13 0x00005555564b52b8 in do_command (thd=0x555558468480) at /home/glorch/dev/ps/facebook-mysql/sql/sql_parse.cc:1193
#14 0x000055555646f598 in do_handle_one_connection (thd_arg=0x555558468480) at /home/glorch/dev/ps/facebook-mysql/sql/sql_connect.cc:1148
#15 0x000055555646ef8d in handle_one_connection (arg=0x555558468480) at /home/glorch/dev/ps/facebook-mysql/sql/sql_connect.cc:969
#16 0x0000555556bd600a in pfs_spawn_thread (arg=0x5555584136f0) at /home/glorch/dev/ps/facebook-mysql/storage/perfschema/pfs.cc:1860
#17 0x00007ffff660c6db in start_thread (arg=0x7fffe87f8700) at pthread_create.c:463
#18 0x00007ffff59f688f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[george-lorch]

email discussions with @lth and some additional testing tells us that the logic here https://github.com/facebook/mysql-5.6/blob/fb-mysql-5.6.35/storage/rocksdb/ha_rocksdb.cc#L11540-LL11541 is slightly off. We get into a situation where new_val is equal to ullong_max, thus new_val + 1 rolls us over to 0, which is what gets stored as the new auto_inc value for the table and then eventually leads to the assertion on the next time through get_auto_increment.

A simple change to this logic solves the problem and prevents us from either underflowing or from giving out any more values. We change:

while (!auto_incr.compare_exchange_weak(last_val, std::min(new_val + 1, max_val)));

to

while (!auto_incr.compare_exchange_weak(last_val, std::min(new_val, max_val - 1) + 1));