crash in innobase_update_index_stats

[sbester]

On trunk/5.6.12 I keep hitting this crash when running random queries involving INDEX_STATISTICS.

(gdb) bt
#0  in pthread_kill () from /lib64/libpthread.so.0
#1  in handle_fatal_signal (sig=11) at ./trunk/sql/signal_handler.cc:248
#2  <signal handler called>
#3  in innobase_update_index_stats (table_stats=0x7f53740766a0) at ./trunk/storage/innobase/handler/ha_innodb.cc:4051
#4  in get_index_stats_handlerton at ./trunk/sql/handler.cc:858
#5  in plugin_foreach_with_mask at ./trunk/sql/sql_plugin.cc:2094
#6  in ha_get_index_stats at ./trunk/sql/handler.cc:866
#7  in fill_index_stats  at ./trunk/sql/table_stats.cc:681
#8  in do_fill_table at ./trunk/sql/sql_show.cc:7195
#9  in get_schema_tables_result at ./trunk/sql/sql_show.cc:7296
#10 in JOIN::prepare_result at ./trunk/sql/sql_select.cc:844
#11 in JOIN::exec at ./trunk/sql/sql_executor.cc:116
#12 in mysql_execute_select  at ./trunk/sql/sql_select.cc:1121
#13 in mysql_select at ./trunk/sql/sql_select.cc:1242
#14 in handle_select at ./trunk/sql/sql_select.cc:125
#15 in execute_sqlcom_select at ./trunk/sql/sql_parse.cc:5534
#16 in mysql_execute_command at ./trunk/sql/sql_parse.cc:2969
#17 in mysql_parse at ./trunk/sql/sql_parse.cc:6694
#18 in dispatch_command at ./trunk/sql/sql_parse.cc:1402
#19 in do_command at ./trunk/sql/sql_parse.cc:1047
#20 in do_handle_one_connection  at ./trunk/sql/sql_connect.cc:1001
#21 in handle_one_connection at ./trunk/sql/sql_connect.cc:917
#22 in pfs_spawn_thread at ./trunk/storage/perfschema/pfs.cc:1855
#23 in start_thread from /lib64/libpthread.so.0
#24 in clone from /lib64/libc.so.6
(gdb) p mysql_parse::thd->query_string
$1 = {
  string = {
    str = 0x7f5374006c40 "select  \t ROUTINE_BODY from\n\t`information_schema`.`INNODB_SYS_FOREIGN` as `INNODB_SYS_FOREIGN` \n\t right outer join `information_schema`.`INDEX_STATISTICS` as `INDEX_STATISTICS` \non 2\n\n \t\n\t natural left outer join `information_schema`.`ROUTINES` as `ROUTINES` \n \t\n\t inner join `test`.`t0004` as `t0004`  \non ( 1  )\n  \n \ngroup by \n\tROUTINES.DEFINER desc",
    length = 351
  },
  cs = 0x1309180 <my_charset_latin1>
}
(gdb)

I have many core files but no exact testcase yet. Rerunning queries didn't crash.
More info later.

[sbester]

Version: '5.6.12'  socket: 'sock'  port: 3306  MySQL Community Server (GPL)
==27811== Thread 24:
==27811== Conditional jump or move depends on uninitialised value(s)
==27811==    at 0x5E87C1: fill_index_stats(THD*, TABLE_LIST*, Item*) (table_stats.cc:692)
==27811==    by 0x757581: get_schema_tables_result(JOIN*, enum_schema_table_state) (sql_show.cc:7195)
==27811==    by 0x73A884: JOIN::prepare_result(List<Item>**) (sql_select.cc:844)
==27811==    by 0x6F8B5B: JOIN::exec() (sql_executor.cc:116)
==27811==    by 0x73EB14: mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, SQL_I_List<st_order>*, SQL_I_List<st_order>*, Item*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:1121)
==27811==    by 0x73F403: handle_select(THD*, select_result*, unsigned long) (sql_select.cc:125)
==27811==    by 0x5A7DEF: execute_sqlcom_select(THD*, TABLE_LIST*, unsigned long long*) [clone .isra.149] (sql_parse.cc:5534)
==27811==    by 0x71C349: mysql_execute_command(THD*, unsigned long long*, unsigned long long*) (sql_parse.cc:2969)
==27811==    by 0x72081E: mysql_parse(THD*, char*, unsigned int, Parser_state*, unsigned long long*, char*) (sql_parse.cc:6694)
==27811==    by 0x721E8A: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1402)
==27811==    by 0x6F0181: do_handle_one_connection(THD*) (sql_connect.cc:1001)
==27811==    by 0x6F022F: handle_one_connection (sql_connect.cc:917)
==27811==    by 0x3537607D13: start_thread (pthread_create.c:309)
==27811==    by 0x35372F168C: clone (clone.S:115)

0x00000000005e87c1 in fill_index_stats (thd=thd@entry=0x13adf890, tables=tables@entry=0x13aef538, cond=<optimized out>) at /home/sbester/build/facebook/mysql-5.6/trunk/sql/table_stats.cc:692
692               index_stats->rows_requested == 0 &&
(gdb) p *index_stats
$1 = {
  name = "PRIMARY", '\000' <repeats 185 times>,
  rows_inserted = 0,
  rows_updated = 0,
  rows_deleted = 0,
  rows_read = 0,
  rows_requested = 0,
  rows_index_first = 0,
  rows_index_next = 0,
  io_perf_read = {
    bytes = 0,
    requests = 0,
    svc_time = 0,
    svc_time_max = 0,
    wait_time = 0,
    wait_time_max = 0,
    slow_ios = 0
  },
  n_pages = 0,
  n_pages_freed = 0,
  n_btr_compress = 0,
  n_btr_compress_failure = 0,
  n_page_split = 0
}
(gdb) p mysql_parse::thd->query_string
$2 = {
  string = {
    str = 0x13ed99a0 "select  IS_HASHED from `information_schema`.`INNODB_SYS_TABLESPACES` as `INNODB_SYS_TABLESPACES` cross join `information_schema`.`PROCESSLIST` as `PROCESSLIST`  on (INNODB_SYS_TABLESPACES.FLAG && PROCESSLIST.DB ) inner join `information_schema`.`INNODB_BUFFER_PAGE_LRU` as `INNODB_BUFFER_PAGE_LRU`  on (INNODB_SYS_TABLESPACES.FLAG is not null ) inner join `information_schema`.`INDEX_STATISTICS` as `INDEX_STATISTICS` on (INNODB_SYS_TABLESPACES.NAME <=> INDEX_STATISTICS.IO_READ_SLOW_IOS )",
    length = 489
  },
  cs = 0x138a1e0
}

[sbester]

Testcase.....

drop procedure if exists p1;
delimiter $
create procedure p1()
begin
  declare continue handler for sqlexception begin end;
  repeat
    if rand()>0.5 then drop table if exists t1; end if;
    if rand()>0.5 then create table if not exists `t1`(`a` int primary key,b char(1), key(b))engine=innodb; end if;
    if rand()>0.5 then select  IS_HASHED from `information_schema`.`INNODB_SYS_TABLESPACES` as `INNODB_SYS_TABLESPACES` cross join `information_schema`.`PROCESSLIST` as `PROCESSLIST`  on (INNODB_SYS_TABLESPACES.FLAG && PROCESSLIST.DB ) inner join `information_schema`.`INNODB_BUFFER_PAGE_LRU` as `INNODB_BUFFER_PAGE_LRU`  on (INNODB_SYS_TABLESPACES.FLAG is not null ) inner join `information_schema`.`INDEX_STATISTICS` as `INDEX_STATISTICS` on (INNODB_SYS_TABLESPACES.NAME <=> INDEX_STATISTICS.IO_READ_SLOW_IOS ) into @a; end if;
    if rand()>0.5 then select  ROUTINE_BODY from `information_schema`.`INNODB_SYS_FOREIGN` as `INNODB_SYS_FOREIGN`  right outer join `information_schema`.`INDEX_STATISTICS` as `INDEX_STATISTICS` on 2  natural left outer join `information_schema`.`ROUTINES` as `ROUTINES`  inner join t1  on ( 1  )group by ROUTINES.DEFINER desc into @a; end if;
    if rand()>0.5 then flush tables; end if;
    if rand()>0.5 then rename table t1 to t2; end if;
    if rand()>0.5 then rename table t2 to t1; end if;
    if rand()>0.5 then alter table t1 add key g(b); end if;
    if rand()>0.5 then alter table t1 add key g(a); end if;
    if rand()>0.5 then alter table t1 drop key g; end if;
    if rand()>0.5 then select * from `information_schema`.`INDEX_STATISTICS`; end if;
  until 1=2 end repeat;
 end $

 delimiter ;
  call p1();

[steaphan-fb-com]

Thanks for the detailed info.

Sorry for the crash bug - we've actually already fixed this in our current dev branch, which is based on 5.6.16. We just haven't had a chance to push an updated version here yet. We should be pushing a completely up-to-date branch up here some time in the next few weeks.

Once we do, we'd certainly love any more feedback you have.

Again, thanks!

[steaphan-fb-com]

Now that WebScaleSQL is public, we also pushed our 5.6.16 branch (which is based on WebScaleSQL now). That includes a fix for this bug.