Defragmentation for partitioned InnoDB table #8

SunguckLee · 2014-11-18T02:55:00Z

Hi Facebook.
Really thanks for your awesome features.

I really interested in defragmentation of InnoDB table.
But for the huge table, we can't split defragmentation job.
Yes for the index level But if PRIMARY KEY itself is huge, we have to run defragmentation whole day.

So, I think defragmentation job can be splitted by partition level if the table is partitioned.
But current facebook version of mysql doesn't support for the partitioned table.

I think it can be implement easily (Just think) and I want to try.
Before implementation, You can also implement it easily (you are expert than me at least ^^), but you didn't.
I thought there's enough reason you did't implement it. Please share the reason to me ?

Really Thanks.

rongrong · 2014-11-18T10:59:43Z

Hi Sunguck,

Thanks for your feedback and we are glad that you are interested in the work.

There are a few reasons:

The feature is developed targeting our largest tier, which does not have many partitioned table, so partitioned table is not a problem for us, as a result we never thought about that. (OK, this is the real reason.)
The speed of defragmentation itself is not as important as its performance impact on production work load. Instead of optimizing on how to finish up defragmenting everything fast, we tried to make defragmentation not affecting normal work load by rate limiting it’s speed, and control how many index are being defragmented in parallel. It might take a even longer time to defragment, but if nobody will notice anything while it’s running it doesn’t matter that much.

How big is your table (or primary index)?

Thanks!
Rongrong

SunguckLee · 2014-11-18T11:36:51Z

Hi rongrong.
Really thanks for your detailed explanation.

The biggest table which I am targeting is about 400GB, and this table is partitioned (hash partition).
But now we can't convert it to non-partitioned table for defragmentation. ^^
So I thought that defragmentation features which can be applied to partitioned-table.

Anyway, I will try it.
I will ask you again if there's problem.

Have a nice day.
Matt.

steaphan-fb-com · 2014-11-18T17:15:44Z

Cool. Let us know how it goes with another comment here.

Or, if you have any more problems, please open a new issue with the details.

SunguckLee · 2014-11-24T06:27:40Z

Hi Rongrong and Speaphan.

I added some code for supporting defragmentation of partitioned innodb table.
I want to the code change will be merged to facebook version of mysql.
We also want to use facebook version mysql on our service if requirement is meet ^^.
(There's some ALTER TABLE syntax change, but if you mind that you can ignore it)

Regards,
Matt.

Summary: I saw this intermittent ASAN failure ==662590== ERROR: AddressSanitizer: SEGV on unknown address 0x000000000098 (pc 0x00000112fc64 sp 0x7f0324c71710 bp 0x7f0324c71760 T21) AddressSanitizer can not provide additional info. #0 0x112fc63 in my_pthread_fastmutex_lock /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/mysys/thr_mutex.c:479 #1 0x18fb4b0 in _ZN23Dropped_indices_manager14remove_indicesERKSt13unordered_setIjSt4hashIjESt8equal_toIjESaIjEEP12Dict_manager /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/include/mysql/psi/mysql_thread.h:688 #2 0x18b031a in _Z17drop_index_threadPv /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/storage/rocksdb/ha_rocksdb.cc:4927 #3 0x7f0337d611e8 in _ZN6__asan10AsanThread11ThreadStartEv ??:0 #4 0x7f03376dbfa7 in start_thread ??:0 #5 0x7f0335a5f5bc in __clone ??:0 Thread T21 created by T0 here: #0 0x7f0337d5121b in pthread_create ??:0 #1 0x18aec6a in _ZL17rocksdb_init_funcPv /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/storage/rocksdb/ha_rocksdb.cc:1765 #2 0x60a039 in _Z24ha_initialize_handlertonP13st_plugin_int /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/handler.cc:673 #3 0xab04e0 in _ZL17plugin_initializeP13st_plugin_int /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/sql_plugin.cc:1137 #4 0xac8fe3 in _Z11plugin_initPiPPci /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/sql_plugin.cc:1431 #5 0x5ef74c in _ZL22init_server_componentsv .cc:5482 #6 0x5f0cce in _Z11mysqld_mainiPPc .cc:6254 #7 0x7f033597defe in __libc_start_main ??:0 #8 0x5d86a8 in _start /home/engshare/third-party2/glibc/2.17/src/glibc-2.17/csu/../sysdeps/x86_64/start.S:123 ==662590== ABORTING It looks like the Dropped_indices_manager mutex is accessed by the thread after the mutex was cleaned up. I'm moving the cleanup of the manager to the thread in hopes that this will fix the failure. Test Plan: ran rocksdb suite, nothing failed Reviewers: hermanlee4 Reviewed By: hermanlee4 Differential Revision: https://reviews.facebook.net/D42909

Summary: Fix yet another memory leak reported by ASAN, in perfschema dynamic array of instrument config. LeakSanitizer reported: ==1556117==ERROR: LeakSanitizer: detected memory leaks Direct leak of 21 byte(s) in 1 object(s) allocated from: #0 0x7fd5cee05151 in __interceptor_calloc (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libasan.so.2+0x9a151) #1 0x120ea19 in my_malloc /home/tianx/mysql/5.6/mysys/my_malloc.c:38 #2 0x1891606 in add_pfs_instr_to_array(char const*, char const*) /home/tianx/mysql/5.6/storage/perfschema/pfs_server.cc:251 #3 0x76bd26 in mysqld_get_one_option /home/tianx/mysql/5.6/sql/mysqld.cc:11033 #4 0x1242157 in my_handle_options /home/tianx/mysql/5.6/mysys_ssl/my_getopt.cc:659 #5 0x77d88e in handle_early_options(char) /home/tianx/mysql/5.6/sql/mysqld.cc:8750 #6 0x78d003 in mysqld_main(int, char**) /home/tianx/mysql/5.6/sql/mysqld.cc:6936 #7 0x7fd5cc942857 in __libc_start_main (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6+0x20857) #8 0x764398 in _start (/data/users/tianx/mysql/5.6/_build-5.6-ASan-PerfSchema/sql/mysqld+0x764398) SUMMARY: AddressSanitizer: 21 byte(s) leaked in 1 allocation(s). Reviewed By: gunnarku Differential Revision: D5210977 fbshipit-source-id: 54e7744

Summary: Fix the following leak reported by ASAN: ==2475771==ERROR: LeakSanitizer: detected memory leaks Direct leak of 40 byte(s) in 1 object(s) allocated from: #0 0x7f97e37fb151 in __interceptor_calloc (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libasan.so.2+0x9a151) #1 0x445229 in my_malloc /home/tianx/mysql/5.6/mysys/my_malloc.c:38 #2 0x44b908 in _lf_alloc_new /home/tianx/mysql/5.6/mysys/lf_alloc-pin.c:440 #3 0x44dfd0 in lf_hash_insert /home/tianx/mysql/5.6/mysys/lf_hash.c:390 #4 0x416220 in find_or_create_file(PFS_thread*, PFS_file_class*, char const*, unsigned int, bool) /home/tianx/mysql/5.6/storage/perfschema/pfs_instr.cc:1353 #5 0x439265 in create_file_v1 /home/tianx/mysql/5.6/storage/perfschema/pfs.cc:1791 #6 0x4099f6 in test_locker_disabled() /home/tianx/mysql/5.6/storage/perfschema/unittest/pfs-t.cc:1150 #7 0x40aef7 in do_all_tests() /home/tianx/mysql/5.6/storage/perfschema/unittest/pfs-t.cc:1659 #8 0x403d38 in main /home/tianx/mysql/5.6/storage/perfschema/unittest/pfs-t.cc:1668 #9 0x7f97e2288857 in __libc_start_main (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6+0x20857) #10 0x404bf8 in _start (/data/users/tianx/mysql/5.6/_build-5.6-ASan-PerfSchema/storage/perfschema/unittest/pfs-t+0x404bf8) SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s). It seems the instrumented file's name is normalized to contain `-instrumented` in it. Previously test wasn't able to clean up the instrumented file "foo". Depends on D5235521 Reviewed By: gunnarku Differential Revision: D5235559 fbshipit-source-id: 8493330

Summary: Fix yet another memory leak reported by ASAN, in perfschema dynamic array of instrument config. LeakSanitizer reported: ==1556117==ERROR: LeakSanitizer: detected memory leaks Direct leak of 21 byte(s) in 1 object(s) allocated from: #0 0x7fd5cee05151 in __interceptor_calloc (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libasan.so.2+0x9a151) facebook#1 0x120ea19 in my_malloc /home/tianx/mysql/5.6/mysys/my_malloc.c:38 facebook#2 0x1891606 in add_pfs_instr_to_array(char const*, char const*) /home/tianx/mysql/5.6/storage/perfschema/pfs_server.cc:251 facebook#3 0x76bd26 in mysqld_get_one_option /home/tianx/mysql/5.6/sql/mysqld.cc:11033 facebook#4 0x1242157 in my_handle_options /home/tianx/mysql/5.6/mysys_ssl/my_getopt.cc:659 facebook#5 0x77d88e in handle_early_options(char) /home/tianx/mysql/5.6/sql/mysqld.cc:8750 facebook#6 0x78d003 in mysqld_main(int, char**) /home/tianx/mysql/5.6/sql/mysqld.cc:6936 facebook#7 0x7fd5cc942857 in __libc_start_main (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6+0x20857) facebook#8 0x764398 in _start (/data/users/tianx/mysql/5.6/_build-5.6-ASan-PerfSchema/sql/mysqld+0x764398) SUMMARY: AddressSanitizer: 21 byte(s) leaked in 1 allocation(s). Test Plan: mtr and the following test: mysqltest.sh --testset=ParallelBig --perfschema --asan --workers=16 auth_sec.secure_file_priv_error Reviewers: gunnarku Reviewed By: gunnarku Subscribers: [email protected] Differential Revision: https://phabricator.intern.facebook.com/D5210977 Tasks: 17217920 Signature: t1:5210977:1497296884:606f6587bfe1595abd7188c1a27f57805b11ee96

Summary: Fix the following leak reported by ASAN: ==2475771==ERROR: LeakSanitizer: detected memory leaks Direct leak of 40 byte(s) in 1 object(s) allocated from: #0 0x7f97e37fb151 in __interceptor_calloc (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libasan.so.2+0x9a151) facebook#1 0x445229 in my_malloc /home/tianx/mysql/5.6/mysys/my_malloc.c:38 facebook#2 0x44b908 in _lf_alloc_new /home/tianx/mysql/5.6/mysys/lf_alloc-pin.c:440 facebook#3 0x44dfd0 in lf_hash_insert /home/tianx/mysql/5.6/mysys/lf_hash.c:390 facebook#4 0x416220 in find_or_create_file(PFS_thread*, PFS_file_class*, char const*, unsigned int, bool) /home/tianx/mysql/5.6/storage/perfschema/pfs_instr.cc:1353 facebook#5 0x439265 in create_file_v1 /home/tianx/mysql/5.6/storage/perfschema/pfs.cc:1791 facebook#6 0x4099f6 in test_locker_disabled() /home/tianx/mysql/5.6/storage/perfschema/unittest/pfs-t.cc:1150 facebook#7 0x40aef7 in do_all_tests() /home/tianx/mysql/5.6/storage/perfschema/unittest/pfs-t.cc:1659 facebook#8 0x403d38 in main /home/tianx/mysql/5.6/storage/perfschema/unittest/pfs-t.cc:1668 facebook#9 0x7f97e2288857 in __libc_start_main (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6+0x20857) facebook#10 0x404bf8 in _start (/data/users/tianx/mysql/5.6/_build-5.6-ASan-PerfSchema/storage/perfschema/unittest/pfs-t+0x404bf8) SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s). It seems the instrumented file's name is normalized to contain `-instrumented` in it. Previously test wasn't able to clean up the instrumented file "foo". Depends on D5235521 Test Plan: mtr mysqltest.sh --asan --perfschema --unit-tests --unit-tests-report empty_table Reviewers: gunnarku Reviewed By: gunnarku Subscribers: [email protected] Differential Revision: https://phabricator.intern.facebook.com/D5235559 Tasks: 17217920 Signature: t1:5235559:1497365801:a1b886d1f09005cd17196dfa44837e4336a5dfe5

Summary: As title. Test Plan: I couldn't reproduce it on my local server, but it fails consistently on sandcastle. Sandcastle filename+dirname may be too long and cause flakiness, while the `-instrumented` postfix has some magic of avoiding/cleaning up leaks. Ran `[build dir]/storage/perfschema/unittest/pfs-t`. Before the fix: ================================================================= ==678962==ERROR: LeakSanitizer: detected memory leaks Direct leak of 40 byte(s) in 1 object(s) allocated from: #0 0x7fd33820a151 in __interceptor_calloc (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libasan.so.2+0x9a151) facebook#1 0x447409 in my_malloc /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/mysys/my_malloc.c:38 facebook#2 0x44dae8 in _lf_alloc_new /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/mysys/lf_alloc-pin.c:440 facebook#3 0x4501b0 in lf_hash_insert /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/mysys/lf_hash.c:390 facebook#4 0x418400 in find_or_create_file(PFS_thread*, PFS_file_class*, char const*, unsigned int, bool) /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/storage/perfschema/pfs_instr.cc:1353 facebook#5 0x441926 in end_file_open_wait_and_bind_to_descriptor_v1 /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/storage/perfschema/pfs.cc:3996 facebook#6 0x40c160 in test_file_instrumentation_leak() /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/storage/perfschema/unittest/pfs-t.cc:1438 facebook#7 0x40d0dc in do_all_tests() /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/storage/perfschema/unittest/pfs-t.cc:1659 facebook#8 0x403d38 in main /data/sandcastle/boxes/trunk-git-mysql-int-tools-primary/mysql-src/storage/perfschema/unittest/pfs-t.cc:1667 facebook#9 0x7fd336c97857 in __libc_start_main (/usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6+0x20857) facebook#10 0x404bf8 in _start (/data/sandcastle/boxes/trunk-git-mysql-int-secondary/_build-5.6-ASan-PerfSchema/storage/perfschema/unittest/pfs-t+0x404bf8) SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s). Clean run after the change. Reviewers: gunnarku Reviewed By: gunnarku Differential Revision: https://phabricator.intern.facebook.com/D5243042 Tasks: 17217920 Signature: t1:5243042:1497460215:c79e007405676c8ea585e1eb09c881fd3ed1ce1f

Under load, if START SLAVE IO_THREAD and STOP SLAVE execute concurrently, the following deadlock is possible: 1) The START thread waits to acquire the channel map RW lock in shared mode and does not check whether it was signaled to quit by the STOP thread: frame facebook#5: 0x000000010075303c mysqld`Checkable_rwlock::rdlock(this=0x0000600001cd80e0) at rpl_gtid.h:464:5 frame facebook#6: 0x0000000100752168 mysqld`Multisource_info::rdlock(this=0x0000000105d2a780) at rpl_msr.h:408:25 frame facebook#7: 0x0000000100751dc8 mysqld`start_handle_slave_stats_daemon() at slave_stats_daemon.cc:233:15 frame facebook#8: 0x00000001020520ec mysqld`handle_slave_io(arg=0x000000011a869000) at rpl_replica.cc:6051:36 2) The STOP thread waits for the previous thread to stop while holding the channel map RW lock in exclusive mode: frame facebook#5: 0x0000000102072a38 mysqld`inline_mysql_cond_timedwait(that=0x000000011a869638, mutex=0x000000011a8694f0, abstime=0x00000001732d9588, src_file="/Users/laurynas/vilniusdb/myr-native-dd-proto/sql/rpl_replica.cc", src_line=2368) at mysql_cond.h:224:16 frame facebook#6: 0x0000000102050128 mysqld`terminate_slave_thread(thd=0x000000011b3ed800, term_lock=0x000000011a8694f0, term_cond=0x000000011a869638, slave_running=0x000000011a8696f4, stop_wait_timeout=0x00000001732d9700, need_lock_term=false, force=false) at rpl_replica.cc:2368:9 frame facebook#7: 0x000000010204fa6c mysqld`terminate_slave_threads(mi=0x000000011a869000, thread_mask=1, stop_wait_timeout=31536000, need_lock_term=false) at rpl_replica.cc:2204:18 frame facebook#8: 0x0000000102048020 mysqld`stop_slave(thd=0x000000011a908000, mi=0x000000011a869000, net_report=true, for_one_channel=true, push_temp_tables_warning=0x00000001732d9a37, invoked_by_raft=false) at rpl_replica.cc:10033:9 frame facebook#9: 0x0000000102048ca8 mysqld`stop_slave_cmd(thd=0x000000011a908000) at rpl_replica.cc:971:13 For the fix, observe that the starting replica I/O thread only tries to signal the stats thread to start, thus move this code to the START REPLICA command-executing thread instead, which already happens to hold the channel map lock. This also forces to move the stopping of the stats thread from the replica I/O thread to the STOP REPLICA command-executing thread. This fixes intermittent but often-seen failures on rpl.rpl_multi_source_channel_map_stress.

Under load, if START SLAVE IO_THREAD and STOP SLAVE execute concurrently, the following deadlock is possible: 1) The START thread waits to acquire the channel map RW lock in shared mode and does not check whether it was signaled to quit by the STOP thread: frame facebook#5: 0x000000010075303c mysqld`Checkable_rwlock::rdlock(this=0x0000600001cd80e0) at rpl_gtid.h:464:5 frame facebook#6: 0x0000000100752168 mysqld`Multisource_info::rdlock(this=0x0000000105d2a780) at rpl_msr.h:408:25 frame facebook#7: 0x0000000100751dc8 mysqld`start_handle_slave_stats_daemon() at slave_stats_daemon.cc:233:15 frame facebook#8: 0x00000001020520ec mysqld`handle_slave_io(arg=0x000000011a869000) at rpl_replica.cc:6051:36 2) The STOP thread waits for the previous thread to stop while holding the channel map RW lock in exclusive mode: frame facebook#5: 0x0000000102072a38 mysqld`inline_mysql_cond_timedwait(that=0x000000011a869638, mutex=0x000000011a8694f0, abstime=0x00000001732d9588, src_file="/Users/laurynas/vilniusdb/myr-native-dd-proto/sql/rpl_replica.cc", src_line=2368) at mysql_cond.h:224:16 frame facebook#6: 0x0000000102050128 mysqld`terminate_slave_thread(thd=0x000000011b3ed800, term_lock=0x000000011a8694f0, term_cond=0x000000011a869638, slave_running=0x000000011a8696f4, stop_wait_timeout=0x00000001732d9700, need_lock_term=false, force=false) at rpl_replica.cc:2368:9 frame facebook#7: 0x000000010204fa6c mysqld`terminate_slave_threads(mi=0x000000011a869000, thread_mask=1, stop_wait_timeout=31536000, need_lock_term=false) at rpl_replica.cc:2204:18 frame facebook#8: 0x0000000102048020 mysqld`stop_slave(thd=0x000000011a908000, mi=0x000000011a869000, net_report=true, for_one_channel=true, push_temp_tables_warning=0x00000001732d9a37, invoked_by_raft=false) at rpl_replica.cc:10033:9 frame facebook#9: 0x0000000102048ca8 mysqld`stop_slave_cmd(thd=0x000000011a908000) at rpl_replica.cc:971:13 For the fix, observe that the starting replica I/O thread only tries to signal the stats thread to start, thus move this code to the START REPLICA command-executing thread instead, which already happens to hold the channel map lock. This also forces to move the stopping of the stats thread from the replica I/O thread to the STOP REPLICA command-executing thread. This fixes intermittent but often-seen failures on rpl.rpl_multi_source_channel_map_stress. Squash with b015dd3

Under load, if START SLAVE IO_THREAD and STOP SLAVE execute concurrently, the following deadlock is possible: - Thread 44 is executing STOP SLAVE, has read-locked the channel map, and is waiting for the slave I/O thread to quit. - Thread 55 is executing START SLAVE, and is blocked waiting for the channel map write lock. - Thread 54 is the SLAVE I/O thread, and is trying to read-lock the channel map lock. The request of T54 is compatible with the current lock state, but according to POSIX, once a write request is pending, it is up to the implementation whether to satisfy them or block. For the fix, observe that the starting replica I/O thread only tries to signal the stats thread to start, thus move this code to the START REPLICA command-executing thread instead, which already happens to hold the channel map lock. This also forces to move the stopping of the stats thread from the replica I/O thread to the STOP REPLICA command-executing thread. This fixes intermittent but often-seen failures on rpl.rpl_multi_source_channel_map_stress under macOS. Squash with b015dd3 Stacktraces: Thread 44: frame facebook#5: 0x0000000106483108 mysqld`inline_mysql_cond_timedwait(that=0x000000014585d438, mutex=0x000000014585d2f0, abstime=0x000000016db61588, src_file="/Users/laurynas/vilniusdb/fb-mysql/sql/rpl_replica.cc", src_line=2367) at mysql_cond.h:224:16 frame facebook#6: 0x0000000106460818 mysqld`terminate_slave_thread(thd=0x00000001458fb800, term_lock=0x000000014585d2f0, term_cond=0x000000014585d438, slave_running=0x000000014585d4f4, stop_wait_timeout=0x000000016db61700, need_lock_term=false, force=false) at rpl_replica.cc:2367:9 frame facebook#7: 0x0000000106460188 mysqld`terminate_slave_threads(mi=0x000000014585ce00, thread_mask=1, stop_wait_timeout=31536000, need_lock_term=false) at rpl_replica.cc:2204:18 frame facebook#8: 0x000000010645873c mysqld`stop_slave(thd=0x0000000145952800, mi=0x000000014585ce00, net_report=true, for_one_channel=true, push_temp_tables_warning=0x000000016db61a37, invoked_by_raft=false) at rpl_replica.cc:10032:9 frame facebook#9: 0x00000001064593c4 mysqld`stop_slave_cmd(thd=0x0000000145952800) at rpl_replica.cc:971:13 Thread 55: frame facebook#2: 0x000000018e5bf73c libsystem_pthread.dylib`_pthread_rwlock_lock_slow + 720 frame facebook#3: 0x0000000104c3d2ac mysqld`native_rw_wrlock(rwp=0x00006000039dc0e8) at thr_rwlock.h:101:10 frame facebook#4: 0x0000000104c3d21c mysqld`inline_mysql_rwlock_wrlock(that=0x00006000039dc0e8, src_file="/Users/laurynas/vilniusdb/fb-mysql/sql/rpl_gtid.h", src_line=473) at mysql_rwlock.h:410:12 frame facebook#5: 0x0000000104c3c6f0 mysqld`Checkable_rwlock::wrlock(this=0x00006000039dc0e0) at rpl_gtid.h:473:5 frame facebook#6: 0x00000001054dc6e0 mysqld`Multisource_info::wrlock(this=0x000000010a182980) at rpl_msr.h:441:25 frame facebook#7: 0x0000000106458bb0 mysqld`start_slave_cmd(thd=0x0000000130008a00) at rpl_replica.cc:832:15 Thread 54: frame facebook#2: 0x000000018e5bf73c libsystem_pthread.dylib`_pthread_rwlock_lock_slow + 720 frame facebook#3: 0x0000000104b62d90 mysqld`native_rw_rdlock(rwp=0x00006000039dc0e8) at thr_rwlock.h:82:10 frame facebook#4: 0x0000000104b62d00 mysqld`inline_mysql_rwlock_rdlock(that=0x00006000039dc0e8, src_file="/Users/laurynas/vilniusdb/fb-mysql/sql/rpl_gtid.h", src_line=464) at mysql_rwlock.h:340:12 frame facebook#5: 0x0000000104b62bcc mysqld`Checkable_rwlock::rdlock(this=0x00006000039dc0e0) at rpl_gtid.h:464:5 frame facebook#6: 0x0000000104b61cec mysqld`Multisource_info::rdlock(this=0x000000010a182980) at rpl_msr.h:415:25 frame facebook#7: 0x0000000104b618f0 mysqld`start_handle_slave_stats_daemon() at slave_stats_daemon.cc:233:15 frame facebook#8: 0x00000001064627dc mysqld`handle_slave_io(arg=0x000000014585ce00) at rpl_replica.cc:6050:36

Summary: After bumping the thread stack size to 10MB for ASAN, this test started failing because the Bison parser ran out of stack before the thread did. This test deliberately tries to cause a thread stack overflow by generating an arbitrarily deeply nested SQL query to exercise the `check_stack_overrun` logic. Because our thread stack is so large in ASAN, we hit the Bison "stack" instead first. From the error log: ``` CURRENT_TEST: main.execution_constants mysqltest: At line 42: Query '$query_head 0 $query_tail' failed with wrong error 3950: 'Out of memory', should have failed with any of '0,1436' errors. ``` Parser OOM Stack: ``` (lldb) bt * thread #42, name = 'connection', stop reason = breakpoint 1.1 * frame #0: 0x000000000a33ea53 mysqld`my_error(nr=<unavailable>, MyFlags=<unavailable>) at my_error.cc:217:3 frame #1: 0x0000000008065a21 mysqld`MYSQLerror(location=<unavailable>, thd=<unavailable>, (null)=<unavailable>, s=<unavailable>) (.__uniq.242601085889238811981650692569837157750) at sql_yacc.yy:302:5 [artificial] frame #2: 0x0000000008034304 mysqld`MYSQLparse(YYTHD=<unavailable>, parse_tree=<unavailable>) at sql_yacc.cc:25323:9 frame #3: 0x0000000008009085 mysqld`THD::sql_parser(this=<unavailable>) at sql_class.cc:3811:7 frame #4: 0x00000000082b1f4c mysqld`parse_sql(thd=0x0000628000300100, parser_state=<unavailable>, creation_ctx=<unavailable>) at sql_parse.cc:7963:34 frame #5: 0x00000000082a060d mysqld`dispatch_sql_command(thd=<unavailable>, parser_state=<unavailable>, last_timer=<unavailable>) at sql_parse.cc:5993:11 frame #6: 0x000000000829bd7b mysqld`dispatch_command(thd=<unavailable>, com_data=<unavailable>, command=<unavailable>) at sql_parse.cc:2445:7 frame #7: 0x000000000829e5e9 mysqld`do_command(thd=<unavailable>) at sql_parse.cc:1636:18 frame #8: 0x00000000086c5e34 mysqld`handle_connection(arg=0x00006030001e0940) (.__uniq.188644437719200549207133117087266310382) at connection_handler_per_thread.cc:307:13 frame #9: 0x000000000b3e123b mysqld`pfs_spawn_thread(arg=0x000061600044f4a0) (.__uniq.322994040891040637281203315118403052672) at pfs.cc:2983:3 frame #10: 0x00007ffff7c9abaf libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:434:8 frame #11: 0x00007ffff7d2d17c libc.so.6`__clone3 at clone3.S:81 ``` sql/sql_yacc.yy: ``` static void MYSQLerror(YYLTYPE *location, THD *thd, Parse_tree_root **, const char *s) { if (strcmp(s, "syntax error") == 0) { thd->syntax_error_at(*location); } else if (strcmp(s, "memory exhausted") == 0) { my_error(ER_DA_OOM, MYF(0)); ... ``` mysql-fork/_build-8.0-ASanDebug/sql/sql_yacc.cc: ``` yyoverflow (YY_("memory exhausted"), &yyss1, yysize * YYSIZEOF (*yyssp), &yyvs1, yysize * YYSIZEOF (*yyvsp), &yyls1, yysize * YYSIZEOF (*yylsp), &yystacksize); ``` Differential Revision: D51233935 fbshipit-source-id: 04391603e7baef0c6e6fa4f308de0aad545474c2

Under load, if START SLAVE IO_THREAD and STOP SLAVE execute concurrently, the following deadlock is possible: - Thread 44 is executing STOP SLAVE, has read-locked the channel map, and is waiting for the slave I/O thread to quit. - Thread 55 is executing START SLAVE, and is blocked waiting for the channel map write lock. - Thread 54 is the SLAVE I/O thread, and is trying to read-lock the channel map lock. The request of T54 is compatible with the current lock state, but according to POSIX, once a write request is pending, it is up to the implementation whether to satisfy them or block. For the fix, observe that the starting replica I/O thread only tries to signal the stats thread to start, thus move this code to the START REPLICA command-executing thread instead, which already happens to hold the channel map lock. This also forces to move the stopping of the stats thread from the replica I/O thread to the STOP REPLICA command-executing thread. This fixes intermittent but often-seen failures on rpl.rpl_multi_source_channel_map_stress under macOS. Squash with b015dd3 Stacktraces: Thread 44: frame facebook#5: 0x0000000106483108 mysqld`inline_mysql_cond_timedwait(that=0x000000014585d438, mutex=0x000000014585d2f0, abstime=0x000000016db61588, src_file="/Users/laurynas/vilniusdb/fb-mysql/sql/rpl_replica.cc", src_line=2367) at mysql_cond.h:224:16 frame facebook#6: 0x0000000106460818 mysqld`terminate_slave_thread(thd=0x00000001458fb800, term_lock=0x000000014585d2f0, term_cond=0x000000014585d438, slave_running=0x000000014585d4f4, stop_wait_timeout=0x000000016db61700, need_lock_term=false, force=false) at rpl_replica.cc:2367:9 frame facebook#7: 0x0000000106460188 mysqld`terminate_slave_threads(mi=0x000000014585ce00, thread_mask=1, stop_wait_timeout=31536000, need_lock_term=false) at rpl_replica.cc:2204:18 frame facebook#8: 0x000000010645873c mysqld`stop_slave(thd=0x0000000145952800, mi=0x000000014585ce00, net_report=true, for_one_channel=true, push_temp_tables_warning=0x000000016db61a37, invoked_by_raft=false) at rpl_replica.cc:10032:9 frame facebook#9: 0x00000001064593c4 mysqld`stop_slave_cmd(thd=0x0000000145952800) at rpl_replica.cc:971:13 Thread 55: frame facebook#2: 0x000000018e5bf73c libsystem_pthread.dylib`_pthread_rwlock_lock_slow + 720 frame facebook#3: 0x0000000104c3d2ac mysqld`native_rw_wrlock(rwp=0x00006000039dc0e8) at thr_rwlock.h:101:10 frame facebook#4: 0x0000000104c3d21c mysqld`inline_mysql_rwlock_wrlock(that=0x00006000039dc0e8, src_file="/Users/laurynas/vilniusdb/fb-mysql/sql/rpl_gtid.h", src_line=473) at mysql_rwlock.h:410:12 frame facebook#5: 0x0000000104c3c6f0 mysqld`Checkable_rwlock::wrlock(this=0x00006000039dc0e0) at rpl_gtid.h:473:5 frame facebook#6: 0x00000001054dc6e0 mysqld`Multisource_info::wrlock(this=0x000000010a182980) at rpl_msr.h:441:25 frame facebook#7: 0x0000000106458bb0 mysqld`start_slave_cmd(thd=0x0000000130008a00) at rpl_replica.cc:832:15 Thread 54: frame facebook#2: 0x000000018e5bf73c libsystem_pthread.dylib`_pthread_rwlock_lock_slow + 720 frame facebook#3: 0x0000000104b62d90 mysqld`native_rw_rdlock(rwp=0x00006000039dc0e8) at thr_rwlock.h:82:10 frame facebook#4: 0x0000000104b62d00 mysqld`inline_mysql_rwlock_rdlock(that=0x00006000039dc0e8, src_file="/Users/laurynas/vilniusdb/fb-mysql/sql/rpl_gtid.h", src_line=464) at mysql_rwlock.h:340:12 frame facebook#5: 0x0000000104b62bcc mysqld`Checkable_rwlock::rdlock(this=0x00006000039dc0e0) at rpl_gtid.h:464:5 frame facebook#6: 0x0000000104b61cec mysqld`Multisource_info::rdlock(this=0x000000010a182980) at rpl_msr.h:415:25 frame facebook#7: 0x0000000104b618f0 mysqld`start_handle_slave_stats_daemon() at slave_stats_daemon.cc:233:15 frame facebook#8: 0x00000001064627dc mysqld`handle_slave_io(arg=0x000000014585ce00) at rpl_replica.cc:6050:36

Summary: add histogram for rpl_semi_sync_master_trx_wait. 8.0 porting notes: Keeps the same histogram status variables as before since these are already being read by various applications. We should eventually remove this. Reference Patch: facebook@d1a1394 Reference Patch: facebook@15333b2e6f9 Differential Revision: D21832889 ---------------------------------------------------------------------- Fix semi_sync histogram reporting Summary: Fix a porting bug with semi_sync histograms. Reviewed By: george-reynya Differential Revision: D40964563 ---------------------------------------------------------------------- Semisync histogram double free (facebook#1290) Summary: Avoid double free on latency histogram data Before this fix, rpl.rpl_semi_sync_alias test under ASan with ``` ================================================================= ==65389==ERROR: AddressSanitizer: heap-use-after-free on address 0x0001742e17d4 at pc 0x000107febaf0 bp 0x00016ea8f710 sp 0x00016ea8f708 READ of size 4 at 0x0001742e17d4 thread T80 #0 0x107febaec in my_free(void*) my_malloc.cc:135 facebook#1 0x103cb9828 in free_latency_histogram_sysvars(SHOW_VAR*) mysqld.cc:4668 facebook#2 0x103cb99bc in prepare_latency_histogram_vars(latency_histogram*, SHOW_VAR*, unsigned long long*) mysqld.cc:4692 facebook#3 0x17c65826c in rpl_semi_sync_master_trx_wait_histogram(THD*, SHOW_VAR*, char*) semisync_source_plugin.cc:581 facebook#4 0x10be1b4cc in PFS_status_variable_cache::manifest(THD*, SHOW_VAR const*, System_status_var*, char const*, bool, bool) pfs_variable.cc:1366 facebook#5 0x10be1ba90 in PFS_status_variable_cache::do_materialize_all(THD*) pfs_variable.cc:1172 facebook#6 0x10c0ab33c in PFS_variable_cache<Status_variable>::materialize_all(THD*) pfs_variable.h:536 facebook#7 0x10c0ab294 in table_session_status::rnd_init(bool) table_session_status.cc:111 facebook#8 0x10bceb790 in ha_perfschema::rnd_init(bool) ha_perfschema.cc:1686 facebook#9 0x1033c7cec in handler::ha_rnd_init(bool) handler.cc:3157 facebook#10 0x103975380 in TableScanIterator::Init() basic_row_iterators.cc:230 facebook#11 0x103a33a18 in FilterIterator::Init() composite_iterators.h:82 facebook#12 0x103982ec0 in MaterializeIterator::MaterializeQueryBlock(MaterializeIterator::QueryBlock const&, unsigned long long*) composite_iterators.cc:845 facebook#13 0x103981410 in MaterializeIterator::Init() composite_iterators.cc:660 facebook#14 0x1049fc518 in Query_expression::ExecuteIteratorQuery(THD*) sql_union.cc:1293 facebook#15 0x1049fd358 in Query_expression::execute(THD*) sql_union.cc:1355 facebook#16 0x1047ae7ac in Sql_cmd_dml::execute_inner(THD*) sql_select.cc:870 facebook#17 0x1047ac344 in Sql_cmd_dml::execute(THD*) sql_select.cc:618 facebook#18 0x1047ffcc8 in Sql_cmd_show::execute(THD*) sql_show.cc:232 facebook#19 0x10480ab58 in Sql_cmd_show_status::execute(THD*) sql_show.cc:894 facebook#20 0x1045cea6c in mysql_execute_command(THD*, bool, unsigned long long*) sql_parse.cc:5323 facebook#21 0x1045c5dcc in dispatch_sql_command(THD*, Parser_state*, unsigned long long*) sql_parse.cc:6093 facebook#22 0x1045bb92c in dispatch_command(THD*, COM_DATA const*, enum_server_command) sql_parse.cc:2444 facebook#23 0x1045c06f8 in do_command(THD*) sql_parse.cc:1636 facebook#24 0x104cc4cc4 in handle_connection(void*) connection_handler_per_thread.cc:307 facebook#25 0x10bd130d4 in pfs_spawn_thread(void*) pfs.cc:2983 facebook#26 0x18ad47fa4 in _pthread_start+0x90 (libsystem_pthread.dylib:arm64e+0x6fa4) facebook#27 0x18ad42d9c in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1d9c) 0x0001742e17d4 is located 4 bytes inside of 40-byte region [0x0001742e17d0,0x0001742e17f8) freed by thread T80 here: #0 0x139ff6de4 in wrap_free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3ede4) facebook#1 0x107febcfc in my_raw_free(void*) my_malloc.cc:269 facebook#2 0x107feba48 in my_free(void*) my_malloc.cc:141 facebook#3 0x103cb9828 in free_latency_histogram_sysvars(SHOW_VAR*) mysqld.cc:4668 facebook#4 0x17c6231e8 in ReplSemiSyncMaster::~ReplSemiSyncMaster() semisync_source.cc:517 facebook#5 0x17c623488 in ReplSemiSyncMaster::~ReplSemiSyncMaster() semisync_source.cc:516 facebook#6 0x17c651484 in semi_sync_master_plugin_deinit(void*) semisync_source_plugin.cc:833 facebook#7 0x10467aa90 in plugin_deinitialize(st_plugin_int*, bool) sql_plugin.cc:1123 facebook#8 0x1046730b0 in reap_plugins() sql_plugin.cc:1192 facebook#9 0x1046863b4 in mysql_uninstall_plugin(THD*, MYSQL_LEX_CSTRING) sql_plugin.cc:2602 facebook#10 0x104685374 in Sql_cmd_uninstall_plugin::execute(THD*) sql_plugin.cc:3731 facebook#11 0x1045cea6c in mysql_execute_command(THD*, bool, unsigned long long*) sql_parse.cc:5323 facebook#12 0x1045c5dcc in dispatch_sql_command(THD*, Parser_state*, unsigned long long*) sql_parse.cc:6093 facebook#13 0x1045bb92c in dispatch_command(THD*, COM_DATA const*, enum_server_command) sql_parse.cc:2444 facebook#14 0x1045c06f8 in do_command(THD*) sql_parse.cc:1636 facebook#15 0x104cc4cc4 in handle_connection(void*) connection_handler_per_thread.cc:307 facebook#16 0x10bd130d4 in pfs_spawn_thread(void*) pfs.cc:2983 facebook#17 0x18ad47fa4 in _pthread_start+0x90 (libsystem_pthread.dylib:arm64e+0x6fa4) facebook#18 0x18ad42d9c in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x1d9c) ``` It seems that the double invocation of `free_latency_histogram_sysvars` is correct in this case, thus protect against the double free with resetting the pointers to nullptr. Pull Request resolved: facebook#1290 Reviewed By: sunshine-Chun Differential Revision: D45277600 Pulled By: hermanlee

steaphan-fb-com closed this as completed Nov 18, 2014

mariadb-DanielLee mentioned this issue Nov 27, 2015

mysqld some times crashes at the end of Random Query Generator test #111

Closed

This was referenced Dec 8, 2015

MyRocks slave crashed because of the wrong errkey. #120

Closed

mysqld crashed when running a RQG transactions test, not reproducible at will #123

Closed

This was referenced Dec 28, 2015

handle_fatal_signal (sig=11) in thd_ha_data #129

Closed

Assertion `v->cfd_->internal_comparator().Compare(start, end) <= 0' failed. #131

Closed

rameshvs02 mentioned this issue Feb 18, 2016

InnoDB: Failing assertion: no_trx || parent_trx->is_primary in file ha_innodb.cc line 11527 #170

Closed

mariadb-DanielLee mentioned this issue Apr 28, 2016

RQG test failed with a core dump #237

Closed

RickPizzi mentioned this issue Oct 21, 2016

MyRocks: 1 hour stall in DelayWrite() during bulk load #348

Closed

spetrunia mentioned this issue Nov 16, 2016

rocksdb.bulk_load test times out #428

Closed

alxyang mentioned this issue Jan 23, 2017

Flush memtable and compact L0 files for read only workloads #427

Open

laurynas-biveinis mentioned this issue Mar 14, 2017

-DWITH_ASAN=ON results in broken server if system AddressSanitizer includes LeakSanitizer #567

Closed

This was referenced Sep 21, 2017

Server asserts on startup with rocksdb-persistent-cache-size-mb #713

Open

Assertion `!rocksdb_db_options->allow_mmap_writes' failed in virtual void myrocks::Rdb_background_thread::run() #714

Open

This was referenced Oct 11, 2017

Assertion `covers_key(b)' failed in int myrocks::Rdb_key_def::cmp_full_keys(const rocks db::Slice&, const rocksdb::Slice&) #728

Closed

Start server with --rocksdb-table-cache-numshardbits=100000000 segfaults #739

Open

safe-net9 mentioned this issue Dec 6, 2017

HHVM stops at segmentation fault. #769

Closed

RoelVdP mentioned this issue Dec 31, 2017

MyRocks crashes when using tmpfs for data directory & --rocksdb-use-direct-reads=1 #778

Closed

george-lorch mentioned this issue Jan 10, 2018

rocksdb: crash & corruption, charsets, indexed cols #784

Closed

george-lorch mentioned this issue Feb 12, 2018

prod201704 AUTO INCREMENT regressions #792

Closed

hermanlee mentioned this issue Aug 23, 2024

Implement LAST_INSERT_IDS function #1469

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Defragmentation for partitioned InnoDB table #8

Defragmentation for partitioned InnoDB table #8

SunguckLee commented Nov 18, 2014

rongrong commented Nov 18, 2014

SunguckLee commented Nov 18, 2014

steaphan-fb-com commented Nov 18, 2014

SunguckLee commented Nov 24, 2014

Defragmentation for partitioned InnoDB table #8

Defragmentation for partitioned InnoDB table #8

Comments

SunguckLee commented Nov 18, 2014

rongrong commented Nov 18, 2014

SunguckLee commented Nov 18, 2014

steaphan-fb-com commented Nov 18, 2014

SunguckLee commented Nov 24, 2014