storage core after exit

[critical27]

Master, reproduce every time.

Core stack

#0  0x00007f8d00436387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f8d00437a78 in __GI_abort () at abort.c:90
#2  0x0000000003ffa1f2 in rocksdb::port::PthreadCall(char const*, int) [clone .part.0] ()
#3  0x0000000003ffa2f0 in rocksdb::port::Mutex::Lock() ()
#4  0x0000000003e675b2 in rocksdb::PeriodicWorkScheduler::Unregister(rocksdb::DBImpl*) ()
#5  0x0000000003d8f1db in rocksdb::DBImpl::CancelAllBackgroundWork(bool) ()
#6  0x0000000003d9cd9a in rocksdb::DBImpl::CloseHelper() ()
#7  0x0000000003da71a3 in rocksdb::DBImpl::~DBImpl() ()
#8  0x0000000003da7202 in rocksdb::DBImpl::~DBImpl() ()
#9  0x0000000003bafd7e in std::default_delete<rocksdb::DB>::operator() (this=0x7f8d0001db30, __ptr=0x7f8d00089800) at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:81
#10 0x0000000003ba52b4 in std::unique_ptr<rocksdb::DB, std::default_delete<rocksdb::DB> >::~unique_ptr (this=0x7f8d0001db30) at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:284
#11 0x0000000003bad4a4 in nebula::kvstore::RocksEngine::~RocksEngine (this=0x7f8d0001dae0) at /data/doodle/Git/nebula/src/kvstore/RocksEngine.h:120
#12 0x0000000003bad5ab in nebula::kvstore::RocksEngine::~RocksEngine (this=0x7f8d0001dae0) at /data/doodle/Git/nebula/src/kvstore/RocksEngine.h:120
#13 0x0000000002574b8e in std::default_delete<nebula::kvstore::KVEngine>::operator() (this=0x7f8d0000a130, __ptr=0x7f8d0001dae0) at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:81
#14 0x000000000251d864 in std::unique_ptr<nebula::kvstore::KVEngine, std::default_delete<nebula::kvstore::KVEngine> >::~unique_ptr (this=0x7f8d0000a130) at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:284
#15 0x0000000002555565 in nebula::storage::StorageEnv::~StorageEnv (this=0x7f8d0000a0e0) at /data/doodle/Git/nebula/src/storage/CommonUtils.h:71
#16 0x000000000255544a in std::default_delete<nebula::storage::StorageEnv>::operator() (this=0x7f8d0004f698, __ptr=0x7f8d0000a0e0) at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:81
#17 0x000000000251a2f4 in std::unique_ptr<nebula::storage::StorageEnv, std::default_delete<nebula::storage::StorageEnv> >::~unique_ptr (this=0x7f8d0004f698)
    at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:284
#18 0x000000000251a928 in nebula::storage::StorageServer::~StorageServer (this=0x7f8d0004f600) at /data/doodle/Git/nebula/src/storage/StorageServer.cpp:57
#19 0x000000000251475a in std::default_delete<nebula::storage::StorageServer>::operator() (this=0x56280c0 <gStorageServer>, __ptr=0x7f8d0004f600) at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:81
#20 0x0000000002504ab4 in std::unique_ptr<nebula::storage::StorageServer, std::default_delete<nebula::storage::StorageServer> >::~unique_ptr (this=0x56280c0 <gStorageServer>)
    at /opt/vesoft/toolset/clang/10.0.0/lib/gcc/x86_64-vesoft-linux/9.2.0/../../../../include/c++/9.2.0/bits/unique_ptr.h:284
#21 0x00007f8d00439ce9 in __run_exit_handlers (status=0, listp=0x7f8d007c76c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#22 0x00007f8d00439d37 in __GI_exit (status=<optimized out>) at exit.c:99
#23 0x00007f8d0042255c in __libc_start_main (main=0x2504ae0 <main(int, char**)>, argc=3, argv=0x7ffc3c38c878, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc3c38c868) at ../csu/libc-start.c:300
#24 0x0000000002503e4e in _start ()

stderr:

pthread lock: Invalid argument
pthread lock: Invalid argument
terminate called without an active exception
*** Aborted at 1634107808 (Unix time, try 'date -d @1634107808') ***
*** Signal 6 (SIGABRT) (0x3e800005604) received by PID 22020 (pthread TID 0x7f3c28d639c0) (linux TID 22020) (maybe from PID 22020, UID 1000) (code: -6), stack trace: ***
./nebula/build/bin/nebula-storaged(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x216f1b1]
./nebula/build/bin/nebula-storaged(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x1b)[0x2166f8b]
./nebula/build/bin/nebula-storaged[0x21652d7]
/lib64/libpthread.so.0(+0xf62f)[0x7f3c2822762f]
/lib64/libc.so.6(gsignal+0x37)[0x7f3c27e80387]
/lib64/libc.so.6(abort+0x147)[0x7f3c27e81a77]
./nebula/build/bin/nebula-storaged[0xe579e1]
./nebula/build/bin/nebula-storaged(_ZN10__cxxabiv111__terminateEPFvvE+0x5)[0x2627935]
./nebula/build/bin/nebula-storaged(_ZSt9terminatev+0x10)[0x2627980]
./nebula/build/bin/nebula-storaged(_ZN6nebula7storage16AdminTaskManagerD2Ev+0xe6)[0xead3c6]
/lib64/libc.so.6(+0x39ce8)[0x7f3c27e83ce8]
/lib64/libc.so.6(exit+0x16)[0x7f3c27e83d36]
/lib64/libc.so.6(__libc_start_main+0xfb)[0x7f3c27e6c55b]
./nebula/build/bin/nebula-storaged[0xea418d]
(safe mode, symbolizer not available)
pthread lock: Invalid argument
pthread lock: Invalid argument
pure virtual method called
terminate called without an active exception
pthread lock: Invalid argument
pthread lock: Invalid argument

[liwenhui-soul]

"pthread lock: Invalid argument" shows that timer_mu_ has already been destructed in PeriodicWorkScheduler::Unregister().
the reason is that:
PeriodicWorkScheduler in rocksdb is a static variable, and gStorageServer is a global variable, and PeriodicWorkScheduler may be destructed first when proccess exit, so PeriodicWorkScheduler::Unregister() may get error in gStorageServer's destruction.
destructing gStorageServer or env_->adminStore_ manually would fix it.