DB Corruption on parity v2.5.5

[gituser]

• Parity Ethereum version: 2.5.5
• Operating system: Linux
• Installation: built from source
• Fully synchronized: yes
• Network: ethereum
• Restarted: yes

Your issue description goes here below. Try to include actual vs. expected behavior and steps to reproduce the issue.

I just had a crash with DB corrupted on v2.5.5 too.

And parity isn't restoring anything it's simply stuck in limbo and requires full resync. Super annoying.

I don't see any errors in dmesg or anywhere else about failed sectors on my NVME drives, so it has to be something to do with parity:

2019-07-16 03:41:11  Verifier #0 WARN kvdb_rocksdb  DB corrupted: Corruption: bad entry in block. Repair will be triggered on next restart


====================

stack backtrace:
   0:     0x563ac446e1ed - backtrace::backtrace::trace::h3b13a3b0bfa8fe24
   1:     0x563ac446b2be - panic_hook::gen_panic_msg::hb3f9980518fbc3e8
   2:     0x563ac4584af7 - panic_hook::set_with::{{closure}}::h2ef65aa679d4a524
   3:     0x563ac44afd04 - std::panicking::rust_panic_with_hook::h057ff03eb4c8000f
                        at src/libstd/panicking.rs:478
   4:     0x563ac44afa7e - std::panicking::continue_panic_fmt::ha6d6ae144369025b
                        at src/libstd/panicking.rs:381
   5:     0x563ac44cc5b5 - rust_begin_unwind
                        at src/libstd/panicking.rs:308
   6:     0x563ac44d250b - core::panicking::panic_fmt::hc4f83bfed80aeabd
                        at src/libcore/panicking.rs:85
   7:     0x563ac41f2686 - core::result::unwrap_failed::h1da4e2a190b03633
   8:     0x563ac3f5776d - ethcore::client::client::Importer::import_verified_blocks::hdb8a24774fb46554
   9:     0x563ac3809281 - <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message::h638abf17257bd545
  10:     0x563ac3d406a9 - ethcore_io::service_mio::IoChannel<Message>::send_sync::h2f200202ec2466bc
  11:     0x563ac3d1991b - ethcore::verification::queue::QueueSignal::set_sync::hda12cace25390049
  12:     0x563ac3d44ced - std::sys_common::backtrace::__rust_begin_short_backtrace::h8b4c2a14eebf2bf4
  13:     0x563ac3dcd104 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6babe2e56d3b5a8d
  14:     0x563ac44cb33e - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h805c3cc89d534c05
                        at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/liballoc/boxed.rs:704
  15:     0x563ac44cde9b - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h9ebda9db5f03daf3
                        at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/liballoc/boxed.rs:704
                         - std::sys_common::thread::start_thread::ha32838ccbb8a8179
                        at src/libstd/sys_common/thread.rs:13
                         - std::sys::unix::thread::Thread::new::thread_start::h6f10b78f26c98dc6
                        at src/libstd/sys/unix/thread.rs:79
  16:     0x7f3960e3c063 - start_thread
  17:     0x7f396095b62c - clone
  18:                0x0 - <unknown>

Thread 'Verifier #0' panicked at 'DB flush failed.: Custom { kind: Other, error: StringError("Corruption: bad entry in block") }', src/libcore/result.rs:999

This is a bug. Please report it at:

    https://github.com/paritytech/parity-ethereum/issues/new


2019-07-16 03:41:11  main INFO parity_ethereum::run  Finishing work, please wait...
2019-07-16 03:41:12  IO Worker #1 WARN kvdb_rocksdb  DB corrupted: Corruption: bad entry in block. Repair will be triggered on next restart
2019-07-16 03:41:14  jsonrpc-eventloop-0 WARN shutdown  unable to get mut ref for engine for shutdown.


====================

stack backtrace:
   0:     0x563ac446e1ed - backtrace::backtrace::trace::h3b13a3b0bfa8fe24
   1:     0x563ac446b2be - panic_hook::gen_panic_msg::hb3f9980518fbc3e8
   2:     0x563ac4584af7 - panic_hook::set_with::{{closure}}::h2ef65aa679d4a524
   3:     0x563ac44afd04 - std::panicking::rust_panic_with_hook::h057ff03eb4c8000f
                        at src/libstd/panicking.rs:478
   4:     0x563ac44afa7e - std::panicking::continue_panic_fmt::ha6d6ae144369025b
                        at src/libstd/panicking.rs:381
   5:     0x563ac44cc5b5 - rust_begin_unwind
                        at src/libstd/panicking.rs:308
   6:     0x563ac44d250b - core::panicking::panic_fmt::hc4f83bfed80aeabd
                        at src/libcore/panicking.rs:85
   7:     0x563ac47ce770 - core::result::unwrap_failed::h96bf7c627fd8285a
   8:     0x563ac47a7800 - <ethcore::verification::queue::VerificationQueue<K> as core::ops::drop::Drop>::drop::hdb58fbf8a0150d7e
   9:     0x563ac48a9400 - core::ptr::real_drop_in_place::h30eb7831c14ae48b.74788
  10:     0x563ac48a9131 - core::ptr::real_drop_in_place::h4d3e2bc2754f6d81.74767
  11:     0x563ac419ca1a - alloc::sync::Arc<T>::drop_slow::h002620ab4abf0bdc
  12:     0x563ac419ca1a - alloc::sync::Arc<T>::drop_slow::h002620ab4abf0bdc
  13:     0x563ac37484e4 - alloc::sync::Arc<T>::drop_slow::h33849f564b3a0419
  14:     0x563ac35f30da - core::ptr::real_drop_in_place::h96b851c9e7b38507
  15:     0x563ac419ca1a - alloc::sync::Arc<T>::drop_slow::h002620ab4abf0bdc
  16:     0x563ac452836e - alloc::sync::Arc<T>::drop_slow::hb49f61fd79773c0d
  17:     0x563ac419ca1a - alloc::sync::Arc<T>::drop_slow::h002620ab4abf0bdc
  18:     0x563ac404cb9c - core::ptr::real_drop_in_place::h30a1235cffa36a64
  19:     0x563ac4526d03 - alloc::sync::Arc<T>::drop_slow::h70b744b231a7451e
  20:     0x563ac46f7625 - core::ptr::real_drop_in_place::habae2440e7bc2b3c
  21:     0x563ac486192b - <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll::h9718e8ab5c38deb3
  22:     0x563ac43620ff - futures::task_impl::std::set::h78da85acf379b616
  23:     0x563ac435fff2 - tokio_threadpool::worker::Worker::run_task::h8707b008c521fd33
  24:     0x563ac435e0c2 - tokio_threadpool::worker::Worker::run::hc14a9b13d6edffae
  25:     0x563ac434408e - tokio::runtime::builder::Builder::build::{{closure}}::h2ce8ae50d62f213e
  26:     0x563ac43626fb - std::sys_common::backtrace::__rust_begin_short_backtrace::h9468191d4a2bdecf
  27:     0x563ac4360d4f - core::ops::function::FnOnce::call_once{{vtable.shim}}::h55d0c5125eea3ba0
  28:     0x563ac44cb33e - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h805c3cc89d534c05
                        at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/liballoc/boxed.rs:704
  29:     0x563ac44cde9b - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h9ebda9db5f03daf3
                        at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/liballoc/boxed.rs:704
                         - std::sys_common::thread::start_thread::ha32838ccbb8a8179
                        at src/libstd/sys_common/thread.rs:13
                         - std::sys::unix::thread::Thread::new::thread_start::h6f10b78f26c98dc6
                        at src/libstd/sys/unix/thread.rs:79
  30:     0x7f3960e3c063 - start_thread
  31:     0x7f396095b62c - clone
  32:                0x0 - <unknown>

Thread 'jsonrpc-eventloop-0' panicked at 'Propagating verifier thread panic on shutdown: Any', src/libcore/result.rs:999

This is a bug. Please report it at:

    https://github.com/paritytech/parity-ethereum/issues/new


2019-07-16 03:41:14  jsonrpc-eventloop-0 WARN kvdb_rocksdb  DB corrupted: Corruption: bad entry in block. Repair will be triggered on next restart

[gituser]

It happened again:

traps: IO Worker #2[10808] trap invalid opcode ip:55cb835ef900 sp:7fd931ffa4e0 error:0 in parity-v2.5.5-stable-3ebc769-20190708[55cb828c0000+2064000]

I checked there is nothing in the logs this time.