sst file data block is not in sorted order and have duplicate key

[xueyonglu]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Our ceph cluster use rocksdb, the version is 5.4.0, we encounter a failure when get() from DB，and it has happened several times.

I analyzed the sst file , it's a Level6 file, the keys in data block as follows:

Data block 1
Restart : auth2300、auth2301、…、auth2315 (16个key)
Restart : auth2316、…、auth2320、authfirst_committed、authformat_version、auth2146、…、auth2154 (16个key)
Restart : auth2155、auth2156 (2个key)

Data block 2
Restart : auth2157、…、auth2172 (16个key)
Restart : auth2173、…、auth2188 (16个key)

Data block 3
Restart : auth2189、…、auth2197、authfirst_committed、authformat_version、authfull_2301 (12个key)

we can see
1）DataBlock1 is not in sorted order；
2）DataBlock1 and DataBlock3 have duplicate key（authfirst_committed、authformat_version）

Expected behavior

the data block of sst file is in sorted order ,and have no duplicate key between different data block

Actual behavior

the data block of sst file is not in sorted order and have duplicate key

Steps to reproduce the behavior

This is encountered in a production environment，I didn't repeat it.

Does anyone know if there are any related bugs in the current version Or how should I locate the root cause?

[ajkr]

Regarding prevention, you can turn on the option check_flush_compaction_key_order, which prevents files with incorrect ordering from being persisted. However this option is only available in versions 6.14 or newer.

Regarding root cause, it will be difficult to confirm. I describe one possible root cause here: #7405 (comment). Other possible root causes, like a rare concurrency bug, memory corruption, undefined behavior, or hardware issue, would be even more difficult to confirm.

[xueyonglu]

Hi @ajkr ,thanks for your replay.
I will check in our environment what you mentioned in #7405 .