Enable rocksdb.enable_blob_files will cause blob files leak

[sryanyuan]

Search before asking

• I had searched in the issues and found no similar issues.

Version

centos 7.9/5.10.109-1.el7.x86_64，kvrocks 2.10.0

Minimal reproduce step

Enabling rocksdb.enable_blob_files and continuing writing data with ttl will cause disk full finally. By checking the file descriptors of the kvrocks process, we can determine that many blob files have been deleted but are still open by kvrocks.

What did you expect to see?

disk usage stable

What did you see instead?

disk full finally

Anything Else?

We checked the issue list of rocksdb, the issue may be related to this. The recent release version 2.10.0, which upgraded rocksdb to 9.6.1, may pose a potential risk for users of old versions who have the option rocksdb.enable_blob_files enabled.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[git-hulk]

@sryanyuan Thanks for your information. Will keep an eye on this issue and apply it in Kvrocks ASAP.

[ilyaevseev]

The same problem with kvrocks 2.10.0 and rocksdb.enable_blob_files=yes:

# lsof -n > /run/lsof.log 2>/dev/null
# egrep -c 'kvrocks.*deleted' /run/lsof.log 
3060

# egrep '^(COMMAND|kvrocks.*deleted)' /run/lsof.log | less
COMMAND       PID     TID TASKCMD              USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
kvrocks    415416                               999    7r      REG              259,1 123878620    2627813 /var/lib/kvrocks/db/000065.blob (deleted)
kvrocks    415416                               999    8r      REG              259,1 123878620    2627787 /var/lib/kvrocks/db/000051.blob (deleted)
kvrocks    415416                               999   50r      REG              259,1 235657931    2627945 /var/lib/kvrocks/db/000189.blob (deleted)
kvrocks    415416                               999   57r      REG              259,1 181792793    2627901 /var/lib/kvrocks/db/000738.blob (deleted)
...

# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        33G   33G   55M 100% /mnt/data

# docker restart kvrocks
kvrocks

# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        33G   13G   21G  38% /mnt/data

[git-hulk]

@ilyaevseev Thanks for your investigation. We would like to have a patch release in a few days. Should we revert the RocksDB to 8.11.4 in this patch release as well? @apache/kvrocks-committers

[PragmaTwice]

Yeah I think we can use rocksdb 8.11 in 2.10.1.

[PragmaTwice]

Also in the future maybe we can expose uncache_aggressiveness to users, according to facebook/rocksdb@b34cef5.

[ilyaevseev]

Yeah I think we can use rocksdb 8.11 in 2.10.1.

Is RocksDB 8.11 compatible with 9.x by data formats?
I.e., should we recreate our databases from scratch after Kvrocks upgrade from 2.10 to 2.10.1 ?

[PragmaTwice]

Yeah I think we can use rocksdb 8.11 in 2.10.1.

Is RocksDB 8.11 compatible with 9.x by data formats? I.e., should we recreate our databases from scratch after Kvrocks upgrade from 2.10 to 2.10.1 ?

From here for forward compatibility it said that:

an older release of RocksDB should be able to open DB generated by releases at least several minor versions higher, if no new features are explicitly enabled by users which are not supported in older versions

So I think we can do some experiments before we do the downgrade. cc @git-hulk
Also, we can choose v9.3.1 which didn't introduce this issue.

I prefer v9.3.1 than v8.11 since v8 seems a far big downgrade.

cc @mapleFU

[git-hulk]

@PragmaTwice Downgrade to 9.3.1 is also good to me since this issue was first introduced in RocksDB 9.4.0

[mapleFU]

I prefer v9.3.1 than v8.11 since v8 seems a far big downgrade.

I agree downgrade to 9.3.1