[DocDB] Block cache key generation algorithm may return already existing key

[arybochkin]

Jira Link: DB-9839

Description

Current implementation of block cache key prefix generation does not provide enough guarantee of returning a unique key for a SST file. It uses the inode information, which can be reused after the file deletion while some old blocks of that deleted file may still be present in the block cache, and the cache will return an old block instead of a new block, where the block handle has the information about the deleted file. Observed effects of such behaviour are: block checksum error during compaction and flush, attempt to read by the offset past the file end, and further crashes in some cases.

Block cache key prefix is composed of (inode device id, inode number, inode generation). In older filesystems (even older versions of ext4) the inode generation is randomly generated. Then if we recycle inode number a bunch of times, which file systems tend to do, we can end up reading a stale block from the cache. Also XFS might generate random number and put it into inode’s generation number instead of reading from disk, when inode info is not in disk cache.

The similar issues were met in the upstream RocksDB.

Potential fixes:

• Provide a strong guarantee of cleaning the block cache by removing all the blocks during a file deletion.
• Port an upstream fix to have more stable cache keys
• Use ikeep mount option for XFS: https://man7.org/linux/man-pages/man5/xfs.5.html. Also it might be even better for SSD to use it: https://superuser.com/a/599944:

Fragmentation and things being “scattered around the disk more” cause significant performance problems for traditional spinning-platter disks. However, on an SSD, those things are no problem at all. Conversely, deleting disk blocks is a significant performance problem on SSDs while being no problem for spinning-disks. So while the patch making it the default to delete empty inodes was a performance improvement for spinning-disks, it actually made performance worse on SSDs. Hence the recommendation to use ikeep on SSDs."

Also from XFS source code:

/*
 * Read the disk inode attributes into the in-core inode structure.
 *
 * For version 5 superblocks, if we are initialising a new inode and we are not
 * utilising the XFS_MOUNT_IKEEP inode cluster mode, we can simple build the new
 * inode core with a random generation number. If we are keeping inodes around,
 * we need to read the inode cluster to get the existing generation number off
 * disk. Further, if we are using version 4 superblocks (i.e. v1/v2 inode
 * format) then log recovery is dependent on the di_flushiter field being
 * initialised from the current on-disk value and hence we must also read the
 * inode off disk.
 */
int
xfs_iread(

The following issues could be related to the conflicting keys problem: #10621, #12681, #13109, #14064

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[rthallamko3]

Reactivating for 2024.1 and 2.20 backports.