feat: use u64 hash in buffer index instead of str literal

[hiltontj]

Closes #25875

[hiltontj]

I made a couple of QoL improvements in 732874a in addition to moving the call to hash the row value after the check to see if it is an indexed column.

[praveen-influx]

Looks good - I can generally follow the code, I guess because we need to keep a tab on the actual hashes (Hashset<u64>?) outside the HashMap, we are not passing the hasher function through Hashmap::with_hasher?

[hiltontj]

Looks good - I can generally follow the code, I guess because we need to keep a tab on the actual hashes (Hashset<u64>?) outside the HashMap, we are not passing the hasher function through Hashmap::with_hasher?

By HashSet<u64> are you referring to the HashSet<usize>? If so, that is the set of row indexes, so we don't use the XX Hasher there - I think we need to use a cryptographically secure hash on that, because the row indices that a given value falls into needs to be correct.

The XX hasher is only for taking the values, which are originally string literals, and converting them to a u64, so that they are moved around, stored, and compared more cheaply.

The previous structure of BufferIndex was:

BufferIndex {
    Column ID -> string literal -> Set of Indexes from the table buffer
}

This changes it to

BufferIndex {
    Column ID -> u64 hash of string literal -> Set of Indexes from the table buffer
}

Consequently, by using XX Hash, there is a chance of hash collisions (though unlikely), but that is acceptable in this case, because the result would be that the buffer produces excess rows, which DataFusion will filter out.

[praveen-influx]

The XX hasher is only for taking the values, which are originally string literals, and converting them to a u64

I see - got it. Happy for it to be merged.

[pauldix]

We shouldn't be hashing the row indexes, they're already in the optimal type for doing set union and intersections on them, as long as they're sorted.

[hiltontj]

We shouldn't be hashing the row indexes, they're already in the optimal type for doing set union and intersections on them, as long as they're sorted.

@pauldix Ah, my changing to a HashSet<usize> to hold row indices in #25866 could be an issue if keeping them in sorted order is important. Previously it was a Vec<usize> which would keep the rows in the order in which they are added - which would be sorted order.

If that is important, we could switch to use an IndexSet.

[pauldix]

Yeah, since the rows always arrive in order (i.e. every row added is always > than any row before) and they're never added to an entry more than once, it's wasteful to use a set rather than just appending to a vec. And doing set operations on two sorted vecs is generally as fast as it gets. No reason to use a hash set.