vecindex: add support for vector index prefix columns #142050
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
This change is |
andy-kimball
force-pushed
the
prefix
branch
3 times, most recently
from
7d0bf2d to
8c8ea92
Compare
DrewKimball
approved these changes
Feb 28, 2025
There was a problem hiding this comment.
Reviewed 24 of 24 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @mw5h)
andy-kimball
force-pushed
the
prefix
branch
4 times, most recently
from
89eb0af to
4ddffbc
Compare
The CREATE VECTOR INDEX syntax allows indexing over multiple columns, as long as the vector column to be indexed is the last column in the index definition. The other "prefix" columns can be used to partition the index by tenants, regions, users, etc. The execution engine encodes prefix columns as a byte slice and passes it as a parameter to vector index operations like Insert and Search. While the index itself treats these bytes as an opaque "TreeKey", the CRDB Store implementation incorporates these prefix bytes into KV keys. This prefixing mechanism has the effect of separating the index into distinct K-means trees, each identified by a unique TreeKey. CRDB partitioning can control where those trees are located, e.g. an app that stores indexed user photo embeddings in a region close to them. Epic: CRDB-42943 Release note: None
DrewKimball
approved these changes
Mar 1, 2025
There was a problem hiding this comment.
Reviewed 13 of 13 files at r3, all commit messages.
Reviewable status:
|
bors r=drewkimball |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The CREATE VECTOR INDEX syntax allows indexing over multiple columns, as long as the vector column to be indexed is the last column in the index definition. The other "prefix" columns can be used to partition the index by tenants, regions, users, etc. The execution engine encodes prefix columns as a byte slice and passes it as a parameter to vector index operations like Insert and Search. While the index itself treats these bytes as an opaque "TreeKey", the CRDB Store implementation incorporates these prefix bytes into KV keys.
This prefixing mechanism has the effect of separating the index into distinct K-means trees, each identified by a unique TreeKey. CRDB partitioning can control where those trees are located, e.g. an app that stores indexed user photo embeddings in a region close to them.
Epic: CRDB-42943
Release note: None