Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to s2 for bulk loader and backup/restore #9315

Merged
merged 1 commit into from
Feb 26, 2025

Conversation

RJKeevil
Copy link
Contributor

@RJKeevil RJKeevil commented Feb 20, 2025

Description

A small change, but i noticed https://github.com/klauspost/compress was already an indirect dependency of the repo, so this uses that version over the golang version as a drop in replacement. According to the repo it offers "better compression and concurrent streams".

Checklist

  • Code compiles correctly and linting passes locally

@RJKeevil RJKeevil requested a review from a team as a code owner February 20, 2025 12:48
@mangalaman93
Copy link
Member

Thank you for the PR @RJKeevil. I don't see a direct benefit of doing this. It doesn't change dependency tree much. Performance for snappy is not clear that it improves it significantly either. Am I missing something?

@RJKeevil
Copy link
Contributor Author

Yeh its a minor one, you had both dependencies anyway so I just thought it might be best to default to the one that claims to be faster and seems to receive more bug fixes and improvements. Where it becomes interesting is it would allow you to consider more compression modes in future, such as s2, which is a more significantly improved snappy.

@mangalaman93
Copy link
Member

For Badger, we already use s2 instead of snappy. I think we can do that here too https://github.com/hypermodeinc/badger/blob/main/table/builder.go#L151

For connections, not sure what the best option is. I wouldn't change that unless we have a solid reasoning for it.

@RJKeevil RJKeevil changed the title Switch to klauspost snappy Switch to s2 for bulk loader and backup/restore Feb 21, 2025
@RJKeevil
Copy link
Contributor Author

@mangalaman93 made those changes

@mangalaman93 mangalaman93 enabled auto-merge (squash) February 24, 2025 08:16
@mangalaman93
Copy link
Member

@RJKeevil could you rebase on latest main? We merged a fix so that the failing tests pass.

auto-merge was automatically disabled February 26, 2025 09:09

Head branch was pushed to by a user without write access

@mangalaman93 mangalaman93 merged commit f4c2345 into hypermodeinc:main Feb 26, 2025
23 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants