Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Archives more robustly #8135

Merged
merged 17 commits into from
Jul 25, 2024
Merged

Build Archives more robustly #8135

merged 17 commits into from
Jul 25, 2024

Conversation

macneale4
Copy link
Contributor

@macneale4 macneale4 commented Jul 22, 2024

This change is long in coming. Several findings from building a large user database are incorporated here. Specifically:

  • No longer storing all chunks in memory
  • Cancellable process with ^C
  • Progress reporting for major stages of the build
  • No chunk grouping by default, --group-chunks to enable. Pathological cgo dictionary building problem needs to be fixed before we enable by default.
  • Writing a useful metadata json block with dolt version and origin table file ID.

@macneale4 macneale4 requested a review from reltuk July 22, 2024 16:47
@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
6ee2664 ok 5937457
version total_tests
6ee2664 5937457
correctness_percentage
100.0

@macneale4 macneale4 marked this pull request as draft July 22, 2024 18:45
@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
8f0f55a ok 5937457
version total_tests
8f0f55a 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000
version result total
672c882 ok 5937457
version total_tests
672c882 5937457
correctness_percentage
100.0

@macneale4 macneale4 marked this pull request as ready for review July 23, 2024 03:05
@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
a001e1e ok 5937457
version total_tests
a001e1e 5937457
correctness_percentage
100.0

Copy link
Contributor

@fulghum fulghum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! The support for canceling and the progress output is really, really nice. I only had a couple of really minor thoughts.

@coffeegoddd
Copy link
Contributor

@macneale4 DOLT

comparing_percentages
100.000000 to 100.000000
version result total
23ea4fd ok 5937457
version total_tests
23ea4fd 5937457
correctness_percentage
100.0

@macneale4 macneale4 merged commit 93dc3a6 into main Jul 25, 2024
21 checks passed
@macneale4 macneale4 deleted the macneale4/archive-build branch July 25, 2024 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants