Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 4271 Replace InMemory statestore with transaction log #4272

Merged
merged 51 commits into from
Feb 18, 2025
Merged

Issue 4271 Replace InMemory statestore with transaction log #4272

merged 51 commits into from
Feb 18, 2025

Conversation

rtjd6554
Copy link
Collaborator

@rtjd6554 rtjd6554 commented Feb 17, 2025 •
edited by patchwork01
Loading

Make sure you have checked all steps below.

Issue

Tests

  • My PR adds the following tests OR does not need testing for this extremely good reason:
    • Tests present as part of the build and refactor

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
    separate issue for that below.
  • If I have added or removed any dependencies from the project, I have updated the NOTICES file.

rtjd6554 and others added 30 commits February 17, 2025 10:55
@rtjd6554
New class and name refactor for state store
97c0aa3
@patchwork01
Adjust Javadoc on InMemoryTransactionLogStateStore
95454a7
@patchwork01
Add InMemoryTransactionLogStateStore.createProvider
17c3e17
@patchwork01
Use new helper in BulkExportQuerySplitterTest
a01279c
@patchwork01
Use new test helpers in bulk import unit tests
7cf6e35
@rtjd6554
Update QueryClientIT refactor statestore usage
e97f6fd
@rtjd6554
Updated missing javadoc
4eac53f
@rtjd6554
Update usages of the InMemoryTransactionLogStore
4001ea8
@patchwork01
Revert formatting in BulkImportExecutorTest
141ca9f
@patchwork01
Use new helper in PartitionsStatusReportTest
cdf3ac2
@patchwork01
Remove unused helpers
2a3cccb
@rtjd6554
Update to refactor out the StateStoreTestBuilder
764a1f4
@rtjd6554
Update instance properties usage
18ffa4d
@rtjd6554
Update to PartitionsStatusReportScreenTest
3003bf1
@rtjd6554
Refactor instance and table variables
3d4ce4f
@patchwork01
Use new helper in TableMetricsTest
1453a70
@patchwork01
Use new test helpers in PartitionsStatusTest
47c3a02
@rtjd6554
Refactor WaitForPartitionSplittingTest
3661338
@rtjd6554
Update PartitionStatusReportIT
a58c5df
@rtjd6554
Delete test helpers no longer required
c7d0c6f
@rtjd6554
Update noProvider method usages
ec0557a
@rtjd6554
Update RustCompaction
aae9dba
@rtjd6554
Remove unused dependancy
908a351
@rtjd6554
Update InMemoryStateStore base
0870f81
@patchwork01
Merge branch 'develop' into 4271-Replace-InMemory-State-Store-With-Tr…
3c071e4 
…ansaction-Log
@patchwork01
Use new provider in InitialiseStateStoreFromSplitPointsTest
8d2e93f
@patchwork01
Use new helper in CreateCompactionJobsTest
f671c64
@rtjd6554
Update to the CompactionJobDispatcherTest
1e86dee
@rtjd6554
Update to CompactioTaskTestBase
7377411
@rtjd6554
New createMethod and adjust splitFileReferences
511cfa2
rtjd6554 and others added 21 commits February 18, 2025 10:57
@rtjd6554
Update assertions and InMemoryStateStore usage
ed54ea8
@rtjd6554
Update GarbageCollectorIT usages of InMemory
2f94c8e
@rtjd6554
Update for GarbageCollectorS3IT
547bcf5
@rtjd6554
Update for IngestRecordsFromIteratorIT
299e43f
@patchwork01
Use new helper in IngestRecordsIT
ad7fbeb
@patchwork01
Use new helper in ingest tests
7fc7185
@patchwork01
Simplify IngestRecordsFromIteratorIT
b1719d0
@patchwork01
Use new helper in backed by Arrow ingest tests
e39e90c
@patchwork01
Simplify IngestCoordinatorCommonIT
827cb98
@patchwork01
Simplify IngestCoordinatorFileWritingStrategyIT
1ee6a96
@patchwork01
Use new helper in IngestJobRunnerIT
1147723
@patchwork01
Use new helper in query tests
456c1c7
@patchwork01
Use new helper in FindPartitionsToSplitTest
83163ca
@patchwork01
Add TableProperties in SplitPartitionTest
ac674df
@patchwork01
Use new helper in SplitPartitionTest
bc2360a
@patchwork01
Use new helper in FindPartitionsToSplitIT
34c4fbe
@patchwork01
Use new helper in StateStoreWaitForFilesTest
168e147
@patchwork01
Delete old in-memory state store implementation
c5d39ea
@patchwork01
Move in memory transaction log code to testutils package
31bba7a
@patchwork01
Rename test package
98e09fc
@patchwork01
Move transaction tests to transaction package
bcd6b34
@rtjd6554 rtjd6554 marked this pull request as ready for review February 18, 2025 14:27
@rtjd6554 rtjd6554 added the pr-base-for-stacking Base for stacked pull requests (a dependency for others, where this PR's branch will be the base) label Feb 18, 2025
@rtjd6554 rtjd6554 merged commit 309162e into develop Feb 18, 2025
9 checks passed
@rtjd6554 rtjd6554 deleted the 4271-Replace-InMemory-State-Store-With-Transaction-Log branch February 18, 2025 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patchwork01 patchwork01 patchwork01 approved these changes

Assignees
No one assigned
Labels
one-review-required pr-base-for-stacking Base for stacked pull requests (a dependency for others, where this PR's branch will be the base)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Replace simple in memory state store implementation with in memory transaction log
2 participants
@rtjd6554 @patchwork01