Track local checkpoint on primary immediately #25434
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today after a primary shard is recovered or promoted from a replica, it will not be tracking its own local checkpoint until the first indexing operation on the shard. This leads to situations where a shard is relocated and the knowledge it sends in the primary context about its own local checkpoint violates the global checkpoint. This commit rectifies this situation by tracking the local checkpoint on such shards immediately on recovery/promotion/relocation.
|
@ywelsch I'm opening this for discussion. |
9 tasks
* master: Do not swallow exception when relocating Docs: Fix typo for request cache (elastic#25444) Remove implicit 32-bit support [DOCS] reworded to prevent code span rendering glitch (elastic#25442) Disallow multiple concurrent recovery attempts for same target shard (elastic#25428) Update global checkpoint when increasing primary term on replica (elastic#25422) Add backwards compatibility indices for 5.4.3 Add version 5.4.3 after release Update MSI installer images (elastic#25414) Add missing newline at end of SetsTests.java Rename handoff primary context transport handler correct expected thrown exception in mappingMetaData to ElasticsearchParseException (elastic#25410) test: Make many percolator integration tests real integration tests [DOCS] Update docs to use shared attribute file (elastic#25403) Add Javadocs and tests for set difference methods Tests: Add parsing test for AggregationsTests (elastic#25396) test: get upgrade status for all indices Mute SignificantTermsAggregatorTests#testSignificance()
…cal-checkpoint * enhance/single-updateshardstate-method: Some cleanup Do not swallow exception when relocating Docs: Fix typo for request cache (elastic#25444) Remove implicit 32-bit support [DOCS] reworded to prevent code span rendering glitch (elastic#25442) Disallow multiple concurrent recovery attempts for same target shard (elastic#25428) Update global checkpoint when increasing primary term on replica (elastic#25422) Add backwards compatibility indices for 5.4.3 Add version 5.4.3 after release Update MSI installer images (elastic#25414) Add missing newline at end of SetsTests.java fix test Rename handoff primary context transport handler Provide single IndexShard method to update state on incoming cluster state
* master: Use a single method to update shard state
ywelsch
suggested changes
Jun 28, 2017
| @@ -360,6 +365,7 @@ synchronized void updateAllocationIdsFromPrimaryContext(final PrimaryContext pri | |||
| .allMatch(e -> e.value == SequenceNumbersService.UNASSIGNED_SEQ_NO) : inSyncLocalCheckpoints; | |||
| assert StreamSupport | |||
| .stream(trackingLocalCheckpoints.spliterator(), false) | |||
| .filter(e -> !e.key.equals(allocationId)) // during primary relocation a shard can already know its local checkpoint | |||
There was a problem hiding this comment.
I think we should not filter out the allocation id, but keep the method as before and instead assert in the IndexShard.updateAllocationIdsFromPrimaryContext method that the local checkpoint that is provided as part of the primary context is equal to the local checkpoint that exists locally.
| final SequenceNumbersService seqNoService = engine.seqNoService(); | ||
| seqNoService.updateAllocationIdsFromMaster(applyingClusterStateVersion, activeAllocationIds, initializingAllocationIds); | ||
| if ((currentState == IndexShardState.POST_RECOVERY && state == IndexShardState.STARTED) || | ||
| recoveryState.getRecoverySource().getType().equals(RecoverySource.Type.PEER)) { |
There was a problem hiding this comment.
With the change I suggested above we can leave out the == PEER condition and instead add && currentRouting.isRelocationTarget == false to currentState == IndexShardState.POST_RECOVERY && state == IndexShardState.STARTED
| * Tests that a primary shard tracks its own local checkpoint after starting. | ||
| */ | ||
| @ESIntegTestCase.ClusterScope(scope = Scope.TEST, numDataNodes = 0) | ||
| public class LocalCheckpointIT extends ESIntegTestCase { |
There was a problem hiding this comment.
I wonder if we need an integration test for this. A test under IndexShardTests would be able to check the same.
There was a problem hiding this comment.
Yeah, now that we are not doing this via indices cluster state service, this is now possible. I will update.
|
This will be superseded by a forthcoming PR. |
clintongormley
added
:Distributed Indexing/Engine
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today after a primary shard is recovered or promoted from a replica, it will not be tracking its own local checkpoint until the first indexing operation on the shard. This leads to situations where a shard is relocated and the knowledge it sends in the primary context about its own local checkpoint violates the global checkpoint. This commit rectifies this situation by tracking the local checkpoint on such shards immediately on recovery/promotion/relocation.
Closes #25415, relates #10708, relates #25355