Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Translog corruption marker #33415

Closed

Conversation

vladimirdolzhenko
Copy link
Contributor

@vladimirdolzhenko vladimirdolzhenko commented Sep 5, 2018

Spin-off of #28434:

Today if Lucene index is not good, we mark its store as corrupted by storing a corruption marker. With the presence of the corruption marker, a corrupted copy won't be used as a primary and requires a full file-based recovery to be used a replica. This restriction should also apply to a copy with corrupted translog. If a copy with corrupted translog is mistakenly selected as a primary, it may not be able to provide all translog operations properly for replicas in the future.
This commit marks the store of a shard copy as corrupted if its translog is not good.

  • on a contrast to Mark store as corrupted if its translog is bad #28434 translog corruption marker is stored in translog folder
  • when we have translog corruption marker elasticsearch-shard remove-corrupted-data could be run without specifying any parameters (like index name and shard id) - it can scan all folders in data paths for the corruption markers

Relates to #31389

@vladimirdolzhenko vladimirdolzhenko added >enhancement WIP :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. v7.0.0 labels Sep 5, 2018
@vladimirdolzhenko vladimirdolzhenko self-assigned this Sep 5, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@vladimirdolzhenko vladimirdolzhenko changed the base branch from master to fix/31389_2 September 5, 2018 09:58
@vladimirdolzhenko vladimirdolzhenko force-pushed the tl_corr_marker branch 4 times, most recently from 0ff923b to 933164e Compare September 12, 2018 06:16
@vladimirdolzhenko vladimirdolzhenko force-pushed the tl_corr_marker branch 3 times, most recently from d0f102f to f89c325 Compare September 14, 2018 11:25
@vladimirdolzhenko vladimirdolzhenko changed the title [WIP] Translog corruption marker Translog corruption marker Sep 17, 2018
@vladimirdolzhenko vladimirdolzhenko changed the base branch from fix/31389_2 to master September 19, 2018 09:33
@vladimirdolzhenko vladimirdolzhenko requested review from ywelsch and removed request for DaveCTurner October 15, 2018 12:13
@ywelsch ywelsch closed this Jan 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >enhancement v7.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants