Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

index.routing.allocation.initial_recovery._id split after shrink #43955

Closed
laf-rge opened this issue Jul 4, 2019 · 6 comments · Fixed by #44053
Closed

index.routing.allocation.initial_recovery._id split after shrink #43955

laf-rge opened this issue Jul 4, 2019 · 6 comments · Fixed by #44053
Assignees
@DaveCTurner
Labels
>bug :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes)

Comments

@laf-rge
Copy link

laf-rge commented Jul 4, 2019

Elasticsearch version (bin/elasticsearch --version): 7.1.1

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

Steps to reproduce:

  1. Create an index
  2. Shrink an index
  3. remove the node the shrink occurred on from the cluster
  4. Wait for index to recover
  5. attempt to split the index

Provide logs (if relevant):
Split fails on allocation. Decider says all primary shards must be on the dead node. Can not remove index.routing.allocation.initial_recovery._id setting from index.
@laf-rge
Copy link
Author

laf-rge commented Jul 4, 2019

Similar to #31787

@laf-rge
Copy link
Author

laf-rge commented Jul 4, 2019

You might be asking yourself: why would you split a node after a shrink? Short answer: bad ILM policy.

@henningandersen henningandersen added the :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Jul 4, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@DaveCTurner DaveCTurner self-assigned this Jul 6, 2019
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Jul 8, 2019
@DaveCTurner
Do not copy initial recovery filter during split
f546cca 
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes elastic#43955
@DaveCTurner DaveCTurner mentioned this issue Jul 8, 2019
Merged
@DaveCTurner
Copy link
Contributor

Thanks for the report @laf-rge. Splitting an index after a shrink is a legitimate thing to do so this does look wrong. I opened #44053 to address this.

pull bot pushed a commit to Pandinosaurus/elasticsearch that referenced this issue Jul 8, 2019
@DaveCTurner
Do not copy initial recovery filter during split (elastic#44053)
18d4e9d 
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes elastic#43955
DaveCTurner added a commit that referenced this issue Jul 8, 2019
@DaveCTurner
Do not copy initial recovery filter during split (#44053)
3129f5b 
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes #43955
@laf-rge
Copy link
Author

laf-rge commented Jul 8, 2019 •
edited
Loading

This is great. Is there a fix for the current version? Will the updated version also fix this?
edit I meant workaround.

@DaveCTurner
Copy link
Contributor

One possible workaround is to use reindex instead of simply splitting the index. This will take longer, unfortunately, but seems likely to work. Another idea, assuming you haven't already shrunk this index down to a single shard, is to shrink it again (which will overwrite the index.routing.allocation.initial_recovery._id setting) and then split it from there.

DaveCTurner added a commit that referenced this issue Jul 9, 2019
@DaveCTurner
Do not copy initial recovery filter during split (#44053)
f3620c6 
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes #43955
DaveCTurner added a commit that referenced this issue Jul 9, 2019
@DaveCTurner
Do not copy initial recovery filter during split (#44053)
f1b73fe 
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes #43955
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
>bug :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Do not copy initial recovery filter during split DaveCTurner/elasticsearch
4 participants
@laf-rge @DaveCTurner @elasticmachine @henningandersen