Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ShardFollowNodeTask fetch operations twice #32453

Closed
dnhatn opened this issue Jul 28, 2018 · 9 comments
Closed

ShardFollowNodeTask fetch operations twice #32453

dnhatn opened this issue Jul 28, 2018 · 9 comments
Assignees
@dnhatn
Labels
>bug :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features

Comments

@dnhatn
Copy link
Member

dnhatn commented Jul 28, 2018 •
edited
Loading

Since #31581, ShardFollowNodeTask may fetch some range twice. The following log indicates that we fetched the range [1680 to 2024] twice.

[ShardFollowNodeTask] fetch from=1620, to=1679, receive [1620 2024]
[ShardFollowNodeTask] fetch from=1680, to=1984, receive [1680 2084]

This bug and if the follower is not using the FollowingEngine (PR #32448) can explain why we have many deletes on the replicas of the follower (but not on the primaries of the follower).

[ShardFollowNodeTask] send shard request from=0, batch=405, max=-1, retry=0, gcp=-1, num_reads=1
[ShardFollowNodeTask] fetch from=0, to=-1, receive [0 404]
[ShardFollowNodeTask] send shard request from=405, batch=405, max=809, retry=0, gcp=1679, num_reads=1
[ShardFollowNodeTask] send shard request from=810, batch=405, max=1214, retry=0, gcp=1679, num_reads=2
[ShardFollowNodeTask] send shard request from=1215, batch=405, max=1619, retry=0, gcp=1679, num_reads=3
[ShardFollowNodeTask] send shard request from=1620, batch=405, max=1679, retry=0, gcp=1679, num_reads=4

[ShardFollowNodeTask] fetch from=810, to=1214, receive [810 1214]
[ShardFollowNodeTask] send shard request from=1680, batch=405, max=1984, retry=0, gcp=1984, num_reads=4
[ShardFollowNodeTask] fetch from=405, to=809, receive [405 809]
[ShardFollowNodeTask] fetch from=1215, to=1619, receive [1215 1619]
[ShardFollowNodeTask] fetch from=1620, to=1679, receive [1620 2024] (*)
[ShardFollowNodeTask] send shard request from=2025, batch=405, max=2322, retry=0, gcp=2322, num_reads=2
[ShardFollowNodeTask] fetch from=1680, to=1984, receive [1680 2084] (**)
@dnhatn dnhatn added >bug :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features labels Jul 28, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn
Copy link
Member Author

dnhatn commented Jul 28, 2018

/cc @martijnvg @jasontedor @bleskes

dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Jul 30, 2018
@dnhatn
ShardFollowNodeTask should fetch operation once
270934f 
Today ShardFollowNodeTask might fetch some operations more than once.
This happens because we ask the leading for up to max_batch_count
operations (instead of the left-over size) for the left-over request.
The leading then can freely respond up to the max_batch_count, and at
the same time, if one of the previous requests completed, we might issue
another read request whose range overlaps with the response of the
left-over request.

Closes elastic#32453
@dnhatn dnhatn mentioned this issue Jul 30, 2018
Merged
@bleskes
Copy link
Contributor

bleskes commented Jul 30, 2018 •
edited
Loading

Good catch @dnhatn. This feels like a bug to me? we should not get ops we didn't ask for?

[ShardFollowNodeTask] fetch from=1620, to=1679, receive [1620 2024] (*)

@bleskes
Copy link
Contributor

bleskes commented Jul 30, 2018

Also, I'm not sure I follow how this explains the deletes, can you clarify? if the primary processes the same ops twice, but ignores the associated seq# and issues it's own (and replicates it) how does it explain that the primary has no deletes but the replica does?

@dnhatn
Copy link
Member Author

dnhatn commented Jul 30, 2018

@bleskes The primary uses version numbers to resolve the indexing plan and rejects the duplicate operations; Whereas the replica uses seq# to resolve the indexing plan, and indexes the duplicate operations as stale documents. Does it sound reasonable to you?

@bleskes
Copy link
Contributor

bleskes commented Jul 30, 2018

Thx @dnhatn I forgot that we don't use the bulk shard request and that exceptions on the primary don't map to lack of replications. It does. Thanks!

1 similar comment
@bleskes
Copy link
Contributor

bleskes commented Jul 30, 2018

Thx @dnhatn I forgot that we don't use the bulk shard request and that exceptions on the primary don't map to lack of replications. It does. Thanks!

@dnhatn dnhatn self-assigned this Jul 30, 2018
dnhatn added a commit that referenced this issue Jul 31, 2018
@dnhatn
ShardFollowNodeTask should fetch operation once (#32455)
8cfbb64 
Today ShardFollowNodeTask might fetch some operations more than once.
This happens because we ask the leading for up to max_batch_count
operations (instead of the left-over size) for the left-over request.
The leading then can freely respond up to the max_batch_count, and at
the same time, if one of the previous requests completed, we might issue
another read request whose range overlaps with the response of the
left-over request.

Closes #32453
@dnhatn
Copy link
Member Author

dnhatn commented Jul 31, 2018

Fixed in #32455

@dnhatn dnhatn closed this as completed Jul 31, 2018
dnhatn added a commit that referenced this issue Aug 2, 2018
@dnhatn
ShardFollowNodeTask should fetch operation once (#32455)
473be53 
Today ShardFollowNodeTask might fetch some operations more than once.
This happens because we ask the leading for up to max_batch_count
operations (instead of the left-over size) for the left-over request.
The leading then can freely respond up to the max_batch_count, and at
the same time, if one of the previous requests completed, we might issue
another read request whose range overlaps with the response of the
left-over request.

Closes #32453
@martijnvg
Copy link
Member

@dnhatn Great catch! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
>bug :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@martijnvg @bleskes @dnhatn @elasticmachine