Fix AdaptiveSelectionStats serialization bug #28718
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ywelsch
added
>bug
:Data Management/Stats
jasontedor
reviewed
Feb 18, 2018
| @@ -96,7 +96,8 @@ public SearchTransportService(Settings settings, TransportService transportServi | |||
| } | |||
|
|
|||
| public Map<String, Long> getClientConnections() { | |||
| return Collections.unmodifiableMap(clientConnections); | |||
| // create a (weakly consistent) snapshot of the map so that it does not get modified when serializing it for stats reporting | |||
| return Collections.unmodifiableMap(new HashMap<>(clientConnections)); | |||
There was a problem hiding this comment.
I think this is subject to the same race, concurrent modification during the copy? It won’t manifest as an inconsistent snapshot of size and number of entries when iterating for serialization, but instead a concurrent modification exception thrown during the copy?
There was a problem hiding this comment.
See https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ConcurrentHashMap.html
Similarly, Iterators, Spliterators and Enumerations return elements reflecting the state of the hash table at some point at or since the creation of the iterator/enumeration. They do not throw ConcurrentModificationException
There was a problem hiding this comment.
The problem is I did not realize the backing map was a concurrent hash map (the perils of reviewing on mobile).
ywelsch
added a commit
that referenced
this pull request
Feb 19, 2018
The AdaptiveSelectionStats object serializes the clientOutgoingConnections map that's concurrently updated in SearchTransportService. Serializing the map consists of first writing the size of the map and then serializing the entries. If the number of entries changes while the map is being serialized, the size and number of entries go out of sync. The deserialization routine expects those to be in sync though. Closes #28713
ywelsch
added a commit
that referenced
this pull request
Feb 19, 2018
The AdaptiveSelectionStats object serializes the clientOutgoingConnections map that's concurrently updated in SearchTransportService. Serializing the map consists of first writing the size of the map and then serializing the entries. If the number of entries changes while the map is being serialized, the size and number of entries go out of sync. The deserialization routine expects those to be in sync though. Closes #28713
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Feb 19, 2018
* master: Enable selecting adaptive selection stats Remove leftover mention of file-based scripts Fix threading issue on listener notification (elastic#28730) Revisit deletion policy after release the last snapshot (elastic#28627) Remove unused method Track deletes only in the tombstone map instead of maintaining as copy (elastic#27868) [Docs] Correct typo in README.textile (elastic#28716) Fix AdaptiveSelectionStats serialization bug (elastic#28718) TEST: Fix InternalEngine#testAcquireIndexCommit Add note on temporary directory for Windows service Added coming annotation and breaking changes link to release notes script Remove leftover PR link for previously disabled bwc tests Separate acquiring safe commit and last commit (elastic#28271) Fix BWC issue of the translog last modified age stats
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Feb 21, 2018
Today, we serialize collections prefixed by their lengths. If the serialized length is inconsistent with the number of objects in the collection then the serialization succeeds but any subsequent deserialization will (almost certainly) fail, reporting something unexpected at some later point in the stream. This can happen, for instance, because of a concurrent modification (e.g. elastic#28718) to the collection in between getting its length and iterating through its objects. This is a royal pain to debug, because the failure is reported a long way from where it actually occurs. See also e.g. elastic#25323. This change adds checks to StreamOutput to verify that it wrote as many objects as it said it would, in order to catch any similar problems in future closer to their sources.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
AdaptiveSelectionStatsobject serializes theclientOutgoingConnectionsmap that's concurrently updated inSearchTransportService. Serializing the map consists of first writing the size of the map and then serializing the entries. If the number of entries changes while the map is being serialized, the size and number of entries go out of sync. The deserialization routine expects those to be in sync though.Closes #28713