Load only required corpora #555
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this commit Rally will only load the required corpora / document sets that are actually referenced by the tasks in the currently selected challenge. This saves race setup time by avoiding unneeded downloads and decompression of document sets. Closes elastic#553
danielmitterdorfer
added
enhancement
dliappis
approved these changes
Aug 24, 2018
esrally/track/track.py
Outdated
| @@ -214,6 +214,13 @@ def filter(self, source_format=None, target_indices=None): | |||
| filtered.append(d) | |||
| return DocumentCorpus(self.name, filtered) | |||
|
|
|||
| def union(self, other): | |||
| assert self.name == other.name, "Both document corpora must have the same name" | |||
There was a problem hiding this comment.
We've discussed this for other cases before, but would it be more consistent with existing behavior to throw an exception here?
There was a problem hiding this comment.
I can address this, sure.
danielmitterdorfer
added a commit
that referenced
this pull request
Aug 27, 2018
With this commit we load track plugins eagerly when preparing the track. We need to do this because the track loader needs to query all parameter sources for the corpora that are in use. Relates #555
danielmitterdorfer
added a commit
that referenced
this pull request
Sep 4, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this commit Rally will only load the required corpora / document
sets that are actually referenced by the tasks in the currently selected
challenge. This saves race setup time by avoiding unneeded downloads and
decompression of document sets.
Closes #553