Optimise YAML config parsing #754
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…dictionary to YAML This greatly speeds up the initial configuration step since the expansion of glob expressions for the inputs into thousands of files creates very large YAML objects that strictyaml is slow to validate. The changes here ensure the validation step is only done once when writing the updated (expanded glob) file lists back to _yaml["inputs"].
Each creation of the 'inputs' dictionary in a PipeLine config object can take a very long time since safeyaml is extremely slow to validate long file lists when converting to a python list. The PipelineConfig.validation method recreates the ['inputs'] dictionary many times. Just do it once at the start then use that dictionary throughout.
Just because I was changing things in this method anyway.
ddobie
approved these changes
Sep 25, 2024
|
Approved - just FYI we usually use a "Squash and merge" when merging PRs rather than a merge commit or a rebase |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Optimise read of YAML configuration at the start of the pipeline by copying the configuration into a dictionary before reading or updating it. This prevents
strictyamlfrom validating the YAML on every read or write and only lets the validation happen on bulk reads or writes.This closes #751