Fix performance regression by parallelizing alias checkout #1101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
simu
force-pushed
the
feat/parallel-alias-setup
branch
from
3ebe0c5 to
3f0e27a
Compare
simu
changed the base branch from
master
to
fix/component-dependency-nonoptional
simu
changed the title
Base automatically changed from
fix/component-dependency-nonoptional
to
master
February 14, 2025 11:50
simu
force-pushed
the
feat/parallel-alias-setup
branch
from
3f0e27a to
0f9b03b
Compare
This change will allow us to parallelize `alias_checkout()`.
simu
force-pushed
the
feat/parallel-alias-setup
branch
2 times, most recently
from
08bfdcb to
46f96a4
Compare
bastjan
approved these changes
Feb 14, 2025
We collect aliases and their backing components and use the same ThreadPoolExecutor approach that we use for the component checkouts. We aggressively parallelize the alias checkouts for aliases which use a dependency URL that we've already cloned during the component checkout phase, and group aliases with the same not-yet-cloned dependency URL to avoid race conditions. We rename the wrapper function which does the parallel work to `do_parallel()` since it doesn't necessarily fetch anything anymore. Additionally we add type annotations to `do_parallel()` to make it more obvious how the arguments are supposed to look like.
We simply assert that `apath` is not None to signal to mypy that there's no possible way that the alias isn't registered on the alias multi-dependency (and therefore `adep.get_component(alias) will never return None) when the alias is present in `_aliases`.
simu
force-pushed
the
feat/parallel-alias-setup
branch
from
46f96a4 to
4d5f0fc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is necessary now that setting up aliases requires Git checkout and potentially even Git clone operations. This reduces the "Fetching components..." stage for a cluster with 60 component instances from ~12s to ~4s on my system (clones unchanged at ~3s, alias setup from ~9s to ~1s).
We change
fetch_componentsto collect aliases and their backing components and use the sameThreadPoolExecutorapproach that we use for the component checkouts.We aggressively parallelize the alias checkouts for aliases which use a dependency URL that we've already cloned during the component checkout phase, and group aliases with the same not-yet-cloned dependency URL to avoid race conditions.
To make this work, we need to refactor the Component alias handling a bit, so that we have enough information to do the actual
alias_checkout()in a parallelized loop.Checklist
bug,enhancement,documentation,change,breaking,dependency,internalas they show up in the changelog