You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's non-trivial to 'dry-run' a pipeline. The main.py --dry-run functionality is very limited, and just prints the stages in the workflow, +/- some first/last/skipping through config
What I really want to test locally is the code in queue_jobs and expected_outputs, as well as the interconnection between stages. This requires me to run the workflow locally, without the --dry_run parameter (so that the Stage code, expected_outputs, queue_jobs actually runs), but with dry run communicated directly to Hail here through config, to prevent launching of the assembled jobs. To do this (AFAIK) I need to run the pipeline normally (without dry-run), and with a substitute config block like this:
It would be nice if this was a behaviour I could trigger with just a CLI setting.
Another glitch here is that even providing these config settings, some methods write directly to GCP during workflow setup (e.g. here or here - the latter responds to workflow.dry_run in config, which is a third different place we can add dry_run, and affects a different subset of behaviours.)
The text was updated successfully, but these errors were encountered:
I feel like this is a cpg-flow User Experience request, rather than a Prod Pipes request... Maybe consolidating down to a single consistent dry_run value, and a way to decorate methods with @inactivate_on_dry_run if they run actual writing during workflow setup?
It's non-trivial to 'dry-run' a pipeline. The
main.py --dry-run
functionality is very limited, and just prints the stages in the workflow, +/- some first/last/skipping through configWhat I really want to test locally is the code in queue_jobs and expected_outputs, as well as the interconnection between stages. This requires me to run the workflow locally, without the
--dry_run
parameter (so that the Stage code, expected_outputs, queue_jobs actually runs), but with dry run communicated directly to Hail here through config, to prevent launching of the assembled jobs. To do this (AFAIK) I need to run the pipeline normally (without dry-run), and with a substitute config block like this:It would be nice if this was a behaviour I could trigger with just a CLI setting.
Another glitch here is that even providing these config settings, some methods write directly to GCP during workflow setup (e.g. here or here - the latter responds to
workflow.dry_run
in config, which is a third different place we can adddry_run
, and affects a different subset of behaviours.)The text was updated successfully, but these errors were encountered: