Update mpox tree builds to current snakemake workflow #1706
Conversation
| @@ -176,7 +179,11 @@ def dump_yaml_template( | |||
| session, phylo_run.pathogen, phylo_run.template_args, group | |||
| ) | |||
| builder: TemplateBuilder = TemplateBuilder( | |||
| phylo_run.tree_type, phylo_run.group, resolved_template_args, **context | |||
| phylo_run.tree_type, | |||
| phylo_run.pathogen, | |||
There was a problem hiding this comment.
This is a straight bugfix. The TemplateBuilder init args expect: tree_type, pathogen, group, template_args, **kwargs. Somehow we've been missing passing the pathogen to the TemplateBuilder. I'd think that would have pretty major ramifications though, so I'm surprised this didn't get noticed sooner.
There was a problem hiding this comment.
Whoops, I got myself a bit confused. This is still a straight bugfix, but it's fixing a seldom used develpment function dump_yaml_template that we can use for only generating the build config yaml. We hadn't used it for awhile and it never got updated while the TemplateBuilder usage where it mattered (later in this same file) did get updated. I forgot that context when I wrote the above comment. But that's why there were no major ramifications and it didn't get noticed: it almost never gets used, and I just wound up using it for this work since a lot of my work was around fixing up the yaml config.
There was a problem hiding this comment.
dump_yaml_template was intended for use during development. nice catch!
|
|
||
| # run snakemake, if run fails export the logs from snakemake to s3 | ||
| (cd /mpox && snakemake --printshellcmds --configfile config/build_czge.yaml --resources=mem_mb=312320) || { $aws s3 cp /mpox/.snakemake/log/ "${s3_prefix}/logs/snakemake/" --recursive ; $aws s3 cp /mpox/results/mpxv/filter.log "${s3_prefix}/logs/mpox/" --recursive ; } | ||
| (cd /mpox/phylogenetic && snakemake --printshellcmds --configfile build_czge.yaml --resources=mem_mb=312320) || { $aws s3 cp /mpox/phylogenetic/.snakemake/log/ "${s3_prefix}/logs/snakemake/" --recursive ; $aws s3 cp /mpox/phylogenetic/results/aspen/logs/ "${s3_prefix}/logs/mpox/" --recursive ; } |
There was a problem hiding this comment.
the log location changed with the subsampling update so I changed it accordingly
…#1706) * Fix yaml dump only to have expected args * WIP: Alter base MPX template for new mpox flow * WIP: Address notes from talking with Dan * WIP: Adjust `max_sequences` to match old MPX template * WIP: Keep `builds` in MPX output template * Remove no longer needed comments * Convert to being compatible with new mpox build * Modify paths to use latest mpox workflow format * Lint roller * Update run_nextstrain_mpx.sh * Fix test after mpox config format changes * Remove arguments in filter differentiation * Lint roller. Again. --------- Co-authored-by: Dan Lu <[email protected]>
Summary:
Updates our mpox workflow kick-off process to be compatible with the new changes present in
nextstrain/mpoxsnakemake workflow and our fork of it.Notes:
chanzuckerberg/monkeypoxwith the branchsubsampling. That was forked in 2022. We are now using chanzuckerberg/mpox and the branchsubsample_by_distance.build_czge.yamlconfig to conform to the required config structure for the above mpox workflow and to shuffle around files so they're where it expects them to be./mpoxinside the workflow repo. Now, running the tree build snakemake happens from inside/mpox/phylogenetic. This one change accounts for most of the shuffling around that had to occur.