Test how variable numbers of inputs and outputs work

[ewels]

Need to look in to how Nextflow DSL2 handles variable numbers of inputs or outputs.

For example - TrimGalore! can optionally save untrimmed reads. If that is enabled, we will have an additional output channel. How do pipelines handle this?

[drpatelh]

Yeah, this is going to be tricky. Pipelines will want to use the module command in different ways so we will have to try and anticipate different numbers/types of inputs and outputs. Should this be done on a case-by-case basis? e.g. add the option for a another input/output whilst maintaining backward compatability?

With DSL1 the way I have provided conditional inputs is:
https://github.com/nf-core/chipseq/blob/21be3149542cdc84431e12d1e092359058aed32a/main.nf#L424

and conditional outputs as:
https://github.com/nf-core/chipseq/blob/21be3149542cdc84431e12d1e092359058aed32a/main.nf#L1087

Note all of the files dont have to be created as an output here and so NF wont fail if they arent. Maybe the solution is to have all outputs from a module in a single channel and then filter as required but this could get messy...

[ewels]

I think DSLv2 can have named outputs, right? So I'm hoping that if pipelines specify the output channels that they are using, they can ignore the ones that they don't need. Then we can just allow those channels to be empty and I think everything should be fine... 🤞

The conditional input ifEmpty() thing shouldn't be necessary... I think... because with modules you don't specify the input channel name. Not sure though.

I think the one channel for everything isn't necessary if we use optional outputs and named outputs in the workflows.

Make sense? Bit scatterbrained today sorry - say if the above is jumbled jibberish 😆

[drpatelh]

No. That makes sense. Like most things "modules" this will need a play around to work out how best to proceed. I need to fix up and look sharp with DSL2 too. Maybe we can try with a simple TrimGalore implementation?

[apeltzer]

Yes I agree - we should really get some "basic" tests for this and play around a bit to get a feeling for how this behaves. During the discussion we all kind of agreed its got lots of potential for problems arising, which is why Phil created the issue here 👍

[drpatelh]

We have a requirement to emit every output file which solves this issue. Where required, we are using the optional declaration for optional outputs.

We still don't have a proper solution for optional inputs but this will have to be announced officially on the Nextflow end