File structure

[ewels]

Suggested during discussion at the Stockholm hackathon about potential repository organisation:

.
├── .github
│   └── wokflows
│       └── test-processes.yml
├── README.md
├── nf-core
└── tools
    ├── bwa
    │   └── mem
    │       ├── main.nf
    │       ├── meta.yml
    │       └── test-action.yml
    ├── fastqc
    │   ├── main.nf
    │   ├── meta.yml
    │   └── test-action.yml
    └── samtools
        ├── index
        │   ├── main.nf
        │   ├── meta.yml
        │   └── test-action.yml
        └── sort
            ├── main.nf
            ├── meta.yml
            └── test-action.yml

• Have a directory for every tool
• Have subdirectories for every subcommand
• Have a yaml meta file with descriptions of the process
• .github/workflows/test-processes.yml will have a step for each process tool.
  • Each step can use path to only run when those files are changed (docs)
  • Each step can reference the test-action.yml file held in the process subdirectory with uses (docs)
  • Need to lint that .github/workflows/test-processes.yml has a step for every process

---

• QUESTION: commands that can be run in very different ways?
  • Should we have a different subdirectory for commands that can be run in a very different manner?
• QUESTION: What happens with variable numbers of inputs and outputs? cf. Test how variable numbers of inputs and outputs work #6

[drpatelh]

Yes, I think that file structure makes sense 👍 Maybe worth adding a command here that can be run in different ways so we can get an idea as to how we will need to deal with this. For most instances though, Im assuming the command will evolve based on updates provided by users of the module - backwards compatibility will have to be a must here.

[ewels]

backwards compatibility will have to be a must here.

I disagree a bit - git histories provide backwards compatibility! 😉

[drpatelh]

Ah, sorry what I meant by backwards compatibility here is that the command should still work as it did before you changed it e.g. If I add in an additional option to the command as part of a revision does it still work? More to do with the module tests we perform with actual data I guess.

Id imagine there will be instances where these changes arent "backward compatible" e.g. having to remove a parameter to be able to use another one in which case we would need a different module file.

[apeltzer]

If we release a pipeline / update it using submodules, this will refer to https://github.com/nf-core/modules@HASH with a specific set of tools/methods at that point in time, thus not breaking things. Once you update to a newer version of modules, you might experience differences (e.g. because certain modules were updated), but that is fine/should be fine I suppose?

The only thing you can't do with that approach is using different module revisions at one time, e.g. two different fastqc module revisions at once in a single pipeline, which you probably won't want to do anyways (or only very rarely?).

Phil's on an example, guess that we can test/clarify/set things there then easily and gain some extra experience quickly :-)

[ewels]

If I add in an additional option to the command as part of a revision does it still work?

Yes, this will break pipelines, but only when the pipelines update their hash reference to the modules repo (eg. before a release). So then the tests will fail and the pipeline can be fixed.

The previous stable versions of the pipeline will still refer to the previous hash before the change so should still be fine.

[drpatelh]

We will need to update this structure because tools used in this context can also cause confusion with nf-core/tools 😏 Should we replace tools with software? Also, given the preliminary pipeline template for DSL 2 as added here I dont think we will need to host any specific modules for the pipeline template on this repository so could maybe remove the nf-core folder too?

Newly proposed structure:

.
├── .github
│   └── wokflows
│       └── test-processes.yml
├── README.md
└── software
    ├── bwa
    │   └── mem
    │       ├── main.nf
    │       ├── meta.yml
    │       └── test-action.yml
    ├── fastqc
    │   ├── main.nf
    │   ├── meta.yml
    │   └── test-action.yml
    └── samtools
        ├── index
        │   ├── main.nf
        │   ├── meta.yml
        │   └── test-action.yml
        └── sort
            ├── main.nf
            ├── meta.yml
            └── test-action.yml

[ewels]

Agreed, software is nicer 👍

I think we should change to this structure ASAP, but maybe worth doing a PR mergeathon first to avoid all of the conflicts that it will cause...

[ewels]

Ok, I've been a complete merge-cowboy this morning and just bulldozed through a load of stuff so that we're vaguely ready for the hackathon next week.

I've restructured everything to fit the above proposed file hierarchy in #36