Fixes for PR #238 #243

drpatelh · 2024-01-04T18:05:33Z

Refactor code introduced in #238 to use collectFile for merging samplesheets.

github-actions · 2024-01-04T18:07:03Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 8fc22ec

+| ✅ 131 tests passed       |+
#| ❔  21 tests were ignored |#
!| ❗   6 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 1.11.0
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in meta.yml: #Add a description of the subworkflow and list keywords
pipeline_todos - TODO string in meta.yml: #Add a list of the modules and/or subworkflows used in the subworkflow
pipeline_todos - TODO string in meta.yml: #List all of the channels used as input with a description and their structure
pipeline_todos - TODO string in meta.yml: #List all of the channels used as output with a descriptions and their structure

❔ Tests ignored:

files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: assets/multiqc_config.yml
files_exist - File is ignored: conf/igenomes.config
files_exist - File is ignored: conf/modules.config
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowFetchngs.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/nfcore_external_java_deps.jar
files_unchanged - File ignored due to lint config: .gitattributes
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: lib/nfcore_external_java_deps.jar
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_ci - actions_ci
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/fetchngs/fetchngs/.github/workflows/awstest.yml
multiqc_config - 'assets/multiqc_config.yml' not found

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-fetchngs_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-fetchngs_logo_light.png
files_exist - File found: docs/images/nf-core-fetchngs_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-fetchngs_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/nf-core-fetchngs_logo_light.png matches the template
files_unchanged - docs/images/nf-core-fetchngs_logo_light.png matches the template
files_unchanged - docs/images/nf-core-fetchngs_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (157 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: cloud_tests_small.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: release-announcments.yml
actions_schema_validation - Workflow validation passed: cloud_tests_full.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.11.1
Run at 2024-01-07 23:07:57

…into fixes

nextflow.config

drpatelh · 2024-01-05T17:20:08Z

nf-test.config

@@ -3,7 +3,7 @@ config {
    testsDir "."

    // nf-test directory including temporary files for each test
-    workDir "/tmp"
+    workDir "nf-test"


Revert to /tmp. Can't use .nf-test due to bug with fasterq-dump not being able to output files with .

I've been battling with @adamrtalbot on this.
But we finally agreed and came up with:
https://github.com/nf-core/modules/blob/master/.github/workflows/test.yml#L620
https://github.com/nf-core/modules/blob/master/nf-test.config#L6

tests/nextflow.config

drpatelh · 2024-01-07T11:27:22Z

Tests are failing due to askimed/nf-test#173

Think we need to remove using snapshots until that is fixed and add explicit tests for the output files.

.github/workflows/ci.yml

maxulysse · 2024-01-08T07:03:45Z

Looking nice to me, just think we need to use the env variable for the default location of the nf-test workdir as we do in modules

drpatelh added 3 commits January 4, 2024 18:02

Use double quotes for closure around projectDir

47a5491

Remove test data for sra_merge_samplesheet process

7afee76

Fix bugs with samplesheet merging from PR nf-core#238

209a7a4

drpatelh added 2 commits January 4, 2024 18:07

Update CHANGELOG

ec8585c

Fix nf-core lint until template updates are merged in

da88dc6

drpatelh mentioned this pull request Jan 4, 2024

Pipeline fails for large studies #236

Closed

drpatelh and others added 13 commits January 4, 2024 18:12

Fix prettier

05b993e

Fix prettier

8fc57d9

Fix nf-test for sra_skip_fastq_download

fb07b32

Change workDir to .nf-test

7e5687a

Fix Black lint

0193bce

Merge branch 'dev' into fixes

d9103e6

Merge branch 'fixes' of https://github.com/drpatelh/nf-core-fetchngs …

1ccc05c

…into fixes

Update snapshots and remove dot from nf-test workDir

06cfcc8

Use @adamrtalbot Docker runOptions fix to get nf-test running

1544d3f

Bump nf-test version to 0.8.2 in CI tests

e94ed03

Test adding docker.runOptions to pipeline nextflow.config

0b738ed

Idiot...add docker in front of runOptions

0a89ccb

Add sort logic to collectFile operator

1f70b27

adamrtalbot mentioned this pull request Jan 5, 2024

Fix nf-test testing directory and add common tag #244

Closed

10 tasks

drpatelh commented Jan 5, 2024

View reviewed changes

nextflow.config Outdated Show resolved Hide resolved

drpatelh commented Jan 5, 2024

View reviewed changes

drpatelh mentioned this pull request Jan 5, 2024

Issues with PATH when Docker is enabled askimed/nf-test#149

Closed

drpatelh added 3 commits January 5, 2024 17:48

Update all workflow snapshots and use outputDir by default

12597f1

Use icdiff tool to show nf-test snapshot diff in Github Actions output

dd67624

Fix prettier

b8d43be

Midnighter reviewed Jan 6, 2024

View reviewed changes

tests/nextflow.config Outdated Show resolved Hide resolved

drpatelh added 2 commits January 6, 2024 20:03

Bump nf-test version to 0.8.3 and remove Docker home mounts

300cb2e

Add ch_ prefix to fastq_files channel

db9d949

drpatelh added 3 commits January 6, 2024 20:34

Use --verbose parameter by default with nf-test

44aa79c

Use relative path for --outdir because outputDir will be unique path

584fb2d

Update snapshots

18a7c27

drpatelh added 9 commits January 7, 2024 14:13

Update snapshots and replace with explicit tests

95b30a6

Fix snapshot for sra_skip_fastq_download

7cc221d

Replace results with outputDir

f08543d

Reorder samples in first column of samplesheet and mappings

b0d4905

Regenerate snapshots and remove metadata test

b7cfca7

Remove all workflow snapshots

f705af8

Tidy up tests and configuration

6a0120d

Use smaller test data to speed up test runtimes

f6510d5

Update pipeline test to load test.config directly

8fc22ec

maxulysse reviewed Jan 8, 2024

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

maxulysse approved these changes Jan 8, 2024

View reviewed changes

drpatelh merged commit addb060 into nf-core:dev Jan 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for PR #238 #243

Fixes for PR #238 #243

drpatelh commented Jan 4, 2024

github-actions bot commented Jan 4, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

drpatelh Jan 5, 2024

maxulysse Jan 8, 2024

drpatelh commented Jan 7, 2024

maxulysse commented Jan 8, 2024

Fixes for PR #238 #243

Fixes for PR #238 #243

Conversation

drpatelh commented Jan 4, 2024

github-actions bot commented Jan 4, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

drpatelh Jan 5, 2024

Choose a reason for hiding this comment

maxulysse Jan 8, 2024

Choose a reason for hiding this comment

drpatelh commented Jan 7, 2024

maxulysse commented Jan 8, 2024

github-actions bot commented Jan 4, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️