Skip to content

POC: topics #9159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
+289 −175
Draft

POC: topics #9159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8d95f7c
POC topic versions
nvnieuwk Sep 30, 2025
6f53cbb
update test snaps
nvnieuwk Sep 30, 2025
48455e8
update meta.yml
nvnieuwk Sep 30, 2025
869d805
add topics to meta yaml
nvnieuwk Oct 1, 2025
0a6c97b
update meta yml
nvnieuwk Oct 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 1 addition & 11 deletions modules/nf-core/bwamem2/index/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ process BWAMEM2_INDEX {

output:
tuple val(meta), path("bwamem2"), emit: index
path "versions.yml" , emit: versions
tuple val("${task.process}"), val('bwa-mem2'), eval("bwa-mem2 version 2>&1 | tail -1"), topic: versions, emit: versions_bwamem2

when:
task.ext.when == null || task.ext.when
Expand All @@ -29,11 +29,6 @@ process BWAMEM2_INDEX {
$args \\
-p bwamem2/${prefix} \\
$fasta

cat <<-END_VERSIONS > versions.yml
"${task.process}":
bwamem2: \$(echo \$(bwa-mem2 version 2>&1) | sed 's/.* //')
END_VERSIONS
"""

stub:
Expand All @@ -46,10 +41,5 @@ process BWAMEM2_INDEX {
touch bwamem2/${prefix}.pac
touch bwamem2/${prefix}.amb
touch bwamem2/${prefix}.bwt.2bit.64

cat <<-END_VERSIONS > versions.yml
"${task.process}":
bwamem2: \$(echo \$(bwa-mem2 version 2>&1) | sed 's/.* //')
END_VERSIONS
"""
}
28 changes: 21 additions & 7 deletions modules/nf-core/bwamem2/index/meta.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,27 @@ output:
pattern: "*.{0123,amb,ann,bwt.2bit.64,pac}"
ontologies:
- edam: "http://edamontology.org/data_3210" # Genome index
versions:
- versions.yml:
type: file
description: File containing software versions
pattern: "versions.yml"
ontologies:
- edam: http://edamontology.org/format_3750 # YAML
versions_bwamem2:
- - ${task.process}:
type: string
description: The process the versions were collected from
- bwa-mem2:
type: string
description: The tool name
- bwa-mem2 version 2>&1 | tail -1:
type: string
description: The command used to generate the version of the tool
topics:
- versions:
- - process:
type: string
description: The process the versions were collected from
- tool:
type: string
description: The tool name
- version:
type: string
description: The version of the tool
Comment on lines +54 to +62
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good!
but if we want to automate it somehow, the values of the topic channel should be the same that we add for versions1: ${task.process}, bwa-mem2 and bwa-mem2 version 2>&1 | tail -1.
It doesn't look as nice, but it's the only way we can automate the generation of the meta.yml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks the main issue here is that the versions topic can consist of multiple tools here. I'm also not sure this needs automation as this will be in the template anyways. I've tried adding the automation, but it's not that trivial to do :/

authors:
- "@maxulysse"
maintainers:
Expand Down
12 changes: 2 additions & 10 deletions modules/nf-core/bwamem2/index/tests/main.nf.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,7 @@ nextflow_process {
then {
assertAll(
{ assert process.success },
{ assert snapshot(
process.out.index,
process.out.versions,
path(process.out.versions[0]).yaml
).match() }
{ assert snapshot(process.out).match() }
)
}
}
Expand All @@ -51,11 +47,7 @@ nextflow_process {
then {
assertAll(
{ assert process.success },
{ assert snapshot(
process.out.index,
process.out.versions,
path(process.out.versions[0]).yaml
).match() }
{ assert snapshot(process.out).match() }
)
}
}
Expand Down
120 changes: 82 additions & 38 deletions modules/nf-core/bwamem2/index/tests/main.nf.test.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,64 +1,108 @@
{
"fasta - stub": {
"content": [
[
[
{
"id": "test"
},
{
"0": [
[
{
"id": "test"
},
[
"genome.fasta.0123:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.amb:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.ann:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.bwt.2bit.64:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.pac:md5,d41d8cd98f00b204e9800998ecf8427e"
]
]
],
"1": [
[
"BWAMEM2_INDEX",
"bwa-mem2",
"2.2.1"
]
],
"index": [
[
{
"id": "test"
},
[
"genome.fasta.0123:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.amb:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.ann:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.bwt.2bit.64:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.pac:md5,d41d8cd98f00b204e9800998ecf8427e"
]
]
],
"versions1": [
[
"genome.fasta.0123:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.amb:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.ann:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.bwt.2bit.64:md5,d41d8cd98f00b204e9800998ecf8427e",
"genome.fasta.pac:md5,d41d8cd98f00b204e9800998ecf8427e"
"BWAMEM2_INDEX",
"bwa-mem2",
"2.2.1"
]
]
],
[
"versions.yml:md5,9ffd13d12e7108ed15c58566bc4717d6"
],
{
"BWAMEM2_INDEX": {
"bwamem2": "2.2.1"
}
}
],
"meta": {
"nf-test": "0.9.2",
"nextflow": "25.04.6"
},
"timestamp": "2025-09-04T08:55:53.219699135"
"timestamp": "2025-09-30T15:48:12.646279004"
},
"fasta": {
"content": [
[
[
{
"id": "test"
},
{
"0": [
[
{
"id": "test"
},
[
"genome.fasta.0123:md5,b02870de80106104abcb03cd9463e7d8",
"genome.fasta.amb:md5,3a68b8b2287e07dd3f5f95f4344ba76e",
"genome.fasta.ann:md5,c32e11f6c859f166c7525a9c1d583567",
"genome.fasta.bwt.2bit.64:md5,d097a1b82dee375d41a1ea69895a9216",
"genome.fasta.pac:md5,983e3d2cd6f36e2546e6d25a0da78d66"
]
]
],
"1": [
[
"BWAMEM2_INDEX",
"bwa-mem2",
"2.2.1"
]
],
"index": [
[
{
"id": "test"
},
[
"genome.fasta.0123:md5,b02870de80106104abcb03cd9463e7d8",
"genome.fasta.amb:md5,3a68b8b2287e07dd3f5f95f4344ba76e",
"genome.fasta.ann:md5,c32e11f6c859f166c7525a9c1d583567",
"genome.fasta.bwt.2bit.64:md5,d097a1b82dee375d41a1ea69895a9216",
"genome.fasta.pac:md5,983e3d2cd6f36e2546e6d25a0da78d66"
]
]
],
"versions1": [
[
"genome.fasta.0123:md5,b02870de80106104abcb03cd9463e7d8",
"genome.fasta.amb:md5,3a68b8b2287e07dd3f5f95f4344ba76e",
"genome.fasta.ann:md5,c32e11f6c859f166c7525a9c1d583567",
"genome.fasta.bwt.2bit.64:md5,d097a1b82dee375d41a1ea69895a9216",
"genome.fasta.pac:md5,983e3d2cd6f36e2546e6d25a0da78d66"
"BWAMEM2_INDEX",
"bwa-mem2",
"2.2.1"
]
]
],
[
"versions.yml:md5,9ffd13d12e7108ed15c58566bc4717d6"
],
{
"BWAMEM2_INDEX": {
"bwamem2": "2.2.1"
}
}
],
"meta": {
"nf-test": "0.9.2",
"nextflow": "25.04.6"
},
"timestamp": "2025-09-04T08:55:45.007921901"
"timestamp": "2025-09-30T15:48:02.557291528"
}
}
15 changes: 2 additions & 13 deletions modules/nf-core/bwamem2/mem/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ process BWAMEM2_MEM {
tuple val(meta), path("*.cram") , emit: cram, optional:true
tuple val(meta), path("*.crai") , emit: crai, optional:true
tuple val(meta), path("*.csi") , emit: csi , optional:true
path "versions.yml" , emit: versions
tuple val("${task.process}"), val('bwa-mem2'), eval("bwa-mem2 version 2>&1 | tail -1"), topic: versions, emit: versions_bwamem2
tuple val("${task.process}"), val('samtools'), eval("samtools --version 2>&1 | head -1 | sed 's/^.*samtools //'"), topic: versions, emit: versions_samtools

when:
task.ext.when == null || task.ext.when
Expand All @@ -46,12 +47,6 @@ process BWAMEM2_MEM {
\$INDEX \\
$reads \\
| samtools $samtools_command $args2 -@ $task.cpus ${reference} -o ${prefix}.${extension} -

cat <<-END_VERSIONS > versions.yml
"${task.process}":
bwamem2: \$(echo \$(bwa-mem2 version 2>&1) | sed 's/.* //')
samtools: \$(echo \$(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*\$//')
END_VERSIONS
"""

stub:
Expand All @@ -73,11 +68,5 @@ process BWAMEM2_MEM {
"""
touch ${prefix}.${extension}
${create_index}

cat <<-END_VERSIONS > versions.yml
"${task.process}":
bwamem2: \$(echo \$(bwa-mem2 version 2>&1) | sed 's/.* //')
samtools: \$(echo \$(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*\$//')
END_VERSIONS
"""
}
39 changes: 32 additions & 7 deletions modules/nf-core/bwamem2/mem/meta.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/meta-schema.json
name: bwamem2_mem
description: Performs fastq alignment to a fasta reference using BWA
keywords:
Expand Down Expand Up @@ -118,13 +119,37 @@ output:
description: Index file for BAM file
pattern: "*.{csi}"
ontologies: []
versions:
- versions.yml:
type: file
description: File containing software versions
pattern: "versions.yml"
ontologies:
- edam: http://edamontology.org/format_3750 # YAML
versions_bwamem2:
- - ${task.process}:
type: string
description: The process the versions were collected from
- bwa-mem2:
type: string
description: The tool name
- bwa-mem2 version 2>&1 | tail -1:
type: string
description: The command used to generate the version of the tool
versions_samtools:
- - ${task.process}:
type: string
description: The process the versions were collected from
- samtools:
type: string
description: The tool name
- samtools --version 2>&1 | head -1 | sed 's/^.*samtools //':
type: string
description: The command used to generate the version of the tool
topics:
- versions:
- - process:
type: string
description: The process the versions were collected from
- tool:
type: string
description: The tool name
- version:
type: string
description: The version of the tool
authors:
- "@maxulysse"
- "@matthdsm"
Expand Down
8 changes: 4 additions & 4 deletions modules/nf-core/bwamem2/mem/tests/main.nf.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ nextflow_process {
{ assert snapshot(
bam(process.out.bam[0][1]).getHeaderMD5(),
bam(process.out.bam[0][1]).getReadsMD5(),
process.out.versions
process.out.findAll { key, val -> key.startsWith("versions") }
).match() }
)
}
Expand Down Expand Up @@ -75,7 +75,7 @@ nextflow_process {
{ assert snapshot(
bam(process.out.bam[0][1]).getHeaderMD5(),
bam(process.out.bam[0][1]).getReadsMD5(),
process.out.versions
process.out.findAll { key, val -> key.startsWith("versions") }
).match() }
)
}
Expand Down Expand Up @@ -107,7 +107,7 @@ nextflow_process {
{ assert snapshot(
bam(process.out.bam[0][1]).getHeaderMD5(),
bam(process.out.bam[0][1]).getReadsMD5(),
process.out.versions
process.out.findAll { key, val -> key.startsWith("versions") }
).match() }
)
}
Expand Down Expand Up @@ -139,7 +139,7 @@ nextflow_process {
{ assert snapshot(
bam(process.out.bam[0][1]).getHeaderMD5(),
bam(process.out.bam[0][1]).getReadsMD5(),
process.out.versions
process.out.findAll { key, val -> key.startsWith("versions") }
).match() }
)
}
Expand Down
Loading
Loading