Skip to content

Standardized "versions" output format and handling #4924

Closed

Description

@stevekm

Outputting the versions of software involved in each Nextflow process has been an important aspect of pipelines for a while now.

A lot of pipelines have been outputting the versions in a manner like this

process FASTQC {

    input:
    tuple val(meta), path(reads)

    output:
    path  "versions.yml", emit: versions

    script:
    """
    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        fastqc: \$( fastqc --version | sed -e "s/FastQC v//g" )
    END_VERSIONS
    """
}

Where a YAML is created inside each process script block, outputted and collected, and then parsed downstream. Often for the purpose of creating a subsequent YAML file (and/or HTML table) for inclusion with MultiQC.

With the new topic Channels available, this method may be obsolete. Some newer methods for outputting the versions may be seen demo'd here;

https://github.com/stevekm/nextflow-demos/tree/master/topic-channels-versions

here is one example;

process FASTQC {
    tag "${id}"
    container "quay.io/biocontainers/fastqc:0.11.9--0"

    input:
    val(id)

    output:
    tuple val(task.process), val(task.container), val("fastqc"), eval('fastqc --version | sed -e \'s/FastQC v//g\''), topic: versions

    script:
    """
    """
}

This is a pretty nice improvement. However, it leaves open the question of "what should we put in these 'versions' topic channels, and how should we use it?"

I was wondering if the Nextflow dev team & community had any thoughts on this? Is this something that might be helpful to have some sort of standard method?

The method I have been prototyping is like this

// using the output schema shown above
// collect all the versions in a list of Map's
// so that it can be parsed easily with some downstream process
// create one single list of all the unique version map's
channel.topic('versions').map { proc, container, tool, vers ->
        return [
            "process": proc,
            "container": container,
            "software": tool,
            "version": vers
        ]
    }.unique().collect().view()

I am wondering what other methods may have been proposed for accomplishing this, and if some method such as this could be decided upon for inclusion with Nextflow official documentation. And by extension, if some method like this could be standardized, perhaps some extra helpers could be built into Nextflow to make it even easier to collect and parse these versions outputs?

Activity

  1. locked and limited conversation to collaborators on Apr 17, 2024
  2. converted this issue into a discussion #4925 on Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions