This article has multiple issues. Please help improve it or discuss these issues on the talk page . (Learn how and when to remove these messages)
|
Original author(s) | Paolo Di Tommaso |
---|---|
Developer(s) | Seqera Labs, Centre for Genomic Regulation |
Initial release | April 9, 2013 |
Stable release | v23.10.1 / January 12, 2024 |
Preview release | v24.02.0-edge / March 9, 2024 |
Repository | https://github.com/nextflow-io/nextflow |
Written in | Groovy, Java |
Operating system | Linux, macOS, WSL |
Type | Scientific workflow system, Dataflow programming, Big data |
License | Apache License 2.0 |
Website | nextflow |
Nextflow is a scientific workflow system predominantly used for bioinformatic data analysis. It establishes standards for programmatically creating a series of dependent computational steps and facilitates their execution on various local and cloud resources. [1] [2]
Many scientific data analyses require a significant amount of sequential processing steps. Custom scripts may suffice when developing new methods or infrequently running particular analyses, but scale poorly to complex task successions or many samples. [3] [4] [5]
Scientific workflow systems like Nextflow allow formalizing an analysis as a data analysis pipeline. Pipelines, also known as workflows, the specify order and conditions of computing steps. They are accomplished by special purpose programs, so-called workflow executors, which ensure predictable and reproducible behavior in various computing environments. [3] [6] [7] [8]
Workflow systems also provide built-in solutions to common challenges of workflow development, such as the application to multiple samples, the validation of input and intermediate results, conditional execution of steps, error handling, and report generation. Advanced features of workflow systems may also include scheduling capabilities, graphical user interfaces for monitoring workflow executions, and the management of dependencies by containerizing the whole workflow or its components. [9] [10]
Typically, scientific workflow systems initially present a steep learning challenge as all their features and complexities are built on in addition to the actual analysis. However, the standards and abstraction imposed by workflow systems ultimately improve the traceability of analysis steps, which is particularly relevant when collaborating on pipeline development, as is customary in scientific settings. [11]
In Nextflow, pipelines are constructed from individual processes that work in parallel to perform computational tasks. Each process is defined with input requirements and output declarations. Instead of running in a fixed sequence, a process starts executing when all its input requirements are fulfilled. By specifying the output of one process as the input of another, a logical and sequential connection between processes is established. [12]
This reactive implementation is a key design pattern of Nextflow and is also known as the functional dataflow model. [13]
Processes and entire workflows are programmed in a domain-specific language (DSL) which is provided by Nextflow which is based on Apache Groovy. [14] While Nextflow's DSL is used to declare the workflow logic, developers can use their scripting language of choice within a process and mix multiple languages in a workflow. It is also possible to port existing scripts and workflows to Nextflow. Supported scripting languages include bash, csh, ksh, Python, Ruby, and R. Any scripting language that uses the standard Unix shebang declaration (#!/bin/bash
) is compatible with Nextflow.
Below is an example of a workflow consisting of only one process:
processhello_world{input:valgreetingoutput:path"${greeting}.txt"script:""" echo "${greeting} World!" > ${greeting}.txt """}workflow{Channel.of("Hello","Ciao","Hola","Bonjour")|hello_world}
To enable easy collaboration on workflows, Nextflow natively support for source-code management systems and DevOps platforms including GitHub, GitLab, and others. [15]
Nextflow's DSL allows you to deploy and run workflows across different computing environments without having to modify the pipeline code. Nextflow comes with specific executors for various platforms, including major cloud providers. It supports the following environments for pipeline execution: [16]
In Nextflow, there is tight integration with software containers. Workflows and single processes can utilize containers for their execution across different computing environments, eliminating the need for complex installation and configuration routines. [3] [20]
Nextflow supports container frameworks such as Docker, Singularity, Charliecloud, Podman, and Shifter. These containers can be automatically retrieved from external repositories when the pipeline is executed. Additionally, it was revealed at Nextflow Summit 2022 that future versions of Nextflow will support a dedicated container provisioning service for better integration of customized containers into workflows. [21] [22]
Nextflow was originally developed at the Centre for Genomic Regulation in Spain and released as an open-source project on GitHub in July 2013. [23] In October 2018, the project license for Nextflow was changed from GPLv3 to Apache 2.0. [24]
In July 2018, Seqera Labs was launched as a spin-off from the Centre for Genomic Regulation. [20] The company employs many of Nextflow's core developers and maintainers and provides commercial services and consulting with a focus on Nextflow. [25]
In July 2020, a major extension and revision of Nextflow's domain-specific language was introduced to allow for sub-workflows and additional improvements. [26] In the same year, monthly downloads of Nextflow reached approximately 55,000. [20]
The nf-core project has been adopted by several sequencing facilities including the Centre for Genomic Regulation, [27] the Quantitative Biology Center in Tübingen, the Francis Crick Institute, A*STAR Genome Institute of Singapore, and the Swedish National Genomics Infrastructure as their preferred Scientific workflow system. [20] These facilities have collaborated to share, harmonize, and curate bioinformatic pipelines, [28] [29] [30] [31] leading to the creation of the nf-core project. [32] Led by Phil Ewels, at the Swedish National Genomics Infrastructure at the time, [33] [34] nf-core focuses on ensuring reproducibility and portability of pipelines across different hardware, operating systems, and software versions. In July 2020, Nextflow and nf-core received a grant from the Chan Zuckerberg Initiative in recognition of their importance as open-source software. [35] As of 2024, the nf-core organization hosts 117 Nextflow pipelines for the biosciences and more than 1382 process modules. With more than 1200 developers and scientists involved, it is the largest collaborative effort and community for developing bioinformatic data analysis pipelines. [36]
Nextflow is the preferred tool for processing sequencing data and conducting genomic data analysis by domain and research subject. Over the past five years, numerous pipelines have been published for various applications and analyses in the genomics field.
One notable use case is its role in pathogen surveillance during the COVID-19 pandemic. [37] Swift and highly automated processing of raw data, variant analysis, and lineage designation were essential for monitoring the emergence of new virus variants and tracing their global spread. Nextflow-enabled pipelines played a crucial role in this effort. [38] [39] [40] [41] [42] [43] [44]
Nextflow also plays a significant role for the non-profit plasmid repository Addgene, using it to confirm the integrity of all deposited plasmids. [45]
In addition to genomics, Nextflow is gaining popularity in other domains of biomedical data processing, where complex workflows on large amounts of primary data are required. These domains include Drug screening, [46] Diffusion magnetic resonance imaging (dMRI) in radiology, [47] and mass spectrometry data processing, [48] [49] [50] the latter with a particular focus on proteomics [51] [52] [53] [54] [55] [56] [57] [58]
Bioinformatics is an interdisciplinary field of science that develops methods and software tools for understanding biological data, especially when the data sets are large and complex. Bioinformatics uses biology, chemistry, physics, computer science, computer programming, information engineering, mathematics and statistics to analyze and interpret biological data. The process of analyzing and interpreting data can some times referred to as computational biology, however this distinction between the two terms is often disputed. To some, the term computational biology refers to building and using models of biological systems.
Genomics is an interdisciplinary field of molecular biology focusing on the structure, function, evolution, mapping, and editing of genomes. A genome is an organism's complete set of DNA, including all of its genes as well as its hierarchical, three-dimensional structural configuration. In contrast to genetics, which refers to the study of individual genes and their roles in inheritance, genomics aims at the collective characterization and quantification of all of an organism's genes, their interrelations and influence on the organism. Genes may direct the production of proteins with the assistance of enzymes and messenger molecules. In turn, proteins make up body structures such as organs and tissues as well as control chemical reactions and carry signals between cells. Genomics also involves the sequencing and analysis of genomes through uses of high throughput DNA sequencing and bioinformatics to assemble and analyze the function and structure of entire genomes. Advances in genomics have triggered a revolution in discovery-based research and systems biology to facilitate understanding of even the most complex biological systems such as the brain.
In bioinformatics, sequence analysis is the process of subjecting a DNA, RNA or peptide sequence to any of a wide range of analytical methods to understand its features, function, structure, or evolution. It can be performed on the entire genome, transcriptome or proteome of an organism, and can also involve only selected segments or regions, like tandem repeats and transposable elements. Methodologies used include sequence alignment, searches against biological databases, and others.
Orange is an open-source data visualization, machine learning and data mining toolkit. It features a visual programming front-end for exploratory qualitative data analysis and interactive data visualization.
The Generic Model Organism Database (GMOD) project provides biological research communities with a toolkit of open-source software components for visualizing, annotating, managing, and storing biological data. The GMOD project is funded by the United States National Institutes of Health, National Science Foundation and the USDA Agricultural Research Service.
GenePattern is a freely available computational biology open-source software package originally created and developed at the Broad Institute for the analysis of genomic data. Designed to enable researchers to develop, capture, and reproduce genomic analysis methodologies, GenePattern was first released in 2004. GenePattern is currently developed at the University of California, San Diego.
Galaxy is a scientific workflow, data integration, and data and analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming or systems administration experience. Although it was initially developed for genomics research, it is largely domain agnostic and is now used as a general bioinformatics workflow management system.
UGENE is computer software for bioinformatics. It works on personal computer operating systems such as Windows, macOS, or Linux. It is released as free and open-source software, under a GNU General Public License (GPL) version 2.
OpenMS is an open-source project for data analysis and processing in mass spectrometry and is released under the 3-clause BSD licence. It supports most common operating systems including Microsoft Windows, MacOS and Linux.
In the fields of molecular biology and genetics, a pan-genome is the entire set of genes from all strains within a clade. More generally, it is the union of all the genomes of a clade. The pan-genome can be broken down into a "core pangenome" that contains genes present in all individuals, a "shell pangenome" that contains genes present in two or more strains, and a "cloud pangenome" that contains genes only found in a single strain. Some authors also refer to the cloud genome as "accessory genome" containing 'dispensable' genes present in a subset of the strains and strain-specific genes. Note that the use of the term 'dispensable' has been questioned, at least in plant genomes, as accessory genes play "an important role in genome evolution and in the complex interplay between the genome and the environment". The field of study of pangenomes is called pangenomics.
LabKey Server is a software suite available for scientists to integrate, analyze, and share biomedical research data. The platform provides a secure data repository that allows web-based querying, reporting, and collaborating across a range of data sources. Specific scientific applications and workflows can be added on top of the basic platform and leverage a data processing pipeline.
Anduril is an open source component-based workflow framework for scientific data analysis developed at the Systems Biology Laboratory, University of Helsinki.
Integromics was a global bioinformatics company headquartered in Granada, Spain and Madrid. The company had subsidiaries in the United States and United Kingdom, and distributors in 10 countries. Integromics specialised in bioinformatics software for data management and data analysis in genomics and proteomics. The company provided a line of products that serve gene expression, sequencing, and proteomics markets. Customers included genomic research centers, pharmaceutical companies, academic institutions, clinical research organizations, and biotechnology companies.
A bioinformatics workflow management system is a specialized form of workflow management system designed specifically to compose and execute a series of computational or data manipulation steps, or a workflow, that relate to bioinformatics.
The BioCompute Object (BCO) project is a community-driven initiative to build a framework for standardizing and sharing computations and analyses generated from High-throughput sequencing. The project has since been standardized as IEEE 2791-2020, and the project files are maintained in an open source repository. The July 22nd, 2020 edition of the Federal Register announced that the FDA now supports the use of BioCompute in regulatory submissions, and the inclusion of the standard in the Data Standards Catalog for the submission of HTS data in NDAs, ANDAs, BLAs, and INDs to CBER, CDER, and CFSAN.
The 'German Network for Bioinformatics Infrastructure – de.NBI' is a national, academic and non-profit infrastructure initiated by the Federal Ministry of Education and Research funding 2015-2021. The network provides bioinformatics services to users in life sciences research and biomedicine in Germany and Europe. The partners organize training events, courses and summer schools on tools, standards and compute services provided by de.NBI to assist researchers to more effectively exploit their data. From 2022, the network will be integrated into Forschungszentrum Jülich.
Debasis Dash is an Indian computational biologist and chief scientist at the Institute of Genomics and Integrative Biology (IGIB). Known for his research on proteomics and Big Data and Artificial Intelligence studies, his studies have been documented by way of a number of articles and ResearchGate, an online repository of scientific articles has listed 120 of them. The Department of Biotechnology of the Government of India awarded him the National Bioscience Award for Career Development, one of the highest Indian science awards, for his contributions to biosciences, in 2014. He was appointed as the director of Institute of Life Sciences, Bhubaneswar on 18 May 2023.
Nvidia Parabricks is a suite of free software for genome analysis developed by Nvidia, designed to deliver high throughput by resorting to graphics processing unit (GPU) acceleration.