From 50eb4c912c570b1bf97cdc02f129c72c77a1fab6 Mon Sep 17 00:00:00 2001 From: giorgiagandolfi Date: Thu, 23 Jan 2025 11:19:05 +0100 Subject: [PATCH] finalize output.md --- docs/output.md | 427 +++++++++++++++++++++++++------------------------ 1 file changed, 222 insertions(+), 205 deletions(-) diff --git a/docs/output.md b/docs/output.md index 92ff31f..9b40db3 100644 --- a/docs/output.md +++ b/docs/output.md @@ -2,37 +2,72 @@ ## Introduction -The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. - This document describes the output produced by the pipeline. All plots generated in each step are summarised into the final report. +The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. ## Pipeline overview The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps: -- [Variant Annotation](#variant-annotation) - annotation of variants and cohort summary visualization -- [Formatter](#formatter) - coversion of files to different formats (unpubblished) -- [Lifter](#lifter) - pileup of private mutations of the other samples in multi-sample setting -- [Driver Annotation](#driver-annotation) - _add description_ (unpubblished) -- [QC](#qc) - quality control of copy-number and somatic mutation calling and creation of multi-CNAqc object -- [Subclonal Deconvolution](#subclonal-deconvolution) - -- [Signature Deconvolution](#signature-deconvolution) - - -Intermediate steps of the pipeline will output unpublished results which will be available for the user in the working directory of the pipeline. --> - -The pipeline is built using [Nextflow](https://www.nextflow.io/) and consists in five main subworkflows: - - [Variant Annotation](#variant-annotation) -- [Driver Annotation](#driver-annotation) +- [Formatter](#formatter) +- [Lifter](#lifter) +- [Catalogue Driver Annotation](#driver-annotation) - [QC](#qc) -- [Signature Deconvolution](#signature-deconvolution) - [Subclonal Deconvolution](#subclonal-deconvolution) - +- [Signature Deconvolution](#signature-deconvolution) -Intermediate steps connetting the main subworkflows will output [unpublished results](#unpublished-results) which will be available in the working directory of the pipeline. These steps consist in: - -- [Formatter](#formatter) -- [Lifter](#lifter) +## Directory Structure + +The default directory structure is as follows: + +``` +{outdir} +├── variant_annotation +| └── vep +│ └── +├── driver_annotation +| └── annotate_driver +│ └── +├── pipeline_info +├── formatter +│ ├── cna2cnaqc +│ └── +│ ├── cnaqc2tsv +│ └── +| └── vcf2cnaqc +│ └── +├── lifter +│ ├── mpileup +│ └── +│ └── positions +│ └── +├── QC +│ ├── tinc +│ └── +│ ├── CNAqc +│ └── +│ └── join_CNAqc +│ └── +├── signature_deconvolution +| ├── SigProfiler +│ └── +| └── SparseSignatures +│ └── +└── subclonal_deconvolution +| ├── mobster +│ └── +| ├── viber +│ └── +| ├── ctree +│ └── , +| └── pyclonevi +│ └── +work/ +.nextflow.log +``` + + ## Variant Annotation @@ -44,13 +79,13 @@ This directory contains results from the variant annotation subworkflow. At the This step starts from VCF files.
-Output files for all samples -Output directory: {publish_dir}/variant_annotation/VEP/dataset/patient/sample/ +Output files for all samples + +**Output directory: `{outdir}/variant_annotation/vep////`** + +- `__.vcf.gz` and `__.vcf.gz.tbi` + - VCF file and tabix index with called mutations -
    -
  • __.vcf.gz and __.vcf.gz.tbi -
  • annotated VCF file with tabix index -
- + -## Driver Annotation +
+Output files for all patients + +**Output directory: `{outdir}/lifter/positions///`** + +- `___all_positions.rds` + - RDS containing shared and private mutations + +
+ +## Catalogue Driver Annotation This directory contains results from the driver annotation subworkflow. ### Tumour-type driver annotation -According to the specified tumour type, potential driver mutations are identified and annotated using [IntOGen database](<(https://www.intogen.org/search)>). +According to the specified tumour type, potential driver mutations are identified and annotated using [IntOGen database](<(https://www.intogen.org/search)>). The user can also provide a custom table of driver genes.
Output files for all samples -**Output directory: `{publish_dir}/driver_annotation/annotate_driver////`** +**Output directory: `{outdir}/driver_annotation/annotate_driver////`** - `___driver.rds` - RDS with annotated mutations @@ -136,7 +245,7 @@ Add description
Output files for all samples -**Output directory: `{publish_dir}/driver_annotation/BuildReference////`** +**Output directory: `{outdir}/driver_annotation/BuildReference////`** * `fit.rds` * add description * `plot.rds` @@ -151,7 +260,7 @@ Add description
Output files for all samples -**Output directory: `{publish_dir}/driver_annotation/DNDSCV////`** +**Output directory: `{outdir}/driver_annotation/DNDSCV////`** * `dnds.rds` * DNDSCV object in RDS format @@ -159,17 +268,17 @@ Add description ## QC -The QC subworkflows requires in input a segmentation file from allele-specific copy number callers (either [Sequenza](https://sequenzatools.bitbucket.io/#/home), [ASCAT](https://github.com/VanLoo-lab/ascat)) and the joint VCF file from [join_positions](#join_positions) subworkflow. The QC sub-workflows first conduct quality control on CNV and somatic mutation data for individual samples in [CNAqc](#cnaqc) step, and subsequently summarize validated information at patient level in [join_CNAqc](#join_cnaqc) step. +The QC subworkflows requires in input a segmentation file from allele-specific copy number callers (either [Sequenza](https://sequenzatools.bitbucket.io/#/home), [ASCAT](https://github.com/VanLoo-lab/ascat)) and the joint VCF file. As a first step, the QC subworkflow provides an estimate of normal and tumour samples contamination in [TINC](#tinc) step, in order to have a measure of experimental quality. Then,it first conducts a quality control on copy number and somatic mutation data for individual samples in [CNAqc](#cnaqc) step, and subsequently summarize validated information at patient level in [join_CNAqc](#join_cnaqc) step. The QC subworkflow is a crucial step of the pipeline as it ensures high confidence in identifying clonal and subclonal events while accounting for variations in tumor purity. ### TINC -[TINC](https://caravagnalab.github.io/TINC/index.html) is a package to calculate the contamination of tumor DNA in a matched normal sample. TINC provides estimates of the proportion of cancer cells, containing the normal sample, and the proportion of cancer cells in the tumor sample (tumor purity). +[TINC](https://caravagnalab.github.io/TINC/index.html) is a package to calculate the contamination of tumor DNA in a matched normal sample. TINC provides estimates of the proportion of cancer cells containing the normal sample (TIN), and the proportion of cancer cells in the tumor sample (TIT).
Output files for all samples -**Output directory: `{publish_dir}/QC/tinc////`** +**Output directory: `{outdir}/QC/tinc////`** - `___fit.rds` - TINC fit containing TIN and TIT estimates in RDS; @@ -182,12 +291,12 @@ The QC subworkflow is a crucial step of the pipeline as it ensures high confiden ### CNAqc -[CNAqc](https://caravagnalab.github.io/CNAqc/) is a package to quality control (QC) bulk cancer sequencing data for validating copy number segmentations against variant allele frequencies of somatic mutations. +[CNAqc](https://caravagnalab.github.io/CNAqc/) is a package that performs quality control of bulk cancer sequencing data for validating copy number segmentations against variant allele frequencies of somatic mutations.
Output files for all samples -**Output directory: `{publish_dir}/QC/CNAqc////`** +**Output directory: `{outdir}/QC/CNAqc////`** - `___data_plot.rds` and `___data.pdf` - CNAqc report with genome wide mutation and allele specific copy number plots in RDS and PDF @@ -200,17 +309,17 @@ The QC subworkflow is a crucial step of the pipeline as it ensures high confiden ### join_CNAqc -This module creates a multi-CNAqc object for patient by summarizing the quality check performed at the single sample level. For more information about the strucutre of multi-CNAqc object see [CNAqc documentation](<(https://caravagnalab.github.io/CNAqc/)>). +This module creates a multi-CNAqc object for patient by summarizing the quality check performed at the single sample level. For more information about the structure of multi-CNAqc object see [CNAqc documentation](<(https://caravagnalab.github.io/CNAqc/)>).
Output files for all patients -**Output directory: `{publish_dir}/QC/join_CNAqc///`** +**Output directory: `{outdir}/QC/join_CNAqc///`** - `__multi_cnaqc_ALL.rds` - - unfiltered multi-CNAqc RDS object + - unfiltered mCNAqc RDS object - `__multi_cnaqc_PASS.rds` - - filtered multi-CNAqc RDS object + - filtered mCNAqc RDS object
@@ -218,7 +327,7 @@ This module creates a multi-CNAqc object for patient by summarizing the quality -Mutational signatures are distinctive patterns of somatic mutations in cancer genomes that reveal the underlying mutational processes driving tumor evolution and progression. These signatures are identified by analyzing aggregated point-mutation counts from multiple samples. Validated mutations from the [join_CNAqc](#join_cnaqc) step are converted into a joint TSV table (see [tsvparse](#tsvparse)) and then input into the signature deconvolution subworkflow, which performs de novo extraction, inference, interpretation, or deconvolution of mutational counts. +Mutational signatures are distinctive patterns of somatic mutations in cancer genomes that reveal the underlying mutational processes driving tumor evolution and progression. These signatures are identified by analyzing aggregated point-mutation counts from multiple samples. Validated mutations from the [join_CNAqc](#join_cnaqc) step are converted into a joint TSV table (see [cnaqc2tsv](#cnaqc2tsv)) and then input into the signature deconvolution subworkflow, which performs de novo extraction, inference, interpretation, or deconvolution of mutational counts. The results of this step are collected in `{pubslish_dir}/signature_deconvolution/`. Two tools can be specified by using `--tools` parameter: [SparseSignatures](#sparsesignatures) and [SigProfiler](#sigprofiler). @@ -229,32 +338,46 @@ The results of this step are collected in `{pubslish_dir}/signature_deconvolutio
Output files for dataset -**Output directory: `{publish_dir}/signatures_deconvolution/SparseSig//`** +**Output directory: `{outdir}/signatures_deconvolution/SparseSig//`** -- `best_params_config.rds` +- `_best_params_config.rds` - signatures best configiration object -- `cv_means_mse.rds` +- `_cv_means_mse.rds` - cross validation output RDS -- `nmf_Lasso_out.rds` +- `_nmf_Lasso_out.rds` - NMF Lasso output RDS -- `plot_signatures.pdf` - - exposure PDF plot -- `plot_signatures.rds` - - exposure RDS plot +- `_plot_signatures.pdf` and `_plot_signatures.rds` + - exposure plot in PDF and RDS
### SigProfiler -[SigProfiler](https://osf.io/t6j7u/wiki/home/) is a python framework that allows _de novo_ extraction of mutational signatures from data generated in a matrix format. The tool identifies the number of operative mutational signatures, their activities in each sample, and the probability for each signature to cause a specific mutation type in a cancer sample. The tool makes use of `SigProfilerMatrixGenerator` and `SigProfilerPlotting`, seamlessly integrating with other `SigProfiler` tools. +[SigProfiler](https://osf.io/t6j7u/wiki/home/) is a python framework that allows _de novo_ extraction of mutational signatures from data generated in a matrix format. The tool identifies the number of operative mutational signatures, their activities in each sample, and the probability for each signature to cause a specific mutation type in a cancer sample. The tool makes use of `SigProfilerMatrixGenerator` and `SigProfilerExtractor`, seamlessly integrating with other `SigProfiler` tools.
Output files for all samples -**Output directory: `{publish_dir}/signatures_deconvolution/SigProfiler//`** - -- `.` _missing_ - - _missing_ +**Output directory: `{outdir}/signatures_deconvolution/SigProfiler//results`** + +- `input/` + - folder containing a copy of the user-provided input files for SigProfilerMatrixGenerator step +- `input_data.txt` + - join table of all mutations in the dataset in TXT +- `logs/` + - folder containing the error and log files for SigProfilerMatrixGenerator step +- `output/` + - folder containing the DBS, SBS, INDEL nucleotide matrices resulting from SigProfilerMatrixGenerator step +- `SBS96/` + - folder containing the results of SigProfilerExtractor step in the SBS96 mutational context. This directory will contain: + - `All_Solutions/` + - subdirectory containing the results from running extractions at each rank within the range of the input. For more details visit the [official website](https://osf.io/t6j7u/wiki/5.%20Output%20-%20All%20Solutions/) + - `Suggested_Solution/` + - subdirectory containing the optimal solution. For more details visit the [official website](https://osf.io/t6j7u/wiki/6.%20Output%20-%20Suggested%20Solution/) +- `JOB_METADATA.txt` + - TXT file containing all the metadata about the system and runtime of the job +- `Seeds.txt` + - TXT file containing the replicate IDs and preset seeds
@@ -272,7 +395,7 @@ The subclonal deconvolution subworkflow requires in input a joint `mCNAqc` objec -The results of subclonal decovnultion step are collected in `{publish_dir}/subclonal_deconvolution/` directory. +The results of subclonal decovnultion step are collected in `{outdir}/subclonal_deconvolution/` directory. - -
- -## Unpublished results - -### Formatter - -The Formatter subworkflow is used to convert file to other formats and to standardize the output files resulting from different mutation (Mutect2, Strelka) and cna callers (ASCAT,Sequenza). Output files from this step are not published. - -#### cna2CNAqc - -This parser aims at standardize into a unique format copy number calls and purity estimate from different callers. - -
-Output files for all samples - -**Output directory: `{work_dir}/formatter/cna2cnaqc////`** - -- `___cna.rds` - - RDS file containing parsed cna output in table format - -
- -#### vcf2cnaqc - -This parser aims at standardize into a unique format single nucleotide variants from different callers. - -
-Output files for all samples - -**Output directory: `{work_dir}/formatter/vcf2cnaqc////`** - -- `___snv.rds` - - RDS file containing parsed vcf in table format - -
- -#### cnaqc2tsv - -This parser aims at converting mutations data of joint CNAqc analysis from CNAqc format (RDS file) into a tabular format (TSV file). This step is mandatory for running python-based tools (e.g. PyClone-VI, SigProfiler). - -
-Output files for all patients - -**Output directory: `{work_dir}/formatter/cnaqc2tsv///`** - -- `__joint_table.tsv` - - TSV file containing cna and variants joint - . - -
- -### Lifter - - - -The Lifter subworkflow is an optional step and it is run when single sample VCF file are provided. When multiple samples from the same patient are provided, the user can specify either a single joint VCF file, containing variant calls from all tumor samples of the patient (see [joint variant calling](<(https://nf-co.re/sarek/3.4.2/parameters/#joint_mutect2)>)), or individual sample specific VCF files. In the latter case, path to tumor BAM files must be provided in order to collect all mutations from the samples and perform pile-up of sample's private mutations in all the other samples. Two intermediate steps, [get_positions](#get_positions) and [mpileup](#mpileup), are performed to identify private mutations in all the samples and retrieve their variant allele frequency. Once private mutations are properly defined, they are merged back into the original VCF file during the [join_positions](#join_positions) step. The updated VCF file is then converted into a `vcfR` RDS object. - -#### mpileup - -At this stage, [bcftools](https://samtools.github.io/bcftools/bcftools.html) is used to perform the pileup in order to retrieve frequency information of private mutations across all samples. - -
-Output files for all samples - -**Output directory: `{work_dir}/lifter/mpileup////`** - -- `__.bcftools_stats.txt` - - TXT file with statistics on the called mutations -- `__.vcf.gz` and `__.vcf.gz.tbi`: - - VCF file and tabix index with called mutations - -
- -#### positions - -This intermediate step allows to retrieve private and shared mutations across samples originated from the same patient. Retrieved mutations are joined with original mutations present in input VCF, which is in turn converted into an RDS object using [vcfR](https://cran.r-project.org/web/packages/vcfR/vignettes/intro_to_vcfR.html). - -
-Output files for all samples - -**Output directory: `{publish_dir}/lifter/positions////`** - -- `__.pileup_VCF.rds` - - RDS containing shared and private mutations -- `__.positions_missing` - - TXT file containing mutations to be retrieved for a given sample +
--> -
+### Pipeline information
-Output files for all patients - -**Output directory: `{publish_dir}/lifter/positions///`** +Output files -- `___all_positions.rds` - - RDS containing shared and private mutations +- `pipeline_info/` + - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. + - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. + - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. + - Parameters used by the pipeline run: `params.json`.
- +### SigProfiler -### Pipeline information +Reference genome for SigProfiler is store in the following folder:
Output files -- `pipeline_info/` - - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. - - Parameters used by the pipeline run: `params.json`. +**Output directory: `{outdir}/subclonal_deconvolution/signature_deconvolution/SigProfiler/genome/tsb/{ref_genome}`** -
- -[Nextflow](https://www.nextflow.io/docs/latest/tracing.html) provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage. +- `{chromosome}.txt` + - genome assemby chromosme level +- `{ref_genome}_proportions.txt` + - genome assemby proportions +