Studying tissue-independent components of cancer and defining pan-cancer subtypes could be addressed using tissue-specific molecular signatures if classification errors are controlled. Since PAM50 is a well-known, United States Food and Drug Administration (FDA)-approved and commercially available breast cancer signature, we applied it with uncertainty assessment to classify tumor samples from over 33 cancer types, discarded unassigned samples, and studied the emerging tumor-agnostic molecular patterns. The percentage of unassigned samples ranged between 55.5% and 86.9% in non-breast tissues, and gene set analysis suggested that the remaining samples could be grouped into two classes (named C1 and C2) regardless of the tissue. The C2 class was more dedifferentiated, more proliferative, with higher centrosome amplification, and potentially more TP53 and RB1 mutations. We identified 28 gene sets and 95 genes mainly associated with cell-cycle progression, cell-cycle checkpoints, and DNA damage that were consistently exacerbated in the C2 class. In some cancer types, the C1/C2 classification was associated with survival and drug sensitivity, and modulated the prognostic meaning of the immune infiltrate. Our results suggest that PAM50 could be repurposed for a pan-cancer context when paired with uncertainty assessment, resulting in two classes with molecular, biological, and clinical implications. Molecular classification of cancer has been a central topic for decades, because it generates grounds for biological research and is directly linked to the development of specific therapies for distinct subtypes—a key element of precision medicine . Cancer In the search for molecular classification criteria able to subdivide any cancer type into relevant subtypes, FDA-approved molecular signatures initially developed for a particular tissue, like PAM50 ,5, may sDespite those results, there are several shortcomings in the previous uses of PAM50. First, it was applied without uncertainty assessment even though it may misclassify around 30% of breast cancer samples , which cWe aimed to evaluate if tissue-specific molecular signatures can be applied in a pan-cancer context and reveal broad molecular patterns with biological and clinical relevance that transcend tissues. Specifically, we used the PAM50 signature with an uncertainty assessment method to build a tumor-agnostic classification criterion that was associated with well-known cancer processes, prognosis, and drug sensitivity, and modulated the prognostic ability of the immune infiltrate.First, primary tumor samples of 33 different cancer types  from TCGA were confidently classified into the LumA, LumB, Her2e, and Basal subtypes. Second, gene set analysis was used to study the pan-cancer differences between those subtypes. The results prompted us to propose a new classification based on PAM50 and permutation-based confidence for molecular classification (PBCMC) ,17. ThirTCGA data were downloaded with TCGA assembler , and rawviations . GEO datviations . Cell liThe normalized and transformed CPM data was used for classification purposes. The PAM50 algorithm and uncertainty assessment analysis was performed with the PBCMC R package  using thp-value for each gene set, for each comparison , in each cohort. The use of MIGSA was restricted to cohorts with at least eight samples per subtype in each comparison, following the author´s suggestion. For both methods, a total of 1195 gene sets with lengths between 15 and 500 genes from MSigDB [p-values obtained with MIGSA were adjusted with Benjamini andand Hochberg correction within each cohort. The differential gene expression analysis was performed with limma package [p-values with Benjamini andand Hochberg method.Two kinds of gene set analysis were performed: Gene Set Variation Analysis (GSVA)  and the m MSigDB  were tesm MSigDB , 188 form MSigDB  and 957 m MSigDB ). Expres package  with logGene signatures were used with entrez gene identifiers when available, else entrezids were obtained from gene symbols using org.Hs.eg.db package. Genes without a valid entrezid were discarded. Signature scores were calculated according to Equation (2), except for the CA20 signature scores which were calculated using Equation (3). Wilcoxon signed-rank test was used to test the significance of the differences between classes, using Benjamini and Hochberg correction for multiple testing within each database. Tumor-normal differences were studied on TCGA participants with tumor and normal paired samples, calculating for each participant and signature a difference according to Equation (1). Tumor-normal differences between classes were tested with Wilcoxon signed-rank test, and Benjamini and Hochberg correction was performed within each database.p” for the tumor “t” and normal “n” samples, respectively.To quantify the difference between tumor and normal samples, a general equation was used in all instances:s”, i-th” GSVA gene set score or expression of the “i-th” gene for sample “s”, and “k” represents the number of genes or gene sets in the signature.The signature value (except for the CA20 signature) for each sample was calculated as the average of the normalized GSVA scores or normalized gene expression:s”, g” from the signature in sample “s”, “m” is the median expression of all the genes in the cohort, “k” is the number of genes in the signature.The CA20 signature value for each sample was calculated as the sum of the expression of the signature genes, centered by the median expression of the cohort and scaled by the standard deviation of the cohort:p-value than the ones observed with the classification under study.Cox proportional hazards models were used. Considering that most randomly generated molecular signatures are associated with survival in breast and other cancer types ,35, the p-values were adjusted with Benjamini and Hochberg correction. Shapiro–Wilk’s and Levene’s tests were used in each model to check the model assumptions. Model p-values were adjusted within each cohort using Benjamini and Hochberg method. Only drugs with adjusted model p-value < 0.01, Shapiro–Wilk’s p-value > 0.05, and Levenne’s test > 0.05 were considered significantly different between classes.A simple lineal model was adjusted for each drug in each cancer type to model the PAM50 combined with PBCMC was used on TCGA cohorts, discarding unclassified samples in order to work only with good representatives of the subtypes . The proportion of unclassified samples was 26% for breast cancer in accordance to previous studies , and betWe then compared the four PAM50 subtypes  to one another using the MIGSA and GSVA methods to search for pathways that transcend tissues. We chose the gene set analysis approach instead of differential gene expression because this approach can account for the genetic heterogeneity between cohorts, providing more concordant results in this pan-cancer scenario ,37,38. TTherefore, in TCGA data, both GSVA and MIGSA suggest that the samples classified with PAM50 and uncertainty assessment fall into two major classes: C1 comprising LumA samples, and C2 comprising LumB, Her2e, and Basal samples. The consistency of the results across tumor types encouraged us to propose a classification strategy based on PAM50 and uncertainty assessment: first apply PAM50 with an uncertainty assessment algorithm (PBCMC) discarding samples with low classification certainty, then LumA samples constitute the C1 class, and LumB, Her2e, and Basal constitute the C2 class.This classification proposal obtained on TCGA cohorts was then validated on GEO and GDSC datasets. The selected cancer types from GEO were bladder (GSE48075 ), colon G2/M checkpoint, activation of ATR in response to replication stress, and mitotic spindle checkpoint indicate alterations in pathways associated with cell-cycle-arrest in response to DNA damage, which are central alterations in cancer [resolution of sister chromatid cohesion, E2F targets, polo-like kinase mediated events, and mitotic G1 phase and G1/S transition indicate alterations in pathways related to cell-cycle progression.To validate the proposed classes, they were studied at the gene set level using MIGSA to compare C1 with C2 in 27 cohorts from TCGA and 6 from GEO (only cohorts with at least 8 samples in each class were used). Out of the 1195 gene sets analyzed, 126 were differentially enriched between C1 and C2 in more than half of the studied cohorts, with 28 gene sets differentially enriched in at least 30 out of the 33 cohorts . Therefon cancer . Other gp < 0.05). Therefore, compared to normal tissue, the C2 class presented a greater difference between normal and tumor samples in terms of cell-cycle checkpoints, cell-cycle progression, and response to DNA damage pathways, which suggests a more aggressive and proliferative profile [Cell-cycle checkpoints and cell-cycle progression pathways are closely associated with proliferation, and together they represent some of the most fundamental differences between normal cells and cancer cells ,46. Sinc profile .To characterize the C1 and C2 classes at the gene expression level, we performed a differential gene expression analysis on each cohort to compare these classes. In order to be consistent with the pathway analysis, only cohorts with at least eight samples in each class were used, resulting in 33 cohorts (27 from TCGA and 6 from GEO). The number of genes present in all TCGA, GEO, and GDSC databases was 8364, 1273 of which were differentially expressed in at least half of the analyzed cohorts , and 95 The differences between tumor and normal samples in terms of the 95 core genes were studied similarly to what was explained in the gene set analysis section (methods: Equations (1) and (2)). Analogously to the pathway results, the molecular profile of the C1 class was more similar to normal samples, while the C2 class presented a higher expression of the 95 core genes b.Gene set and gene expression analyses revealed that the main differences between C1 and C2 are associated with cell-cycle progression, cell-cycle checkpoints, and DNA damage response. To further validate and characterize the proposed classification, we studied some well-known biological processes for which we could find previously published molecular signatures.The differences between C1 and C2 observed so far were closely associated with proliferation, which is probably the most fundamental trait of cancer  and an iFor each of those signatures, a sample-based score was calculated in cohorts with at least two samples in each class (C1 and C2), which accounts for a total of 56 cohorts: 32 from TCGA, 6 from GEO, and 18 from GDSC. The signature scores were calculated as the average of the normalized expression of the signature genes (methods: Equation (2)), except for the CA20 signature (methods: Equation (3)). To study the difference between normal and tumor samples, we used the same approach explained during gene set and gene expression analysis (Methods: Equation (1)).p < 0.05) in 30 of the 32 TCGA cohorts  in 27/32 of the TCGA cohorts, 6/6 of the GEO cohorts, and 6/18 of the GDSC cohorts : LGG, LIHC, LUAD, PAAD, MESO, KIRC, and BRCA, with the C2 class having a decreased survival  with different IC50 values in C1 compared to C2 in non-breast cohorts  that the hazard ratio (between the low and high immune infiltrate groups) in the C1 class was different than the hazard ratio in the C2 class (p = 0.012), but a reduced survival in the C2 class  (p = 0.049). Moreover, the C1/C2 classification improved the prognostic ability of the immune infiltrate in 55 cases from nine cohorts , where the complete Cox model  performed significantly better  than the univariate model . Therefore, our results suggest that cell-cycle markers, which are closely associated with the C1/C2 classification, can improve the prognostic ability of immune markers. This has been observed in non-small-cell lung carcinoma, where a classification based on proliferation and PD-L1 expression was associated with survival in patients treated with immune checkpoint inhibitors [In 28 cases we found evidence  . Despite= 0.033) , this clhibitors . Overallhibitors —our resuExpression-based signatures that can effectively classify tumor samples independently of their histology may reveal tissue-independent components of cancer that can push forward our knowledge of the disease and help in the design of new therapies and clinical trials. In this sense, molecular signatures originally devised and validated for a single cancer type could provide useful information in a pan-cancer scenario.This study was focused on the FDA approved breast cancer PAM50 signature paired with an uncertainty assessment method. We subdivided each cancer type into two classes: C1 comprising confidently assigned luminal A samples, and C2 comprising confidently assigned luminal B, Her2e, and Basal samples. This grouping of PAM50 subtypes emerged from a pathway analysis performed on 18 TCGA non-breast cancer types, and it was studied from multiple perspectives in 32 non-breast cohorts from TCGA, 6 cohorts from GEO, and over 1000 cell line profiles from GDSC. This reduction in the number of groups (from four subtypes to two classes) suggests that the genes that PAM50 uses to subdivide the more proliferative tumors into luminal B, Her2-enriched and basal-like (ESR1- and ERBB2-associated genes)  do not sThe molecular differences between the C1 and C2 classes were remarkably consistent between methods and across cohorts from all different data sources. We identified 28 gene sets and 95 core genes exacerbated in the C2 class, and two transcription factors (E2F1 and FOXM1) that are known to affect prognosis and response to therapy in different cancer types, and might be responsible for most of the differences between the two classes. The C1/C2 classification was also closely associated with multiple molecular signatures that describe other well-known cancer processes such as centrosome amplification, retinoblastoma protein pathway, and differentiation. Therefore, this study shows that a tissue-specific molecular signature can be used in a pan-cancer context and reveal components of the disease that have the same biological meaning regardless of the cancer type.We additionally studied the prognostic relevance of the C1/C2 classification, finding several cancer types where the C2 class had a significantly shorter survival. Even when not directly prognostic on its own, the C1/C2 classification modulated the prognostic ability of the immune infiltrate, revealing that in some cases the infiltrate of a particular cell type can be either a favorable or unfavorable indicator, depending on the class under study. This strengthens the growing idea that the immune infiltrate needs to be studied taking into account other variables and shows pathways and genes that deserve further investigation.The differences observed between C1 and C2 also had implications in compound sensitivity. Gene set analysis predicted 255 compounds that could affect C1 and C2 differentially, and 70 drugs showed differential activity in GDSC cell lines. The main aim of performing such drug sensitivity analysis was to find pan-cancer patterns that could be used as a proxy for cancer therapy selection based on C1 and C2 profiles. While we could not find any clear sensitivity patterns when considering general mechanism of action (MoA) of anticancer drugs, we did have some intriguing findings. As expected, most type of cancers presented clear patterns of pan-resistance to therapies associated to C2 profiles . However, some cancers  showed the opposite scenario: an increased sensitivity of C2 cells vs. C1 to several types of drugs. In addition, such situation was observed for particular drugs in cancers with the opposite pattern . These findings suggest that these cancers cells have acquired actionable vulnerabilities to certain anticancer agents that could be exploited for successful treatment. Moreover, since C1 profiles are more similar to those of normal tissue, these findings also suggest the acquisition of such vulnerabilities should allow targeted treatment without major collateral effects on the normal cells of the patient. Future prospective studies using ex-vivo drug sensitivity screenings with patient-derived cells subjected to C1/C2 determination should help to validate these findings.https://github.com/Dario-Rocha/tumor.agnostic.git.In summary, this study shows the use of a tissue-specific molecular signature as a tumor-agnostic classification tool with the aid of a proper uncertainty assessment method, resulting in consistent biological patterns. Our observations, added to the fact that PAM50 is an FDA approved signature for breast cancer, indicate that PAM50 could be repurposed for other cancer types with potential clinical implications that encompass drug sensitivity and immune infiltrate. This opens the door to studying any molecular signature that proved to be useful in a specific cancer type in other cancer types, which could result in a better understanding of this disease, more therapeutic options that could be offered, and improved patient stratification. A shiny app and R code to explore our findings in user-defined expression data is available at


Chromatin dysregulation is associated with developmental disorders and cancer. Numerous methods for measuring genome-wide chromatin accessibility have been developed in the genomic era to interrogate the function of chromatin regulators. A recent technique which has gained widespread use due to speed and low input requirements with native chromatin is the Assay for Transposase-Accessible Chromatin, or ATAC-seq. Biologists have since used this method to compare chromatin accessibility between two cellular conditions. However, approaches for calculating differential accessibility can yield conflicting results, and little emphasis is placed on choice of normalization method during differential ATAC-seq analysis, especially when global chromatin alterations might be expected.Using an in vivo ATAC-seq data set generated in our recent report, we observed differences in chromatin accessibility patterns depending on the data normalization method used to calculate differential accessibility. This observation was further verified on published ATAC-seq data from yeast. We propose a generalized workflow for differential accessibility analysis using ATAC-seq data. We further show this workflow identifies sites of differential chromatin accessibility that correlate with gene expression and is sensitive to differential analysis using negative controls.We argue that researchers should systematically compare multiple normalization methods before continuing with differential accessibility analysis. ATAC-seq users should be aware of the interpretations of potential bias within experimental data and the assumptions of the normalization method implemented. Genome-wide quantitative sequencing methods for measuring genomic features have been recently developed to address various biological questions previously limited to locus-level interrogation. Current applications include gene expression , DNA metThe latest major technique for effectively measuring genome-wide chromatin accessibility is the Assay for Transposase-Accessible Chromatin, or ATAC-seq. This assay makes use of a Tn5 transposase reaction which preferentially inserts a 9-bp adapter fragment into accessible regions of the genome . AdapterATAC-seq has been used to both identify basal accessible chromatin regions in a given cellular context as well as determine regions differentially accessible (DA) between two cellular states –20. The csaw which permits testing of multiple normalization methods. In whole, researchers should be aware of differing biological interpretations resulting from different normalization methods and any biases, which may not be considered or eliminated during analysis. This is especially true wherein widespread chromatin structure alterations might be expected, such as when disrupting chromatin regulators. We further provide computational methodology that serves as a “one size fits all” guideline for ATAC-seq data analysis from reads to differential accessibility analysis.Using our recently reported in vivo sorted mouse ATAC-seq data set, we show that different tools and normalization methods for calculating significant DA regions yield distinct results following disruption of a chromatin remodeler . We showMACS2, DiffBind, csaw, voom, limma, edgeR, and DESeq2 [I and II follow the DiffBind protocol, originally intended for ChIP-seq data, which constructs a consensus read count matrix from MACS2 replicate peak sets of m query regions by n samples. Briefly, MACS2 constructs an ATAC fragment pileup from aligned paired-end data, then builds a local bias track through a series of parameters to estimate background noise, and finally compares ATAC signal to the local background at each genomic bp using a Poisson test. Significant nearby regions are then merged into a peak. DiffBind then calculates linear scaling factors from either the total number of reads in each library, which assumes that true global differences may be expected and technical bias is small, or the total number of reads in queried peak regions, which should eliminate global differences in favor of reducing any technical biases. The former method is applied in I and the latter in II. The count matrix with normalization factors is then subject to the DESeq2 framework of dispersion estimation and negative binomial generalized linear model (GLM) fitting for hypothesis testing, according to the design matrix. Analyses III through VI follow approaches described in the csaw manual [III and IV count reads in query regions defined by MACS2 peak sets then filter low-abundance windows, while V and VI use the csaw sliding window approach to quantify ATAC signal in the 300-bp interval query windows across the genome. The de novo query windows in V and VI then pass low-abundance filtering and are tested for signal enrichment greater than threefold over the surrounding 2 kilobase local neighborhood. For normalization, III and V implement the trimmed mean of M values  normalization method. This highly conservative method normalizes the signal distribution locally based on extent of ATAC signal abundance. As a result, the loess fit assumes a symmetric global distribution in which there are no true biological global differences in ATAC reaction efficiency or distribution, and any evidence of these biases are technical and should be removed. The count matrices with respective normalization factors or offsets from all four of these csaw analyses are then subject to the edgeR statistical framework of estimating dispersions by empirical Bayes and quasi-likelihood GLM fitting for hypothesis testing, according to the design matrix. Finally, analyses VII and VIII follow the same procedure as III and IV with respect to using MACS2 peak query regions filtered by abundance in csaw, but instead they follow a voom transformation to log2 counts per million (log2CPM) in VII which are further quantile normalized in VIII. The log2CPM transformation simply scales by full library size and maintains those assumptions, while quantile normalization equalizes the signal distribution across all libraries [limma linear modeling and hypothesis testing by empirical Bayes-moderated statistics.To determine if choice of ATAC-seq DA analysis method influences experimental results, we compared 8 different DA approaches Table  using thd DESeq2 –33. Analw manual . III andues (TMM ) method ibraries . Quantilibraries . The R26Pik3ca*H1047R; Arid1afl/* mutant mice. Mutant and wild-type endometrial epithelial cells were positively selected by a surface marker, EPCAM, and purified by magnetic bead separation [We recently reported an ATAC-seq data set in which chromatin accessibility was compared between sorted mutant and control mouse endometrial epithelial cells following disruption of a common tumor suppressor and oncogene . In thisometrial  and periometrial  invasionparation . ATAC-sey-axis (M) and signal intensity as the x-axis (A). For example, TMM normalization of the MACS2 peaks read count matrix in III shows the majority of these genomic regions are increasing accessibility in mutant cells compared to control cells, whereas a small minority are decreasing. The TMM approach is similar to the default normalization method of DiffBind in I which scales counts based on library size, but TMM only considers regions expected to be unchanged for generating normalization factors. These observations are in contrast with the DA results of a more conservative loess-based count adjustment in IV. The loess normalization yielded more significant DA regions that were decreasing rather than increasing accessibility. MA plots for DA regions following all varying analyses show how these global patterns are affected through different normalization methods. The DA distribution is shifted upwards in the TMM normalized windows, which may or may not be technical in nature, and is corrected with the loess normalization.We next compared chromatin accessibility patterns between mutant and control endometrial epithelial populations using the 8 DA approaches. Different patterns of DA measurements were observed through MA plot visualization depending on the choice of DA approach used Fig. . MA plotHOMER [We next performed a detailed comparative analysis of the DA outputs from all 8 different approaches. The number of significant DA regions identified by an FDR < 0.10 threshold ranged from 33 to 24,450 with varying proportions of regions increasing vs. decreasing accessibility  in mutant vs. control endometrial epithelial populations. One issue is that quantifying direct overlap between promoter DA and gene DE is sensitive to the DA FDR threshold. To overcome this, we implemented a precision–recall (PR) curve to predict gene DE status  with promoter DA FDR values yielded by a given DA analysis. The PR curves showed modest predictive ability of promoter DA, though analyses I) to FDR = 0.936 (VI) . An advantage to this data set is the inclusion of two control groups, one containing NaCl in the wash buffer and one without NaCl, which permits comparisons between two negative control groups. Schep et al. also utilized a published expression microarray data set by Ni et al. [In addition to comparing the effects of genetic mutations or other treatment conditions, examining temporal changes in chromatin accessibility in cell populations is another application of ATAC-seq DA analysis. We utilized the Schep et al.  osmotic i et al.  from thei et al. .n = 2) vs. 0-min control (n = 4) comparison, in which between 491 and 1082 regions were determined significantly DA , ranging from 0 to 85 regions , the displayed chromatin accessibility profiles over the time course did not match the direction of gene expression change  with modifications. Example applications include calling baseline accessible regions in naïve cells and identifying DA genomic regions. The field typically accepts data sets with at minimum two biological replicates for standard peak calling, as was established by ENCODE for ChIP-seq data [naiveOverlapBroad.sh, Additional file The workflow presented in Fig. seq data , and at k-means clustering [A major addition to our workflow is quantifying and normalizing library molecular complexity. Often times, it is desired to quantify and compare ATAC signal at different genomic loci which are not typically part of an ATAC-seq or differential ATAC-seq analysis framework. This could involve genomic tiling, i.e., quantifying signal in evenly distributed genomic intervals, or quantifying ATAC at regions defined through other assays, such as ChIP-seq, or ustering , 43–45. preseqR and a wrapper ATACseqQC have implemented functions to estimate complexity by calculating a duplicate frequency matrix then estimating the number of unique molecules sequenced  in each library sample [samtools view can then be used to subsample libraries based on these estimates. In support that the stochastic subsampling process should not greatly affect experimental results, a replicated analysis with two different random subsampling seeds yielded highly similar and overlapping results from peak calling   were also retained in the naïve overlap peak set indicating replicate peak region concordance > 50% , where 5508 were concordantly identified between both, 2585 were uniquely called in the Buenrostro et al. ZINBA peak set, and 161 were uniquely called in the naïve overlap peak set  showed better performance for differential ChIP-seq analysis than  domains . At leas domains . While o domains , 66. CurWe present data indicating that ATAC-seq is sensitive to bias when comparing chromatin accessibility across multiple conditions. We compared several commonly used, published methods for calculating differential accessibility to our previously reported in vivo ATAC-seq data set as well as a yeast ATAC-seq time series data set, and we observe conflicting results dependent upon the normalization method used. We provide intuitive, standardized bioinformatics methodology for analyzing ATAC-seq data by non-computational scientists. Our validated workflow also includes a critical, complexity-normalization step. Altogether, we argue that researchers should properly normalize ATAC-seq data before calculating differential accessibility.Sequencing data used for analyses presented in this manuscript were downloaded from GEO accessions GSE121198, GSE66386, GSE47753, and GSE26312. Yeast genes with distinct expression response patterns following 0.6 M NaCl exposure were defined and extracted from Supp. Table 4 reported by Ni et al. DiffBind, DESeq2, csaw, edgeR, voom, and limma as described in the “Results” section [See Figs.  section –33. WorkLtfCre0/+; (Gt)R26Pik3ca*H1047R; Arid1afl/fl vs. control sorted mouse endometrial epithelial cells were extracted from GEO accession GSE129784. 3481 significant DE genes were selected by FDR < 0.05 filtering. 24,097 total expressed genes were used as gene universe for enrichment analyses.Differential gene expression results from previously reported RNA-seq data of affy package and normalized via the Robust Multi-Array Average (RMA) expression measure. RMA values are reported as log2 scale. The mean RMA value of both replicates was used for analyses in this manuscript. The 22,277 measured probes were collapsed to 11,369 genes with a unique Ensembl and symbol identifier [Raw data were downloaded from GEO for both GM12878 replicates generated by Affymetrix HT Human Genome U133A Array. CEL files were read into R through the entifier . ExpressHOMER [cis-promoter classification as within 3000 bp of a canonical gene TSS. Yeast genomic regions were annotated by TxDb.Scerevisiae.UCSC.sacCer3.sgdGene R package using the genes and promoters functions with default settings (yeast promoters are defined as − 2000 to + 200 bp around TSS) [PRROC R package using the pr.curve function [DESeq2 FDR < 0.05 threshold. Read counts in ATAC-seq peaks were calculated by HOMER for correlation and box dot plot quantification. MACS2 was used to generate ATAC-seq signal tracks for display in IGV [ComplexHeatmap [biomaRt was used for all gene nomenclature and mouse–human ortholog conversions [ggplot2 was used for certain plotting applications throughout this manuscript [Mouse and human ATAC-seq peak coordinates were annotated by HOMER  with a mund TSS) . Unweighfunction . For PR y in IGV , 70. MSiy in IGV . PathwayxHeatmap . biomaRtversions . The cumnuscript .Additional file 1. Supplementary Information .Additional file 2.ATACseq_workflow.txt—Example machine-readable Fig. Additional file 3. bedpeTn5shift.sh—Bash script for shifting coordinates in standard 10-column format BEDPE files to compensate for Tn5 adapter insertion as described in Buenrostro et al. See Fig. Additional file 4.bedpeMinimalConvert.sh—Bash script for converting standard 10-column format BEDPE to the “minimal” format defined by MACS2. See Fig. Additional file 5.naiveOverlapBroad.sh—Bash script for calculating naïve overlap broad peak set from 2 individual replicate peak sets and a pooled replicate peak set. Can be modified for to accept more replicates as desired. See Fig. Additional file 6. csaw_workflow.R—Example R workflow for differential accessibility analysis with csaw as graphically displayed in Fig. MACS2 peak sets as query regions or identifying de novo locally enriched windows.


Differential splicing (DS) is a post-transcriptional biological process with critical, wide-ranging effects on a plethora of cellular activities and disease processes. To date, a number of computational approaches have been developed to identify and quantify differentially spliced genes from RNA-seq data, but a comprehensive intercomparison and appraisal of these approaches is currently lacking. In this study, we systematically evaluated 10 DS analysis tools for consistency and reproducibility, precision, recall and false discovery rate, agreement upon reported differentially spliced genes and functional enrichment. The tools were selected to represent the three different methodological categories: exon-based , isoform-based  and event-based methods . Overall, all the exon-based methods and two event-based methods (MAJIQ and rMATS) scored well on the selected measures. Of the 10 tools tested, the exon-based methods performed generally better than the isoform-based and event-based methods. However, overall, the different data analysis tools performed strikingly differently across different data sets or numbers of samples. Differential splicing (DS) enables production of various messenger RNAs (mRNAs), and thereby various protein products, from one gene . This prAlternative splicing (AS) events are currently divided into five main types: skipped exons (SE), alternative 5′ (donor) splice sites (A5SS), alternative 3′ (acceptor) splice sites (A3SS), retained introns (RI) and mutually exclusive exon usage (MXE) . In exonThe technology of RNA-sequencing (RNA-seq) has enabled the detailed analysis of the transcriptome  and its Two major strategies are currently applied for DS analysis: isoform-based  [n = 100) [All 10 tools included in our comparison were tested using four RNA-seq data sets. The first two: human prostate cancer (PCa) data set (n = 28)  and human = 100)  were chon = 100) . The secn = 100) .Isoform-based methods first seek to reconstruct full-length transcripts and estimate their relative abundances in each sample based on the sequencing reads. Statistical testing is then applied to identify significant differences in the relative transcript abundances between the different experimental conditions. The performance of this approach depends on accurate transcript quantification.Cufflinks is a pipeline consisting of different programs including cufflinks itself , cuffmerab initio approach; it first reconstructs the transcriptome based on the aligned reads, then quantifies the abundance of alternative paths through the graph and finally identifies the alternative splicing modules (ASMs) [DiffSplice takes a graph-based s (ASMs) . ASMs arCount-based methods include both exon-based and event-based approaches. In exon-based methods, read counts are assigned to different features, such as exons or junctions. The limitation of this approach is that it does not infer the type of the splicing event occurring in a gene but only identifies the differentially expressed exons/junctions between experimental conditions. In event-based methods, splicing events themselves are quantified by calculating the percentage spliced in (PSI) values for each event, which measure the fraction of mRNAs expressed from a gene that contains a specific form of that event .Exon-based method DEXSeq is an R/Bioconductor package developed to detect DS from RNA-seq data. The method uses a generalized linear model to model the differential usage of exons in different sample groups . It assuedgeR is an R/Bioconductor package that can be used to analyse differential expression at the gene, exon or transcript level . The exoJunctionSeq is an R/Bioconductor package, which utilizes a similar statistical strategy as DEXSeq. It enables estimation of differential exon usage as well as known or novel exon junctions .limma is an R/Bioconductor package that is widely used for differential gene expression analysis and has been extended to perform DS using exon-level count data . It fitsdSpliceType is an event-based method designed to find DS by utilizing base-wise read coverage signal data . It extrMAJIQ (Modeling Alternative Junction Inclusion Quantification) uses local splicing variations (LSVs) to quantify RNA splicing in genes. LSVs are splits in a splice graph where several edges come to or from a single exon called a reference exon . The LSVrMATS  is an evSUPPA is an event-based method that uses transcript abundances to estimate the PSI values for each DS event . Transcrwww.ebi.ac.uk/arrayexpress/) under accession number E-MAT-567 (accessed on November 2015). The second data set, referred to as HCa data set, involves 100 human normal and tumor samples from hepatocellular carcinoma metastasis [In the present study, four different publicly available RNA-seq data sets were analysed . The firtastasis  and was tastasis . The foutastasis , 32 DS gHomo sapiens) and NCBIM37 (Mus musculus) genomes were downloaded from Illumina igenomes (https://support.illumina.com/sequencing/sequencing_software/igenome.html). Sequences for transcriptome reference file of GRCh37 were downloaded in fasta format from Ensembl (GRCh37: release 83) and indexed using bowtie2 [The reference sequences and annotation files for Ensembl GRCh37  with STAR v2.6.1b [The sra files downloaded from SRA or GEO were converted to fastq files using the sratoolkit.2.8.0  and the  v2.6.1b  using deThe tools used to identify DS required different types of input files and the reference files for each tool were prepared according to the description available with the tool. Samtools v1.2  was usedFor the purpose of general quality overview, we produced gene-level read counts using the featureCounts tool  and normIn this study, we systematically evaluated 10 DS analysis tools. The tools were selected to represent the three different methodological categories: isoform-based , exon-based  and event-based methods . The DS tools were run using the default settings. The description and the commands used to run the programs are provided in the In our first comparison setting, the consistency of the splicing tools was tested using the PCa and HCa human data sets. The results provided by different tools at the level of isoforms, exons or events were aggregated to gene level in order to compare the methods. Details of the aggregation approach are given in the P-value due to very few findings with FDR below 1. Where needed, test statistic was used as a secondary ranking criteria.For selecting the DS genes for each method the FDR threshold was set at 0.05. Where ranks of genes were considered, ranking was done based on FDR for most of the methods. For DiffSplice, the method’s own test statistic was used since this is the only statistic it provides whereas for cuffdiff2, the results were ranked according to full denote the DS genes in the complete data set and let DSsubset denote the DS genes in the subset of the data. The precision was defined asTo calculate the precision and recall of the methods, we compared samples between tumor and normal group with varying numbers of replicates in each data set. Let DSCorrespondingly, the recall was defined asFDR was assessed in mock comparisons by selecting randomly without replacement from the normal group two artificial sample groups to be compared. The process was repeated 10 times for each different subset size. The findings of these mock comparisons were considered as false positives (FP). To estimate the FDR, the number of FPs detected were divided by the median number of detections in the real comparison with the same number of samples. FDR values were truncated to 1, if they were greater than 1.In our second comparison setting, we compared the detections in the mouse data set (MVS) and in the human data set (HVS) to the 28 and 32 qPCR-validated DS genes, respectively.For evaluating the performance of the tools at different sequencing depths, the HVS data set was downsampled to subsets of 20 to 100 million reads with increments of 20 million reads. Precision and recall were calculated as above with the genes detected by each tool in the original HVS data set considered as the complete set.P-values of the GO terms that were among the top 10 most enriched terms with at least one of the methods. The P-values were then used as input for hierarchical clustering and heatmap visualizations.Functional enrichment analysis was carried out to detect the most enriched Gene Ontology (GO) biological processes across the different methods. The top 500 genes from each method were analysed for enrichment using the R/Bioconductor package topGO  using thWe tested 10 different DS tools on four different RNA-seq data sets. The PCa and HCa data sets were used for the assessment of consistency and reproducibility of the tools by performing true and mock comparisons. In the true comparisons, the analyses were performed on the complete data sets as well as on smaller subsets of samples chosen at random without replacement from both sample groups. In the mock comparisons, the randomly chosen samples were all from the same  experimental condition. Both the true and the mock comparisons were repeated 10 times for each subset size. In the MVS and HVS data sets, which contained qPCR-validated genes, the tools were assessed in their ability to retrieve the validated genes. The overall experimental design is illustrated in Strikingly, large variability in the numbers of detections was found between the tools in the PCa and HCa data sets . In the For most of the tools, the number of samples had a substantial effect on the number of DS genes reported; edgeR was the only tool that showed consistent increase in the number of DS genes in both data sets when the number of samples was increased. DEXSeq, JunctionSeq, limma and rMATS also followed this trend with minor deviations between consecutive subsets. DiffSplice did not show any consistent trend in the number of DS genes when the number of samples increased, while dSpliceType and SUPPA/SUPPA2 actually detected fewer DS genes with larger numbers of samples in both data sets, the newer version of the tool reporting significantly fewer DS genes than the older version. The newer version also showed increased relative SD. MAJIQ reported almost the same number of DS genes with all sample sizes.Next, we assessed the precision and recall of the different tools by comparing the DS genes reported among the random subsets to those reported in the complete PCa and HCa data sets. Recall for a given subset was defined as the proportion of DS genes in the complete data set that were also detected in the subset (see Methods). Correspondingly, precision was defined as the proportion of DS genes detected in the subset that were also detected in the complete data set. In general, both precision and recall increased with the increase of the number of samples, except for cuffdiff2, which reported no DS genes in most subsets . Overalln = 10 or n = 25). However, the variability of the FDR values was noticeable for DiffSplice, edgeR, JunctionSeq, limma and SUPPA/SUPPA2 even at the largest sample size in the PCa data set. Similar to the real comparisons, isoform-based cuffdiff2 reported very few DS genes also in the mock comparisons, regardless of the number of samples. Isoform-based DiffSplice overall reported the lowest FDRs in both data sets, followed by the exon-based tools DEXSeq, JunctionSeq and limma. Out of the event-based tools MAJIQ and rMATS (both versions) performed better than SUPPA/SUPPA2 and dSpliceType, which reported similar numbers of DS genes across the runs in both the true and the mock comparisons, leading to very high FDR.To investigate the FDR reported by the different tools, we performed mock comparisons in the PCa and HCa data sets by randomly sampling two groups from the normal experimental condition. Normal sample group was used here, as differences between the random subsets were expected to be minor compared to differences in the tumor group and thus DS genes reported in the derived mock comparisons can be considered to be FP. The FDR for each tool was estimated by scaling the number of FP by the median number of DS genes detected with the same number of samples in the corresponding real comparison. Generally, FDR and its variability across the 10 random subsets decreased as the number of samples increased , being cAs the different tools reported a hugely variable number of DS genes , we decided to focus on the top 500 ranking genes for evaluating the overlap across the tools. dSpliceType was excluded from the analysis as it did not allow reporting DS genes with FDR above 0.05 and thus provided less than 500 genes altogether. Notably, the overlap of the 500 top-ranking genes between the tools in general was strikingly low 3. The hiFor the event-based tools, we also calculated the proportions of the different types of events reported among the top 500 detections from each tool, summarized in The major reason for the inclusion of the MVS and HVS data sets in this study was to bring the chosen DS analysis tools to bear on data sets wherein we have, even if only a limited number of qPCR-validated splicing events (28 in MVS and 32 in HVS data set). It should, however, be noted here that rMATS was used in the original studies to determine the splicing events selected for the qPCR validation. While all tools recovered a clearly higher proportion of validated genes in the larger HVS data set, the number of the detected, validated and total DS genes varied considerably between the different tools . MAJIQ aP-value  enriched biological processes found in at least nine of the tools were GO:0000375 , GO:0000377 , GO:0000398 , GO:0006397 (mRNA processing) and GO:0016071 (mRNA metabolic process) in the PCa data set and GO:0019752 (carboxylic acid metabolic process) and GO:0044281  in the HCa data set. Overall, the most significantly enriched of GO terms were detected by rMATS and MAJIQ in both data sets. These methods also clustered together according to their GO enrichment. Although this work does not focus on biological mechanisms, this preliminary result nevertheless warrants further investigation as to the DS of the splicing machinery genes themselves as drivers of cancer or other disease processes.To further investigate the DS gene lists provided by the different tools, we ran GO  enrichmeP-value . The oven = 3 and n = 5), its memory usage increased rapidly with the increasing sample size .Analysis of high-throughput sequencing data is a computationally intensive task. Major measures of performance of the tools are their maximum memory consumption and total runtime with increasing numbers of samples, which were here ascertained for all tools . All tooTo study the effect of the sequencing depth on the performance of the methods, we compared the number of DS genes and the number of qPCR-validated genes in the HVS data set that had more than 100 million reads per sample . This alTo investigate whether there were differences in the proportions of differentially expressed genes between the DS gene lists reported by the different tools, we performed gene-level differential expression analysis in the complete PCa and HCa data sets with limma requiring FDR < 0.05 and absolute fold-change >2 . The proIn this study, we evaluated 10 tools  for the The isoform, exon and event-level results were aggregated to the gene level in order to compare the different methods  and two event-based methods (MAJIQ and rMATS) overall performed robustly, showing low FDR, high precision and moderate recall 2. AlthouWe observed that with most methods, the number of detected DS genes increased and the relative variation across the randomly sampled subsets decreased with larger numbers of replicates, as expected. However, cuffdiff2 reported very few DS genes in all but HVS data set and with dSpliceType and SUPPA, the number of reported DS genes consistently decreased when the number of replicates increased in PCa and HCa data sets. Similar poor performance (no results or decreasing number of detections with an increasing number of replicates) has been earlier shown for cuffdiff2 in previous studies both in the context of differential gene expression , 52, 53 In our comparison, the tools were run using default settings, as this is how most users would in practice do. Initially, we performed all the analyses based on alignments produced using Tophat2  but lateFor all methods, we used the same complete annotation file (including definitions of gene structures), except for DiffSplice that does not use any annotation. Some of the methods  detect also unannotated novel splicing events, but we limited our comparisons to the known annotations. Considering the low overlap observed between the different methods and the fact that significant performance decrease with incomplete annotation has been shown earlier , compariThe effect of sequencing depth and read length in regard to DS has been studied earlier. The study by Liu et al.  suggesteIn this study, our aim was to compare methods that claim to perform DS analysis from RNA-seq data using different approaches. The different tools provide the results at the exon, event or isoform level depending on the strategy employed by them. Our comparison showed how the results vary largely across the different methods even for the same approach category. The low general overlap of the results especially across the event-based tools may, in fact, be partly explained by the large differences in the approaches taken by the different methods, which may yield to varying strength in identifying different types of splicing events, as depicted in our analysis . While rThus, our main conclusion is that running several tools is advisable in order to generate a comprehensive view of the DS among the studied samples. While the isoform-based methods compare the relative isoform abundances and exon-based methods compare exon and exon-junction read counts, the event-based methods compare the quantitated splicing events. All of the three approaches have the general goal to reveal differences in the gene expression as the outcome of the operation of the splicing machinery. Currently, it remains an open question which of these approaches (or which combination of these approaches) will prove to be most useful in elucidating the underlying biological phenomena and more application studies will be needed to answer this question.To our knowledge, this work provides the first independent cross-comparison of DS analysis tools across real vertebrate data sets. However, it would be interesting to repeat the evaluation when more real data sets with fairly large numbers of replicates, sufficient sequencing depth and read length and/or more qPCR-validated DS genes becomes available.In our comparison, all the exon-based methods  and the two event-based methods, MAJIQ and rMATS, overall performed well in terms of the number of detections, as well as precision and recall in detecting DS across the four data sets analysed. These methods also achieved a moderate FDR and recovered a reasonably high proportion of the previously qPCR-validated DS genes, thus presenting these tools as the currently best available candidates for DS analysis in RNA-seq data. In practice, the limitation of rMATS in requiring equal read length across the input data may need to be considered. Where computational performance is a concern (maximum memory and run time), limma and edgeR are recommended over the other tools. However, currently we would recommend running DS analysis using more than one tool due to the relatively large variability of the results reported by the different tools.A.M. performed the analysis and wrote the manuscript. A.L. and M.V. participated in writing the manuscript and in supervising the study. A.J.M. participated in writing the manuscript. N.W. participated in performing the analysis. L.L.E. conceived and supervised the study and participated in writing the manuscript.Supplementary_Figures_bbz126Click here for additional data file.Supplementary_Tables_bbz126Click here for additional data file.Supplementary_File_bbz126Click here for additional data file.


Mass cytometry (CyTOF) is a technology that has revolutionised single-cell biology. By detecting over 40 proteins on millions of single cells, CyTOF allows the characterisation of cell subpopulations in unprecedented detail. However, most CyTOF studies require the integration of data from multiple CyTOF batches usually acquired on different days and possibly at different sites. To date, the integration of CyTOF datasets remains a challenge due to technical differences arising in multiple batches. To overcome this limitation, we developed an approach called CytofRUV for analysing multiple CyTOF batches, which includes an R-Shiny application with diagnostic plots. CytofRUV can correct for batch effects and integrate data from large numbers of patients and conditions across batches, to confidently compare cellular changes and correlate these with clinically relevant outcomes. Mass cytometry or Cytometry by Time-Of-Flight (CyTOF)  is a higTo overcome this challenge, a normalisation method was created to improve the comparability between measurements . BrieflyIn this study, a single CyTOF dataset, barcoded sample set, or run is referred to as a CyTOF batch. We call the batch effects we seek to remove ‘unwanted variation’, and their causes include: differential antibody staining across samples within a batch, different batches of reagents, different machines or the inevitable lab differences found in multicentre studies. The ability to accurately distinguish true biological changes from technical artefacts like those just mentioned is critical, and has already been done to an extent for flow cytometry assays .Two methods have recently been published that aim to achieve consistency between samples across batches which make use of shared reference samples across batches. BatchAdjust  offers mwww.github.com/mtrussart/CytofRUV. We begin by examining the batch effects found when comparing CyTOF data from samples replicated across batches. To do so, we built an R-Shiny application that exhibits any batch effects present in such samples using four different diagnostic plots and their associated numerical metrics based on protein expression distributions and clustering results. Then, we compare the unadjusted data with the normalised data using either CytofRUV, BatchAdjust or CytoNorm, all on three different datasets. Our results suggest that not only does CytofRUV do better at removing unwanted variation from measured protein expression, it also makes the distributions of these quantities more uniform across batches, and enhances the detection of biologically important changes embodied in the data across batches.In recent years, a class of methods called Remove Unwanted Variation (RUV) has been developed to remove unwanted variation such as batch effects, from high-dimensional genetic and genomic data. They have been applied to microarray , RNA-seqA known source of unwanted variation in CyTOF datasets is the time-drift in signal intensity . Howeverarcsinh (intensity/5) in all that follows.We created a dataset based on 24 samples in total, consisting of peripheral blood mononuclear cell (PBMC) samples from three patients with chronic lymphocytic leukaemia (CLL) and PBMC from nine healthy controls (HC), each replicated across two batches of 12 samples . All samThe first class of diagnostic plots we use is based on median protein expression. The multi-dimensional scaling (MDS) plot  computedWe next examined the magnitude of the batch-to-batch differences in the distributions of the protein expression across replicates, as our second class of diagnostic plots. We observed that batch effects not only affect each protein differently  but alsoTo assess how batch effects affect cell subpopulations, we used our first dataset of samples from 3 patients with CLL and 9 PBMC healthy controls (HC). We partitioned the entire collection of cells into clusters using FlowSOM as described in To optimise the dimensionality reduction and visualise the extent to which discrete subsets of cells are separated from each other in the 31-dimensional space, we performed a linear discriminant analysis (LDA) on four subsets of cells: the predominant CLL cell subpopulation (cluster 9) and CD8 killer cell subpopulation (cluster 2) identified in the study from batches 1 and batch 2. The first dimension (LD1) separates the cell types well, while the second dimension (LD2) embodies batch differences: cells that originate from batch one are located at the top of the plot while cells from batch two are at the bottom of the plot .CyTOF has the unique ability to provide a deeper understanding of the molecular changes induced by targeted therapies at the single-cell level and how their efficacy is influenced by cancer heterogeneity. However, the assessment of patient heterogeneity and its correlation with the clinical outcome necessitates a reliable detection of changes in protein expression across patients. As such, we compared the median expression of BCL-2 for the 3 patients with CCL in the main CLL cluster nine and found that the variation from a single sample across batches is higher than that observed in the BCL-2 median expression across patients . To overIn CyTOF studies, the analysis of cell subpopulation abundances as well as that of protein expression can be used to identify sets of proteins that are associated with response to a treatment. Comparing the proportions of inferred cell types across different drug treatments highlights the subpopulations that change across experimental conditions. To assess whether batch differences affect cell subpopulation abundances, we compared the cluster frequencies across replicates. We detected a noticeable difference in the proportions of CLL cancer cells  among cells that originate from the batch one compared to those from batch 2 . Our R-SIn summary, we examined the reproducibility of samples replicated across batches. To facilitate the identification of batch effects across replicates, we developed an R-Shiny application that produces the four diagnostic plots previously mentioned: Median Protein Expression, Protein Expression Distributions, Clustering Results and Cluster Proportions. We found that batch differences affect not only the protein expression levels, but also the cluster proportions. Such differences are problematic in large-scale studies with multiple patients, cell types and treatments, as they compromise the detection of biologically important changes. The integration of datasets from multiple CyTOF batches is therefore an important challenge to be addressed.To integrate data from multiple CyTOF batches, we have developed a normalisation method that removes batch effects in CyTOF data. In order to estimate and adjust for such unwanted variation, CytofRUV exploits the concept of pseudo-replicates (here cells) in the RUV-III method that has been successfully applied to the Nanostring nCounter gene expression platform  and to sWe first used CytofRUV on data from the 12 samples replicated across two batches, using two samples as our known replicated reference samples. Specifically, we used HC1 for the HC samples and CLL2 for the CLL samples, defining all cells from those samples in any given cluster to be pseudo-replicates. Assuming that all 20 clusters have cells from HC1 and CLL2 in both batches, any differences in protein expression between cells within the cluster but in different batches will represent unwanted variation. We summarized the performance of our CytofRUV normalisation method on all the CLL samples using three metrics and compare them with the corresponding ones for BatchAdjust  and CytoTo assess the quality of the our CytofRUV normalisation, we first compared the distributions of proteins across the two batches for the designated replicated reference samples. We found that, for all the proteins, these distributions become more similar across batches . Also, wAltogether EMD revealed that CytofRUV is the method with the optimal reduction in batch differences from protein expression for 11 out of 12 samples. CytoNorm gave the lower EMD when compared with BatchAdjust for 7 out of 12 samples .The proportions of the different cell subpopulations in replicates in different batches should be consistent. This consistency among the replicates ensures that a differential analysis of their abundances will confidently detect robust cellular changes across experimental conditions. To assess how well CytofRUV corrects for differences in cluster proportions across batches, we computed the Hellinger distance (see Methods) between the paired cluster proportions of all replicated samples. For the HC samples, CytofNorm gave the lowest Hellinger distances for 4 samples out of 9 compared to both CytofRUV and BatchAdjust. Two remaining samples are the ones where CytofRUV is able to make those proportions more consistent across replicates . For allSilhouette scores (see Methods), which are a combined measure of the degree of separation of cells within clusters and that between clusters. We compute two Silhouette scores sbiology and sbatch, based on the cell subpopulations defined by the clusters, and batch grouping, respectively. A normalisation method that successfully removes batch effects should lead to adjusted data with a small sbatch score, while one which preserves or enhances the biology should have high value of sbiology, at least as high as that before adjustment. To assess this aspect of the performance of normalisation methods, we combine the sbiology for clusters with sbatch for the batches in a single plot  CytofRUV not only removes the batch effects but also improves biological accuracy of the clusters compared to the two other methods. Likewise, CytoNorm is performing well according to Silhouette scores in the second analysis of the third dataset. We also conclude that BatchAdjust has the least ability to remove batch effects in the three datasets we have explored here, according to those metrics. This conclusion might be a result the single batch adjustment performed by BatchAdjust to the measurements on all cells, whereas CytoNorm and CytofRUV both take into account the impact of batch differences at the cluster level.biology to determine the tightness and separation of the clustering, as it provides an evaluation of the clustering validity.We envisage that in some cases removing batch effects might lead to removal of clusters or to the identification of new biologically relevant clusters. Some clusters could be artefacts of the batch effects. We use sWe observed that the B cell cluster  to relatTo further quantify the extend of batch effects in CyTOF datasets, it might be interesting to perform differential analyses of replicates across batches. Recent methods like diffcyt  have beeOne limitation of our method is that CytofRUV does rely on the assumption that at least one cell subpopulation is shared across the batches that would be used as our pseudo-replicates.In experiments where large batch effects occur and no cell subpopulation is shared between batches, our method would not be applicable.In this study, we considered three different datasets containing up to 24 samples and up to 12 batches. RUV methods have already been applied to hundreds and thousands of samples with Nanostring technology  and hundA requirement of the current CytofRUV method is the availability of enough material to have replicated reference samples in each batch. As previously mentioned for CytoNorm, and as we also observed for CytofRUV, in order to remove batch effects from all samples, it might be necessary to include more than one set of replicated references samples in the batch, in particular including samples that are similar to each of the main types of samples. In future work, we plan to explore estimating the unwanted variation using different replicated reference samples, each present in only some of the batches, to avoid the need for replicated reference samples in every batch. For example, the use of a carefully designed ‘bridging’ set of replicated reference samples analogous to long term reference samples in metabolomics  should lIn summary, we proposed here a computational algorithm called CytofRUV that effectively enables batch effect reduction in mass cytometry with an adaptable normalisation method that detects heterogeneity of cellular responses across large-scale studies with multiple patients, cell types and conditions (e.g. treatment).To remove the unwanted variation across multiple datasets and batches, we used fastRUV-III as previously described . Brieflyj in a cell i with m cells and n proteins. The standardisation is to have zero mean and unit standard deviation across all cells for all protein measurements.Here, i. kj in a cell i and N.m x n matrix of these residuals. Next the k-vectors of all proteins as 'negative controls'. Finally, the estimates of The data are normalised in six steps. First, all the data are clustered, typically using FlowSOM, although other clustering methods can be used. Second, group of cells are defined to be pseudo-replicates if they belong to the same subpopulation (i.e. cluster) but are in different batches, which can be done either on specific clusters or on all clusters. Third, cell-level residuals are computed by averaging all the protein measurements across those pseudo-replicates, and subtracting these averages from each measurement on each cell. In essence, differences of measurements on pseudo-replicates are considered as unwanted variation. The quantities For all the datasets, BatchAdjust was performed using the R code and usage instructions as described in CytoNorm was performed as described in To quantify the similarity of protein expression distributions across batches, we computed the Earth Movers Distance, also called the Wasserstein metric, as described in To quantify the differences between cluster proportions across batches, we computed the Hellinger distances between the proportions found in samples replicated across different batches. This distance is defined for two probability We computed these distances for both the original and the normalised datasets.i and all other cells in the group to which cell i is assigned, and i and any cells in other groups not containing cell i, then the silhouette coefficient of cell i is calculated asTo assess the extent to which the data are grouped based on the batch effects as opposed to biological signals, we computed batch and biology Silhouette scores. Given a partitioning of all cells into groups, if ib is high and ia is low) indicate that cells are well matched to their own group. In this way, we computed the silhouette score sbatch based on the batches as groups and the silhouette score sbiology based on the grouping of the cells by subpopulation (i.e. clusters).The average of the silhouette values across cells using a particular grouping is called the silhouette score for that grouping. Silhouette score ranges from −1 to +one where positive values   . InstallBlood was obtained from healthy donors (via the Victorian Donor Blood Registry) and patients with CLL . All patients consented under Melbourne Health HREC 2016.305 and samples were analysed under HREC 2016.066. PBMCs were isolated using standard Ficoll-based methods and cryopreserved.Cells were thawed and stained for viability with cisplatin. Cells were then fixed with paraformaldehyde  to a final concentration of 1.6% for 10 min at room temperature. Cells were pelleted and washed once with cell staining medium  to remove residual PFA and stored at −80°C.191Ir/193Ir DNA intercalator  in PBS with 1.6% PFA at 4°C overnight. Cells were washed once with CSM, washed three times with double-distilled water and filtered to remove aggregates and resuspended with EQ normalisation beads immediately before analysis using a Helios mass cytometer . Throughout the analysis, cells were maintained at 4°C and introduced at a constant rate of ~300 cells/sec.Cells were barcoded using 20-plex palladium barcoding according to manufacturer’s instructions . Following barcoding, cells were pelleted and washed once with cell staining medium (PBS with 0.5% BSA and 0.02% sodium azide) to remove residual PFA. Cells were stained with CD16/CD32 for 10 min and surface antibodies  for 30 mData concatenation, normalisation and debarcoding are done using the R Catalyst package merging the two batches (RUV_1B and RUV3B) when applying the Finck normalisation. The R script used to generate this preprocessing (CytofRUV_preprocessing_dataset.R) is also available in the supplementary data.The FCS files from this study are available at flow repository ID FR-FCM-Z2L2. In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.Acceptance summary:The work is an important contribution to the field of cytometry. It provides an objective and well thought out normalization procedure for mass cytometry and potentially even fluorescence flow cytometry.Decision letter after peer review:eLife. Your article has been reviewed by four peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by Detlef Weigel as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Nima Aghaeepour (Reviewer #1); Anna Belkina (Reviewer #3); Sofie Van Gassen (Reviewer #4).Thank you for submitting your article "CytofRUV: Removing unwanted variation to integrate multiple CyTOF datasets" for consideration by The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.eLife papers without additional data, even if they feel that they would make the manuscript stronger. Thus the revisions requested below only address clarity and presentation.We would like to draw your attention to changes in our revision policy that we have made in response to COVID-19 (https://elifesciences.org/articles/57162). Specifically, we are asking editors to accept without delay manuscripts, like yours, that they judge can stand as Summary:The authors present a new Cytof normalization approach based on RUV III that has proven useful for other technologies including RNASeq, single-cell RNAseq and nanostring. The reviewers all agreed that this was a strong manuscript that makes an important contribution to an area of the field that remains under-served, and they unanimously recommended publication.Essential revisions:Several concerns arose during review that the authors should address to strengthen the results and improve the presentation.1) Given the size of typical cytometry data sets (millions of cells and hundreds of samples), how does this approach scale? What are the limitations? The Discussion mentions large-scale studies, but how large in practice?biology , nor how and if the proposed figures and evaluation measures could be interpreted to detect this situation. The reviewers felt this aspect needed further exploration and discussion.2) Most importantly, reviewers raised questions around the evaluation of the normalization procedures. Specifically, since the data are re-clustered after normalization, and performance of normalization was assessed against the re-clustered data (and while the reviewers agreed this made sense), they were concerned about how a negative impact of normalization could be assessed. Specifically, if normalization failed, leading to fusion of biologically relevant clusters post-normalization, how could that be detected? It was not clear that this would be captured by S3) Finally the reviewers were looking for more guidance from the presentation around the relative ranking of the different methods. The authors should more clearly present their conclusions about the relative performance of the methods and state which methods are performing well or poorly and why, and if the results are not conclusive, this should be stated and explained. Essential revisions:Several concerns arose during review that the authors should address to strengthen the results and improve the presentation.1) Given the size of typical cytometry data sets (millions of cells and hundreds of samples), how does this approach scale? What are the limitations? The Discussion mentions large-scale studies, but how large in practice?We thank the reviewers for their question and we updated the Discussion of the manuscript with more details regarding large-scale studies and their limitations in the revised manuscript.“In this study we considered three different datasets containing up to 24 samples and up to 12 batches. […] For example, the use of a carefully designed “bridging” set of replicated reference samples analogous to long term reference samples in metabolomics  should lead to the normalization of large studies that is just as effective as that achieved using the same replicated reference samples throughout.”biology , nor how and if the proposed figures and evaluation measures could be interpreted to detect this situation. The reviewers felt this aspect needed further exploration and discussion.2) Most importantly, reviewers raised questions around the evaluation of the normalization procedures. Specifically, since the data are re-clustered after normalization, and performance of normalization was assessed against the re-clustered data (and while the reviewers agreed this made sense), they were concerned about how a negative impact of normalization could be assessed. Specifically, if normalization failed, leading to fusion of biologically relevant clusters post-normalization, how could that be detected? It was not clear that this would be captured by Sbiology and provide an example of how users can validate their clusters. We expanded the exploration and discussions of the questions raised here in the Discussion in the revised manuscript.The reviewers are raising an important aspect to take into account in the evaluation of the normalization. We acknowledge that in some cases there might be loss of clusters or the identification of new biologically relevant clusters. An important aspect to take into account in this discussion is that the same number of clusters is conserved before and after normalization. To assess the performance of the normalization, we clarify out use of S3) Finally the reviewers were looking for more guidance from the presentation around the relative ranking of the different methods. The authors should more clearly present their conclusions about the relative performance of the methods and state which methods are performing well or poorly and why, and if the results are not conclusive, this should be stated and explained.We added more descriptions in the relative performance of each method for all three datasets (Results) and clearly stated the conclusions from those analysis in the Discussion in the revised manuscript.


Colon cancer is common worldwide and is the leading cause of cancer-related death. Multiple levels of omics data are available due to the development of sequencing technologies. In this study, we proposed an integrative prognostic model for colon cancer based on the integration of clinical and multi-omics data.In total, 344 patients were included in this study. Clinical, gene expression, DNA methylation and miRNA expression data were retrieved from The Cancer Genome Atlas (TCGA). To accommodate the high dimensionality of omics data, unsupervised clustering was used as dimension reduction method. The bias-corrected Harrell’s concordance index was used to verify which clustering result provided the best prognostic performance. Finally, we proposed a prognostic prediction model based on the integration of clinical data and multi-omics data. Uno’s concordance index with cross-validation was used to compare the discriminative performance of the prognostic model constructed with different covariates.Combinations of clinical and multi-omics data can improve prognostic performance, as shown by the increase of the bias-corrected Harrell’s concordance of the prognostic model from 0.7424  to 0.7604 . Additionally, 2-year, 3-year and 5-year Uno’s concordance statistics increased from 0.7329, 0.7043, and 0.7002  to 0.7639, 0.7474 and 0.7597 , respectively.In conclusion, this study successfully combined clinical and multi-omics data for better prediction of colon cancer prognosis. Colon cancer, which is a subset of colorectal cancer (CRC), is common worldwide and is the leading cause of cancer-related death. Although incidence and mortality rates have declined in recent years due to changes in risk factors and recent improvements in screening tests and treatments, there are large differences in 5-year colon cancer survival rates across countries and regions , 2.Because of the development of sequencing technology, precision medicine has become a popular field in cancer research. Omics data have been widely used for cancer classification based on identified gene signatures, gene pathways, and protein-protein interaction networks, among others –5. Such The American Joint Committee on Cancer (AJCC) tumor, node and metastasis (TNM) staging system is an important tool used for clinical colon cancer prognostic predictions. However, no molecular factors or omics features were included in the TNM system for colon cancer in the recently published 8th version . In contAs cancer research has entered the fields of precision medicine and personalized medicine, non-molecular features have become insufficient, whereas the inclusion of molecular features is becoming an increasingly popular research direction. Scientists have been attempting to integrate multiscale omics data to gain deeper insight into cancer mechanisms as the human body is a complex system. The Cancer Genome Atlas (TCGA) conducted a series of comprehensive integrative molecular analyses with multiscale data types to identify the genomic alterations in several cancer types; five genome-wide platforms were used to identify somatic alterations in colorectal carcinoma . Kim D. The performance of cancer prognostic analyses may benefit from the integration of clinical features and molecular features . PreviouTo improve the prediction performance of colon cancer prognosis, an integrative prognostic analysis of colon cancer was proposed in this study based on clinical, gene expression, DNA methylation and miRNA expression data from TCGA.https://portal.gdc.cancer.gov/repository) with the provided data-transfer tool. Then, the downloaded raw data files were reprocessed following the flowchart shown in Fig. Normalized and preprocessed clinical data and omics data  of primary tumors included in the TCGA-COAD (colon adenocarcinoma) project were downloaded from the new TCGA data portal  and age at initial diagnosis were chosen as clinical features, as previous studies have shown that these four features have significant effects on CRC prognosis . Patienthttp://www.genome.jp/kegg/) database and other omics features that showed potential relationships with colon cancer prognosis in previous articles [Prior knowledge was based on the pathways and microRNAs involved in CRC according to the Kyoto Encyclopedia of Genes and Genomes  normalized data were chosen for further analysis, and the original Ensembl IDs were converted to gene symbols with the biomaRt R package. Then, each feature with more than 5% NA or 0 values was removed. Features with the top 25% of coefficients of the variable along with features in the prior knowledge list of gene expression were selected. Ultimately, 4691 features were selected for further analysis.For miRNA expression data, reads per million mapped reads (RPM) normalized data were chosen for further analysis. Each feature with more than 5% NA or 0 values was removed. Features with the top 25% of coefficients of the variable along with features in the prior knowledge list of microRNA expression were selected. Ultimately, 111 features were selected for further analysis.For DNA methylation data, the beta values were used in the analysis, whereas various probes  were converted into 34,040 gene symbols. The conversion procedure was performed by calculating the average DNA methylation beta value of the CpG sites in a particular region of a gene based on TCGA-assembler 2 . In addiPatients with both clinical data and the three types of omics data were chosen for our analysis. Overall, 344 patients had both clinical data and the three types of omics data, resulting in 4691 features in the gene expression profiles, 4441 features in the DNA methylation profiles and 111 features in the miRNA expression profiles. Detailed information regarding the clinical and omics data is shown in Table The overall pipeline for the construction of the prognosis prediction model is shown in Fig. Hierarchical clustering was used for the cluster analysis because this approach is an unsupervised cluster method that is widely used in omics data analyses. The following three parameters were employed in this unsupervised clustering step: the distance method was used as the metric, the linkage method was used as the linkage criteria, and the cluster number was used as the cut-off condition. In addition, each cluster should contain at least 10 patients. Overall, seven distance methods, including Euclidean, Maximum, Manhattan, Canberra, Binary, Minkowski and Correlation; eight linkage methods, including Ward.D, Ward.D2, Single, Complete, Average, Mcquitty, Median and Centroid; and cluster numbers ranging from 2 to 11 were applied to evaluate different combinations of the cluster parameters.Clustering of the three types of omics data was performed. Different combinations of distance methods, linkage methods and cluster numbers were used, and the combination that provided the best prognostic information was selected by fitting a single-covariate Cox proportional hazards (PH) model. The cluster labels generated by the identified clustering parameters were used as new omics features for the three types of omics data. In addition, we investigated whether the integrated multi-omics data could improve the prognostic prediction performance over that of separate multi-omics data combined with clinical features. Then, we conducted a cluster-of-clusters (C-o-C) approach based on these new omics features to integrate the three types of omics data because this approach has shown excellent performance in previous single-cancer and pan-cancer studies , 29, 30.h(t) is the hazard (risk of death) at time t, h0(t) is the baseline hazard , βn is the regression coefficient of the clinical features, βm is the regression coefficient of the omics features, zn represents the different clinical features, and zm represents the different omics features. The T, N, and M stages and age at initial diagnosis were used as the clinical features. The omics features consisted of different types of omics data, including three features of the single types of omics data and one feature of three integrated types of omics data. Cox PH models with different combinations of clinical features and omics features were constructed. The prognostic performance of each model was compared to verify that the integration of clinical data and multi-omics data provided the best prognostic performance.A multi-covariate Cox PH model was used for the prognostic analysis, and different combinations of clinical features and omics features were used as covariates in the Cox PH model. The formula of the Cox PH model used in our study was as follows:p-value greater than 0.05 suggests no time-variation issue [p-value of less than 0.05 indicates that the covariate in the model has a significant effect [https://cran.r-project.org/web/packages/rms/index.html) and survC1 (https://cran.r-project.org/web/packages/survC1/index.html) R package.We focused on the discriminative performance of the prognostic model. Therefore, the concordance index (C-index) was used as the main evaluation metric, with a C-index of 1 indicating perfect discrimination and a C-index 0.5 indicating a random guess. We used Harrell’s C-index during the model construction procedure to select the model with best overall discriminative performance . Then Unon issue . In addit effect . These mConsidering the moderate sample size, we preferred a bootstrapping analysis, which revealed the ability to optimize the estimation of the C-index caused by overfitting, to generate the bias-corrected C-index rather than the original Harrell’s C-index –39. RandOverall, eight combinations of distance methods, linkage methods and cluster numbers were identified for clustering of different types of omics data while combining with clinical features, including two combinations for gene expression, three combinations for DNA methylation and three combinations for miRNA expression. Cluster parameters of the three types of omics data for prognostic models with different covariates are listed in Table Overall, we included four clinical covariates, three omics covariates and one integrated omics covariate. The four clinical covariates were used as clinical covariates; alongside the three omics covariates, they formed seven different combinations. The prognostic model we proposed was constructed with clinical, gene expression, DNA methylation and miRNA expression as covariates. In addition, we constructed three models with clinical covariates and two types of omics data, three models with clinical covariates and one type of omics data, four models with clinical covariates or one type of omics data alone, and one model with clinical covariates and C-o-C results.First, all models passed the PH assumptions test, as shown in Fig. p-values of the likelihood ratio test, the score test and the Wald test are plotted in Fig. The p values of the likelihood ratio tests between different prognostic models. The results of the Wilcoxon signed-rank test for the differences in the C-indexes of different models further confirmed that our prognostic model showed the best prognostic performance, as summarized in Table We calculated 3-year and 5-year Uno’s C-index values for all 12 prognostic models. In addition, as the median survival time of the dataset was approximately 26 months, the 2-year Uno’s C-index was also inferenced. The 2 year Uno’s C-indexes without and with cross-validation are shown in Fig. In summary, these results clearly indicate that among all models, our prognostic model showed the best discriminative performance. Both the likelihood ratio test and the Wilcoxon signed-rank test of difference between distributions of C-index of these two models suggest our prognostic model owned better prognostic performance. In addition, paired comparisons of prognostic model with only clinical covariates, the best prognostic model with clinical covariates and one type of omics data and the best prognostic model with clinical covariates and two types of omics data suggested that the more types of omics data that were used, the better the prognostic performance was.We investigated how the performance of our prognostic model changed after one of the covariates was removed. The same evaluation procedures were used to compare the performance of our prognostic model with that of the model with one covariate removed.The results of the comparison of the Uno’s C-index without cross-validation and the likelihood ratio test are summarized in Table The results of the comparison of Harrell’s C-index with bootstrapping and Uno’s C-index with cross-validation are summarized in Table Based on current analysis pipeline and results, removing miRNA expression from our prognostic model may have a minimal effect on short-term prognostic prediction, but would have a significant effect on long-term prognostic prediction. In addition, removal of T stage had a positive prognostic effect on our model.Considering the overall evaluation, T stage was the least important covariate of the clinical covariates in our prognostic model, while miRNA expression was the least important covariate of the omics covariates in our prognostic model.First, we successfully performed unsupervised clustering to aggregate patients in our dataset into different groups based on different types of omics data profiles. According to the evaluation of the single-covariate Cox PH model shown in Fig. p-values of the three tests of the prognostic model were improved, especially the score test, which is consistent with the findings from a previous study [https://bioconductor.org/packages/release/bioc/html/edgeR.html) and the Database for Annotation, Visualization and Integrated Discovery database . In total, 57 differentially expressed genes were identified, as listed in Additional file Then, we integrated the clinical data and used different combinations of omics data by fitting a multi-covariate Cox PH model, and the results confirmed that we had successfully integrated the clinical features, gene expression, DNA methylation and miRNA expression to improve the colon cancer prognostic performance. The evaluation metrics used on the models based on different covariates suggested that combining clinical features and all three types of omics data could offer the best prognostic performance. The us study . In TablOur study indicated that combining clinical covariates with omics data could improve prognostic performance, and that the more types of omics data that were used, the better the improvement was. Compared to a previous study conducted by Zhao et al. , our stuThis study highlights our ongoing work. Colon cancer prognosis may benefit from the integration of clinical and omics features. However, cooperation among biomedical scientists, oncologists and biologists is necessary for implementing the practical application of a personalized prognostic model. A platform that offers integrative analyses of clinical and omics data and management and storage of clinical and omics data is essential. Our current study approach offers a fundamental framework for this type of platform. In addition, our approach can be easily extended to other types of omics data, such as copy number variations or somatic mutations. We aim to build a classifier based on the identified cluster labels of different omics data and develop a web-based tool for practical application in the coming future. Moreover, we aim to identify patients with good versus poor prognoses with integrated clinical and multi-omics features.However, our current study was limited by the TCGA-COAD datasets and the use of only three types of omics data and four clinical features. Therefore, we hope to collect omics data from patient follow-ups performed at the hospital. Such collection may provide access to more clinical features, including treatment, larger samples and more types of omics data for analysis. Such advancements may validate the extensibility of this integrative analytic approach. The finding that miRNA expression had a negligible prognostic contribution to the short-term prognosis indicated that we might need a better analysis method for miRNA expression. Based on Uno’s C-index, the contribution of T stage to our prognostic model was questionable, though Harrell’s C-index suggested that T stage had a positive contribution to our prognostic model. These inverse results might be due to the limitation of Uno’s C-index, as there were no events for different T stages except for the T3 stage before the 2-year time point, as shown in Additional file The results suggested that the current C-o-C approach may not be suitable for integrating multi-omics data in our current study , 29, 30.In conclusion, we applied a pilot integrative prognostic analysis of colon cancer based on clinical features, gene expression, DNA methylation and miRNA expression data. This approach successfully increased the predictive performance of an integrated prognostic model of colon cancer patients compared to the performance achieved using clinical features alone, and all types of omics data had significant effects on the prognostic model. Therefore, our study has the potential to help colon cancer oncologists treat patients more accurately.Additional file 1. List of prior knowledge. List of gene symbols for gene expression, gene symbols for DNA methylation and miRNA ids for miRNA expression used as prior knowledge.Additional file 2: Additional table for the detailed concordance of different prognostic models. Table S1. Origin concordance and bias-corrected concordance of the Cox models with different covariates.Additional file 3. Differentially expressed genes identified based on cluster labels of gene expression. Differentially expressed genes identified based on cluster labels of gene expression.Additional file 4. KM-curve of the dataset based on the covariate T stage. KM-curve of the dataset based on the covariate T stage.Additional file 5. KM-curve of the dataset based on the covariate age. KM-curve of the dataset based on the covariate age.