PRSS1, PNLIP, CLPS, and/or CELA3A). Fourteen additional highly expressed genes from other tissues also indicate contamination. Sample contamination is strongly associated with a sample being sequenced on the same day as a tissue that natively expresses those genes. Discrepant SNPs across four contaminating genes validate the contamination. Low-level contamination affects ~40% of samples and leads to numerous eQTL assignments in inappropriate tissues among these 18 genes. This type of contamination occurs widely, impacting bulk and single cell (scRNA-seq) data set analysis. In conclusion, highly expressed, tissue-enriched genes basally contaminate GTEx and other datasets impacting analyses.A challenge of next generation sequencing is read contamination. We use Genotype-Tissue Expression (GTEx) datasets and technical metadata along with RNA-seq datasets from other studies to understand factors that contribute to contamination. Here we report, of 48 analyzed tissues in GTEx, 26 have variant co-expression clusters of four highly expressed and pancreas-enriched genes ( Sample contamination has been reported in high throughput RNA sequencing. Here the authors analyze the RNA sequencing data from the Genotype-Tissue Expression project and describe how highly expressed, tissue specific genes contaminate across samples, which is corroborated in other data sets. These data sources are generally publicly available and have been used by hundreds of researchers for secondary analyses of high impact6.The rise of next-generation sequencing has allowed for unparalleled data generation for a variety of nucleic acid studies including RNA expression. As cost per basepair decreases, more large-scale transcriptome projects can be performed that will inform on tissue expression patterns in health and disease9. Another challenge of high-throughput RNA-Seq is contamination, leading to the presence of sequence data within a data set of one sample that originates from a separate sample. This contamination can come from many different aspects of the modern sequencing process, such as human error, machine or equipment contamination, intrinsic preparation and sequencing errors, and computational errors, including errors that can occur based on the multiplexing methods used 11. For single cell RNA-Seq (scRNA-Seq), doublets or multiplets (2+ cells partitioned together) can cause cross-contamination resulting in expression hybrids12. Compared with RNA-Seq, contamination has been better characterized for DNA sequencing projects15.Limitations exist for all –omics technologies, including RNA sequencing (RNA-Seq). Issues of hybridization biases, library preparation biases, and computational biases such as positional fragment bias are known limitations of RNA-Seq experiments1. It is an ongoing project with over 700 individuals and 11,000 tissue samples. GTEx identifies eQTLs by associating genotypes called from whole-genome sequencing with gene expression levels obtained from bulk RNA-Seq. GTEx has made their RNA-Seq, phenotype, genotype, and technical data available for public access with permission.The Genotype-Tissue Expression project (GTEx) aims to create a large publicly available database of tissue-specific expression quantitative trait loci (eQTL) from over 40 tissuesIn an analysis of variation in the GTEx RNA-Seq data (V7), we detect unexpected sources of variation that we hypothesize are likely contaminating sequence reads found at low, but variable levels across different tissues. Herein, we describe how we identify the source of contamination and establish basal rates of contamination in the GTEx bulk RNA-Seq data. We further demonstrate the universality of highly expressed genes contaminating other samples.16. We used DESeq2 variance stabilizing transformation (VST) to normalize read counts from 11,340 samples17. Then we filtered genes in each tissue keeping those with a mean transformed count >5. The median number of genes above the expression threshold was 17,729 with the highest and lowest gene counts being 23,930 and 13,807 in the testis and whole blood, respectively. As previously described, we correlated and hierarchically clustered variable genes (more than four variance across samples) for all tissues with >70 samples (N = 48) in the GTEx data set V716. Our algorithm identified multiple gene clusters per tissue, based on their Kendall’s tau correlations  identified in 26 of the 48 tissues that did not have an intuitive biological explanation. These genes are highly expressed and pancreas acinar-cell specific18. To identify other highly expressed tissue-enriched genes appearing variably in other samples, we cross-referenced a list of tissue-enriched proteins generated by the Human Protein Atlas (HPA) to the GTEx transcripts per million (TPM) data  was significantly higher in non-pancreas samples if they were sequenced on the same day as a pancreas sample  was on average 0.175 higher if the tissue was isolated on the same day as a pancreas sample , but on average 0.863 higher if the tissue was sequenced on the same day as a pancreas sample  . The same model as above showed only sequencing a tissue on the same day as a esophagus sample to be significant (p = 5e-260)  and 190 in esophagus associated gene coding sequences . As a comparison group, 287 variants were investigated in two control gene coding sequences (GAPDH and RAB7A) that have near ubiquitous expression across all tissues. Of 1010 variants obtained from the combined VCF files, 11 had some degree of allelic heterogeneity (Table To prove that pancreas/esophagus transcripts were contaminating from other (non-self) samples we identified incongruencies between a person’s genotype (from DNA data) and the genotype in matching loci in the pancreas/esophagus contaminated RNA-Seq samples. We required both the individuals’ DNA genotype and RNA-Seq files from contaminating tissues, in order to account for both RNA editing and preferential allele expression. Based on these sample requirements and limited by available raw sequencing files, we identified 11 contaminated tissues to evaluate. For each, we obtained and processed their raw RNA-Seq FASTQ sequences to identify nucleotide variants in both their contaminated tissues and their matched pancreas or esophagus tissue (depending on the gene source of contamination). In addition, we used the GTEx filtered variant call format (VCF) file from the individual’s sequenced DNA to further establish their SNP allele patterns. Across all tissues, 533 SNPs, rare variants, and private variants, were investigated in pancreas associated gene coding sequences . One individual (arbitrarily GTEX-1) was homozygous C at rs7956809 in both its DNA (VCF file) and matched esophagus (RNA-Seq FASTQ data) , located in ta) Fig. . HoweverWe further investigated the relationship between the GTEX-1 fibroblast sample and the GTEX-2 esophagus sample finding no clear connection. The two samples were sequenced on different machines and in different flow cells. Of some interest, the sequencing sample adapters (molecular indexes) were similar  and “outside of hospital”  deaths  was fully corrected for by PEER factors  factor to correct for possible confoundersths Fig. . This refor Fig. . Indeed,Using the GTEx eQTL browser, we identified 72 tissues reported as having significant eQTLs for the 18 genes listed in Table 24. Neither study sequenced the pituitary, which is the organ with the highest levels of prolactin (PRL) expression , contaminating reads are minimal and do not correlate. This was the opposite of what was observed in GTEx adipose tissue , suggested otherwise.We then investigated this type of highly expressed gene contamination in scRNA-Seq. Although doublets are well-characterized, and barcode swapping has been reported, other forms of contamination are not well-described29. From the pancreas sample, insulin (INS) represented a highly expressed, cell type-enriched gene that is exclusive to beta cells  = 1671). INS expression was modestly elevated in endothelial cells and mesenchymal cells from the same data set (GSE84133), but is absent in endothelial and mesenchymal cells from the other data sets in which beta cells were not sequenced  from normal pancreas, metastatic melanoma, and head and neck squamous cell carcinoma were usedced Fig. . Contami31. We gathered endothelial cell data from nine tissues and generated tSNE plots of this data based on the inclusion of either 6 or 10 principal components (PCs) . However, both owing to contamination being noted in some samples that are sequenced a few days apart from a possible contaminating source and the SNP-based evidence, we suspect the majority of the contamination occurred during library preparation rather than the sequencing itself. Library preparation dates were not publicly documented.We found that contamination is best linked to the date of sequencing for both pancreas genes and esophagus genes . This has been demonstrated in scRNA-Seq, and seems to occur more often on a HiSeq 4000 platform26.At the stage of library preparation or sequencing where our data indicate most of the contamination occurred, there are multiple steps that could be implicated. The library preparation was completed automatically in 96-well plates with a mix of tissues and individuals to prevent batch effects33.Specific to this study, GTEx samples were run on either HiSeq 2000 or 2500 machines in high output mode using cbot clustering. The flowcells were not patterned, and therefore not prone to index-swapping . GTEx’s use of custom i7 (later Illumina kit-based) dual indices also reduced the amount of index hopping that can occur34. If true, this would occur before the library preparation and sequencing steps, would cause only a modest expression of the gene, and would not be detectable by current doublet correcting mechanisms35. This method of contamination might have implications for Human Cell Atlas studies, as we demonstrated it clearly affects cross-tissue comparisons of common cell types in Tabula Muris36.Contamination of scRNA-Seq, may or may not be occurring by yet an additional mechanism. Ambient RNA, from disrupted (“broken”) cells may be a major source of contamination, as recently describedUsing other bulk RNA-Seq data sets with similar sequencing methods and scRNA-Seq data sets, with different preparation methods, we validated that it is predominately contamination, not low-level transcription, which results in non-zero expression values in inappropriate tissues/cells Fig. . These ePRSS1 (TPM > 100). Thus, for many uses of GTEx data, this level is irrelevant. However, for experiments that involve differential expression profiling, these genes will repeatedly appear due to their variable levels of contamination. We additionally note that the GTEx standard normalization pipeline using PEER factors did not entirely eliminate this source of variation and numerous eQTLs that were identified for the 18 genes described herein were located in incorrect tissues (>40%). We caution that our results do not suggest that one should design a study in which sample type  is perfectly confounded with library preparation or sequencing date. In fact, it was only because the GTEx study sequenced samples from different individuals on the same date that we were able to definitively show that contamination between samples occurred.So how big is the contamination problem? It depends on how one uses the data. Fortunately, in the GTEx data, the levels are overall low with only 2.85% of samples having relatively high levels of 37. Our findings call these reports into question. The extent of cross-contamination, where one laboratories’ samples are prepped and sequenced at the same time as a different laboratories’ unrelated samples through a university core sequencing facility or sequencing company is unknown, but likely frequent39. The xenomiR story, that rice miRNAs are found in human blood through dietary means40, was shown to result from library preparation contamination42. For scRNA-Seq, this contamination falsely implies some gene expression is ubiquitous across cells, which influences computational heterogeneity analyses of cell types, beyond the known challenges of other biological and technical artifacts43. Also, our work highlights the fact that work flows must be considered carefully in very-low DNA mutation detection analysis in clinical cancer samples as samples with higher tumor burdens may contaminate samples with lower tumor burdens and falsely suggest treatment approaches45. In particular, GTEx data are available in many outlets, including the UCSC Genome Browser. Variable, low-level expression of PRSS1, CELA3A, and others may falsely intrigue researchers, particularly within the reported eQTLs.In other scenarios, this basal contamination may be more worrisome. Many publications have reported rare, but variable gene expression in their samples claiming their importance or disease-related behaviorsWe described low-level, variable expression contamination in the GTEx RNA-Seq data set. This variation was most noticeable for 18 highly expressed, tissue-enriched genes and strongly correlates with the library preparation and sequencing of the samples. Similar contamination was observed in other bulk and scRNA-Seq data sets, suggesting a universality to this type of contamination. Evaluating low-level variable gene expression in RNA sequencing data sets must be performed with precaution and awareness of potential sample contamination.All human data were publicly available or used with approval of the GTEx consortium. Consent was obtained by those studies.https://gtexportal.org/home/datasets), along with the de-identified sample annotations (GTEx_v7_Annotations_SampleAttributesDS.txt). The FASTQ files of the tissue samples and the VCF files of appropriate individuals were downloaded from dbGaP (phs000424.v7.p2) with the required permissions.The gene read counts of the RNA-Seq GTEx version 7 data set  were downloaded from the GTEx Portal .We obtained the HPA tissue-enriched genes by downloading a CSV file from this filtered site (https://jhubiostatistics.shinyapps.io/recount/) to find data sets where only one tissue was sequenced, with the exception of the ovary sample, which was obtained from the Illumina BodyMap 46. Owing to our inability to find salivary gland data we used prostate tissue, the third highest expressing tissue for ZG16B. Heart (ERP009437), adipose tissue (SRP053101), lung (SRP032833), prostate (SRP003611), stomach (ERP010889), and colon (SRP029880) TPMs were acquired through Recount2 and were selected for normal samples. The testis data were downloaded from GEO (GSE103905). Vagina and ovary FASTQ data were downloaded from SRA run selector (GSE68229 and GSE30611). FASTQ files were mapped to the Genome Reference Consortium Human Build 38 (GRCh38) using the software HISAT2 version 2.1.047. The output SAM files were turned into BAM files and indexed using samtools version 1.949. Assembly was completed using StringTie v1.3.4d and StringTie TPMs were used from the output51.Using the GTEx portal (v7) we noted the TPM for the second highest expressing tissue with >60 samples. We then used GEO, SRA Run Selector, and Recount2  to reduce noise and lessen the inflated effect of low expressing genes on correlations.The acquired raw read counts were segmented into separate tissue subsets (48 tissues with ≥70 samples each) and their read counts were normalized using the VST feature in DESeq2 version 1.22.1 in R version 3.6.1All analyses were completed in R version 3.6.1. In each tissue, a threshold of a >4 variance of VST normalized read counts was used as our cutoff for highly variable transcripts. These genes were then clustered using hierarchal clustering on a distance generated by 1−Kendall’s rank-correlation coefficient. A tau critical value was calculated based on the number of samples and genes expressed. The correlation-based dendrogram was cut to produce gene clusters with average within cluster correlation of at least the tau critical value.Normalized expression scores allow one to summarize the expression of a gene cluster in a sample by the average normalized score of the genes in that cluster. These normalized scores were calculated by subtracting the mean expression and dividing by the median absolute deviation of the expression values for each gene across all samples within a given tissue. The equation is as follows, where Equation 1PRSS1, CELA3A, PNLIP, and CLPS, across all tissues. An “esophagus” normalized score was defined as the average normalized score using genes KRT4 and KRT13 in a given tissue. All normalized scores were calculated independently for each tissue, to account for tissue-specific between-sample variation in the expression of these genes.A “pancreas” contamination score was defined as the average normalized score using the genes N = 6258). For the model, tissue was used as a covariate along with “pancreas sequencing day” and “pancreas nucleic acid isolation day” were coded binarily as True/False based on whether a given non-pancreas sample was sequenced or underwent RNA isolation on the same day as a pancreas sample. Our model identified an association between the pancreas contamination score with pancreas sequencing day , pancreas nucleic acid isolation day , and tissue as covariates, with the subject IDs included as a random effect. The equation for our model is below where i, and Linear models were used to find associations between nucleic acid isolation or sequencing on the same day as a potential contaminating tissue (source of gene expression). These analyses were completed in R version 3.6.1 using the lmer function in package lme4 (v1.1-21). In the linear mixed models we used all available tissues with > 40 samples not sequenced on the same day as a pancreas sample . The model used esophagus nucleic acid isolation day , esophagus sequencing day , and tissue as covariates along with subject IDs as a random effect on the esophagus contamination score.We repeated the above analysis with the esophagus mucosa, coding shared sequencing day and shared nucleic acid isolation day similarly. In the linear mixed model we used all tissues with a sample size > 40 samples not sequenced on the same day as the esophagus mucosa , were manually selected using IGV as a reference. Using the tool bam-readcount version 0.8.0 in combination with a Python 3.6.2 script, a list of RNA-Seq and genomic incongruencies were generated for the acquired sample BAM files.Base pair incongruency analysis required a contaminated tissue expression FASTQ, a native tissue expression FASTQ, and the individual’s VCF file. FASTQ files were mapped to the Genome Reference Consortium Human Build 37 (hg19) using the software HISAT2 version 2.1.0N = 427). The data underwent trimmed mean of m-values (TMM) normalization and filtering out of lowly expressed genes (< 0.1 TPM for 80% or more of the samples) before running PEER to identify potential confounders21. Following GTEx’s pipeline , we then performed an inverse-normal transformation on the expression values for each gene in order to reduce the effect of outliers22. Normalized scores for each gene are based on TMM-normalization, inverse-normal transformation, and scaling/centering at zero using the base R scale function. A linear regression was performed, with either raw inverse-normalized expression values or expression normalized scores (after correcting for PEER factors) as the outcomes and same-day pancreas (/esophagus) sequencing status as the predictor. Beta estimates represent how many standard deviations greater the mean expression of a gene is when samples are sequenced on the same day as a contaminating tissue, even when accounting for variance explained by 60 PEER factors.We obtained the GTEx RNA-Seq data set from lung . eQTLs were identified by tissue association and conservatively placed in one of three categories: appropriate expression, possible expression, and inappropriate expression. The appropriateness of expression in any tissue was based on the evaluation of TPM levels in the tissue and immunohistochemistry staining patterns as noted in the HPA54.We obtained and tallied eQTL reports for the 18 genes in Table 19. The RNA Atlas was acquired by downloading their raw RNA-seq counts from Gene Expression Omnibus (GSE120795)23. The HPA RNA-Seq was performed across 27 tissues and the RNA Atlas was across 20 tissues. We filtered samples down to the shared tissues of liver, lung, pancreas, prostate, skin, and stomach as well as the potential sources of PRL contamination of pituitary, placenta, and uterus. Only sun exposed skin was used for GTEx analysis. TPM RNA-seq data from the METSIM study was downloaded through the GEO (GSE135134)25.Using the R package recount version 1.8.2, we downloaded HPA RNA-Seq data, accession ERP00361329. All three sets had median read counts/cell >100,000. The processed read counts from the three studies were obtained from the Gene Expression Omnibus (GEO) . For each study we used the supplied cell type information to identify endothelial and mesenchymal-type cells. Mesenchymal cells were labeled as “activated stellate” (GSE84133), “fibroblast” (GSE103322), and “cancer-associated fibroblast” (GSE72056). The GSE103322 and GSE72056 read data were generated using Smart-Seq2 on a NextSeq 500 instrument55. Raw FASTQ data sets were normalized using RSEM to calculate Ei,j = log2, where TPMi,j refers to transcript-per-million for gene i in sample j56. This was done to create TP10K values to account for single cell read depth and the addition of 1 was to limit the effect of 0 s in downstream analysis27. GSE84133 was generated using the inDrop method and sequenced on a HiSeq 2500 instrument57. Human islets were isolated using a modified Ricordi method, which includes collagenase, mechanical agitation, pumping, and centrifugation58. To make single cells, the islets were then centrifuged twice at 250 rpm and treated with TrypLE Express before mechanical dispersion with a P100 pipette, followed by centrifugation at 500 rpm. The two cancer sample libraries  were generated using Smart-Seq2 and sequenced on a NextSeq 500 instrument55. Read counts were scaled using TP10K to equalize expression level counts to the other two studies. The calculation of the percent contamination of endothelial cells by INS was the ratio of average normalized INS read counts per endothelial cell divided by the normalized average read count in beta cells and excluding the nine cells with >1000 TP10K.We identified three publications with human scRNA-Seq data sets that all contained endothelial and mesenchymal cellshttps://figshare.com/articles/Robject_files_for_tissues_processed_by_Seurat/5821263 (visited on 8/8/19)31. We used the R package Seurat (v3.1.1) and selected endothelial cells from nine organs/tissues  based on the annotation of the seurat object in the metadata slot cell_ontology_class labeled “endothelial cell”59. tSNE plots were generated based on the top 6 or top 10 PCs. The average number of PCs used by Tabula Muris per tissue/organ to create tSNE maps was 12 and the median was 10. Images of protein expression of contaminating pancreas genes were obtained from the HPA54.We obtained the Tabula Muris read data from Further information on research design is available in the Supplementary InformationPeer Review FileReporting SummaryDescription of Additional Supplementary FilesSupplementary Data1Supplementary Data 2Supplementary Data 3Supplementary Data 4Supplementary Data 5Supplementary Data 6


Oncogenesis and cancer can arise as a consequence of a wide range of genomic aberrations including mutations, copy number alterations, expression changes and epigenetic modifications encompassing multiple omics layers. Integrating genomic, transcriptomic, proteomic and epigenomic datasets via multi-omics analysis provides the opportunity to derive a deeper and holistic understanding of the development and progression of cancer. There are two primary approaches to integrating multi-omics data: multi-staged (focused on identifying genes driving cancer) and meta-dimensional . A number of ready-to-use bioinformatics tools are available to perform both multi-staged and meta-dimensional integration of multi-omics data. In this study, we compared nine different integration tools using real and simulated cancer datasets. The performance of the multi-staged integration tools were assessed at the gene, function and pathway levels, while meta-dimensional integration tools were assessed based on the sample classification performance. Additionally, we discuss the influence of factors such as data representation, sample size, signal and noise on multi-omics data integration. Our results provide current and much needed guidance regarding selection and use of the most appropriate and best performing multi-omics integration tools. Alterations in the genetic make-up of a cell, such as mutations, copy number aberrations and epigenetic changes, can all drive the development of cancer . Copy nuDespite these well-understood mechanisms, two major challenges remain for the studies of oncogenesis. The first challenge is the identification of genes and aberrations that serve as drivers for disease development, which is complicated by the evolutionary nature of the disease and the presence of passenger aberrations that do not contribute to disease development . IdentifMulti-omics data integration allows the joint analysis of multiple omics data types to provide a global view of the biological system and offers insights into the nature of the interactions between the different dataset layers. As cancer is a heterogeneous genetic disease, it is imperative that such multi-omics data integrations be performed in order to fully appreciate the complex inter-layer regulatory interactions governing the development and progression of this disease. Decreasing costs and technological advancement have enabled the multi-platform sequencing of thousands of tumours by consortia such as International Cancer Genome Consortium  and The cis relationships), such as the effects of copy number aberration on the expression of the affected gene. Tools that employ multi-staged integration approaches perform gene-centric integration in order to uncover the potential driver genes and help understand the effects of the interplay between different gene regulatory levels on oncogenesis and cancer progression. There are numerous ready-to-use tools available which employ this approach, including CNAmet [Multi-omics integration strategies commonly employed to address the aforesaid challenges can broadly be classified into two types: multi-staged and meta-dimensional integrative approaches . Multi-sg CNAmet , iGC [13g CNAmet , Oncodrig CNAmet  and Methg CNAmet .Meta-dimensional integration (or simultaneous integration) is based on the hypothesis that interactions between the multiple biomolecular layers are non-hierarchical and complex . This apGiven the increasing availability of multi-omics datasets and the continuous development of analytical approaches and objectives, novel multi-omics integration tools are continuously being developed, and selection of the most appropriate and best performing integration tool from the numerous tools available can be a tedious and time-consuming task. Although previous comparative studies have investigated the performance of multi-omics integration tools, they have focused only on either the multi-staged or the meta-dimensional approaches and often failed to investigate the effects of gene (or feature) selection , influenFive multi-staged  and four meta-dimensional  tools were compared in this study and are summarized in the following sections and in t-tests [This study assessed the performance of the CNAmet, iGC, PLRS, Oncodrive-CIS and MethylMix tools designed to carry out the multi-staged integration. A generalized workflow implemented in these tools to achieve integration and identify driver genes involves quantification of the strength of the cause and effect association between aberration in the DNA (copy number and methylation aberration) and mRNA expression at the gene level, using a statistical approach followed by correction for multiple hypothesis testing in order to identify the statistically significant genes. The statistical approaches include variations of linear regression techniques , Studentt-tests  and corrt-tests .While iGC, PLRS and Oncodrive-CIS tools attempt to discover the genes that have dysregulated expression due to copy number change by integrating copy number and gene expression data, the MethylMix tool integrates methylation and gene expression data to identify the genes for which expression is influenced by methylation status. The CNAmet tool is capable of modelling both the combined as well as independent effects of a given gene’s copy number and methylation status on its expression. In this study, we used the CNAmet tool to identify the independent effects of the copy number and methylation aberrations on the gene expression. The meta-dimensional integration tools assessed in this study include the SNF, BCC, iClusterPlus and mixOmics. These tools employ one or more mathematical techniques including matrix factorization, correlation, Bayesian, network and transformation , 32, forThe SNF tool integrates multi-omics data using transformation-based network fusions. Briefly, each omics data type is transformed into a network wherein the nodes represent samples and weighted edges represent the similarity between the samples. Next, these individual networks are fused into a single network using message-passing theory method , where hhttp://firebrowse.org/) databases, were used in this study. These cancers were selected as they have varying sample size and are widely studied. The following omics data were downloaded for each cancer  transformed RSEM [Gene expression: Gene-level transcription estimates, as in logmed RSEM  normalizDNA methylation: Gene-level mean beta values were downloaded from the firebrowse database.Data processing for each cancer multi-omics dataset involved selection of tumour samples common across all omics, removal of genes with NA, zero variance and near-zero variance  from gene expression omics data using caret R package  followedMeta-dimensional tool analyses were performed using 2 real multi-omics cancer datasets and 16 simulated datasets.Hepatocellular carcinoma (HPC) dataset: This dataset  compriseGlioblastoma (GBM) dataset: Protein expression measured using reverse phase protein array technology, gene expression measured using AffyU133a array, DNA methylation data measured using Illumina Infinium HumanMethylation27 platform and phenotype information for glioblastoma tumour samples were downloaded from UCSC Xena database. Common samples across all omics data and having information on the clinical subtype were selected for analysis. Feature selection was based on inter-sample variance, with the top 25% variable genes and probes selected . The finThe miRNA gene expression data, DNA methylation data and the top 1000 inter-sample variable genes from mRNA gene expression data from the TCGA breast cancer dataset  Supplem were usei,sdi) + NormalNormal  + NormalNormal  + NormalNormal . Additionally, independent Gaussian noise  was added to the features profiles. The profiles of the features in the samples belonging to each class with added independent noise are given below: Class 1 − To assess the ability of the tools to combine the patterns identified in individual omics data types, certain classes in two or all of the omics data types of the simulated datasets B, C and D were simulated to be a hidden class . SimulatTo understand the effect of sample size on the performance of these tools, all the datasets were recreated with 150 samples, with each class containing 50 samples. The 150-sample datasets will be referred to as Large Group, while the previously created 60-sample datasets will be referred to as Small Group. To assess the influence of noise, Gaussian noisy features with means equal to that of the genes selected randomly from the real dataset and with a standard deviation of 2 were added to the datasets in both the Small and Large Groups and are referred to as Small Noisy Group and Large Noisy Group, respectively. As these (noisy) features have a constant standard deviation across all three classes, they do not have the information to discriminate the classes and hence, are noisy. For all the datasets, 100 noisy features were added to Omics data 1, while 20 noisy features were added to Omics data 2 and 3.http://mixomics.org/mixdiablo/case-study-tcga/). The runtimes of these tools were recorded as the time taken to execute the integration function in each tool, and the data pre-processing time prior to running these tools was not included. We have provided the R code to reproduce the pre-processing, variable selection, creation of simulated datasets, running of tools and performance evaluation presented in this study on GitHub .The CNAmet, iGC, PLRS, MethylMix, SNF and BCC tools were run in R on a 64-bit Ubuntu desktop computer. The iClusterPlus and mixOmics tools were also run in R but on a high performance computing Linux cluster hosted by Queensland University of Technology, Brisbane, Australia. The Oncodrive-CIS tool, which is available as a Python script, was executed via the terminal (command line) in Python version 2.7 on a 64-bit Ubuntu desktop. All the tools were run using their default parameter settings obtained from the respective manuals whereas the parameter values for the mixOmics tool were obtained from their case study  and the tools integrating methylation intensity and gene expression data (MethylMix and CNAmet tools).http://www.cbioportal.org/cancer_gene_list.jsp) [The tools were evaluated based on the overlap estimates of genes, function terms (GO terms) and pathways with that of gold standard cancer gene list—cBioPortal cancer gene list   which coThe selected gene result from each tool and the cancer gene list were enriched with GO terms for Biological Process (BP), Cellular Component (CC) and Molecular Function (MF) categories using clusterProfiler R package  and the The meta-dimensional tools were evaluated based on the sample classification performance in various scenarios: (i) binary and multi-class classification, (ii) presence of hidden classes, (iii) small and large sample sizes and (iv) noise in small and large sample sizes. The HPC dataset was used for the binary classification analysis in which the sample classes were Normal and PVTT/Tumour, with the latter class formed by merging the PVTT and Tumour samples. For multi-class classification analysis, the HPC and GBM datasets were used. The sample classes for the HPC dataset were Normal, PVTT and Tumour, while for the GBM dataset the classes were Classical, Mesenchymal, Neural and Proneural. As the mixOmics tool performs supervised classification, the samples in both the real and simulated datasets were divided into training (75%) and test (25%) datasets prior to analysis. The tool performance was assessed using the classification results obtained for the test datasets.The sample classification powers of the meta-dimensional integration tools were assessed using the following metrics:1 scores and accuracies. Refer to In the following section, we have highlighted the performance of meta-dimensional tools mainly based on their average FThe performance of the copy number and gene expression integration tools in the mesothelioma, pancreatic cancer, colon cancer and melanoma datasets were tested at three evaluation levels: gene, function and pathway. The 3000 most informative and statistically significant unique genes identified by these tools were utilized for the evaluations. For these gene lists and the CCGL, significant GO function terms and pathways were identified. For each evaluation, overlaps of the results from the tools with the CCGL in each dataset were identified and are shown in The PLRS tool achieved the highest overlap with CCGL in 11/20 overlap tests, demonstrating the highest sensitivity compared to other tools. The Oncodrive-CIS and iGC tools achieved the highest overlap in 6/20 and 2/20 tests, respectively. All tools displayed poor performance for the mesothelioma dataset. The comparatively small sample size of the mesothelioma dataset (87 samples) likely reduced the power to identify significant genes and subsequently the significant GO function terms and pathways associated with the disease. Overall, all tools showed an increasing trend in the performance with increasing sample size of the datasets.t-test, PLRS uses piecewise linear regression splines and Oncodrive-CIS uses Z-test. The high similarity between these statistical methods resulted in the observed high overlap of implicated genes. However, in general highest congruence was observed between PLRS and iGC tools , colon cancer (67–74%) and melanoma (60–66%) datasets . The goaGC tools –S4.Similar to copy number and gene expression integration tools, methylation and gene expression integration tools were also assessed at three levels—gene, function and pathway. The top 3000 methylation-driven genes were selected from the results provided by the MethylMix and CNAmet tools to perform 20 overlap tests with CCGL. The MethylMix tool, which integrates the omics datasets using a continuous data and linear regression approach, outperformed the CNAmet tool, which integrates using a categorical data and signal-to-noise ratio approach, in 18/20 overlap tests . The sha1 scores in multi-class classification are preferable as tissue-specific tumours often comprise more than two molecular subtypes that may differ in prognosis and treatment strategies.The goal of binary and multi-class classification using meta-dimensional integration is to accurately identify the different tumour subtypes or patient types using multi-omics datasets. Binary classification involves segregating the samples into two subtypes/classes, while multi-class classification involves identifying more than two classes. Tools achieving high accuracies and F1 scores and accuracies among all the tools. The performance of all the tools was significantly better in the binary classification compared to the multi-class classification analyses. In the former analysis, all the tools were capable of retrieving the two classes with high precision and recall. However, during the multi-class classification using the HPC dataset, only the mixOmics tool could delineate between the samples from Tumour and PVTT classes. The poor precision and recall in classifying between these two classes could have been due to the high heterogeneity in these classes, as reported by Yang et al. [The performances of the tools during binary and multi-class classification tested using the HPC and GBM datasets is shown in g et al. . In the g et al.  caused lg et al. . Hence, g et al. . Multi-ciClusterPlus is a popular integration tool that has been used for multi-omics analysis of numerous cancers , 54, 55.Independent analyses of multiple omics data types may identify subtypes that do not conform to one another as a subtype identified in one omics data may be undetectable in another omics data type. Such subtypes are referred to as the hidden classes, and the meta-dimensional tools aim to identify these classes through joint analysis of multi-omics data. The ability of the tools to retrieve a hidden class was tested using datasets A, B, C and D from the Small Group, where all datasets contain 60 samples and no noisy features. The dataset A that contained no hidden classes is simple compared to the other datasets, which are complex due to the presence of hidden classes in two or more omics data types.The accuracies obtained by the tools for these datasets are shown in 1 scores achieved by these tools for individual classes in each dataset provide insights into the identification of the classes. A high F1 score, particularly for a hidden class, implies effective combining of shared and complementary subtype information from multiple omics data types to retrieve the underlying class. Our results showed that the recoverability of a hidden class by the tools was lower in comparison to a class that was not hidden. For example, the F1 scores achieved by the tools for Class 1 in datasets B and D were lower than that in datasets A and C (1 score < 0.5) in dataset C. However, in dataset D, the SNF and mixOmics tools were capable of identifying this class (F1 score > 0.5). For dataset D, which contained non-overlapping hidden classes in all three omics data types, only the mixOmics tool retrieved all three classes (F1 > 0.5), while the SNF and iClusterPlus tools retrieved two out of the three classes. The BCC tool performed the worst for complex datasets, such as datasets C and D, producing only a single cluster solution where all samples belonged to one class.The F A and C . The tooThe influence of sample size on the performance of the tools was assessed by comparing their performances using datasets from Small and Large Groups. Both groups contain the A, B, C and D datasets but differ only by sample size. The Small Group contains datasets with 60 samples whereas the Large Group contains datasets with 150 samples. The iClusterPlus and mixOmics tools accomplished more accurate classifications in the majority of datasets compared to SNF and BCC tools in the Large Group . When coAs noise is inherent in biological data, it is crucial to identify the robustness of the selected classifiers while processing noisy data. To test the robustness of these tools, we recreated the A, B, C and D datasets in Small and Large Groups with noisy features and referred to as Small Noisy Group and Large Noisy Group, respectively. Gaussian noisy features (mean = 0 and standard deviation = 2) were added to each of the omics data types in the datasets. As noisy features do not have profiles similar to that of the informative features, their presence confounds the classification process. A good classifier identifies the underlying classification despite the noisy features.1 scores for datasets in the Small Noisy Group compared to that in the Small Group, demonstrating the lack of robustness to noise in these tools. The mixOmics tool was the most affected by the presence of noise compared to the other tools by showing the highest decrease of 43.75% for dataset C.First, we compared the accuracies achieved by the tools for the datasets in Small Group and Small Noisy Group . The SNF1 scores of all the tools for datasets in Large Noisy Group compared to that in Large Group were minimal or non-existent  identification of cancer driver genes and (ii) tumour subtype discovery or patient/sample classification.In general, a multi-omics integration study encompasses the following steps: feature selection , selection of an integration tool, processing of input data to fit the tool specifications, running the tool with optimized parameter settings, filtering the output  and finally, interpretation of results. Identification of an optimal integration tool will minimize the challenges encountered at different steps and hence enable researchers to meet the integration objectives. Assessing the performance of the tools in identifying the cancer-associated genes, functions and pathways, and classification of tumour samples, are critical factors to be considered while choosing an efficient tool.For multi-staged integration studies which seek to identify the cancer driving genes, our comparisons revealed that regression-based integration methods are most effective, identifying most cancer-associated genes, GO terms and pathways. For such analyses, we therefore recommend the PLRS and MethylMix tools as they implement regression-based integration and outperformed the other tools compared in this study. The regression-based integration approaches, which utilize continuous values for copy number or methylation status, prevent the loss of information due to the use of non-integer or sharp cut-off values. Using ordinal values can lead to reduced variability between the samples and hence miss the true relationship between the omics datasets analysed . Cancer cis interactions between gene expression and copy number/methylation of genes and many cancer genes are implicated in the disease through multiple mechanisms, it is possible to miss the cancer genes in the tool results if small cut-offs, such as top 200 [As multi-staged integration tools focus on  top 200 , are use top 200 . Our ana top 200 . Prior tcis relationship between copy number or methylation and gene expression of genes and do not address the trans relationship, which is the regulation of distant genes, as has been reported in many studies [To potentially improve the accuracy in identification of the true associations, we recommend the tools that utilize additional biological information during integration. The PLRS tool (the best performing method) uses both segmented copy number and thresholded copy number, as well as call probabilities when available. These data provide information on the heterogeneity among the samples as well as the precision of the copy number call, leading to the identification of statistically reliable associations that drive cancer . Use of  studies , 61.To classify the cancer subtypes using meta-dimensional integration, we recommend the mixOmics tool as it provided the most accurate classifications for the majority of the real and simulated datasets. Alternatively, when the sample labels are not known prior to integration, we recommend iClusterPlus as it provided most accurate classification for the majority of the simulated datasets. However, it should be noted that the performance of iClusterPlus tool can reduce in the presence of a large number of uninformative features . Hence, feature selection prior to integration is critical. The iClusterPlus tool is also computationally intensive, and the runtime increases with an increase in the number of input features. A recent version of the iClusterPlus tool, iClusterBayes , endeavoet al. [An important characteristic of a good meta-dimensional integrative tool is to identify the subtypes that are common or unique to different omics datasets and effectively link this information to provide the best classification. Our study showed that the meta-dimensional tools have difficulties in identifying a class that is unidentifiable in the majority of the omics data types used for integration and classification. For the Tumour and PVTT classes in the HPC dataset and Neural class in GBM dataset, which are not clearly distinguished in the omics data types , all tooet al.  recommenOur analyses showed that noise can have a substantial negative impact on the classification. Hence, use of advanced feature selection techniques is critical  prior toAll tools were run using default parameters to avoid bias. However, optimization of the parameters might have improved the performance of the tools, such as the SNF tool . Only coOur results provide much needed advice regarding selection and use of multi-omics integration tools. Tools implementing the regression-based approach for multi-staged integration effectively capture the association between the omics data types. The potential for false positive findings from multi-staged integration can be minimized using control datasets, feature selection prior to integration and/or filtering down the output after integration. For the meta-dimensional integration tools, the use of larger sample sizes enables more accurate identification of tumour and patient groups as well as diminishing the adverse effect of noise. Additional factors such as feature selection and parameter optimization are also critical and can improve the integration for both multi-staged and meta-dimensional integration tools. Although the multi-omics tools investigated improved the classification and feature identification that were undetectable in some individual omics data types, they still have much room for improvement with respect to their performance and ease of use.SupplementaryInformation_bbz121Click here for additional data file.


Benchmarking RNA-seq differential expression analysis methods using spike-in and simulated RNA-seq data has often yielded inconsistent results. The spike-in data, which were generated from the same bulk RNA sample, only represent technical variability, making the test results less reliable. We compared the performance of 12 differential expression analysis methods for RNA-seq data, including recent variants in widely used software packages, using both RNA spike-in and simulation data for negative binomial (NB) model. Performance of edgeR, DESeq2, and ROTS was particularly different between the two benchmark tests. Then, each method was tested under most extensive simulation conditions especially demonstrating the large impacts of proportion, dispersion, and balance of differentially expressed (DE) genes. DESeq2, a robust version of edgeR (edgeR.rb), voom with TMM normalization (voom.tmm) and sample weights (voom.sw) showed an overall good performance regardless of presence of outliers and proportion of DE genes. The performance of RNA-seq DE gene analysis methods substantially depended on the benchmark used. Based on the simulation results, suitable methods were suggested under various test conditions. High-throughput cDNA sequencing (RNA-seq) has been commonly used for transcriptome analysis for the last decade . WhereasSeveral comparative studies for DE analysis methods have already been conducted to suggest suitable methods for different test conditions. Some have used real RNA-seq data to test false positive control, compared the reproducibility of DE analysis results using a reduced number of samples, and analyzed similarity of results between different methods –18. OtheIn this study, (1) we compared the performance of 12 DE analysis methods using both RNA spike-in  and NB-bijR], i = 1,2,…,n, j = 1,2,…,(m1+m2) composed of non-negative integers where n is the number of genes analyzed, and m1 and m2 are sample sizes for the test and control groups, respectively.In this article, only two-sample group comparison and ‘gene’-based analysis will be considered to focus on the performance of DE analysis methods. We consider a raw read count matrix R = [ijR of the ith gene in the jth sample has often been modeled by discrete counting probability, such as Poisson or NB distribution. Poisson distribution is only able to address the experimental variation between replicates caused by random sampling and is appropriate for modeling the counts obtained from technical replicates [A read count plicates , 10. Howijμ≥0 and dispersion iφ≥0, and its variance is represented as ijμ = jμis, where js is scaling factor (or depth) of sample j, and iμ is the mean expression of gene i. If iφ is zero, NB becomes a Poisson distribution where mean and variance have the same value. Zero-inflated NB distribution was considered to model the sparse single-cell RNA-seq data [To incorporate the increased variability in biological replicates, NB distribution has been widely used for modeling RNA-seq read counts –13, 24. seq data , 26, butWe selected such DE analysis methods for RNA-seq data that exhibited relatively good performances in a previous comparative study , 18, 21.Performance of the aforementioned DE analysis methods was compared using their area under receiver operating curve (AUC), true positive rate (TPR), true false discovery rate (FDR), and false positive counts (FPCs). Here, true FDR is the proportion of non-DE genes among the significant genes and indicates the extent of reliability of predicted DE genes. We calculated true FDR only when five or more significant genes were detected in each method. FPC is the number of significant genes detected from the datasets where no DE genes were included (Type I error). Compared to the previous simulation study , we usedSequencing Quality Control (SEQC) data pertained to the Microarray Quality Control (MAQC) study and contained replicated samples of universal human body reference RNA and human brain reference RNA with spike-in controls. We used the same count data as used in a previous DE analysis benchmark study (GEO accession GSE49712)   Different proportion of DE genes: pDE = 0.27%, 5%, 10%, 30% or 60% DE genes were included in the simulation datasets. 0.27% was considered to compare with the results for SEQC data, and 60% DE genes represent, for example, complex disease conditions such as cancer vs. normal conditions.(SS) Sample sizes of three or ten were used in each condition.(OL1) Random outliers: each read count could become an outlier with a probability of 0.05 or less. An outlier count was regenerated with five to ten times larger mean value.(OL2) Low-quality sample: dispersions of 30% of samples were increased fivefold.(OL3) Composite condition of 3 and 4: 30% of samples were generated with fivefold increased dispersions, and the 3% of read counts in the other samples were regenerated as a random outlier. Different balances of up and downregulated genes: Bal = 50%, 70% or 90% of DE genes were upregulated, while the remainder were downregulated.(Disp) The same or different dispersion values between conditions were assigned for each gene.(EF) Weak effect sizes  were used.These conditions include diverse situations encountered in RNA-seq data analysis and the test results can provide a guideline for selecting an appropriate method in each situation. Our tests have the following several differences from the previous simulation study by Soneson and Delorenzi :Four additional edgeR methods, two additional limma methods, PoissonSeq, DESeq2, and ROTS were included in our study.Wider range of DE gene proportions from 0.27% to 60% was tested, whereas only 10% and 33% DE genes were considered previously. We demonstrated that both low (0.27% or 5%) and high (60%) proportions of DE genes have substantial effects on performance of some methods . The low pDE (e.g. 5%) represents the case, such as gene-knock-out experiment where only a small number of genes are expected to be differentially expressed, whereas the high pDE  represents the case of complex disease, such as cancer as compared to normal samples where the majority of genes are expected to be differentially expressed .Effects of dispersions and mean counts on DE analysis methods were analyzed. We specifically showed that dispersions have great impacts on both the relative and absolute performances of each method.In our study, weak effect sizes (1.2 or larger fold changes) were used to analyze subtle differences in performance, whereas 1.5 or larger fold changes were used previously.http://gdac.broadinstitute.org) (denoted as KIRC) and inbred mouse dataset (denoted as Bottomly) [independent samples denote those obtained from different individuals with possibly different genetic backgrounds, such as KIRC. Independent samples typically exhibit large dispersion values [https://github.com/unistbig/compareDEtools.To simulate read counts, we first estimated mean and dispersion parameters for each gene from two RNA-seq datasets with different parameter distributions: TCGA Kidney Renal Clear Cell Carcinoma/normal dataset, which is available from GDAC (URL: ottomly) . KIRC isn values . We addin values , respectWe tested the 13 DE analysis methods including two different implementations of baySeq for both, SEQC spike-in data and KIRC-based simulation data and compared their AUCs. The AUCs for SEQC data are shown in Hence, we performed further simulation tests to investigate the cause of such differences, by adjusting dispersion parameters and pDE. First, the dispersions of KIRC simulation data were decreased to the level of SEQC data (22.5 times lower) and AUCs were compared . BecauseOverall, this experiment shows the extremely small dispersion and proportion of DE genes in SEQC data may not provide fully reliable evaluation of DE analysis methods and simulation tests based on parameters for biological replicates should also be considered. A similar concern of benchmarking SEQC data has also been raised when normalizing RNA-seq data . Thus, wThe test results are depicted as boxplots in Read counts of 10,000 genes were generated using the parameters estimated from KIRC data. Three or ten samples were generated in each sample group. To begin with, we tested for the ‘base’ condition where no outliers and no low quality samples were included, and same dispersions between sample conditions and equal proportions of up and down regulated DE genes  were assumed. Then, we further tested for alternative test conditions. Under the base condition for three samples, DESeq2 performed best . DiffereIn presence of outliers (OL1 = 5%), the best AUCs were observed for DESeq2 and edgeR.rb followed by voom.tmm, and voom.sw when three samples were used. Especially with ten samples, edgeR.rb surpassed every other method, and SAMseq was the second best. DESeq2, voom methods, and ROTS showed similarly good AUCs. We further compared DESeq2 and edgeR.rb by additionally introducing lower proportion of outliers (1% and 3%) to the base condition . edgeR.rwithout any normalization or transformation was much less affected by the varying proportion and imbalance of DE genes and showed outperforming AUCs when Bal ≥ 0.7 and pDE ≥ 0.3 , but they were dramatically improved when pDE or sample size was increased. True FDRs were considerably increased as random outlier counts were introduced . They weWe then compared the FPCs in each case by randomly sampling two groups from the same sample group ; in otheWe also tested the composite conditions of random outliers and increased dispersions (OL3) for a large number of samples (10 and 30 samples in each condition) . The oveThe same conditions were tested using the parameters estimated from Bottomly data. It is known that large mean counts and low dispersions increase the statistical power of DE analysis , 22. DesTo compare the impacts of mean and dispersion parameters, we additionally created simulation data with hybrid parameters by combining KIRC means and Bottomly dispersions (denoted as mKdB) and Bottomly means and KIRC dispersions (denoted as mBdK), respectively. As shown in We have shown through simulation tests that dispersion values, proportion of DE genes, and outlier counts largely impact the performance of each method. However, it is not always clear which condition in For outlier counts, DESeq2 automatically truncates them or simply remove genes that contained a potential outlier count depending on the sample size. However, for edgeR package, the user has to choose a method between edgeR and its robust version. If the given data are highly noisy, edgeR.rb may be most effective. For moderately noisy data, both DESeq2 and edgeR.rb perform better than their original methods. Recently developed methods can help detect outlier counts in RNA-seq data , 43.Lastly, the proportion of DE genes are not known to the user. However, we can roughly estimate the proportion from the test condition. In complex disease, such as cancer, the majority of genes are differentially expressed compared to the normal condition, where the condition ≥30% can be considered. If a single gene with potentially limited effects was knocked out, much less proportion of DE genes are expected compared to complex disease conditions. In ambiguous cases, several DE analysis methods can be tested together to roughly estimate DE genes from significantly detected genes.We also compared the 12 DE analysis methods by analyzing the two RNA-seq datasets (KIRC and Bottomly) that exhibited different distribution of parameters. As the true DE genes are not known for real data, we only compared the number of significantly detected genes and FPCs. KIRC contained 72 normal and 72 cancer samples. Bottomly contained ten C57BL/6J strain and eleven DBA/2J strain samples. Three, five, ten, and twenty (KIRC only) sample sizes were considered. After removing the genes with average read count of less than ten, 16,621 genes and 8,550 genes remained for KIRC and Bottomly, respectively. We used q-value < 0.1 to select DE genes. To compare FPCs, we used only the normal sample group for KIRC, where two contrasting groups were randomly sampled for DE analysis. The same analysis was done for the DBA/2J sample group of Bottomly. Each experiment was repeated fifty times and the results are represented as boxplots in Most results showed similar patterns between the two datasets except that PoissonSeq detected relatively small number of DE genes for Bottomly. SAMseq and voom.sw detected the largest numbers of DE genes, while DESeq.pc and ROTS detected the smallest numbers of DE genes for both datasets. We assumed that KIRC included many DE genes between cancer and normal conditions , and found high similarity between the number of detected DE genes , true positive rate (TPR) and true false discovery rate (true FDR) are shown. Three and five samples were used in simulation tests. (A) Test results for SEQC spike-in data (5 samples). 0.25, 1.5, and 2 fold changes were introduced to DE genes. (B) KIRC based simulation. Not less than 1.5 or 1.3 fold changes are introduced to 0.27% of 10,000 genes for 3 and 5 samples, respectively. (C) Two conditions are different from B: fixed fold changes of 0.625, 1.15, and 1.3 were introduced to DE genes, and dispersions were also lowered to SEQC level (22.5 times) (D) The same condition as A but DE proportion was increased to 1%.(EPS)Click here for additional data file.S2 FigDispersions between test and control groups are different (A) Three samples per each group and four different proportions of DE genes  were used. (B) Sample size is increased to ten.  Each read count can be a random outlier with 5% probability.(EPS)Click here for additional data file.S3 FigNo random outlier was included, and same dispersions were used between conditions. (A) Three samples per each group and Bal = 70% were used for four different proportions DE genes . (B) Bal = 90%.  The same condition as A-B, but sample size is increased to ten.(EPS)Click here for additional data file.S4 Fig(A) 1% probability for each read count to be an outlier was applied. Three samples per each group and four different proportions DE genes  were used. (B) The same condition as A, but the sample size is increased to ten.  The same as A and B, but the outlier probability of 3% is used.(EPS)Click here for additional data file.S5 Fig30% of each sample group have fivefold increased dispersion parameters and 3% of counts in the other samples were regenerated as outliers. Ten and 30 samples were used in each sample group for (A) and (B), respectively.(EPS)Click here for additional data file.S6 FigArea under ROC curve (AUC), true positive rate (TPR) and true false discovery rate (true FDR) are shown. Three and ten samples were used in each sample group for  and , respectively.  Same dispersion between test and control groups, Bal = 50%, four different proportions DE genes  were used.  Random outlier counts: the same condition as A-B but each read count can be a random outlier with 5% probability.  Low quality samples: the same condition as  but one and three samples of each sample group have fivefold increased dispersion parameters.(EPS)Click here for additional data file.S7 FigDispersions between test and control groups are different (A) Three samples per each group and four different proportions of DE genes  were used. (B) The sample size is increased to ten.  The same condition as A-B but each read count can be a random outlier with 5% probability.(EPS)Click here for additional data file.S8 FigSame dispersions were used between conditions. (A) Three samples per each group and Bal = 70% were used for four different proportions of DE genes . (B) Bal = 90%.  Sample size is increased to ten.(EPS)Click here for additional data file.S9 FigThree and ten samples were used in each sample group for  and , respectively.  Same dispersions between test and control groups are assumed.  Random outlier counts: the same condition as A-B but each read count can be a random outlier with 5% probability.  Low quality samples: the same condition as  but one and three samples of each sample group have fivefold increased dispersion parameters.(EPS)Click here for additional data file.S10 FigSame dispersions between test and control groups are used. (A) Three samples per each group and four different proportions of DE genes  were used. (B) Sample size is increased to ten.  Random outlier counts: the same condition as A-B, but each read count can be a random outlier with 5% probability.(EPS)Click here for additional data file.S11 FigSame dispersions between test and control groups are used. (A) Three samples per each group and four different proportions of DE genes  were used. (B) Sample size is increased to ten.  Random outlier counts: the same condition as A-B, but each read count can be a random outlier with 5% probability.(EPS)Click here for additional data file.S12 FigPrincipal Component Analysis for (A) KIRC and (B) Bottomly datasets.(EPS)Click here for additional data file.S13 FigHierarchical clustering of DE genes based on spearman rank correlation. The union of top 3000 and 300 DE genes obtained from 12 DE analysis methods were clustered between different methods based on spearman rank correlation for (A) KIRC and (B) Bottomly datasets, respectively.(EPS)Click here for additional data file.


Non-targeted cytotoxics with anticancer activity are often developed through preclinical stages using response criteria observed in cell lines and xenografts. A panel of the NCI-60 cell lines is frequently the first line to define tumor types that are optimally responsive. Open data on the gene expression of the NCI-60 cell lines, provides a unique opportunity to add another dimension to the preclinical development of such drugs by interrogating correlations with gene expression patterns. Machine learning can be used to reduce the complexity of whole genome gene expression patterns to derive manageable signatures of response. Application of machine learning in early phases of preclinical development is likely to allow a better positioning and ultimate clinical success of molecules. LP-184 is a highly potent novel alkylating agent where the preclinical development is being guided by a dedicated machine learning-derived response signature. We show the feasibility and the accuracy of such a signature of response by accurately predicting the response to LP-184 validated using wet lab derived IC50s on a panel of cell lines.We applied our proprietary RADR® platform to an NCI-60 discovery dataset encompassing LP-184 IC50s and publicly available gene expression data. We used multiple feature selection layers followed by the XGBoost regression model and reduced the complexity of 20,000 gene expression values to generate a 16-gene signature leading to the identification of a set of predictive candidate biomarkers which form an LP-184 response gene signature. We further validated this signature and predicted response to an additional panel of cell lines. Considering fold change differences and correlation between actual and predicted LP-184 IC50 values as validation performance measures, we obtained 86% accuracy at four-fold cut-off, and a strong (r = 0.70) and significant  correlation between actual and predicted LP-184 sensitivity. In agreement with the perceived mechanism of action of LP-184, PTGR1 emerged as the top weighted gene.Integration of a machine learning-derived signature of response with in vitro assessment of LP-184 efficacy facilitated the derivation of manageable yet robust biomarkers which can be used to predict drug sensitivity with high accuracy and clinical value. Chemotherapeutic drugs target cancer-specific characteristics, such as alterations in DNA repair, which has complicated roles in oncogenesis and tumor progression. Illudins are a class of sesquiterpene secondary metabolites initially researched in the 1950s due to potent cancer cytotoxicity, particularly illudins S and M , 3. The Mechanisms of illudin and acylfulvene cytotoxicity include DNA alkylation, resulting in cell-cycle arrest and apoptosis –9, generIn addition to key mechanistic knowledge on tumor specificity related to acylfulvene activation by PTGR1 and tumor resistance determined by TC-NER pathway components, there remain additional pathways known to be associated with their activity. We are therefore not fully aware of all possible mechanisms of action in tumor types under consideration for treatment with acylfulvenes, which may impede our capability to leverage this mechanistic knowledge into ultimate therapeutic decisions. An additional barrier remains in determining precisely how raw mechanistic knowledge is eventually translated into a useful measurement guiding therapeutic choices. Promising classes of compounds have not been further studied in specific subsets of cancers due to lack of a drug-tumor interaction knowledge base. These challenges will be more pronounced with novel acylfulvene derivatives that, while possibly showing appealing therapeutic indices, lack extensive mechanistic studies. A solution to the dual problems of having unknown determinants of therapeutic potential, and untested predictive value with currently known mechanisms, is the development of a signature designed to make these predictions using machine learning approaches. Machine learning approaches for gene signature development save significant amount of time from preliminary data generation to clinical application with rapid iterations allowing for hypothesis building and testing.A gene signature can be used to predict expected outcomes of different patients or predict the likelihood of a successful therapeutic response with a given drug. With regard to cancer, development of a signature is possible through use of a combination of clinical features, gene expression, genomic alterations, and other data types. A gene signature can be similar to a biomarker in its characteristics, and, in some cases, a signature may be regarded as a larger set of biomarkers. Like a biomarker, a signature aims to have robust and reliable prognostic value, but both the methods used for derivation of a gene signature, and for computing its predictive power, will often differ. While traditional biomarkers might be based on extensive study and expert analysis with a cancer type, machine learning is ideally suited to the task of uncovering features that can successfully predict outcomes of interest, using various data sources. Data on mutations and gene expression are often used to develop signatures, and several studies have concluded that the mRNA expression signature of a tumor often has the most predictive power. With the recent expansion of high-quality databases in cancer and the reduced cost of doing whole-genome analyses in patients, there is great potential for improving accuracy and utility of gene signatures used in clinical predictions. With potential acylfulvene derivative therapies, machine learning methods have potential to identify the tumor characteristics that may be missed in human studies, and provide better predictions of the tumors where they will be therapeutically effective. Traditional gene signature of drug response could take several years after millions of dollars of laboratory resources, and may even fail to have clinical significance. Early integration of machine learning is critical to compress the time, risk and cost of exploring and achieving optimal positioning of compounds in cancers that can be particularly sensitive.N-hydroxy-N-(methylacylfulvene)urea. LP-184 is an alkylating prodrug that damages DNA and has broad antitumor activity that is preferentially activated in cancer cells expressing certain biomarkers. Another acylfulvene analog, Irofulven (hydroxymethylacylfulvene) [LP-184 is a next generation acylfulvene (AF) drug class member related to Illudin S, a toxin that occurs naturally in certain mushrooms . The chefulvene) , has beefulvene) –18. IrofApplications of machine-learning to genomic analysis in cancer have covered diagnosis, prognosis, and therapeutic response predictions –25. EarlIn this paper we describe the leveraging of LP-184 sensitivity and cancer cell line gene expression from the NCI-60 panel as training data to establish a set of candidate biomarkers forming an LP-184-specific gene signature predictive of response in solid tumors. Using RADR® (Response Algorithm for Drug positioning and Rescue), we implemented an optimized set and sequence of gene feature filtering and machine learning algorithms to derive a gene expression-based signature of response to LP-184. This signature was further validated in vitro and in silico in independent testing datasets, demonstrating its predictive value in identifying suitable tumor models as well as sensitive tumor types for advanced experimental validation and clinical translation. Notably, the signature includes the anticipated candidate biomarker PTGR1 that is highly correlated with LP-184 sensitivity and implicated in the induction of bioactivation of LP-184 which provides a strong biological rationale for clinical application. Further, our machine learning approach is able to rank the relative variable importance of each of the signature genes which would be useful in developing a scalable clinical companion diagnostic assay associated with LP-184 response. Using machine learning, starting from a pool of 18,000 genes, we have a reduced set of only 16 genes forming an LP-184 response signature. This provides a manageable biomarker panel as a basis for patient stratification and clinical advancement of LP-184.50  and RNA-seq  data from 59 cancer cell lines in the NCI-60 database . Using t50 values (as a regression), or a sensitivity classification. Gene input either used all genes, or a subset of genes determined by a Pearson’s correlation threshold, which was further filtered by selecting subsets from Pathway enrichment analysis or gene-network analysis  that covered those having either the low- or high-end of PTGR1 expression in microarray data. Using expression data from CCLE, our model predicted sensitivity of these cell lines, and, as shown in Fig. We further analyzed the relationship of gene expression and the signature to LP-184 activity. Hierarchical clustering of top 5 known sensitive and top 5 known resistant cell lines from the NCI-60 panel clearly divided the cells (with one exception) between sensitive and resistant, and an obvious pattern with the top-20 most correlated genes emerged   , and the full 1000 lines of the CCLE  . Highly significant differences were also observed with PTPN14 (p < 3.2e−13), SDC4 (p < 3.2e−13), CKAP4 (p < 3e−12), ADAM9 (p < 3.3e−11), and PIK3CG (p < 2.3 e−10). Some of these may have novel roles in relation to cancer and/or the LP-184 mechanism of action, while some have been previously reported to play roles in cancer malignancy – e.g., PIK3CG which is tightly associated with growth and migration in several cancers.We further looked at the association of the top-20 sensitivity correlates and signature genes, comparing differential expression in the top-25 ranked cell lines classified by either predicted sensitivity and resistance, which corresponds to cell lines having a range of IC50 from 14.3 nM to 35.9 μM, respectively. Individually looking at expression of the top-20 sensitivity gene correlates, we observed a stark contrast between groups. Each gene was significantly differentially expressed in resistant and sensitive lines using a Wilcoxon test  had p values > 0.05. This might indicate that performance of the predictive model on the blind set of 37 CCLE lines , while some are general purpose . We used a Driverless AI platform from H2o.ai—the automatic machine learning platform, as it performs automated model tuning and facilitates derivation of suitable models capable of making predictions on independent datasets. As described in the Methods section, data were prepared and submitted to build the model. It ran a total of 199 tuning and 372 feature evolution models. The final model was a “bagged ensemble (pasting) of 1 LightGBM model across 3 folds”. Using the “Score on Another Dataset” option from the H2o interface, we obtained predicted IC50 values on the test set of 37 CCLE cell lines using the expression values of the same number of 1899 genes used in the training set. Table In summary, our key findings demonstrate that the machine learning derived signature from the best trained and tuned model—with a selected set of optimized parameters—predicted LP-184 sensitivity with > 85% accuracy. It is important to note that when using a relatively small training dataset i.e. NCI-60 screening data, it is challenging to derive a robust predictive gene signature. Here we have shown that our proprietary A.I. platform RADR® can be used effectively to derive a gene signature predictive of LP-184 response from NCI-60 data. This 16-gene signature appeared to be optimized for drug prediction outcomes on the entire (~ 1000 cell lines) CCLE dataset, based on the strong similarity of correlated gene pathways with the genes correlated to experimentally determined sensitivity. Toxicity of acylfulvenes requires intracellular reduction that is believed to be mainly mediated by PTGR1, which is highly expressed in a variety of cancer models and patient samples, and appears as a part of various prognostic or predictive models in cancer that are not necessarily drug-specific –40. ThisHere we have shown that LP-184, a next generation acylfulvene, exhibits broadly potent nanomolar toxicity in a spectrum of cancer types. LP-184 is a next-generation acylfulvene that is related to illudin S . AcylfulMachine learning in cancer research has used data from tumor and/or patient features to predict clinical outcomes, ranging from general prognosis to response to a specific therapeutic option. This could be applied to improve conventional biomarker and diagnostic measures, and provide better informed, personalized patient-specific recommendations. Using biomarkers to inform chemotherapeutic choices has remained difficult, even in cases where the measured marker is known to be related to a drug target –48. ThisHere we first attempted a variety of machine learning approaches, including SVM, ENR, RF, deep neural networks, that were evaluated alone or in combination. We found our initial training performed best when using a gene-correlation filter step, followed by initial selection with a random-forest method, and a final round of feature selection. This method generated the gene signature shown in Fig. Application of machine-learning to clinical cancer analysis has been promising but faces a number of challenges in practice. Depending on the training data available, there may be a limited sample quantity, which may also have substantial confounding properties—such as may occur when using a small number of observations from patients with diverse backgrounds. Training with certain predictive models also encounters challenges with regard to the type of prediction used. For example, predicting patient outcomes with a certain chemotherapeutic has been confounded with selection patterns that identify patients having generally better outcomes, regardless of drug or therapy. It has been found that in vitro measurement of drug sensitivity, by IC50 measurement, adequately reflects actual tumor sensitivity [To train a model predicting sensitivity from this data, we used the baseline transcriptome of cancer cells to train a model that selects which features  together will determine the level of sensitivity to LP-184. Various features from cancer, including other forms of whole-genome omic measurements, have potential to be used as predictive markers. We compared other potential feature inputs, from DNA somatic mutations, copy-number variation, methylation, and others, for their capability to produce an accurate prediction model of LP-184 sensitivity in a test set, and found RNA expression to provide the greatest accuracy. This parallels what has been recently reported in cancer-sensitivity predictive model research , 52. We Our model determined PTGR1 to be the most important feature for LP-184 sensitivity prediction, with PTGR1 alone having accounted for over half of all feature importance in the model  of CCLE cell lines to LP-184. The downloaded files from CellMinerCDB platform are (1) data_nci60_xai.txt and (2) data_ccle_exp.txt.We have used NCI CellMinerCDB platform to download the microarray log2 expression data of NCI-60 and CCLE (Cancer Cell Line Encyclopedia). The LP-184 IC50 values were derived by NCI-60 screening of the compound. From the CCLE set of cell lines, we selected 37 cell lines as the blind testing set in which we generated LP-184 ICThe LP-184 sensitivity and gene expression data were available for a total of 58 cell lines. Of these, 6 blood cancer cell lines were excluded from solid tumor specific data analyses. The resulting data served as training data from which the LP-184 response gene signature was derived. There were a total of 23,059 genes and 52 cell lines in the training data.50) values. The blind testing data were generated by extracting expression data for only those 37 cell lines from CCLE. There were a total of 19,851 genes and 37 cell lines in the blind testing data.37 cell lines from CCLE were selected to be tested at REPROCELL USA, Inc. for LP-184 activity in order to derive drug sensitivity . Hence, we removed those cell lines. There were a total of 19,851 genes and 1000 cell lines in the CCLE data, that we used to identify potential indications for LP-184.Human tumor cell lines were grown in RPMI 1640 medium containing 5% fetal bovine serum and 2 mM L-glutamine. Cells were inoculated into 96 well microtiter plates in 100 μL at plating densities ranging from 5,000 to 40,000 cells/well depending on the doubling time of individual cell lines. After cell inoculation, the microtiter plates were incubated at 37° C, 5% CO2, 95% air and 100% relative humidity for 24 h prior to addition of LP-184. After 24 h, LP-184 from a 10 mM stock solution in DMSO was added in triplicate wells to the desired test concentrations in the range of 1 nM to 10 uM. Following drug addition, the plates were incubated for an additional 48 h at 37 °C, 5% CO2, 95% air, and 100% relative humidity. The assay was terminated by the addition of cold TCA. Cells were fixed in situ by the gentle addition of 50 μl of cold 50% (w/v) TCA  and incubated for 60 min at 4 °C. The supernatant was discarded, and the plates were washed five times with tap water and air dried. Sulforhodamine B (SRB) solution (100 μl) at 0.4% (w/v) in 1% acetic acid was added to each well, and plates are incubated for 10 min at room temperature. After staining, unbound dye was removed by washing five times with 1% acetic acid and the plates were air dried. Bound stain was subsequently solubilized with 10 mM trizma base, and the absorbance was read on an automated plate reader at a wavelength of 515 nm. Using the absorbance measurements, Inhibitory Concentration of 50% (IC50), which is the drug concentration resulting in a 50% reduction in the net protein increase (as measured by SRB staining) in control cells during the drug incubation, was calculated.p value <  = 0.05) correlated both positively and negatively with LP-184 IC50 values using a Pearson correlation method. Pathway analysis (using Pathcards database covering 3840 pathways) and gene network analysis  were performed separately on the set of correlated genes in order to retain the genes that appear to be biologically relevant. In pathway analysis, the known drug target pathway is used to narrow down the correlated genes and to output only the genes that are directly or indirectly related to the drug's target pathway. The gene network analysis outputs the network of genes that are connected with correlated genes. Lastly, another inductive learning algorithm, termed the ‘Relief algorithm’ ranks and assigns weights to these filtered genes based on drug sensitivity (IC50). The highly ranked genes (top 100) are used in the subsequent prediction module. The prediction module runs the ‘Boruta’ feature selection algorithm on these ranked genes to obtain the significant gene features . In order to reduce the set of potential response predictor genes even further, based on the ranking, genes get eliminated one by one, and the model gets built to compare the prediction accuracy on the test set. This step performs 5-time repeated tenfold cross validation using RMSE as the validation metric. Prediction accuracy is calculated by dividing the number of cell lines with predicted LP-184 IC50 within two or four fold from actual LP-184 IC50, by the total number of cell lines in the blind set and expressed as a fraction ranging from 0 to 1. Performance is evaluated based on the sum of the ‘Two-Fold’ and ‘Four-Fold’ testing accuracy during the iterative loop. The final model and signature set chosen is the one with the highest accuracy sum. The 16 gene signature was selected as the best signature. Flow chart of the entire process is given in the Additional file We have used Lantern’s proprietary RADR® platform in order to step by step identify and reduce the statistically significant and biologically relevant genes and finally derive both robust and manageable biomarkers. The platform first identifies the genes that are significantly . In order to evaluate the prediction, we compared the outcome with NCI-60 data for which the LP-184 sensitivity is known.We have used the expression levels of 16 signature genes on the entire CCLE data described in the “CCLE data” section of the data processing step and tuned the model to predict LP-184 sensitivity . In order to select the highly correlated genes, we first performed the Pearson correlation (between the gene expression and IC50 from 52 cell lines) and selected the genes that are statistically significant  followed by matching the genes that are present in the blind testing data set. In the resulting training data, there were 52 cell lines and 1899 genes. The Experiment was conducted with train data and default regression model settings from the H2o platform.Additional file 1: Figure S1. Flow chart describing statistical methodology.Additional file 2: Figure S2. Comparison of the different model accuracy on blind test data.Additional file 3: Table S1. List of tumor types covered in the test set of 1000 CCLE cell lines.Additional file 4: Table S2. List of 52 cell lines with experimental IC50 (Molar).Additional file 5: Table S3. Enrichment analysis of top 200 highly correlated genes from 58 NCI-60 cell lines using experimental IC50.Additional file 6: Table S4. Enrichment analysis of top 200 highly correlated genes from 1000 CCLE data using predicted IC50.