Public repositories of large-scale omics datasets represent a valuable resource for researchers. In fact, data re-analysis can either answer novel questions or provide critical data able to complement in-house experiments. However, despite the development of standards for the compilation of metadata, the identification and organization of samples still constitutes a major bottleneck hampering data reuse. We introduce Onassis, an R package within the Bioconductor environment providing key functionalities of Natural Language Processing (NLP) tools. Leveraging biomedical ontologies, Onassis greatly simplifies the association of samples from large-scale repositories to their representation in terms of ontology-based annotations. Moreover, through the use of semantic similarity measures, Onassis hierarchically organizes the datasets of interest, thus supporting the semantically aware analysis of the corresponding omics data. In conclusion, Onassis leverages NLP techniques, biomedical ontologies, and the R statistical framework, to identify, relate, and analyze datasets from public repositories. The tool was tested on various large-scale datasets, including compendia of gene expression, histone marks, and DNA methylation, illustrating how it can facilitate the integrative analysis of various omics data. However, most published studies covering few regulatory factors and/or epigenetic marks profiled over a limited number of biological conditions rely on newly generated omics data, thus ignoring the fact that massive collections of publicly available data could be exploited to study those very same factors and their interactions in identical or very similar conditions. Data re-use is mainly hindered by technical challenges in data retrieval, analysis and integration. In particular, biologists have little familiarity with the necessary tools3, which are seldom user friendly. Moreover, despite the introduction of standards such as MINSEQE and MIAME4, sample metadata are typically highly heterogeneous thus making samples retrieval complicated. Available integrative approaches are oriented to the development of public accessible interfaces5 where users can query databases for their data of interest or download pre-compiled datasets. However, these are typically limited in terms of the available datasets, strongly depend on the willingness and ability of the developers to maintain and update the system and the available data. Ultimately, these solutions offer a fixed, pre-defined set of functionalities, and as a result, they are suboptimal in terms of flexibility. Instead, federated databases use semantic web technologies5 to represent their data in RDF (Resource Description Format) to be simultaneously queried using the language SPARQL. In this case the drawback is that knowledge of a very specific query language is needed to make proper queries. More advanced methods, like heterogeneous information networks6, provide data structures and analytical methods to handle data heterogeneity in integrative analyses. However, these methods require solid computational skills to correctly develop and mine such networks. Finally, key repositories, like GEO, are not represented in RDF format and exploratory queries can be tricky.The plummeting cost of high-throughput sequencing experiments has led to a rapid accumulation of omics datasets in public repositories. For example, more than 600 K RNA-seq and 80 K ChIP-seq samples are currently available in Gene Expression omnibus , a tool that leverages NLP techniques, biomedical ontologies, and the R statistical framework, to identify, relate, and analyze datasets from public repositories. First, with a process known as named entity recognition, Onassis associates free textual descriptions of publicly available samples to the concepts belonging to ontologies where entities of a given domain of interest are associated to a standard representation. Second, semantic similarity measures, quantifying the similarity of ontology concepts based on the underlying structure of ontological relationships, can be used by Onassis to determine the closeness of samples based on their ontological annotations. Finally, statistical tests with any custom function can be effortlessly applied within Onassis to analyze the corresponding omics data, driven by the semantic aware samples organization.14, a very popular software repository for the analysis of genomic data, used by both bioinformaticians and biologists. The package functionalities assist users in performing the following tasks retrieving metadata associated with collections of GEO samples (dataset)configuring and running a named entity recognition pipeline to annotate retrieved metadata  with the previously built dictionary, therefore associating them with ontology conceptsstructuring the dataset of interest by hierarchically organizing its samples according to the semantic similarity between their associated ontology termsperforming a semantically-driven statistical analysis of the actual omics data in the dataset.Onassis is available as a package within the R/Bioconductor project16. This package stores the metadata of GEO samples and datasets in an SQLite database, with tables containing information such as the title of the experiment, the list of samples, their descriptions, details about the organism, sequencing libraries, description of treatments, and other attributes. While queries through SQLite statements are permitted, as originally intended in GEOmetadb, Onassis implements simplified functions to easily query and subset the metadata tables avoiding SQLite statements. For a proper annotation, particularly important is the choice of the fields to be passed to the annotator12. A visual inspection of GEOmetadb tables revealed that the most informative fields for the description of experiments are title and summary, while for samples source_name_ch1 for sample source, organism_ch1, characteristics_ch1 and description were selected. Finally, filters are provided for the type of platform (e.g. Illumina HiSeq 2000), experiment , and organism. Ensuring the full flexibility, metadata from any other resource can be provided to Onassis in textual files.Metadata can be retrieved from large-scale repositories of omics data, such as GEO and SRA. In particular, the metadata of GEO samples, a reference repository in the field, can be conveniently accessed in Onassis via GEOmetadb17, which we and others previously showed that outperforms other available tools18. XML dictionaries, required by Conceptmapper, can be automatically generated by Onassis starting from OBO ontologies (in both OBO v1.2 and OWL formats). Entries of the dictionary are specified by a canonical name and one or more variants (synonyms).The process of text annotation using controlled vocabularies relies on the ccp-nlp implementation of Conceptmapper19. Eventually, duplicated annotations are removed and a table associating each sample with a list of unique ontology concepts (that we define as semantic set) is returned.The annotation pipeline consists of a sequence of steps: detection of sentences in the metadata, tokenization of both metadata and individual dictionary entries which can be composed of multiple words (tokens), optional stemming of the tokens to reduce word inflections and lookup of the dictionary tokens within sentences. When it comes to building dictionaries and named entity recognition pipelines, various configuration parameters can be set, including: the type of synonyms to include in the dictionary , case sensitivity, stop words removal to ignore uninformative tokens, stemmers, different lookup strategies to consider ordered tokens or their permutations, finding all possible matches or only the longest, allowances for overlaps between subsequent matches18. In particular, in the former, we evaluated the performances of applying Conceptmapper to GEO metadata based on the Brenda20 and Disease Ontology21, to associate GEO samples to specific tissues and disease conditions. The accuracy of Conceptmapper was also compared to the performance of Metamap22, another popular tool in the field. As a result, Conceptmapper reached an accuracy of 0.8 and 0.9, for the disease and tissue ontologies, respectively, while Metamap was in the 0.56–0.65 range12.Importantly, configuration parameters were set to default values based on recent benchmarking studiesFinally, in order to identify in the metadata terms such as gene/protein names and epigenetic modifications, Onassis can automatically build dictionaries for those terms, using Bioconductor libraries available for multiple species. This is particularly useful for example to associate the metadata of ChIP-chip or ChIP-seq experiments to the specific targeted factors.13. Several tools for the computation of semantic similarities have been proposed, including GOSemSim23 and DOSE24, which are tailored on Gene Ontology and Human Disease Ontology, respectively. Their functions allow for the calculation of functional similarity between genes and gene groups, but are limited on a single ontology, implement only a limited number of similarity measures and rely on Bioconductor annotation databases, which are not available for every OBO ontology. Another package, MeSHSsim25 allows the computation of semantic similarities between MeSH headings, but this is tailored for MEDLINE documents.Once samples of large dataset are associated to concepts of a biomedical ontology, it could become evident that the various experiments are not independent. Indeed, the associated concepts are likely to reveal samples that can be considered equivalent or very similar, and others that can be considered far from each other. In other words, relations between samples emerge, which can give a structure to the dataset and can be quantified. Indeed, the relationships encoded in an ontology structure can be used to quantify the semantic similarity between its conceptsIn Onassis, samples annotated with one or more ontologies can be related to each other based on the semantic similarity between their associated semantic sets (i.e. sets of associated concepts). Pairwise samples similarities can be used for their hierarchical clustering, and samples associated with similar semantic sets (based on a user-defined cutoff of similarity) can be grouped (collapsed) into semantically homogenous clusters. For large datasets this can be important to reduce the complexity of resulting dendrogram, and to increase the number of observations (samples) which can be brought back to a common semantic set.26. Specifically, the ontology, represented as a graph, can be traversed to calculate the semantic similarity between pairs (pairwise similarity) or groups (group-wise similarity) of concepts. This is important for determining the similarity between samples that can be easily associated to multiple concepts. Noteworthy, since most of the semantic similarity measures consider only one type of edge and given that “part-of” relationships are widely used in current ontologies, we converted them into the most common “is-a” type. Finally, Onassis allows users to combine similarities from multiple ontologies by using any aggregating function of choice . The choice of the semantic similarity measure can influence the way in which samples are combined. However, difficulties in assessing the performances of semantic similarities arise due to the lack of a gold standard. In fact, the performance of these measures is expected to be highly dependent on the specific task and on the features of the considered corpus, such as annotation size and depth or specificity of annotation classes28.Various similarity measures are available within Onassis by wrapping functions from the Semantic measures library and toolkitcompare function, in order to direct the analysis of the actual omics data , Onassis uses it within the ata Fig. .This requires that the omics data are stored within a score matrix, whose rows represent genomic units and whose columns represent samples. We reasoned that the two most likely genomics unit would be either genes or genomic regions. In the former case, the score may be any omics data that can be associated with individual genes, such as gene-level RNA-seq expression data. In the latter case, the score may be any omics data that can be associated with individual genomic regions, such as the enrichment of ChIP-seq for a given factor or mark within each promoter.Onassis takes then advantage of the semantic information to compare these scores among samples. Specifically, the semantic similarity determines the grouping of the semantic sets, and the scores will be compared between those sets. The function used for comparing the scores could be any of the R-statistical tests, typically returning a test statistic and a multiple-testing corrected p-value. Importantly, users are allowed to define and use their own personalized test function. Tests can be applied by columns, for a pair-wise comparison of semantic sets based on global scores distributions. Instead, the comparison by rows perform the analysis at the level of each genomic unit, leading to the identification of units that are differential among the semantic sets. Finally, in case samples were annotated using two different ontologies, such as cell types and disease ontologies, the corresponding semantic sets can be passed on to Onassis simultaneously. This enables Onassis to perform the above-mentioned analyses by automatically iterating over the semantic sets defined on the primary ontology (e.g. tissue types), while comparing, by row or column, the scores among the states defined on the secondary ontology (e.g. disease conditions identified for each tissue condition). The following use cases will illustrate these analyses in detail.We adopted Onassis for the integrative analysis of large-scale omics datasets in three different contexts: (i) a large dataset of H3K27ac ChIP-seq samples; (ii) a compendia of 620 RNA-seq samples from GEO; (iii) a large-compendia of ~15 K DNA-methylation samples. These analyses, which are presented below, illustrate how Onassis functionalities can be used for a semantically-aware analysis of omics data.29. As illustrated by Fig. 30, a database describing experiments and samples in SRA, the main public repository for high throughput sequencing data. Similarly to GEOmetadb, SRAdb can be explored through SQLite queries. Metadata of human ChIP-seq samples were obtained by querying the database according to library strategy, library source and organism attributes of experiments. The text from informative fields, which included information about the target of the ChIP-seq experiment, cell type or tissue and disease conditions (if any), was passed as input to Onassis. H3K27ac samples metadata were thus annotated with Cell Ontology31 (CL) and Human Disease Ontology21 (DOID) concepts. The default parameter configuration was adopted for the annotation pipeline, since it is the best performing according to previous analyses18. Generic or uninformative terms such as ‘cell line’ or ‘disease’ were filtered out, and cell line/tissue and disease semantic sets were defined for each sample  values higher than 0.9 were associated to a unique tissue semantic set  was bound by the mark in a specific sample (column) were filled with the corresponding peak intensities. The resulting score matrix and the annotations were fed into Onassis compare function, to automatically apply two-tailed Wilcoxon rank sum tests, thus revealing global differences in H3K27ac within promoters between pairs of diseases for each tissue semantic set , were retrieved through GEOmetadb queries. The actual expression data were obtained from the recount2 project36, following their homogeneous re-analysis in order to minimize the possibility of batch effects. Metadata were annotated with Onassis based on the CL and DOID ontologies, to determine cell line/tissue(s) and disease(s) associated with each dataset. The annotation procedure resulted in 26 tissue and 24 disease semantic sets. RNA-seq samples within each semantic set were then used to identify genes that were post-transcriptionally regulated for a given tissue and disease as described in35. Without the use of Onassis we could only have independently identified genes within each of the 620 samples. Rather, having assigned each sample to a semantic state, and having grouped samples associated to very similar semantic states, our analysis benefited by having multiple samples per condition , and by having associated those samples to biologically meaningful, discrete, and related conditions .With the aim of determining the extent of post-transcriptional regulation in human we analyzed large-scale37 and are associated with enhancers and regulatory factors driving the expression of cell identity genes39. Altered DNA methylation in these regions has been related to various diseases and contributes to cancer development42. We used Onassis functionalities to semantically annotate and organize a large DNA methylation dataset with the aim of identifying tissue or cell line specific LMRs, thus pointing to altered methylation patterns in specific disease conditions. We focused on the Marmal-aid repository43, which provides base-resolution information on the methylation of cytosines for 14584 samples from the human Infinium Illumina platform, profiling 450 K individual cytosines. As for the recount2 project, also Marmal-aid data were subjected to a re-analysis with homogenous conditions. We discarded the LMRs that could be found within hypomethylated promoters of expressed genes, and focused on 21209 LMRs that were distal from genes, since these represent potential regulatory elements. A manually curated table of samples’ metadata was included in Marmal-aid. We used Onassis to annotate those metadata with concepts belonging to the CL extended with concepts from Uber Anatomy ontology (UBERON)44 for anatomical parts. Eventually Onassis allowed to associate 13071 samples to 141 different cell type, which were eventually collapsed to a simplified set of 78 semantic states, each including one or more samples, i.e. sets of LMRs. We then proceeded to characterize tissue-specific LMRs. Finally, samples associated with each tissue, could also be associated to one or more disease conditions through the DOID Disease ontology. For each cell line/tissue, and for each disease condition, we focused on the LMRs that were present or absent compared to matched healthy samples. This analysis returned tissue specific catalogues of LMRs representing candidate regulatory elements that are epigenetically active in specific disease conditions. The ability of Onassis to assign samples to semantic conditions, and to determine their relatedness, was critical to identify the samples to be compared in their DNA methylation patterns (Supplementary File Low methylated regions (LMRs), with an average methylation of 30%, usually occur at distal regulatory elements12.Onassis allows users to build large-scale structured datasets enabling the exploration of multiple conditions simultaneously. On one hand, this can allow to test hypothesis on previously published omics data. For example, searching for diseases that differ from the normal counterpart in terms of binding of a transcription factor. On the other hand, this can be instrumental for complementing and expanding in-house generated datasets. For example, a scientist working on a given regulatory factor, could study how its binding relate to the binding of other factors profiled by ChIP-seq in the same or very similar conditions32 (for ChIP-seq data), recount236 (for gene expression data), and Marmal-aid43 (for DNA methylation data), or public consortia such as ENCODE45 provide large datasets that have been processed with homogeneous pipelines and settings. These data are particularly suitable to the integrative analysis approaches implemented in Onassis, since batch effects due to alternative data analysis and normalization strategies are reduced.Onassis wraps different NLP tools and makes their functionalities conveniently available within the R/Bioconductor environment, where a number of packages could be used to perform additional analyses downstream the output of Onassis. The package focuses on and is particularly advantageous to the integrative analysis of omics data, but it can accommodate any corpus of text and ontology. Nowadays different projects such as CistromeIn conclusion, Onassis greatly facilitates the identification and exploitation of publicly available datasets, thus promoting their re-use and providing biologists with the ability to analyze their own data within the context of the enormous wealth of knowledge nowadays available in public repositories.Supplementary File1.


Furthermore, users can upload subtyping results of their own methods to compare with the built-in methods. CEPICS is implemented as an R package and is freely available at https://github.com/GaoLabXDU/CEPICS.Cancer subtypes can improve our understanding of cancer, and suggest more precise treatment for patients. Multi-omics molecular data can characterize cancers at different levels. Up to now, many computational methods that integrate multi-omics data for cancer subtyping have been proposed. However, there are no consistent criteria to evaluate the integration methods due to the lack of gold standards . Since comprehensive evaluation and comparison between different methods serves as a useful tool or guideline for users to select an optimal method for their own purpose, we develop a scalable platform, CEPICS, for comprehensively evaluating and comparing multi-omics data integration methods in cancer subtyping. Given a user-specified maximum number of subtypes,  With the development of high-throughput technologies, huge amounts of multi-omics data for cancers have been generated, such as genomics, epigenomics, and transcriptomics data. Researchers have opportunities to integrate different types of omics data to advance cancer research, including identifying cancer driver mutations , prioritCancer subtyping is one of the most important issues in cancer research. In precision medicine, patients need to be accurately diagnosed and treated based on their cancer subtypes due to the difference in therapeutic strategy and drug response. In recent years, many computational methods that integrate multi-omics data for cancer subtyping have been proposed. These approaches can be mainly divided into two categories based on data modeling strategies , includihttps://github.com/GaoLabXDU/CEPICS. For convenience, we also construct a docker image of CEPICS. Please see Since comprehensive evaluation and comparison between different methods serves as a useful tool or guideline for users to select an optimal method for their own purpose , we deveHere, we introduce the framework of CEPICS and present its three different application scenarios based on genomics, transcriptomics, and epigenetics data.k-nearest neighbor imputation  of cancer subtypes, CEPICS requests users to specify a maximum possible number of subtypes  so that CEPICS generates subtyping results from two to k-max respectively. In order to evaluate and compare the five built-in methods, we make some improvements on their method implementation; please see Methods for details.In the second step  and PFA Finally, for evaluation and comparison (k) in each method but also makes horizontal comparisons across different methods. Most importantly, as a scalable platform, CEPICS allows users to upload their own subtyping results so that it is very convenient to evaluate and compare their own methods to the state-of-the-art ones. For providing a high-confidence subtyping result, CEPICS employs a vote-based strategy to conduct a sample consensus clustering. The result represents a robust prediction for sample pairwise similarities because it considers all choice of k and all user-selected built-in methods.CEPICS not only makes vertical comparisons on results generated by specifying different numbers of subtypes , KM survival curves, and patient similarity heatmaps for different numbers of subtypes in each method. CEPICS generates two versions of reports depending on the availability of true labels of patients.CEPICS is an easily-used R package. Here we present its three application scenarios using four different omics datasets of Colon Adenocarcinoma (COAD) from The Cancer Genome Atlas (TCGA), including mRNA expression, miRNA expression, DNA methylation, and copy number variation data.K-nearest neighbor method was used to impute missing values and z-score was used to normalize each dataset. Top 10% features based on the MAD strategy were selected for subtyping. We set five as the maximum possible number of subtypes  and chose all five built-in integration methods and all performance metrics for evaluation and comparison. CEPICS main results are elaborated below and complete reports are provided in For all application scenarios, CEPICS selected samples belonging to the same patient cohort for four omics datasets. Then CEPICS filtered the features that had more than 20% missing values across all patients and filtered the patients that had more than 20% missing values across all features for each dataset. k = 2 to k-max. Then, CEPICS evaluates and compares these results using both survival analysis and clustering performance metrics.Given multi-omics datasets without any subtype information, CEPICS uses each user-selected method to generate subtyping results based on k = 4, 4, and 3, respectively . Based on a voting strategy, CEPICS calculated a consensus matrix considering results of all methods and all choices of k, and then generated a consensus similarity heatmap by hierarchical clustering  and NMI, while iClusterBayes had the highest ARI and silhouette coefficient. In this situation, considering both clustering and clinical metrics, CEPICS set different weights for four metrics and scored each method to suggest the best one (please see Methods for details). Then, CEPICS made comparisons across different methods from k = 2 to k-max  of cancer subtypes, CEPICS requests users to define a maximum possible number of subtypes  instead of k in order to perform a vertical comparison between different k ranging from two to k-max, which comprise a large amount of work.Overall sample analysis. CEPICS employs a vote-based strategy to give a sample consensus clustering result, which represents a robust prediction for sample pairwise similarities because it considers all choices of k and all user-selected methods.Comparison with existing subtyping results. CEPICS allows users to upload their own subtyping results so that it is very convenient for researchers to evaluate and compare their own methods to the state-of-the-art ones. Moreover, an empirical pre-determination of subtypes can also be inputted to CEPICS, which can be used as the gold standard to evaluate and compare results generated by different methods.Using CEPICS to make comparisons between different methods, we found that the two statistics-based methods, iClusterBayes and LRAcluster, were relatively inefficient. For example, the proportion of time consumption for iClusterBayes shown in the pie charts of reports were all over 80%  constructs a sample-sample similarity network for each data type and then integrate them based on message-passing theory. Spectral clustering is performed on the integrated similarity network to discriminate disease subtypes. Pattern Fusion Analysis (PFA) aligns local sample-pattern from each data type into a global sample-pattern based on an adaptive optimization strategy and then uses k-means clustering to identify subtypes. There are also two statistics-based methods in the second category. Low-rank Approximation Based Multi-omics Data Clustering (LRAcluster) uses a low-rank approximation-based probabilistic model to integrate different omics data and then performs k-means clustering to identify disease subtypes. iClusterBayes uses the Bayesian latent variable regression model to integrate different omics data and also performs k-means clustering to discriminate subtypes. Perturbation Clustering for Data Integration and Disease Subtyping (PINS) is an integration framework as the third category. It considers the resilience of patient connectivity and cluster ensembles to integrate different omics data, and then uses hierarchical clustering, PAM or Dynamic Tree Cut to identify subtypes. Since both PFA and LRAcluster packages only output an integrated sample-feature matrix after data integration rather than clustering (subtyping) results, CEPICS uses In order to evaluate and compare the five built-in methods, we format inputs and outputs of all methods consistently and also make some improvements on them.When using the LRAcluster R package for dimension reduction, users must manually determine the dimension of the reduced subspace used for clustering according to an explained variance (EV) plot. For convenience, CEPICS implements an automatic procedure to perform dimension reduction.n, a user-specified maximum dimension (default as 10). Typically, the reduced dimension can be determined based on a turning point on the EV curve according to the criterion that the increase of model fitness is much slower after it  is the x-coordinate  of point i or j, and ev(·) is the EV value of that point. δ is a parameter which controls the impact of the difference in x-coordinates between points i and j on their relative difference in EV values, and δ ≥ 0. When δ = 0, h represents their difference in EV values. When δ = 1, h indicates the slope of the curve between points i and j when x(j) > x(i), and the negative slope when x(j) < x(i). When δ > 1, the larger difference in x-coordinates between the two points, the smaller their relative difference in EV values. To consider the global information, we set δ = 2.2. Then, we define a turning score aswhere i, defined as µ,v are the mean and the variance of x-coordinates, respectively. Then, an adjusted turning score is defined asTo avoid the bias of the turning score on terminal points of the EV curve, we introduce a penalty to each point i* with the highest adjusted turning score is chosen as the reduced dimension. The x-coordinate of the turning point The process of choosing the reduced dimension can be parallel computed. We bring the parallel computation to LRAcluster which can save much time. iClusterBayes can be parallel computed only on Linux system or macOS because of the limitation of parallel function the authors used. We employ a different parallel function that is also supported on Windows system and make it possible to run iClusterBayes in parallel without operating system limitation.Since only the MATLAB source code is available for the PFA algorithm, CEPICS uses R.matlab R package to initialize a MATLAB server to run PFA. The principle of the PFA method has no limitation on the number of datasets. However, the code provided by the author requires three datasets as input. We improve the code in order to integrate multi-omics data without the limitation of the number of datasets by modifying the structure of the code to fit different numbers of datasets and calling the PFA core code as the author did.For comprehensive evaluation and comparison, both survival analysis and clustering performance metrics are applied to each method.Kaplan-Meier survival curves and p-value of the Cox proportional hazards model evaluate the significance of difference among different subtypes identified by a method. Generally, p–value < 0.05 suggests that subtyping results are reasonable.Similarity heatmap shows similarity patterns of patient samples.Silhouette coefficient measures the similarity between one sample and its classified subtype compared to other subtypes to determine how appropriately the samples have been clustered. For a sample i, the silhouette coefficient can be calculated asa(i) is the average distance between i and other samples in its own cluster, b(i) is the minimum distance between i and other samples which are not in its own cluster. Then, the silhouette coefficient of the result can be defined as n is the total number of samples. SC has a range  and higher values represent better performance.Normalized mutual information (NMI) measures the concordance of two subtype results. It can be calculated as Adjusted rand index (ARI) measures the agreement between two subtype results. Suppose that we have two partitions, U={u1,u1,…,ru} and V={v1,v2,…,sv}, of N samples. We take U as true labels and V as a clustering result. A matrix can be constructed with each entry ijn is the number of samples that are both in clusters iu and jv, i=1,2,…,r, j = 1,2,…,s. Define i and column j, respectively, and U and V is defined as  runs in total. CEPICS calculates a consensus matrix n is the number of samples, and i and j are clustered in the same cluster, otherwise, ijcm∈. Applying hierarchical clustering to CM, CEPICS generates a clustering result which represents a robust prediction for sample pairwise similarities because it considers all choices of k and all user-selected methods.m, CEPICS calculates z-score of the result x(i) of each method i as µ and σ are the means and standard deviations of all results of the metric m, respectively. For the Cox p-value, CEPICS transforms z-scores. Considering that both clinical survival analysis and clustering performance metrics should have the same weight, a performance score for each method i is defined asWhen a user uploads a pre-determined subtyping result, CEPICS considers it as the gold standard to make comparisons and suggests the best method according to the combination of the Cox p-value, NMI, ARI and silhouette coefficient. For a performance metric where https://github.com/GaoLabXDU/CEPICS; Project name: CEPICS; Operating system(s): Windows, Linux, and macOS; Programming language: R; License: General Public License (GNU GPLv3); Any restrictions to use by nonacademics: None. The datasets analyzed for this study are available at the Genomic Data Commons Data Portal: https://portal.gdc.cancer.gov/The source code and binaries of CEPICS, and the demo datasets are available at RD and LG conceived the study. HX and KS developed the program and wrote the codes. RD, HW, YD, CZ, and SJ tested the R package. RD wrote the manuscript. YH, LG, and RD revised the manuscript. All authors read and approved the manuscript.This work was supported by the National Key R&D Program of China No. 2018YFC0910400 to LG, the National NSFC , and the Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), LCNBI, and ZJLab.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Given that the biological processes governing the oncogenesis of pancreatic cancers could present useful therapeutic targets, there is a pressing need to molecularly distinguish between different clinically relevant pancreatic cancer subtypes. To address this challenge, we used targeted proteomics and other molecular data compiled by The Cancer Genome Atlas to reveal that pancreatic tumours can be broadly segregated into two distinct subtypes. Besides being associated with substantially different clinical outcomes, tumours belonging to each of these subtypes also display notable differences in diverse signalling pathways and biological processes. At the proteome level, we show that tumours belonging to the less severe subtype are characterised by aberrant mTOR signalling, whereas those belonging to the more severe subtype are characterised by disruptions in SMAD and cell cycle-related processes. We use machine learning algorithms to define sets of proteins, mRNAs, miRNAs and DNA methylation patterns that could serve as biomarkers to accurately differentiate between the two pancreatic cancer subtypes. Lastly, we confirm the biological relevance of the identified biomarkers by showing that these can be used together with pattern-recognition algorithms to accurately infer the drug sensitivity of pancreatic cancer cell lines. Our study shows that integrative profiling of multiple data types enables a biological and clinical representation of pancreatic cancer that is comprehensive enough to provide a foundation for future therapeutic strategies. In general, the approaches that are currently used for making both outcome predictions and treatment decisions are based on histological subtyping and clinical parameters such as the disease stage, metastasis, and the resectability of tumours5. Recently, however, the advent of molecular profiling has laid the foundation for quantitatively profiling tumours based on their genome-wide gene transcription profiles, protein expression profiles and/or mutational landscapes9. These profiling methods promise a more accurate and precise definition of tumour subtypes and better predictions of how particular tumour types will respond to different treatments.Pancreatic cancer is a heterogeneous disease that is characterised by poor clinical outcomes and few effective treatment options. Attempts to define a standard classification for tumours of the pancreas have been ongoing for decades11.Further, molecular data that is used to construct the molecular profiles of particular cancers have been used to identify the perturbances in the cellular regulatory networks that characterize these cancers: often revealing numerous potential drug targets within various signalling pathways. This molecular data together with the known molecular profiles of numerous well characterized cancer cell lines can even be leveraged using machine learning methods to predict the responses of particular patient tumour subtypes to different anticancer drugs13.A crucial resource for the discovery of useful diagnostic biomarkers and potential anticancer drug targets are large-scale datasets comprising, among other data types, extensive genomic, transcriptomic and proteomic profiles of matched healthy and tumorous tissues. These datasets, which are compiled and maintained by The Cancer Genome Atlas (TCGA) and the International Cancer Genome Consortium (ICGC) are helping us uncover the molecular characteristics and signalling pathway perturbations that define specific cancer subtypes18. Although the biomarkers that differentiate between different pancreatic cancer subtypes could eventually inform treatment decisions, there are as yet no available subtype-specific treatment options for this type of cancer. There is, therefore, a pressing need to, firstly, find a set of biomarkers that can be used to accurately and sensitively diagnose pancreatic cancer subtypes and, secondly, to identify suitable targets for drug development among these biomarkers.Among the cancers that are well represented in these data collections is pancreatic cancer. Molecular profiling analyses of the pancreatic tumour datasets have identified both distinct pancreatic cancer subtypes, and mutations of the genes, KRAS, TP53, SMAD4 and CDKN2A as potential drivers of pancreatic cancerDefinitions of disease subtypes is a perpetual process, with classifiers and cut-offs that differentiate between the subtypes, essentially needing to be continually re-defined and refined as more molecular data and better molecular profiling tools become available. As classification schemes for pancreatic cancers improve, it is expected that additional specific molecular correlates of patient survival, responses to anticancer drugs, and tumour aggressiveness will be uncovered. Armed with such knowledge, we could develop better prognostic and diagnostic methods, and select the best drugs to treat specific pancreatic cancer subtypes. Further, more subtype-specific molecular features could potentially enhance the accuracy with which machine learning methods could predict the drug response profiles of specific pancreatic tumours, thus leading to improved disease outcomes.21. These difficulties include, but are not limited to, inconsistent classifications of patient tumours when the tumours are subtyped using different types of molecular data, and the efficient integration and analysis of different data types to yield consistent identifications of the causal disruptors of the molecular processes that underlie the observed differences between pancreatic cancer subtypes19. Ultimately, these difficulties undermine efforts to predict the responses of tumours to drugs: an endeavour involving comparisons between the relevant molecular features of a novel tumour with those of well-characterized tumour subtypes or tumour cell lines.However, it remains technically difficult to effectively leverage the diverse and ever-increasing data relating to pancreatic tumoursWith these issues in mind, we attempted to identify clinically relevant subtypes of pancreatic cancer accounting for the full spectrum of molecular and clinical data available for pancreatic cancer tumours in the TCGA dataset. We address the problem of inconsistent tumour classifications that are obtained using different types of molecular data, by applying an integrative classification approach that considered all the available molecular data types. As expected, our analyses identified discrepancies between various classification schemes but ultimately supported the existence of two major pancreatic cancer subtypes. Besides uncovering the likely molecular causes of altered biological processes within the tumours of these two subtypes, we identified biomarker sets that can be used to accurately and sensitively classify novel pancreatic tumours. Further, in the face of multiple high-dimensional data types, we show that statistical models that capture the complexity of disease can aid in the identification of relevant drugs and drug targets that might offer substantial benefits for patients afflicted with tumours belonging to either of the pancreatic cancer subtypes.22. Then, we compared this clustering of pancreatic cancer samples to other subtypes that are reported in the literature for various other molecular data types  and established that the samples clustered differently depending on the specific molecular data type used  determined proteomics data of the 45 high-purity pancreatic cancer samples that are available in the TCGA database to identify two coherent clusters of patient tumours Fig. 22. Then,sed Fig. .Figure 119. Using DNA methylation status, protein expression, mRNA transcription and miRNA data of the 45 high purity cancer tumour samples available in TCGA, we applied the SNF clustering method to identify two-cluster and three-cluster clustering solutions . SNF solves the disparate clustering problem by constructing similarity networks of samples for each available molecular data type and then efficiently fuses these into one network that represents clustering based on all the underlying data types Fig. 19. Usingons Fig. .The pancreatic cancer subtypes in the two-cluster solution comprised 25 and 20 tumours, which we provisionally named as subtype-1 and subtype-2, respectively. Interestingly, the SNF clustering solutions were highly concordant with each of the clustering solutions obtained using individual molecular data types but were most similar to that obtained using the proteomics data  in the tumours of both subtypes  biological process classifications of proteins that were upregulated within tumour belonging to each of the subtypes using Enrichrpes Fig. .Figure 2−19). Within the mTOR pathway of subtype-1 tumours, we found increased expression of well-documented oncoproteins including MTOR and BRAF: both of which have previously been linked to pancreatic carcinogenesis  between these subtypes  and tumour suppressor proteins (such as PTEN and CDK2A1)30 Fig. . Despite30 Fig. . Such a 30 Fig. . Likewispes Fig. .Figure 3−03), MTOR (p = 0.011), and RPS6KA1 (p = 0.0499)  that subtype-1 tumours display increased mTOR signalling. Conversely, for subtype-2 tumours, we found elevated phosphorylation levels of proteins such as CDK1-pY15, p27-pT158 and p27-pT198 , we found a subset of kinases that might drive pancreatic carcinogenesis, including, among others (Supplementary File 99) Fig. 31. We obrms Fig. . Furtherrms Fig. 32,33. Th198 Fig.  which arOverall, our findings suggest that for tumours of the two major pancreatic cancer subtypes, oncogenesis may be primarily driven by perturbation in either SMAD4 or mTOR signalling.24. We found that the over-transcribed genes in subtype-2 tumours were enriched for, among others, molecular functions associated with transmembrane transporter and G-protein coupled receptor activities  for classification with regularisationata Fig. 38.40. Specifically, we observed five-fold cross-validation classification accuracies of 99% for the mRNA-based KNN classifier and 98% for the DNA methylation-based SVM classifier, with an agreement of 97% (see methods sections).Using NCA, we identified biomarker sets comprising 50 mRNAs, 49 methylated genes, 14 proteins, and 20 miRNAs. For these biomarker sets, we separately applied hierarchical clustering to each of the different molecular data categories to consistently and accurately reproduce the pancreatic cancer subtype classifications Fig. . Also, wDecreasing the number of biomarker genes needed to accurately classify tumours from new pancreatic cancer patients would improve the utility of these sets in a clinical diagnostic setting. To identify smaller biomarker gene sets, we used supervised machine learning methods (see methods section) to define a biomarker set of fewer than ten genes, miRNA or proteins that would minimise incorrect classifications Fig. . Also, w12. Using the mRNA-based KNN classifier that was trained on TCGA data, we tested the reproducibility of the two-subtype classification scheme by classifying 96 ICGC pancreatic cancer patients into subtypes-1 and subtypes-2 .Concerning histological features of tumours that might be useful for differentiating between the subtypes, we observed that only subtype-1 tumours displayed evidence of intraductal papillary mucinous neoplasm, whereas only subtype-2 tumours were categorised by histological inspection as being pancreatic adenocarcinomas Fig. . Further10. We attempted to subtype these cell lines using the biomarker gene set identified using the KNN classifier that we trained on the TCGA mRNA data ata Fig. . It is k47. The Searcher model quantified and stored information concerning similarities between the transcription profiles of all the cell lines. Next, we retrieved the drug response profile of a “query” cell line and those of its nearest neighbours based on squared Euclidean distances from the Searcher model. To infer the drug response of the query cell line, we calculated the median drug response of the retrieved nearest neighbour cell lines to each of the 24 anticancer drugs that were profiled by the CCLE  to each “query” cell line as determined using an exhaustive KNN searcher modelCLE Fig. . For exaCLE Fig. .After predicting the drug responses of all the pancreatic cancer cell lines that also had observed drug response data, we compared the predictions to the observed drug responses. Our drug response predictions displayed substantial agreement with the actual drug responses in that they yielded an average Kappa statistic of 0.67 Fig. .48. Here, we attempted to resolve this issue by employing a multidimensional clustering method capable of simultaneously utilising protein expression, mRNA transcription, DNA methylation and miRNA synthesis data. We found that by integrating across all these molecular data types, pancreatic cancer tumours could be classified into two clinically distinct subtypes: which we have simply named subtype-1 and subtype-2.We conducted a comprehensive analysis of clinically relevant patterns of mutation, methylation, transcription, protein expression, and miRNA synthesis within pancreatic cancer tumours. Several pancreatic cancer studies have previously highlighted the limitations of utilising a single molecular data type to accurately classify pancreatic cancers Fig. 8,9,14,4851. Further, it is well established that some pancreatic cancer subtypes respond well to drugs which inhibit the mTOR pathway53. Accordingly, we anticipate that subtype-1 tumours will likely be more responsive to such therapies than will subtype-2 tumours.We observed that subtype-1 tumours were characterised by alterations of the mTOR signalling pathway, and the expression levels of different mTOR pathway proteins were positively correlated to each other Fig. . This fi57.Interestingly, subtype-2 tumours display unique alterations to cell cycle pathways Fig. . This is60.We noted that, in addition to differences in patterns of protein expression, the two pancreatic cancer subtypes differ with respect to patterns of protein phosphorylation, implying that the kinases that are involved in oncogenic transformation differ between the subtypes. Specifically, whereas subtype-1 tumours show upregulation of mTOR signalling associated kinases , subtype-2 tumours display upregulation of cell cycle associated kinases  and miRNA signatures. These suggest that epigenetic and/or miRNA variations may be primarily drivers of the differences in the transcriptome and proteome profiles of subtype-1 and substype-2 tumours.77. We are optimistic that any of these four biomarker sets could be individually used to obtain accurate subtype classifications for new pancreatic tumours. Nevertheless, the utility of these four biomarkers sets for predicting clinical outcomes and guiding treatment strategies will need to be evaluated in future studies.In line with other studies that have identified biomarkers to classify tumour subtypes, some of which have important treatment and prognostic implications, we identified biomarker mRNA, DNA methylation, protein or miRNA sets that could be used to accurately subtype pancreatic tumours78, our approach is novel in that it utilizes tumour subtyping based on all available molecular data to mine for biomarkers that differentiate disease subtypes: biomarkers which are then used to inform our KNN exhaustive search model with respect to quantifying the similarity of cell lines. What this means is that our approach is capable of utilizing matched molecular data and drug responses from either cancer patients or cell lines to predict, with reasonable accuracy, the drug responses of tumours for which we have only information on the concentrations of the mRNAs, proteins or miRNAs that are included within the biomarker sets which we have identified. As with other machine learning based inference schemes, the accuracy of the predictions that are made should improve given additional matched molecular and drug response data79.Encouragingly, we were able to demonstrate that, by focusing on just the transcription levels of the mRNA molecules that are represented in our mRNA biomarker set, we could accurately predict the drug responses of cancer cell lines based on the drug responses of other cancer cell lines with similar mRNA expression profiles. Although others have also been able to predict the drug responses of cell lines using similar machine learning approachesAltogether, our analyses have revealed the molecular underpinnings of, and potential treatment strategies for, two clinically distinct forms of pancreatic cancer. We are optimistic that an approach such as we have used, where multiple different molecular data types are leveraged to subtype and characterise particular tumour variants, could yield valuable insights into the management of other difficult to treat cancers such as those of the lungs and triple negative breast cancer.13. Data on these patient samples within the TCGA included: reverse phase protein array-based proteomics data , whole exome sequencing data (n = 76), transcriptome data determined using RNAseq (n = 76); DNA copy number and mutation data (n = 76), miRNA data (n = 56), and comprehensive clinical data. For our analyses, we only considered the 76 “high purity” samples for which transcriptome and whole exome sequencing data was available. Out of these 76 samples only 45 have RPPA data. All data used in our analyses were obtained from cBioPortal 80.We analysed data from 185 of the pancreatic cancer patients who had contributed samples to the TCGA project39. To find the most informative number of clusters, K-means clustering was run over 500 iterations for cluster sizes  of two, three, four, and five . The average silhouette values for each value of K were compared, revealing that the two-cluster solution had the highest mean silhouette value and was therefore deemed to be the most coherent  is a clustering method that considers information from multiple molecular profiles. It has previously been used to segregate tumours of various cancer types based on multiple different sources of molecular datarks Fig. . Next, wrks Fig. 19.23.The Kaplan-Meier method was used to compare overall survival and the duration of progression-free survival of patients with tumours belonging to the different pancreatic cancer subtypest-test with unequal variance and with the Benjamin-Hochberg correction applied to p-values84. Further, we queried Enrichr with two lists of 60 and 30 proteins found to be upregulated in subtype-1 and subtype-2 tumours, respectively, to return enriched KEGG pathways for each subtype , the Kinase Enrichment Analysis , and Chromatin Immunoprecipitation Enrichment Analysis 2016 88. We visualised the resulting networks in yEd  molecular functions and KEGG pathways enriched for each disease subtype  with regularisation method was used to select the most useful features for each molecular data typeion Fig. . To appl95. This improved the overall classification accuracy of the models on the cross-validation set to 100% for the mRNA-based KNN model and 99% for the DNA methylation-based SVM model. The trained models were then used to classify 31 other high-purity pancreatic tumours from the TCGA   Fig. .96. The exhaustive searcher model takes as input the training data (in this case the mRNA biomarkers), distance metrics, and parameter values of the distance metrics for an exhaustive nearest neighbour search and can then be used to identify the nearest neighbours to a particular patient tumour or cell line within a specified radius of the distance matric. Here, the nearest neighbours to a particular cell line suggest similarity at the molecular level based on mRNA, DNA methylation, protein and miRNA data encoded in the SNF subtyping. The ten nearest neighbouring cell lines or tumours were determined using a nearest neighbour search algorithm based on a squared Euclidean distance metric  IRB00001938 approved the protocol of this study.Supplementary Information.Supplementary File 1.Supplementary File 2.Supplementary File 3.


Long non-coding RNAs (lncRNAs) are emerging as important regulators of various biological processes. While many studies have exploited public resources such as RNA sequencing (RNA-Seq) data in The Cancer Genome Atlas to study lncRNAs in cancer, it is crucial to choose the optimal method for accurate expression quantification.In this study, we compared the performance of pseudoalignment methods Kallisto and Salmon, alignment-based transcript quantification method RSEM, and alignment-based gene quantification methods HTSeq and featureCounts, in combination with read aligners STAR, Subread, and HISAT2, in lncRNA quantification, by applying them to both un-stranded and stranded RNA-Seq datasets. Full transcriptome annotation, including protein-coding and non-coding RNAs, greatly improves the specificity of lncRNA expression quantification. Pseudoalignment methods and RSEM outperform HTSeq and featureCounts for lncRNA quantification at both sample- and gene-level comparison, regardless of RNA-Seq protocol type, choice of aligners, and transcriptome annotation. Pseudoalignment methods and RSEM detect more lncRNAs and correlate highly with simulated ground truth. On the contrary, HTSeq and featureCounts often underestimate lncRNA expression. Antisense lncRNAs are poorly quantified by alignment-based gene quantification methods, which can be improved using stranded protocols and pseudoalignment methods.Considering the consistency with ground truth and computational resources, pseudoalignment methods Kallisto or Salmon in combination with full transcriptome annotation is our recommended strategy for RNA-Seq analysis for lncRNAs. Other lncRNA databases including NONCODE [cis ortrans; and (iv) scaffolds, which serve as central platforms where relevant molecular components in cells are assembled [Long non-coding RNAs (lncRNAs) are a diverse class of RNA molecules that are >200 nucleotides in length and do not encode proteins . While f NONCODE  both col NONCODE . lncRNAs NONCODE . They ar NONCODE . They massembled .lncRNAs have been shown to be important in the pathogenesis of human diseases, especially in cancer, and many cancer-relevant lncRNAs have been identified ,9. For eThe discovery of oncogenic and tumor suppressor lncRNAs has led to an increased interest in the investigation of lncRNAs as potential cancer drug targets and biomarkers. Hence, it is critical to accurately determine lncRNA expression in cancer research. RNA sequencing (RNA-Seq) has been widely used for massive-parallel gene expression quantification. There have been many studies that explore lncRNA expression profile in cancer using publicly available RNA-Seq datasets such as those generated by The Cancer Genome Atlas (TCGA), which provide a rich source of lncRNA expression data in large cancer patient populations ,17. AmonThere are 2 types of RNA-Seq protocols, depending on whether strand specificity information of transcripts is retained in the library preparation step . The staMultiple tools for processing RNA-Seq data have been developed in recent years. While some studies have benchmarked RNA-Seq analysis workflows ,21, theiBecause of the lower expression and different properties of lncRNAs with respect to protein-coding genes, we hypothesized that the processing and analysis of RNA-Seq data for lncRNA expression may be subjected to different technical biases and challenges, and that special considerations may be necessary to optimize the pipeline specifically for lncRNAs.To investigate the performance of different methods on the quantification of lncRNAs as well as the effect of different RNA-Seq library preparation protocols, we applied 5 popular quantification methods, Kallisto , Salmon Both un-stranded and reverse-stranded RNA-Seq data from TCGA samples were downloaded from the ISB Cancer Genomics Cloud. The other reverse-stranded dataset was downloaded from NCBI SRA under the accession PRJEB11797. Read quality control was performed with Trim galore , with thWe used RSEM  to simulUsing the lncRNA-only annotation overestimates lncRNA expression compared to full annotation Fig. . The ovePseudoalignment methods detect expression of more genes than alignment-based methods Fig. . The aveR2 value, and a slope value of close to 1  [Antisense and long intergenic non-coding RNAs (lincRNAs) are the 2 major types of lncRNAs. In un-stranded samples, the mean proportion of antisense lncRNAs in the expressed lncRNAs is 54%, which is much higher than the proportion of antisense lncRNAs in GENCODE (39%) and the expressed antisense lncRNAs in reverse-stranded libraries (25–48%) Fig. . More thWe further investigated the features of discordant lncRNAs (Spearman’s correlation <0.7 compared with respect to the ground truth), especially for alignment-based methods, because pseudoalignment methods are highly concordant with the ground truth. In un-stranded samples, the majority of discordant lncRNAs are antisense Fig. . ApproxiTo demonstrate the importance of accurate lncRNA expression quantification, we investigated the expression profile of a number of well-known lncRNAs with important functions in cancer: HOTAIR, CDKN2B-AS1, TERC, LINC01106, and LINC01123 , aligners  or transcriptome annotation (GENCODE and NONCODE). Further evaluation of the methods, including RSEM, on datasets generated by Polyester showed that RSEM's performance was similar to that of Kallisto and Salmon. Pseudoalignment methods detected more lncRNAs in each sample, which is similar to those levels in the ground truth for the simulated datasets. They were also highly concordant with the ground truth in terms of having the highest Spearman’s correlation and the lowest Euclidean distance, especially at sample-level comparisons. When each method was linearly regressed with the ground truth, almost all the points from Kallisto and Salmon fell on the diagonal line, with very few outliers . The supR2 values as compared with the ground truth, especially for the 2 reverse-stranded datasets . The results are deposited online for the wider research community studying lncRNAs in TCGA samples, and we also provide a web interface to investigate and visualize gene expression in these samples.The comparison in this analysis was performed at the gene level. If the goal is to examine transcript-level expression, HTSeq and featureCounts are not suitable for the purpose. They are developed explicitly for gene-level read counts. When they count the reads mapped to transcripts rather than genes, reads mapped to exons shared by several transcripts will then be considered ambiguous and discarded by default. Kallisto, Salmon, and RSEM are able to produce both transcript- and gene-level expression output.We further addressed the problem of using partial transcriptome annotation Fig. . Full trOne limitation of this study is that the simulated datasets are based on polyA-selected RNA-Seq. It would be helpful to evaluate RNA-Seq methods that capture more lncRNAs. However, this study focuses on using existing RNA-seq datasets for profiling lncRNAs in cancer. More importantly, in cancer research most of the available datasets, including TCGA, were generated using polyA-selected RNA-Seq. Thus, our study still provides valuable guidelines for researchers studying lncRNAs in cancer. Another limitation is that only simulated data were used in the study. Simulated data may not capture the complexity of real data and true experimental variability. A more comprehensive approach to complement the simulated data with experimental data should be considered in further benchmark studies . More viReads from the lncRNAs that cannot be quantified accurately by HTSeq and featureCounts are either aligned poorly to the genome, or they can be properly aligned, but HTSeq and featureCounts cannot determine where to assign the reads because of overlapping annotation with other genes . The supIn summary, considering the consistency with the ground truth, flexibility at both gene- and transcript-level analysis, and the computational resource use, pseudoalignment methods Kallisto and Salmon are recommended for RNA-Seq analysis for lncRNAs, with Kallisto performing slightly better than Salmon. The full transcriptome annotation including protein-coding genes, lncRNAs, and others is also the recommended strategy for RNA-Seq analysis.The large amount of data produced by next-generation sequencing techniques has posed great challenges for fast and scalable data analysis. Our study findings imply that for RNA-Seq datasets, incorporating pseudoalignment methods into the analytical framework can achieve high accuracy with minimum computing requirements. Moreover, with more and more RNA-Seq datasets specifically for studying lncRNAs becoming available, our work also lays the basis for a more comprehensive evaluation of tools for lncRNA expression quantification.Here we clarify and define relevant terms. (i) Genes and transcripts: a transcript, sometimes also referred to as an isoform, is composed of exons. An exon is any part of a gene that will encode a part of the final mature RNA. A gene is a collection of transcripts. Transcripts of the same gene often share exons. In this study the analysis was performed at the gene level. (ii) Expressed genes: when describing a single sample, “expressed genes” refer to genes with FPKM ≥ 1 in that particular sample. When describing multiple samples in a dataset, “expressed genes” refer to the genes with median FPKM ≥ 1 across the cohort of samples. (iii) Discordant genes: these are defined as genes whose Spearman’s correlation of FPKM values with the ground truth is <0.7, when compared across a cohort of samples.Human transcriptome version GENCODE release 27 (GTF file and transcriptome fasta file) were downloaded from the GENCODE FTP site. The GENCODE release 27 collects 58,288 genes and 200,401 transcripts, among which are 19,836 protein-coding genes . If lncRThe NONCODE database  collectiThe indexes for Kallisto and Salmon were built using the transcriptome fasta file. The indexes for RSEM and STAR were built using transcriptome GTF file and GRCh38 genome sequences. The indexes for Subread and HISAT2 were built using GRCh38 genome sequences.k-mers of genes were generated using script from Computational Genomics Analysis and Training (CGAT).Gene features  were generated from GTF file using in-house script. Unique Nine pipelines were applied to process the datasets, including 2 pseudoalignment methods, Kallisto and Salmon; RSEM with bowtie as the aligner; and a combination of read aligners  and quantification tools (HTSeq and featureCounts).Kallisto, version 0.44.0, quant mode. Default parameters were applied. Stand-specific option was set as "–rf-stranded" for reverse-stranded samples.Salmon, version 0.9.1, quant mode. Default parameters were applied. Stand-specific option: "-l A." Kallisto and Salmon measure the expression level of each transcript by default. To get gene-level expression results, the package tximport  was usedSTAR, version 2.5.4a. Two-pass mode was used for mapping.Subread, version 1.6.1. Stand-specific option was set as "-S ff" for un-stranded samples and "-S rf" for reverse-stranded samples. Other settings: "–multiMapping -B 4 -t 0 ."HISAT2, version 2.1.0. Stand-specific option was set as "–rna-strandness RF" for reverse-stranded samples.HTSeq, version 0.7.2. Mapped reads from STAR, Subread, and HISAT2 were counted for each gene according to the GTF file from either GENCODE or GENCODE+NONCODE annotations. Stand-specific option was set as "-s no" for un-stranded samples and "-s yes" for reverse-stranded samples.featureCounts, version 1.6.1. Mapped reads from STAR, Subread, and HISAT2 were counted for each gene according to the GTF file from either GENCODE or GENCODE+NONCODE annotations. Stand-specific option was set as "-s 2" for reverse-stranded samples. HTSeq and featureCounts output read count for each gene. FPKM values were generated from read counts with in-house scripts.RSEM, version 1.3.0. The command "rsem-calculate-expression" was used together with bowtie aligner to obtain transcript- and gene-level quantification.To compare the computational resources required by each tool, RNA-Seq reads from 5 original samples were chosen and processed with each tool, with the number of threads set at 4.The gene expression measured by each tool was compared with ground truth at both sample level and gene level. In sample-level comparison, gene expression from a single sample was compared with ground truth for each method. In gene-level comparison, gene expression for a single gene across the samples was compared with ground truth for each method.Hierarchical clustering was performed for the sample method matrix. Euclidean distance and average linkage were used for both the columns and rows. The clustering dendrogram was cut into 4 groups and the number of times that every 2 methods are clustered together was counted and used to construct a similarity matrix.U test was used to test the difference between 2 continuous variables, and χ2 test was used to test the difference of 2 ratios. Unless otherwise specified, all the tests have P-values <0.001; thus, they are not explicitly explained in the main text.The Mann-Whitney The raw RNA-Seq sequencing data of TCGA samples were downloaded from the ISB Cancer Genomics Cloud and processed with Kallisto, using GENCODE (version 27) as transcriptome reference.Project name: RNASeq_pipelinehttps://github.com/gevaertlab/RNASeq_pipeline (licence: MIT)Project home page: Operating system(s): GNU/LinuxProgramming language: Linux/Bash, R, and PythonThe TCGA RNA-Seq re-analysis results and the simulated datasets are available in the Stanford Medicine Box .The web interface for investigating and visualizing individual gene expression can be found in Zheng and Gevaert .GigaScience database, GigaDB [Supporting data and code are also available via the , GigaDB .Additional File 1—The genes and transcripts in GENCODE release 27.Additional File 2—The percentage of expressed genes using each method and different gene annotation sets.Additional File 3—The effect of incomplete transcriptome annotation on the expression quantification of protein-coding genes.Additional File 4—The percentage of expressed lncRNA genes using each method and full annotation.Additional File 5—Examples of sample-level comparison of each method and the ground truth.Additional File 6—Sample-level comparison of gene expression.Additional File 7—Statistical tests for gene-level comparison between pseudoalignment methods and alignment-based methods.Additional File 8—Gene-level comparison of gene expression.Additional File 9—Features of total and expressed lncRNAs.Additional File 10—Feature (lncRNA type) of discordant lncRNAs.Additional File 11—Features of discordant lncRNAs.Additional File 12—Number of transcripts of discordant lncRNAs.Additional File 13—Transcript length of discordant lncRNAs.Additional File 14—Number of exons of discordant lncRNAs.Additional File 15—Sequence uniqueness of discordant lncRNAs.Additional File 16—Overall feature breakdown of GENCODE, expressed, and discordant lncRNAs.Additional File 17—Read mapping of discordant lncRNAs.giz145_GIGA-D-19-00113_Original_SubmissionClick here for additional data file.giz145_GIGA-D-19-00113_Revision_1Click here for additional data file.giz145_GIGA-D-19-00113_Revision_2Click here for additional data file.giz145_GIGA-D-19-00113_Revision_3Click here for additional data file.giz145_GIGA-D-19-00113_Revision_4Click here for additional data file.giz145_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giz145_Response_to_Reviewer_Comments_Revision_1Click here for additional data file.giz145_Response_to_Reviewer_Comments_Revision_2Click here for additional data file.giz145_Response_to_Reviewer_Comments_Revision_3Click here for additional data file.giz145_Reviewer_1_Report_Original_SubmissionBo Li, Ph.D. -- 6/2/2019 ReviewedClick here for additional data file.giz145_Reviewer_1_Report_Revision_1Bo Li, Ph.D. -- 10/5/2019 ReviewedClick here for additional data file.giz145_Reviewer_2_Report_Original_SubmissionSerghei Mangul -- 6/5/2019 ReviewedClick here for additional data file.giz145_Reviewer_2_Report_Revision_1Serghei Mangul -- 10/23/2019 ReviewedClick here for additional data file.giz145_Reviewer_2_Report_Revision_2Serghei Mangul -- 11/4/2019 ReviewedClick here for additional data file.giz145_Reviewer_3_Report_Original_SubmissionAndrey D. Prjibelski, M.Sc. -- 6/11/2019 ReviewedClick here for additional data file.giz145_Supplemental_FileClick here for additional data file.bp: base pairs; CPU: central processing unit; FPKM: fragments per kilobase million; HOTAIR: Hox transcript antisense RNA; ISB: Institute for Systems Biology; lincRNA: long intergenic non-coding RNA; lncRNA: long non-coding RNA; NCBI: National Center for Biotechnology Information; NIH: National Institutes of Health; RNA-Seq: RNA sequencing; SRA: Sequence Read Archive; TCGA: The Cancer Genome Atlas; TERC: telomerase RNA component; TERT: telomerase reverse transcriptase.The authors declare that they have no competing interests.This work was supported by the National Institute of Dental and Craniofacial Research (NIDCR) (U01 DE025188), the National Institute of Biomedical Imaging and Bioengineering (R01 EB020527), and the National Cancer Institute (U01 CA217851), all of the NIH. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. The funders did not play a role in the design and execution of this study.H.Z. and O.G. conceived and designed the study. H.Z., K.B., and M.H. performed data analysis. H.Z. wrote the manuscript, and all authors participated in preparing the manuscript.


Batch effects were not accounted for in most of the studies of computational drug repositioning based on gene expression signatures. It is unknown how batch effect removal methods impact the results of signature-based drug repositioning. Herein, we conducted differential analyses on the Connectivity Map (CMAP) database using several batch effect correction methods to evaluate the influence of batch effect correction methods on computational drug repositioning using microarray data and compare several batch effect correction methods.limma) software demonstrated little agreement. The external validity of the gene signatures was evaluated by connectivity mapping between the CMAP database and the Library of Integrated Network-based Cellular Signatures (LINCS) database. The results of connectivity mapping indicate that the genes identified were not reliable for drugs with total sample size (drug + control samples) smaller than 40, irrespective of the batch effect correction method applied. With total sample size larger than 40, the methods correcting for batch effects produced significantly better results than the method with no batch effect correction. In a simulation study, the power was generally low for simulated data with sample size smaller than 40. We observed best performance when using the limma method correcting for two principal components.Differences in average signature size were observed with different methods applied. The gene signatures identified by the Latent Effect Adjustment after Primary Projection (LEAPP) method and the methods fitted with Linear Models for Microarray Data (limma when sample size is sufficient (larger than 40 drug and controls combined).Batch effect correction methods strongly impact differential gene expression analysis when the sample size is large enough to contain sufficient information and thus the downstream drug repositioning. We recommend including two or three principal components as covariates in fitting models with The online version of this article (10.1186/s12859-019-3028-6) contains supplementary material, which is available to authorized users. Drug repositioning is the process of finding new indications for existing drugs. If successful, it has advantages over de novo drug development in terms of potentially shorter development times, less costs and risks . Facilitcombining batches of microarray data) applies Empirical Bayes estimation to adjust the mean and the variance by pooling information across multiple genes in order to perform gene-wise batch corrections for mean and variance [Despite CMAP having been demonstrated to be valuable and successful, it still has some limitations. These include a limited number of cell-lines and the fact that batches were required to generate all the data. Both factors can lead to biased analyses and here we focus on batch effects. Batch effects are defined as technical variations that have been introduced by time varying external factors during handling of the samples or effects of sample handling itself. Such factors include various sources, such as personnel effects, environmental conditions, different experiment times, etc. , some ofvariance , which ivariance . While Rvariance . SurrogaHowever, in many drug repositioning studies, gene expression profiles were directly used from either CMAP or LINCS without correcting for batch effects , 10, 13.limma, and the LEAPP method. We evaluate the quality of the gene signatures generated by these methods by gene set enrichment analyses on the shared drugs between the CMAP database and the LINCS database  was performed for every gene expression data matrix jointly for the treatment and control gene expression profiles corresponding to each drug. As is shown in Fig. limma package (version 3.32.5). The null model always contained the log-transformed concentrations. Subsequently, we tested if adding either the batch id or PCs improves the external validity of the genes identified as differentially expressed, as discussed below in the section on connectivity mapping. We included several sets of covariates to adjust for batch effects: i) null; ii) batch id (corresponding to the plate id); iii) one or more largest PCs (continuous variable). The linear associations between features and the drugs were also assessed by fitting models with the LEAPP method, for which no batch information was provided.Gene expression differences between drugs and vehicle controls were analyzed with linear models fitted by the limma , indicating numeric convergence problems, which prevented meaningful gene set enrichment analysis. Therefore, these results were not further analyzed.After filtering out the genes with coefficients of variation outside the 20 and 80% quantiles, the number of genes included in the differential expression analysis decreased to 8131. Due to insufficient sample size, which did not allow some linear models to be fitted, some of the drugs do not have results produced in differential expression analysis, as illustrated in Table mma Fig. . Comparidel Fig. a. Simila00% Fig. b. Table limma methods correcting either for two, three or four PCs, or batch id . The superior performance of sample size > 40 is further demonstrated by the plot of the high proportion of drugs (19–36%) having the connectivity scores of the same drugs ranked within top 3 for every method stratified by group, compared to the low proportion of drugs within rank 3 (2–13%) observed in the group with sample size ≤40  calculated by Gene Set Enrichment Analysis using function ConnectivityScore implemented in PharmocoGx package . For eac id Fig. a. The medrugs 19–6% havingWe also performed the gene set enrichment analysis using sets of significant genes with FDR ≤ 10% and the results are plotted on Additional file An increasing trend was observed for the methods that correct for two to four PCs. When the cut-off rank was relaxed from top 3 to top 10, similar results were obtained  when the medium batch effects simulated from principal component loadings were added to the expression data, the power decreased by 10% for the method without batch effect correction, while the power of the methods correcting for two and three PCs only decreased by no more than 3% . Therefore, the drug signatures generated by the differential expression methods investigated in this study were probably not reliable when the total sample size was smaller than 40, in the sense that the drug signatures are probably unable to perform well in downstream analysis of drug repositioning, no matter whether batch effects were corrected for or not. This was also supported by the simulation results, where we observed extremely low power in every simulated data with total sample size smaller than 40. Therefore, we conclude that more than 40 total samples are needed to generate reliable drug signatures from CMAP data.limma methods, but we were unable to validate the quality of the drug signatures generated by the method or determine if the method was better in analyzing the CMAP dataset than the limma methods. The LEAPP method generated many estimates being exactly 0 in both the CMAP and the LINCS datasets, which prevented us from running the gene set enrichment analysis. Most likely, sample size in our application was too low for LEAPP to work reliably but we did not investigate this hypothesis in detail. We were unable to run simulations for LEAPP as it was too time consuming.The LEAPP method was not effective in our analysis—at least in the way we used it—although the method is theoretically advantageous and convenient . The dilimma methods, correcting for two and three PCs performed equally well as correcting for batch id when analyzing data with large sample size, as was indicated by the analysis of connectivity mapping. Nevertheless, the method adjusting for PCs has the potential to outperform the method adjusting for batch id for the following reasons: i) PC scores are continuous, which could detect relatively small technical differences within batches, such as, the temperature gradient on plates, and thus could have benefits over categorical variables like batch id; ii) PCs can be directly generated from the gene expression data so that the researcher does not need to rely on accurate batch labels; iii) PCs can be analyzed in a more refined fashion. For example, control genes could be introduced in the analysis, as is applied in the RUV-2 method [Among the 2 method . SecondlIn the simulation study, correcting for two PCs achieved relatively higher power and fewer false positives than correcting for other numbers of PCs in the simulated data of sufficiently large total sample size. In gene set enrichment analysis, though, correcting for three PCs performed relatively better. Based on the results, we recommend correcting for two or three PCs in data with sufficiently large sample size.The simulation study also suggested less conservative FDR cutoff value should be considered. We speculate that the increase of the FDR threshold could increase power with small trade-off on the proportions of false positives, which might improve the results in the gene set enrichment analysis.It is likely that PCs are unstable when the sample size is small, which may be one of the reasons that it did not perform well in data with small sample size in our analysis. PC correction can be adapted by applying weights in PCA by borrowing information from other data, such as, data from the same batch or by shrinking the covariance matrix towards the identity matrix . We herep-values, or introducing another parameter determining the degree of weighting.In our study, we performed the analysis of connectivity mapping between two databases on the shared drugs. On the one hand, we implemented the practical procedure of computational drug repositioning. On the other hand, we provided a method to evaluate the quality of the drug signatures generated by differential expression methods, where we sought to find the same drugs back in the top of the lists ordered by the connectivity scores. Because the same drug is expected to affect the same cell line in different databases similarly, this could be considered a “gold standard”. However, there were also some limitations to this approach. Firstly, not all the cell types used in CMAP are available in LINCS. The analysis was done without matching the cell types between the two databases. Although ignoring the cell types may add noise to the analysis, the results are unbiased and robust. On the other hand, matching cell types would remove several samples from CMAP and thus take the analysis further away from real applications. Secondly, we imposed a fixed number of 15 genes in the signature for the analysis, which was rather small and may negatively affect results. The minimum number for gene set enrichment analysis was suggested to be 25 so as to avoid inflation of scorings . We chosP-values for these cases , was extremely low. As can be seen in Additional file In the simulation study, we simulated both the case and control data under the null hypothesis by extracting the variance-covariance matrix from the real data of the vehicle controls, thereby capturing both biological and batch effects in the covariance matrix. Instead of interpreting such data as batch-effect free, we see it as a starting point for the simulations with a realistic covariance structure which is not necessarily identical to that of the actual drug. Moreover, the drug effects simulated were based on point estimates of differential expression analysis of real data which only reflect the truth up to uncertainty in estimates. The absence of further modifications of drug effects implies that some effects are overestimated and are exaggerated in the simulations. In the simulations, where the simulated drug effects were small, the power to detect the differences between simulated cases and simulated controls was expected to be small as well. Lastly, the batch effects were simulated from the first two PCs of the PCA, which is probably the reason that the method correcting for two PCs performed better than the other methods. On the other hand, the real data analysis supports the more general conclusion put forward in this discussion.limma when the total sample size is large enough (at least > 40 drug and controls combined), which applies to most of the drugs in LINCS . However, for drugs of smaller total sample size, if analyzed with the methods discussed in this paper, the results should be interpreted with caution. Dealing with small sample sizes seems to require more method development.Our study highlighted the importance of batch effect correction in computational drug repositioning, especially in generating gene expression signatures with the CMAP dataset, which has been used in at least 2800 studies. We recommend exercising caution in selecting proper batch effect correction methods. In applying the methods discussed in this study, sufficient sample size is essential to assure the validity of results. It is advisable to adjust for two or three PCs in the models fitted by Future work can include: i) applying weights and/or regularization in PCA on data with small sample size; ii) evaluating the optimal number of genes to be used in gene set enrichment analysis; iii) conducting simulations with various sizes of drug effects.combat function in the SVA package (version 3.10.0) [CMAP database (build 2) was downloaded using the PharmacoGx package (version 1.6.1) . Pre-pro 3.10.0) . The CMAThe LINCS database in the level 3 format was obtained from the NCBI Gene Expression Omnibus (GEO) dataset (GSE92742), which was pre-processed by invariant set scaling and quantile normalization . The numTo avoid the effect of influential observations on the analyses, for every drug, samples were excluded if the concentration value used for the perturbation was more than 1.5 times the interquartile range above the third quartile or below the first quartile of the concentration values. The vehicle controls, i.e. samples containing only solvents for the active drug, from the same batches as the excluded drug samples were excluded as well. The total sample size of certain drug is therefore the sum of the number of drug samples and the number of the corresponding vehicle controls after exclusion.To reduce the computational burden of the analyses while increasing the statistical power, we applied non-specific gene filtering by removing genes with coefficients of variation outside the 20 and 80% quantiles (coefficient of variation is the ratio of the standard deviation to the mean).prcomp.PCA was used both as a descriptive tool to evaluate the existence of batch effects and as a correction method . Scores e plus one transformed (i.e. the mean of the log-transformed concentrations of vehicle controls was zero).Concentrations of the vehicle controls were set to zero, while the concentrations of the drugs were rescaled to molar concentrations, and subsequently logLimma is an R/Bioconductor software package that fits linear model to each row that represents a gene in an gene expression matrix, as well as borrows information from the other genes analyzed, thus providing more reliable statistical results [ results .The model without batch effect correction:yi is the expression value of sample i, β0 is the intercept, β1 is the drug effect, Xi is loge plus one transformed molar concentrations of the drug, ξi is the residual for sample i.where limma package (version 3.32.5).We assessed the following batch effect correction methods that adjusted for covariates in linear models fitted by the By including batch id (corresponding to the plate used to incubate the sample) while fitting linear model, this method adjusts the mean of the expression levels by the contrast of a batch with the reference batch. yi is the expression value of sample i, β0 is the intercept, β1 is the drug effect, Xi is loge plus one transformed molar concentrations of the drug, β2,..., βj + 1 are the coefficients of the dummy variables for batch IDs, Zi1, ..., Zji are the dummy variables for batch IDs (j indexes the batches) of sample i, ξi is the residual for sample i.where The method adjusts for batch effect by including several PCs starting from the first as covariates while fitting the linear model. These PCs are believed to capture batch effects under the assumption that the variation caused by batch effects is much larger than the variation caused by drug effects. The method is similar to RUV-2 but without applying the PCA on negative control genes, as drug specific control genes have not been determined. The optimal number of PCs needed to capture the batch effect is part of the evaluation. In formula.yi is the expression value of sample i, β0 is the intercept, β1 is the drug effect, Xi is loge plus one transformed molar concentrations of the drug, β2,..., βp are the coefficient of the principal components, Ci1, ..., Cpi are the scores of the first 1 to p principal component(s) of sample i, p = 1, 2, 3 or 4, ξi is the residual for sample i.where limma package was employed to moderate estimated gene variances generated by limma models.Empirical Bayes procedures implemented in the e plus one transformed concentrations as primary variables, assuming sparsity of the primary parameter. IPOD algorithm in Owen and She was applied to enforce sparsity [The LEAPP method attempts to automatically separate batch effects from the biological effects of interest by an estimation procedure. An attractive feature of the method is that it obviates the need of a list of control genes. The model estimates latent vectors corresponding to PCs so that residuals become uncorrelated, i.e. clustering in the data is removed. The number of latent variables is subject to variable selection and the method can be seen as PCA correction that searches control genes implicitly. The detailed description of the method can be found in the paper of Sun, et al. . The anasparsity ; hard thP-values were adjusted with Benjamini-Hochberg approach to control the false discovery rate (FDR). The significance level is defined at FDR 10% but other commonly used FDR levels  were also assessed.The resulting The connectivity scores were calculated by Gene Set Enrichment Analyses (GSEA) with the function ConnectivityScore in PharmacoGx package (version 1.6.1) . The Benlimma, and ii) the models fitted by limma with different number of PCs included as covariates. We based our simulated data on the correlation structure of real data corresponding to a representative selection of drugs and its vehicle controls. By simulating the same sample size as for the real data, simulations closely follow a realistic setting (significance level defined at FDR ≤ 10%). Simulated data under the null hypothesis were generated from real expression data of vehicle controls, with noise added by sampling from multivariate distribution with mean 0 and covariance matrix extracted block-wise from the data (1000 genes per block). Afterward drug effects extracted from the linear models fitted by limma on log-transformed drug concentration and adjusted for two PCs, were added to the simulated treatment group. Further, rescaled loadings of the first two PCs from the PCA on the real expression data of both the treatment and the control groups were used to simulate batch effects. The rescaling factors were 0, 2 and 4, representing no, medium and large batch effects respectively. The batch effects were simulated in four scenarios: 1) without loadings, 2) only the first principal component (PC1), 3) only the second principal component (PC2), and 4) PC1 and PC2. These four scenarios were always applied to the complete simulated case data, but the percentage applied to the control data thus modified depending on whether the batch effect was simulated as balanced or not. Thirty percent differences in batch allocation were imposed to simulate unbalanced designs. FDR cutoff values at 10 and 20% were evaluated. The simulation was conducted 10 times per drug and per setting.Simulation studies were performed to compare i) the null model fitted by Additional file 1:Figure S1. Score plots of the first two principal components for four typical drugs . Colors indicate batch (plate id) and shapes indicate cell type. (PDF 1116 kb)Additional file 2:Figure S2. Using connectivity mapping to evaluate batch effect correction methods illustrated by ciclopirox. First, both CMAP and LINCS underwent the differencial expression analyses with the same batch effect correction methods, which resulted in drug signatures for all the drugs; second, the drug signature of ciclopirox in CMAP matched to all the drug signatures in LINCS, and the resulted connnectivity scores were ranked, where we expect that ciclopirox appears within the top three of the ranked list when the drug signature generated by the method is of high validity and good quality. (PDF 85 kb)Additional file 3:Figure S3. Results of connectivity score analysis with a fixed number of 15 genes with the lowest FDR. The y axis is the proportion of drugs having the same drug ranked within top 10 in connectivity mapping between shared genes of CMAP and LINCS dataset. The error bars are the 95% confidence levels as estimated by binomial test. The x-axis is grouped by the total sample size in CMAP dataset. The colors indicate the differential gene expression analysis methods. (PDF 488 kb)Additional file 4:Figure S4. Results of connectivity score analysis with all significant genes (FDR ≤ 10%). Only drugs with at least 10 significant genes yielded were included in the analysis. The y axis is the proportion of drugs having the same drug ranked within top 3 or 10 in connectivity mapping between shared genes of CMAP and LINCS dataset. The error bars are the 95% confidence levels estimated by binomial test. The x-axis is grouped by the differential gene expression analysis methods. The colors indicate the total sample size in CMAP dataset. (PDF 1461 kb)Additional file 5:Figure S5. Results of connectivity score analysis with all significant genes (FDR ≤ 5%). Only drugs with at least 10 significant genes yielded were included in the analysis. The y axis is the proportion of drugs having the same drug ranked within top 3 or 10 in connectivity mapping between shared genes of CMAP and LINCS dataset. The error bars are the 95% confidence levels estimated by binomial test. The x-axis is grouped by the differential gene expression analysis methods. The colors indicate the total sample size in CMAP dataset. (PDF 1492 kb)Additional file 6:Figure S6. Results of connectivity score analysis with all significant genes (FDR ≤ 20%). Only drugs with at least 10 significant genes yielded were included in the analysis. The y axis is the proportion of drugs having the same drug ranked within top 3 or 10 in connectivity mapping between shared genes of CMAP and LINCS dataset. The error bars are the 95% confidence levels estimated by binomial test. The x-axis is grouped by the differential gene expression analysis methods. The colors indicate the total sample size in CMAP dataset. (PDF 1433 kb)Additional file 7:Figure S7. Results of the simulation study without batch effects and FDR < 10%. A, log10 transformed number of significant genes averaged over 10 simulations; B, Proportion of false positives among the significant genes averaged over 10 simulations; C. the power of the analysis averaged over 10 simulations. (PDF 95 kb)Additional file 8:Figure S8. Results of simulation study with medium batch effects and FDR < 10%. A, log10 transformed number of significant genes averaged over 10 simulations; B, Proportion of false positives among the significant genes averaged over 10 simulations; C. the power of the analysis averaged over 10 simulations. (PDF 95 kb)Additional file 9:Figure S9. Results of simulation study with large batch effects and FDR < 10%. A, log10 transformed number of significant genes averaged over 10 simulations; B, Proportion of false positives among the significant genes averaged over 10 simulations; C. the power of the analysis averaged over 10 simulations. (PDF 94 kb)Additional file 10:Figure S10. Results of simulation study with medium batch effects, FDR < 10% and unequal allocation of cases and controls. Medium batch effect simulated with 0.3 differences between cases and controls. A, log10 transformed number of significant genes averaged over 10 simulations; B, Proportion of false positives among the significant genes averaged over 10 simulations; C. the power of the analysis averaged over 10 simulations. (PDF 94 kb)Additional file 11:Figure S11. Results of simulation study with medium batch effects and FDR < 20%. A, log10 transformed number of significant genes averaged over 10 simulations; B, Proportion of false positives among the significant genes averaged over 10 simulations; C. the power of the analysis averaged over 10 simulations. (PDF 94 kb)Additional file 12:Figure S12. Negative log 10 of P-values plotted against absolute estimates of extracted drug effects of felodipine, fulvestrant and vorinostat. (PDF 391 kb)Additional file 13:Figure S13. Histograms of P-values resulted from differential expression analyses on one set of data simulated from colchicine with balanced batch design and median batch size (parameter settings see Table Additional file 14:Figure S14. Histograms of P-values resulted from differential expression analyses on one set of data simulated from vorinostat with balanced batch design and median batch size (parameter settings see Table