The ultra-high risk (UHR) state was originally conceived to identify individuals at imminent risk of developing psychosis. Although recent studies have suggested that most individuals designated UHR do not, they constitute a distinctive group, exhibiting cognitive and functional impairments alongside multiple psychiatric morbidities. UHR characterization using molecular markers may improve understanding, provide novel insight into pathophysiology, and perhaps improve psychosis prediction reliability. Whole-blood gene expressions from 56 UHR subjects and 28 healthy controls are checked for existence of a consistent gene expression profile (signature) underlying UHR, across a variety of normalization and heterogeneity-removal techniques, including simple log-conversion, quantile normalization, gene fuzzy scoring (GFS), and surrogate variable analysis. During functional analysis, consistent and reproducible identification of important genes depends largely on how data are normalized. Normalization techniques that address sample heterogeneity are superior. The best performer, the unsupervised GFS, produced a strong and concise 12-gene signature, enriched for psychosis-associated genes. Importantly, when applied on random subsets of data, classifiers built with GFS are “meaningful” in the sense that the classifier models built using genes selected after other forms of normalization do not outperform random ones, but GFS-derived classifiers do. Data normalization can present highly disparate interpretations on biological data. Comparative analysis has shown that GFS is efficient at preserving signals while eliminating noise. Using this, we demonstrate confidently that the UHR designation is well correlated with a distinct blood-based gene signature. Prodromal (early) intervention has reportedly beneficial effects in attenuating, delaying, and even preventing psychosis onset  as objective data. Earlier studies reported identifiable gene expressional differences between psychotic and healthy individuals in both blood and brain tissues , a prospective observational study on youths susceptible to psychosis  and is stored at − 80°C until RNA extraction. Total blood RNA is extracted using a Tempus Spin RNA isolation kit (Applied Biosystems) and amplified using an Illumina TotalPrep RNA amplification kit . mRNA expression profiles are assessed on Illumina HumanHT-12 v4 Expression BeadChip arrays. Experimental quality controls are performed prior to RNA amplification and before RNA hybridization to ensure that RNA concentrations are ≥100 ng/ml, Ap values below 0.05 or below background are considered undetectable and discarded. A total of 84 subjects (56 UHRs + 28 non-UHRs) across 18,029 gene probes meets QC standards.Probe intensities on the Illumina Microarray BeadChips are summarized using GenomeStudio . To ensure read quality, sample-based and probe-based quality control (QC) is performed. In sample QC, samples with signal-to-noise ratio below 10 or samples with intensity of negative control probes higher than control probes are considered unreliable and discarded. In probe QC, probes with detection Following background subtraction, probe-to-gene name mapping, and following median centering of expression, four normalization approaches are applied on the quantified data: Log-conversion (None), quantile normalization (Quantile), gene fuzzy scoring  and improves the distribution symmetry. However, None is usually insufficient to render samples cross comparable and is usually accompanied by a second data transformation, typically z scaling, linear interpolation, or quantile normalization.Strictly speaking, None is not true normalization; we use this primarily as a negative control and to facilitate parametric statistics. We use the natural log, such that for each observed measurement z scaling, these are already known to be inferior to Quantile anyway and detract from more interesting new methods that explicitly deal with heterogeneity. GFS is an unsupervised signal-boosting transformation  be the rank of a biological feature gi in patient pj and q be the rank corresponding to the upper θth level of feature ranks in pj. The GFS score s assigned to feature gi for patient pj is determined by the functionQuantile is a gold-standard data transformation procedure and reportedly performs well in stabilizing variance in Illumina BeadChip HT-12  into a lower number of orthogonal dimensions, or principal components .We also use the PCA-generated loading matrix to determine the association of individual PCs against each factor . Assuming that a factor may have more than two levels, association is evaluated using the nonparametric Kruskal–Wallis (KW) test  multiple-test correction.The F statistic is expressed asYi,j is the jth observation in the ith group over K groups. N is total sample size, and ni is the size of the ith group. i, and F is the ratio of intergroup over intragroup variances and is large if the groups are strongly different. The p value is determined against the nominal F distribution with  degrees of freedom.The Samples are evenly split into training and validation sets. All features are used to train a shrunken-centroid classifier Dabney, . Cross-vF test/BH correction (p ≤ 0.01).The same procedure was performed as described earlier, except feature selection in the training set (signature) is performed using To demonstrate robustness, given each split, an equal number of random features was selected, and the cross-validation accuracy based on randomly selected features  was determined.http://www.genemania.org/) is used to evaluate functional relationships among differential genes  should be ignored. Each PC uniquely encapsulates a distinct portion of total variance and is mutually orthogonal. We may deploy the PCs more effectively by quantifying the association of each PC against known factors . If sample class is associated significantly among the top PCs, then this supports the scatterplot analysis. Conversely, we may also reverse engineer interesting PCs and identify which genes load strongly onto it , then we assume that this factor can explain this PC. As a rule, for good normalization methods, the top PCs should be significantly associated with class, followed by other important factors.To test association, the KW test is used, accounting for scenarios where a factor has more than two levels. If a PC exhibits strong differential behavior between factor levels . Note that GFS is unsupervised and robust against heterogeneity and technical bias . The F test is used to ensure cross-comparability of the other methods with SVA as it reduces overall variability, while the degrees of freedom, given any statistical test, remain the same. While this amplifies effect size , it also increases the false-positive rate . Thus SVA requires an appropriate correction (provided within the SVA package on the F test).If normalization has limited impact on feature selection, then given a single feature-selection method, selected features from differently normalized datasets should be similar. Genes are selected using the p value distributions (extreme right skew), it is clear that None is problematic , because rank instability increases as p values get smaller  is shared. Quantile, GFS, and SVA exhibit deeper overlaps with each other. In particular, GFS shares 80% (four out of five) of its selected features with Quantile and SVA, but not with None. And these overlapping genes are important: Their associated p values (based on SVA) are far lower (and therefore more significant) than nonoverlapping genes . The stringent cutoff is used for limiting the high number of selected features due to log-conversion. For phenotypic correlation, we may bolster sensitivity by relaxing the p value cutoff to 0.05 .GFS selected five genes . It is interesting that feature selection with GFS reduces classifier accuracy . We should not rely on feature-selection methods to work properly without proper data cleaning. As it turns out, most feature-selection methods are not resistant to technical bias or noise  alone is insufficient and may produce false effects. The oft-used quantile normalization (Quantile) provides considerable improvement but may not be good enough for practical applications. Finally, the highly sophisticated technical variability removal method, SVA, is powerful, but as a supervised method, it is susceptible to the veracity of the class labels: If wrong, for example, owing to misdiagnosis, then SVA can make mistakes. Finally, we introduce the unsupervised signal-boosting transformation GFS, which provides the highest data-cleaning utility .Although GFS’s performance suffers when combined with a feature-selection approach, it is the only method that can strongly outperform random signatures . This is not always the case where, following certain normalization approaches, the distribution of variance among PCs becomes more balanced . AlthougWe may use PCs for conducting correlation checks with known factors: If technical batches are known, we may determine which PC is associated with batch. If correlated with PC1, batch effect dominates, and skews feature selection. Similarly, examining PC correlations with other factors determines if they contribute strongly and meaningfully to the model. In particular, we expect gender and racial effects to be present. Indeed, these are correlated with top PCs, except in SVA. Detailed PC examination reveals the influence of various factors, but we need to figure out which genes are responsible. This is possible (but we did not deploy this technique explicitly here): Suppose PC1 is strongly associated with batch; we can go back to the PCA loading matrix and isolate genes that load strongly onto PC1, thus isolating the batch effect–susceptible genes  are reported consistently, they are immune associated and do not seem consistent with the UHR phenotype. However, the immune system and schizophrenia have a long history of intricate association  with high predictive power.Given good normalization, and demonstrated independent reproducibility, blood-based gene signatures have the potential to help identify UHRs accurately and, it is hoped, improve clinical outcomes via early intervention.W. W. B. Goh and L. S. Wong designed and conducted the bioinformatics analyses. J. C. G. Sng performed the sample preparations. J. Y. Yee and Y. M. See collected patient samples. T. S. Lee and J. Lee profiled the patient cohort.The Singapore Translational and Clinical Research in Psychosis  is supported by the National Research Foundation Singapore under the National Medical Research Council Translational and Clinical Research Flagship Program (grant NMRC/TCR/003/2008). L. W. is supported in part by a Kwan Im Thong Hood Cho Temple chair professorship.


Clustering is a prevalent analytical means to analyze single cell RNA sequencing (scRNA-seq) data but the rapidly expanding data volume can make this process computationally challenging. New methods for both accurate and efficient clustering are of pressing need. Here we proposed Spearman subsampling-clustering-classification (SSCC), a new clustering framework based on random projection and feature construction, for large-scale scRNA-seq data. SSCC greatly improves clustering accuracy, robustness, and computational efficacy for various state-of-the-art algorithms benchmarked on multiple real datasets. On a dataset with 68,578 human blood cells, SSCC achieved 20% improvement for clustering accuracy and 50-fold acceleration, but only consumed 66% memory usage, compared to the widelyused software package SC3. Compared to k-means, the accuracy improvement of SSCC can reach 3-fold. An R implementation of SSCC is available at https://github.com/Japrin/sscClust. Single cell RNA sequencing (scRNA-seq) has revolutionized transcriptomic studies by revealing the heterogeneity of individual cells with high resolution ad hoc computational strategies have been proposed by downsampling or convoluting large datasets to small ones 2) to O(n) while maintaining high clustering accuracy. Moreover, flexibility of the new computational framework allows our methods to be further extended and adapted to a wide range of applications for scRNA-seq data analysis.To handle multiple large-scale scRNA-seq datasets, i.e., O(n). Such a framework generally consists of four steps (k-means k-nearest neighbors 2) to O(n) by leveraging the efficiency of supervised machine learning algorithms. However, classifiers built on the original gene expression data of a small subset of cells may be flawed and biased due to noise of the raw data and small number of cells, thus impairing the accuracy of label assignment for the total cells.Among the available solutions to handle large scRNA-seq datasets, clustering with subsampling and classification ur steps A. (1) a Here we proposed a new computational framework for clustering large scRNA-seq data by adding a feature engineering/projecting step into SCC B. SimilaFeature engineering techniques involved in this study include distance-based methods (Euclidean and cosine), correlation-based methods  using the Fluidigm C1 system and applying the SMARTer Kit to obtain cDNA and the Nextera XT Kit for Illumina library preparation.The Kolodziejczyk dataset etc. Either low or high sequencing depth based on the C1 Single-Cell Auto Prep Integrated Fluidic Circuit, the SMARTer Ultra Low RNA Kit, and the Nextera XT DNA Sample Preparation Kit was used to depict the gene expression profiles of individual cells .The Pollen dataset i.e., peptidergic nociceptor-containing, nonpeptidergic nociceptor-containing, neurofilament-containing, and tyrosine hydroxylase-containing cells. The neuronal cells were picked with a robotic cell-picking setup and positioned in wells of 96-well plates before RNA-seq .The Usoskin dataset The Zeisel dataset The Zheng dataset The PBMC 68 k dataset The Macoskco dataset i.e., the Kolodziejczyk, Pollen, Usoskin, and Zeisel datasets) have been widely used for evaluating clustering algorithms, of which the preprocessed data have been included in the SIMLR software package for test use (https://github.com/BatzoglouLabSU/SIMLR). We downloaded these four datasets from the Matlab subdirectory of the SIMLR package, and then selected the top 5000 most informative genes  for subsequent analysis. If the number of genes in a dataset was smaller than 5000, then all the genes in the dataset were retained for further analysis. For the Zheng dataset, one patient (P0508) was selected for comparison of different clustering algorithms, which had 1020 T cells with eight subtypes defined by the tissue sources and the cell surface markers. Genes with both the average and the standard deviation of log2-transformed expression values >1 were retained and then the transcripts per million (TPM) values were used for clustering evaluation. For the PBMC 68 k dataset, the preprocessing pipeline described in the original report https://github.com/10XGenomics/single-cell-3prime-paper). For the Macoskco dataset, the UMI counts were used for evaluation without gene filter.The first four datasets  i.e., affinity propagation (AP) k-means k-medoids i.e., SC3 k-means and k-medoids are pure clustering algorithms that partition samples into groups while AP, SC3, and SIMLR inherently include feature construction techniques. All these clustering algorithms were evaluated on five small-scale datasets , while only SC3 was evaluated on the PBMC 68 k dataset and only k-means was evaluated on the Macoskco dataset for simplicity. Parameters  were used for SC3 (default), whereas parameters  were used for SC3 + SSCC. On the Macoskco dataset, ∼5% and 10% cells were randomly picked out for clustering analyses. We used the k-nearest neighbor algorithm for classifying unsubsampled cells, which is robust to parameter selection.Many clustering algorithms are available. We selected five widelyused clustering algorithms in this study to evaluate the impacts of Spearman correlation-based feature construction method. These five algorithms include three general clustering algorithms which were designed initially not for scRNA-seq data, First we evaluated whether feature extraction methods can improve clustering results of scRNA-seq data. We calculated the silhouette values to evaluate the consistency between cell features extracted using various methods and the reference labels. Silhouette values are frequently used to indicate whether a sample is properly clustered. But here we can use silhouette values to reversely indicate whether the extracted features are properly consistent with the reference cell labels. By comparing with silhouette values of the original scRNA-seq data, we observed that most of the evaluated feature-extracting methods can improve the silhouette values for many cells in multiple datasets . For theWhile subsampling can greatly boost the efficiency of clustering of large scRNA-seq data, it often compromises the clustering accuracy. We observed that the improvements of silhouette scores by SSCC were robust to subsampling fluctuations . For allk-means, k-medoids, AP, SC3, and SIMLR, we observed that compared to SCC, SSCC can significantly improve the clustering accuracy in terms of NMI, for all the five clustering algorithms on all the benchmark datasets tested  between two subsampling schemes (−0.80 with 5% cells and −0.51 with 10% cells), whereas the difference using SSCC became negligible (0.01). The NMI values between the two subsampling schemes were 0.60 and 0.69 when using SCC and SSCC, respectively. Pearson correlation coefficients of silhouette values between the two subsampling schemes were increased from 0.47 to 0.58 when switching from SCC to SSCC  to O(n). Here we introduce feature engineering and projecting techniques into the SCC framework and propose SFCC as an alternative. Specially, with Spearman correlations as the feature engineering and projecting methods, we formulate a framework named as SSCC, which can significantly improve clustering accuracy and consistency for many general and speciallydesigned clustering algorithms. Evaluations on real scRNA-seq datasets, which cover a wide range of scRNA-seq technologies, sequencing depths, and organisms, demonstrate the robustness of the superior performance of SSCC. Therefore, SSCC is expected to be a useful computational framework that can further unleash the great power of scRNA-seq in the future.The availability of large-scale scRNA-seq data raises urgent need for efficient and accurate clustering tools. Currently a few scRNA-seq data analysis packages have been proposed to address this challenge. Of these tools, SC3 XR and ZZ designed the study. XR and LZ collected the data, implemented the software, and did the analysis. XR and ZZ wrote the manuscript. All authors read and approved the final manuscript.The authors have declared no competing interests.


The study of high-throughput genomic profiles from a pharmacogenomics viewpoint has provided unprecedented insights into the oncogenic features modulating drug response. A recent study screened for the response of a thousand human cancer cell lines to a wide collection of anti-cancer drugs and illuminated the link between cellular genotypes and vulnerability. However, due to essential differences between cell lines and tumors, to date the translation into predicting drug response in tumors remains challenging. Recently, advances in deep learning have revolutionized bioinformatics and introduced new techniques to the integration of genomic data. Its application on pharmacogenomics may fill the gap between genomics and drug response and improve the prediction of drug response in tumors.50 values of 265 drugs. We trained and tested the model on a dataset of 622 cancer cell lines and achieved an overall prediction performance of mean squared error at 1.96 . The performance was superior in prediction error or stability than two classical methods (linear regression and support vector machine) and four analog DNN models of DeepDR, including DNNs built without TCGA pre-training, partly replaced by principal components, and built on individual types of input data. We then applied the model to predict drug response of 9059 tumors of 33 cancer types. Using per-cancer and pan-cancer settings, the model predicted both known, including EGFR inhibitors in non-small cell lung cancer and tamoxifen in ER+ breast cancer, and novel drug targets, such as vinorelbine for TTN-mutated tumors. The comprehensive analysis further revealed the molecular mechanisms underlying the resistance to a chemotherapeutic drug docetaxel in a pan-cancer setting and the anti-cancer potential of a novel agent, CX-5461, in treating gliomas and hematopoietic malignancies.We proposed a deep learning model to predict drug response (DeepDR) based on mutation and expression profiles of a cancer cell or a tumor. The model contains three deep neural networks (DNNs), i) a mutation encoder pre-trained using a large pan-cancer dataset  to abstract core representations of high-dimension mutation data, ii) a pre-trained expression encoder, and iii) a drug response predictor network integrating the first two subnetworks. Given a pair of mutation and expression profiles, the model predicts ICHere we present, as far as we know, the first DNN model to translate pharmacogenomics features identified from in vitro drug screening to predict the response of tumors. The results covered both well-studied and novel mechanisms of drug resistance and drug targets. Our model and findings improve the prediction of drug response and the identification of novel therapeutic options. Due to tumor heterogeneity and intra-tumor sub-clones, an accurate prediction of drug response and an identification of novel anti-cancer drugs remain challenging tasks , 2. PharDeep learning (DL) is the state-of-the-art machine learning technology for learning knowledge from complex data and making accurate predictions. It features the ability to learn the representation of data without the need for prior knowledge and an assumption on data distributions. The DL technology has been successfully applied to bioinformatics studies of regulatory genomics, such as predicting binding motifs , investiAddressing the unmet demands, the present study is aimed to predict the response of tumors to anti-cancer drugs based on genomic profiles. We designed DeepDR, a deep neural network (DNN) model to learn the genetic background from high-dimensional mutation and expression profiles using the huge collection of tumors of The Cancer Genome Atlas (TCGA). The model was further trained by the pharmacogenomics data developed in human cancer cell lines by the GDSC Project and their corresponding genomic and transcriptomic alterations, and finally applied to TCGA data again to predict drug response of tumors. Collectively, DeepDR is a novel DL model that translates cell line-derived pharmacogenomics knowledge via tumor genomic and transcriptomic abstraction to predict tumors’ response to compound treatment.2 Data Portal [C, T, G, respectively, we metricized the expression data by g  in cell line c , and t . Genes with low information burden (mean < 1 or st. dev. < 0.5) among TCGA samples were removed. Mutation Annotation Format (MAF) files of mutation data were downloaded directly from CCLE (1463 cells) [g in c and t, respectively. Genes with no mutations in CCLE and TCGA samples were eliminated.We downloaded gene-level expression data of 935 cell lines of the Cancer Cell Line Encyclopedia (CCLE) and 11,078 TCGA pan-cancer tumors from the CTDa Portal  and UCSCa Portal , respect3 cells) , 20 and 50) from the GDSC Project [50 were measured in μM and represented in log scale  and missing data were imputed by a weighted mean of IC50 of 5 nearest drugs using R packages VIM and laeken [50 data and 9059 tumors with expression and mutation profiles.We also downloaded drug response data of 990 CCLE cell lines to 265 anti-cancer drugs measured by the half maximal inhibitory concentration  connected layers for all networks. At a neuron xi is the output of neuron i at the previous layer of j, wij and bj denote the synaptic weight and bias, respectively, and F represents an activation function. The notation of all neurons at a layer can thus be written as, where enc) and expression encoder (Eenc) in the complete model were initialized by the synaptic parameters learned from the pre-training on TCGA data. Neuron activation function was set as rectified linear unit (ReLU) except for the output layer of P as linear in order to better fit the distribution of log-scale IC50.During training, synaptic weights and biases are adjusted to minimize a loss function. We hereafter refer to the two parameters as synaptic parameters because they represent the model and can be used to transfer a learned model to another. In this study, DNNs were optimized using the Adam optimizer with a loss function of mean squared error (MSE). We used the He’s uniform distribution  to initi50 values based on genomic profiles of a cell or a tumor. Given the pair of mutation and expression vectors of sample c, {MCCLE, ECCLE}, the model predicts a D-length vector of IC50, enc), ii) an expression encoder (Eenc), and iii) a prediction feedforward network (P). The first and second components are the encoding parts of two autoencoders pre-trained using TCGA data to transform high-order features of mutation and expression data into a lower dimensional representation. The encoded representations of mutation and expression profiles were linked into P and the entire model was trained on CCLE data to make prediction of IC50 values. Details of DeepDR are described below.DeepDR was developed to predict ICmentclass2pt{minimAutoencoder is an unsupervised DL architecture that includes a symmetric pair of encoder and decoder. By minimizing the loss between input and reconstructed  data, it reduces the dimension of complex data and captures crucial features at the bottleneck layer (the layer between encoder and decoder) , ETCGA} was fed into the model and 50 indicates an adverse response of a patient to the corresponding drug.We applied the final model to predict drug response of TCGA tumors. For a tumor enc and Eenc, we randomly initialized both encoders using the He’s uniform distribution and calculated MSE of the entire model. Second, dimension reduction of the Menc and Eenc networks was replaced by principal component analysis (PCA). Last two models were built without Menc or Eenc to study whether they jointly improved the performance. In each iteration, CCLE samples were randomly assigned to training (80%), validation (10%), and testing (10%) and each model was trained and tested. Performance in terms of the number of consumed epochs and MSE in IC50 were summarized and compared across the 100 iterations. We also analyzed two classical prediction methods, multivariate linear regression and regularized support vector machine (SVM). For each method, top 64 principal components of mutations and gene expression were merged to predict IC50 values of all (using linear regression) or individual drugs (SVM).Performance of DeepDR was compared to four different DNN designs. First, to assess the effect of TCGA pre-training on M50 values) using genome-wide mutation and expression profiles. We included mutation and expression profiles of 622 CCLE cell lines of 25 tissue types and 9059 TCGA tumors of 33 cancer types. After data preprocessing, 18,281 and 15,363 genes with mutation and expression data, respectively, available in both CCLE and TCGA samples were analyzed. Log-scale IC50 values of all cell lines in response to 265 anti-cancer drugs were collected from the GDSC Project [50 was from − 9.8 to 12.8 with a standard deviation of 2.6  and Eenc , to construct our final model (detailed in Methods). Output neurons of the two encoders were linked to P , of which the last layer outputs predicted IC50. Architecture of the complete neural networks is shown in Fig. The study is aimed to predict drug response  were used for testing. The model achieved an overall MSE in IC50 of 1.53, corresponding to 1.48 and 1.98 in training/validation and testing data, respectively. Generally, the distribution of predicted IC50 was similar to original data  and rank of drugs  of a sample , PCA , Menc only (Eenc removed from DeepDR), and Eenc only (Menc removed from DeepDR). The two classical methods seemed to suffer from high MSE in testing samples . Our data echoed the biological premise that gene expressions are more directly linked to biological functions and thus richer in information burden than mutations.To test the stability of DeepDR, we ran 100 training processes each of which training, validation, and testing cells were reselected. Overall, the model converged in 14.0 epochs . Also, two EGFR inhibitors, afatinib and gefitinib, achieved better performance in non-small cell lung cancers (NSCLC) with mutated EGFR (P = 2.0 × 10− 7 and 6.6 × 10− 3). While the promising results on these well-characterized drugs showed the applicability of our model to tumors, we noted that the magnitude of differences in predicted IC50 levels was modest, underlining the fundamental differences between cell lines and tumors. In order to prioritize mutations underlying drug response, we systematically analyzed all cancer–mutation–drug combinations and tested the significance of differences in IC50 between samples with and without a mutation for each cancer. Here only genes with a mutation rate higher than 10% and harbored by at least 10 patients in a cancer were analyzed. With a stringent criterion of Bonferroni-adjusted t-test P < 1.0 × 10− 5, we identified a total of 4453 significant cancer–mutation–drug combinations involving 256 drugs and 169 cancer–mutation combinations , lung squamous cell carcinoma , and stomach adenocarcinoma  (Table TP53 was one of the most frequently mutated and well-studied genes in many cancers. The mutation has been shown to be associated with cancer stem cells and resistance functions and thus regulates drug resistance [50 (ΔIC50) = 0.95; P = 2.2 × 10− 109; Fig. 50 = 0.68; P = 7.4 × 10− 71; Fig. IDH1 , CSMD3 (223), SYNE1 (218), TTN (206), and RYR2 (199) were the top drug response-modulating genes (Table TP53 (9 sensitive and 242 resistant drugs) and TTN mutations (44 and 162) were associated with the largest numbers of resistant and sensitive drugs, respectively . DAVID analysis of the top 300 differentially expressed genes highlighted differential mechanisms between solid and non-solid tumors, such as extracellular matrix and cell motion . FutureThis study addresses the need for a translation of pharmacogenomics features identified from pre-clinical cell line models to predict drug response of tumors. We developed a DNN model capable of extracting representative features of mutations and gene expression, and bridging knowledge learned from cancer cell lines and applications to tumors. We showed the reliability of the model and its superior performance than four different methods. Applying our model to the TCGA collection of tumors, we identified both well-studied and novel resistance mechanisms and drug targets. Overall, the proposed model is widely applicable to incorporate other omics data and to study a wider range of drugs, paving the way to the realization of precision oncology.


ELMER (Enhancer Linking by Methylation/Expression Relationships) provides a systematic approach that reconstructs altered gene regulatory networks (GRNs) by combining enhancer methylation and gene expression data derived from the same sample set.DNA methylation has been used to identify functional changes at transcriptional enhancers and other cis-regulatory modules (CRMs) in tumors and other disease tissues. Our R/Bioconductor package ELMER that provides numerous new features including an optional web-based interface and a new Supervised Analysis mode to use pre-defined sample groupings. We show that Supervised mode significantly increases statistical power and identifies additional GRNs and associated Master Regulators, such as SOX11 and KLF5 in Basal-like breast cancer.We present a completely revised version 2 of ELMER v.2 is available as an R/Bioconductor package at http://bioconductor.org/packages/ELMER/.Bioinformatics online. ELMER (Enhancer Linking by Methylation/Expression Relationships) to infer regulatory element landscapes and GRNs from cancer methylomes (Motivated by the identification of transcription factor binding sites (TFBSs), enhancers and other cis-regulatory modules (CRMs) from DNA methylation data in tumor samples , we prevthylomes , and remthylomes . Other tthylomes  and RegNSupervised mode where the user can explicitly define sample groups for comparison. In this brief Note, we highlight several of these new features by analyzing TCGA Breast Cancer data to identify molecular subtype-specific networks. A complete description of new methods and features, along with computational benchmarking, is presented in the http://bit.ly/ELMER_reports.We present here a substantially re-written ELMER v. 2 A that imn% of tumors with the most extreme methylation values . The rationale was that any particular GRN might only be altered in a subset of tumors with a specific molecular phenotype, which would not always be known a priori. While 20% was an arbitrary definition, we found this to be a useful exploratory strategy given the heterogeneity of cancer molecular phenotypes.ELMER first identifies Differentially Methylated CpGs (DMCs) occurring at distal (non-promoter) probes (Step 1), then searches for downstream gene targets for each DMC (Step 2), and finally identifies Master Regulator TFs based on enriched binding motifs and TF expression (Step 3), as shown in Unsupervised strategy. However, we have found many practical use cases where the group structure is known in advance, and a Supervised search strategy is preferable. This is especially true for “case–control” experimental designs such as treated versus untreated samples. The major difference is that in Supervised mode, all samples must be contained in one of the two comparison groups, whereas Unsupervised mode still uses only the n% most extreme. Furthermore, this subset of samples with the most extreme methylation values changes from one genomic locus to the next.In ELMER v. 2, we continue to support this original Supervised versus Unsupervised modes, we used ELMER v. 2.4.3 to analyze TCGA BRCA (Breast Invasive Carcinoma) data  data . When coal power B, and idal power . SpecifiSupervised versus Unsupervised analyses, we gained insight into the question by comparing ELMER-predicted enhancer-gene links to pairs identified using PolII looping (ChIA-PET) in Luminal type MCF7 cells , the pairs from the Unsupervised analysis were more strongly enriched based on both Precision and Recall values  analysis showed the likely family member to be FOXA1  for subtype definitions, but any multi-omic unsupervised clustering method can used, depending on what data types are available.ELMER v. 2 has been substantially re-written based on Bioconductor standards and user needs. The new Supervised mode, our improved TF analysis identified additional known and novel Master Regulators candidates in TCGA BRCA analyses. ELMER v. 2 has only been tested on data from Illumina methylation arrays, which cover only 5-15% of all enhancer regions based on whole-genome bisulfite sequencing (WGBS). While ELMER does not currently support WGBS due to lack of sufficient test data, the number of WGBS datasets is quickly growing, and we expect the same basic ELMER approach will scale well in the future to take advantage of this more comprehensive data type.In addition to the new The project was funded by the Cedars-Sinai’s Samuel Oschin Comprehensive Cancer Institute, by the São Paulo Research Foundation (FAPESP) (2016/01389-7 to T.C.S. & H.N. and 2015/07925-5 to H.N.), by the NIH/NCI Informatics Technology for Cancer Research  and Genomic Data Analysis Network (1U24CA210969 to B.P.B & T.C.S) programs, as well as NIH/NCI grant R01CA190182 to D.J.H.Conflict of Interest: none declared.bty902_Supplementary_MethodsClick here for additional data file.


Staphylococcus aureus. By combining supervised and unsupervised machine learning methods, we firstly show that the mass spectroscopy data contains strong signal for the clustering of MSSA and MRSA. Then we concentrate on applying supervised learning to extract and verify the important features. A new workflow is proposed that allows for extracting a fixed set of reference peaks so that any new data can be aligned to it and hence consistent feature matrices can be obtained. Also note that by doing so we are able to examine the robustness of the important features that have been found. We also show that appropriate size of the benchmark data, appropriate alignment of the testing data and use of an optimal set of features via feature selection results in prediction accuracy over 90%. In summary, as proof-of-principle, our integrated experimental and bioinformatics study suggests a novel intact cell MALDI-TOF to be of great promise for fast and reliable detection of MRSA strains.Fast and reliable detection coupled with accurate data-processing and analysis of antibiotic-resistant bacteria is essential in clinical settings. In this study, we use MALDI-TOF on intact cells combined with a refined analysis framework to demonstrate discrimination between methicillin-susceptible (MSSA) and methicillin-resistant (MRSA)  Staphylococcus aureus, which is the most common and important pathogen in both community-associated and clinical-associated infectionKock, Schaumburg [S. aureus isolates is time-consuming (up to 48 hours) and laborious even if PCR-based methods are used to increase the speed of identification and susceptibility[Staphylococcus aureus infection, based on direct peptide fingerprints of intact heat-inactivated S. aureus clinical isolates using an improved data analysis workflow and machine learning.As more antibiotic-resistant pathogens emerge, fast, accurate, robust and reliable detection of bacterial pathogens as well as their resistance to antibiotics is urgently needed to contain the spread of these so-called “superbugs”. This is the case for haumburg . Currentptibility, 3. Howeng up to 8 hours aptibility, 6. In tS. aureus.We introduced an improved workflow to analyse our mass spectroscopy (MS) datasets. In particular, our new workflow extracts a fixed set of reference peaks from the training data. Then given any new test data, they can be pre-processed independently and mapped onto the reference peaks. Furthermore, we examined two different ways to generate a feature matrix (intensity-based and peak area-based methods) and two different statistical learning models (random forest and binaS. aureus (MSSA) and 10 methicillin-resistant S. aureus (MRSA) clinical isolates recovered from individual patients with bacteraemia by the diagnostic microbiology service at Charing Cross Hospital. The strains were grown in Tryptic Soy Broth (TSB) medium at 37°C for 12 hours to reach an OD600 of 0.4–0.6.For the present proof-of-principle study, we examined 10 methicillin-susceptible 4 bacteria per μl.Bacteria were harvested, heat-inactivated 30 mins in at 80°C, washed three times and suspended in double distilled water at a final concentration of 10As proof of principle, we generated a total of 7 datasets corresponding to four biological replicates and three of these contained two technical replicates see . Each daPrior to mass spectrometry analysis, the 2,5-dihydroxybenzoic acid (DHB) matrix was added to a final concentration of 10 mg/mL in chloroform/methanol at a ratio of 90:10 v/v. 0.4 μL of cell solution and 0.8 μL of the matrix solution were deposited on the MALDI Target, mixed with a micropipette and left to dry. MALDI-TOF analysis was performed on a 4800 Proteomics Analyzer  using the reflectron mode. Samples were analyzed operating at 20 kV in negative and positive ion mode, and three independent experiments were performed. Mass spectrometry data were analyzed using both Data Explorer version 4.9 from Applied Biosystems and R [m/z, I) values, where m/z stands for the mass to charge ratio and I stands for the intensity value. m/z values range from around 399 to 4012  and N samples and several steps of pre-processing is required as explained in “Data pre-processing”.The MS data for each sample consists of around 130,000 pairs of , the peaks found at m/z 1650–1850 can be assigned to the peptide fragment identified as the phenol-soluble modulin α1Δ1–5 (PSMα1Δ1–5) with the sequence GIIKVIKSLIEQFTGK with a theoretical m/z of 1774.083 [m/z 2100–2300 are assigned to formylated peptides such as PSMα1 at m/z 2287.3 . In the case of MRSA  method. Then the data were aligned with half window size set to 20, tolerance set to 0.02, “minFrequency” set to 0.75 and signal-to-noise-ratio (SNR) set to 10.After the above procedure, we were able to extract reference peaks. Firstly, we detected peaks based on the aligned training data by setting SNR to 2 and half window size to 20 (using median absolute deviation (MAD) method). Then the reference peaks were created by calling the function “referencePeaks” from MALDIquant with method “strict”, “minFrequency” set to 0.75 and tolerance 0.02. To be more specific, the mass of peaks detected from each sample were gathered as a vector and ordered. Then the differences between each neighbor were calculated. After that, starting from the largest gap, the vector of masses was repeatedly divided into bins until the bin does not contain two or more peaks of the same sample and all peaks in a bin are near to a specific position which was determined by the parameter tolerance .For the preprocessing of testing data, it was also smoothed, baseline corrected, intensities and calibrated using the same parameter setting as illustrated above. In alignment, the reference peaks extracted from the training data were specified.After preprocessing the data, we were able to generate either peak area or intensity based feature matrix.M1 ranked features from the M0 full features and building a statistical model for testing (M1≤M0).Given a feature matrix and corresponding true labels, there are also different ways of performing feature selection. Support Vector Machine with Recursive Feature Elimination (SVM-RFE) was proposed by Guyon et al . Tim et Each one of three evaluation procedures was performed using the following 4 different variants of the second workflow:Generating an intensity-based feature matrix and transforming it to a binary matrix for supervised learning using BinDA (“I+BI” for short).Generating an intensity-based feature matrix and building a random forest model for testing (“I+RF” for short).Generating a peak area-based feature matrix and transforming it to a binary matrix for supervised learning using BinDA (“P+BI” for short).Generating a peak area-based feature matrix and building a random forest model for testing (“P+RF” for short).Apart from the above four variants, all the parameter settings used in MALDIquant for intensity smoothing, baseline correction, intensity calibration and alignment are kept the same.We also tried to incorporate the first workflow into supervised learning framework using the same parameter settings as illustrated above. Procedures with BinDA or random forest are termed as “MALDI_BI” and “MALDI_RF” respectively. We combine all training samples and one testing sample (leave one out) and obtain the feature matrix using MALDIquant. Then a subset of feature matrix was used for training and selecting the top ranked features for testing the one testing sample. This procedure was repeated for each one of samples used for testing. The “leave one out” procedure could potentially reduce the impact of testing sample on the alignment step. We should note that the limitation of using the first workflow in supervised learning is that the features could change due to different input data. Because different peaks can be detected and hence reference peaks are not the same. The procedure with.For supervised learning, there are two ways of creating training and testing data: by splitting datasets or samples, which correspond to training with or without biological replicates of testing data in training procedure .K∈{5,10,15,30,50,100,300,500,1000,Full features}) ranked features were kept for training and testing.The evaluation of the workflows in terms of supervised learning focuses on three parts. The first part is deciding upon the top ranked features used in supervised learning. There are 8 different choices for selecting 4 out of 7 datasets such that the 4 datasets are all biological replicates. In order to create training and testing data, if we split datasets, for each of the 8 subsets, we conducted supervised learning based on the 3 datasets , each one of 7 datasets were set as testing data. We gradually increase the size of subset of the remaining datasets for training and make sure that technical replicate of the testing data does not appear in the training data. For the latter case (splitting samples), for each one of 8 subsets as mentioned before, we split samples into training and testing data by the following procedure: In each biological replicate, 3 MRSA and MSSA samples were randomly selected for constructing testing dataset (hence testing data contains (3+3)×4 = 24 samples). As above, during splitting we made sure that technical replicates of testing samples were also excluded from training procedure. This way of constructing testing data results in 56 samples left. Among the 56 samples, we randomly selected subset of samples with size 8, 16, 24, 32, 48 and 56 for training. In addition, we made sure that in both training and testing data, MRSA/MSSA samples were evenly spread. This splitting was repeated under each one of 8 subsets 5 times, which results in 40 accuracy rates for each one of 4 different variants of the second workflows as discussed below.The third part concerns the evaluation of the stability of the top ranked features. We expect a consistent pattern of extracted features based on different training datasets. The top 100 ranked features were extracted based on training datasets which were obtained by splitting datasets in the first part of evaluation.m/z 1650–1850 that could act as a biomarker for identifying MRSA vs MSSA strains. To systematically obtain biomarkers, in the following, we will use careful pre-processing of this data and apply statistical learning methods.Visual inspection of the raw spectra in m/z values above 1200, which further suggests that informative peaks mainly lie in the range above 1200. These findings are also consistent with what we observed in m/z values less than 1200 tend to be noisier.The first workflow  is used The first workflow used thus far requires all the datasets to determine the feature sets. In practice, we would like to determine the biomarkers based on some benchmarking datasets and then test it on new datasets. In the following, we will use our proposed second workflow to test reliability of this data in classification of MSSA and MRSA samples. Ideally, the top ranked features should not vary too much when training datasets change.Firstly, we evaluated the effect of the top ranked features used on supervised learning. As In order to check that whether the different performance results from variants of features, we compared features selected by either BinDA or random forest using both Jaccard Index  and MeanThe averaged accuracy results as a function of the number of datasets or samples used in training procedure are shown in the We next test the importance of alignment of new data according to the training data in the reference data-based workflow . In contm/z. The result also indicates some signal in around 500m/z, but the density plot shows that the signal there is noisier. This is again consistent with what we observed in The top 100 ranked features were extracted based on the m/z ranging from 500–10,000 using the matrix 5-chloro-2-mercaptobenzothiazole dissolved in water:methanol:acetonitrile 1:1:1 and containing 0.1% formic acid and 0.01 M 18-Crown-6 ether[S. aureus strains and MRSA was correctly distinguished from MSSA using cluster analysis with a dendrogram[S. aureus had different peak intensities at m/z 3048, m/z 3086 and m/z 3124[m/z 2302 and m/z 3871[The use of MALDI-TOF MS to differentiate MRSA from MSSA has been already investigated. As for example, Edwards-Jones et al. used seven MSSA reference strains and seven MRSA clinical isolates for a n-6 ether. Du et a m/z 3124. Drake eIn terms of preprocessing, in this study, we propose a modified workflow that allows defining a set of fixed reference peaks based on some predefined set of benchmarking dataset. Then, we describe how to map new testing datasets to the reference dataset. This approach is required for practical applications as one would like to classify newly obtained clinical samples based on a validated prior classification model. In the literature, although some methods such as “Peak Probability Contrasts” (PPC)  and BinDFor the first workflow, if we want to apply supervised learning, we can pre-process all samples and then divide the final feature matrix to training and testing data separately. However, if we incorporate new data and apply the first workflow again, the features could change. The second workflow allows us to generate a consistent feature matrix based on a fixed reference and hence evaluate the robustness of the extracted features.We evaluated the second workflow in three different ways. We have tested training data with or without biological replicates of testing data. In addition, we carefully avoid the potential impact between technical replicates in supervised learning by excluding technical replicates of testing data from training procedure. The promising results indicate that we are able to extract robust information from direct peptide fingerprints based MS data. We also learned it is optimal to use between 20–100 top ranked features and to use peak area based methods in combination with random forest for classification.There still remains room for improving the workflow. As illustrated before, for reference based workflow, it is important to align new spectra according to the reference well. Only then can we further summarize MS data into the feature matrix. Hence a robust alignment method is necessary. Given that the pre-processing of MS data contains several steps and for each step there exists different methods, the workflow has potential to be further optimized.S. aureus.As proof-of-principle, our integrative analysis offers a unique and unbiased data analysis and classification of new MALDI samples based on existing benchmarked datasets and could discriminate MSSA from MRSA with over 90% accuracy. Last but not least, our improved analysis workflow is generally applicable to identification by MALDI MS for any biological sample including bacteria, viruses and fungi. In summary, our approach promises a rapid and reproducible method to determine the resistance to methicillin in S1 TableTab 1, 2, 3 and 4 represent the Top 100 features for I_BI, I_RF, P_BI, P_RF conditions respectively.(XLSX)Click here for additional data file.


E. coli. The transcriptome data provides clear expression signals and thus represents an extremely favorable test setting. The methods estimate activity changes of all TFs, which we expect to be highest in the KD TF. However, only in 15 out of 54 cases, the KD TFs ranked in the top 5%. We show that this poor overall performance cannot be attributed to a low effectiveness of the knockdown or the specific regulatory network provided as background knowledge. Further, the ranks of regulators related to the KD TF by the network or pathway are not significantly different from a random selection. In general, the result overlaps of different methods are small, indicating that they draw very different conclusions when presented with the same, presumably simple, inference problem. These results show that the investigated methods cannot yield robust TF activity estimates in knockdown schemes.Numerous methods have been developed trying to infer actual regulatory events in a sample. A prominent class of methods model genome-wide gene expression as linear equations derived from a transcription factor (TF) – gene network and optimizes parameters to fit the measured expression intensities. We apply four such methods on experiments with a TF-knockdown (KD) in human and  Gene regulation also plays an important role in the development and progression of various diseases3. Thus, the elucidation of human regulatory relationships is an important research field and many methods attempting to infer the actual regulatory events in a given sample have been proposed.The regulation of gene expression is a fundamental biological mechanism in all living species. It determines the cells’ unique properties and enables them to adapt to the organism’s development, cellular function, the environment and external stimuli1. Several algorithms, such as biRte4 (Bayesian inference of context-specific regulator activities), ISMARA5 , RABIT6  and RACER7 , have been presented to model genome-wide gene expression and regulation via the activity and relationships of transcription factors. These models allow for the application of mathematical optimization to find parameters that minimize the divergence of predicted and measured expression intensities8. They all consider the topology of the regulatory TF – gene network to be given and try to infer the actual TF activity developed in a certain disease or under a specific experimental condition. Their primary output is a ranked list of TFs, sorted by their activity in a given group of samples. A detailed description of the specific methods evaluated in this paper is given in the materials and methods section at the end of the paper. Several studies reported that such methods can be used to identify biomarkers for specific phenotypes in human cell lines and in vivo samples, for example in innate immunity5, ageing related changes5 or acute myeloid leukemia7.In eukaryotes, gene expression is mainly regulated by a complex network of transcription factors (TFs) which activate or repress gene transcription8. We used a publicly available human TF – gene network9 together with experimental data from TCGA10 (The Cancer Genome Atlas) for three cancer types to identify key biomarkers for these specific diseases. The results showed that all methods seemed to detect strong signals and find biologically relevant information, but sensitivity was low and the mutual result overlaps from different methods were small. We suspected that the complexity of gene regulation in cancer was one reason for the questionable performance and low consistency of different methods’ results. Therefore, we here focus on much less complex data and use knockdown experiments to evaluate different methods on estimating TF activity changes. We suppose that the highest change in activity will occur in the knocked down TF when comparing case and control samples. Many datasets of such high-throughput experiments for certain experimental conditions and different species have been published and are available in public repositories like GEO11 (Gene Expression Omnibus). In this straightforward and for the methods favorable setting, we expected that the methods would consistently be able to identify the knocked down TF.We previously compared different methods for estimating regulatory activity qualitatively and quantitatively in detail4, ISMARA5, RABIT6 and RACER7, to infer transcription factor activity from gene expression data in knockdown (KD) experiments. We downloaded transcriptome data of four publically available KD experiments from the GEO11 repository including different TF knockdowns in human and E. coli cell lines. To better distinguish KD TFs from cell lines or other abbreviations, we will set TFs in italics. The three chosen experiments in human cell lines  contain data from 8 knocked down genes  and the double knockdown C/EBPβ & STAT3. The selected experiment in E. coli (GEO identifier GSE112116) comprises 5 knocked down genes  and the double knockdown ArcA & Fnr. Some of these experiments were conducted in different cell lines or conditions . Overall, we study 25 data sets , 13 from human and 12 from E. coli. For an overview of the composition of the experiments, see Figs E. coli) as input to the methods biRte, RABIT and RACER, whereas ISMARA employs an own, inaccessible underlying network. The network including information on human regulatory relationships is based on a text-mining approach9 complemented with TF – gene interactions from the public TRANSFAC database17. The network was built by text mining the entire Medline and an additional manual curation step of the top-ranking sentences. It thereby combines the content of regulatory databases with more than 300 validated regulatory relationships. The network for E. coli was retrieved from RegulonDB18. The considered datasets are described in detail in the materials and methods section at the end of the paper.Here, we compare four different methods, namely biRte19 (Algorithm for the Reconstruction of Accurate Cellular Networks) to assess the influence of the network on the results. To test whether the mere differential expression is a better predictor for TF activity changes, we calculated ranks of differential expression for the knocked down TFs and compared those to the activity ranks.We predominantly assessed the rank of the TF that was knocked down and the total number of ranked TFs. We additionally checked for aliases and we determined the ranks of neighbor TFs in the network, of co-members in a pathway and of interacting TFs. To examine whether the methods were able to detect a common signal in the data, we compared the overlap of the top 100 ranked regulators of all methods within one data set. We additionally performed the activity estimation on smaller networks and on networks inferred de-novo by ARACNE8, we are not aware of any other independent study on the performance of optimization-based algorithms for the estimation of whole genome transcription factor activity. Our previous results compared the performance on multi-omics data sets where no clear expectations could be formulated and also found very low coherence in the results of different methods. This lack of quality estimation of results led us to the new experimental design of using knock-down. However, also on this presumable much simpler problem the result overlaps are very low and the knocked-down transcription factor was only very rarely identified. We conclude that the investigated methods do not yield robust estimates of TF activity in a knockdown scheme.Apart from our own previous workE. coli, the identification of the KD TF by activity estimation yielded slightly better results compared to human cell lines. When looking not only at the KD TF but at the regulators related to the KD TF in the network or a pathway, we identified only a single case where the mean of the ranks of all related TFs was significantly smaller than expected by chance. The overlap of the top 100 ranked regulators of all methods within one data set was small and statistically insignificant. The reduction of the network size or the use of ARACNE’s inferred networks did not improve the results.Our results show that, although almost all KD TFs showed differential expression, their activity ranks were only in 15 out of 54 cases within the top 5% of all ranked TFs , both in the single KD and double KD together with STAT3, see Fig. RUNX1 was significantly upregulated in SNB19 case samples compared to the control samples. Nonetheless, we included RUNX1 in our analyses since we were only interested in finding absolute TF activity changes. In E. coli  TFs themselves were differentially expressed, which was the case for all human KD TFs except see Fig. , all KD E. coli, the number of KD TFs in the top 5% TFs was 3 in the aerobic and 5 in the anaerobic condition out of 6 data sets in either condition. As expected, the KD TFs are in about two third of the considered data sets amongst the TFs with the highest changes in differential expression.Additionally, we checked whether the differential expression per se would be a good predictor for the determination of knocked down TFs in a data set. Therefore, we computed differential expression separately in each data set, contrasting the expression of the corresponding case and control samples and evaluated the ranks of differential expression for the KD TFs. The results are shown in Supplementary Fig. E. coli). Of the 54 cases, where ranks were provided, 27 resulted from biRte, one from ISMARA, 13 from RABIT and 13 from RACER. Due to stringent filtering thresholds within the methods, no activity score was assigned to the KD TF in 37 cases, hence the ranking could not be computed in those cases.We next applied biRte, ISMARA, RABIT and RACER to determine the respective KD TFs’ ranks. The KD TFs were only in 15 out of 54 cases within the top 5% of all ranked TFs . In E. coli, the results from biRte were better with 8 out of 14 TFs in the top 5% and another two TFs in the top 10%. In all other data sets, the ranks provided by biRte for the TFs in question were quite low. ISMARA could only be applied to GSE45838 and GSE17172 since the chips from the other experiments were not supported by the online interface. In one data set (MYB) the KD TF was highly ranked (10th out of 602), but ISMARA did not provide any ranks for the two other KD TFs (BCL6 and FOXM1). Since the underlying network from ISMARA is not accessible we cannot discern whether the TF is not present in the network or was not considered important by the ranking procedure. RABIT removes TFs with insignificant cross-sample correlation from the results and therefore only provides the ranks of, on average, 56 TFs in our analyses. It did not provide any activity score for the KD TF in over half of the data sets . In human, not a single KD TF was ranked in the top 20%. However, in E. coli RABIT was able to identify AppY (rank 1) and ArcA as knocked down TFs in the anaerobic condition (rank 2 in the single KD and rank 1 in the combined KD ArcA & Fnr). In contrast, RACER ranked only one KD TF for the human data sets at all (BCL6) and did not rank any KD TF highly in E. coli. In some human data sets, RACER even reported the total number of important regulators to be zero.We observed that biRte provided ranks for nearly all KD TFs. In 3 out of the 13 human data sets, where ranks were specified, biRte ranked the knocked down TF in the top 5% (20 for human respectively EcoCyc21 for E. coli), direct neighbors in the TF – gene network, directly interacting TFs  and presumed aliases from the GeneCards23 (human) and EcoCyc21 database (E. coli)  in Fig. JUN), which is directly connected to MYB in the human regulatory network, was ranked among the top 20% TFs by two of the four methods . Previously, it was shown that JUN contributed to the transcriptional activation of MYB25. For each method and data set individually, we evaluated whether the mean of the resulting ranks of all related TFs was significantly smaller than the average rank expected at random . Only one out of 54 of the mean ranks of the estimated activity changes was significantly below the average rank: in E. coli, biRte ranked OxyR and a related TF highly in the anaerobic condition (p = 0.002). However, since this result was obtained with quite a small sample set (only two ranked TFs), we consider it not representative.We show the resulting ranks and according p-values of the KD TF and related TFs for one exemplary result . These results make sense, as the expression of FOXM1 and MYBL2 as its downstream factor were significantly associated with clinical stages and overall survival of glioma patients26 and is very high in Burkitt lymphoma27. Further, MYBL2 deregulations occurred in a broad spectrum of cancer entities29. FOXM1 is a direct target of repression by FOXO proteins. An inactivation of FOXO or overexpression of FOXM1 was associated with tumorigenesis and cancer progression30. Nevertheless, the overlap is extremely small and not significantly larger than expected at random .We found very little overlap in human cell line data. The highest overlap among three methods  occurred in E. coli, the number of common TFs from biRte, RABIT and RACER was higher, but also not significant , with a maximum overlap of 18 TFs (ArcA & Fnr knockdown in the anaerobic condition). The overlap contained, for example, ArcA, which is activated in anaerobic conditions31, NtrC, which was shown to be upregulated during the transition from anaerobic to aerobic conditions32, and AdiY, which was maximally induced under anaerobic conditions33. Although the methods do not find the knocked down TF itself, at least in our E. coli datasets they commonly find TFs biologically relevant for the condition under consideration. The results are exemplarily shown for FOXM1 and the combined ArcA & Fnr KD (anaerobic condition) in Fig. In 7, is that the regulatory networks used in the original work were much smaller compared to our networks. To assess whether the usage of a smaller network improves the results, we restricted the underlying TF – gene network to the neighborhood of each knocked down TF with a distance of two. Note that this design gives a very favorable prior to the analysis. An exemplary restricted network for FOSL2 is presented in Fig. RUNX1 and STAT3 were ranked within the top 5% and FOXM1 in the top 10% using biRte. This result was already obtained using the full network  and AppY KD (E. coli). We used these networks as input to the investigated TF activity estimation methods and ranked the resulting TF activity scores. Although the networks inferred by ARACNE have a higher density compared to our original networks, the resulting TF activity rankings are comparable  did not improve or change substantially the results.For the evaluation of the experiments based on human data, we used a gene regulatory network constructed by a text mining approachIn general, the selection of experiments might affect the outcome of the methods. We used experiments from the GEO platform, an established and extensive repository for genomic data sets, to ensure an easy and public access to the data and to allow other researchers to replicate our results. We chose data from different species, different cells of origin and cell lines, from various contributors, data measured on different arrays, and of 13 different KD TFs to ensure that our results are generalizable and depend less on the specific datasets. However, the chosen experiments had to fulfill certain criteria: obviously, they had to contain a KD scenario and we chose to include only experiments with at least three samples per condition. Further, as we wanted to include ISMARA as a method for estimating TF activity, we had to choose experiments whose Affymetrix chips were supported by its web service. All these constraints limited the number of possible data sets. Of course, the use of other experimental data, different underlying networks or additional methods might affect the results. However, since we draw our conclusions from a total of 25 evaluated data sets, amongst which we did not detect a pattern justifying an especially good or bad performance, we argue that our results show not only individual artefacts but are generalizable to the estimation of TF activity in KD studies.E. coli data. We believe that the main reason for this deficiency is the simplistic model of cellular processes used in the more complex methods like ISMARA. We can only speculate which aspects are primarily responsible for the limited performance. All considered methods only used gene expression data whereas other important regulatory processes such as epigenetic mechanisms like DNA methylation, chromatin remodeling, complex promoter structures, and posttranscriptional regulatory processes via microRNAs are disregarded. The inclusion of further data types would probably change the outcome of the methods and might improve results8. Also, all models assume linear relationships between TFs and lack a notion of kinetic or temporal effects36. Although time series expression data from TF knockdown or TF induction experiments exist38, the selected methods cannot make use of this type of data. Another possible reason for the failure of the methods might be their inability to model TF self-regulation and feedback loops despite their known importance for gene regulation40.Altogether, the investigated methods for estimating TF activity were not able to robustly detect knocked down TFs neither in human nor in E. coli cell lines from the GEO repository11. We chose three different experiments for TF silencing in human cell lines  and one experiment in E. coli (GSE112116). They contain data from 8 human knocked down genes  and 5 knocked down genes  from E. coli. PCA plots for all data sets are provided in Supplementary Fig. E. coli). When multiple probes mapped to one gene we computed a t-test comparing case and control group and kept the probe with smallest p-value.We downloaded publicly available transcriptome data for different TF knockdowns in human and GSE4583812 contains data from the knock-down of BCL6 expression in human diffuse Large B-Cell Lymphoma cell lines. This experiment was performed in OCI-Ly7 and Pfeiffer GCB-DLBCL cell lines as triplicates, providing three case and three control samples per cell line. Gene expression was profiled on H-GU133plus2 Affymetrix gene chips. We analyzed the samples in dependence of their cell line origin and treated them as two independent data sets since they were clearly separated in a PCA plot . cRNA was hybridized in Affymetrix Human Genome U95 Version 2 Arrays. We used the MAS541 normalized data as provided on GEO.GSE1911415 includes 74 samples from knockdown experiments in human glioma cell line SNB19 and glioblastoma multiforme-derived brain tumour initiating cells (BTICs). shRNA-mediated silencing targeted bHLH-B2, FOSL2, RUNX1, C/EBPβ and STAT3. For SNB19, 10 control samples were available together with 4 samples with bHLH-B2 knockdown, 4 with FOSL2 knockdown and 3 samples each for C/EBPβ, STAT3 and the combined C/EBPβ & STAT3 knockdown. Data was available for C/EBPβ, STAT3, combined C/EBPβ & STAT3 knockdown and a control condition for 11 samples in each group in BTICs. RNA was hybridized on Illumina HumanHT-12v3 expression BeadChip. Since the samples were clearly separated in a PCA plot by their cell type  in both aerobic and anaerobic conditions. Additionally, three (aerobic condition) and four (anaerobic condition) wild type samples were available. Gene expression was profiled on Affymetrix E. coli Antisense Genome Arrays. We analyzed the data from the two oxygen conditions independently.9 and one from E. coli18 as input for biRte, RABIT and RACER, which are available as supplementary files. ISMARA can only be used with its own underlying regulatory network model, which is not accessible explicitly.We provided a human TF – gene network9 based on a text-mining approach complemented with TF – gene interactions from the public TRANSFAC database17 . This network includes 2894 interactions between 429 TFs and 1218 genes. The network is provided as supplementary material.We used a publicly available TF – gene network18 version 9.0, Release 9.4 and kept those interactions for which at least one entry in the column “Evidence that supports the existence of the regulatory interaction” was mentioned. The network contains 4273 interactions between 206 TFs and 1798 genes and is provided as supplementary material.We downloaded TF – gene interactions from RegulonDB8 or the according original paper. The investigated methods are publically available as web service, downloadable program or R package.The considered methods for estimating TF activity model genome-wide gene regulation as sets of equations over the activity of transcription factors. All methods assume both the set of TFs and the topology of the regulatory network to be given. By combining this background knowledge with transcriptome data, they try to infer the activity of regulators in a certain experimental condition or disease using mathematical optimization to find parameters minimizing the divergence of predicted and measured gene expression intensities. The methods predominantly produce a ranked list of TFs, sorted by their activity in a given group of samples. Here, we briefly explain the functioning of each method and state our parameter settings if applicable. For a detailed description of each method refer our previously published review4 (Bayesian inference of context-specific regulator activities) uses a probabilistic framework to estimate regulatory activities from differential gene expression data and a TF – gene network. The set of active regulators can be seen as hidden state variables which are estimated with help of the Markov Chain Monte Carlo method. Thereby, the posterior probability for each regulator and condition to influence the expression of its target genes is estimated. Simultaneously, a variable selection procedure is applied to achieve sparsity of the model.BiRteBiRte is available as a bioconductor R package. We used R version 3.2.0 with biRte version 1.10.0 and applied the method “birteLimma” to estimate regulatory activities with the options “niter” and “nburnin” set to 10000. As biRte has a randomized component, the resulting TF activities are not exactly the same for different runs. We averaged the final activity scores over 100 iterations of birteLimma.5  infers the activity of regulatory motifs (short nucleotide sequences) and thereby indirectly deduces the effects of TFs. The input signal levels, which are computed from gene expression data, are modelled linearly in terms of binding site predictions and unknown motif activities. ISMARA employs a Bayesian procedure with a Gaussian likelihood model and a Gaussian prior distribution for inferred motif activity profiles to avoid overfitting.ISMARAIn contrast to all other investigated methods, TF – gene relationships are not provided by the user. ISMARA can only be run with its proprietary underlying regulatory network model. Further, only raw data provided as CEL, FASTQ, BED, BAM or SAM files can be uploaded. Therefore, the results from ISMARA are only partly comparable to the results from other methods here.https://ismara.unibas.ch/fcgi/mara). We uploaded raw CEL files and grouped the samples according to their origin or treatment to compare the average regulatory activity between different conditions. Since Illumina chips in general and the Affymetrix E. coli Antisense Genome Array in particular are not supported, we could only run ISMARA on the data sets with BCL6 (GSE45838) and FOXM1/MYB (GSE17172) knockdown.ISMARA is available via a web service  applies a linear regression model to estimate TF activities. First, RABIT tests in each sample whether the target genes of each TF are differentially expressed. A score indicating regulatory activity is defined by the t-value (regression coefficient divided by standard error). To find a subset of TFs among those screened before, a stepwise forward selection is applied optimizing the model error. Lastly, TFs with insignificant cross-sample correlation are removed from the results.RABIThttp://rabit.dfci.harvard.edu which we adopted with the FDR option set to 1. We used the difference of expression values between case and control group as input and ordered the activity of TFs by t-value as proposed in the RABIT paper.The authors of RABIT published a C++ implementation accessible under 7  consists of a two-stage linear regression. Optimization is applied twice to reduce model complexity, where the method first infers sample-specific TF activities and uses these, in a second step, to compute general TF – gene interactions. Sparsity of the solution is obtained through elastic-net regularized generalized linear models. A supplementary feature selection procedure comparing the full model to a restricted model leaving one TF out provides the most predominant regulators.RACERWe used the available R scripts from http://www.cs.utoronto.ca/~yueli/racer.html to run RACER and set miRNA expression data, copy number variation and methylation scores, which have to be provided, to zero. The obligatory miRNA – gene network was artificially created where all dummy miRNAs and genes were connected. We computed separate models for case and control group and extracted the resulting sample-specific regulatory activities. TFs were ranked by their activity difference between the two groups.For all ranking assignments, we appointed TFs that compared equal the same rank. Subsequently, a gap was left in the ranking numbers which size was equal to the number of items that compared equal minus 1.We calculated the differential expression between case and control group for all genes in all data sets via a two-sided t-test. We ranked the genes according to the p-value of the t-test . We did not apply any multiple test correction, since we were not interested in the precise p-value but only the order of p-values to assign ranks.directly connected TFs in the network23 database version 4.8.0 Build 5 (available under www.genecards.org) for human TFs respectively synonyms from the EcoCyc21 database (ecocyc.org) for E. coli.aliases provided in the GeneCards20 2.0  for human TFs respectively from the EcoCyc21 database (ecocyc.org) for E. coli.TFs directly connected in a pathway from SignaLink22 version 2.2.2 available under http://tools.sschmeier.com/tcof/home for human.TFs directly interacting with the KD TF given in TcoF-DBIn each data set, we compared the results of each method by ranking the absolute values of the computed TF activity scores . Activities equal to zero were not considered. Therefore, the total number of ranked TFs is different in each method and data set. We predominantly assessed the rank of the TF that was knocked down in particular. Additionally to the KD TF, we evaluated the ranks (if existing) ofWe call the union of these TFs related TFs. An overview is available in Supplementary Table We computed the number of overlapping TFs in the top 10 and top 100 lists comparing all applied methods within one data set. The resulting Venn diagrams are shown in Supplementary Fig. To evaluate the influence of the human gene regulatory network, we reduced the network to the close neighborhood of the knocked down TF with a distance of 2, thus giving smaller gene regulatory networks for each knocked down TF.19 (Algorithm for the Reconstruction of Accurate Cellular Networks) to reconstruct a gene regulatory network using the FOXM1 (human) and AppY (E. coli) KD transcriptome data. We used the implementation of the “minet” bioconductor package42 in R (version 3.38) and built the mutual information matrix with Spearman’s correlation. The threshold for removing an edge in the aracne function was set to 0.1. We used the resulting gene regulatory networks as input to biRte4, RABIT6 and RACER7 and ranked the estimated TF activity scores as described above.We applied ARACNESupplementary InformationDataset 1Dataset 2


Unfortunately, as we demonstrate empirically, such problematic situations of correlated and linearly dependent variables often exist in genomic datasets and lead to under-performance of classical methods of variable selection.Association studies to discover links between genetic markers and phenotypes are central to bioinformatics. Methods of regularized regression, such as variants of the Lasso, are popular for this task. Despite the good predictive performance of these methods in the average case, they suffer from stable and consistent variable selection in simulated data with highly correlated and linearly dependent variables. We then demonstrate the effectiveness of the Precision Lasso to select meaningful variables from transcriptomic profiles of breast cancer patients. Our results indicate that in settings with correlated and linearly dependent variables, the Precision Lasso outperforms popular methods of variable selection such as the Lasso, the Elastic Net and Minimax Concave Penalty (MCP) regression.To address these challenges, we propose the Precision Lasso. Precision Lasso is a Lasso variant that promotes sparse variable selection by regularization governed by the covariance and inverse covariance matrices of explanatory variables. We illustrate its capacity for https://github.com/HaohanWang/thePrecisionLasso.Software is available at Bioinformatics online. High-throughput technology for profiling gene expression levels and assaying genetic variations at a genome-wide scale produces massive data, creating an opportunity to study the genetic causes of complex diseases by statistical methods. Computationally screening for putatively causal genes is a key step in hypothesis generation, and many such techniques have been proposed. These can be categorized into several generations, starting with hypothesis testing . Unfortuhigh-dimensional regime that is common in genomic applications. To solve this problem, regularization via the Lasso  and dependent variable Y (of size n × 1), the Lasso solves the problemt > 0 controls the amount of regularization. With the To jointly consider associations between all biomarkers and a phenotype, linear regression-based methods , when in reality the evidence is shared between all elements of the pathway.Gene expression profiles can result in unstable variable selection. As expression levels of genes within a regulatory pathway are highly correlated  is the set of active variables, X(2) is the set of non-active variables, 1 is a vector of ones, To show this, we must first formalize the definition of ondition , which snon-irrepresentable . These datasets are moderately large: 617, 504 and 595 patients for the Glioblastoma, Lung and Breast datasets, respectively. The datasets also contain a high number of explanatory variables: 17 814 for gene expression, 27 577 for methylation and 4201 for miRNA. As the data are drawn from multiple assay and cancer types, they can be considered a representative sample of genomic datasets one might encounter in practice.A matrix which breaks this condition called sentable , and leaX, we selected K random genes to be ‘active’ in a simulated linear regression model. Then we checked the irrepresentability condition  = 10– 5 based on this active set. We replicated this 100 times for inconsistent for the Lasso. In all datasets, at least one active set broke this condition for each K and for K > 10, almost all active sets broke this condition.We tested the non-irrepresentable condition on these datasets as follows: Using the real data as the design matrix instability condition, we also calculated the frequency of highly correlated variables in the data. Six of the nine datasets checked had at least one pair of variables that had an empirical correlation coefficient of 0.99 or higher. These experiments indicate that both non-irrepresentability and high correlation—and hence inconsistency and instability—are both prevalent and a serious nuisance in working with real genomic datasets, motivating the need for more powerful methods that are robust to these problems.To check the instability and inconsistency separately, and then combine these two approaches in order to derive the final model.In this section, we introduce the Precision Lasso, a regularized regression method that mitigates the problems exposed in the previous section. We first introduce techniques to handle instability of the Lasso refers to its inability to distinguish the effects of correlated explanatory variables. Since correlated explanatory variables cannot be separated by observational statistics, one solution is to force the model to simultaneously identify correlated variables. A straightforward way to fulfil this goal is to assign similar weights to correlated variables.As introduced previously, the A convenient set of weights to use are the sum of each explanatory variable’s correlation with other explanatory variables. Following the trace Lasso  and X(2) with sufficiently large coefficients.Evidently the iX and the rest of the variables. Thus, in order to address inconsistency, we use the inverse covariance structure in an analogous manner to TXX invertible when p > n. In our experiments, we follow the protocol of Grave et al. to select μ  or to inconsistency   to a lightweight version that only uses the inverse covariance matrix in the regularizer (IC), which amounts to setting γng (SIS) , Lasso rng (SIS) , it was To compare these methods, we ran the following experiments: (i) Simulated data with a continuous response, (ii) simulated case-control data using logistic regression and (3) breast cancer gene expression data. We report here the results for binary case-control data, although the results for continuous data are similar . We report here the average AUC score (area under ROC curve) for case-control data. Following the convention of  AIC/BIC . MoreoveOn data simulated from a binary logistic model, we see higher AUC for the Precision Lasso methods than for the baseline methods in the cases when auto-regressive coefficient is strong . InteresWe also tested the performance of these methods for continuous response data. Here, we observed that the Precision Lasso outperformed other methods, even more so relative to our experiments on case-control data . SimilarK = k∕2 and K = 2k). We see that the Precision Lasso again outperforms the baseline methods in settings of large auto-correlation. Finally, we also compare the strategies for tuning hyperparameters to verify the argument that cross-validation is a less favourable parameter tuning strategy for variable selection. Detailed results are reported in Supplementary Section S6 of the We also tested Precision Lasso in several other settings, including mis-specified number of active variables (i.e. http://cancergenome.nih.gov/) as explanatory variables and the case-control status as the phenotype. These data consist of RNA-seq assays for 532 breast cancer samples and 63 matching control samples. While it is typical to perform significant pre-processing on these data, here we are interested in the performance of statistical methods to extract signal from data with structured noise. For this reason, we perform variable selection on the FPKM-normalized RNA-seq counts of 10 000 genes.To investigate the performance of Precision Lasso on real genomic data, we use breast cancer mRNA expression data from TCGA (http://oasis-genomics.org/) to generate a list of potential driver mutations in breast cancer and cross-check each selected gene for evidence of driver function. The results are shown in For each variable selection method, we select exactly 100 genes. To evaluate the quality of these selections, we seek to identify genes that are likely to be involved in the causation of the breast cancer. First, we filter the selected genes by intersection with the Catalogue of Somatic Mutations in Cancer  which coFOXA1 and AR, which both have been implicated as potential driver mutations. In contrast, the closest baseline method (Trace Lasso) only selected 1 gene (FOXA1) with a somatic mutation known to be associated with breast cancer. While the performance of all methods is likely underestimated due to unknown cancer associations, we expect this effect to be consistent across the sets of variables.As seen in These results suggest that the Precision Lasso can select meaningful variables when the data contains highly correlated or linearly dependent variables, as in the case of breast cancer RNA-seq assays. Furthermore, a gene-specific investigation suggests that while many genes are correlated with breast cancer oncogenesis, few have significant evidence of causality. In this setting, in which there are many variables that are linearly dependent or highly correlated, baseline methods tend to select variables which mediate the causal relationship between genotype and phenotype, as opposed to potential driver mutations that are of more interest to biologists. In contrast, the Precision Lasso tends to select these underlying driver mutations.n = 1000 with p = 5000 on a modern laptop  or n = 5000 with p = 50 000 on a modern server . Furthermore, even though the Precision Lasso shows improvements in variable selection, it does not necessarily outperform traditional methods such as the Lasso in prediction. This is unsurprising, since the much simpler task of prediction does not suffer from the unstable and inconsistent problems that are unique to variable selection.While the Precision Lasso has been shown to outperform existing methods on variable selection tasks with highly correlated and linearly dependent data, it is also of interest to compare the relative computational efficiency of the compared methods. Owing to the complexity of the regularizer employed by the Precision Lasso, it is not surprising that it requires more computational resources, scaling cubically with the number of samples and linearly with the number of explanatory variables. Empirically, we find our implementation feasible of a dataset as large as unstable and inconsistent selection. We first showed that these issues are quite real and arise in genomic datasets as a rule rather than an exception. To overcome these challenges, we proposed the Precision Lasso, a novel form of sparse regularization that overcomes many of the drawbacks of traditional methods such as the Lasso. In our experiments, the Precision Lasso outperformed these traditional methods in the presence of highly correlated and linearly dependent variables. With real breast cancer gene expression data, we demonstrated the effectiveness of the Precision Lasso to select more meaningful genes.In this paper, we studied the problem of variable selection for genomic data in which a portion of the explanatory variables are either highly correlated or linearly dependent. In this setting, traditional variable selection methods such as the Lasso struggle with The Precision Lasso also offers the potential for extension to other structured methods. In particular, we are interested to see how this variable selection method can help improve structured variable selection methods such as the group lasso , as wellThe Precision Lasso is open-source and freely available as a command line tool that is compatible with either.csv or PLINK files. We also plan to make the methodology available via a point-and-click interface by integrating it in the visual platform GenAMap (Supplementary DataClick here for additional data file.