Purpose: Gene expression data generated from microarray technology is often analyzed for disease diagnostics and treatment. However, this data suffers with missing values that may lead to inaccurate findings. Since data capture is expensive, time consuming, and is required to be collected from subjects, it is worthwhile to recover missing values instead of re-collecting the data. In this paper, a novel but simple method, namely, DSNN (Doubly Sparse DCT domain with Nuclear Norm minimization) has been proposed for imputing missing values in microarray data. Extensive experiments including pathway enrichment have been carried out on four blood cancer dataset to validate the method as well as to establish the significance of imputation.Methods: A new method, namely, DSNN, was proposed for missing value imputation on gene expression data. Method was validated on four dataset, CLL, AML, MM (Spanish data), and MM (Indian data). All the dataset were downloaded from GEO repository. Missing values were introduced in the original data from 10 to 90% in steps of 10% because method validation requires ground truth. Quantitative results on normalized mean square error (NMSE) between the ground truth and imputed data were computed. To further validate and establish the significance of the proposed imputation method, two experiments were carried out on the data imputed with the proposed method, data imputed with the state-of-art methods, and data with missing values. In the first experiment, classification of normal vs. cancer subjects was carried out. In the second experiment, biological significance of imputation was ascertained by identifying top candidate tumor drivers using the existing state-of-the-art SPARROW algorithm, followed by gene list enrichment analysis on top candidate drivers.Results: Quantitative NMSE results of the DSNN method were compared with three state-of-the-art imputation methods. DSNN method was observed to perform better compared to these other methods both at high as well as low observable data. Experiment-1 demonstrated superior results on classification with imputation compared to that performed on missing data matrix as well as compared to classification on imputed data with existing methods. In experiment-2, cancer affected pathways were discovered with higher significance in the data imputed with the proposed method compared to those discovered with the missing data matrix.Conclusion: Missing value problem in microarray data is a serious problem and can adversely influence downstream analysis. A novel method, namely, DSNN is proposed for missing value imputation. The method is validated quantitatively on the application of classification and biologically by performing pathway enrichment analysis. High dimensional gene expression data helps in determining gene-to-gene interactions in different biological pathways. Molecular techniques such as “Microarrays” are used to measure expression levels of genes. Different machine learning algorithms and statistical methods are applied to gene expression data to extract relevant information for applications such as disease diagnosis and classification of clinical subtypes. These analyses assist in developing effective drugs for specific diseases because treatment procedures differ from disease to disease and case to case. For example, in blood cancer, the drugs are required to be targeted toward the specific type of cancer. Similarly, in personalized therapy, the response of a subject to a particular drug can be captured and its correlation with the mutation profiles of the subject can be examined to design targeted medicine. Thus, the therapy can be customized according to the genetic built of the subject.A persistent problem associated with microarray dataset is the presence of varying number of missing values in the data that may arise owing to poor slide quality (dusty or scratchy), poor image quality, or insufficient resolution . Subsequk nearest-neighbor imputation (KNNimpute)    is one oNimpute)  and IKNNNimpute)  are variNimpute) , least sNimpute) , and varNimpute) , iteratiNimpute) , robust Nimpute) , BayesiaNimpute) , collateNimpute) , and autNimpute)  are all Nimpute) , BayesiaNimpute)  are the Nimpute) , HPM-MI Nimpute) , and triNimpute) . GOimputNimpute) , HAimputNimpute) , and (iMNimpute)  are knowNimpute) . A briefMost of the methods perform missing value imputation in gene expression data at comparatively higher observability, say, when 70% or more data is available . Recent developments have made it possible to predict expression data values when the observed data is as low as 10%. Gene expression data is a highly correlated data because of the high level of interdependence between the genes. This interdependence is due to functional relationship between the genes as the group of genes interact together in any biological process. Therefore, it is evident that gene expression matrix is very similar to a low rank matrix that can be embedded into a lower dimensional subspace. Hence, imputation of missing values in data matrix has been projected as the matrix completion problem.Matrix completion is a popular and challenging area of research in various domains. Many matrix completion methods exist in the literature and out of these methods, LMaFit (Low Rank Matrix Fitting) , LogDet In this paper, a novel 2-stage method, DSNN (Doubly Sparse DCT domain with Nuclear Norm minimization), has been proposed for predicting missing values in gene expression data using Compressive Sensing (CS) based formulation. In the first stage, missing values were recovered in gene expression data by formulating it as the CS-based reconstruction with double sparsity in the Discrete Cosine Transform (DCT). It has been shown in Gupta et al.  that DCTMatrix obtained in first stage is considered a noisy version of the original matrix. Therefore, in Stage-2, denoising of the matrix recovered from Stage-1 is done by utilizing nuclear norm minimization. It exploits the low rank property of the data matrix. Missing value imputation was performed on four blood cancer dataset at different observability of data (10–90%) using NMSE as evaluation metric. Significance of imputation was validated by two experiments. In the first experiment, classification of normal vs. cancer subjects was carried out. In the second experiment, biological significance of imputation was ascertained by first identifying top 500 genes using SPARROW algorithm , followeGSE50006) submitted by Dana-Farber Cancer Institute, USA. CLL dataset contains expression values of 220 subjects across 54675 probe-ids and consists of two classes depending on whether the subject has CLL or not. There are 188 tumor samples and rest 32 are normal samples. Dataset-2 is Acute myeloid leukemia (AML) cancer dataset (GSE9476) (GSE47552) (GSE125361) belonging to Indian population. It contains gene expression data of 48 MM subjects across 58341 probe-ids.Four publicly available microarray gene expression dataset of different cancer types and different population have been used. Dataset-1 is Chronic lymphocytic leukemia (CLL) cancer dataset (GSE9476)  from FreSE47552)  from Cen6) for the CLL dataset, data was log transformed to reduce its dynamic range and to ensure that the smaller values were not shadowed by the higher values during the missing data recovery method.Data was pre-processed to convert probe-ids to gene symbols because gene vs. sample information is required for SPARROW analysis. It was observed that several probe-ids showed same gene names. To overcome this problem, gene expression levels of the probe-ids corresponding to the same gene name were averaged and gene vs. sample matrix was created. After pre-processing, CLL dataset had 220 samples with expression values of 23,348 genes. AML dataset had 64 samples with expression values of 13,650 genes. MM-Spanish dataset had 99 samples with expression values of 23,307 genes. MM-Indian dataset had 48 samples with gene expression values of 33,973 genes. Since the range of gene expression values was very high  method consists of two stages. Stage-1 imputes missing values using a CS-based framework and DCT-based sparsity, while Stage-2 removes noise from the matrix obtained from Stage-1 by using a simple denoising framework.Y of size r × s, where r represents the number of subjects and s denotes the number of genes. Since the expression value of any gene will not vary much across subjects, data within a column would be sparse in some transform domain. Similarly, for a sample, gene expression levels of the gene will also be sparse in some transform domain. Columns and rows of the gene expression matrix were studied in the DCT domain and were observed to be highly sparse as shown in In this stage, missing value problem was projected as compressive sensing based reconstruction problem. To understand it better, consider an incomplete matrix r × s and had “0” entries for missing values in data matrix Y, while rest of the entries were “1.” Corresponding to each observed entry (that is not missing) of the ith column, there is a row in Φi with an entry “1” for the corresponding position and zeros in the rest of the positions. For example, assume xmissing=x1, x3, and x6 are available and, x2, x4, and x5 are missing (denoted as “.” in the vector). Then, the vector xmissing can be re-written as y:Thus, the missing data recovery problem was formulated in a compressive sensing framework, where the sensing matrix Φ was of size x is the desired vector to be recovered. This is the standard formulation in compressive sensing literature, where it is assumed that only few values of data are sensed. In the above example, these values are x2, x4, and x5. Thus, we have recast the problem of missing values in vector xmissing as the compressively sensed vector y. Now, the task is to recover full data x from compressively sensed data y that will lead to missing value recovery.where the sensing matrix is written as Gene expression data was interpreted as a matrix with few observed samples, where the goal was to reconstruct the original matrix from the observed entries using DCT-based sparsity of gene expression data.YThe following optimization problem was solved to recover the missing values in Dc is columnwise DCT matrix applied on columns of the Dr is the rowwise DCT matrix applied on rows of the where 1 was chosen automatically by the “SPGL1” solver.where ” solver , 39, wheX and hence, the recovered matrix was denoised in Stage-2. Before denoising, It was assumed that the recovered j ranges from 1 to m (number of rows/ subjects), |.| denotes the absolute value and, mean(yi) and std(yi) denote the mean and the standard deviation of the ith column of the initial observed (but incomplete) matrix Y. Parameter λ2 was determined empirically and was set to value 0.2 for experiments on CLL dataset, MM-Spanish dataset, and MM-Indian dataset. It was set to 0.1 for experiments on AML dataset. Denoising was formulated in the Split-Bregman type optimization aswhere where where “°” represents the Hadamard product of two matrices with the elements of Equation (5) was solved in Split Bregman type iterations as3 was determined using grid search and was set to 100 in all experiments. The complete algorithm for the proposed DSNN method is presented below.where “SVT” denotes the soft singular value thresholding method  and X~obFor assessing the performance of the proposed DSNN method, some data were dropped randomly to create incomplete matrices with available data ranging from 10 to 90%. Next, incomplete matrices were imputed using the DSNN method. Results were simultaneously generated using three state-of-the-art matrix completion methods for comparative analysis. Normalized mean squared error (NMSE) was used as the evaluation metric and was calculated between the original and the recovered matrix. NMSE is defined as:Semi-log plots of NMSE at different stages are shown in In order to determine the significance of the DSNN method, two separate experiments were carried out on the original data, incomplete data, and imputed data matrices. In experiment-1, classification of normal vs. cancer subjects was carried out. In experiment-2, biological significance of imputation was ascertained by first identifying top candidate tumor drivers from SPARROW algorithm, followed by gene enrichment analysis on the top-ranked genes using web based application Enrichr.k nearest neighbor (KNN) method with k = 3. Both the dataset had large number of features, therefore, feature reduction was performed to extract important features from the data. Three different methods of feature reduction were used, Mutual Information criterion, Principal Component Analysis (PCA) and Chi-square method. Optimal number of features in each method were estimated by grid search. Further, five-fold cross validation was performed and average accuracy over 20 iterations was reported. Experiments were performed in Python 3 environment with Sklearn 0.20 library. Classification code was written in Python programming language. Scikit-learn is a Python module for machine learning and contains various algorithms related to regression, classification, and clustering. Examples of these algorithms are support vector machines (SVM), random forest (RF), k-means. Classification accuracy and F1 score were calculated at different sampling ratios from 10 to 90%. The accuracy and F1 score are defined asSimulation results on missing value recovery were validated by performing classification on original matrices, matrices with random missing values, and imputed matrices of the CLL and AML dataset. Classification can be either supervised or unsupervised depending on the availability of ground truth labels. In these dataset, ground truth labels were available. Hence, supervised classification was performed to distinguish between two classes, normal and cancer using two different classifiers: linear Support Vector Machine (SVM) and M is the total number of samples in the dataset, xi is the ground truth class label of the ith sample, and F1 score was used in order to account for label imbalance arising out due to unequal number of tumor and normal samples. CLL dataset had 188 tumor and 32 normal samples and AML dataset had 26 tumor samples and 38 normal. F1 scores for incomplete matrices are low as compared to the values obtained on imputed matrices. Classification accuracy and F1 scores were also computed on imputed matrices obtained from the three existing methods on both the dataset and compared with the results of DSNN method as shown in where For biological validation of the results, SPARROW was applied on the original matrix, incomplete matrices, and imputed matrices to identify top candidate tumor driver genes. SPARROW  was proposed by Logsdon et al.  and aimsym,k is the value of expression of the nth gene for the mth subject, em,n is a normally distributed error, xm,k is the value of expression of the kth SER for the mth subject and βk,n is the additive effect of the expression of the kth SER on the expression of the nth gene. m ranges from 1….M, where M is the total number of subjects and n ranges from 1…….N, where N is the total number of genes. Total SERs used in the analysis were around 3,400 and they were downloaded from the link provided in the original paper. This algorithm provides a rank to each SER based on the gene expression values of the samples. The top-ranked genes from the list can be further studied by gene enrichment analysis.where p-value) whereas for imputed matrices, cancer pathways were predicted with a higher significance due to decrease in p-value. Results from KEGG analysis on all dataset are presented in tabular form showing the p-values, combined score for original data and the incomplete and complete matrices in tabular form in p-value was computed from the Fisher exact test. Fisher test was run on random gene sets and ranks were derived at each run. Mean rank was calculated from the different runs and standard deviation of the rank obtained from the expected rank was also calculated for each term in the gene-set library. Finally, a z-score was calculated to estimate the deviation from the expected rank. z-score and p-value were used to compute combined score which is obtained by multiplying z-score with the logarithm of p-value. A detailed analysis for CLL dataset consisting of z-score and combined score has also been shown in For finding top 500 candidate driver genes, only the tumor samples from the data matrices were considered for SPARROW analysis. Algorithm was applied on original complete data matrices of all the dataset to identify the top-ranked candidate tumor drivers. This served as the ground truth for our analysis. Further, SPARROW was applied on incomplete and imputed data matrices of both the dataset at sampling ratios of 50 and 70%. Top-ranked candidate drivers from the incomplete and imputed data matrices were obtained. Gene enrichment analysis was performed on top 500 genes. KEGG and GO pathways were studied using web based application, Enrichr, developed, and maintained by Chen et al.  and KuleDSNN, a two stage method proposed for matrix recovery in the paper, was based on Compressive Sensing Framework. In Stage-1, it utilized column and row sparsity of the gene expression matrix in DCT domain for missing value imputation, while in Stage-2, it exploited low rank nature of the matrix for denoising. Expression values of any particular gene would vary slowly across subjects, thereby, exhibiting sparsity in columns in some transformed domain. Similarly, expression values of a subject for most of the genes will also be slowly varying, thereby, exhibiting sparsity in the rows. Since there is a high inter-dependence between the expression levels of the genes, one may consider gene expression matrix as a low rank matrix. Thus, as discussed earlier, both the assumptions used in Stage-1 (of sparsity in DCT domain) and Stage-2 (low rank of matrix) hold true for the given gene expression data. The concept of DCT-based sparsity was recently applied on biological data in methods, TV-DCT  and CT-NF1 scores reduced as the number of missing values increased. There were 220 samples in CLL dataset and 64 samples in AML dataset. For smaller dataset like AML, missing values affected the classification accuracy and F1 scores greatly. Thus, it is necessary to impute missing values in gene expression data to prevent incorrect downstream analysis of the data. When the classification was performed on the imputed data, there was considerable improvement in the classification accuracy, thereby, validating our hypothesis. Classification accuracy and F1 scores calculated on original complete data matrices (100% sampling ratio) were considered as ground truth values. For CLL dataset, ground truth values of classification accuracy and weighted F1 score were 0.99 and 0.99, respectively, as shown in F1 score obtained for 50% observed data was 0.86 and 0.80, respectively as shown in F1 score were 1.0 and 1.0, respectively as shown in F1 score for 50% observed data was 0.56 and 0.57, respectively, as shown in F1 score improved considerably to 0.99 and 0.99, respectively. For every sampling ratio, consistent results were obtained that validates our method.It was evident from the results shown in p-value < 0.05 were only considered. When KEGG analysis was done on top-ranked genes from incomplete matrices, there was significant decrease in the p-value of the most significant pathways. “Wnt signaling pathway” (p-value for “Notch signaling pathway” was 2.00E-01 at ground truth and it was 5.76E-02 at 70% observed data for CLL dataset. Values were insignificant in both the cases. However, after imputation, p-value became significant with value 1.56E-02 which was <0.05 as shown in KEGG and GO enrichment analysis was performed on the top 500 ranked genes obtained from SPARROW algorithm to biologically validate our results. As mentioned earlier, KEGG pathways obtained by the top-ranked genes of original matrices were considered the ground truth values. Pathways with pathway” , 42 and pathway” , 44 are p-value for “Wnt signaling pathway” was 8.33E-05 on original dataset, as shown in p-value became significant with value 2.13E-03. Similarly, p-value became 1.90E-05 after matrix imputation on 70% observed data which was more significant than the p-value 6.66E-5, observed at 70% data. “Fc epsilon RI signaling pathway” is an important pathway in AML cancer (p-value-2.12E-01. At 70% observed data, p-value was 9.40E-02 which was again greater than 0.05. After matrix imputation, the value became significant at 2.75E-02, which was less than 0.05 as shown in p-value-0.0052 for original data but became insignificant with p-value-0.23 when 70% data was observed as shown in p-value 0.04. For MM-Indian dataset,“Transcriptional misregulation in cancer” was found to be insignificant with p-value 0.47 as shown in p-value decreased to 1.37E-03 and became more significant than ground truth p-value, 7.8E-03. Additional KEGG analysis results on the dataset CLL, AML and MM Spanish data are provided in the Similarly, L cancer . This paThus, DSNN method not only imputed missing entries but also performed some denoising to improve the results. It is quite evident from the analysis that gene enrichment analysis results were partially inaccurate due to incomplete matrices. This was because the genes identified as top-ranked genes by performing SPARROW analysis on complete data matrix were not identified in the top-ranked list obtained from incomplete data matrix. However, when the incomplete matrix was imputed using the proposed DSNN method, top-ranked list of genes obtained from SPARROW analysis was quite similar to the ground truth. Our observations demonstrate the importance of imputing missing values in gene expression data.Microarray data generally has a lot of missing values that can adversely influence the downstream analysis. In this paper, a new method, namely DSNN, is proposed that imputes missing values in the gene expression data using discrete cosine transform based double sparsity and nuclear norm minimization. Method was also validated quantitatively based on the application of classification approach as well as biologically by performing pathway enrichment analysis and showed consistent findings.GSE9476, GSE50006, and GSE47552 and GSE125361. Code of the DSNN method proposed in this manuscript is available publicly at github:https://github.com/AkankshaFarswan/DSNN_MatrixImputation.Data was downloaded from GEO database with accession numbers, AF and AG conceptualized the method of data imputation. RG and GK contributed to the ideas on method application and validation in bioinformatics. AF prepared all codes, generated results, and prepared the first draft of the paper. All authors edited the manuscript for final submission.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Lung cancer is one of the most malignant tumors, causing over 1,000,000 deaths each year worldwide. Deep learning has brought success in many domains in recent years. DNA methylation, an epigenetic factor, is used for model training in many studies. There is an opportunity for deep learning methods to analyze the lung cancer epigenetic data to determine their subtypes for appropriate treatment.Here, we employ variational autoencoders (VAEs), an unsupervised deep learning framework, on 450K DNA methylation data of TCGA-LUAD and TCGA-LUSC to learn latent representations of the DNA methylation landscape. We extract a biologically relevant latent space of LUAD and LUSC samples. It is showed that the bivariate classifiers on the further compressed latent features could classify the subtypes accurately. Through clustering of methylation-based latent space features, we demonstrate that the VAEs can capture differential methylation patterns about subtypes of lung cancer.VAEs can distinguish the original subtypes from manually mixed methylation data frame with the encoded features of latent space. Further applications about VAEs should focus on fine-grained subtypes identification for precision medicine. Lung cancer is one of the most malignant tumors with the fastest growth in morbidity and mortality, causing over 1,000,000 deaths each year. There are two common histological subtypes of lung cancer, lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). In order to understand the heterogeneity of lung cancer, many researchers have done a lot of work based on immune-response genes, DNA mutations and DNA methylation . As a weIn recent years, deep learning has been performed and achieved state-of-art performances in many domains, including speech, image classification, text and natural language processing, but has seen slow adoption for in bioinformatics. NeverthVariational autoencoders (VAEs), which are unsupervised deep learning approaches, have become more and more popular in the research area. Interestingly, through feature compression and nonlinear activation functions, the VAEs can capture an underlying data manifold from input data . CompareHere, we use a VAE model in the study of lung cancer - including two subtypes: LUAD and LUSC - epigenetic data. Although the samples of lung cancer from TCGA are labelled, we chose unsupervised learning here instead of supervised learning to verify whether the reconstructed features by VAEs can represent the original data labels. We demonstrate that the encoded 100-dimensional latent space holds meaningful information of the original methylome. It is showed that the features of latent space represents the patterns of LUAD and LUSC epigenetics and the VAE model may be available for analysing DNA methylation data to extract features associated with subtypes.In order to verify the feasibility of VAEs to extract a biologically meaningful latent space from DNA methylation data, we employed a VAE model on the top 300,000 probes that were chosen by median absolute deviation (MAD) of methylation beta values across 917 samples containing LUAD and LUSC subtypes. The 300,000 features were encoded to an intermediate layer with 100 dimensions which were then encoded back to 300,000 dimensions by a non-linear combination. For an insight into the 100 dimensions space, t-Distributed Stochastic Neighbor Embedding (t-SNE)  method wIt took about 2 hours to complete the model training on a server Ubuntu 16.04.6) with 1T memory and no GPUs. The process of model training is shown in Fig. 6.04.6 wiIn order to further investigate the relative information of the latent features, a well-known feature compress and visualization method, the t-SNE, was performed on the 100-dimensional features of the latent space resulting in 2D features.A scatter for the 2D features was plotted which showed an obvious distribution with four main classes showing separation Fig. . The disResults of unsupervised hierarchical clustering with the 2D t-SNE features on the merged data frame are shown in Fig. To test the utility of the compressed 2D features, logistic regression classifiers were performed on the merged data frame with “1 vs The Rest” analyses by sklearn module . A half Tybalt, which was developed to learn a latent space on pan-cancer RNA-seq data by Way and Greene, on epigenetic data from lung cancer to extract a meaningful relevant space. The above analyses show that epigenetic data of lung cancer is suitable for unsupervised deep learning to mine its subtypes. Moreover, it shows that the VAE model could extract a biologically relevant space and the meaningful information still can be captured by the further compressed features after dimensionality reduction with the t-SNE method.We transfer the application of The successful applications of deep learning on many domains give us a clue that it should be used for precision medicine for more effective treatments in the context of cancer rates and cancer-related mortality fast increasing.With the data of TCGA released publicly, large-scale dataset and pan-cancer analyses can be achieved by deep learning methods. In the future work, pan-cancer 450K DNA methylation data can be trained by the VAEs to get a latent representative space relevant to large-scale datasets from multiple tumors. To better understand the tumors, further analyses and multi-omic data should be involved, for example, fine-grained subtypes of a specific tumor should be identified, and gene ontology (GO) enrichment analyses can be run on CpGs with high weights, and non-coding RNA data can be added into input , 13–18.In this work, we demonstrate that the epigenetic data of lung cancer samples is capable of unsupervised deep learning with VAEs. A biologically meaningful latent space can be extracted by the VAE model from the manually merged dataset, which represents the distribution about different subtypes of samples credibly. By comparing the results of unsupervised hierarchical clustering with the original labels of samples, VAEs can capture the different methylation expression patterns for various subtypes.https://portal.gdc.cancer.gov/) through the GDC data transfer tool.All level-3 plain files of Illumina HumanMethylation450 (450K) DNA methylation data for LUAD and LUSC samples were downloaded from the The Cancer Genome Atlas (TCGA) project  and 475 tumor samples. For tumor samples, there are 473 primary solid tumor samples (short for LUAD-01) and 2 recurrent solid tumor samples (short for LUAD-02).TCGA-LUSC dataset contains 412 LUSC samples. It consists of 42 tumor-adjacent normal samples (short for LUSC-11) and 370 solid tumor samples (short for LUSC-01).Table Tybalt [Tybalt was developed for extract a relevant latent space from cancer transcriptomes of 10,459 tumors. The original model consisted of an encoder and a decoder where 5,000 input selected genes were encoded to 100 latent features and reconstructed back to the 5,000 genes. Way et al. chose Keras (version 2.2.2) [Tybalt with the following values: 50 for batch size, 0.0005 for learning rate, 1 for κ, 90/10 for training/validation ratio. We changed the epochs from 50 to 100, expecting better training.We extend Tybalt , a VAE mn 2.2.2)  to buildn 2.2.2)  and traiThe latent space consisted of 100 features compressed from 300,000 CpGs. First, the 100-dimensional latent features were evaluated whether they may represent the different methylation patterns for LUAD/LUSC subtypes. So, we performed unsupervised hierarchical clustering on the merged data frame with the 100-dimensional latent features then compared the clustering results with the original labels of each sample.For a more intuitive representation, we performed dimensionality reduction on the 100-dimensional latent features. The t-SNE method converts the Euclidean distance to probability distribution using Gaussian distribution making it suitable for feature compression and visualization . We perfIn order to test the utility of the 2D t-SNE features, “1 vs The Rest” logistic regression classifiers were trained with the 2D features from t-SNE analyses. Specially, to simplify the model training, the labels of the only 2 LUAD-02 samples were converted to “LUAD-01”. So, four bivariate classifiers  were developed to classify samples using the 2D features. The merged data frame was randomly split into 50/50 using Pandas module for training/testing sets where 50% of the samples from each subtype were included.


Data errors need to be identified and corrected before integrative data analyses where different types of data are merged on the basis of the annotated labels. Data with labeling errors dampen true biological signals. More importantly, data analysis with sample errors could lead to wrong scientific conclusions. We developed a robust cis-associations was small and/or the number of samples was large. Application of our proMODMatcher to multi-omics datasets in The Cancer Genome Atlas and International Cancer Genome Consortium identified sample errors in multiple cancer datasets. Our procedure was not only able to identify sample-labeling errors but also to unambiguously identify the source of the errors. Our results demonstrate that these errors should be identified and corrected before integrative analysis.Application to simulated datasets suggests that proMODMatcher achieved robust statistical power even when the number of Our results indicate that sample-labeling errors were common in large multi-omics datasets. These errors should be corrected before integrative analysis. With advances in high-throughput technologies in the past 2 decades, diverse types of omics data at multiple layers of regulation have been generated to survey complex human diseases , which aRecently, many large projects, such as The Cancer Genome Atlas (TCGA) and International Cancer Genome Consortium (ICGC), have generated diverse types of omics data for public use. However, data errors, including sample swapping, mis-labeling, and improper data entry, are almost inevitable in the process of large-scale data generation and management. Westra et al.  found ∼2Identifying and ultimately correcting these sample errors are critical for statistical data analysis, especially for integrative analysis. Data errors need to be identified and corrected before extensive efforts are devoted to data analysis. Analyzing data with sample errors is a waste of limited public resources. More importantly, data analysis with sample errors could lead to wrong scientific conclusions. Furthermore, sample errors have more significant effects on integrative data analysis where different types of data are merged on the basis of the annotated labels. Some types of sample errors can be detected during data quality control on each individual type of data, whereas such sample errors as sample swapping or mis-labeling are difficult to detect by means of data quality control on that individual type of data alone.cis-associations” between 2 types of omics data and then to use these biological cis-associations as intrinsic barcodes to match different types of omics data. The types of biological cis-associations are different when different pairs of omics data are mapped, but they all reflect general biological regulations. For example, when mapping genotype and gene expression data, the method is based on cis-genetic regulation of expression traits (or expression quantitative trait loci—cis-eQTLs), where a genetic polymorphism at a gene's promoter or regulatory region affects the binding of transcription factors or co-factors, which in turn affects the abundance of the gene's transcript ) for assessment. Additionally, because a pair of omics profiles mostly has unbalanced samples, we mimic this by adding 10% of M samples for Type A and Type B omics profiles.Simulated datasets for testing alignment between a pair of omics profiles were generated. Given a set of cis-associations. Samples were initially matched based on annotated sample ID and potential cis-associations  . Thus, MODMatcher was not powered to accurately align these types of profile pairs.Numbers of significant curacies . This reciations . On the cis-associations and samples  were simulated by randomly assigning some samples’ labels; then proMODMatcher and MODMatcher were applied to identify aligned profile pairs. As expected, when a larger number of FPR Fig. . Across her Fig. . The topher Fig.  but betther Fig. . These scis-associations between miRNA and mRNA profiles was small. We used the TCGA breast cancer (BRCA) dataset as an example to illustrate the profile alignment results in detail. There were mRNA expression profiles based on 2 different platforms, Agilent microarray and RNAseq technology. There were 519 tumor samples with both mRNA expression measured in Agilent microarray and miRNA expression measured by small-RNA sequencing method, and 1,041 tumor samples with both mRNA expression measured in RNAseq and miRNA measured by small-RNA sequencing method. A small portion of miRNAs are embedded in gene regions  and frequently co-transcribed with host genes [cis-associations. A total of 1,222 miRNAs were profiled, and 227 and 271 of them were mapped to host genes, for Agilent microarray and RNAseq data, respectively. Among them, 138 of 227 and 175 of 271 miRNA-host gene pairs were significantly associated with each other at q-value < 0.05, for Agilent microarray and RNAseq data, respectively. For example, in the case of miR-452 located in the gene body of GABRE, its expression was highly associated with mRNA expression of GABRE  for self-aligned RNAseq-miRNA profiles were much higher than for other possible pairs in general Fig. . An examP-value of 7.3 × 10−6.Comparing MODMatcher and proMODMatcher, the proMODMatcher identified an additional 91 self-aligned profile pairs that were missed by MODMatcher. For example, the similarity score of self-alignment for TCGA-AO-A0JF-01 was among the highest when the miRNA profile was compared to RNAseq profiles of other samples   Fig. . The croow) Fig.  indicateow) Fig. . Our preow) Fig.  and C. Iow) Fig. , consistson Fig. .The proMODMatcher identified a cross-aligned pair between the mRNA Agilent array profile of TCGA-BH-A0BZ-01 and the miRNA profile of TCGA-E2-A15K-01 (See Table cis-associations can accurately identify sample-labeling errors and unambiguously correct labeling errors.These results together suggest that proMODMatcher with 138 q < 0.05) with the corresponding mRNA's expression level were defined as significant cis-associations based on Agilent microarray and RNAseq data, respectively  in TCGA to align miRNA Table  and RPPAFor alignments between mRNA and RPPA profiles, the self-alignment rates were lower than alignments between mRNA and miRNA Fig.  for mostC4-01 and TCGA-AX-A1CI-01 after considering other types of omics data , DNA methylation profiles based on Illumina HM450, miRNA expression profiles, and copy number somatic mutation profiles. The ICGC datasets used and the associated alignment results are summarized in Table cis-associations. We showed that the number of candidate intrinsic cis-associations between mRNA-miRNA profiles or mRNA-RPPA profiles was low. Application of our proMODMatcher to alignment between mRNA-miRNA profile pairings and mRNA-RPPA profile pairings from 22 different cancer datasets in TCGA demonstrated that sample-labeling errors occurred even in datasets of high quality and our procedure was not only able to identify sample-labeling errors but also to unambiguously identify the source of the errors.We developed a sample alignment method, proMODMatcher, for detecting and correcting sample-labeling errors by aligning omics profiles. The proMODMatcher extended our previous method MODMatcher by estimating probabilities of potential matches rather than using ranks of similarity scores. Applied to simulated datasets, proMODMatcher outperformed MODMatcher when aligning the omics data profiles with a relatively small number of Integrating multi-omics data into comprehensive network models is essential to elucidate complex molecular mechanisms of cancers. After correcting sample-labeling errors, associations between different profiles were stronger. For example, mis-labeled samples were outliers when comparing significant pairs between mRNA and miRNA expression levels in the TCGA BRCA dataset Fig. .cis-associations is small.We showed that some potential cross-aligned profile pairs in the TCGA BRCA dataset were missed by proMODMatcher. The sensitivity and accuracy of multi-omics profile-matching methods need further improvement. Integrating >2 types of profiles in probability estimation may yield more robust sensitivity and specificity when the number of cis-associations, and the information content (Shannon entropy) of each cis-association depends on the randomness of the genotype at each locus or gene expression of each gene. For example, if there were 2 possible genotypes at a locus, then randomness or Shannon entropy is maximized when the probability of each genotype is 50%. When the probabilities of the 2 genotypes deviate from equal, the randomness or Shannon entropy at the locus decreases. Thus, in our analyses, we excluded biological cis-associations that are driven by extreme values (rare events). For example, in eQTL analyses, we only included loci of minor allele frequency >0.05. Missing values commonly occur in high-throughput omics data. In our analyses, we do not explicitly impute missing values. Instead, we filtered out probes or genes of >25% missing value in the data pre-processing step.The proMODMatcher depends on a set of biological The computational cost of applying proMODMatcher is small. For example, mapping mRNA and miRNA expression profiles for 408 samples took 802 seconds of CPU time with maximum memory usage of 503 MB on a machine with CPU processor of 3.50 GHz.Our results demonstrated that sample-labeling errors were common in large multi-omics datasets. Our method has improved statistical accuracy to identify and curate these errors over the previous method and is generally applicable to other datasets. Application of our general framework for automated curation of public databases and properly merging omics data would be the fundamental basis for the development of effective integrative approaches.cis-associations. Probes in different types of data were matched by intrinsic biological relationships. For example, probes in methylation, miRNA, and CNV profiles were mapped to a close transcript based on hg19 reference genome. Samples were initially matched on the basis of annotated sample ID and potential cis-associations  adjusted q-value < 0.05  and RT, where n,iA and n,iB represent the measurements of sample i and nth cis-related probes for Type A and B profiles, respectively  cis-eQTLs for mapping genotype and gene expression data: a genetic polymorphism at a gene's promoter or regulatory region affects binding of transcription factors or co-factors, which in turn affects the abundance of the gene's transcripts [cis-eQTL. (2) cis-methylations for mapping DNA methylation and gene expression data: increased DNA methylation at CpG sites near a gene promoter region is associated with gene repression [cis-methylation probe. (3) cis-CNVs for mapping DNA CNVs and gene expression profiles: amplified or deleted genomic regions can regulate the expression levels of genes within that genomic region [cis-CNV. (4) cis-miRNA-gene pairs for mapping miRNA and gene expression profiles: a small portion of miRNAs are embedded in gene regions  and frequently co-transcribed with host genes [cis-miRNA-gene pair. (5) cis-mRNA-protein pairs for mapping protein and gene expression profiles: the abundance of a protein depends on the corresponding mRNA transcript level and other factors [cis-mRNA-protein pair.“Biological nscripts . If the pression . A methyc region . If a gest genes , 18. If  factors . If theiself-aligned, where j of Type A and sample k of Type B, cross-aligned.In the “determine self-aligned vs cross-aligned” step Fig. , the simi of Type A and profile j of Type B, mA and mB represent the Type A and Type B profile of the mth matched profile pairs, respectively. Given the bivariate normal distribution, we calculated the distance of a point The characteristics  of 2 types of profiles may be different. The rank-based cut-off was not able to reflect similarity score differences in a specific similarity score distribution with a large or small variance . In the rix Fig. . The proiA and jB and their similarity score i,jr, we calculated the p-value of the 2 profiles matched by chance asAdditionally, we introduced a prior probability of self-alignment i indicates the samples with both Type A and Type B profiles. And the In this study, the prior probability p-value p-value p-value p-value The similarity score les Fig. . Each roTo assess improvement of signals after labeling error correction, we calculated the Spearman correlation between miRNA expression and its host genes with initially matched pairs based on sample ID and with aligned sample pairs. To avoid bias due to different number of samples, we matched the number of samples of initially matched pairs to the number of aligned pairs. We randomly selected the samples with the same number of aligned pairs, and calculated the Spearman correlation. We performed random selection 100 times and calculated the mean of correlation.Project name: ProMODMatcher (passcode to decrypt the zipped file is “password123”)https://github.com/integrativenetworkbiology/proMODMatcher) and http://labs.icahn.mssm.edu/zhulab/software/Project home page: Github site (Operating system: Platform independentProgramming language: R (R 3.5.1 or later)Other requirements: R package mnormtLicense: GNU General Public LicenseRRID: SCR_01 7219GigaScience repository, GigaDB [Data supporting the results of this article are publicly available at the Firehose database, TCGA data portal, and ICGC data portal (see Data Description). Data further supporting this work and snapshots of our code are available in the , GigaDB .Supplementary Figure S1. Similarity scores of matched pairs based on simulated datasets.Supplementary Figure S2. Similarity scores of cross-aligned pairs detected by only MODMatcher based on aligning RNASeq and miRNA profiles of BRCA data.Supplementary Figure S3. Similarity scores of cross-aligned pairs based on aligning RNA and miRNA profiles of BRCA data. (A) Alignment of RNAseq and miRNAseq profiles. (B) Alignment of Agilent array and miRNAseq profiles.Supplementary Figure S4. Similarity scores of cross-aligned pairs detected by only MODMatcher based on aligning Array and miRNA profiles of BRCA data.Supplementary Figure S5. Similarity scores of cross-aligned pairs based on aligning RNA and RPPA profiles of BRCA data. (A) Aligning of RNAseq and RPPA profiles. (B) Aligning of Agilent array and RPPA profiles.Supplementary Figure S6. The frequency of genes of which the number of samples with non-assigned values are greater than each percent of samples.Supplementary Figure S7. Examples of distribution of similarity scores. The red asterisk indicates the values of the highest similarity score.Supplementary Table S1. Sample alignment results by MODMatcher for mRNA (Array and RNAseq) -miRNAseq profiles based on BRCA data.Supplementary Table S2. Sample alignment results by MODMatcher for mRNA (Array and RNAseq) -RPPA profiles based on BRCA data.giz080_GIGA-D-19-00039_Original_SubmissionClick here for additional data file.giz080_GIGA-D-19-00039_Revision_1Click here for additional data file.giz080_GIGA-D-19-00039_Revision_2Click here for additional data file.giz080_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giz080_Response_to_Reviewer_Comments_Revision_1Click here for additional data file.giz080_Reviewer_1_Report_Original_SubmissionEttore Mosca -- 3/13/2019 ReviewedClick here for additional data file.giz080_Reviewer_1_Report_Revision_1Ettore Mosca -- 5/31/2019 ReviewedClick here for additional data file.giz080_Reviewer_2_Report_Original_SubmissionSijia Huang -- 4/16/2019 ReviewedClick here for additional data file.giz080_Reviewer_2_Report_Revision_1Sijia Huang -- 5/31/2019 ReviewedClick here for additional data file.giz080_Supplemental_FileClick here for additional data file.BH: Benjamini-Hochberg; BLCA: bladder urothelial carcinoma; BRCA: breast invasive carcinoma; CESC: cervical and endocervical cancers; CNV: copy number variation; CPU: central processing unit; eCOAD: colon adenocarcinoma; DLBC: lymphoid neoplasm diffuse large B-cell lymphoma; eQTL: expression quantitative trait locus; FPR: false-positive rate; GBM: glioblastoma multiforme; HM27: Illumina HumanMethylation27 Beadchip; HM450: Illumina HumanMethylation450 Beadchip; HNSC: head and neck squamous cell carcinoma; ICGC: International Cancer Genome Consortium; KIRC: kidney renal clear cell carcinoma; KIRP: kidney renal papillary cell carcinoma; LGG: brain lower grade glioma; LIHC: liver hepatocellular carcinoma; LUAD: lung adenocarcinoma; LUSC: lung squamous cell carcinoma; miRNA: microRNA; MODMatcher: Multi-Omics Data matcher; mRNA: messenger RNA; OV: ovarian serous cystadenocarcinoma; PRAD: prostate adenocarcinoma; proMODMatcher: probabilistic Multi-Omics Data matcher; READ: rectum adenocarcinoma; RNAseq: RNA sequencing; RPPA: reverse phase protein array; SARC: sarcoma; SKCM: skin cutaneous melanoma; SNP: single-nucleotide polymorphism; STAD: stomach adenocarcinoma; TCGA: The Cancer Genome Atlas; THCA: thyroid carcinoma; UCEC: uterine corpus endometrial carcinoma.The authors declare that they have no competing interests.This work was partially supported by the National Institutes of Health .E.L. and J.Z. designed research. E.L. performed research and analyzed data. S.Y. contributed to download data and analyzed data by MODMatcher method. W.W. contributed design of simulation. Z.T. contributed revising paper. E.L. and J.Z. wrote the paper. All authors read and approved the final manuscript.


In silico quantification of cell proportions from mixed-cell transcriptomics data (deconvolution) requires a reference expression matrix, called basis matrix. We hypothesize that matrices created using only healthy samples from a single microarray platform would introduce biological and technical biases in deconvolution. We show presence of such biases in two existing matrices, IRIS and LM22, irrespective of deconvolution method. Here, we present immunoStates, a basis matrix built using 6160 samples with different disease states across 42 microarray platforms. We find that immunoStates significantly reduces biological and technical biases. Importantly, we find that different methods have virtually no or minimal effect once the basis matrix is chosen. We further show that cellular proportion estimates using immunoStates are consistently more correlated with measured proportions than IRIS and LM22, across all methods. Our results demonstrate the need and importance of incorporating biological and technical heterogeneity in a basis matrix for achieving consistently high accuracy. Cell type deconvolution from bulk expression data rely on a reference expression matrix. Here, the authors introduce a basis matrix built using data from both healthy and diseased samples profiled on 42 platforms, reducing biases introduced by single-platform matrices built using healthy samples. Multiple computational methods have been developed6 to estimate the proportions of immune cells in blood6 and tissue biopsies7, as well as cell-type specific expression profiles from bulk expression data9. The underlying assumption in virtually every deconvolution approach to date is that the observed expression of any given gene in a mixed-tissue sample is a combination of its expression across each cellular subset1. Based on this assumption, methods for estimating cellular frequencies from mixed tissue data use a variant of a regression model, such as linear regression3, quadratic programming4, robust regression7, or support vector regression7. Irrespective of the type of statistical model, each method requires a reference expression matrix, called a basis matrix, that is composed of genes specifically expressed in the expected cell subsets found in the tissue of interest8.Cell-mixture deconvolution is an established in silico approach for quantifying cell subpopulations directly from bulk gene expression data of mixed cell samples3) or more datasets , which are profiled using a single microarray platform to ensure homogeneity in expression data. However, it is possible that this approach can introduce a technical bias in a basis matrix towards the microarray platform used for transcriptome profiling, resulting in lower deconvolution accuracy for samples that are profiled using different platforms. Furthermore, basis matrices to date have been created using expression data solely from healthy subjects7, which can further introduce biological bias that could affect deconvolution accuracy and limit their applicability to samples from patients with disease.Typically, a basis matrix is constructed from sorted cell expression data by combining expression profiles of sorted immune cells from one  measured across eight microarray platforms from two different manufacturers (see Methods and Supplementary Table 21. We estimated accuracy of deconvolution across samples by computing their goodness of fit as previously described (see Methods section)7. Briefly, if the original mixed-tissue sample expression could be reconstituted by combining the estimated proportions of individual cell types with the expression from basis matrix, we would observe high goodness of fit and expect good deconvolution accuracy.We hypothesized that a basis matrix created using gene expression data generated from a single microarray platform would exhibit significant platform-dependent bias  in deconvolution accuracy. To test our hypothesis, we used IRIS and LM22 as basis matrices, both of which are constructed using only healthy samples profiled only on Affymetrix microarrays22, across samples from different platforms. We calculated MAD as difference in goodness of fit for each sample from mean goodness of fit across all platforms for a given basis matrix, and estimated its statistical significance against the null hypothesis that there was no technical variation between samples (see Methods section). We observed significant heterogeneity in goodness of fit between platforms for both IRIS  and LM22 , irrespective of the method used  irrespective of the method used.Using these data, we created a new basis matrix consisting of 317 cell-type specific genes, called immunoStates,  will have lower goodness of fit when deconvolving a disease sample, and hence, lower deconvolution accuracy, whereas a basis matrix created using both healthy and disease samples  will have higher goodness of fit and accuracy. To test this hypothesis, we used E-MTAB-62 , which was profiled using GPL10558, using these two Illumina-specific basis matrices and immunoStates, and correlated estimated cellular proportions with those measured by cytometry. Compared to immunoStates, both Illumina-specific basis matrices had lower correlations, irrespective of the method used  using data only from a specific Illumina microarray (GPL10558) and (2) using data from all Illumina microarrays. We deconvolved GSE65133  was downloaded, processed, and annotated from Array Express (http://www.ebi.ac.uk/arrayexpress) using the ArrayExpress R package. Dataset GSE65133 [https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE65133] and its paired flow-cytometry data were directly downloaded from the CIBERSORT website (https://cibersort.stanford.edu). Paired flow-cytometry data for GSE59654 [https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE59654] was downloaded from ImmPort . Data from the Stanford-Ellison cohort from years 2011 to 2013 was collected and processed as previously described26. Dataset E-MTAB-62 was downloaded, processed, and annotated from Array Express (http://www.ebi.ac.uk/arrayexpress) using the ArrayExpress R package.Unless otherwise noted, we downloaded all datasets from Gene Expression Omnibus  against all remaining cells within that lineage (controls). We applied a correction for small sample size bias to Hedge’s g as needed. For each gene g and cell type i within lineage l, we computed an effect size asj is any cell type within lineage l such that j ≠ i. A high effect size indicates a strong separation between our target cell type i and its closest cell type j.We collected and processed 165 publicly available gene expression datasets comprising 6160 microarray samples profiling sorted human leukocytes. Datasets used to build immunoStates are described in Supplementary Data i from every other cell type j in l. We first estimated the classification accuracy of the first gene of the list by estimating the area under receiver operating characteristic (AUROC) curve. Next, we incrementally added one gene at a time following their ranking and recomputed the AUROC corresponding to the new gene set. We repeated this process until we identified the minimal gene set that produced an AUROC proximal to the maximum (with ε = 0.005), requiring a minimum of 5 genes per signature. We performed this strategy on each cell type across each lineage and obtained a signature of 201 genes.We then ranked genes in decreasing order according to their effect sizes, and performed a step-wise search to identify the smallest gene signature able to accurately classify cell type g with our selection strategy to be more accurate in distinguishing cell types. All code was written and run using the R programming language.We then applied the same strategy to distinguished lineages from one another. We excluded all the signature genes that were used to separate cell types due to their confounding contribution in separating lineages. We performed the same gene selection strategy described above and obtained a second gene set of 116 genes. Together, these sets make the 317 genes that form immunoStates. To build the basis matrix with expression values, we computed the mean expression for every gene in every cell type (317 genes by 20 cell types) from the quantile normalized expression matrix 7. We used PERT as is from its source code in Octave21. We replicated all the pre-processing steps that CIBESORT performed on both the expression sample and basis matrix  in order not to be confounded when we compared different methods. We assessed the effect of re-scaling for the linear model and robust regression, two methods that did not require re-scaling in their original implementation, by computing the pairwise correlation of estimated cell proportions with and without re-scaling  and the IRIS matrix from the CellMix R package. Statistical comparisons were performed using the Wilcoxon’s rank sum test. Analysis and plots were generated using the R programming language.We performed deconvolution with support vector regression using the CIBERSORT algorithm (v1.03)7. Briefly, for a given basis matrix M and a mixture sample gene expression vector s, deconvolution estimates the known cell proportion vector p such thatTo quantify the extent of platform-dependent technical bias in deconvolution, we analyzed a collection of 1071 microarray samples profiling human PBMCs in diseased and healthy individuals profiled on platforms GPL96, GPL571, GPL570, GPL5715, GPL6244, GPL6104, GPL6947, GPL6102, and GPL6480  of the median goodness of fit across every platform and deconvolution method. We chose MAD because of its robustness to outliers in estimating the heterogeneity of the distribution of interest. To estimate significance of our MAD scores, we generated a background distribution of MAD scores for each platform, which represent an expected distribution of homogenous MAD values, and then performed a Z-test, asking whether the observed MAD was significantly higher than expected. All analysis and plots were generated using the R programming language.Goodness of fit of the basis matrix https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-62/] which contains samples profiled on Affymetrix HG-U133A arrays (GPL96). For the purpose of this analysis, we excluded all cell line samples and removed samples that had been used to generate immunoStates. Briefly, we deconvolved each sample across every matrix and method combination, and estimated a p-value indicative of the significance of its goodness of fit7. We then grouped all samples based on whether they originated from a healthy or a disease affected donor. Within each group, we compared the significance of goodness of fit between samples originating from blood and those originating from solid-tissue biopsies. Using the p-values as scores, we measured their accuracy in distinguishing blood-derived samples from solid tissue biopsies by computing an area under the receiver operating characteristic curve (AUROC), as previously described7, for all combinations of basis matrices and deconvolution methods. Analysis and visualization was performed using the R programming language.To estimate the effect of disease to sample deconvolution, we analyzed dataset E-MTAB-62 [https://khatrilab.stanford.edu/immunostates. All analyses and visualization were performed using the R programming language.Code for all the analysis can be found at Supplementary InformationPeer Review FileDescription of Additional Supplementary FilesSupplementary Data 1Supplementary Data 2


Deciphering the meaning of the human DNA is an outstanding goal which would revolutionize medicine and our way for treating diseases. In recent years, non-coding RNAs have attracted much attention and shown to be functional in part. Yet the importance of these RNAs especially for higher biological functions remains under investigation.In this paper, we analyze RNA-seq data, including non-coding and protein coding RNAs, from lung adenocarcinoma patients, a histologic subtype of non-small-cell lung cancer, with deep learning neural networks and other state-of-the-art classification methods. The purpose of our paper is three-fold. First, we compare the classification performance of different versions of deep belief networks with SVMs, decision trees and random forests. Second, we compare the classification capabilities of protein coding and non-coding RNAs. Third, we study the influence of feature selection on the classification performance.As a result, we find that deep belief networks perform at least competitively to other state-of-the-art classifiers. Second, data from non-coding RNAs perform better than coding RNAs across a number of different classification methods. This demonstrates the equivalence of predictive information as captured by non-coding RNAs compared to protein coding RNAs, conventionally used in computational diagnostics tasks. Third, we find that feature selection has in general a negative effect on the classification performance which means that unfiltered data with all features give the best classification results.Our study is the first to use ncRNAs beyond miRNAs for the computational classification of cancer and for performing a direct comparison of the classification capabilities of protein coding RNAs and non-coding RNAs. Lung cancer is one of the most common cancers in humans worldwide among both men and women, as well as the leading cause of cancer-related deaths . There aNon-coding RNAs (ncRNAs) is a broad class of transcripts, consisting of well known transcripts with structural  and regulatory  roles, and transcripts whose functions remain largely unknown –12. The Among the many categories of ncRNAs the most well understood are miRNAs , siRNAs and piRNAs, which guide effector Argonaute proteins to genomic loci or target RNAs in a sequence-specific manner. lncRNAs, on the other hand, are implicated mostly in the regulation of gene expression and many are functionally validated by now to be involved in different cellular and developmental pathways , 23. DysRegarding the classification of disorders, lncRNAs have been used to distinguish different subtypes in human breast cancer and glioblastoma , 26. HowOur paper contributes to these diagnostic investigations by studying the classification capabilities of protein coding and non-coding RNAs from lung cancer. Specifically, we are using RNA-seq data from lung adenocarcinoma patients  generateWe perform this analysis for a number of different state-of-the-art classification methods, including deep learning neural networks, decision trees, random forrests and support vector machines –34. We sFor our classification task, we make use of recent progress in deep learning models based on neural networks . DespiteDeep learning models have been successfully used in many application areas, most notably in image recognition , 42 and Our paper is organized as follows. In the next section we present details about the lung cancer data and the methods for our analysis. Then we present our results and a discussion of these. We finish the paper with concluding remarks.http://www.ncbi.nlm.nih.gov/geo/), accession number GSE40419.For our analysis, we are using RNA-seq data from lung cancer in Koreans . The datThe processing pipeline for the data include the following steps : The datTopHat and STAR would be alternative choices instead of Bowti2, which are quite commonly used in transcriptomics studies. Their major difference from BowTie2 is an opportunity to account for reads in splice-affected regions. However, in  it was aUsing the human genome annotation databases Reference Sequence (hg38) as reference to map RNA sequencing reads to protein coding RNAs, we find expression levels for 36,742 RefSeq genes. After filtering out redundant entries this results in 22,427 gene transcripts. Removing genes with low expression values  or genes with a small standard variation results in 12,360 genes corresponding to protein coding RNAs we used for our analysis.microRNA: miRNAribosomal RNA: rRNAsmall interfering RNA: siRNAlong non-coding RNAs: long ncRNAs or lncRNAssmall nuclear ribonucleic acid: snRNAFor a similar analysis for non-coding transcripts we find 3124 non-coding RNAs (ncRNAs) for RPKM and 1398 ncRNAs for TPM (16828 before filtering). That means the majority of ncRNAs is expressed at a very low level. Among these the most important ncRNAs are :microRNFor our analysis, we need to assess the performance of a binary classification. The results form such a classification can be summarized by a confusion matrix shown in Table A) Accuracy (True positive rate (TPR)  True negative rate (TNR)  From the confusion matrix in Table For our analysis a true positive (TP) indicates a correctly predicted lung adenocarcinoma sample and a true negative (TN) a correctly predicted control sample. Hence, the true positive rate is with respect to lung adenocarcinoma and the true negative rate for the control samples. The true positive rate (TPR), also called sensitivity, evaluates the proportion of all positives correctly identified and the true negative rate (TNR), also called specificity, evaluates the proportion of all negatives correctly identified. With respect to the confusion matrix the TPR and the TNR are symmetrically defined by exchanging the class labels. Overall, the TPR has a focus on positive labels which correspond in our case to lung adenocarcinoma patients and the TNR has a focus on negative labels corresponding to control patients. In contrast, the accuracy assesses the overall classification performance for both classes in a weighted manner. In addition, we evaluate the area under the receiver operator characteristics (AUROC) curve . For theDue to the fact that the number of samples in both classes is exactly the same, none of our measures suffers from negative consequences of imbalanced classes .For assessing the variability of our results, we use a 10-fold cross validation (CV) in order to estimate the standard errors of the performance measures. CV splits the data into 10-folds whereby one fold is used for training (estimation of parameters of the models) and 9-folds are used for testing. CV is a resampling method that is the gold standard for error estimations in order to avoid a high bias , 60.stochastic gradient descent either in combination with the basic backpropagation (Bprop) algorithm or the resilient backpropagation (Rprop) algorithm, whereas the resilient backpropagation (Rprop) algorithm is a more efficient, faster variant of the Bprop.For our analysis, we are using Deep Belief Networks (DBNs) . DBN modIn the following, we describe the pre-training and fine-tuning steps briefly.In principle, it is possible to learn neural network models solely by supervised learning methods skipping a pre-training step entirely. However, it has been shown that pre-training with a suitable initialization of the model parameters, i.e., the weights of the network, can make the supervised learning step much faster and improve the overall performance . The ResTechnically, the pre-training step of DBNs consists in stacking RBMs, so that the next RBM in a chain is trained by using the previous hidden layer as its input layer, in order to initialize parameters for each layer. In previous studies this approach has been found to be efficient . FurtherIn the supervised training step the parameters of the model are optimized by fine-tuning the model. For this step, the class labels of the training data are used, which makes this step supervised. In contrast, the pre-training step does not make use of these labels and for this reason is unsupervised.+ algorithm. iRprop+, as well as all other realizations, are available in the darch package [+, we are using the basic backpropagation algorithm (Bprop) for reasons of comparison.The resilient backpropagation (Rprop) algorithm is a motified version of the backpropagation algorithm. The purpose for introdicing this algorithm was to speed-up the backpropagation (Bprop) algorithm . There a package . In addiThe architecture of the neural network is a parameter of the model that needs to be defined. From previous studies it is known that there is not one type of network architecture that is best under all conditions but the choice of the architecture is data and problem dependent. For instance, some studies use a decreasing architecture (the number of neurons in the hidden layers decreases) , whereasIn the following, we will analyze RNA-seq data from lung cancer patients in two ways. First, we will only use gene expression values from protein-coding RNAs corresponding to mRNAs. Second, we will only use gene expression values from non-coding RNAs (ncRNAs) including miRNAs.The RNA-seq data for our analysis consist of 62 samples from lung adenocarcinoma and 62 samples from adjacent normal tissues corresponding to control samples. For our first analysis will only use gene expression data from protein-coding genes. For our data set this corresponds to 12360 mRNAs (genes). We will use these data for a binary classification separating adenocarcinoma samples from control samples. The results of this classification are summarized in Table Our results are interesting for several reasons. First, the DBN classifier in combination with SVM or alone outperform the SVM, decision tree (DT) and random forrest (RF) for almost all combinations. Second, Bprop and Rprop perform similarly with only a slight advantage for Rprop. This is somehow surprising because it is know that Rprop performs in general better than Bprop. Third, the deep learning classifiers outperform the SVM but only slightly.%.In Table In order to compare our results with previous findings, we performed a literature search. From this we found two related studied, the results are shown in Table There are further studies about the classification of lung cancer, but they are less close to our setting. For instance, in  lung adeAnother study by  used 86 We want to highlight that in addition to the differences mentioned above, all of these less close studies used DNA microarray data but no RNA-seq data.Regarding the application of deep learning classifiers, in  samples As a result from this literature overview its seems that our study is the first to investigate the classification of deep learning classifiers without feature selection.In this section we are repeating a similar analysis as in the previous section, however, with one important difference. Instead of using RNA-seq data from genes coding for proteins we will use RNA-seq data from genes that do not code for proteins (non-coding genes) leading to non-coding RNAs. In our data set, we have in total 3124 non-coding RNAs.Our results are summarized in Table In Table Recently, it was shown that information about the presence and absence of miRNA isoforms (isomiRs) can be used to discriminate between 32 cancer subtypes . The aveAll of the results from the literature have in common that they all used feature selection and they all performed worse than our best results. Also to the best of our knowledge there is no study that included ncRNAs beyond miRNAs in their analysis.Next, we are comparing the results we obtained for different classification methods . In Fig. Overall, the accuracy values for data from non-coding RNAs are for all compared classification methods higher than for data from coding RNAs Fig. a. The diDue to the fact that deep learning networks do not require a feature selection mechanisms for reducing the dimension of the input data we did not use such a mechanism as a preprocessing step for our analysis so far. However, now we want to study the effect of feature selection , 83 in aIn Figs. The joint impurity filter (JIM) is defined as,Xj is the expression value of feature j, Yj is its the outcome variable (class label), S is the set of already selected features and G is the Gini impurity gain,Here J(Xj).The method starts with the feature that maximizes the impurity gain and then greedily adds new features that maximize The joint mutual information (JMI) is defined as ,3\documS is again the set of already selected features and I is the joint mutual information between Xj,Wi and Yj. The method adds new features Xj∉S in a greedy way by maximizing J(Xj).Here In Fig. Overall, these investigations demonstrate that a feature selection does not have a positive effect on the classifiers.From analyzing the classification of lung adenocarcinoma patients by using a number of different state-of-the-art classification methods, we found that data from non-coding RNAs have a comparable classification performance as data from coding RNAs, whereas for DBN we found an even better performance. This demonstrates that (I) both data sources (coding and non-coding RNAs) contain a comparable amount of information regarding the underlying disease and (II) the results are robust and do not depend on a particular classification method. From this we conclude the equivalence of predictive information as captured by non-coding RNAs and protein coding RNAs and their utility for computational diagnostics tasks in lung adenocarcinoma.An intuitive biological explanation for this observation is given by the idea underlying the epigenetic landscape  or the mvalid features, which are all supposed to carry information, and select from these a small subset to which the analysis is limited. Even for the data sets we analyzed without using any feature selection mechanism we applied a feature filtering to remove noise from the data.In our study, we showed that for the classification of lung adenocarcinoma patients a feature selection is not beneficial, but reduces the prediction accuracy for the DNN and SVM , SVMs and combinations.⇒ This may point to a new application area for ncRNAs in the computational diagnostics of lung cancer and potentially other disorders.Feature selection reduces this predictive ability in both cases and the best prediction accuracy is obtained without feature selection.⇒ This eliminates a general problem all biomarker studies suffering from, which is the definition of the optimal biomarker set .Deep belief networks perform competitively to SVMs.⇒ Despite this positive finding, compared to the overwhelming success of DBN for image analysis, the result differences are not large enough to claim a dominating performance.In our study, we assessed the entire information content of coding RNAs (mRNAs) and non-coding RNAs provided in RNA-seq data by studying the data with and without feature selection. Overall, from analyzing a large-scale data set from lung adenocarcinoma patients we found:From reviewing the literature it seems our study is the first to use ncRNAs beyond miRNAs for the computational classification of cancer. All previous studies limited their focus on miRNAs when classifying disease stages, e.g., , 78, 79.For future studies, it would be interesting to expand our analysis to other cancer types and general complex disorders. It would be interesting to see if also other diseases exhibit the same behavior as we observed for lung cancer. We expect similar results to hold for other cancers but for general complex disorders predictions are more difficult.In summary, our investigations underline the importance of general ncRNAs in understanding the complex etiology of lung cancer and suggest to conduct similar studies for other cancer types and possibly other complex disorders.