Single-cell RNA-seq (scRNA-seq) is quite prevalent in studying transcriptomes, but it suffers from excessive zeros, some of which are true, but others are false. False zeros, which can be seen as missing data, obstruct the downstream analysis of single-cell RNA-seq data. How to distinguish true zeros from false ones is the key point of this problem. Here, we propose sparsity-penalized stacked denoising autoencoders (scSDAEs) to impute scRNA-seq data. scSDAEs adopt stacked denoising autoencoders with a sparsity penalty, as well as a layer-wise pretraining procedure to improve model fitting. scSDAEs can capture nonlinear relationships among the data and incorporate information about the observed zeros. We tested the imputation efficiency of scSDAEs on recovering the true values of gene expression and helping downstream analysis. First, we show that scSDAE can recover the true values and the sample–sample correlations of bulk sequencing data with simulated noise. Next, we demonstrate that scSDAEs accurately impute RNA mixture dataset with different dilutions, spike-in RNA concentrations affected by technical zeros, and improves the consistency of RNA and protein levels in CITE-seq data. Finally, we show that scSDAEs can help downstream clustering analysis. In this study, we develop a deep learning-based method, scSDAE, to impute single-cell RNA-seq affected by technical zeros. Furthermore, we show that scSDAEs can recover the true values, to some extent, and help downstream analysis. Single-cell RNA sequencing (scRNA-seq) has revolutionized the study of transcriptomes since the earliest scRNA-seq technique was developed by Tang in 2009 . scRNA-sSeveral methods aim at solving the missing data imputation problem in single-cell RNA-seq data. MAGIC  is the fThe autoencoders are manifold learning algorithms assuming that high-dimensional data actually lie on a low-dimensional manifold. Specifically, autoencoders first compress high-dimensional data to a low-dimensional hidden space and then reconstruct it using artificial neural networks. For their abilities to regenerate and denoise the input data, autoencoders are often utilized to do dimension reduction and missing data imputation. The loss of classical autoencoders is the reconstruction error on all features, while for imputing missing data, the reconstruction error on the non-missing features is often used. There are several pure autoencoder-based methods developed to impute missing data in scRNA-seq data. SAUCIE  uses theHere, we propose sparsity-penalized stacked denoising autoencoders (scSDAEs), a method that adopts layer-wise pretrained stacked denoising autoencoders as the imputation framework and the mixture of mean squared error (MSE) and sparse penalty as the loss to impute the missing data in scRNA-seq data. Our method has three advantages. First, it has weak assumptions on data distribution. Second, it incorporates the information of zeros but treats them differently from non-zero data. Third, layer-wise pretrained stacked denoising autoencoders improve the fitting ability of the model and enable us to fit the model better without filtering genes out, especially on small-size datasets. We compare the recovery ability of our method with three state-of-the-art methods, including MAGIC, scImpute, DCA, and other autoencoder-based methods, such as SAUCIE, SDAE and SDAE0.SAUCIE imputes and denoises all the values in the expression matrix without distinguishing zero values and non-zero ones. Because autoencoder-based methods assume the data lie on a low-dimensional manifold, replacing all the values makes the expression matrices lose their biological heterogeneity and induces many false signals. SAUCIE sometimes imputes all the cells into similar vectors with little variation . To makeFirst, we removed the cells in which no genes expressed and the genes that were expressed in no cells. After filtering, tens of thousands of genes still remained in the expression matrix, which needed handling by relatively deep neural networks. Then, we normalized the data by library size and took the logarithm of the library-size-normalized expression data with Pesudocount 1:The raw count vector of a cell with n genes expressing is denoted as An autoencoder is an artificial neural network trained to reconstruct the input data itself. It usually consists of three layers: an input layer, a hidden layer, and an output layer. The output layer has the same dimension as the input layer, while the hidden layer is of much lower dimension in order to facilitate leaning a low dimensional representation of the input data. Specifically, an autoencoder maps the input data to the hidden layer Vincent et al.  developeer noise . In our We used stacked denoising autoencoders (SDAEs)  as the iThe structure of this neural network is symmetrical where the input of the We adopt the following mixture loss function for scSDAE:The first part of the loss function is the same as others ,25 to miAlgorithm 1. Sparsity penalized stacked denoising autoencoders for single cell RNA-seq data (scSDAE).Input: normalized expression vector Output: imputed expression vector fordoifdo                         corrupt else do                         corrupt ifdopredict end forbuild neural network predict calculate the mask vector calculate the output The word “denoising” reflects in the training strategy of scSDAE, which is from Vincent et al. . The traThe encoder network is set as a fully connected multilayer perceptron (MLP) with dimensions n-500-500-2000-10 for all datasets. In addition, the decoder network is an MLP with dimensions 10-2000-500-500-n. While the output and the embedding layer are not activated by any activation functions, all other layers are activated by ReLU nonlinear function. Parameter ptimizer  with defhttps://github.com/Vivianstats/scImpute), and the default parameters were used. The DCA Python package (v.0.2.2) was installed with pip. The hidden_size and batch_size parameters were set the same as those of scSDAE, and other parameters were set as default. The SAUCIE package was downloaded from GitHub (https://github.com/KrishnaswamyLab/SAUCIE), and the default parameters were used. The Seurat R package (v.3.0) for clustering analysis was downloaded from R cran (https://CRAN.R-project.org/package=Seurat). We used the first 30 principal components to do clustering analysis and visualization and set the resolution parameter to make the number of communities as close as possible to the given cluster number. Please refer to the code to implement scSDAE with Python in The computations and analyses were carried out using R version 3.6.1  and Python version 3.7.3 . The R package “splatter” (v.1.8.0) for simulation and the packages for normalization were installed from Bioconductor. The normalization packages, including scran (v.1.12.1), Linnorm (v.2.8.0), and DESeq2 (v.1.24.0) were performed with default settings, while for scone (v.1.8.0), we set the maximum number of unwanted variation components to zero. The Python version of the MAGIC package (v.1.5.99) was installed with pip, and the default parameters were used. The scImpute R package (v.0.0.8) was downloaded from GitHub  [We first downsampled the bulk data and regarded it as the golden standard to evaluate the performance of the imputation methods. To obtain the simulated data, we adopted the relationship observed by  that thece (CMD)  between 3–104. Half of the genes were differentially expressed (DE) between the two cell-types, half were drawn independently. Spearman correlation with a conservative Bonferroni multiple testing correction (family-wise type I error rate q < 0.05) was used to identify significant gene–gene correlations. To test if false signals are induced by imputation methods, we followed the simulation strategy of . We firsNext, we assessed the accuracy of different imputation methods to identify differentially expressed genes using simulated data with Splatter . The simWe adopted the RNA mixture dataset of  to validIt is hard to directly evaluate the accuracy of imputation methods since the true expression of genes in real scRNA-seq data is unknown. However, spike-in control external RNAs with known concentrations can serve as a golden standard for comparison. Ziegenhain et al.  sequenceHere, we analyze a CITE-seq dataset  of 8617 We did a clustering analysis of five datasets with Seurat . These dTo more explicitly demonstrate the difference of different loss functions, we also analyzed another dataset with simulated genes. In , two subroup MAE . From throup MAE , althougIn this study, we proposed a deep learning-based method, scSDAE, to impute the missing values in single-cell RNA-seq data. We utilized stacked denoising autoencoders that can capture the complex nonlinear relationships among data. The loss of traditional SDAE is MSE on all the features, and the loss of SDAE0 only contains the MSE loss of the non-missing features. Different from SDAE and SDAE0, we adopted a mixture loss of MSE on non-missing values and sparse penalty on zero values, which incorporated the information of the zeros and distinguished them from the observed values. We also adopted layer-wise pretraining to train the stacked denoising autoencoders to improve the fitting ability of the model. We compared scSDAE with MAGIC, scImpute and DCA, as well as other autoencoder-based methods, on both simulation study and real data analysis. scSDAE not only obtained better performance on recovering the true values influenced by technical variations and doing clustering analysis, but also induced less false signals. The results not only showed that the SDAE framework could work well on single-cell RNA-seq data imputation comparing with traditional methods, but also demonstrated that the mixture loss proposed in this paper fitted the problem well by comparison with SDAE and SDAE0. Nevertheless, DCA and SAUCIE which impute and denoise all the values including non-zero ones often induced severe false signals. In addition, DCA without pretraining performed not well enough on small-size datasets for clustering, while scSDAE performed well on several small-size datasets, which demonstrates the power of layer-wise pretraining of the deep neural networks when the data size is small. Overall, we suggest people take an imputation step before analyzing scRNA-seq data since the severe technical zeros obstruct the analysis of it and the results in the paper showed that scSDAE could help recover the true values. To impute scRNA-seq data, there are two things whereof people need special caution. First, people should avoid over-imputing and introducing false signals, especially when using methods like MAGIC, DCA, and SAUCIE. Second, procedures to improve the model fitting of deep neural networks need to be taken when using them to impute small-size datasets.


Sputum induction is a non-invasive method to evaluate the airway environment, particularly for asthma. RNA sequencing (RNA-seq) of sputum samples can be challenging to interpret due to the complex and heterogeneous mixtures of human cells and exogenous  material. In this study, we develop a pipeline that integrates dimensionality reduction and statistical modeling to grapple with the heterogeneity. LDA-link connects microbes to genes using reduced-dimensionality LDA topics. We validate our method with single-cell RNA-seq and microscopy and then apply it to the sputum of asthmatic patients to find known and novel relationships between microbes and genes. RNA sequencing (RNA-seq) has become a standard method of analyzing complex communities. Depending on the sample type, these data can be very heterogeneous. A key problem tackled in this paper is dealing with the heterogeneity and noise in RNA-seq data in samples such as sputum. This can be appreciated by comparing sputum RNA-seq to a more traditional experiment, e.g., blood RNA-seq, where the sample can be collected consistently and that contains relatively well-defined cell types Fig. . In the Asthma is a disease of the airway that can present with diverse clinical phenotypes. Much work has focused on identifying subgroups of the disease and how each subgroup responds to treatment. For example, Yan et al. introduced transcriptional endotypes of asthma, and the Severe Asthma Research Program defined five subtypes of asthma , 2. SomeThe pathology of microbes is often inferred by the number and type of immune cells observed in samples, such as sputum total leukocyte counts , 6. A stEscherichia are human commensals, whereas Escherichia coli OH157:H7 causes hemorrhagic colitis. Alternatively, one can group sequences by the metabolic pathways observed, although this requires high-depth sequencing. Here, we propose a method to reduce the dimensionality of microbes by first linking the microbes to human genes and then applying supervised gene dimensionality-reduction methods .While deconvolution to cell fractions effectively de-noises human RNA-seq data, an equivalent supervised dimensionality reduction method does not exist for microbes. One can reduce the dimensions by collapsing microbial strains to different taxonomic ranks ; however, taxonomy is notoriously imprecise at defining behavior. For example, many bacteria in the genus A number of algorithms have been developed to infer higher-order structures within high-dimensional datasets. Latent Dirichlet allocation (LDA) was first used in text mining applications to learn the set of topics in documents and has since been used in a broad range of applications including marketing, genomics, drug-pathway relationships, and others –12. UnliIn this paper, we use RNA-seq of sputum samples from asthmatic patients to demonstrate dimensionality-reduction strategies and identify microbe-host relationships. We map RNA-seq reads onto human or microbial genomes and relate the resulting abundance matrices to each other and to clinical data. Further, we deconvolve the human reads into fractions of the various cell types that make up the sputum. Finally, we relate the human genes and microbes using a method we call the LDA-link, which identifies relationships between genes, microbes, and cell types. These methods represent a general strategy for dealing with heterogeneous RNA-seq data that is applicable to other sample types beyond sputum.We collected induced sputum samples from 115 patients with heterogeneous asthma phenotypes and sequenced these samples using RNA-seq. The median read depth per sample was 47.5 million, which meets the depth recommendations for analyses of this type . We procC, including age, weight, and lung function tests, with rows indexed by patient (Np = 115) and roughly 200 clinical variables (Nc). Alignments to human protein-coding regions created the gene table, G, with Np rows, as above, and roughly 20,000 genes (Ng). Alignments to exogenous genomes created the microbe table (M) with Np rows and roughly 1000 microbes (Nm). Given these three tables , the basic analysis framework is to correlate columns or rows within or between tables. We represent this by a matrix of correlations, R, where X∙,i is the ith column of table X and X∙,j is the jth column of table Y. This correlation is summed over the other index, usually p. For example, we test the relationship between age and the abundance of each microbe R across all patients. Similarly, we correlate the expression of a gene  with microbe  R.The goal of the analysis was to infer meaningful relationships between the numbers and origins of the RNA-seq reads and relate them to clinical phenotypes. We conceptualized the clinical information and RNA-seq alignments as a series of tables Fig. . The cliG, Npby Ng) into a cell-type fraction table  and a cell-type signature table . However, an analogous supervised method does not exist for the microbes. Therefore, we applied an unsupervised dimensionality-reduction approach, latent Dirichlet allocation (LDA), which provides topic distributions in patients  across a smaller number (k = 10) of gene topic . This can also be done to the microbe table M and get θM and φM, and the gene and microbe topic can be correlated .Individual correlations can be difficult to interpret, particularly in heterogeneous, sparse, or noisy datasets. Organizing the genes into relevant pathways or cell types can reduce the dimensionality and de-noise the analysis. To this end, we deconvolved the gene table (φG and φM). We call this method the LDA-link. Further, we relate the gene and microbe links identified to cell fractions and thereby relate how the host is responding to microbes with regard to immune cell type response with a particular gene.The framework described above is useful for identifying linear relationships, but non-linear relationships are also possible. For example, a microbe sensed by a human immune cell could lead to the activation of a transcription factor and the expression of several genes, each of which would have a non-linear relationship to microbe abundance. To identify such relationships, we applied a non-linear ensemble learning algorithm , 16, usiG) into a table of fractions of component cells type (F) and cognate cell-type signatures (S) by solving the formula G = FS. This method relies on knowing the signature gene set in each cell type, for which we used high-quality reference sets from experimentally isolated and sequenced circulating immune cells ), that is analogous to the supervised deconvolution tables F and S. We correlated the cell-type fraction table with the gene topics fraction table  for all patients and found agreement between LDA and the cell signature-based deconvolution for the most prominent cell type, neutrophils  . The top genes in each topic can be found in Additional file We compared the signal captured by cell-type deconvolution to an unsupervised decomposition method: LDA. Using LDA, we factored the gene expression table into ten topics that conceptually represent gene expression programs. This resulted in a gene-topic-fraction-in-patient table . Another method using a very different algorithm than LDA, non-negative matrix factorization (NMF), showed strong agreement with LDA  . The exogenous sequences aligned to mostly bacteria and fungi, although we also observed a few arthropod and helminth reads. The dominant phyla observed were from the bacterial kingdom: Proteobacteria, Firmicutes, and then Bacteroidetes. The abundance of Proteobacteria is in contrast to observations from the gut where Bacteroidetes predominate  is the topic distribution of documents; and φ [Nk by Nv] is the distribution of words in topics. z is a categorical integer between 1 and Nk, denoting the topic of a word in a document.In this model, φ and θ are integrated out as:The joint distribution of the LDA model is whereZ. Given Zm, n denotes the topic of the nth word in the mth document and also assume that its word symbol is the vth word in the dictionary; ¬m, n denotes all the word except the nth one in the mth document. The conditional probability can be inferred as follows:The Gibbs sampling equation can be derived from θ (doc → topic)and φ(topic → word) matrix can be inferred as follows given the symmetric hyper-parameters α and β were used:After Gibbs sampling, the expectation of the θ and φ to We instantiated the variables z-scores. Clusters were created via principal component analysis and then t-SNE of the principal component loadings. The optimal number of clusters was identified by calculating the gap statistic (method = “Tibs2001SEmax”) using the cluster package in R.Sputum cells were separated on a Fluidigm C1 medium-sized channel. The mRNA was purified from approximately 500 pg to 1 ng of total RNA using the Clontech SMARTer Ultra Low RNA Kit and poly-dA-selected using SPRI beads and dT primers. Full-length cDNA was sheared into 200–500 bp DNA fragments by sonication  and then indexed and size validated by LabChip GX. Two-nanomolar libraries were loaded onto Illumina version 3 flow cells and sequenced using 75-bp single-end sequencing on an Illumina HiSeq 2000 according to Illumina protocols. Data were cleaned, processed, aligned, and quantified following the SINCERA pipeline . BrieflyGi∙,  for i gene and Mj∙,  for j microbe:Microbe relative abundances and gene TPM values were correlated as follows, with p values were corrected by the method of Benjamini and Hochberg [Correlation Hochberg .p values less than 1e−5 (FDR = 0.016 and absolute correlation greater than 0.4)). Negative links in the training set were defined as an absolute correlation of less than R = 0.05. Each gene and microbe could be linked to many microbes or genes, respectively.We employed a self-supervised learning framework to learn labels from the data. The links between gene and microbes that were very tightly correlated, ergo they were in one cluster, were labeled as the positive links after being confirmed by functional enrichment  The microbe network modules, which were densely connected to themselves but sparsely connected to other modules, were clustered based on betweenness [The raw abundance weenness  and the weenness . We alsoAdditional file 1:Figures S1-S11. Supplemental Figures.Additional file 2:Table S1. Gene-topic membership. A tab-delimited text file showing the complete topic distributions for all genes.Additional file 3:Table S2. Gene-microbe links. A tab-delimited text file showing all genes-microbe pairs and the confidence of their link.Additional file 4:Table S3. Patient characteristics. A Microsoft Excel spreadsheet providing summary statistics for the patients included in the study.Additional file 5: Review history.


Also, we show how a tuning process may or may not significantly improve algorithms’ accuracies. Our results demonstrate an efficient and accurate classification method based on gene expression (microarray data) and ML/DL algorithms, which facilitates tumor type prediction in a multi-cancer-type scenario.Cancer classification is a topic of major interest in medicine since it allows accurate and efficient diagnosis and facilitates a successful outcome in medical treatments. Previous studies have classified human tumors using a large-scale RNA profiling and supervised Machine Learning (ML) algorithms to construct a molecular-based classification of carcinoma cells from breast, bladder, adenocarcinoma, colorectal, gastro esophagus, kidney, liver, lung, ovarian, pancreas, and prostate tumors. These datasets are collectively known as the 11_tumor database, although this database has been used in several works in the ML field, no comparative studies of different algorithms can be found in the literature. On the other hand, advances in both hardware and software technologies have fostered considerable improvements in the precision of solutions that use ML, such as Deep Learning (DL). In this study, we compare the most widely used algorithms in classical ML and DL to classify the tumors described in the 11_tumor database. We obtained tumor identification accuracies between 90.6% (Logistic Regression) and 94.43%  using  Cancer is one of the most deadly diseases in human health caused by the abnormal proliferation of cells, leading to malignant malformations or tumors with different pathology characteristics . Cancer-Impaired gene expression is a characteristic of carcinogenic cells . Accordihttps://github.com/simonorozcoarias/ML_DL_microArrays/blob/master/data11tumors2.csv. This dataset is a good example of the curse of dimensionality due to the high number of characteristics and few registers of this database. Therefore, most studies use it to test specific data science techniques, such as feature selection methods  or response (regression) of each data point by using a provided set of labeled training examples. In unsupervised learning, such as clustering and principal component analysis, the goal is to learn inherent patterns within the data .The main goal of any ML task is to optimize model performance not only on the training data but also on additional datasets. When a learned model displays this behavior, it is considered to generalize well. With this aim, the data in a given database are randomly split into at least two subsets: training and validation . Then, aIn this study, we compare the performance of the most commonly used ML and DL algorithms in bioinformatics  in the tML and DL techniques can learn the characteristics of a given problem from a certain amount of data. These data are usually randomly subdivided into two groups: training and validation. A training dataset is used to calibrate the parameters of the model, and a validation dataset is utilized for evaluating model performance .K-means algorithm, in which a given number of prototype samples, also known as cluster centers, are estimated by iteratively assigning data points to prototype samples and updating them as the mean of the assigned samples. The second method tested is hierarchical clustering, which is better suited for irregular shapes than K-means. After, we tested eight different classification algorithms. The most popular one, and the standard baseline in classification problems, is K-Nearest Neighbors (KNN), where classification decisions are done through a voting mechanism and model training stores the dataset in a way that queries can be done efficiently. Another family of classification methods comprises the so-called linear models, for which a learning algorithm estimates as many weights as features from the training data so classification prediction is done as a function of the dot product between the weights and a test sample. Linear models are fast to train, fast to predict, and also scale well to datasets in which the number of features is large compared to the number of samples. The linear methods we tested are Linear Support Vector Classifier (SVC), Logistic Regression (LR), Linear Discriminant Analysis (LDA), Naive Bayesian Classifier (NB), and Multi-Layer Perceptron (MLP).In this article, we compared results obtained from classifying 11 different tumor classes through different approaches of ML and DL. We began by evaluating two unsupervised methods; the first method is the popular We also included Decision Tree-methods (DT) such as Random Forests (RF). Unlike linear models, DTs and RFs are invariant to data scaling and work well with features on different scales. Finally, we applied Deep Neuronal Networks (DNN), such as fully connected neural networks, also known as Multi-Layer Perceptron (MLP) and Convolutional Neural Networks (CNNs). MLPs are well-suited for non-linear data, whereas CNNs automatize the costly task of engineering features; an unavoidable task in classical ML approaches. The above algorithms are extensively explained in https://github.com/simonorozcoarias/ML_DL_microArrays/blob/master/data11tumors2.csv). This database consists of 174 samples with 12,533 gene expression microarrays for 11 different types of cancer. The 12,533 microarrays of genetic expression are integers with positive and negative values; these values represent the characteristics that allow the ML and DL algorithms to learn how to classify by cancer type. The types of cancer and the number of patients for each type are shown in The datasets used represent measurements of gene expression using cancer microarrays and normal biopsies , and arem × n and the classes are a vector of size n × 1, where m is the number of samples and n is the number of genes for each class . The dataset, containing 174 samples, is randomly subdivided into two subsets , including 139 samples for training and 35 samples for validation. Initial calibration of ML and DL algorithms (training) was done using the training set; then, hyperparameter tuning was performed with the validation set and measured the accuracy of the algorithms. We calculated the accuracy of each algorithm using tuned hyperparameters with k-fold cross-validation and k = 10 to avoid overfitting.For the experiments, we divided the information into two groups; the first group corresponds to the features (X) and the second group to the classes (Y). The features compose a matrix of size The dataset used in this paper has the curse of dimensionality since the number of characteristics  is higher than the number of samples (174) . TherefoFour different datasets were created for the training and validation of each ML or DL algorithm. For the first dataset, we did not apply any preprocessing operations; for the second, we performed a scaling process; for the third, we applied PCA with a retained variance of 96% to reduce data dimensionality, obtaining a dimensional reduction from 12,533 to 83 features. Finally, for the last dataset, we applied both scaling and PCA, obtaining a dimensional reduction from 12,533 to 113 features .Classification performance is highly correlated with the degree of separability of a dataset; therefore, we analyzed performance using clustering techniques. Based on data labels, we can gain a priori insight into the algorithm that works best on the distribution of the gene expression microarray dataset.K-means algorithm with k values of one to eleven clusters using all datasets. We plotted the behavior in terms of accuracy and as a confusion matrix.Before applying the classification algorithms (supervised learning), we performed a hierarchical analysis to better understand the dataset. This hierarchical clustering used different distance metrics, such as ward, average, single, complete, weighted, centroid, and median. Further, as input, we used a dataset with no preprocessing. These distance metrics serve to capture the differences between the data samples and vary in their capacity to deal with large outliers  or if they allow choosing the number of clusters to consider  . After tWe evaluated the performance of well-known ML classification algorithms, including KNN, SVC, LR, LDA, NB, MLP, RF and DT. Subsequently, we evaluated DL architectures, such as fully connected neural networks (FNNs) and convolutional neural networks (CNNs).Two types of networks were used for DL; the first is a fully connected neural network and the second is a convolutional neural network. The FNN consists of three fully connected layers of 100 neurons each and the Softsign activation function; then, a final layer of 11 neurons is generated with the Sigmoid activation function to generate the probability of the type of cancer. The CNN consists of three convolutional layers with 128 filters each, with a kernel size of 3 and a linear activation function; followed by a layer of 100 fully connected neurons with the Softsign activation function and, finally, a layer of 11 neurons with the Softmax activation function to generate the probability of the type of cancer. Several algorithms were tested by varying or tuning parameter values to find the best performance . With thx and the number of test instances N, accuracy is defined as follows:We performed a test for difference in proportions to determine whether the difference between accuracies of the algorithms is significant. We calculated the differences between the observed and expected accuracies under the assumption of a normal distribution. Given the number of correct test predictions https://github.com/simonorozcoarias/ML_DL_microArrays.The algorithms were executed using Python programing language and scikit-learn libraries , which aBefore evaluating the classification algorithms, we visualized the intrinsic groupings in the data and determined how these groups are influenced by the different preprocessing methodologies applied to our data . Using tWard’s method created four groups, while the other methods clustered the individuals into fewer groups and, in most cases, these groups are largely unbalanced. On the other hand, the raw data and data transformed by PCA performed better in the hierarchical clustering analysis. Employing these datasets, we were able to obtain four and five clusters, respectively. Finally, the heatmaps plotted in Based on a priori knowledge that the number of cancer types is eleven (11), we were interested in determining how the hierarchical clustering algorithm created the cluster assignments. Therefore, we applied the best parameters found previously . The results shown in K-means algorithm. We used all datasets and changed the number of clusters iteratively from one to eleven, increasing by one cluster at a time. Then, we calculated the accuracy in each iteration and a confusion matrix was plotted with the best results . Also, classes 6, 7 and 9 showed precisions of 100% and class 5 of 91% .We performed a test of significant differences, with a 95% confidence level, between the two best-performing ML algorithms (LG and CNN). Accordingly, we found no significant differences between the accuracies of these two algorithms . One could assume there is overfitting since the confusion matrix showed an extremely good behavior; however, the comparison of the training and validation accuracies between parameters using the entire dataset may indicate perfect accuracy in both training and validation datasets. Additional tests with independent data should be done to discard potential overfitting.On the other hand, MLP and LDA showed a high accuracy value of 97.14% in the validation dataset. This improvement in accuracy was obtained by optimizing several parameters (number of neurons in MLP) and preprocessing the dataset with PCA.After tuning four parameters, RF obtained high results, with a maximum accuracy of 85.71%. In contrast, DT obtained 51.14% accuracy, demonstrating that DT does not work properly for the datasets used in this study, despite tuning several parameters .K-means worked better using no preprocessing. Only SVC improved when using scaling and, interestingly, none of the other algorithms showed better results using scaling and PCA on the datasets.Our findings demonstrate that the various algorithms work better by preprocessing the datasets differently. Our results show that MLP, DT, and LDA improved in performance if PCA was applied in advance. However, LG, KNN, NB, RF, and K-means) showed significant changes. We conclude that LG is the best ML algorithm for the test dataset in this study, providing an accuracy of 90.6% with a standard variation of 5.94 from the cross-validation analysis based on ten times. Nevertheless, we recommended using it with moderation. On the other hand, for DL architectures, CNN obtained the best accuracy with 94.43% (Parameter tuning can improve the accuracy of the algorithm used . For insh 94.43% . The grih 94.43% . FinallyCancer is predicted to become the most deadly disease for humans in the future ; therefo10.7717/peerj-cs.270/supp-1Supplemental Information 1These commands were executed in Rstudio to calculate if the accuracies of the ML algorithms are statistically different. Also, these commands were used to find out if the tuning process produced significant changes in the accuracies at a confidence level of 95%.Click here for additional data file.10.7717/peerj-cs.270/supp-2Supplemental Information 2These hierarchical maps were generated by data without transformation and deleting their labels. Clustering approaches demonstrate whether the data contain relevant patterns for grouping.Click here for additional data file.


The ability to confidently predict health outcomes from gene expression would catalyze a revolution in molecular diagnostics. Yet, the goal of developing actionable, robust, and reproducible predictive signatures of phenotypes such as clinical outcome has not been attained in almost any disease area. Here, we report a comprehensive analysis spanning prediction tasks from ulcerative colitis, atopic dermatitis, diabetes, to many cancer subtypes for a total of 24 binary and multiclass prediction problems and 26 survival analysis tasks. We systematically investigate the influence of gene subsets, normalization methods and prediction algorithms. Crucially, we also explore the novel use of deep representation learning methods on large transcriptomics compendia, such as GTEx and TCGA, to boost the performance of state-of-the-art methods. The resources and findings in this work should serve as both an up-to-date reference on attainable performance, and as a benchmarking resource for further research.l2-regularized regression methods applied to centered log-ratio transformed transcript abundances provide the best predictive analyses overall.Approaches that combine large numbers of genes outperformed single gene methods consistently and with a significant margin, but neither unsupervised nor semi-supervised representation learning techniques yielded consistent improvements in out-of-sample performance across datasets. Our findings suggest that using Transcriptomics-based phenotype prediction benefits from proper normalization techniques and state-of-the-art regularized regression approaches. In our view, breakthrough performance is likely contingent on factors which are independent of normalization and general modeling techniques; these factors might include reduction of systematic errors in sequencing data, incorporation of other data types such as single-cell sequencing and proteomics, and improved use of prior knowledge. The potential to tailor therapies for individual patients rests on the ability to accurately diagnose disease and predict outcomes under various treatment conditions. Predictors based on high-throughput ’omics technologies hold great promise, but a number of technical challenges have limited their applicability . PhenotyMachine learning methods often benefit from large datasets where learning complex relationships is feasible. Although individual biological experiments tend to be small, relatively large amounts of ’omics data are available in public repositories. For example, hundreds of thousands of samples from human RNA sequencing (RNA-seq) experiments are available from the recount2 and ARCHS4 databases –8. StillIf these learned features capture biologically relevant processes, then predictive models built from those features should outperform models built directly from relative transcript abundances.Genes rarely act in isolation, so it is reasonable to expect that combinations of genes may be more effective than individual genes for predicting phenotypes. For example, linear models operating on RNA-seq data create predictors from a weighted combination of gene expression values. However, some of these features could reflect biological processes that are involved in multiple phenotypes. Many previous analyses have explored this possibility by creating complex features that incorporate biological knowledge from gene sets , 11, ontl2-regularized regression methods applied directly to the centered log-ratio transform of transcript abundances performed consistently well relative to the other methods. Therefore we recommend treating that particular combination as a baseline method for predictive analysis on RNAseq data. Throughout this text we refer to the combination of l2-regularized regression methods applied directly to the centered log-ratio transform of transcript abundances as the recommended model.In this work, we present a comprehensive analysis of phenotype prediction from transcriptomics data with a particular emphasis on representation learning. Using the recount2 database , we systA high-level description of our quality control, data processing, and machine learning analyses is provided in Fig. Briefly, our dataset is sourced from the recount2 database , and conWe considered four different normalization methods to correct for variance introduced in the data collection and measurement process. We first converted the samples from counts to Transcripts Per Million (TPM) , a normaOT), protein coding genes annotated as with biological processes or molecular functions in the Gene Ontology 12 (denoted O), and all genes provided by recount2 . The O and OT gene sets are substantially smaller than the all gene set and allow exploration of the dependence on the number of genes. In total, we examined twelve different normalization-gene set combinations for each predictive problem.For each of these normalization approaches, we also explored three gene sets corresponding to transcription factors  , a Stacked Denoising Autoencoder (SDAE), and a Variational Autoencoder (VAE) trained on the 37k samples in the training set without any supervising information.l2-regularized multinomial Logistic Regression (LR) on the normalized and transformed data using 5-fold nested cross validation. Using nested cross validation ; multiclass tasks were compared using the accuracy, and survival tasks were compared using the concordance-index (C-index) [For each binary or multiclass prediction task, we trained a k-Nearest Neighbor (kNN) classifier, a Random Forest (RF), and an idation “” sectionC-index) , 32.Our systematic model search covered four normalization methods, three gene sets, four representations, and three supervised algorithms totaling 144 comparison models for each of the 24 binary and multiclass tasks. For the survival tasks we used the same normalization methods, gene sets, and representations, but considered only one supervised algorithm . For comparison, we also trained linear predictors using the recommended method that were only allowed to use a single gene. The choice of gene was treated as a hyperparameter and optimized using 5-fold nested cross validation.The predictive performance assessed through 5-fold nested cross validation varied considerably across and within the predictive problems see Fig. . Gene exIn order to compare the effects of the gene set size and transformation, it is helpful remove between-task variance and then to aggregate results across tasks. To remove the between-task variance, we defined a shifted statistic in which we subtracted the median value of all models on the same task. For example, the AUC for the random forest classifier on the STAD stage dataset was shifted by subtracting the median AUC for all of the binary classifiers trained on the STAD stage dataset. Averages of the shifted statistics across predictive problems can be easily interpreted: if the value is less than zero then the method underperformed the median, whereas the method outperformed the median if the value is greater than zero.Within-task variance in predictive performance was partially explained by the choice of gene set and normalization method see Fig. . BecauseNext, we examined differences in absolute performance between the kNN, RF, and LR models on the classification problems . As shown in Fig. Gene expression data are very high dimensional, with the number of genes ranging from ∼1.5k in the transcription factor gene set to ∼56k in the gene set consisting of all genes annotated in recount2. In contrast, the supervised task datasets typically consisted of only a few hundred samples. Moreover, it seems unlikely that genes actually coordinate in a linear fashion to generate complex phenotypes. Therefore we hypothesized that predictive performance could be improved by training predictors on lower dimensional representations derived from unsupervised analyses of the ∼37k unlabeled samples in the training set. One could also view these analyses as a type of transfer learning, in which biological knowledge derived from the analysis of one dataset is used to inform the analyses of another.The first feature representation that we considered was a Principal Components Analysis (PCA) with 512 latent dimensions. These principal components are orthogonal linear combinations of expression values that represent the directions of largest variance in the training set. Together, the 512 principal components we used explained the majority of the variation in the transcriptional datasets see . We founTraining a linear model on top of representations derived from a linear transformation like PCA is equivalent to a regularized linear model trained on the unembedded data. Deep neural network-based architectures like SDAEs and VAEs, by contrast, process an input expression vector through a series of nonlinear transformations to learn more complex features. Therefore, we also trained a 512-dimensional SDAE and VAE on the training set for each gene set-normalization combination and used the representations derived from these neural networks as features for downstream prediction tasks. Nevertheless, we found that preprocessing the expression data using these networks decreased the out-of-sample performance of downstream prediction tasks relative to just using the normalized expression data directly  can be used to train a highly accurate predictor. However, if this difference is much smaller than the variance in the expression of other non-predictive genes, then it will be ignored by most unsupervised representation learning algorithms. One way to avoid this problem is use a semi-supervised method to learn the representation.The goal of semi-supervised representation learning is to derive a common set of features that are useful for multiple downstream predictive tasks. Our semi-supervised model consists of an autoencoder along with a number of logistic regression classifiers, one for each supervised task involved in the training set. The predictors operate on the 512-dimensional latent space embedding of the autoencoder. For any expression vector the autoencoder contributes a reconstruction loss. Furthermore, if there is a predictive task label associated to the expression vector, then the associated linear predictor contributes a classification loss as well. We trained the model to minimize a loss function that was a weighted combination of the autoencoder loss and the supervised loss averaged across each of the predictive tasks. We considered the out-of-sample predictive performance of four representations: the unembedded data, data embedded by a model trained using only autoencoder loss, data embedded by a model trained on the combined autoencoder and supervised losses, and data embedded by a model trained using only the supervised loss. More details are provided in the online “In order to test the semi-supervised model, we divided the larger labeled training datasets into two halves. The first half of the labeled training datasets were combined with the unlabeled data from the training set and used to train the semi-supervised autoencoder. The second half of the training datasets were held out as validation. We also held out the validation and testing labeled datasets as in the analyses of the representations learned by unsupervised algorithms. This strategy provided two types of validation tasks: those in which the representation was trained on similar data , and those in which the representation had not been trained on similar data. The results are shown in Fig. The hypothesis that gene expression measurements can be combined into higher level features that should be useful for predicting phenotypic characteristics has intuitive appeal. Indeed, we believe that genes act together as coordinated pathways that control cellular processes. Moreover, changes in expression at the tissue level could reflect higher level changes due to differences in cellular composition. As a result, one would expect that it is possible to define useful high-level features for expression data; this intuition has driven the development of pathway analyses –35, geneMultivariate predictors outperformed predictors based on the best single gene.Larger gene sets performed better than smaller gene sets.CLR and tissue-specific Z-score normalization were better than TPM.Logistic regression and random forests performed equally well.Our key results can be summarized in a few bullet points:l2-regularized regression applied to the CLR transformed relative transcript abundances is generally the best choice for predictive analyses using transcriptomics data. The Z-score and Z-ternary normalizations generally perform comparably to CLR, but require the GTEx data as a reference and hence CLR is recommended.Representations derived from unsupervised or semi-supervised methods did not improve out-of-sample performance for phenotype prediction. Based on these key results, we conclude that Figures Our first conclusion, that multivariate predictors outperform predictors based on single gene expression measurements, was the clearest cut. This has some practical consequences when combined with our other conclusion that larger gene sets are better, especially for the fitting of proportional hazards models used for survival analyses. First, using multivariate predictors on large gene sets means that the number of covariates will almost always vastly outnumber the subjects in a study. Therefore, it is absolutely necessary to regularize these models by adding penalties to the coefficients. Moreover, nested cross validation should be used for all performance assessments to mitigate overfitting to hyperparameter choices and to minimize variance in the performance metric. Second, it is often impractical —or even impossible— to fit these models using standard methods on typical computing architectures. For example, open source packages for survival analyses typically use second-order methods to optimize the objective function. This works for a single gene, but fitting the multivariate model requires computing a very large matrix of second derivatives, e.g., 56,000 x 56,000 in this study. As a result, it was necessary to implement first-order optimization methods and perform most of the matrix operations using graphical processing units to make the survival analyses in this study feasible.Overall, we found that choices such as the normalization method, the gene set, the type of supervised prediction algorithm, and the use of a learned representation made surprisingly little impact on out-of-sample predictive performance. Moreover, we could not identify any clear trends. For example, it is not necessarily better to use smaller gene sets or other lower dimensional representations for studies with smaller sample sizes. In light of these results, it is not clear that features derived from either prior knowledge or from representation learning methods have much value in the analyses of bulk RNA-seq data. If the relationship between bulk gene expression and phenotype is not one-to-one, then there is already a limit on how well one could predict phenotype from gene expression. Relatively simple methods may be already very close to this limit.l2 coefficient rankings. When looking at the distribution of coefficient rankings we did not see examples of obvious outliers. Rather we saw distributions which looked roughly normal, consistent with the imposition of an l2 prior on the coefficients.This study consists of an analysis of methods for phenotype prediction in which methods are compared against each other according to their performance across a large number of predictive problems. The overarching question is how well different methods perform in finding a predictive signal in the datasets. Once such a method demonstrates that a useful predictive signal is present it may become beneficial to attempt an interpretation of the model features. Such analysis can yield useful insights about biological function, and the presence of the predictive signal provides credence to such interpretations. Although it is beyond the scope of this work to attempt interpretations of model features for each of the model types, we performed an interpretive analysis of the recommended model as it operates on the binary predictive tasks included in this study. We included tables  in the fl2-regularized linear predictors will yield results which are at or near state-of-the-art.Our work provides strong justification for recommendations that guide other researchers working on similar problems with RNAseq datasets. A researcher confronted with the problem of predicting phenotypes from RNAseq data ought to feel confident that Our work provides a collection of curated datasets and benchmarks which can provide terra firma on which to further develop techniques for predictive analyses on RNAseq datasets. Benchmark datasets coupled with standardized testing protocols are used extensively in machine learning research to assess technological improvements. This work provides such a resource to the computational biology community.Our work encourages researchers to direct more energy towards the reduction of systematic errors which appear “higher” in the data chain – from improved lab techniques for handling tissue samples to controlling errors in assay technology itself. Because such sources of error are likely just as present in other technologies such as single-cell RNAseq, improvements in these regards effect more than just bulk RNAseq data.We believe our analysis will have three important effects on the wide community engaged in the project of extracting insights from RNAseq data.breakthrough performance is likely contingent on factors such as reduction of systematic errors in sequencing data, incorporation of other data types such as single-cell sequencing and proteomics, and improved use of prior knowledge.In summary, transcriptomics-based phenotype prediction clearly benefits from proper normalization techniques and state-of-the-art regression approaches. However, The analysis presented here and depicted in Fig. The recount2 database  is a repThe data comprising recount2 can be divided into three broad groups according to their sources: GTEx, TCGA, and SRA. The GTEx group was sourced from the Genotype Tissue Expression program and contains 9538 samples from healthy individuals across 30 tissue types. The TCGA group was sourced from the Cancer Genome Atlas project and contains 11284 samples from individuals with cancer across 21 tissue types. In that group samples were taken from tumor sites as well as normal tissue adjacent to tumor (NAT) sites. GTEx and TCGA are each single, large collaboration projects with high quality control standards and protocols for sample processing. Metadata for these projects is extensive. The SRA group contains 49638 samples from 2033 smaller, distinct experiments collected in the Sequence Read Archive. Metadata for experiments in SRA are sparser, with tissue labels occasionally absent.Remove samples in which the reported cell type is a cell line. 9644 samples fit this criterion.Remove studies in SRA from single-cell sequencing. Examining metadata from GEO, 38 studies in SRA with 5865 samples in recount2 have single-cell transcriptomic data.Remove samples in which the reported tissue does not match any tissue in GTEx. 6824 samples fit this criterion .Remove samples in SRA which have duplicate GEO accession numbers (GSMs). There were 9601 samples that met this criterion.Remove samples in which more than 30% of genes listed in the Gene Ontology (GO) 12 under the “biological process” or “molecular function” categories have a counts value of 0. 15390 samples met this criterion.In total, 70460 samples were available in recount2 when the database was downloaded. However, many of these samples are not ideal for representation learning with transcriptomics data. We developed a quality control (QC) pipeline to remove samples or entire SRA studies. The number of samples remaining after the QC pipeline is 39848. The QC steps are as follows:The number of matching samples in each step are non-exclusive, meaning a sample can match more than one of the exclusion criteria. These effect of these exclusion criteria are depicted in the It is useful to present some detailed commentary on the duplicate GEO accession number criterion. We observed that several samples have duplicate GSMs, and that many such samples had the same number of reads . This suggests that the individual samples could be chunks of reads from the same underlying sample. However, we could find no satisfying reason for duplicate GSMs or the round number of read counts for these samples, and therefore excluded them from the dataset.The QC pipeline determines which samples are admitted to the final dataset; there are also choices to be made about which genes to consider in the analysis, and which normalizing procedures to apply to the expression data.all genes: (57992 genes).O genes: genes in GO under the “biological process” or “molecular function” categories (17970 genes at the time of dataset creation).OT genes: O genes also labeled as transcription factors 29 (1530 genes at the time of dataset creation).We considered three different gene sets in the analysis:TPM: The counts are transformed into transcripts per million (tpm), which account for gene length to normalize reads. The TPM value is determined in terms of the counts as,i and j index genes.CLR: A centered-log-ratio transform is carried out on the TPM vectors. The CLR value is determined in terms of the TPM values as,N is the number of genes.Z-score: The Z-score transform is carried out on the CLR features. The Z-score is the CLR value standardized by the mean expression of a gene in healthy tissue, determined by the GTEx samples for the same tissue. The Z-score value is determined in terms of the CLR value as,Z-ternary: The Z-ternary transform is carried out on the Z-score features. The Z-score values are ternarized based on their value, and the ternarization indicates whether the gene’s expression is increased, decreased, or unchanged relative to the mean expression in healthy tissue. Since the distribution of Z-score values is expected to be approximately normal for healthy tissue, any Z-score value below -2 is assigned the Z-ternary value of -1; any Z-score value above 2 is assigned the Z-ternary value of 1, and any Z-score value between -2 and 2 is assigned the Z-ternary value of 0.In addition, we considered four different normalizing transformations of the counts data:genemunge [We made use of the open-source python library enemunge  for makiThe above procedure describes the preparation of the gene expression datasets. In addition to the expression data, some samples have one or more labels suitable for predictive modeling. TCGA has rich metadata with natural label types, available in the TCGA Pan-Cancer Clinical Data Resource ; some SRWe divided the 50 predictive tasks into three groups, “training”, “validation”, and “test”. We then built a “training” gene expression dataset consisting of any samples with a label in the training task group, as well as any samples with no label. This dataset, which has 36794 gene expression samples, was used to train the unsupervised models. A sample’s inclusion in this dataset distinguishes the training and validation task groups. Both the training and validation tasks were used in the analysis, whereas tasks in the test group were held out until the end of the project so that no model selection criteria might influence performance on these tasks in any way. The supervised tasks are summarized in Tables Three different types of unsupervised models were trained on the gene expression datasets: principal components analysis (PCA), stacked denoising autoencoders (SDAE), and variational autoencoders (VAE). These three types were chosen because they offer a range of sophistication, features, and expressive power. PCA is a simple means of generating a linear feature embedding. SDAEs and VAEs are examples of deep neural networks which have significant expressive power. The SDAE is capable of learning an implicit distribution of feature embeddings of a dataset; the VAE learns an explicit distribution of feature embeddings which can be sampled from.k principal components of a suitably large collection of vectors of dimension n admits an analytic solution. But the computation required to perform this calculation is in The problem of finding the a et al.  called “l2 weight penalty, with these hyperparameters selected by sweeping a range.We employed denoising autoencoder architectures of “hourglass” shape with seven layers. The hourglass narrows to a middle layer of 512 dimensions, yielding a 512-dimensional encoder. The details of the architecture are recorded in the We also included a deep generative model among our unsupervised model types, the variational autoencoder . In partIn addition to these unsupervised models, we also employed a kind of control comparator: a “no-embedding” model which does nothing to the expression data. The dimension of the gene expression data is not reduced under the no-embedding model; the features are the normalized gene expression vectors themselves.O and OT gene sets excluding the all genes set. The lack of an improvement in performance on smaller gene sets indicated the dataset with all genes was unlikely to provide quality embedding models.We trained the PCA model on each of the four normalizations for each of the three gene sets. Due to computational constraints we applied the SDAE and VAE models to each of the four normalizations for the We evaluated the ability of an unsupervised model to learn useful representations across transcriptomics data by assessing the performance of supervised models operating on the learned representations. For each unsupervised model and predictive task, we trained and evaluated supervised models using nested cross validation. The performance of these predictive models gave an indication of how well the learned representation captured features in the data useful for various kinds of phenotype prediction. Before presenting the different kinds of supervised models, we present a small primer on nested cross validation.K groups, or folds (we use K=5). For each fold, the data for that fold is held out and the remaining K−1 folds are used for the inner loop. In the inner loop, this data is divided into K folds, and on each fold the data for that fold is held out and the model is trained on the remaining K−1 folds for each hyperparameter value. The held-out fold is used to estimate the model performance for each hyperparameter value, and this performance is averaged over all folds in the inner loop. The best performing hyperparameter value is selected, and the model is re-trained on all data used in the inner loop. The model performance is then evaluated on the held out data from the outer fold. This value is averaged over all folds in the outer loop, and this final average is the estimated model performance. Note that a different optimal hyperparameter may be selected for each outer fold. Nested cross validation is resistant to hyperparameter overfitting, as the model is evaluated on data completely held out from the process of selecting the optimal hyperparameter. With this robustness comes increased computational complexity —if there are N hyperparameter values tested, nested cross validation requires training K(KN+1) individual models.Nested cross validation is designed to provide a robust estimate of the expected (predictive) model performance on new data, optimizing over a set of hyperparameter values (such as the maximum depth in a random forest). In nested cross validation, there are two loops over the data, the outer and inner loop. The inner loop is used to select an optimal hyperparameter value, and the outer loop is used to estimate the performance of the model with this hyperparameter value. In the outer loop, data is divided evenly into Logistic regression (LR). Logistic regression with an l2 penalty, trained via stochastic gradient descent. The logistic regression model is a single layer neural network with a softmax activation on the output. The hyperparameter optimized was the l2 penalty, logarithmically spaced between 10−6 and 103 in ten steps. The model was implemented in pytorch [ pytorch . HyperpaRandom forest (RF). Random forest models with 100 trees per forest. The hyperparameter optimized was the maximum depth of the random forest, logarithmically spaced between 2 and 27 in seven steps. We relied on the scikit-learn [it-learn  implemenK-nearest neighbors (kNN). The hyperparameter optimized was the value of k, the number of neighbors used, taking a value of 1, 3, 5, 7, or 9.For classification tasks we applied three different types of supervised models:Although there are countless types of predictive models, we chose these three because they cover a wide range of features and characteristics of predictive models. LR is the canonical example of a generalized linear model. A random forest is a decision tree-based, non-linear classifier which is known to achieve state-of-the art performance on a large number of difficult classification problems . And finl2 penalty whose strength, logarithmically spaced between 10−6 and 103 in ten steps, was optimized in the inner cross-validation loop. Even with this computational speedup, evaluating the survival tasks requires the bulk of compute time. It bears noting that these models were still trained with a fixed initial learning rate which was small enough to guarantee controlled gradient descent across all tasks. It is certain that absolute performance on individual contrasts could be improved by also optimizing the learning rate in the nested cross validation. However, because the study concerns the relative performance of this algorithm across gene sets, embeddings, and normalizations, we avoided this additional multiplier on the computational time.For survival tasks, in which the overall survival time or the progression free interval time were predicted, we trained a Cox proportional hazard (CPH) model. The standard solvers for CPH models use second-order methods, such as versions of Newton’s method, making them unsuitable for use with a large number of features. The computation time required for the 512-dimensional embedding, using nested cross validation, is already immense; training CPH models on data without an embedding is completely impractical. Instead, we implemented a CPH model in pytorch, and trained it via stochastic gradient descent by backpropagating through the Cox-Efron pseudolikelihood. Such moAll of the above supervised models are trained on features from multiple genes. In order to compare our embedding models to single-gene analysis, we also trained a set of models on single genes with no-embedding model. For these models the hyperparameter optimized in the inner cross validation loop is the gene selected for the model. We had no need to run these comparators across all transform/predictor combinations so we restricted these examples to clr-transformed data and used only the univariate logistic regression models for classification tasks. For survival tasks, single-gene CPH models were trained. No regularization term was necessary in either case because these models have so few parameters. These results provide a direct comparison to the multi-gene, CLR-transformed, no-embedding results.Task. There are 24 classification tasks and 26 survival tasks.Gene set. There are three gene sets, all genes, O genes, and OT genes.Normalization. There are four data normalizations, TPM, CLR, Z-score, and Z-ternary.Unsupervised model. There are four types of unsupervised model, PCA, SDAE, VAE, and no-embedding. SDAE and VAE were only trained on the O and OT gene sets.Supervised model. For classification tasks, three different supervised models were trained, LR, RF, and kNN. For survival tasks, a CPH model was trained.In total, we evaluated a very large number of (multi-gene) supervised models. There are five different characteristics of a single result:This amounts to 3920 results. In terms of individual models trained during nested cross validation, there are 807600 models.The semi-supervised models are designed to learn a feature embedding which is co-adapted to the purpose of reconstruction as well as the performance of supervised models operating on the embedding. Each of the semi-supervised models consists of a linear, single-hidden layer autoencoder coupled to a number of logistic regression predictors —one for each supervised task involved in the training dataset. The supervised predictors operate on the autoencoder’s 512-dimensional encoding. Both a schematic diagram of the model architecture and the details of the architectures are recorded in the Supple-In order to be able to assess the performance of semi-supervised representation learning within-task, we had to further subdivide some of the labeled expression data. In particular, we subdivided into two halves the binary predictive tasks within the “training set” which contained at least 200 samples. The first half was used in the training of the semi-supervised model; the second half was held out for validation. We called these sets the “divided tasks”; they were drawn from the following binary tasks:{CESC grade, COAD stage, KIRC grade, KIRC stage, LGG grade, LIHC stage, LIHC grade, LUAD stage, SKCM stage, STAD stage, STAD grade, THCA stage, UCEC stage, UCEC grade}.The rest of the tasks which constituted the original “validation” and “test” sets were used for validation. So the training set for each semi-supervised model consisted of all expression data from the first halves of the fourteen divided tasks along with their associated binary labels.x from the training set we can compute the autoencoder reconstruction loss on that sample, specifically as the squared reconstruction error, R(x)i:=(AE(x)i−xi)2, in which AE(x) denotes the action of the autoencoder on x. Supposing that x has a class label lx∈{0,1} from the jth predictive task, we can also compute a classification error of the associated binary logistic regression classifier Pj,Given any sample expression vector π, which ranged from 0 to 1. Our training algorithm allowed different batch sizes for the autoencoder loss and the predictor losses; let these be denoted by BR, and BC, respectively. Let We trained our semi-supervised models (via stochastic gradient descent) to minimize a convex combination of these two error terms. The constant controlling the interpolation of these two losses we called the “predictor strength,” J denotes the total number of predictive tasks. The last two terms are l2 weight penalties on the model parameters; these are controlled by adjustable constants λAE and λP.Here, π=0.0, i.e. the model is an autoencoder only.π=0.1, the model is trained with a mixture of both losses.π=1.0, i.e. the model is a purely supervised shared-embedding model.We compared three scenarios for the predictor strength in our analysis,We also compared results to a no-embedding model as a kind of control.λAE and λP in the cartesian product {0,0.1,0.01,0.001}2. We selected the l2 coefficient pair which minimized average error on the held-out half of the divided contrasts.For each gene set, data normalization, and predictor strength scenario, we performed a sweep over all 16 pairs of values for Finally, we assessed the performance of predictive models  operating on the learned data embedding to compare the effect of semi-supervised representation learning across these three scenarios. Those results are displayed in Fig. Additional file 1 Supplementary Figures and Tables.


Cancer of unknown primary (CUP) is a type of malignant tumor, which is histologically diagnosed as a metastatic carcinoma while the tissue-of-origin cannot be identified. CUP accounts for roughly 5% of all cancers. Traditional treatment for CUP is primarily broad-spectrum chemotherapy; however, the prognosis is relatively poor. Thus, it is of clinical importance to accurately infer the tissue-of-origin of CUP.  We developed a gradient boosting framework to trace tissue-of-origin of 20 types of solid tumors. Specifically, we downloaded the expression profiles of 20,501 genes for 7713 samples from The Cancer Genome Atlas (TCGA), which were used as the training data set. The RNA-seq data of 79 tumor samples from 6 cancer types with known origins were also downloaded from the Gene Expression Omnibus (GEO) for an independent data set.  400 genes were selected to train a gradient boosting model for identification of the primary site of the tumor. The overall 10-fold cross-validation accuracy of our method was 96.1% across 20 types of cancer, while the accuracy for the independent data set reached 83.5%.  Our gradient boosting framework was proven to be accurate in identifying tumor tissue-of-origin on both training data and independent testing data, which might be of practical usage. Cancer of unknown primary (CUP) is a type of malignant tumor, histologically diagnosed as a metastatic carcinoma with no confidently anatomical primary site even after comprehensive evaluation. CUP accounts for approximately 3% to 5% of all tumors –4. In geThe patients diagnosed as CUP are treated by using traditional chemotherapy; however, prognoses of these patients are relatively poor. For a physician, accurate diagnosis can be a direct guide to individual surgical intervention as well as medication regimen. Furthermore, identification of the primary site of the tumor is relatively helpful for clinicians to design a targeted treatment plan, as well as improving survivals and quality of life , 13.Currently, the diagnostic techniques primarily include comprehensive evaluation, imaging examination, pathological analysis, immunohistochemistry (IHC) panels, and genetic testing . A gene CancerTYPE ID was a gene expression-based test, focusing on identifying the tissue of origin. This molecular test was based on real-time PCR technology by using the differential expression data of 92 genes in the tumor cells and classified tumors by matching the gene expression partem of tumor specimens to a database of 50 known tumor types and subtypes. The test compared genomic information from tumor samples with reference databases of more than 2000 tumors with definitive diagnoses. Gene expression profile analysis by using microarray data provided diagnoses of cancer types with high accuracy . AnotherRNA-seq is a high-throughput sequencing approach that sequences mRNA, small RNA, and noncoding RNA by using high-throughput sequencing technology. RNA-seq, characterized with more exact quantification, higher repeatability, wider examination area, and more credible analysis, can be used to study genome-wide differences in gene expression. In addition, it is considered as cost-effective. TOO was based on Array data, and CancerTYPE ID was conducted on the RT-PCR data; however, application of RT-PCR or Array has not only a higher cost but also a limited accuracy. Here, we conducted an experiment to identify the tissue of origin with a gradient boosting classifier  and RNA-https://dcc.icgc.org/releases/release_26/) download. In order to facilitate the follow-up work, we generated a M∗N matrix where M represents the sample size and N represents the number of genes. The matrix was generated by normalizing the expression value of each sample and each gene from TCGA. An independent data set, including 79 tumor samples from 6 cancer types with known origins, was also downloaded from the Gene Expression Omnibus (GEO). These samples belong to GSE8352, GSE8734, GSE11107, GSE11132, GSE4895, GSE6491, GSE7966, GSE7766, and GSE11843. The samples not included in the 20 cancers were excluded.The Cancer Genome Atlas (TCGA) RNA-seq and array data include 20,501 genes from the ICGC Data Portal (ft−1(x) and the loss function is L). The goal of this round of iterations is to find a weak-learner ht(x) of the CART regression tree model and minimize the loss function L = ft‐1(x) + ht(x)) of this cycle. This iteration finds the decision tree, and therefore, the sample loss is as small as possible.We employed a gradient boosting algorithm for gene feature selection and final classification with cross-validation. Gradient boosting (GBDT) is a machine learning method for regression and classification in studies, which combines multiple weak learners into prediction models . FurtherL through optimization. In the t-th iteration, the first t − 1 base learners are all fixed,Major step in this machine learning method is to minimize the loss function Minimize loss functiont wheel is expressed asThe negative gradient of the loss function of the sample of the Input cancer sample training set:Nis the number of 7633 cancer samples, the maximum number of iterations is T, the loss function is L, and output maximum learner is f(x) = fT(x).T, the following formula for the importance of each feature Xl is used:Q is the number of leaf nodes, Q‐1 is the number of internal nodes, Xv(t) is the splitting characteristic associated with the internal nodetwheretis for the cancer type, andlis the number of features. For each internal node t, the feature Xv(t) is used to simulate and divide the feature space to obtain a square error reduction after splitting, that is, Xl is summed up by the error reduction on all internal nodes. The more the total error is reduced, the more important this feature is. Because similar response values are in the same set, every node in the decision trees is a condition on a single gene. The more the total error decreases, the more important the feature becomes. For the integration of M trees, feature importance is the average of corresponding values of each tree.For a single tree L) = log(1 + eyf(x)−2). We expected to minimize the loss of the function; therefore, we used the derivative of the function to find the minimum value of the function. After getting fT(x), we have to do the probability estimation by P = P(y = 1|x) = 1/(1 + ef(x)−2).Unlike GBDT, AdaBoost selects an exponential loss, while GBDT uses the classifying loss of function from the logistic loss The study process for identifying the tumor-of-origin was shown in From TCGA  data set, we downloaded expression profiles for 7713 RNA-seq samples covering 21 common cancers without metastasis . Two sam372 metastasis samples containing 352 cases with SKCM were originally included in the test data set. However, the metastatic cases that originated from SKCM are relatively higher than those from other cancers. In order to reduce impact on the results, SKCM data were removed during data analysis.20,501 genes across 7713 samples from the TCGA data set were included in the study. In order to reduce the model complexity, we performed feature selection. First, we ranked genes by importance scores calculated by gradient boosting algorithm, and the order was defined from high to low. We conducted a series of experiments, and the experimental results are shown in In the gene selection part, we got a 7633 × 400 matrix as the input matrix, and the corresponding gene expression profile of each sample was extracted. By using the GBDT method, we set n_estimators to 200. In fact, we also tested the estimator value from 100 to 300, and the results showed an upward trend followed downward trend and reached the maximum value at 200. Therefore, we finally chose 200 weak classifiers, which meant the number of decision trees was 200. The trained tree was used to select each cancer and returned the cancer which has been selected more times. We used the gene expression values as the training features to fit the cancer type as labels.We adopted a 10-fold cross-validation in this study, which divided the data set into 10 subsets. Nine subsets were merged to a training set, and one subset was used as the test set. We repeated the algorithm ten times using the same gene features, and the average precision was 96.1%.M∗M matrix. A confounding matrix can be used to judge the accuracy of the classifier classification and is presented in the form of a graph, so it is widely used to measure the success rate of classification. The confusion matrix is a summary of the predicted results of the classification problem. It can find errors in the classification model and understand the types of errors that are occurring [The confusion matrix is a standard format for precise evaluation, which is represented by an ccurring , 21. TheK-nearest neighbor (K = 5) [K-nearest neighbor (K = 5) are closer to GBDT; GBDT is significantly higher than the other methods.We also made a comparison with  (K = 5) , decisio (K = 5) , AdaBoos (K = 5) , and sup (K = 5) . The resK-nearest neighbor (K = 5), decision tree, AdaBoost, and support vector machine. The results are shown in K-nearest neighbor (K = 5) are closer to GBDT; GBDT is significantly higher than the other methods.In order to verify the generalization and robustness of the approach, we also downloaded data sets from GEO for independent validation. The data sets covered 6 cancer types, including BRCA, LUAD, PAAD, PRAD, STAD, and THCA. And the overall accuracy rate from the gradient boosting classifier reached 83.5%.We also made a comparison with Biological validation of the optimal biomarker signature was done by GO enrichment analysis. The Gene Ontology (GO) Consortium was formed to address the limited interoperability of genomic databases due to lack of progress . FiguresIn Figures 8A heat map is a visualization method to analyze the distribution of experimental data, which directly reflected the expression of 400 characteristic genes in cancer species. As shown in The treatment of cancers with unknown primary origin is mainly empirical chemotherapy, but the prognosis of patients is generally poor. A clear diagnosis directly determines the surgical method and scope, as well as the drug regimen of physicians. Because the method in this paper is based on sequencing, this approach guides medication in patients who are sequenced. For patients who are not sequenced, the next step of diagnosis and treatment should be determined according to the guidance of doctors. The diagnostic techniques primarily include comprehensive evaluation, imaging examination, pathological analysis, immunohistochemistry (IHC) panels, and genetic testing, but the treatment is less effective. The method proposed in this paper can be used to identify tumor tissue of origin, so as to provide doctors with help and appropriate drugs according to this.We used GBDT to predict the tissue origin of the metastatic samples. GBDT can flexibly handle all kinds of data, including continuous value and discrete value. GBDT uses some robust loss functions and is relatively robust to outliers such as the Huber loss function and the quantile loss function. Because of the dependence among weak learners, it is difficult to carry out parallel training. Therefore, if the program runs too slowly with a large amount of data, it can achieve partial parallelism by adding self-sampling SGBT. The training data in this experiment was not parallel to the training data. Therefore, the results of this study might be influenced by the training method.It was demonstrated that GBDT is a powerful method of ensemble learning. Breast cancer has a high mortality rate and is the most common cancer among women worldwide. Because of the high mortality rate of breast cancer patients, the most urgent need is to find appropriate biomarkers to determine the prognosis of breast cancer, especially BRCA (invasive breast cancer) . BecauseNeither the TCGA test set nor GEO's independent test set was 100 percent accurate because a small percentage of cancers were misdiagnosed. The main reason for this error is that the two cancer species have similar characteristics and are easy to misjudge during classification, which is a key point that can be improved in the future.Since this study was researched on gene expression profiles, it is easy to make an error prediction if the gene expressions of samples are similar. Therefore, the next step was to address this problem by increasing the sample number in types of cancer.In conclusion, we applied a gradient boosting classifier to identify 20 tumor types based on expression profiles with a high accuracy, which might assist the pathologists in the diagnosis of cancers of unknown primary origins. Subsequent work has been to improve accuracy by increasing the number of samples of cancer types and improving methods.


The discovery of cancer of unknown primary (CUP) is of great significance in designing more effective treatments and improving the diagnostic efficiency in cancer patients. In the study, we develop an appropriate machine learning model for tracing the tissue of origin of CUP with high accuracy after feature engineering and model evaluation. Based on a copy number variation data consisting of 4,566 training cases and 1,262 independent validation cases, an XGBoost classifier is applied to 10 types of cancer. Extremely randomized tree (Extra tree) is used for dimension reduction so that fewer variables replace the original high-dimensional variables. Features with top 300 weights are selected and principal component analysis is applied to eliminate noise. We find that XGBoost classifier achieves the highest overall accuracy of 0.8913 in the 10-fold cross-validation for training samples and 0.7421 on independent validation datasets for predicting tumor tissue of origin. Furthermore, by contrasting various performance indices, such as precision and recall rate, the experimental results show that XGBoost classifier significantly improves the classification performance of various tumors with less prediction error, as compared to other classifiers, such as K-nearest neighbors (KNN), Bayes, support vector machine (SVM), and Adaboost. Our method can infer tissue of origin for the 10 cancer types with acceptable accuracy in both cross-validation and independent validation data. It may be used as an auxiliary diagnostic method to determine the actual clinicopathological status of specific cancer. Recent advances in molecular biology, e.g., genomics, proteomics, and metabolics, have resulted in a more accurate and specific prediction of tumor response to treatment, as well as trends in metastasis recurrence and prognosis. However, traditional detection methods, e.g., clinical, impact, and pathological examination, can only determine 50–80% of patients of metastasis cancer, while the remaining 20–50% of patients still cannot be determined  as more Many studies have attempted to use cancer biomarkers to predict the locations of primary tumors in CUP such as gene expression, miRNA and DNA methylation . Gene exCopy number variations (CNVs) usually refer to the individual difference in continuous DNA fragments with length ranging from several thousand to several trillion base pairs in genome. It can affect the individual phenotype by changing the number of gene copies, thus affecting the expression dose, disrupting the coding region structure of gene, changing the position or length of gene regulatory sequence, exposing recessive mutation, etc. . So far,In this research, an efficient multiclass model of XGBoost is established to assist in the identification of tumor origin for the entire data group from 10 types of tumors based on copy number variations. Extremely randomized tree (Extra tree) is used to design an algorithm to select fewer variables replacing the original high-dimensional variables for training. Principal component analysis (PCA) is subsequently applied to feature extraction for eliminating noise. Our method is compared to algorithms, such as SVM, KNN, Bayes, and Adaboost in terms of its multiclass classification performance. Experimental results indicated that XGBoost classifier can achieve an overall higher accuracy of more than 0.8913 for all training dataset and 0.7421 for independent dataset with 300 features, significantly improving the classification performance of other classifiers. Refer to This paper is organized as follows. A detailed description of CNV datasets are given first, then feature selection method and XGBoost classifier are introduced in section “Materials and Methods.” In section “Result and Discussion,” five classifiers including SVM, KNN, Bayes, Adaboost, and XGBoost are used to assess the prediction performance. The conclusions and discussion of the work are given in section “Conclusion.”1. Each sample has 24,174 genes with discrete copy number value denoted as “−2,” “−1,” “0,” “1,” “2,” where “−2” is homozygous deletion, “−1” is heterozygous loss, “0” is diploid, “1” is one copy gain, and “2” is high-level amplification or multiple-copy gain. To evaluate the performance of CNVs_origin, we perform the experiments on another copy number datasets, which can be downloaded from http://gdac.broadinstitute.org/ released by TCGA in 2016 . The copy number signal is produced by Affymetrix SNP 6.0 arrays for the set of samples in each TCGA study. These datasets are from primary solid tumor samples released by MSKCC in 2013, which are available from TCGA. The original data consist of 9,743 samples in total in 2016 , 2020.We carry out data preprocessing on all datasets. In the first step, genes with missing values in the samples are deleted. Then, genes contained in both MSKCC and TCGA datasets are selected. Finally, the TCGA samples that exist in MSKCC datasets are removed. After preprocessing, the MSKCC datasets have 4,566 samples and 19,895 genes, which are used as training datasets. The TCGA datasets have 1,262 samples and 19,895 genes as independent datasets. The latter includes 112 BLCA samples, 235 BRCA samples, 170 COADREAD samples, 28 GBM samples, 216 HNSC samples, 41 KIRC samples, 120 LUAD samples, 212 LUSC samples, 22 OV samples, and 106 UCEC samples. For details of all tissue samples of 10 cancers including tumor status and sample sizes after filter, refer to Each sample contains 19,895 variables whose large number may cause overfitting. As such, we carry out feature selection to improve the generalization ability of classifiers and reduce the time to train the classifier. In the research, Extra tree  is used A Wrapper feature selection method based on Extra tree is proposed. Similar idea has been proposed by some researchers utilizing random forest method . First, The steps in feature selection using Extra tree are given as follows. Refer to Input: Original dataset with all features and samples.Output: All features with their ranks and corresponding scores from the most important to the least.Step 1. The original dataset is randomly divided into k equal parts. We suppose it contained m features. k-1 of them are used to train an Extra tree, and the remaining one is used as test dataset. Therefore, we have k extreme random trees R1,R2,⋯,Rk in total.Step 2. For each Extra tree, a sequential backward selection method is used to sort the m features from the most important to the least. We delete the lowest ranking feature and recorded the average classification accuracy of Ri.Step 3. m iterations are carried out in Step 2, and Ri with the highest classification accuracy is selected as the classification accuracy of the ith random tree, and its feature set is obtained.Step 4. For the k random trees, the extreme random tree with the highest classification accuracy is used as the final feature selection.After feature selection, we obtain and rank all features according to their scores from the most important to the least; then, we use principal component analysis (PCA)  to perfoXGBoost classifier is a gradient boosting method incorporating the regression tree . It usesDefining the function as follows.ith sample, xi represents the ith sample, T represents the total number of trees, fk represents the kth tree model, and q represents the structure of each tree that maps an example to the corresponding leaf index. The objective function of XGBoost classifier is defined as T and w refer to the number of leaf nodes and the weight of leaf nodes, respectively. γ controls the number of leaf nodes, and λ is used to prevent overfitting. Each time when a tree is added, it automatically learns a new function to fit the residuals arising from the last prediction. If we obtain k trees after training, it is necessary to sum the scores corresponding to each tree to get the predicted value of a sample. In the research, we choose XGBoost classifier, the maximum depth of which is nine, learning rate γ is 0.1, and λ is 0.3 by 10-fold cross-validation as the model for numerical experiments.in which For the training dataset, first, we choose the proper classifier and feature numbers using 10-fold cross-validation with respective overall prediction accuracy. In this research, the metrics to assess the efficacy of our model included true positives, true negatives, false positives, and false negatives. We also used recall rate, precision, F1 score, and overall prediction accuracy of each cancer for assessing the multiclass predictive performance . The recC = 0.01.Furthermore, other classifiers, e.g., KNN, SVM, Bayes, and Adaboost, are used to compare with our model with the same benchmark datasets using the same 300 features. In our experiment, we compare the performance of the algorithm to other classical classifiers with the same benchmark datasets. We set the parameters of XGBoost classifier with the learning_rate γ = 0.1, max_depth = 9, andλ = 0.3. The number of categories in KNN is chosen to be equal to five. The parameter n_estimators for Adaboost classifier are 200. For SVM, linear kernel function is chosen with an optional constant Furthermore, we compare the overall accuracy with the classifiers on all 10 cancer datasets. It is obvious that XGBoost achieves the best results whether on the training datasets or the independent validation datasets. The overall accuracy is 0.7421 and 0.8913, respectively, which is 17 and 22% higher than that of SVM, higher than KNN by 22%, and higher than Adaboost by 37 and 33%. Refer to NKIRAS1 -kappa B activity by encoding a Ras-like protein. In addition, it is known that the copy number of NKIRAS1 is usually lower in RCC, and this gene is downregulated in malignant renal tumors.CDK4 or CDK6, CDKN2B encodes a cyclin-dependent kinase inhibitor. It is found that transforming growth factor (TGF) beta can drastically induce the expression of this gene, suggesting that CDKN2B might play roles in TGF-beta-related functions. In addition, the aberrant methylation of p15 (INK4B), a protein encoded by CDKN2B, can silence TRβI. Thus, CDKN2B and other tumor suppressor genes, such as CDKN2A, might be used as biomarkers for early detection in HNSC patient.By forming a complex with RBFOX1 is a member of the Fox-1 family of RNA-binding proteins, which regulates tissue-specific alternative splicing in metazoa. RBFOX1 mutations are found in COADREAD cell lines, and the loss of RBFOX1 may account for the anomalous splicing activity associated with COADREAD.p <0.01 and an enrichment factor >1.5 are identified and grouped into clusters. Log10(p) is the p-value in log base 10. All genes mainly participated in four primary biological pathways to be specific: adhesion via plasma–membrane adhesion molecules (4.61%), cell morphogenesis involved in differentiation (7.45%), positive regulation of establishment of protein localization (4.96%), and cell growth (4.96%).Function enrichment analysis is performed on the 300 genes. Specifically, all genes have been used as the enrichment background in The traditional clinical pathology experiment has its limitation. Our model recovers the original signal under the condition of oversampling, and the additional signal samples generated by the cancer sample need to be taken into account. Due to the limited data, only 10 types of tumors are selected in this manuscript for the construction of the classifier, and further studies can incorporate more types of tumors into the cancer of unknown primary model in the future work.All datasets generated for this study are included in the article/YZ conceived the experiments. TF carried out the experiments. SW designed the algorithm. RD analyzed the data. BW contributed reagents, materials, and analysis tools. JY designed the experiments. JS wrote the manuscript. All authors contributed to the article and approved the submitted version.BW, RD, and JY were employed by the company Genies (Beijing) Co., Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.