Simulation studies showed that for data sets with small  or large sample size  limma based on data from the l, l2, and r2 transformations performed better than limma based on data from the voom transformation in term of accuracy, FDR, and FNR. For datasets with moderate sample size , limma with the rv and rv2 transformations performed similarly to limma with the voom transformation. Real data analysis results are consistent with simulation analysis results: limma with the r, l, r2, and l2 transformation performed better than limma with the voom transformation when sample sizes are small or large; limma with the rv and rv2 transformations performed similarly to limma with the voom transformation when sample sizes are moderate. We also observed from our data analyses that for datasets with large sample size, the gene-selection via the Wilcoxon rank sum test (a non-parametric two sample test method) based on the raw data outperformed limma based on the transformed data.We propose eight data transformations ( RNA-seq technology can help identify new genes, with high-sensitivity, high signal-to-noise ratio and small sample requirements. Also, RNA-seq technology can measure read counts at exons, genes, or gene units. Therefore, RNA-seq sequencing technology has been widely used in many different research fields4.With the rapid development of next-generation high throughput RNA sequencing technologies in recent years, genomics studies have seen tremendous advancement. RNA-seq technology is a type of next generation sequencing technology to estimate the expression levels of genes in whole-genome scale studies and has become the standard technology for the study of genomics5. According to the gene annotation and genome build, numbers of features might be different. Different pipelines, such as Cufflink pipeline, Hisat2-StringTie pipeline, and Star-FeatureCount pipeline could result in different properties of the count matrix. Two common properties are sparsity and skewness. Sparsity means that many counts in the count matrix are zero. Skewness means that the histogram of all counts in the count matrix is usually skewed. Skewness indicates that data transformation is required before applying linear regression analysis, which assumes data from normal distributions. Sparsity indicates that the log2 transformation, which is commonly used in gene microarray data, could not be directly applied to RNA-seq data analysis since log2(0) does not exist. It is still expensive to collect RNA-seq data for large sample size. Hence, existing RNA-seq datasets usually have small sample size. To address these two common properties, count distributions, such as Poisson, negative binomial, and inflated Poisson distributions, have been proposed to fit RNA-seq data7. Commonly used R Bioconductor packages that fit RNA-seq data using count distributions include edgeR9, DESeq10, and DESeq211. These methods could borrow information across genes to increase the power of the tests for detecting genes differentially expressed between two conditions .RNA-seq data are usually represented by a matrix of counts showing the expression levels of mRNAs (rows) for a set of samples (columns) after processes such as adapter remove step, alignment step, and quantification step. For each sample, millions of reads can be measured by the RNA-seq technique12. Moreover, there are much fewer analytic tools for count distributions than there are for normal distributions in statistical analysis. Law et al.12 proposed the voom transformation to transform the count distribution to a distribution close to the normal distribution in RNA-seq data analysis and demonstrated that using limma13 with the voom-transformed count data performed comparable to count-based RNA-seq analysis methods, such as edgeR9, DESeq10, baySeq14 and DSS15.The distributions of counts are not as statistical tractable as normal distributionsvoom transformation is a sample-specific transformation, defined as log-counts per million (log-cpm):gi is the count of the g-th mRNA transcript for the i-th sample, Ri is the total counts (Ri = r1i + r2i+ … +rGi) for the i-th sample, g = 1, …, G, i = 1, …, n, G is the number of mRNA transcripts, and n is the number of samples.The voom transformation is to make the empirical distribution of transformed RNA-seq data closer to a normal distribution so that the moderate t tests (limma) could be used. However, it is not always possible to transform count data to have a distribution closer to a normal distribution in real data analysis16. For example, for the SEQC data that was analyzed in real data analyses part of12, the empirical distribution  of the voom transformed data is still far from a normal distribution  are based on root transformations. The other 4 proposed transformations  are based on log transformations. The transformations r, rv, l, and lv do the same transformation to each read count, while the transformations r2, rv2, l2, and lv2 are sample-specific . To evaluate the effects of sample size on the performances of limma with data transformed by each of the 8 proposed data transformations and to compare them with the performance of limma with data transformed by the voom transformation, we performed eight simulation studies based on the simulation scheme in12. Since real datasets seldom have equal library size, we only consider to simulate datasets with un-equal library sizes in our simulation studies.In this article, we proposed 8 data transformation methods to improve the Wilcoxon rank sum test (denoted it as Wilcoxon) for each gene transcript based on the untransformed counts. We then adjusted p-values to control false discovery rate < 0.05.In addition, we would like to evaluate if using non-parametric approaches would have better performance than using parametric approaches in analyzing RNA-seq data, the distribution of which is non-normal. Specifically, we applied the l, l2, and r2 transformations performed better than limma based on datasets from the voom transformation in term of accuracy, false discovery rate (FDR), and false discovery rate (FNR). For datasets with moderate sample sizes , limma with the rv and rv2 transformations performed similarly to limma with the voom transformation. The accuracy is measured by the Jaccard index, which is defined as the ratio d/(b + c + d), where d is the number of truly differentially expressed (DE) gene transcripts having been detected as DE gene transcripts, b is the number of truly DE gene transcripts having been detected as non-DE gene transcripts, c is the number of truly non-DE gene transcripts having been detected as DE gene transcripts. The Jaccard index is a better measurement than commonly-used accuracy measurement acc = (a + d)/(a + b + c + d), where a is the number of truly non-DE gene transcripts having detected as non-DE gene transcripts, for datasets with highly imbalanced proportions  of truly DE gene transcripts and truly non-DE gene transcripts. When a is much larger than d, then a will dominate in the calculation of acc. Hence, acc tends to close to one, which is misleading. FDR is the proportion of truly non-DE gene transcripts among detected DE gene transcripts. FNR is the proportion of detected non-DE gene transcripts among truly DE gene transcripts. A gene transcript is detected as a DE gene transcript if its FDR adjusted p-value is <0.05; it is detected as a non-DE gene transcript if its FDR adjusted p-value ≥0.05.Figure Wilcoxon test without data transformation performed best in terms of accuracy when the sample size was large and samples had unequal library sizes in our simulation studies, in which we used Law et al.’s12 simulation setting. Wilcoxon also had the lowest FDR and low FNR. However, Wilcoxon had significant higher FNR than limma with voom, rv, lv, rv2, or lv2, indicating that Wilcoxon had lower power than limma with voom, rv, lv, rv2, or lv2.Surprisingly, the Wilcoxon test without data transformation can be used to detect differentially expressed gene transcripts for RNA-seq data when sample size is large and sample library sizes are unequal. If data transformation is preferred, then limma with voom, l, r2, or lv2 can be used in this scenario.Overall, the η are similar and are ranged from around 5.7 to 9.5. For the 4 proposed log transformations, the medians of the estimated δ are similar and are ranged from around 0.08 to 0.32.Supplementary Fig. r, l, rv, lv, rv2, and lv2 had the mean-median difference close to zero. However, r2 and l2 had much larger mean-median difference than zero. This is as what we expected since r2 and l2 aims to minimize the sum of sample-specific squared difference between sample mean and sample median, not to minimize the squared difference between sample mean and sample median of the pooled data. While l2 had smaller difference than voom, the r2 transformation had much larger mean-median difference than voom. Supplementary Fig. r2 and l2 transformations had much smaller mean-median difference than r and l transformations.As we mentioned in the Background section, we aimed to transform the RNA-seq count data so that the sample mean would be representative of the center of the transformed empirical distribution. Hence, the difference between sample mean and sample median after transformation is an important judging criterion of RNA-seq transformation methods. We used the difference between sample mean and sample median based on the pooled expression levels of all gene transcripts and all samples to check if sample mean is close to the sample median Fig. . The sma17. We applied the Wilcoxon test, limma after the TMM scale normalization18, quantile normalization, and the voom transformation (We still denoted the method as limma with voom), and limma after the 8 proposed transformations to the SEQC RNA-seq dataset. The information about which gene transcripts are truly differentially expressed between two groups were determined based on qRT-PCR  experimental data. Figure r, l, rv, r2, l2, rv2 had higher accuracy than limma with voom and showed that limma with lv and lv2 had equal accuracies to limma with voom. We noticed that Wilcoxon, a non-parametric method, failed to detect any true positives, although Wilcoxon had slighly higher accuracy to limma with voom. Since we know the true DE status of each gene transcript in SEQC dataset, we calculated the FDR and FNR values for the 10 methods . Figure r, l, r2, and l2 had higher median validation proportion than Wilcoxon and limma with other transformations in two analyses. Note that GSE95640 dataset is from a paired design. There are 191 subjects. Each subject has two observations (after 8 weeks and after 6 months). Within a subject, the two observations are dependent. However, GSE95640 dataset does not provide subject id info. Hence, we ignored the within-subject correlations in this real data analysis. As a consequence, less numbers of DE gene transcripts would be detected than the analyses in which subject id info is known and statistical tests for paired data are applied.GSE95640 is an RNA-seq dataset from a study investigating adipose tissue during low-caloric diet (LCD) that has relatively large sample size (n = 382) with 53343 gene transcripts. We used all 382 samples  and 191 samples from 6-month after LCD) and conducted 100 random partitions of the 382 samples. In each random partition, we randomly split the 382 samples into roughly 2 equal sets: discovery set and validation set. We then calculated the proportion of validated DE gene transcripts for each of the 10 methods. Figure l and l2 had slightly higher median proportion of validation than voom.GSE95587 is an RNA-seq dataset from a study investigating Alzheimer’s disease that has relatively large sample size (n = 117). We used all 117 samples  and conducted 100 random partitions of the 117 samples. In each random partition, we randomly split the 117 samples into roughly 2 equal sets: discovery set and validation set. We then calculated the proportion of validated DE gene transcripts for each of the 10 methods. Figure voom transformation for RNA-seq data analysis. The simulation results showed that for data sets with small  or large sample size  limma based on data from the l, l2, and r2 transformations performed better than limma based on data from the voom transformation in term of accuracy, FDR, and FNR. For datasets with moderate sample size , limma with the rv and rv2 transformations performed similarly to limma with the voom transformation.In this article, we proposed 8 new RNA-seq data transformations to improve the Having sample mean close to sample median for pooled data could not guarantee that for each gene transcript, the sample mean is close to the sample median for cases and for controls, respectively. Also, the empirical distribution of the pooled data after data transformation might still be skewed even if the sample mean is very close to sample median. Hence, robust linear regression might improve the performance of limma after data transformation.voom transformation was proposed by12 and has been implemented in the limma package. Law et al.12 focused on small sample size (nCases = nControls = 3) because RNA-seq data were expensive to obtain at that time. Since then the cost of RNA sequencing become lower and lower. Hence, we evaluated the performances of limma with voom and limma with the 8 new transformations in scenarios where sample sizes are relatively large (nCases = nControls = 100). We also applied the Wilcoxon test to the raw count data to check if data transformation could perform better than the non-parametric test. Interestingly, the Wilcoxon test without data transformation performed better than the limma analysis based on the 9 data transformations in simulation studies when sample sizes are not too small and sample library sizes are unequal. The analysis of the GSE95640 dataset also demonstrated this finding. Further investigation is warranted.The limma with count-based methods, such as edgeR and DESeq since Law et al.12 did the comparison and showed the good performance of the data transformation approach. However, Law et al.12 did the comparisons based on datasets with small sample sizes. Moreover, new count-based methods, such as DESeq2, have been proposed since 2014. Hence, it would be a future research to compare the data transformation approach with all available count-based methods using datasets with large sample sizes .In this article, we did not compare 19 and sleuth20 since this article focuses on comparing count transformation methods in RNAseq analysis, while NOIseq and sleuth provide methods for differential expression analysis. NOIseq R package provides useful tools for quantifying gene expression, assessing the quality of the expression data, choosing appropriate normalization or filtering methods according to the biases detected, performing non-parametric differential expression analysis, and visualizing the results. Sleuth utilizes kallisto quantification21 and bootstrapping and response error linear modeling to detect differentially expressed genes. It would be an interesting future research topic to compare NOIseq, Sleuth, and limma combined with different count transformation methods.We also did not compare the two useful RNAseq analysis tools: NOIseqvoom in most scenarios , but could not beat limma with rv when sample size is moderate . Another limitation of our study is that in our real data analyses, no independent cohorts are available to do validation. However, we did 100 times of random splits. In each split, we had discovery set and validation set. In future, we will do validation studies when independent validation sets are available. The third limitation of this study is that the 8 proposed data transformations aim to make the sample mean closer to the distribution center after data transformation. However, the transformed distribution might not be close to a normal distribution. Hence, robust linear regression models are needed since ordinary linear regression requires the normality assumption. Future research is warranted on this subject as well.We observed that none of the 8 transformations could dominate each other, although they performed better than ios Fig. . For exaet al. (2014) investigated this scenario; and (2) pilot studies usually have small sample sizes. We also considered the scenario with large sample size (nCases = nControls = 100), which are rare in real applications due to the high expenses to obtain RNA-seq data. The main reasons why we consider this scenario are (1) it would be interesting to know the performances of different methods for datasets with large sample size; and (2) in some real application, sample sizes are large, e.g., GSE95640 (nCases = nControls = 191).In the simulation studies, we considered the scenario with small sample size (nCases = nControls = 3), which could not have adequate power. The main reasons why we consider this scenario are (1) Law We observed from both simulation and real data analyses that when sample size is moderate , all the 8 proposed data transformation methods could not beat voom transformation. Further investigation is warranted.l, l2, and r2 transformations performed better than limma with the voom transformation for data with small (nCases = nControls = 3) or large sample size (nCases = nControls = 100). For moderate sample size (nCases = nControls = 30 or 50), limma with the rv and rv2 transformation performed better than limma with the voom transformation. We hope these novel data transformations could provide investigators more powerful differentially expression analysis using RNA-seq data.In simulation and real data studies, limma with the 22: 4 root transformations  and 4 log transformations . The following two properties of the root transformation motivate us to use root transformations: (1) root transformation of zero exists; (2) root transformation could stabilize the variance of count data.We proposed 8 new data transformations based on the Box-Cox transformationr transformation (root transformation) is defined as:xgi is the count of the g-th gene transcript for the i-th sample. The optimal value for the parameter η is to minimize the difference between the sample mean and the sample median of the pooled data:ygi, i = 1, …, n, g = 1, …, G. That is, for a given η, we perform the r transformation to each count xgi, i = 1, …, n, g = 1, …, G. We then can obtain the squared difference η having minimum squared difference.The rv transformation (root and voom transformation) is defined as:The η is to minimize the squared difference between the sample mean and the sample median of the pooled data. That is, for a given η, we perform the rv transformation to each normalized count tgi, i = 1, …, n, g = 1, …, G. We then can obtain the squared difference ygi, i = 1, …, n, g = 1, …, G. Finally, we choose the value of η having minimum squared difference.That is, we do the root transformation for the sample-specific counts per million. The optimal value for the parameter r2 transformation (root transformation minimizing sum of sample-specific squared difference) has the same form as the r transformation:The η is different from the r and rv transformations. The optimal value for the parameter η is to minimize the sum of the squared difference between the sample mean and the sample median across n samples:i-th sample and i-th sample.However, the criterion to estimate the optimal value of rv2 transformation (root and voom transformation minimizing sum of sample-specific squared difference) is a combination of the rv transformation and the r2 transformation, defined as:tgi is the sample-specific counts per million. The optimal value for the parameter η is to minimize the sum of the squared difference between the sample mean and the sample median across n samples.The l transformation (log transformation) is defined as:The δ is to minimize the squared difference between the sample mean and the sample median of the pooled data:ygi, i = 1, …, n, g = 1, …, G.The optimal value for the parameter lv transformation (log and voom transformation) is defined as:tgi is the sample-specific counts per million. The optimal value for the parameter δ is to minimize the squared difference between the sample mean and the sample median of the pooled data.The l2 transformation (log transformation minimizing sum of sample-specific squared difference) has the same form as the l transformation:The δ is different from the l and lv transformations. The optimal value for the parameter δ is to minimize the sum of the squared difference between the sample mean and the sample median across n samples:i-th sample and i-th sample.However, the criterion to estimate the optimal value of lv2 transformation (log and voom transformation minimizing sum of sample-specific squared difference) is a combination of the lv transformation and the l2 transformation, defined as:tgi is the sample-specific counts per million. The optimal value for the parameter δ is to minimize the sum of the squared difference between the sample mean and the sample median across n samples.The nCases cases and nControls controls. An inverse chi-square distribution with 40 degrees of freedom was used to generate a modest amount of gene-wise biological variation12. We set the number of cases equal to the number of controls .In each simulation study, we generated 100 datasets. Each dataset contains 10,000 genes, among which 200 genes are differentially expressed between limma to detect differentially expressed genes. We also compared the results based on transformed data with the results of the Wilcoxon rank sum test based on the original counts.After data transformation, we used Bioconductor package et al.’s12 simulation settings to generate the RNA-seq counts of 10,000 genes for samples with un-equal library sizes. In our simulation studies, we evaluated the effects of sample size on the performances of the 10 methods  in detecting differentially expressed genes. We tried four different sample sizes  with un-equal library size, respectively.Since almost all real RNAseq datasets have unequal library sizes, we applied Law d/(b + c + d), where d is the number of truly differentially expressed (DE) genes having been detected as DE genes, b is the number of truly DE genes having been detected as non-DE genes, c is the number of truly non-DE genes having been detected as DE genes. FNR is the percentage of detected non-differentially expressed (non-DE) genes among truly differentially expressed (DE) genes. FDR is the percentage of truly non-DE genes among detected DE genes. Large accuracy and small FNR, FDR, and DIFF indicate good performance.The criteria to evaluate performance are accuracy (Jaccard index), false negative rate (FNR), false discovery rate (FDR), and the difference (DIFF) between the sample mean and the sample median of the pooled expression levels for all samples and all genes. The accuracy is measured by the Jaccard index, which is defined as the ratio 23. This project provided 6 RNA samples, each sample has 4 replicates, samples A and B were obtained from two well-characterized reference human RNA samples UHR  and HBR (Human Brain Reference RNA). A small amount of Ambion ERCC  Spike-in Mix was added into both Sample A and Sample B. Samples C and D were constructed by mixing Samples A and B to known ratios, 3:1 and 1:3, respectively. The pure ERCC Spike-in Mix 1 and 2 were used as Samples E and F. Gene expression levels of Samples A, B, C and D were analyzed by using TaqMan RT-PCR technology.Sequencing Quality Control (SEQC) is the third phase of the MAQC project (MAQC-III), aimed at assessing the technical performance of next-generation sequencing platforms by generating benchmark datasets with reference sampleshttps://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE56457) provides details about the qRT-PCR data for the SEQC project. We regarded the expression levels of these genes measured by qRT-PCR as the true expression levels. If a gene has mean log2 fold-change (LFC) greater than 2 between two RNA samples in GSE56457, we claimed it as a truly differentially expressed gene. If a gene has mean LFC less than 0.004, we claimed it as a truly non-differentially expressed gene24. Based on this criterion, there were 390 DE genes and 457 non-DE genes. We evaluated the performance of the 9 data transformations using SEQC data based on these 847 genes. We applied limma to detect differentially expressed genes after data transformation. We also applied the Wilcoxon test to detect DE genes based on the raw count data. A gene was estimated as a DE gene if it had FDR-adjusted p-value < 0.05. We then calculated the proportion of agreement  between the true gene significance of the 847 genes and the estimated gene significance.Our first analysis is based on Samples A (UHR) and B (HBR). The dataset GSE56457 on Gene Expression Omnibus (GEO) website  are pre-formulated sets of 92 poly adenylated genes from the ERCC plasmid reference library, three quarters of the genes were truly DE and the remaining quarter were not. The genes are traceable through the manufacturing process to the NIST plasmid reference material. This dataset provided concentrations of the two mixes, the log2 fold change of concentration can be used for determining if a gene is DE. The analysis procedure of spike-in data is consistent with SEQC data. We calculated the accuracy to compare the transformation methods performance.We downloaded ERCC data from https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE95640) is an RNA-seq dataset to evaluate transcriptome alterations in adipose tissue (AT) during LCD based on 191 obese, non-diabetic patients. This RNA-seq dataset is sequenced on Illumina HiSeq 2000 platform. We downloaded the RNA-seq raw data and annotations from Gene Expression Omnibus . The dataset contains 382 samples with 53343 gene transcripts, 191 of which are from the transcriptome after 8-week LCD  (CID1) and 191 of which are from the transcriptome 6-month after LCD (CID2). We randomly split the 392 samples into two roughly equal parts: a discovery set and a validation set. We then applied limma after data transformations to the discovery set and the validation set to detect differentially expressed (DE) genes. For the discovery set, we claimed a gene is DE if its FDR-adjusted p-value < 0.05. For the validation set, we claimed a gene is validated DE if it had a raw p-value < 0.05 in the validation set and it had FDR-adjusted p-value < 0.05 in the discovery set. We repeated the above split-validation procedure 100 times. For each of the 10 methods, we calculated the proportion of the validated DE genes among the DE genes detected in the discovery set. The higher the proportion is, the better performance, the method is.GSE95640 (https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE95587) is an RNA-seq dataset obtained from fusiform gyrus tissue sections of autopsy-confirmed Alzheimer’s cases and neurologically age-matched normal controls. The matching information was not provided in GSE95587. We downloaded the RNA-seq raw data and annotations from Gene Expression Omnibus . The dataset contains 117 samples, 84 of which are from Alzheimer’s cases (ADs) and 33 of which are controls (CONs). We randomly split the 117 samples into two roughly equal parts: a discovery set and a validation set. The discovery set has 42 ADs and 17 CONs. The validation set has 42 ADs and 16 CONs. We then applied limma after data transformations to the discovery set and the validation set to detect differentially expressed (DE) genes. For the discovery set, we claimed a gene is DE if its FDR-adjusted p-value < 0.05. For the validation set, we claimed a gene is validated DE if it had a raw p-value < 0.05 in the validation set and it had FDR-adjusted p-value < 0.05 in the discovery set. We repeated the above split-validation procedure 100 times. For each of the 10 methods, we calculated the proportion of the validated DE genes among the DE genes detected in the discovery set. The higher the proportion is, the better performance, the method is.GSE95587 (voom transformation in all real data analyses in this article, we followed Law et al.12 by first applying TMM scale-normalization18 and quantile normalization before applying for the voom transformation.For the Supplementary Document


We present DeepImpute, a deep neural network-based imputation algorithm that uses dropout layers and loss functions to learn patterns in the data, allowing for accurate imputation. Overall, DeepImpute yields better accuracy than other six publicly available scRNA-seq imputation methods on experimental data, as measured by the mean squared error or Pearson’s correlation coefficient. DeepImpute is an accurate, fast, and scalable imputation tool that is suited to handle the ever-increasing volume of scRNA-seq data, and is freely available at  The RNA sequencing technologies keep evolving and offering new insights to understand biological systems. In particular, single-cell RNA sequencing (scRNA-seq) represents a major breakthrough in this field. It brings a new dimension to RNA-seq studies by zooming in to the single-cell level. Currently, various scRNA-seq platforms are available such as Fluidigm- and Drop-Seq-based methods. While Drop-Seq can process thousands of cells in a single run, Fluidigm generally processes fewer cells but with a higher coverage. In particular, 10X Genomics’ platform is gaining popularity in the scRNA-seq community due to its high yield and low cost per cell. Consequently, an increasing amount of studies have taken advantage of these technologies to discover new cell types , new marDespite these advantages, scRNA-seq data are very noisy and incomplete –12 due tCurrently, several imputation algorithms have been proposed, based on different principles and models. MAGIC  focuses Here, we present a novel algorithm, DeepImpute, as the next generation imputation method for scRNA-seq data. DeepImpute is short for “Deep neural network Imputation”. As reflected by the name, it belongs to the class of deep neural-network models –29. Recedropout layer with 20% dropout rate (note: not the dropout rate in the single cell data matrix) of neurons which avoid overfitting . scImpute and VIPER give the two highest MSEs at the cell level, whereas VIPER consistently has the highest MSE at the gene level  from GSE102827, composed of 48,267 annotated primary visual cortex cells from mice and which had 33 prior cell type labels . Using tGiven lack of absolute truth of class labels in experimental data, we next generated a simulation data using Splatter. This simulation dataset (sim) is composed of 4000 genes and 2000 cells, which are split into 5 cell types (proportions: 5%/5%/10%/20%/20%/40%). DeepImpute successfully separates cell types on the simulation, closely followed by scImpute . We ran the imputations three times and measured the runtime (for both training and testing steps) and memory load on an 8-core machine with 30 GB of memory. DeepImpute, DCA, and MAGIC outperformed the other four packages on speed Fig. a, and DeUnlike the other imputation methods, DeepImpute first fits a predictive model and then performs imputation separately. The model fitting step uses most of the computational resources and time, while the prediction step is very fast. We then asked the question what is the minimal fraction of the dataset needed to train DeepImpute and obtain efficient imputation without extensive training time. Hence, we used the neuron9k dataset and evaluated the effect of different subsampling fraction  in the training phase on the imputation prediction phase. We randomly picked a subset of the samples for the training step and computed the accuracy metrics  on the whole dataset, with 10 repetitions under each condition. Model performance improvement begins to slow down at around 40% of the cells  on another cell type (or cell state) decently. This pre-training process is very valuable in some cases, such as when the number of cells in the dataset is too small to construct a high-quality model. Pre-training can also largely reduce the overall computation time, since DeepImpute spends most of the time on training the samples. Thus, it is also a good strategy when the new, large dataset is very similar to the dataset used in pre-training.An enduring imputation method has to adapt to the ever-increasing volume of scRNA-seq data. DeepImpute is such a method, implemented in deep learning framework where new solutions for speed improvements keep appearing. One example is the development of neural network-specific hardware , which are deemed interesting for downstream analyses , 48. Fordropout layer (note: different from dropout data in scRNA-Seq), and an output layer of S target genes. A gene is selected to the input layer, if it satisfies these conditions: (1) it is not one of the target genes and (2) it has top 5 ranked Pearson’s correlation coefficient with a target gene. The dropout layer is included after the hidden layer, as a common strategy to prevent overfitting [dropout rate as 20%, after experimenting the dropout rates from 0 to 90%  activation function, a rfitting . We optiThe training starts by splitting the cells between a training (95%) and a test set (5%). The test set is used at each epoch to measure overfitting. We use a weighted mean squared error (MSE) loss function that gives higher weights to genes with higher expression values. This emphasizes accuracy on high confidence values and avoids over penalizing genes with extremely low values . For a given cell c, the loss is calculated as follows:Yi is the value of gene i for cell c and where ptimizer , since iptimizer . The traOnce the network weights are properly trained, we impute the data by filling zeros in the original matrix with the imputed values.c1... cn has a dropout probability p1... pn given by the logistic function previously fitted. The masked cells are sampled from a multinomial distribution with parameters , where qi = pi/∑ipi  are the normalized probability such that ∑iqi = 1.To evaluate the accuracy of imputation, we apply a random mask to the real single-cell datasets. The masking probability function is estimated in a similar fashion as in splatter . For eaccj and between genes gi. Speed and memory comparison: we run comparisons on a dedicated 8-core, 30-GB RAM, 100-GB HDD, Intel Skylake machine running Debian 9.4. We record process memory usage at 60-s intervals. For testing data, we use the Mouse1M dataset since it has the largest number of single cells  is the probability of cell i belonging to both cluster C and K.It is an entropy-based metric that calculates the shared entropy between two clustering assignments, and is adjusted for chance. The mutual information is calculated by It is the ratio of all cell pairs that are either correctly assigned together or correctly not assigned together, among all possible pairs. It is also adjusted for chance.It is a metric derived from the true positives (TP), false positives (FP), and false negatives (FN) as follows: It is a clustering metric derived by comparing the mean intra-cluster distance and the mean inter-cluster distance.t test of one group against the rest groups. We used Benjamini-Hochberg correction for multiple hypothesis testing to obtain adjusted p value . Since each method has generated different differentially expressed genes, we extracted the top 500 differentially expressed genes for each group and pooled the differentially expressed genes for all of the groups. Using 1-pvaladj as the DE calling probability and the true differentially expressed genes (by Splatter) as the truth measure, we calculated the area under the curve (AUC) for the ROC curve for each method using the scikit-learn python package.We perform the differential expression analysis using the scanpy package on the simulation as the groups are pre-defined. For each method, we extracted the differentially expressed genes for each cell group by performing a We obtain a Drop-Seq dataset GSE99330) and its RNA FISH dataset from a melanoma cell line, as described by Torre et al. . The sum30 and itFor GINI coefficient calculation, we first normalize the cells in each dataset using a housekeeping gene -based factor, as done by others . We remoThen, we compute GINI coefficient, as done in SAVER . For disX is the input matrix of gene expression from RNA-FISH or Drop-Seq, Cov is the covariance, and Var is the variance.where Additional file 1. Explanatory figures for DeepImpute’s preprocessing and for the masking experiment.Additional file 2. Dropout and activation function optimization experiments for DeepImpute’s architecture.Additional file 3. Summary table of the dataset used in this paper.


The similarities lead to dataset-to-dataset networks visualizing the landscape of a large portion of biological data. Potentially interesting similarities connecting studies of different diseases are assembled in a disease-to-disease network. Exploring it, we discover numerous non-trivial connections between Alzheimer’s disease and schizophrenia, asthma and psoriasis, or liver cancer and obesity, to name a few. We then present a method that identifies the molecular quantities and pathways that contribute the most to the identified similarities and could point to novel drug targets or provide biological insights. The proposed method acts as a “statistical telescope” providing a global view of the constellation of biological data; readers can peek through it at:  Different phenotypes  may be distinct on a macroscopic level, however, they may share several common biological mechanisms on a molecular level. The molecular similarities should be reflected as statistical similarities in corresponding -omics data measurements. A method is presented to identify these statistical similarities. It is then applied to hundreds of public -omics datasets stemming from different studies, creating a network of diseases  whose statistical molecular patterns are similar. We discover numerous non-trivial connections on a molecular level between Alzheimer's disease and schizophrenia, asthma and psoriasis, liver cancer and obesity, to name a few. The quantities and pathways that contribute the most to the identified similarities are also reported. They could point to novel drug targets or biological insights. For example, as of October 2019, the NCBI Gene Expression Omnibus (GEO)1 contains 3,263,365 microarray and RNA-Seq profiles, grouped into 119,386 data series. Each dataset studies a specific biological question, regarding a disease, a treatment, or a phenotype. Examples include finding the gene expression differences between malignant and benign breast tissue or creating a diagnostic model between primary and metastatic lung cancer tumors. Data analysis methods then typically focus on individually analyzing each dataset, like a “statistical microscope”. However, the question arises: how do the measurements from these studies compare against each other, what are their relations, and what is the collective, emerging picture and biological intuition they provide? Can we construct and look through a “statistical telescope” instead? Could it be that different diseases, treatments, other experimental or sampling conditions induce similar biological molecular patterns pointing to common pathophysiological pathways? Their identification could accelerate the deeper understanding of human pathology and the exploitation of clinical study results.Public biological data repositories currently hold tens of thousands of . To account for shared profiles, in this work, we remove all datasets with at least one shared profile. However, when the method was applied, unexpected similarities between studies of different tissue or related to different pathologies were discovered, possibly attributed to a common etiology in their underlying biology. To enable expert inspection, we arranged the similarities into dataset-to-dataset networks, one for each measuring platform, where edges connect statistically significantly similar studies. The networks visualize the landscape of a sizable portion of the biological dataome and can be interactively explored online. To focus attention on the potentially most interesting similarities, we assemble the ones relating studies pertaining to different pathologies or phenotypes into a new type of network: a disease-to-disease network. Once an interesting dataset similarity is identified, it is natural to inquire the reason why. A second method is proposed to identify the molecular quantities  and corresponding enriched pathways, which contribute the most to the identified similarity, thus providing biological intuition regarding the common biological mechanisms involved. The overall approach is depicted in Fig. A distributional similarity, referred simply as “similarity” hereafter, implies intuitively that molecular quantities measured in the two datasets are inter-correlated in a similar way, i.e., the covariance matrices are similar. We propose a generally applicable method, able to robustly estimate distributional distances, even when the dimensionality ranges into hundreds of thousands of observed molecular quantities  and sample sizes as low as 40. It is emphasized that we compute the similarity of two datasets based only upon the statistical properties of the molecular measurements and ignoring any textual information. Similarities may arise due to several factors, such as concerning the same disease or treatment, measuring the same type of tissue, or employing sample analysis protocols and equipment that induce similar systematic batch effects; similarities may even arise between studies that share the same molecular profiles, a phenomenon that is surprisingly prevalent2 Their descriptive presentation is shown in Supplementary Figs We used all the datasets from six measurement platforms that comply with certain criteria for sample size; datasets that share profiles with other datasets have been removed from consideration SM1.1). Datasets include both micro-array technologies on transcriptomes and DNA methylomes, as well as RNA-seq technology; they also include measurements on subject tissues and cell lines. Specifically, the following platforms were included: (1) Homo Sapiens, gene expression, Affymetrix Human Genome U133 Plus 2.0 Array - (GPL570), (2) Homo Sapiens, gene expression, Affymetrix Human Genome U133A Array - (GPL96), (3) Homo Sapiens, gene expression, Affymetrix Human Gene 1.0 ST Array - (GPL6244), (4) Homo Sapiens, gene expression, Illumina HiSeq 2000 - (GPL11154), Homo Sapiens, DNA methylation, Illumina HumanMethylation450 BeadChip - (GPL13534), and Mus Musculus, gene expression, Affymetrix Mouse Genome 430 2.0 Array - (GPL1261). Overall 103,088 Homo Sapiens and Mus Musculus samples were employed in the subsequent analyses and results, grouped in 978 datasets and spanning more than 500 different diseases and phenotypes, as revealed by automated text analysis..1. Datasc-SKL as described in methods. The results are visualized as networks (graphs), where each node corresponds to a dataset (study) and each edge corresponds to a statistically significant c-SKL. As only datasets within the same platform can be compared, i.e., datasets measuring the same sets of variables, a different network is constructed for each platform. The network for GPL570 and all statistically significant edges is shown in Fig. c-SKL with all the other human datasets. There are two communities of breast cancer shown that are not connected; on closer inspection the largest one contains case-control studies on human subjects, while the other contains treatment-control studies on cell lines, justifying why the c-SKL is high among studies belonging in these two different groups. It is well known that significant differences between cell lines and human tumors exist.3For each pair of datasets within the same platform, we computed the It is also interesting to study the evolution of these networks, e.g., using the on-line app provided, as one varies the number of edges to include. In general, as we increase the number of edges in the network, similar communities tend to connect and merge together. For example, if we go beyond the 300 edges shown in Fig. c-SKL metric cannot distinguish among the sources of observed similarities, yet, it can point-out unexpected similarities. For example, the similarity networks (SM 5) demonstrate that tissue is not the only reason of similarity, as we notice significant similarities in datasets measuring different tissues . At the same time, not all datasets providing data from the same tissue interconnect, e.g., datasets of blood tissue measurements do not fully interconnect. To systematically extricate non-trivial and potentially interesting similarities, we focused on the similarities that connect macroscopically different diseases. We construct a new type of network where each node corresponds to a different disease, or phenotype in a more general sense, a disease-to-disease network: two nodes A and B in this network are connected by an edge , whenever a dataset of disease A is found similar to a dataset of disease B. We construct one disease-to-disease network for each different species (Homo Sapiens and Mus Musculus) and each type of -omics (transcriptomics and methylomics) by assembling all edges from all related platforms. Figure  in Fig. A and one from B are found similar. Thus, edges of high weight indicate a molecular similarity between different diseases found multiple times across studies and platforms and that could be attributed to an underlying common biological mechanism. Figure Even though the proposed 4 common elements shared with pathways identified in breast cancer.5 Also, a connection between breast cancer and thyroid cancer appears (weight = 7), a relationship that has attracted substantial attention in the medical research community.6 Alzheimer’s disease and schizophrenia, a similarity found in eight different data-pairs, is known to share psychiatric symptoms that suggest some common cerebral pathophysiology.7 In,8 a network that spatially recapitulates the pattern of brain abnormalities observed in both schizophrenia and Alzheimer’s disease was revealed by a data driven analysis. Psoriasis and asthma (weight = 6), both chronic immune-mediated inflammatory diseases, is another example of a rather unexpected similarity, which has attracted much attention recently.9 Melanoma and breast cancer, an already known similarity12 also appears frequently (weight = 5). Other interesting, frequent similarities that can be further supported by evidence found in the literature, include the links between acute myeloid leukemia and viral infectious diseases (weight = 7),13 asthma and malaria (weight = 5),14 asthma and hepatitis C (weight = 6),15 systemic lupus erythematosus and viral infectious disease (weight = 6),16 or Parkinson’s disease and Alzheimer’s disease .17 It is also interesting to note that in the Mus Musculus transcriptome network the most frequent similarity was found between obesity and liver cancer (weight = 6). See ref. 18 for a review on the existing evidence on the links between obesity and liver cancer incidence and survival and19 for a pooled analysis from all U.S. based studies in the NCI Cohort Consortium. For a ranked list of all disease associations that have been found at least four times in any of the Homo Sapiens platforms see Table 2 in SM6.Focusing on Fig. c-SKL achieved using the B(k) features (shown in red), obtained by the proposed optimization method that explains c-SKL (see Methods), dominates (obtains a lower c-SKL) the c-SKL achieved by randomly selecting k features (gray color). The graph starts at value k = 100 with a step size of 1000. More examples are shown in SM4.1, demonstrating that the curves are qualitatively similar over the whole range of similarities and platforms. The results clearly show that B(k) selects better explaining features than random guessing. Next, we consider the hypothesis that, for the same disease, the biological mechanisms that best explain similarities between datasets at the transcriptome level, should be common, independently of the measuring technology. As an example, we first identified an acute myeloid leukemia (AML) clique of five datasets measured by GPL570 (microarray gene-expression) and an AML clique of five datasets measured by GPL11154 (RNA-seq) and compared the gene sets that explain their similarity  all 45 pair-wise similarities found within a clique of ten breast-cancer datasets and (b) eight similarities of Alzheimer’s disease to schizophrenia. In both cases, the datasets are gene-expression micro-arrays measured by GPL570. The probe-sets are then mapped to their corresponding gene symbols and an enrichment analysis of KEGG pathways is performed. Figure 20) groups diseases with similar clinical symptoms and phenotypic traits. One of the first attempts to group diseaes with common genetic origin was the work by Goh et al.,21 who created the Diseasome, a visual reference of the genetic links between disorders and disease genes. The human disease network they created, displayed many connections between both individual disorders and disorder classes and proved that studying diseases and phenotypes from a systems approach offers the possibility to discover general patterns and principles of human disease. Since then, several network based approaches to understand the molecular basis of human disease, appeared. Hidalgo et al.,22 for example, exploited disease phenotypes of more than 30 million elderly Americans aged 65 or older, to create a Phenotypic Disease Network (PDN) where nodes represented disease phenotypes, as defined by the ICD9 and edges a significant comorbidity according to the measures they introduced. Later, Žitnik et al.23 found relationships between diseases, some of them undiscovered at that time, by fusing molecular interaction and ontology data from several biological databases, such as Gene Ontology, Disease Ontology, DrugBank and others. Sun et al.24 estimated the similarity score of diseases, by analyzing four publicly available disease-gene association datasets  and demonstrated their similarity measure by identifying diseases associated with diabetes mellitus which were further confirmed bibliographically. Yang et al.25 proposed a more data driven approach, by applying differential co-expression analysis to explore the architecture of disease relationships in terms of dysfunctional regulation mechanism. Their results proved that their approach can be a complement to the disease networks generated from symptoms, disease concepts and biomedical data. Later, Menche et al.26 identified common mechanistic pathways between diseases by exploiting the overlap of disease modules (connected subgraphs formed by the interaction of disease proteins). Recently, Halu et al.27 developed and analyzed the human disease multiplex network by considering genotypic and phenotypic information simultaneously and showed, among other results, that Mendelian disorders predispose individuals to more common, complex diseases. Other approaches that compare biological datasets rely on creating a molecular signature list for each dataset containing the differentially expressed quantities with respect to a target variable (supervised approach). A typical such target variable in case-control studies is the disease status, and in treatment-control studies it is the presence or absence of treatment. Dataset similarities are then computed as similarities between the signature lists.30The traditional disease classification system . The latter is technically difficult to perform automatically and scale up to thousands of datasets. Instead, in this work, we propose a method that is unsupervised and depends only on the omics data distributions. Hence, it can easily be applied to the entire collection of available omics datasets.data-based information retrieval, i.e., retrieving all datasets that are similar to a particular query dataset . Data with similar statistical properties as the query data could be potentially pooled together for meta-analysis or identify similar control samples from totally unrelated studies. It is worthwhile to note that quite recently, Google launched a new service, called Dataset Search, also aiming at dataset retrieval. However, Dataset Search performs retrieval based on the metadata tags, while the proposed method identifies statistically similar datasets independently of any text or tag annotations.The proposed method could also be employed for what we call As most human public -omics datasets are case-control studies, in this work we focused on disease-to-disease similarities; however, in the context of drug design, one could examine treatment-control studies to create treatment-to-treatment networks depicting similarities between expression patterns induced by different compounds. Compounds, tested for different diseases that yet induce similar expression patters for unexpected reasons, are prime candidates for drug repurposing.If one annotates datasets with the tissue measured (or some other datasets characteristic), then it is possible to estimate the principal component corresponding to the tissue and remove it. Recomputing similarities after this removal will highlight similarities due to all other reasons except tissue, further revealing common biological mechanisms among different diseases.Finally, while the proposed method requires two datasets to be measuring the same set of variables (molecular quantities), it is possible to develop methods of comparing distributions over different sets of variables (human genes vs. mice genes), an interesting line of future work that we are actively exploring.DSP and DSQ with rows ranging over the samples and columns over the variables . For -omics data, such as micro-array or RNA-seq data, the number of variables ranges into the tens to hundreds of thousands, while often the sample size is less than a few dozens. To compare such high dimensional, low-sample size empirical distributions, we propose the following methodology: First, each dataset is standardized by subtracting the mean and dividing by the standard deviation of a variable for each measured value. Standardization removes the effect of the mean and the scale of the measurements, which could depend on the sensitivity of the measuring instrument and the concentration of the reagents and may be sensitive to other nuisance factors. This transformation retains the linear correlations among variables. Second, each dataset DSP’s distribution is assumed to be multivariate normal and thus fully determined by its mean (which is transformed to zero) and its symmetric, positive semidefinite covariance matrix ΣP (coinciding with the correlation matrix for standardized data). Given that the number of variables is larger than the number of samples, the sample covariance matrix is rank deficient. A third assumption (which enables the estimation of the rank-deficient covariance matrix) is that ΣP can be expressed as ΣP = PΛPPT + σPI, where Pn×k is an orthonormal matrix, ΛPk×k a diagonal matrix with the ith value in the diagonal denoted as σP a non-negative real. In other words, it is assumed that the data lie in a k << n dimensional subspace, where n is the number of variables, with an isotropic noise component (see SM2.1). The decomposition of ΣP is computed using the standard Principal Component Analysis. Alternatively, one could employ Probabilistic PCA to directly compute the maximum likelihood decomposition of ΣP31 at the expense of higher computational cost. P is the matrix of the first k = cP principal axes of the sample covariance matrix, DSP). Similarly, for dataset DSQ we compute the decomposition ΣQ = QΛQQT + σQI, where Q is the matrix of the first cQ principal axes and ΛQ contains the cQ eigenvalues of the covariance matrix in its diagonal. Based on the above ideas, we develop a measure of (dis)similarity between the distributions of the two datasets DSP and DSQ, called the curated symmetric Kullback–Leibler (c-SKL) divergence, and denoted as c-SKL . This measure depends only on the covariance matrices given the above assumptions. The smaller the c-SKL between a pair of datasets, the more similar the two distributions and their covariance matrices are . The value of cP is determined in such a way that the first term explains just about αP percent of the variance, i.e., αP dictates the level of compression of the data. As αP increases, more principal axes are employed, enlarging the axes subspace and allowing finer differences (dissimilarities) to other datasets to be discovered; at the same time, as αP increases the additional axes entering the equation are less reliably estimated. The parameter αP is the variance-per-variable unexplained by the principal components and distributed to the isotropic noise components. We set it to 1 − αP so that the sum of variances in the decomposition equals the variance of the original dataset. Employing the same value for αP in all c-SKL computations among different pairs of datasets (leading to drop the index P from the notation), makes two c-SKL values comparable as they are all computed using the same compression level. In fact, after experimentation we found that results improve when eigenvalues are scaled so that they sum exactly to α, i.e, DS1 and DS2. We computed c-SKL among each pair of all available datasets, agnostic to the above partitioning. By construction, the sibling datasets come from the same distribution and should be discovered to be closer in c-SKL than with any other dataset. The validation experiment and its results are presented in detail in SM3, where we show that the sibling of a dataset is found closer in terms of c-SKL than any other dataset 95% of times. From the experiment we also determined a reasonable value for α to be 50%, but results are robust with respect to the specific numerical value used. We then estimate the statistical significance of similarities and keep only the ones with a q-value  less than 0.05 using a semi-parametric bootstrap technique. The semi-parametric bootstrap testing procedure is as follows: Let J be the distribution of the pooled samples across all datasets; let P and Q be the distributions of two datasets DSP and DSQ. The alternative hypothesis tested is that the observed c-SKLs between P and Q is smaller (and thus their similarity is larger) than the c-SKLs of P and J, as well as Q and J. In other words, a statistically significant similarity indicates that two datasets are more similar to each other, than with the rest of the samples . We note that, there exist procedures in the literature testing the exact equality of high-dimensional distributions,35 i.e., H0: P = Q or equivalently, H0: s = 0, where s is their c-SKL. However, in the context of this work, the null hypothesis of exact equality of distributions is meaningless and should typically be rejected; studies may be related, but do not necessarily employ exactly the same population under the same experimental and sampling conditions. Instead, we propose to test for a relatively high similarity with respect to the rest of the samples in the pooled collection of all datasets.Equation  is an apk molecular quantities, to which the similarity can be mostly attributed. As proved in SM2.2 (Lemma 2), a low Kullback–Leibler divergence corresponds to a similarity between the corresponding covariance matrices ΣP and ΣQ. The most contributing features correspond to the rows and columns of the covariance matrices, where the matrices agree the most  given two datasets with principal axes in P and Q, and eigenvalues in vectors λP and λQ one needs to solve the following optimization problem:S is a vector of exactly k ones and everything else zero, and diag(S) produces a diagonal matrix with S on the diagonal. The S vector serves as the selector of features in the computation of Eq. . In addition to identifying the indexes of the k best-explaining features B(k), one can instead minimize the equation above to find the top k features that make the distributions most different, denoted by W(k) (standing for Worst).It is not straightforward to explain why two distributions over tens of thousands of dimensions are similar. As an explanation, we propose to report the top mentclass2pt{minimB(k) explains a single dataset-to-dataset similarity; however, one could extend this method to find the features that simultaneously best explain a set of pair-wise dataset similarities, trying to pick the common biological mechanisms that are jointly responsible for all similarities observed in the set. To find the best set explanation among a set of pair-wise similarities, we solve a similar optimization problem maximizing the sum of all pair-wise c-SKLs within the set . There are at least two scenarios where a set explanation could be useful. For example, one may wish to explain all pair-wise similarities among datasets pertaining to the same disease. These datasets may differ slightly in terms of patient population, experimental conditions, and sampling methodology; identifying the genes that explain the patterns shared by all of them, accentuates the common cause for the similarities in the group, presumably those genes involved in the common disease mechanism. A second use of a set explanation is when trying to explain an edge in the disease-to-disease network. Each such edge corresponds to a set of similarities whose set explanation sheds light to the common mechanism involved in making the two different diseases similar.As presented, Further information on research design is available in the A data driven approach revealing disease similarity at the molecular level_SMReporting summary


Logic Learning Machine (LLM) is an innovative method of supervised analysis capable of constructing models based on simple and intelligible rules.In this investigation the performance of LLM in classifying patients with cancer was evaluated using a set of eight publicly available gene expression databases for cancer diagnosis.sROC) analysis and estimated by the area under an sROC curve (sAUC). Its performance was compared in cross validation with that of standard supervised methods, namely: decision tree, artificial neural network, support vector machine (SVM) and k-nearest neighbor classifier.LLM accuracy was assessed by summary ROC curve  and outperformed any other method except SVM.LLM showed an excellent accuracy (LLM is a new powerful tool for the analysis of gene expression data for cancer diagnosis. Simple rules generated by LLM could contribute to a better understanding of cancer biology, potentially addressing therapeutic approaches. Furthermore, the capability of LLM to exploit the complex correlation structure of highly dimensional gene expression data for feature selection tasks and to combine information from clinical features and gene expression for classification purposes was reported in the analysis of both simulated and real data sets [Logic Learning Machine (LLM) is an innovative method of supervised data mining based on an efficient implementation of the Switching Neural Network model. In recent years LLM has been applied to a variety of biomedical settings –6. The aata sets , 6. ThesThe present study is aimed at evaluating the performance of LLM through an extensive analysis of microarray gene expression data of cancer patients in diagnostic settings. A comparison with standard methods of supervised analysis is also provided.Fifty-two datasets were retrieved from the GEO web site. After a careful examination of their content and the related documentation, 44 were excluded because they did not fully comply with the selection criteria, thus leaving eight data sets available for the analyses. The complete list of excluded datasets and the reason for their exclusion are shown in Table GDS4968, GDS4296 and GDS3952) included multiple-class comparisons , while the remaining five databases  included two classes.Table GDS4968 data set included 99 samples and 33,297 gene features of pre-malignant  and malignant (multiple myeloma) bone marrow plasma cells [n = 20), multiple myeloma (n = 41) and smoldering  multiple myeloma (n = 33); data related to five healthy controls were excluded from the analyses due to the insufficient sample size.In more details, the GDS4887 included 40 samples and 54,675 features of hepatocellular carcinoma diagnosed in patients with chronic hepatitis C [GDS4794 included 23 samples of small cell lung cancer and 42 normal tissues [GDS4762 included 37 breast cancer cells and 43 fibroblasts [roblasts , and genGDS4471 included 76 samples of medulloblastoma  from patients aged 3 to 16 years [16 years . The anaGDS4296 included 54,675 features and 174 samples from 9 different cancer tissues [ tissues . Among tGDS3952 included 162 samples from breast cancer patients and a variety of other malignancies and normal tissues [n = 37), ectopic cancers (n = 22), healthy controls (n = 31) and malignant breast cancer (n = 51); other smaller groups were not considered in the present analysis. tissues . CompariGDS3945 included 33,297 features in 42 samples of renal clear cell carcinoma [Finally, arcinoma . CompariGDS3945 data set. In two cases, including GDS4794 and GDS4762, all machine learning methods performed equally well with the maximum attainable accuracy (100%), except for ANN in the later data set. With regard to the other six comparisons, performance of SVM (measured by the Cohen’s kappa index) was the highest in each analysis, except for GDS3952, where kNN showed the best accuracy, and GDS3945, where LLM performed better. However, in the GDS4887 data set ANN and kNN had the same performance of SVM, and in the analysis of GDS4296 also kNN performed equally high.Table sAUC = 0.996 and 0.995, respectively). kNN and DT also showed an excellent accuracy (sAUC = 0.991 and 0.964), respectively, while ANN had a quite lower accuracy (sAUC = 0.904).Figure GDS4887, GDS4794, GDS4762 and GDS3945 data sets, the LLM classifier included very few simple rules, based on only one condition and two features (one for each class at comparison). Conversely (and not surprisingly), more complex rules were needed to classify samples from the three datasets that included multiple classes . Three conditions based on a small set of features were sufficient to reach a satisfactory classification accuracy, except for the analysis of the GDS3952 data set , thus indicating that the classification was partly based on an artifact; covering of the corresponding rule was rather high, i.e., 60%. In the absence of available reliable measures of the concentration of the corresponding mRNA  it is impossible to establish if such an artifact can be attributable either to a defective extraction of the LLM algorithm or to noising data related to the intrinsic variability of the microarray technique. Finally, in the GDS4471 data set four classification rules, including a maximum of two conditions each and 6 different features, were identified to separate 41 samples of the classic form of a childhood brain cancer (medulloblastoma) for a pool of 25 heterogeneous samples of rare variants of the same malignancy.Table LLM is an innovative method of supervised analysis that can identify simple and intelligible rules for classification tasks. In previous investigations its accuracy was comparable to that of most common supervised methods based on black-box algorithms and also outperformed DT, which shares with LLM the capability to generate intelligible rules , 15, 16.In the present investigation LLM showed an excellent performance in the analysis of dataset for cancer diagnosis. Comparisons included different types of malignancies or different classes of cancer and non-cancer cells. Carcinogenesis is characterized by the disruption of several biochemical paths; then, many to several genes are expected to be strongly differently expressed between different tumor types and, even more so, between neoplastic and non-neoplastic tissues . ContrarAqp7 gene in a set of 20 hepatocellular carcinoma samples (Table GDS4887 data set). Aqp7 encodes for a transmembrane channel protein belonging to the aquagliceroporins family, whose function is the regulation of transcellular movements of water and glycerol in many mammalian tissues [An advantage of LLM and DT compared to black-box classifiers is that classification rules can provide useful insights about cancer biology. In some instances, the potential biological role of those genes identified by the simple one-feature LLM rules can be confirmed by evidence from recent studies on independent cohorts of cancer patients. For example, in our study LLM revealed a downregulation of the  tissues . A signi tissues .GDS4794 data set LLM found an overexpression of Cbx3 in 23 small cell lung cancer samples. CBX3 is a member of the heterochromatin protein 1 family that plays a fundamental role in multiple cellular functions, including DNA repair, transcriptional regulation, and telomere function [GDS4794 database belonged to another subtype of lung cancer . Our finding, if confirmed by further independent investigations, could provide some insights to the knowledge of the carcinogenesis of the different subtypes of lung cancer.In the function . CBX3 hafunction  found a GDS4471 data set, LLM found an overexpression of the Efhd2 gene that, combined to a downregulation of Loc100132891, was able to separate most samples of classic medulloblastoma (88.3%) from the other subtypes of the same tumor. Medulloblastoma is a childhood cancer, whose different subtypes can be associated with different origins and patient survival [Efhd2  is not a surprising finding, in that it encodes for a calcium ion binding protein involved in several cellular functions, including neurodegeneration and cancer [Loc100132821 is a long non-coding RNA whose function is, at least to our knowledge, still unknown (http://www.igenebio.com/gene/hs-loc100132891.html). In the two last decades many long non-coding transcripts have been found to play a crucial role in carcinogenesis [Loc100132821 in cells with a high expression of swiprosin-1 could be involved in the carcinogenesis of classic medulloblastoma might be a new finding that deserves further investigations.Composite rules generated by combining different features are more difficult to interpret, but they could shed some light on the complex interaction at the basis of carcinogenesis and tumor progression. In the survival . Differed cancer . Conversogenesis , then thThe present study represents the first analysis comparing LLM with different supervised methods for cancer diagnosis using a set of microarray data including different cancer types. LLM showed an excellent accuracy, but our results should be interpreted at the light of some limitations of this study. First of all, this should be considered as a pilot study comparing classification methods under similar simplified conditions and the proposed method should be applied to larger sets of microarray data, including more difficult clinical context . In particular, the tuning methods have produced a very high classification performance that, in the absence of an independent validation cohort, should be interpreted as an upper limit of the actual expected classification accuracy.In spite of the extensive search in the GEO repository, only eight data sets with a sufficiently high dimension have been identified, then the excellent performance found for LLM should be confirmed by further independent studies. Baseline conditions that can influence the LLM performance are still to be elucidated, and further analyses based on large data sets including both real and ad hoc simulated data are needed, in particular to assess the effect of different sample size, unbalanced vs. balanced design, noisy data, and poorly defined gold standards.Finally, in this study, similarly to some other previous investigations , 2, 16, LLM is an innovative method of supervised analysis that has shown a very high accuracy in classifying patients with different oncological conditions.The LLM classifier is typically based on a small set of simple and intelligible rules that could contribute to a better understanding of carcinogenesis process, hopefully supporting the adoption of new therapeutic approaches.http://www.ncbi.nlm.nih.gov/gds/). Selection criteria were: a) inclusion in the GEO data bank from January 2010 to December 2014; b) presence of at least two classes potentially useful for cancer diagnosis, including at least 20 samples each; c) availability of a scientific paper in English language, published on PubMed, and fully describing the experiment and the related study design.Data were drawn from a set of publicly available databases of gene expression microarrays, stored in the GEO repository bank  of conditions and < consequence> provides a class assignment for the output , 6, 15.LLM produces rules through a three-step process, namely: latticization (binarization), monotone Boolean function reconstruction and rule generation Fig. . In the r generated by LLM [C(r), called the covering, or the false positive fraction E(r). In a binary classification task, depending on the class identified by the rule r, C(r) will correspond to either the sensitivity or the specificity. Let r’ represent the rule obtained from r by removing the condition c from its premise part. A simple measure of the relevance R(c) of that condition is then provided by:A set of quality measures has been defined for any rule d by LLM , 15, sucRv(xj) for each variable xj can be obtained by applying the following equation:k varies on the indices of rules rk that includes a condition ckl on the variable xj.Finally, a measure of relevance Rv(xj) ≤ 10% is used to identify a predictor xj providing a marginal contribution to the accuracy of LLM classifiers, while a rule with C(r)(1 – E(r)) ≤ 10% often covers subjects with anomalous values (possible outliers).As a rule of thumb, the inequality E(r) parameter were selected, ranging from 2.5 to 7.5% (step 0.5%). Parameter tuning for the competing methods will be described in the dedicated paragraphs.Performance of LLM was compared with that of four selected competing methods of supervised learning  in leave-one-out cross-validation (LOOCV). Standard measures of quality were obtained for each analysis and proper comparison techniques were adopted to evaluate the overall performance of each classification method in the whole set of analysis. A parameter tuning procedure was adopted to enhance the performance of each selected method and only the models with the highest accuracy were retained. For instance, for LLM a set of values for the TP of positive samples correctly classified by the resulting model,The number FN of positive samples wrongly classified by the resulting model,The number TN of negative samples correctly classified by the resulting model,The number FP of negative samples wrongly classified by the resulting model.The number Consider a two-class classification problem, where the output can assume two different values identified as positive and negative. Each analysis with any supervised learning method is characterized by four values:sensitivity SE and the specificity SP:the From these four values other quality measures for the analysis can be derived, among which:Youden index Y:the Cohen kappa coefficient K:the odds ratio OR:the OR to exploit its asymptotic Normal distribution [2 of the logarithm log(OR) of the odds ratio is readily obtained through the equation [In the present investigation we have log-transformed the ribution . An asymequation :\documen is null .m multiple outcomes, the definition of the Cohen kappa coefficient can be generalized as follows [In the presence of  follows :\documenij represents the counting of elements in the i row and j column of the confusion matrix andwhere aSPm for m multiple outcomes was simply obtained by selecting a category as the reference and computing the proportion of correctly classified samples inside that category.The specificity OR) and of its related variance σ2 were then retrieved by applying the Mantel Haenszel (MH) method [i = 1 be the index of the reference category in the confusion matrix; the MH estimate of log(OR) is obtained as:2 is obtained through the following equation [The corresponding estimate of log method . Let i =equation :\documenSE for multiple outcomes is obtained by exploiting the relationship between OR, SE and SP reported in eq. (1):Finally, the sensitivity A “natural” reference category for multiple outcomes was adopted, whenever possible, selecting either the group of subjects without any disease, if any, or the class with the  less severe illness. Otherwise, in the case of comparison between groups of severely diseased patients  the reference was arbitrarily defined as the class with the highest number of individuals.K index.For each dataset, comparison between the considered supervised classification methods was based on the N studies was obtained by employing the method of the summary ROC (sROC) curves [sAUC under the sROC curve was adopted to evaluate the quality of any classification technique. A proper model was considered, which is described by the following equation:sOR is the summary odds ratio given by:ORi the odds ratio of the ith study and σi2 the variance of log(ORi).For each classifier a common measure of accuracy across the ) curves . In partsOR) can also be obtained assOR, the related 95% confidence intervals (95%CI) of this estimate are obtained as follows:An estimate of the standard error for log. Summary ROC analysis was performed by Stata for Windows statistical software .All supervised analyses were carried out by using Rulex, a software suite developed and commercialized by Rulex Inc. . Furthermore, the highest impurity by node was let to vary between 0.0 and 0.1 (step 0.01).DT is obtained by a “divide-and-conquer” approach that provides disjoint rules. At each iteration, a new node is added to the DT by choosing the condition that subdivides the training set x is sent to the first layer. The remaining layer receives input from the previous one and the last layer produces the output class to be assigned to x. Weights for each neuron are estimated by suitable optimization techniques and form the set of parameters for the ANN. The Levenberg-Marquardt version of the back propagation algorithm was applied to train the ANN [ANN is a connectionist model formed by the interconnection of simple units (neurons), arranged in layers. Each neuron computes a weighted sum of the inputs applying a proper activation function, which provides the output value that will be propagated to the following layer. The input vector  the ANN . Parametn be the number of pairs  in the training set S, where xj is the input vector and yj the output class for the jth sample. When a new subject described by the input vector x is to be classified, the nearest k samples in S, according to a suitable distance measure, are determined and the class y associated with the majority of the k nearest samples is assigned to x [Let ned to x .x to reduce the effect of biases possibly caused by unbalanced domain intervals for different input variables. Tuning procedure was applied to the number of nearest samples letting the k parameter vary between 1 to 10.In the present investigation the standard Euclidean distance was employed, after having normalized the components of the input vector l of input vectors xj in the training set S, called support vectors, and their corresponding outputs yj ∈ {− 1,1}. The class y for any input vector x is then given by:αj and the offset b are evaluated through a proper training algorithm.SVM is a non-probabilistic binary linear classifier based on the identification of an optimal hyperplane of separation between two classes . Given aK is a kernel function used to perform a non-linear classification by constructing an optimal hyperplane in a high dimensional projected space. A linear kernel function was tested on each dataset. The training algorithm was performed by using the LIBSVM library, which is featured by the Rulex Analytics software. The performance of SVM with linear and RBF kernels was tested. Tuning procedure also included the degree of the kernel function that was let range from 1 to 10.