Single-cell RNA sequencing technologies have enabled us to study tissue heterogeneity at cellular resolution. Fast-developing sequencing platforms like droplet-based sequencing make it feasible to parallel process thousands of single cells effectively. Although a unique molecular identifier (UMI) can remove bias from amplification noise to a certain extent, clustering for such sparse and high-dimensional large-scale discrete data remains intractable and challenging. Most existing deep learning-based clustering methods utilize the mean square error or negative binomial distribution with or without zero inflation to denoise single-cell UMI count data, which may underfit or overfit the gene expression profiles. In addition, neglecting the molecule sampling mechanism and extracting representation by simple linear dimension reduction with a hard clustering algorithm may distort data structure and lead to spurious analytical results. In this paper, we combined the deep autoencoder technique with statistical modeling and developed a novel and effective clustering method, scDMFK, for single-cell transcriptome UMI count data. ScDMFK utilizes multinomial distribution to characterize data structure and draw support from neural network to facilitate model parameter estimation. In the learned low-dimensional latent space, we proposed an adaptive fuzzy k-means algorithm with entropy regularization to perform soft clustering. Various simulation scenarios and the analysis of 10 real datasets have shown that scDMFK outperforms other state-of-the-art methods with respect to data modeling and clustering algorithms. Besides, scDMFK has excellent scalability for large-scale single-cell datasets. In the past decade, high-throughput sequencing technology has been widely used in various fields of biology and medicine, greatly promoting research in related areas , n is cell number, and m is the feature(gene) number. Then i-th cell. Assume that the i-th cell contains ti total mRNA transcripts and that Yij is the underlying mRNA transcripts matrix. Then, when we process and lyse the i-th cell on the sequencing protocol, ti mRNA transcripts are attached by barcodes and UMIs such that they are transformed to cDNA molecules by reverse transcription. After removing PCR duplicates, we generate the ni UMI counts in the i-th cell. Because reverse transcriptase is an inefficient and error-prone enzyme, causing some fraction of the cDNA molecules to be lost, ni is usually much less than ti. Thus, the process of successfully converting mRNA to UMI is actually a random sampling process. We defined pij as the relative abundance of the amount of mRNA expressed by j-th gene shared in total mRNA of i-th cell, namely,We begin with some notations. Suppose single-cell UMI count data matrix is ni ≪ ti and true transcripts counts yij are unknown, we supposed that UMI counts Xij are samples of yij with relative abundances remaining constant; thus, the probability distribution function of Xi =  is multinomial distribution with parameter vector pi =  to be estimated,Considering pi(1 ≤ i ≤ n) corresponding to each cell are not completely free from statistical theory. However, these dependencies cannot be captured by a simple generalized linear model because unknown parameters pij actually fall on a low-dimensional manifold. Therefore, instead of designing a specific non-linear association expression or bayesian priors, like mixture dirichlet distribution, we utilized deep autoencoders to approximate the underlying manifold and learn parameter vectors pi(1 ≤ i ≤ n). Furthermore, owing to frequent dropout events, we could not ignore the impact of this extra noise on parameter estimation. To model the dropout events, we introduced binary random variables Uij, where Uij = 0 represents that the j-th gene drops out in the i-th cell. Letting πij = P(Uij = 1), we haveBecause of non-linear dependencies among genes and complex associations between cells, the parameter vectors ij is positively correlated with true expression level of j-th gene in i-th cell. Assume that Vij is the expected relative expression level of j-th gene in i-th cell. Then, given the dropout phenomenon, pij should be the element-wise product of πij and Vij and then normalized to sum to one.Obviously, low-expressed genes have high probabilities of dropping out, which implies that πij, Vij) make up the parameters to be estimated.In this case,  can be modeled as a logistic regression function of underlying true relative expression level. We selected the exponential function as the activation function of V for its non-negativity. To construct the data reconstruction loss function, we assumed that, given latent variable Z, the samples X are conditionally independent. Therefore, we can naturally took the negative log-likelihood of multinomial distribution as the reconstruction loss, aswhere pij can be calculated by Equation (4). When the proportion of zero counts in the data is not so high, we also considered estimating pij directly; that is to say, we chose the softmax activation function at the last layer of decoder and outputted the estimation of pij directly. For distinguishing, we called the standard model scDMFK and the alternative model D-scDMFK.where the value of Z learned by the encoder. The K-means algorithm and its extensions have been the most commonly used clustering methods because of their efficiency. Assuming K subpopulations in latent space, we utilized a fuzzy k-means algorithm with adaptive distance measurement and entropy regularization to perform clustering  and cluster center μj(1 ≤ j ≤ K) are known, wij has an explicit close-form solution for above optimization problem, which iswhere μypes see . The entTherefore, we naturally constructed the following adaptive fuzzy k-means loss function,wij is given by Equation (14) and is the adaptive weighted coefficient representing the soft allocation probability of assigning the i-th cell to the j-th cluster membership. After each iteration of training, we can assign the i-th cell to the cluster label corresponding to the largest wij(1 ≤ j ≤ K).where Having finished whole model construction, we summarize the two components: denoising autoencoder based on multinomial modeling and fuzzy soft k-means clustering with adaptive loss. The total training objective function is given asZ through the encoder and hidden layer, which can calculate the L2 clustering loss. Then the low-dimensional representation Z forwards to the model output through the decoder, which is brought into L1 loss to calculate the negative log-likelihood. The weight of neural network and cluster centers can be jointly optimized and updated by stochastic gradient descent and a back-propagation algorithm. We implement our model in Python 3 using deep learning software Tensorflow. During the following simulation and real data experiments, we took α as 1 by default. The default values of hyperparameter σ and λ were also 1, and their optimal setting for particular datasets have been discussed in section 3. The optimizer for whole model is Adam with a learning rate 0.0001. For autoencoder network architecture, the sizes of two hidden layers were set to 256 and 64 in the encoder. The decoder is the reverse structure of the encoder, and the bottleneck layer (the latent space) had a size of 32. The minibatch size was set to 256 during the training process. As for the model training strategy, we first pre-trained L1 loss by 1,000 epochs and then initialize cluster centers μj(1 ≤ j ≤ K) by standard k-means algorithm in the learned latent space. Lastly, we trained the whole L loss function until cluster membership assignments did not change.where hyperparameter α controls the relative importance of data generation and data clustering. A simple model schematic is shown in We collected the scRNA-seq UMI count data matrix (cells by genes) after passing quality control. We first discarded the genes that are not expressed in any cell and then filtered the cells without gene expression. Then, we calculated the normalized dispersion value of each gene and pick the top 500 highly variable genes from the original whole genes based on dispersion ranking. The expression of these genes was used as input to the model. Based on the consideration of neural network numerical stability, we transformed discrete count data into continuous smooth data. Specifically, we first normalized the total count amount of each cell to their median value level and then utilized a log-transformation for data. Finally, we transformed the logarithm data into z-score data, which means that each selected gene has zero mean and unit variance. All the above preprocessing procedures can be accomplished by using the scanpy package  by 10x Genomics sequencing and designed two groups of noise experiments, one to sample a portion of the cells for clustering and another to randomly add zero elements. For the former, we downsampled 60 and 80% cells apart from the total dataset and then ran each clustering algorithm again. For the latter, we randomly masked some non-zero counts into zeros with probability 0.15. We repeated these procedures 10 times and then calculated the median ARI and NMI value to compare with the undisturbed original dataset. In downsampling experiments see , scDMFK With the rapid development of single-cell sequencing technology, the scale of single-cell RNA sequencing data is getting larger and larger, challenging the scalability of clustering algorithms. We therefore chose a mouse kidney dataset, “Park”  provided by the author on the “Chen” dataset. We also performed clustering analysis on its coarse division (11 cell types) given by the author see . Except L1 loss as model optimization objective function and then extracted the well-trained latent representation to perform cell clustering by a standard k-means algorithm. For convenience, we called them scDM+kmeans and D-scDM+kmeans, respectively. Based on the results of 10 real datasets . Although we set them both as 1.0 in default, they may not be the optimal parameter setting for some datasets in terms of clustering performance. Thus, we used the popular grid search method in the deep learning model fine-tuning strategy to determine the optimal parameter setting of six small real datasets. Specifically, σ was searched for in the grid of , and λ was searched for in the grid of . Single-cell sequencing technology allows researchers to explore tissue heterogeneity at the cellular level; of these, transcriptome sequencing is the most commonly used one since it can identify cell types by directly analyzing gene expression data and then discover the key regulatory genes that cause cell heterogeneity. During this process, clustering analysis of cell population is an essential and indispensable procedure. Therefore, in this paper, we have combined deep learning with statistical modeling to construct a novel model that fuses data denoising, data dimensionality reduction, and clustering. For data modeling, we made full use of the generation mechanism of UMI count data and adapt multinomial distribution modeling instead of the widely used negative binomial distribution or its zero-inflation version.For data dimensionality reduction, we used the neural network (autoencoder) that can efficiently capture the inherent non-linear structure of the data to learn the low-dimensional manifolds where those marker genes are located instead of the linear dimensionality reduction method, PCA, or local structure preservation dimensionality reduction algorithm t-SNE. For cell clustering, we used soft clustering with an adaptive loss function instead of hard clustering. In multiple groups of simulation data generated by the Splatter simulator, including various numbers of cell types, different degrees of sparsity, and different sizes of cell types, our methods were more effective and robust than other deep learning-based and traditional statistic-based scRNA-seq data clustering algorithms. In terms of real data analysis, we selected datasets from different organs, which have diverse population sizes and cell type number. The performance of our methods always ranked in the top three, whether by overall comparison, downsampling comparison, or disturbance factor comparison. On large-scale datasets, our model scDMFK was fast, accurate, and has perfect scalability. It is an effective algorithm for the rapid development of single-cell transcriptome data clustering analysis.During the analysis of the whole real datasets, we selected 500 highly variable genes as the input of the neural network. We filtered out those genes carrying insufficient information for identifying cell types owing to high noise, and we also hoped to speed up the model calculation process. In fact, we tested the clustering performance of selecting 300, 1,000, 1,500, and 2,000 highly variable genes on real datasets. From the ARI and NMI results in In addition to CIDR and SIMLR, we also compared scDMFK with other commonly used scRNA-seq data clustering methods, such as Seurat  or generalized linear model fitting estimation. DIMM-SC is a such method for single-cell UMI count data clustering based on mixture dirichlet multinomial distribution . The source codes of scDMFK are available at: https://github.com/xuebaliang/scDMFK. For convenience, we also place the whole real datasets in https://github.com/xuebaliang/scDMFK/tree/master/dataset.The publicly available real datasets for this study can be downloaded from the data repository NCBI Gene Expression Omnibus(LC and MD conceived and designed the scDMFK and D-scDMFK models. LC and WW implemented the simulation study and real data set analysis. LC and YZ wrote the whole specific codes. Lastly, LC and MD wrote the whole manuscript. All authors read and approved the final manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


ComBat) or have the potential to overcorrect, by removing true but unknown biological differences . It is well known that experimental conditions such as array or reagent batches, PCR amplification or ozone levels can affect the measured expression levels; often the direction of perturbation of the measured expression is the same in different datasets. However, there are no BE correction algorithms that attempt to estimate the individual effects of technical differences and use them to correct expression data. In this manuscript, we show that a set of signatures, each of which is a vector the length of the number of probes, calculated on a reference set of microarray samples can predict much of the batch effect in other validation sets. We present a rationale of selecting a reference set of samples designed to estimate technical differences without removing biological differences. Putting both together, we introduce the Batch Effect Signature Correction (BESC) algorithm that uses the BES calculated on the reference set to efficiently predict and remove BE. Using two independent validation sets, we show that BESC is capable of removing batch effect without removing unknown but true biological differences. Much of the variations due to batch effect is shared between different microarray datasets. That shared information can be used to predict signatures (i.e. directions of perturbation) due to batch effect in new datasets. The correction can be precomputed without using the samples to be corrected (blind), done on each sample individually (single sample) and corrects only known technical effects without removing known or unknown biological differences (conservative). Those three characteristics make it ideal for high-throughput correction of samples for a microarray data repository. We also compare the performance of BESC to three other batch correction methods: SVA, Removing Unwanted Variation (RUV) and Hidden Covariates with Prior (HCP). An R Package besc implementing the algorithm is available from http://explainbio.com.Microarray batch effect (BE) has been the primary bottleneck for large-scale integration of data from multiple experiments. Current BE correction methods either need known batch identities ( Batch effect (BE) has been the primary bottleneck for the large-scale integration of data from multiple experiments. BE, defined as the systematic biases between microarray data generated by different labs at different times or under different experimental conditions , 2, can batch effect as a general term for any heterogeneity due to experimental factors between samples that are putative experimental replicates. The heterogeneity can extend to different samples within the same sample collection, i.e. considering only average difference between two collections will likely underestimate the BE.Unknown or unrecorded experimental or biological differences can add a systematic difference between putative replicates within or between two batches. Thus, we use the term In practice, it has proven difficult to separate heterogeneity due to technical differences from that due to unknown biological differences. The usual approach in batch correction –6 is to Additionally, current batch correction methods are intended to be used each time a new composite dataset is created. It is known that specific differences in sample condition, experimental technique , 10 and batch effect signatures (BES) in the form of orthogonal components from a large reference dataset of samples. We develop an algorithm for computing the BES using the reference dataset such that the BES are unlikely to model known or unknown biological differences. We introduce a novel batch-correction method called Batch Effect Signature Correction (BESC) that uses the batch effect signatures for blind prediction and correction of BE in new samples and compare the performance to SVA.In this article, we take another approach: instead of estimating and removing all differences between two batches, we only remove those differences that are known to be associated with technical variations. We show that a large proportion of the BE in Affymetrix U133 Plus2 array data can be captured by a relatively small set of signatures, defined as the directions in which the measured expression has been perturbed by batch effects. We estimate The measured expression for a set of samples can depend on one or more biological factors  and unknown or unmodeled experimental factors . Following  we can mxij is the measured expression of gene i (out of m genes) for sample j (out of n samples), μi is the overall mean expression of gene i, fi(yj) is a (possibly non-linear) function that models the dependence of the expression of the ith gene on the jth known biological factors yj. The batch effect is modelled as a linear combination L experimental factors each composed of glj the effect of the lth experimental factor on the ith gene multiplied by weight γil. The last term eij is uncorrelated noise.Where fi(yj) and a space spanned by systematic batch effects Data-specific batch correction methods (such as ComBat and SVA) assume that the BE space (third term) is unique to each composite dataset and has to be recalculated each time a new composite dataset is created. However, we show that an orthogonal basis derived from the BE space of a reference dataset can be used to estimate and remove the variation in the BE space for other test datasets (i.e. blind batch correction).i and global means μi using linear or non-linear regression, we can express SVA  has beenSVA is to find a set of K≤L orthogonal vectors (surrogate variables) that span the same linear space as glThe aim of hk = T is a vector of length equal to the number of samples that models one hidden covariate that is not present in the known covariates, and λik is the influence of hk the measured expression of the ith gene. Together they can be used to model heterogeneity from unknown sources in any future statistical analysis on that dataset. SVA proceeds to find these surrogate variables by doing a Singular Value Decomposition of the residuals rij.Each surrogate variable SVA is that we do not need to know the actual batches. The disadvantages of SVA are that, firstly, it has to be recomputed on each new dataset, secondly, it can remove unknown but important biological differences between samples.The advantage of ν that has to be adapted to the dataset being corrected. RUV can be used without knowing the actual batches, but the selection of the control genes and ν remain challenging. Furthermore, these two selections are only valid for a particular dataset; they have to be re-evaluated for each new dataset.RUV  is a batλ,σ1,σ2 that have to be optimized for each dataset. HCP has the disadvantage that these parameters are specific to a particular dataset and have to be re-computed for each new dataset. Furthermore, without some ground truth to compare to , it is not possible to find the values for the three parameters that minimize the batch effect.HCP  is an apBESC that aims to learn the variation due to unknown technical differences using a reference dataset and apply it to correct batch effect in other datasets. To motivate our approach, note that λik is the difference in the expression of the ith gene for each unit change in the kth surrogate variable. Thus, the vector λk = , for the kth surrogate variable and each genes 1…m, is a signature that quantifies the dependence of the expression of each gene on the kth surrogate variable. For example, if the surrogate variable captures the effect of a particular type of sample preparation, the signature is the differential expression between samples that use that preparation method vs. samples that do not. We call the set of K vectors λk the Batch Effect Signatures (BES). Each vector is a zero-mean unit vector  that is as long as the number of probes on the array. We can consider batch effect to be a perturbation in a certain direction for all samples in the same batch. These perturbations can be estimated by looking at the expressions of the same cell line in different batches. Each perturbation is a sum of contributions from multiple sources of technical variation. BES decomposes the individual contribution of each source of technical variation by looking for a set of vectors that spans the batch effect perturbation-space.We introduce a novel approach i.e. signature λk) between samples that differ on a certain experimental factor (e.g. RNA amplified vs. un-amplified) would be more likely to remain the same, multiplied by some coefficient even if the surrogate variables do not remain the same. Given a dataset of reference samples that is large and diverse enough, we can compute the signatures of the various known or unknown factors that contribute to BE. Those signatures can then be used to estimate and remove BE from new samples. We call this approach the Batch Effect Signature Correction (BESC).Our claim is that the expression difference , D is the m×K matrix of batch effect signatures (λik the element at ith row and kth column) and U is the K×n matrix of surrogate variables (hkj the element at kth row and jth column). SVA constrains the rows of U to be orthogonal and estimates the matrix U using singular value decomposition (SVD) on the residual matrix R . On the other hand, we constrain the columns of D to be orthogonal and estimate D by taking the Principal Components (PCA) of the transpose of the residual matrix, RT. Note that there is no substantial difference between using SVD or PCA for decomposing R .Where SVA is not a batch correction algorithm. The surrogate variables also capture heterogeneity due to unknown biological differences since they are calculated without reference to the batch. Other batch correction methods such as ComBat , but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.2. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.  http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” . The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: YesReviewer #2: Yes**********4. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: YesReviewer #2: Yes**********5. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: This manuscript presents a novel computational method (BESC) to correct for batch effects in microarray experiments. The novel idea of the author is that he argues that scanning a large representative set of data should be informative enough for parametrizing the "general" batch effects that are inherent to a particular microarray chip once and for all.If this works, the author points out that one could run BESC once and apply it to new data sets in the future without having to rerun BESC.I find this a novel, appealing idea. I am not aware of a comparable approach.The manuscript is well written, clearly understandable and structured.Major pointshttp://explainbio.com(1) The manuscript states that the method is available from But I did not find the software there when I accessed that site.Hence, I/we could not test the tool ourselves, which is standard when announcing a new bioinformatics tool.(2) The presented results (figures + suppl figures) appear convincing.To illustrate the generality and limitations of BESC I suggest to add the following 2 tests:(a) In addition to the 5-fold CV already done, I suggest that the same 5 data splits should be processed as follows:-> First, BESC should be parametrized on sets 1 + 2 and then applied to set 5.-> Alternatively, BESC should be parametrized on sets 3 + 4 and then applied to set 5.It this manner, two halves of the training data would be used to predict batch-effect corrected versions of set 5. I am curious how different are the results of this exercise.If one does this for the TCGA data , how large is the overlap coefficient of the differentially expressed genes between the two runs?(b) The author argued he used data for a large panel of representative cell lines so that the parametrized BESC should be of general nature.I wonder what the limitations of this approach are?Does this also work for tissues and conditions that are not represented in the training data set?This should be demonstrated on a suitable example.(3) Another research group told us about their experience with TCGA data from the Illumina 27k CpG methylation chip (breast cancer). They found a strong batch effect between separate charges of the chip produced at different times.Such a case would not be covered by BESC. If you parametrized on MA data collected up to a certain moment in time and apply this to process "all" future data processed with this chip type, one would have to assume that the chip is homogeneously manufactured over many years.But this may not be the case, and may remain undetected if users simply rely on BESC.Discussion of this and related effects should be added in the discussion section.Minor points(4) throughtout the text many nouns start with capital letters, but need not to do so.(5) line 317: is to much more -> is much more(6) line 337: which can used -> which can be usedReviewer #2: The paper presents a novel method using signatures of batch effects generated from an external reference to eliminate batch effects from microarray data without harming the biological signal.The idea and method is very interesting and well presented.1. My main criticism is regarding the validation of the method. the method was compared to SVA which removes batch effects as well as the biological signal. The current validation might just show that SVA removes also biology with the batch effects which is worse than the present method or even the raw uncorrected data. Another comparison should be used for validation. For example, comparison with Combat (RUV etc) or other method that deals with removing batch effects without the biological signal, should be added. It might be also compared to the raw data, showing that it outperform the raw data. In case the new method doesn't correct the data, still it will outperform SVA which overfits the data and eliminated the DEGs.2. Im missing a presentation of other recent methods that deal with removing batch effects without removing the biological signal. For example, HCP by Mustafavi, BCeF by Somekh etc.Minor issues:p.1. line 9 "Even though the effects..." - unclear sentencep.1. line 22 - very long sentence with too many parentheses.p.2. l 34 - "or biological" - should be just experimental here?p.4 - some mistake for gij on the jth gene? not enough explanation on gamma and the factors.p 5 - need to add more explanation on the SVA formulas. so (3) is currently not fully clear. I had to read the SVA paper to understand.p 5 l.90 - sometimes ki and sometimes ik are used.p 5 l.92 - should be "each gene" instead of "all genes"?l 96 - length one is unclearl 98 - refine the English sentencel 102 - please add: to remain the same, multiplied by some coefficient.p 12 line 246 - Figure not Fig. Figure 2A - please add explanation on what can be seen for SVA.p. 15 line 294 - unclear sentenceline 299 - for SVA, please add reference**********what does this mean?). If published, this will include your full peer review and any attached files.6. PLOS authors have the option to publish the peer review history of their article  digital diagnostic tool,  6 Dec 2019Response to reviewersFirstly, I would like to thank the reviewers for the thorough review and very constructive suggestions. I have taken their suggestions to heart and performed extensive work to do the comparisons to other batch effect methods as well as validation methods. I believe this makes the manuscript stronger and improves the presentation considerably.I have added my responses below in light orange font.Reviewer #1: This manuscript presents a novel computational method (BESC) to correct for batch effects in microarray experiments. The novel idea of the author is that he argues that scanning a large representative set of data should be informative enough for parametrizing the "general" batch effects that are inherent to a particular microarray chip once and for all.If this works, the author points out that one could run BESC once and apply it to new data sets in the future without having to rerun BESC.I find this a novel, appealing idea. I am not aware of a comparable approach.The manuscript is well written, clearly understandable and structured.Major pointshttp://explainbio.com(1) The manuscript states that the method is available from But I did not find the software there when I accessed that site.Hence, I/we could not test the tool ourselves, which is standard when announcing a new bioinformatics tool.http://explainbio.comThat was due to a misconfiguration of the web server. The package should be available to download now, on clicking the “BESC R package” link on the top right-hand side. Please make sure to clear your browser cache before reloading (2) The presented results (figures + suppl figures) appear convincing.To illustrate the generality and limitations of BESC I suggest to add the following 2 tests:(a) In addition to the 5-fold CV already done, I suggest that the same 5 data splits should be processed as follows:-> First, BESC should be parametrized on sets 1 + 2 and then applied to set 5.-> Alternatively, BESC should be parametrized on sets 3 + 4 and then applied to set 5.It this manner, two halves of the training data would be used to predict batch-effect corrected versions of set 5. I am curious how different are the results of this exercise.I’ve included a plot that explores this . I’ve plotted the uncorrected samples from split 5 along with the same samples corrected by batch effect signatures (BES) calculated on splits 1+2 and splits 3+4 (i.e. two sets of BES). As long as the number of BES is smaller than 10, the corrected samples from the two sets of BES stay close together (while moving away from the uncorrected samples). After 10 BES, the two sets of corrected samples start separating.Furthermore, Supplementary Figure 7 shows the mean correlation between the corrections performed by the sets of BES computed on splits 1+2 vs 3+4. The correlation increases with increasing number of BES, reaching a maximum at 10 BES and then goes down.Together, the two results show that two sets of reference data splits (splits 1+2 and splits 3+4), pick up similar batch effect signatures (until ~ 10 BES) indicating that the algorithm is detecting a robust signal.It seems reasonable that we can find 10 robust BES from a bit less than half the reference data (splits 1+2 or split 3+4) compared to ~20 robust BES from the full reference set. It is possible that additional data in the reference set can give us a larger number of BES covering more of the batch effect If one does this for the TCGA data , how large is the overlap coefficient of the differentially expressed genes between the two runs?I’ve included a plot of the fraction of common genes between the colon cancer/normal dataset  when corrected by sets of BES calculated on the 1st + 2nd split combined and the 3rd + 4th split combined . There is a high degree of overlap (>80% intersection) till the number of BES reaches 10. The BES start to diverge from 20 onwards and the overlap decreases.(b) The author argued he used data for a large panel of representative cell lines so that the parametrized BESC should be of general nature.I wonder what the limitations of this approach are?Does this also work for tissues and conditions that are not represented in the training data set?This should be demonstrated on a suitable example.The two validation sets are actually different tissues and conditions compared to the reference set that was used to compute the BES. The reference set contains only cell line samples while the validation sets contain primary tissue samples . (3) Another research group told us about their experience with TCGA data from the Illumina 27k CpG methylation chip (breast cancer). They found a strong batch effect between separate charges of the chip produced at different times.Such a case would not be covered by BESC. If you parametrized on MA data collected up to a certain moment in time and apply this to process "all" future data processed with this chip type, one would have to assume that the chip is homogeneously manufactured over many years.But this may not be the case, and may remain undetected if users simply rely on BESC.Discussion of this and related effects should be added in the discussion section.I do detect a year-dependent effect of one of the BES. The coefficient calculated for samples with different scan years varies smoothly from year to year. It is true that any abrupt large change in the array manufacturing would not be detected by BESC until we have a number of reference samples on the new array. I’ve added a remark in the discussion on this issue.Minor points(4) throughtout the text many nouns start with capital letters, but need not to do so.(5) line 317: is to much more -> is much more Corrected line 372(6) line 337: which can used -> which can be used Corrected line 395Reviewer #2: The paper presents a novel method using signatures of batch effects generated from an external reference to eliminate batch effects from microarray data without harming the biological signal.The idea and method is very interesting and well presented.Major issues:1. My main criticism is regarding the validation of the method. the method was compared to SVA which removes batch effects as well as the biological signal. The current validation might just show that SVA removes also biology with the batch effects which is worse than the present method or even the raw uncorrected data. Another comparison should be used for validation. For example, comparison with Combat (RUV etc) or other method that deals with removing batch effects without the biological signal, should be added. It might be also compared to the raw data, showing that it outperform the raw data. In case the new method doesn't correct the data, still it will outperform SVA which overfits the data and eliminated the DEGs.Firstly, I would like to clarify that the comparison to the raw data is present in each figure . I’ve added a dotted horizontal line in each figure to show the value for the uncorrected data.I’ve added comparisons to the results using RUV [1] and HCP [2]. I find that RUV does outperform BESC in validation set 2 for correction with a small number of factors . However, this best performance is selected after choosing between two different sets of control genes and varying the value of ν (a tunable parameter for RUV). In contrast BESC (and SVA) does not need to be tuned. The results with HCP was disappointing for both validation sets . In both cases HCP does not do any better than the raw data. Note that these are the best results as selected from a range of values for the three HCP parameters λ,σ_1,σ_2.The tuning of the parameters for RUV and HCP are described in the Supplementary Methods.2. Im missing a presentation of other recent methods that deal with removing batch effects without removing the biological signal. For example, HCP by Mustafavi, BCeF by Somekh etc.I’ve added comparisons to RUV [1] and HCP [2] in the manuscript and discussion of the results from those algorithms. BCeF [3] appears to be a framework to evaluate and compare different batch correction methods based on how closely they recapitulate a known gene co-expression matrix. They report that none of the batch correction methods is able to fully recapitulate all the unmodelled confounders. As I understand it, they do not propose any new batch correction methods.1. Gagnon-Bartsch JA, Speed TP. Using control genes to correct for unwanted variation in microarray data. Biostatistics. 2012;13: 539–552. doi:10.1093/biostatistics/kxr0342. Mostafavi S, Battle A, Zhu X, Urban AE, Levinson D, Montgomery SB, et al. Normalizing RNA-Sequencing Data by Modeling Hidden Covariates with Prior Knowledge. PLOS ONE. 2013;8: e68141. doi:10.1371/journal.pone.00681413. Somekh J, Shen-Orr SS, Kohane IS. Batch correction evaluation framework using a-priori gene-gene associations: applied to the GTEx dataset. BMC Bioinformatics. 2019;20: 268. doi:10.1186/s12859-019-2855-9Minor issues:p.1. line 9 "Even though the effects..." - unclear sentence I’ve rewritten the sentence (p.1. line 9)p.1. line 22 - very long sentence with too many parentheses. I’ve rewritten the sentence (p.1. line 23)p.2. l 34 - "or biological" - should be just experimental here? I have tried to differentiate between batch effect caused due to unknown technical effects from that due to biological effects (in p.2. line 37). Most batch correction methods (e.g. ComBat) remove both, but BESC attempts to remove only the technical differences.p.4 - some mistake for gij on the jth gene? not enough explanation on gamma and the factors. I’ve explained gamma and tried to make the batch effect term clearer. glj  is the effect of the batch l on sample j. The dependence on the gene “i” is modeled by γ_"il" . (p.4. line 77)p 5 - need to add more explanation on the SVA formulas. so (3) is currently not fully clear. I had to read the SVA paper to understand. I’ve expanded the description (page 5)p 5 l.90 - sometimes ki and sometimes ik are used. Corrected (p.6. line 122)p 5 l.92 - should be "each gene" instead of "all genes"? Corrected (p.6. line 124)l 96 - length one is unclear I’ve made it clearer (p.6. line 128)l 98 - refine the English sentence I’ve re-written the sentence to make it clearer (p.7. 129-134)l 102 - please add: to remain the same, multiplied by some coefficient. I’ve added that phrase (p.7. line 137)p 12 line 246 - Figure not Fig. Figure 2A - please add explanation on what can be seen for SVA. I’ve updated the figure with results for RUV and HCA with description of the results for each algorithm (p.14. line 290)p. 15 line 294 - unclear sentence I’ve re-written the sentence (p.16. line 349)line 299 - for SVA, please add reference Added for SVA, RUV and HCP (p.17. line 353)AttachmentResponse-to-reviewers.docxSubmitted filename: Click here for additional data file. 16 Jan 2020PONE-D-19-17279R1Blind estimation and correction of microarray batch effectPLOS ONEDear Dr. Varma,Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that your manuscript may become acceptable after fixing a few remaining minor issues. Therefore, we invite you to submit a revised version of the manuscript that addresses the following 4 points:(1) The supplemental figures are not referenced in the main text. It is up to readers to detect that there are some supplemental figures and to try to figure out what they show. So please add some sentences in the main text and explain what the supplemental figures show, like you did in your reply to the reviewers.(2) Also, the legends of supplemental figures S5 to S7 should be improved.For example, the current legend of Fig. S5"S5 Fig: Consistency of calculated BES – Overlap between significant genes selected forMSS/MSI differences in validation set 2 for BES calculated on different subsets ofthe reference set."does not reflect your reply to the comment of reviewer #1:"I’ve included a plot of the fraction of common genes between the colon cancer/normaldataset  when corrected by sets of BES calculated on the 1st + 2ndsplit combined and the 3rd + 4th split combined .There is a high degree of overlap (>80% intersection) till the number of BES reaches 10.The BES start to diverge from 20 onwards and the overlap decreases."Similarly, the legends Figs. S6 and S7 could be improved.For example, the legend of Fig. S6 does not explain what the black, red and blue symbols are.The following small points refer to the line numbering of the revised manuscript with tracked changes(3) line 108: the disadvantages ... is -> the disadvantages ... are(4) line 387: correct plural/singular in"We show that the batch effect between different datasets occupy a space"https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.We would appreciate receiving your revised manuscript by Mar 01 2020 11:59PM. When you are ready to submit your revision, log on to If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocolsTo enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.We look forward to receiving your revised manuscript.Kind regards,Volkhard HelmsAcademic EditorPLOS ONEAdditional Editor Comments (if provided):The author has done a commendable job in addressing the points raised by the two reviewers.He has performed all requested additional analysis (reviewer #1) and comparisons to other tools (reviewer #2).The new analysis is convincing. The reply letter properly addresses all raised points.However, when looking at the revised version of the manuscript I felt that the presentation of the additional analysis (reviewer #1) in the manuscript could be improved, see points listed above.[Note: HTML markup is below. Please do not edit.]https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,  16 Mar 20201) The supplemental figures are not referenced in the main text. It is up to readers to detect that there are some supplemental figures and to try to figure out what they show. So please add some sentences in the main text and explain what the supplemental figures show, like you did in your reply to the reviewers.Response:I have added text to the manuscript describing those analyses and their results (lines 223-235 and 297-309)2) Also, the legends of supplemental figures S5 to S7 should be improved. For example, the current legend of Fig. S5 "S5 Fig: Consistency of calculated BES – Overlap between significant genes selected forMSS/MSI differences in validation set 2 for BES calculated on different subsets ofthe reference set."does not reflect your reply to the comment of reviewer #1:"I’ve included a plot of the fraction of common genes between the colon cancer/normaldataset  when corrected by sets of BES calculated on the 1st + 2ndsplit combined and the 3rd + 4th split combined .There is a high degree of overlap (>80% intersection) till the number of BES reaches 10.The BES start to diverge from 20 onwards and the overlap decreases."Similarly, the legends Figs. S6 and S7 could be improved.For example, the legend of Fig. S6 does not explain what the black, red and blue symbols are.Response: I’ve update the legends and descriptions3) The following small points refer to the line numbering of the revised manuscript with tracked changesa. line 108: the disadvantages ... is -> the disadvantages ... areb. line 387: correct plural/singular in "We show that the batch effect between different datasets occupy a space"Response: I have corrected these errorsAttachmentResponse.docxSubmitted filename: Click here for additional data file. 25 Mar 2020Blind estimation and correction of microarray batch effectPONE-D-19-17279R2Dear Dr. Varma,We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at onepress@plos.org.If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact With kind regards,Volkhard HelmsGuest EditorPLOS ONEAdditional Editor Comments :The author has now properly addressed my remaining minor points.Reviewers' comments: 27 Mar 2020PONE-D-19-17279R2 Blind estimation and correction of microarray batch effect Dear Dr. Varma:I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. onepress@plos.org.If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact plosone@plos.org. For any other questions or concerns, please email Thank you for submitting your work to PLOS ONE.With kind regards,PLOS ONE Editorial Office Staffon behalf ofProf. Volkhard Helms Guest EditorPLOS ONE


For lung and many other cancers, prognosis is essentially important, and extensive modeling has been carried out. Cancer is a genetic disease. In the past 2 decades, diverse molecular data (such as gene expressions and DNA mutations) have been analyzed in prognosis modeling. More recently, histopathological imaging data, which is a “byproduct” of biopsy, has been suggested as informative for prognosis. In this article, with the TCGA LUAD and LUSC data, we examine and directly compare modeling lung cancer overall survival using gene expressions versus histopathological imaging features. High-dimensional penalization methods are adopted for estimation and variable selection. Our findings include that gene expressions have slightly better prognostic performance, and that most of the gene expressions are weakly correlated imaging features. This study may provide additional insight into utilizing the two types of important data in cancer prognosis modeling and into lung cancer overall survival. For most if not all cancers, various prognosis outcomes, such as overall survival, progression free survival, and time to metastasis, are of essential importance. Accordingly, extensive modeling research has been conducted. In “classic” prognosis studies, low-dimensional demographic, clinical, and environmental risk factors are analyzed, and “standard” regression-based techniques (such as Cox model) are usually sufficient. Despite some successes, it has been well recognized that the complexity of cancer prognosis demands additional data and more sophisticated modeling.1. Since then, hsa-mir-155 over expression has been reported in thyroid carcinoma, breast cancer, colon cancer, and cervical cancer, indicating its potential for serving as a biomarker for tumor detection and evaluation of prognosis outcome2. As another example, the study of genome-wide expression of 100 Non-Small-Cell lung cancer (NSCLC) FFPE samples identified a signature composed of 59 genes, which was strongly associated with prognosis for stage I lung cancer patients. This signature was later proven to be robust for clinical usage3. Molecular data are high-dimensional and contain substantial “noises”, that is, the majority of measurements are not associated with prognosis. To effectively remove noises, identify relevant effects, and build reliable models using “signals” only, a myriad of high-dimensional statistical techniques has been developed. A popular family of approaches conducts regularization and applies techniques such as penalization, boosting, Bayesian, and thresholding, which can simultaneously achieve estimation and variable selection. Such techniques have demonstrated statistical, numerical, and empirical successes. We refer to published literature6 for reviews and more extensive discussions. With the accumulation of clinical and experimental data, there is increasing knowledge on the functionality of molecular changes. As such, studies have also been conducted using molecular changes that have “prior information”, for example, with evidence of being relevant from previous studies. In this line of work, multiple gene panels have been developed and utilized. For example, Jablons and others aimed at developing a prognostic risk score for patients with completely resected lung adenocarcinomas based on genes previously identified in microarray models of NSCLC prognosis. They suggested narrowing the 61-gene panel down to four genes7. A drawback of molecular data is that it is not as easy to collect: many patients are still concerned with providing tissues for molecular profiling, not all hospitals can conduct profiling and process such data routinely, and the cost of high-throughput profiling is still not “friendly”.Cancer is a genetic disease. In the past 2 decades, with the fast development of high-throughput sequencing techniques, molecular data have been extensively collected in cancer studies. Accordingly, molecular data-based prognosis modeling has been accumulating. For example, an investigation of miRNA expression in 104 pairs of primary lung cancers and corresponding noncancerous lung tissues revealed that high hsa-mir-155 and low hsa-let-7a-2 expressions were correlated with poor survival. The signatures were cross validated using an independent set of adenocarcinomas8. They contain rich information on tumors’ “micro” properties and surrounding microenvironment, which play important roles in cancer development. Traditionally, pathologists would examine specimens on slide glass for hours using microscopes and make judgement on a handful of features such as tumor-infiltrating lymphocytes (TIL) and tumor cell intensity. This process can be highly time-consuming, and have poor inter-laboratory, inter-observer, and intra-observer reproducibility9. More recently, the development of digital imaging processing algorithms and software has made it possible to automatedly extract features from histopathological images. Compared to the traditional approach which highly relies on human capability, the new approach is much less labor-intensive and can extract more features that are “hidden” from human eyes and have not been traditionally studied, hence containing possibly different information. With less dependence on human, these high dimensional features can also be more objective and reliable. In a handful of recent studies, histopathological imaging features, especially those extracted using automated imaging processing software, have been used for modeling cancer prognosis (as well as other outcomes and phenotypes)11. However, such studies are still relatively scarce. With the consideration that tumor properties as reflected in histopathological images can be affected by molecular changes, there have been studies modeling the relationships between imaging features and molecular changes13. Such studies are biologically well-grounded. In particular, morphological features of tumor cells and microenvironment can be caused and regulated by molecular changes. As a testament, the successful prediction of microsatellite instability from histopathological images of gastrointestinal cancer14 and colorectal cancer15 suggests that such a genotype–phenotype correlation is consistent enough to robustly infer genotypes by observing histopathological imaging features. A recent pan-cancer study confirmed this finding by analyzing the histopathological images of more than 5,000 patients across 14 solid tumor types using deep learning. This study demonstrated the feasibility of identifying genetic variants, gene expression signatures, and clinical biomarkers from images16. There are also a small number of recent studies showing that collectively analyzing molecular and imaging data can improve prediction. For example, to predict the prognosis of Glioblastoma Multiforme (GBM), Kang et al. integrated histopathological imaging and gene expression data with a deep learning approach. The integrated data achieved a C-index of 0.702 in comparison to 0.640 by using only histopathological imaging data17. Similar data integration has also been pursued for breast cancer19, glioma20, lung cancer21, and prostate cancer22. These studies have suggested the great potential of high dimensional histopathological imaging features for cancer research. Overall, with the cost-effectiveness and routineness of biopsy and histopathological images can potentially play an important role in cancer modeling. As a “side note”, we distinguish between histopathological images and radiological images—the latter are generated by CT, PET, and other radiological techniques and inform “macro” properties of tumors such as size, shape, and density.A more recent type of data for cancer modeling comes from histopathological imaging. In cancer clinical practice, biopsy is routinely conducted, which generates histopathological images. Such images have been long used for definitive diagnosis and stagingA common limitation of the existing studies is that information has been scattered. More specifically, studies that analyze both histopathological imaging features and molecular changes using the same data and on the same ground are very limited. With differences in patient characteristics and data generation, processing, and analysis procedures, findings from different studies may not be directly comparable. In the integration studies, there is often a lack of attention to the direct comparison of molecular and imaging data analysis results.The objective of this study is multi-fold. Specifically, it intends to further demonstrates cancer prognosis modeling using histopathological imaging and molecular data, taking advantage of high-dimensional regularization techniques (which may have a more lucid interpretation than the deep learning and some other techniques). More importantly, it provides a direct and fair comparison of modeling using these two types of highly important and popular data—this differs from most of the published studies. To be comprehensive, we also examine integrating these two types of data for modeling prognosis as well as modeling their relationships, as in some of the aforementioned studies. With the analysis of TCGA LUAD and LUSC data, this study may also provide additional insight into lung cancer prognosis.23. We choose to analyze TCGA data because of its high quality, comprehensiveness, and public availability. In particular, we analyze data on LUAD (lung adenocarcinoma) and LUSC (lung squamous cell carcinoma), two subtypes of NSCLC. Lung cancer patients in general have poor prognosis, and as such, prognosis modeling can be especially important. For prognosis outcome, we choose overall survival, as in Radzikowska et al.24, Collins et al.25, and quite a few other studies.TCGA (The Cancer Genome Atlas) is one of the largest and most comprehensive cancer projects organized by the NCI  and NHGRI . For over thirty different types of cancer, it has published comprehensive phenotypic, demographic, molecular, and imaging datahttps://portal.gdc.cancer.gov). These tissue slides are formalin-fixed and paraffin-embedded, and the cell morphology is well-preserved and suitable for image feature recognition. They are captured at 20× or 40× magnification by the Aperio medical scanner. In recent studies, we13 and others26 have developed and implemented a pipeline for extracting high dimensional imaging features, which is sketched in Fig. 27, a publicly available software package that has been adopted in quite a few recent studies28. In the next step, for each patient, features are averaged. We refer to Zhong et al.13 and Luo et al.26 for more detailed discussions on this imaging processing pipeline as well as alternatives. With this processing pipeline, a total of 299 features can be obtained. We note that this is significantly higher than in studies such as Wang et al.29 and Romo et al.30. As briefly mentioned above, some imaging studies, especially the early ones, utilize low dimensional imaging features. Comparatively, high dimensional features may have less lucid interpretations but can contain information not reflected in the low dimensional features. With their advantages such as cost-effectiveness and reliability, it can be of higher interest to examine their prognosis modeling performance. With the extracted features, we further conduct quality control. In particular, irrelevant features, such as file size and execution information, are removed. We also remove features with severe missingness (> 25%) and no or little variation. A total of 221 features are included in downstream analysis.Whole-slide histopathological images in the svs format are downloaded from the TCGA website  randomly split all samples into a training and a testing set with sizes roughly 3:1; (b) conduct survival analysis as described above using the training set; (c) for subjects in the testing set, compute the predicted risk scores. For example, for the analysis with imaging features, the risk scores are 21. We note that this analysis is unsupervised in the sense that it does not involve survival. As such, the most direct goal is not to improve prognosis modeling but rather to understand, in a broad sense, overlapping information contained in the two distinct types of data.As briefly discussed in “With normalization to zero means, consider the model:Similar to above, to more objectively evaluate the relationship, we consider the following approach: (a) randomly split data into a training and a testing set in the same way as above; (b) conduct the group Lasso estimation using the training set; (c) for the testing set subjects, predict imaging feature values using gene expressions and the training set estimate. For each imaging feature, compute the correlation coefficient between the predicted and estimated values; (d) to avoid an extreme split, repeat Steps (a)–(c) 100 times, and compute the average correlation values. We note that penalization may introduce shrinkage towards zero. As such, we adopt correlation coefficient as the criterion, which is less affected by shrinkage.The first set of analysis regresses survival on the imaging features and clinical characteristics. For the variables included in the final models, their estimated regression coefficients are shown in Tables 37 and RND338. In addition, there are also genes that have not yet been well examined in the literature, such as DNMT2 and UQCRC2.In the next set of analysis, we regress survival on gene expressions. The identified gene expressions and clinical characteristics as well as their estimated coefficients are shown in Tables When integrating the combined imaging effect with the combined gene expression effect in one Cox model, for the LUAD data, we obtain regression coefficients 0.9842  and 0.4726 . For the LUSC data, we obtain regression coefficient 0.9709  and 0.8769 .In the random-splitting based prediction evaluation, for the LUAD data, the median prediction C-index values are 0.6202 (imaging features), 0.6864 (gene expressions), and 0.6823 (combined). For the LUSC data, the median prediction C-index values are 0.5466 (imaging features), 0.5606 (gene expressions), and 0.5511 (combined). More detailed information, for example on the prediction C-index of each split, is available from the authors.Remarks8. In the analysis of gene expression data, the “familiarity” of findings may provide support to the validity of analysis to a certain extent. However, it is noted that more definitive validation will be needed to confirm the findings. The survival analysis with both imaging and gene expression signatures as covariates seems to suggest that the two types of measurements have independent effects. In the random splitting-based evaluation, it is observed that for LUAD, gene expression has moderate predictive performance, and imaging data has moderate/weak predictive performance. For LUSC, both types of measurements have weak predictive performance. For both datasets, gene expression has better performance, which is sensible considering the genetic nature of lung cancer (and other cancers too). Although both LUAD and LUSC are lung cancer subtypes, we observe significantly different results, which can be attributable to the complexity of cancer and suggest that there may not be a definitive conclusion applicable to all cancers. The random splitting evaluation further suggests that integrating the two types of signatures in an additive manner may not further improve prediction, which seems to “contradict” the analysis above. There can be multiple interpretations for this finding. First, the distinction between estimation and prediction should be made – a “good” estimation result may not directly translate into a good prediction. Second, the estimation analysis is repeatedly based on the same data, and there is a risk of over fitting. Third, in the random splitting evaluation, both the training and evaluation are based on fewer observations. An improvement that can be potentially observed with a larger dataset may not be observable with a smaller dataset. It is also noted that penalization and some other sparse approaches have been designed for estimation and may not be ideal for prediction, which may explain the less satisfactory prediction performance observed here.In the separate survival analysis with imaging features and gene expressions, relevant effects have been identified. For imaging features, extensive additional research will be needed to annotate and fully comprehend the identified variables. We note that this issue has been noted in the literatureWe first regress imaging features on gene expressions. Detailed information on the identified gene expressions and their estimated coefficients are provided in the Supplementary Materials The random-splitting based prediction evaluation results are summarized in Fig. 41. It is unclear whether such a difference is attributable to the complexity of cancer, difference in analysis approach, or other factors. More exploration, especially a direct comparison, will be needed.The regression analysis suggests that certain gene expressions are connected to imaging features. This observation is sensible considering, as described in “To complement the above analysis, we conduct additional exploration and present the findings in the Supplementary Materials. In particular, (1) in some cancer studies with high dimensional variables, marginal screening is conducted prior to modeling to reduce dimensionality to a more manageable level. In the above analysis, as the dimensionalities are not as high, screening is not conducted. Results presented in the Supplementary Materials suggest that, for our particular data and analysis, screening can change estimation and identification results, but has no substantial impact on prediction performance. (2) The above penalized estimations involve a tuning parameter, which is selected using cross validation. In the literature, there are many tuning parameter selection methods, and cross validation has been among the most extensively used. In the Supplementary Materials, we show that varying the tuning parameter values near the cross-validation-selected optimal has some moderate impact on estimation. But the findings on prediction are not strongly impacted.Accurately modeling prognosis and other cancer outcomes has been and will remain an important problem for a long time to come. Molecular and histopathological imaging data have played important roles in cancer prognosis modeling. In particular, with unique advantages including broad availability and high cost-effectiveness, it will be of interest to develop more histopathological imaging-based prognosis modeling. In this study, we have analyzed and integrated molecular and imaging data on the same ground using regularization techniques. More analysis of this kind will be needed to better understand the relative roles that molecular and imaging data play for other cancer types. Some of our findings are “negative”: for example, we have found that integrating data using the additive approach cannot improve prediction. More sophisticated methodological development will be needed to conclude whether this lack of improvement should be attributable to data/cancer type or analysis approach. The revealed interconnections between imaging and molecular features warrants additional investigation.Supplementary information 1Supplementary information 2


This study was designed to identify lncRNA biomarker candidates using lung cancer data from RNA-Seq and microarray platforms separately.Lung cancer datasets were obtained from the Gene Expression Omnibus  and The Cancer Genome Atlas  repositories, only common lncRNAs were used. Differentially expressed (DE) lncRNAs in tumors with respect to normal were selected from the Affymetrix and TCGA datasets. A training model consisting of the top 20 DE Affymetrix lncRNAs was used for validation in the TCGA and Agilent datasets. A second similar training model was generated using the TCGA dataset.First, a model using the top 20 DE lncRNAs from Affymetrix for training and validated using TCGA and Agilent, achieved high prediction accuracy for both training (98.5% AUC for Affymetrix) and validation (99.2% AUC for TCGA and 92.8% AUC for Agilent). A similar model using the top 20 DE lncRNAs from TCGA for training and validated using Affymetrix and Agilent, also achieved high prediction accuracy for both training (97.7% AUC for TCGA) and validation (96.5% AUC for Affymetrix and 80.9% AUC for Agilent). Eight lncRNAs were found to be overlapped from these two lists. Lung cancer is the leading cause of cancer-related mortality worldwide , 2. AccoAdvances in high-throughput technologies in recent years have brought a massive increase of multi-omics  data . PotentiAlthough the central dogma of biology states that the flow of genetic information hardwired in the DNA occurs by transcription into RNA and translation into proteins, non-coding RNAs (ncRNAs) are not translated. The many types of ncRNAs are broadly classified into long and small ncRNAs . In recehttps://software.broadinstitute.org/cancer/cga/target).The majority of papers reporting meta-analyses assembled differentially expressed gene (DEG) lists from published experimental studies and then articulated consistently reported DEGs; or integrated multiple datasets from different microarray platforms and then executed statistical tests to discover consistently expressed DEGs . By contPatient information from the downloaded datasets are summarized in Based on the PCA, we merged the two Affymetrix datasets, increasing the sample size to a total of 247. We used this merged microarray Affymetrix dataset for training to identify lncRNA biomarkers that are differentially expressed between lung cancer and normal samples, then used the Agilent and TAGA datasets for validation. For the alternative analysis, we used the RNA-seq TCGA dataset, which had a comparable sample size of 216, for training and validated on the Affymetrix and Agilent datasets. The Agilent database was only used for validation in both analysis streams because of its small sample size (n=40). Results of the first analysis where training was done on the Affymetrix dataset using a two-sample t-test are listed in Results of the same workflow where the TCGA dataset was used for training are also shown in As stated above, we first trained on the merged Affymetrix dataset and validated in both Agilent and TCGA datasets. We used the top 20 differential lncRNAs  to buildSimilarly, when the top 20 differentiated lncRNAs from training done on the TCGA dataset were used to build a classification model using the Voted Perceptron algorithm, we also achieved very good accuracy in separating cancer from normal samples. The training sensitivity was 0.991, specificity was 0.954 and AUC was 0.995 . ValidatComparing the top 20 lncRNAs from the Affymetrix and TCGA datasets revealed 8 overlapped lncRNAs  which weWe sought to determine if lncRNAs could predict patient survival . LINC025We also performed functional analysis using the common 8 lncRNAs. This analysis revealed three significant pathways for 3 lncRNAs : AC00826Biomarkers from easily accessible tissues such as blood and other body fluids are useful and economical screening tools for various diseases. Such tools are especially important for diseases such as lung cancer where existing diagnostic methods are not able to identify patients at early stages when intervention can be more effective. Indeed, a growing number of studies are using high throughput next-generation sequencing (NGS), especially microarray and RNA-Seq data, to identify diagnostic or prognostic biomarkers for lung cancer , 44–48. Microarray and RNA-Seq are two popular ways to measure gene expression. These two technologies have been compared in terms of technical reproducibility, variance structure, absolute expression levels, detection of isoforms, and the ability to identify DEGs and develop predictive models . In geneThe top 20 lncRNAs from each training were then used in machine learning to build two training models. We used two classifiers, Voted Perceptron algorithm and Bayes Network learning, for machine learning. All the classifiers were tried in Weka and the best results were chosen for each training dataset. Interestingly, we noted that the best results for the two training datasets were from different classifiers, with Bayes Network learning working better for the Affymetrix dataset and Voted Perceptron algorithm for the TCGA dataset. Overall, we found that the model built from Affymetrix was better than the model from TCGA. We also noted that the Agilent dataset as a validation dataset performed comparatively worse in all models. The sample size of the Agilent dataset was small compared to the others and it did not cluster with the other two datasets when PCA was done after normalization. This could be due to Affymetrix and Agilent are different platforms and the ways for them to design the arrays are not the same, either. Also maybe the Agilent dataset sample size is not big enough. The batch effects even exist for the same platform coming from different labs and more often existed from different platforms. Still, because this study aims to find biomarkers for global lung cancer, we decided to include the Agilent dataset in our analysis.The training models built separately from the Affymetrix and TCGA datasets and validated on the rest of the datasets resulted in two lists of 20 lncRNAs which include 8 common ones that could be diagnostic lncRNAs. Given that both the sensitivity and specificity are greater than 0.9, we can say that these 8 biomarkers can help predict whether the tissue sample is lung cancer or healthy. Moreover, the good performance of these 8 lncRNA biomarkers strongly suggests that they should work as biomarkers for all lung cancer samples, perhaps including subtypes, although subtypes were not explored in this study. Some of these 8 lncRNAs have previously been described in connection with cell biology and cancer. For example, Qiao et al. reported that TBX5-AS1 was down-regulated in lung cancer tissues compared to non-tumor lung tissues, and its expression was linked to unfavorable prognosis in never-smoking female lung cancer patients . Liu et Previous studies have described the function of lncRNAs, but in general, their clinical potential is underexplored . Here, wOur study is significant because it reports promising biomarker candidates with solid cross-validation bioinformatics data analysis on different platforms of a pretty large sample size. These biomarkers could be tested in blood serum or plasma samples in future studies. Besides providing potential novel diagnostic biomarkers for lung cancer, our study also provides novel candidate molecules and pathways for mechanistic studies on lung cancer development and carcinogenesis and for the development of new targets for lung cancer treatment.We identified 8 lncRNAs as potential diagnostic biomarkers for NSCLC through integrative cross-platform data analyses. This data mining and machine learning approach would be an efficient and economical screening method for tumor biomarker discovery. Moreover, we are now in an exciting time in bioinformatics when both high-throughput tools and data are increasingly accessible for tumor biomarker discovery. Our study can also help understand the development of lung cancer and provide potential novel targets for lung cancer treatment.http://cancergenome.nih.gov/). For the datasets from array-based platforms, GSE19188 and GSE18842 were combined as Affymetrix dataset and GSE70880 was named as Agilent dataset.To detect lncRNAs differentially expressed between healthy and lung cancer tissues, we employed a one-factor  experimental design in which datasets containing lung cancer samples and adjacent normal tissue samples were selected . This apLncRNA names were obtained from BioMart and a total of 963 lncRNAs common to every dataset were identified for further analyses. At first, we used the Affymetrix dataset for training. Differentially expressed lncRNAs at FDR adjusted p-value by the Benjamini-Hochberg procedure less than 0.05 using Student’s t-tests were selected. We uploaded the data to Weka (version 3-8-2) , then usNext, we used the TCGA dataset for training and selected differentially expressed lncRNAs at FDR adjusted p-value less than 0.05 using Student’s t-tests. We applied Correlation Attribute Eval feature selection on the statistically significant results to find the most related lncRNAs and used the top 20 to build the model using the Voted Perceptron algorithm incorporated in Weka. This time, validation was performed on the Affymetrix and Agilent datasets. Of the top related 40 lncRNAs from the two analyses, we identified 8 overlapping lncRNAs. These were further interrogated for survival and function analysis.Lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC) datasets from TCGA incorporating RNA-Seq FPKM data from 216 lung tissue samples and matched adjacent normal tissue samples were downloaded. The annotations of the TCGA dataset were acquired from BioMart in R (version 3.4.3).We conducted a search of the GEO Microarray database using keywords “ncRNA, lung cancer and RNA seq*” to find microarray datasets. We also searched for papers in PubMed that were related to lncRNA, RNA sequencing and lung cancer. We only downloaded microarray datasets with at least 20 lung cancer samples and adjacent normal tissue samples because studies with a smaller sample size would be challenging to merge due to batch effects. The downloaded GEO datasets are: GSE70880, GSE19188, GSE18842. GSE19188 and GSE18842 contain NSCLC samples but the lung cancer types in GSE70880 are unknown. The annotations of each dataset were obtained from BioMart in R (version 3.4.3). The TCGA or GEO datasets were merged based on transcript names and a total of 963 lncRNAs common in all the datasets were included for further analyses.First, we used per sample every sample median value across all the lncRNAs in GSE19188, GSE70880, and GSE18842 datasets. Then we performed per gene normalization based on every lncRNA expression median value across all the samples in these three microarray datasets. We used RBoxPlot in Array Studio software to check the normalization results. LncRNAs in the GSE70880 dataset with missing data were excluded. Principal component analysis based on the common 963 lncRNAs was done on the combined datasets. GSE19188 and GSE18842, both from the Affymetrix platform, clustered well, but GSE70880 from the Agilent platform could not cluster with the other two. So, GSE19188 and GSE18842 were combined and named as Affymetrix dataset and GSE70880 was named Agilent dataset for further analyses. RBoxPlot were obtained using Array Studio 10 The lung cancer RNA-seq data from TCGA was normalized based on the Fragments Per Kilobase of transcript per Million mapped reads (FPKM). The FPKM data from TCGA were log 2 transformed after adding 0.1.For each dataset, the difference in expression of lncRNAs between cancer and normal was examined by a two-sample t-test. The fold change and regulation direction were then reported. Each of the datasets was tested for differential expression by a two-sample t-test using Array Studio 10. Statistically significant differentially expressed lncRNAs were selected with a False Discovery Rate (FDR) adjusted p-value less than 0.05 and fold change greater than 1.3 in at least one dataset.The Affymetrix dataset containing differentially expressed lncRNAs was used as a training dataset first. Feature selection correlation method was done using Weka (version 3.8.2) and the top 20 lncRNAs were selected for further analysis. Hierarchical cluster analysis was performed using Array Studio 10. Bayesian network classifier (BayesNet) was used on the top 20 Affymetrix lncRNAs and validated in TCGA and Agilent datasets. Sensitivity and specificity were calculated based on the Bayesian network results. The formulas for sensitivity and specificity are:Sensitivity (True Positive Rate) = True Positive / ;Specificity (True Negative Rate) = True Negative / .Likewise, the TCGA dataset with differentially expressed lncRNAs was used as a training dataset and Weka (version 3.8.2) feature selection correlation method was used to identify the top 20 lncRNAs. Voted Perceptron was used for classification, and then validated in both Affymetrix and Agilent datasets separately. Sensitivity and specificity were calculated based on Voted Perceptron results. Subsequently, hierarchical cluster analysis was performed using Array Studio to check the expression levels of the eight overlapping lncRNAs. The Receiver Operating Characteristic (ROC) curves were plotted using Weka software.Cox regression analysis was performed using the survival package in R. The lncRNAs with a p-value of less than 0.05 were considered associated with survival. This analysis includes all lung cancer samples with survival information available from TCGA.Overlapping lncRNAs from the top 20 lncRNAs obtained after feature selection from Affymetrix and TCGA datasets respectively, were used for functional analysis.We used TANRIC to find correlated mRNA, miRNA, protein and somatic mutation with the common lncRNAs in LUSC and LUAD datasets, respectively. The lists of correlated genes were then used in TARGET and DAVID for functional analysis.Supplementary Figure 1Supplementary Table 1