The requirement of innovative big data analytics has become a critical success factor for research in biological psychiatry. Integrative analyses across distributed data resources are considered essential for untangling the biological complexity of mental illnesses. However, little is known about algorithm properties for such integrative machine learning. Here, we performed a comparative analysis of eight machine learning algorithms for identification of reproducible biological fingerprints across data sources, using five transcriptome-wide expression datasets of schizophrenia patients and controls as a use case. We found that multi-task learning (MTL) with network structure (MTL_NET) showed superior accuracy compared to other MTL formulations as well as single task learning, and tied performance with support vector machines (SVM). Compared to SVM, MTL_NET showed significant benefits regarding the variability of accuracy estimates, as well as its robustness to cross-dataset and sampling variability. These results support the utility of this algorithm as a flexible tool for integrative machine learning in psychiatry. Biological research on psychiatric illnesses has highlighted the scale of investigations required to identify reproducible hallmarks of illness ,2. In scThe combined “mega-analysis” of data across cohorts and modalities has advantages compared to the more traditional meta-analysis ,7, as itThe application of machine-learning techniques on biological problems in psychiatry has already yielded impressive results, including on the prediction of genetic risk, the identification of biomarker candidates, or the exploration of etiological mechanisms . For exaIn the present study, we used MTL to differentiate schizophrenia patients from controls across multiple transcriptome-wide expression datasets. We hypothesized that MTL is particularly suited for this task, since it allows the consideration of different cohorts as separate classification tasks. As MTL aims to identify predictive patterns that are shared across tasks, it should uncover expression patterns that are biologically reproducible across cohorts. This may result in better and biologically more relevant classifiers compared to those derived from conventional single task learning (STL), which may be unduly influenced by strong signals present in individual cohorts. To test this, we performed a comparative analysis of different MTL and STL approaches in five transcriptome-wide datasets of schizophrenia brain expression. A ‘leave-dataset-out’ procedure was applied to explore and compare the generalizability of the models, with specific focus on classification accuracy, and variability thereof, as well as model sensitivity to cross-dataset and sampling variability.In dn = 4 (four datasets used for training), suggesting that MTL required a higher dataset number to identify a reproducible biological pattern. However, we observed that the variation of accuracies for MTL_NET substantially decreased with increasing numbers of training datasets . First, raw expression data were read using the ‘ReadAffy’ function. Then RMA . For each dataset, propensity score matching was used to obtain a sample with approximate 1:1 matching for diagnosis, sex, ph, age, and post-mortem interval (pmi). Next, all datasets were concatenated for quantile normalization and covariate correction. Specifically, the ‘Combat’ function from the R library sva [2, pmi, and a dataset indicator). Finally, datasets were separated again for feature standardization (z-score) to remove bias from the expressed genes with large variance and for downstream machine learning analysis.Preprocessing was performed using the statistical software, R ( Average ) was apprary sva  was applFor MTL, multiple across-task regularization strategies were tested, such as MTL with network structure (MTL_NET), sparse network structure (MTL_SNET), joint feature learning (MTL_L21), joint feature learning with elastic net (MTL_EN), and low-rank structure (MTL_Trace). As a comparison, we selected logistic regression with lasso (LR), linear support vector machines (SVM), and random forests (RF) as representatives of conventional STL methods. For all models (except for RF), stratified five-fold cross validation was used to select hyper-parameters. Methodological details of the respective methods are described below. All machine-learning analyses were performed using Matlab (R2016b).For all MTL formulations, the logistic loss , as illustrated in To compare the consistency and stability of markers between algorithms, we used the correlation coefficient as the similarity measure of pairwise transcriptomic profiles  learnt by algorithms. A high similarity between profiles implied that models shared important predictors with respect to their weights and signs. Using this similarity measure, ‘consistency’ and ‘stability’ were defined, respectively. These measures were derived from 100-fold stratified bootstrapping of subjects from a set of datasets. In each bootstrapping sample, we tested across the number of training sets  datasets. For this, we differentiated two types of consistencies: ‘Horizontal’ and ‘vertical’ consistency as illustrated in Stability: To quantify the stability of an algorithm against the sampling variability, we observed the variation of transcriptomic profiles learnt from different bootstrapping samples as illustrated in Success rate: In addition to consistency and stability, to perform a side-by-side comparison of algorithms, we defined the success rate as the proportion of cases where one algorithm outperformed the other. For example, we quantified the success rate of consistency as the proportion of bootstrapping samples where the first algorithm demonstrated higher consistency than the second (see supplementary methods for details). The success rate of stability was quantified as the proportion of models, which were more stable for the first algorithm than that for the second (see supplementary methods for details). This work evaluates the performance of MTL and STL for biomarker analysis across five transcriptomic schizophrenia expression datasets. Several quality control procedures were employed to remove unwanted variation in the investigated datasets and to improve the biological generalizability of the obtained results. Despite this, the presented results should be interpreted in the light of the specific datasets investigated. Since other data modalities, including neuroimaging or gene methylation, show similar cross-dataset heterogeneity and correlation structures across variables, the present results may not be limited to expression data, although this remains to be empirically demonstrated. Furthermore, future investigations should include systematic simulation studies to explore the performance of MTL and its robustness against factors typically affecting machine learning performance, including data dimensionality, predictor effect sizes, and biological as well as experimental variability across datasets.The present study demonstrates the utility of MTL for integrative machine learning in high-dimensional datasets, compared to classical single-task learning. Mega-analyses that require integration of data across numerous datasets are becoming more frequent, but thus far, have rarely used machine learning approaches. The present study shows that MTL bears substantial promise for such applications. This particularly applies for scenarios where inter-dataset heterogeneity far outweighs the illness associated signal, a typical case for high-dimensional datasets in psychiatric research.


Megalobrama amblycephala (BSB) × Culter alburnus (TC) generates the variations in phenotypes and genotypes, but it is still a challenge to deeply mine biological information in the transcriptomic profile of this lineage owing to its genomic complexity and lack of efficient data mining methods. In this paper, we establish an optimization model by non-negative matrix factorization approach for deeply mining the transcriptomic profile of the sub-genomes in hybrid fish lineage. A new so-called spectral conjugate gradient algorithm is developed to solve a sequence of large-scale subproblems such that the original complicated model can be efficiently solved. It is shown that the proposed method can provide a satisfactory result of taxonomy for the hybrid fish lineage such that their genetic characteristics are revealed, even for the samples with larger detection errors. Particularly, highly expressed shared genes are found for each class of the fish. The hybrid progeny of TC and BSB displays significant hybrid characteristics. The third generation of TC-BSB hybrid progeny (It has been shown that reciprocal cross allodiploid lineage with sub-genomes derived from the cross of  Taxonomy aims to define and name groups of biological organisms on the basis of their shared similarity in morphological structure and physiological functions . It playIn the past two decades, with an increasing number of genome-wide sequencing and fine mapping, extensive data on transcriptomics, proteomics and metabolomics are available in the literature . To mineDistant hybridization is a hybrid between two different species . For thiDifferent from protein (DNA) sequences, the transcriptome of a cell or a tissue is the collection of RNAs transcribed in it, and is often dynamic and a good representative of the cellular state . Ease ofBasically, our new research method originates from optimization techniques, called a nonnegative matrix factorization method (NMF). By this method, we attempt to approximately factorize the small-size and full-length transcriptome sample data of the distant multi-generation hybrid fishes such that their classification and the gene-expression characteristic of each class can be revealed. As a result, it is associated with solution of large-scale optimization problems with nonnegativity constraints. Therefore, we also aim to develop an efficient algorithm for solving this large-scale optimization problem.k-mean clustering method and the principal component analysis (PCA), are inappropriate to analyze this type of data (8 samples with 20093 features). As stated in  of the distant multi-generation hybrid fishes, which constitutes three different groups corresponding to the three sampling periods. Each group consist of 20093 genes expression levels of eight different fish. Actually, the classical statistical methods, such as tated in , if the tated in , the bootated in , the Jactated in , the SVMtated in , the gretated in , and thetated in . Howeverk-mean clustering method and PCA. However, the performance of NMF depends significantly on the development of efficient algorithms to solve the generated large-scale optimization problem such that the deviation of nonnegative matrix (sample data) factorization is minimized. Especially, if we need to classify 8 full-length transcriptome data of distant multi-generation hybrid fishes, it is necessary to factorize a matrix in R20093×8. Suppose that there are r classes of fishes, then the number of design variables is 20093 × r + 8. For solving such a large-scale optimization model, it is still a challenge to develop an efficient algorithm. In this research, we intend to modify the spectral conjugate algorithm in  and Culture Alburnus or Topmouth culter  at sexual maturity in natural waters of the Yangtse River in China were collected for creating the allodiploids BT (BSB (♀)× TC (♂) ) and TB ( TC (♀) × BSB (♂) ) F1 individuals through intergeneric reciprocal crosses of BSB and TC, respectively. Then, the allodiploid F2 − F3 (2n = 48) hybrid offspring were obtained by self-mating of F1 – F2 populations, respectively. The chimeric offspring was identified based on 45S rDNA sequencing characteristics  with the default parameters .The collected data of 24 samples (liver tissues) of the distant multi-generation hybrid fishes in this research have been uploaded to A ∈ nR × m, we need to decompose A into two nonnegative matrices W and H, i.e.We first propose an optimization model for classifying the hybrids fishes on the basis of NMF. Mathematically, NMF is stated as follows. For a given matrix W ∈ nR × r and H ∈ rR × m. In particular, if the matrix A in (2.1) is the full-length transcriptome data of the distant multi-generation hybrid fishes, and A = WH, then r can represent the number of classes for this classification of fishes in the case that each column of H has only a unique element 1, while the other elements are zeros. Clearly, in this ideal case, the k-th column of W stands for the gene expression level of the k-th class of fishes, and its elements show the expression levels of different genes for each class. Therefore, W in Model (2.3) is called a base matrix in view of its practical meanings, while H is called a coordinate matrix.where A = WH by A ≈ WH. In this case, each column of the matrix A is approximately equal to the linear combination of all column vectors of the matrix W, and the combination coefficients are given by the corresponding column vector of the matrix H, i.e. A:,j denotes the j-th column of the matrix A, W:,k stands for the k-th column of the matrix W, and k,jh represents the element of the k-th row and the j-th column in the matrix H. In other words, For real sample data, it is often difficult to obtain the above ideal result of factorization. Therefore, we relax R ∈ rR × m:Thus, if we define a membership matrix j-th column of R represents the membership degrees of the j-th sample being affiliated all the different classes. Therefore, for all the samples, distinct differences of all the elements in each column of R imply an approximate classification result. By definition, the matrix R shows the result of classification in term of membership degrees, while each column of the matrix H exactly stands for the coordinate of each sample in the r-dimensional space linearly expanded by the r columns of W. In the case that all the r elements in each row of W have the same orders of magnitude, the classification results by H or R are same.Clearly, the n is very large, let alone the requirement of finding the unknown optimal number of classes r. To solve Problem (2.1), we first transform (2.1) into the following optimization model:Unfortunately, it is very difficult to solve Problem (2.1) when F is the Frobenius norm. It has been shown that (2.3) is non-convex and NP-hard  and (2.5). Since both of them are large scale (the size of the problem is over 80000), we will extend the spectral conjugate gradient algorithm in  to solveF in (2.4) and (2.5) with respect to the matrices W and H, respectively.In need of modifying the developed algorithm in  such thai and j,By direct calculation, it is easy to see that for any F with respect to the matrices W and H, respectively: Then, we denote the following two matrices the gradients of S and T with the same size, we define their inner product byFor two given matrices k = 0, a search direction of F at a given initial point W(0) isThen, for k ≥ 1, we define four matrices:And for H(k), W(k) and W(k − 1) are two given matrices. Similar to , determined bywhere H(k).The following algorithm is developed to solve the subproblem (2.4) with the given W and H by H and W in Algorithm 1, respectively. Particularly, we need to computeSimilarly, to solve the subproblem (2.5), we only need replace andwhereandWith the above preparation, we now develop an overall algorithm to solve Model (2.3) in the end of this section.Algorithm 2, we present the results on classification of the distant multi-generation hybrid fishes based on their transcriptome data.In this section, in virtue of Model (2.3) and With the given transcriptome data of the distant multi-generation hybrid fishes, we easily get Model (2.3). Then, we implement Algorithm 2 to solve this model by choosing the same values of algorithmic parameters as in :Wρ, H ρ∊  we can obtain the almost same results in our numerical experiments, which indicates our algorithms are robust for classifying the fishes.In addition, for any choice of, https://github.com/TJY0622/TJY.All codes of the computer procedures are written in MATLAB and run in a MATLAB R2016b, and are carried out on a PC with the Windows 10 operation system environment. All the codes have been uploaded to L2 as a training set, which were from the liver tissue of eight different fish. Since it is unclear how many classes can be identified for the fish samples before our research, we make a trial setting on the number of classes r = 2, …, 7 such that the best number of classes is found.For the sake of better understanding the inherent characteristics of the data, we take the 2nd-group samples with superscripts In r = 6, all the samples are clearly classified owing to existence of greater deviation of elements in the same column of H. In contrast, when r is equal to the other values, there are at least one sample that can not be clearly classified. r = 6, As r = 6 as the number of classes to calculate the membership matrix R defined by (2.2). The numerical results are listed in For the sake of better understanding the above classification result, we use H or by R, r = 6, we choose the 1st-group and the 3rd-group samples  as two test sets to see whether the results are the same or not.To further test robustness of the above trained results, given In W have different orders of magnitude for the 1st-group samples, which can explain inconsistence of the classification results by H and R for the 4 samples: From To further validate the proposed model and algorithms in this paper, we use them to classify more test samples generated by mixing the training set and the test sets.We first mix the training set and the 3rd-group test set. The obtained results are listed in Next, we compute the classification result of all 24 samples . The results are given in In summary, by all of the above test experiments, the average accuracy rate is 75.52% even if there exists larger detection error of the input initial sample data . These tests further verifies that the proposed model and algorithm in this paper can be used to efficiently classify the distant multi-generation hybrid fishes based on their transcriptomic profile.To find out the reasons why the replicated samples are incorrectly classified such that the accuracy rate may be reduced, we calculate the correlation matrix of the sample data to reveal possible detection errors of the input initial data. In From From Similarly, because the sample of For the same reason of weaker correlation, in lass see 5, whichW gives the feature of gene expression for each class of fish. Since the sample of each class consists of 20093 genes, we only list a part of the highly expressed genes for each fish. When r = 6, the highly expressed genes are reported in In the end of this section, based on our classification result from the 2nd-group samples, we answer what are the differently expressed genes in all the six classes. By definition, we know that each column of the base matrix From the numerical results in Apart from one-by-one comparison in It is also noted that in In addition, from A is not unique. In particular, if we choose different initial matrices W0 and H0, the base and coordinate matrices W and H may be different. However, our numerical experiments show that for Algorithms 1 and 2, different choices of W0 and H0 do not affect the final result of classification. For example, as r = 6, the result of classification always is the same for any W0 and H0, which can show robustness of our classification method.In our numerical experiments, it is found that the nonnegative factorization of the matrix Hybridization is considered as the rapidly driving forces that shape epigenetic modifications in plants and parts of lower vertebrate . The merIt is noted that in this research, we choose the 2nd-group samples as the training set, instead of the 1st-group or 3rd-group, and the latter is regarded as test samples to verify the trained result. One of the reasons for our doing so lies in that correlation analysis of the three-group samples indicates that each sample in the second-group is better correlated with the other replicated samples than those in the other two groups.The proposed model and algorithms in this paper can be extended to solve more practical engineering problems from other fields. For example, if we can collect sufficient transcriptome data of patients possibly suffering from breast cancer, we can apply the proposed model and algorithms to identify the classes of patients, even development of the relevant smart aided-system of diagnosis for the sufferers.In this paper, we have constructed a classification model for the distant multi-generation hybrid fishes based on transcriptome data, and developed an efficient algorithm, called the modified spectral conjugate gradient algorithm, for solving such a model.In virtue of our model and algorithm, we have obtained a satisfactory classification for a given full-length transcriptome data of fish samples, and the differently expressed genes of each class have been identified. Our results are first obtained by a training set of samples, then are tested by many test samples generated by different ways.Main results are stated as follows.Even for input data with larger detection error, the average accuracy rate of classification still achieves 75.52% in all the test experiments. It suggests that our model and algorithms are promising in classifying the distant multi-generation hybrid fishes.Owing to the weakest intersection of highly expressed genes between BSB and TC, they are deterministically divided into two classes. However, there exists a higher transcript intersection between them and their hybrids. These findings have further deeply mined the biological genetic characteristics of distant hybridization generated by BSB and TC, based on optimization techniques and transcriptome data.Although the hybrids of TC and BSB have been divided into different classes, the hybrids display higher transcript intersection. Since the transcript intersection of the hybrids and the parents is smaller than that among the hybrids, it can be concluded that the hybrid progeny of TC and BSB has significant hybrid characteristics, which may be useful to carry out trait improvement in practice.Since Since there are no shared genes of high expression between http://www.ncbi.nlm.nih.gov/bioproject/) under accession numbers PRJNA269572. All raw mRNA-seq data were downloaded from the NCBI Sequence Read Archive (http://trace.ncbi.nlm.nih.gov/Traces/sra/) under accession number SRP050891.The genome assembly used in this study was downloaded from NCBI BioProject database , National Key Research and Development Program of China (2018YFD0901202), National Science Foundation of China (31772902), and State Key Laboratory of Developmental Biology of Freshwater Fish (2018KF003).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Gene expression profiling has been widely used to characterize cell status to reflect the health of the body, to diagnose genetic diseases, etc. In recent years, although the cost of genome-wide expression profiling is gradually decreasing, the cost of collecting expression profiles for thousands of genes is still very high. Considering gene expressions are usually highly correlated in humans, the expression values of the remaining target genes can be predicted by analyzing the values of 943 landmark genes. Hence, we designed an algorithm for predicting gene expression values based on XGBoost, which integrates multiple tree models and has stronger interpretability. We tested the performance of XGBoost model on the GEO dataset and RNA-seq dataset and compared the result with other existing models. Experiments showed that the XGBoost model achieved a significantly lower overall error than the existing D-GEX algorithm, linear regression, and KNN methods. In conclusion, the XGBoost algorithm outperforms existing models and will be a significant contribution to the toolbox for gene expression value prediction. Characterizing gene expression patterns in cells under various conditions is an important problem . Gene exGenes expression are closely related, and some methods for gene co-expression have also been extensively studied in recent years to further explore the relationship between gene expression. . ConsideHowever, this task is very difficult because, in principle, gene expression value prediction is a multi-task regression problem. In 2016, Yifei Chen et al. proposed the D-GEX algorithm based on Back Propagation neural network , in whicIn addition to deep network, some researchers also used linear regression, KNN and other classical algorithms for target gene expression prediction , but theAmong the Boosting Tree models, XGBoost  has a veBased on the above research background, we proposed a new gene expression value prediction algorithm based on XGBoost, and established a regression prediction model for each target gene independently. The results showed that the XGBoost algorithm significantly improved the prediction accuracy, which is superior to D-GEX, LR, KNN, and other algorithms. It also had better predictive ability and generalization ability. Lastly, the XGBoost algorithm had stronger interpretability than other algorithms.In this section, we first introduced the dataset we used for this task. Then, we gave an introduction of XGBoost algorithm, and finally, we showed three competing methods.The dataset used in this paper is the same as the dataset used by Yifei Chen et al. in the proposed D-GEX algorithm in 2016, which is the GEO  dataset selected by the Broad Institute from the published gene expression database , and theThe GEO dataset has a total of 129,158 gene expression profiles of cell line samples, and it should be noted that we refer to each profile as a sample in this article. The original GEO dataset was generated by the Affymetrix microarray platform, and the expression values are in a numerical range between 4 and 15. Since some of the samples are repetitive or highly similar, we first removed the duplicate samples from the 129,158 samples in order to avoid unnecessary calculations. All samples were clustered into 100 classes using the k-means algorithm . In eachWe used the training set to train the models, and adjusted the parameters based on the performance on the validation set. Finally, we used the results on the test set to evaluate the model.We also performed experiments on RNA-Seq expression data to further evaluate the reliability of the model. The RNA-Seq expression data includes GTEx expression data and 1,000 Genomes expression data . The GTEx expression data consist of 2921 profiles, which were obtained from various tissue samples , and theLike Chen et al. designed before, we still used the training set of the GEO dataset as the training set, then used 1,000G data as the validation set, and finally employed GTEx dataset as the test set to further evaluate the generalization ability of the models based on this cross-platform experiment .However, the GEO dataset and the RNA-seq dataset were obtained from different platforms, so the numerical scales were different as well. Therefore, we performed quantile normalization on all the datasets, which means that all the datasets were standardized by subtracting the mean and then dividing by the standard deviation of each gene .XGBoost (Extreme Gradient Boosting) is a model that was first proposed by Tianqi Chen and Carlos Guestrin in 2011 and has been continuously optimized and improved in the follow-up study of many scientists . The modthn tree, it is difficult to implement distributed training because the residual of the former n-1 trees is used. XGBoost performs a second-order Taylor expansion on the loss function and it can automatically use the multithreading of the CPU for parallel computing. Besides, XGBoost uses a variety of methods to avoid overfitting.The traditional Boosting Tree models uses only the first derivative information. When training the The XGBoost algorithm is briefly introduced as follows , and theK trees, and use F to represent the basic tree model, then:Integrate the tree model with addition method, assuming a total of The objective function is:l is the loss function, which represents the error between the predictive value and the true value; Ω is the function used for regularization to prevent overfitting:where T represents the number of leaves per tree, and w represents the weight of the leaves of each tree.where After the second-order Taylor expansion of the objective function and other calculations which are detailed in γ is added. The leaf node is allowed to split if and only if the information gain is greater than γ. This is equivalent to pre-pricing the tree while optimizing the objective function.As can be seen from (4), in order to suppress the growth of the tree and prevent the model from overfitting, a splitting threshold In addition, we also used the following two excellent techniques of XGBoost to avoid overfitting in the experiment:If all sample weights on the leaf nodes are less than the threshold, the splitting is stopped. This prevents the model from learning special training samples.Sample features randomly when building each tree.These methods all make XGBoost more generalizable and get better performance in practical applications.In the experiment, the regression model based on XGBoost was independently trained for each target gene, and the number of input landmark genes was 943, which means the input feature dimension was 943, and this dimension is very high. However, many techniques in XGBoost for avoiding overfitting can help reduce the degree of overfitting and improve the accuracy of regression prediction.When the XGBoost model was actually used in the experiment, the following parameters were adjusted to make the model perform its best performance:n_estimatorsn_estimators is the number of iterations in training. A too small n_estimators can lead to underfitting, which makes the model not fully perform its learning ability. However, a too large n_estimators is usually not good either, because it will cause overfitting.min_child_weightmin_child_weight defines the sum of sample weight of the smallest leaf nodes to prevent overfitting.As we mentioned earlier, max_depthIt is the maximum depth of the tree. The greater the depth of the tree, the more complex the tree model is, and the stronger the fitting ability is, but at the same time, the model is much easier to overfit.subsampleThis parameter means the sampling rate of all training samples.colsample_bytreecolsample_bytree. It is the feature sampling rate when constructing each tree. In this task, this is equivalent to the sampling rate of the landmark gene.The last parameter that we need to config is learning_ratelearning rate is a very important parameter that needs to adjust, as well as in XGBoost. It greatly affects the performance of the model. We can reduce the weight of each step to make the model more robust.In most algorithms, The details of parameters configuration were introduced in Section 3.There are other methods that researchers have previously proposed that could be used in the gene expression value prediction task. In this section, we briefly describe these methods, and in next section, we evaluate the performance of XGBoost model by comparing the predictive results of XGBoost model with results of these existing models.D-GEX  is the aBesides, the network used mean square error as the loss function:T was the number of target genes and N was the number of training samples. The D-GEX algorithm selected one, two, or three hidden layers, respectively. The number of neurons in each hidden layer of the network was all the same, 3,000, 6,000, or 9,000,respectively. In addition, they added Dropout Layer (where ut Layer  to the nut Layer  was usedA linear regression model was independently established for each target gene t as follows :(t)w and tb can be calculated by the following formula:where On the basis of (16), by adding the L1 or L2 regularization term, the LR-L1 model and LR-L2 model can be obtained.KNN is a non-parametric learning algorithm. For each target gene, the training samples were used to calculate the Euclidean distance of this target gene to all the landmark genes during training, and the k landmark genes with the smallest Euclidean distance were determined as the k-nearest neighbor landmark genes of the target gene . The aveThe range of k value we tried in the experiment were integers between 2 and 20. We found that when the k value changed from 2 to 5, the prediction error was gradually decreasing; and from 5 to 20, the error was gradually increasing. Therefore, the optimal k value we found in the KNN model is 5.In this section, we firstly introduced the process of parameters configuration of XGBoost algorithm and its high interpretability. Then, we showed the results of XGBoost model on both the GEO data and the GTEx data, and compared it with the previous methods.GridSearchCV, a sub-module of the sklearn module in Python , the more important the feature is.In general, importance scores measure the value of features in tree model construction. In the experiment, we trained six models: LR, LR-L1, LR-L2, KNN, D-GEX, and XGBoost, respectively on the training set, and optimized parameters according to the performance on the validation set. Finally, we evaluated the prediction ability of various models according to their performance on the test set.t, we define the Mean Absolute Error as follows:For each target gene N is the number of samples.where Besides, we further explored MAE score in In addition, we define overall error as follows, which represents the mean value of MAE on all target gene:where N is the number of samples and T is the number of target genes.To further study the practicality of XGBoost model in this task, we conducted a cross-platform experiment the same as Chen et al. . We usedThe overall errors on the RNA-seq expression data further indicate the XGBoost model surpassed all the other learning models. Although for this specific task, the training set and the test set were generated from different platforms. This suggested that the XGBoost model performs well in this task and has a good generalization ability.The gene expression value prediction algorithm based on XGBoost outperforms the D-GEX algorithm, and is better than the traditional machine learning algorithms such as Linear Regression and KNN.In the task of predicting gene expression values, the number of landmark genes is large, which leads to the high dimensionality of input features. This makes the model very easy to fall into overfitting. For the deep network of D-GEX, not only the input dimension is very high, the output dimension is even higher. Therefore, it is difficult to train a very accurate model, and the processing of parameter adjustment is extremely complicated as well. Apart from this, poor interpretability is also a disadvantage of deep network.In the XGBoost algorithm, the control of the complexity of the model is added. Random sampling of samples and features during training time makes the trained model less likely to overfit, which improves the generalization ability of the model, and eventually, the predictive errors for the validation set and test set are significantly reduced. Furthermore, XGBoost is more focused on the interpretability of the model, so we can learn which landmark genes have greater influence on the expression value of each target gene.At the same time, although there is a serial relationship between trees in the XGBoost algorithm, the same level nodes can be parallelized, and the multi-threading of the CPU is automatically used for parallel computing, which makes the XGBoost model faster than traditional tree models, and the XGBoost model has a higher practical value.All datasets generated and analyzed for this study are included in the article/HZ conceived the research. WL, YY, HZ, and XQ designed the research. WL implemented the research. WL, HZ, and YY wrote the manuscript. All authors read and approved the final manuscript.This study is supported by the Major Program of the National Social Science Fund of China (Grant No. 18ZDA362).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


RNAs play important roles in the pathology processes of multiple cardiovascular diseases. Additionally, lncRNAs can function as ceRNAs by sponging miRNAs to affect the expression level of mRNAs, implicating in numerous biological processes. However, the functional roles and regulatory mechanisms of lncRNAs in heart failure are still unclear. In our study, we constructed a heart failure‐related lncRNA‐mRNA network by integrating probe re‐annotation pipeline and miRNA‐target interactions. Firstly, some lncRNAs that had the central topological features were found in the heart failure‐related lncRNA‐mRNA network. Then, the lncRNA‐associated functional modules were identified from the network, using bidirectional hierarchical clustering. Some lncRNAs that involved in modules were demonstrated to be enriched in many heart failure‐related pathways. To investigate the role of lncRNA‐associated ceRNA crosstalks in certain disease or physiological status, we further identified the lncRNA‐associated dysregulated ceRNA interactions. And we also performed a random walk algorithm to identify more heart failure‐related lncRNAs. All these lncRNAs were verified to show a strong diagnosis power for heart failure. These results will help us to understand the mechanism of lncRNAs in heart failure and provide novel lncRNAs as candidate diagnostic biomarkers or potential therapeutic targets.Heart failure has become one of the top causes of death worldwide. It is increasing evidence that lnc Various cardiovascular diseases could increase the risks of heart failure, such as pathological cardiac hypertrophy, cardiac ischaemia and so on. Until now, a large number of studies have found that non‐coding RNAs (ncRNAs) play crucial roles in the biological processes of heart failure, implicating in activation of pathway or key protein. For example, SERCA2a gene therapy of heart failure restores miR‐1 expression by involving in Akt/FoxO3A‐dependent pathway, which is associated with normalized NCX1 expression and could improve cardiac function.Importantly, more and more studies have found that lncRNA was a new regulatory factor in cardiovascular diseases.https://www.ncbi.nlm.nih.gov/geo/). StarBase also provided the CLIP‐seq supported miRNA‐target interactions. All these data could enable us to achieve a global analysis to reveal the functional lncRNAs in heart failure.Competing endogenous RNA (ceRNA) is a novel proposed mechanism, which elucidates the regulatory maps between miRNA targets in post‐transcriptional regulation levels.In this study, to investigate the function of lnRNAs in heart failure, we constructed a heart failure‐related lncRNA‐mRNA network by integrating miRNA‐target interactions and differentially expressed information of genes/lncRNAs Figure . First, 22.1We downloaded the gene expression profile of heart failure with the accession number of GSE1145 from GEO database. GEO is a public functional genomics data repository supporting MIAME‐compliant data submissions. Array‐ and sequence‐based data are accepted. Tools are provided to help users query and download experiments and curated gene expression profiles. In our study, we analysed 26 biological samples that contained 11 control samples and 15 heart failure samples from platform GPL570. All samples were collected from patients undergoing cardiac transplantation whose failure arose from different aetiologies and from normal organ donors whose hearts cannot be used for transplants.2.2Recently, study has found that crosslinking and Argonaute (Ago) immunoprecipitation coupled with high‐throughput sequencing (CLIP‐Seq) could help to identify endogenous interactions between miRNAs and their targets.2.3http://www.affymetrix.com) website. We then aligned these probe sequences to the human lncRNA transcript sequence and the human protein coding transcript sequences from GENCODE (http://www.gencodegenes.org/), using BLASTn. The alignment results were filtered by the following steps:Only the probes that matched to one transcript were retained, the probes that matched to both the protein coding transcripts and lncRNA transcripts were removed, thus resulted in two sets of probes‐transcripts pairs.In each set of probes‐transcripts pairs, removed the probes matched to more than one transcript.Each transcript should be perfectly matched to more than three probes.Studies found that re‐analysis of microarray probes could identify new transcripts’ expression data.2.4P‐value < .01 as differentially expressed. Second, we mapped all these differentially expressed lncRNAs and mRNAs to the Ago CLIP‐supported miRNA‐target interactions and extracted the miRNA‐differentially expressed mRNAs interactions and miRNA‐differentially expressed lncRNA interactions. Third, we used hypergeometric test to identify lncRNA‐mRNA pairs at the threshold of P‐value < .01. The P‐value of a candidate lncRNA‐mRNA pair was measured as follows:m represents the total number of miRNAs in starBase database, t represents the number of miRNAs that interacted with the mRNA, n represents the number of miRNAs that interacted with the lncRNA and r represents the number of miRNAs shared between mRNA and lncRNA.First, we calculated the differentially expressed levels of mRNAs and lncRNAs in normal and heart failure samples using SAM test, based on the re‐annotation results. We considered that lncRNA or mRNA with |fold change| >2 or http://www.cytoscape.org/).The heart failure‐related lncRNA‐mRNA network was constructed by merging all these significant lncRNA‐mRNA pairs. Network was viewed by Cytoscape (2.5https://david.ncifcrf.gov/) and PATHWAX (http://pathwax.sbc.su.se/).Many studies have demonstrated that lncRNAs tend to exert functions in a module.2.6disease represents the Pearson correlation coefficient between gene A and gene B in disease samples, Corrnormal represents the Pearson correlation coefficient between gene A and gene B in normal samples.To investigate the crucial role of dysregulated ceRNA interactions between normal and disease status, we developed a new computational pipeline by integrating lncRNA and mRNA expression levels in normal and disease samples. First, we calculated the Pearson correlation coefficients for all lncRNA‐mRNA interactions that were identified from previous step in normal samples and disease samples, respectively. All the lncRNA‐mRNA pairs with Pearson's correlation coefficients >0.5 in normal samples or disease samples were defined as ceRNA pairs. We used the change of expression correlation of the lncRNA‐mRNA pair in disease samples compared with normal samples to define the extent of dysregulation. Specifically, if the ceRNA pair's expression for disease samples shows a more positive correlation than for normal samples, the ceRNA pair is defined as “gain” dysregulation. If the ceRNA pair's expression for normal samples shows a more positive correlation than for disease samples, the ceRNA pair is defined as “loss” dysregulation. And the change of correlation was defined as follows:R of each ceRNA pair, we performed 1000 times permutation for each ceRNA pair by randomly permuting normal/disease sample labels. The significant P‐value of each ΔR was given as the frequency of the ΔR values in random conditions, which was greater than the ΔR value in the real condition. We only reserved the ceRNA pairs with ΔR > 0.5 or P‐value < .05 for further analysis.To obtain the statistically significant Δ2.7i). In this study, random walk that we performed had capacity of restart with probability r in every time step at “Node i.” The random walk with restart was defined as follows:W represents the column‐normalized adjacency matrix of the network, pt is a vector whose size is equivalent to the number of nodes in the network and the i‐th element holds the probability of being at node i at time step t.A random walk in network was defined as an iterative walker's transition from a certain node to a randomly selected neighbour that started from a given node  by choosing a cut‐off. And the receiver operation characteristic (ROC) curve was used for classifier evaluation which was drawn by plotting sensitivity against the false‐positive rate. This procedure was performed using the R package pROC. In addition, we selected same number of lncRNAs as random biomarkers to perform the last step.To evaluate the diagnosis power of the dysregulated lncRNAs, the scoring classifier was constructed. For each lncRNA, we first performed 33.1To identify the differentially expressed lncRNAs and genes in heart failure, we downloaded the gene expression profile from GEO database. According to previous studies, we decided to use a probe re‐annotation strategy to detect lncRNA and gene expressions. In our study, we re‐annotated the probe annotation data for platform GPL570 that is developed by Affymetrix. Specifically, probe sequences were downloaded from Affymetrix. Using the alignment tool Blastn to alignment the sequences between probes and lncRNAs/genes, the probe‐gene, probe‐lncRNA pairs were annotated. As a result, we re‐annotated ~14 000 probe‐gene pairs and ~4000 probe‐lncRNA pairs for platform GPL570.P‐value < .01.We downloaded the heart failure associated gene expression profile on platform GPL570 (accession number GSE1145), which contained more than 100 samples with failure arising from different aetiologies. We only retained 11 control samples and 15 heart failure samples for analysis. Based on the re‐annotation results, we performed SAM test to identify the differentially expressed genes and lncRNAs. Totally, 1081 differentially expressed genes and 993 differentially expressed lncRNAs were identified as the potential regulators in dysfunction heart at the threshold of |Fold change|>2 or 3.2P‐value < .01 and merged all these pairs into the heart failure‐related lncRNA‐mRNA network, including 66 lncRNA nodes and 451 mRNA nodes   Figure C, indica) Figure . Interes) Figure D, indica3.4Many studies have demonstrated that lncRNA often exerted functions by involving in functional modules. Thus, we performed module analysis to identify the lncRNA‐associated functional modules from the heart failure‐related lncRNA‐mRNA network, using bidirectional hierarchical clustering Figure A. In theSimilarly, pathway analysis was also performed on the genes in module 2 Figure E. Result3.5R > 0.5 or P‐value < .05 as the dysregulated ceRNA pairs (see in Methods). As a result, 135 “gain” ceRNA pairs and 178 “loss” ceRNA pairs were identified as dysfunctional in heart failure for further analysis. We merged all these dysregulated ceRNA pairs into the heart failure‐related dysregulated ceRNA network  Pearson's correlation coefficients were calculated for all lncRNA‐mRNA interactions in normal samples and heart failure samples, respectively. Those lncRNA‐mRNA pairs with Pearson's correlation coefficients >0.5 in normal samples or disease samples were defined as ceRNA pairs; (ii) the changes in Pearson's correlation coefficients between normal and disease conditions (“gain” or “loss” dysregulation) were calculated; (iii) 1000 times permutations for step i were performed to identify the statistical significant dysregulated ceRNA pairs. We only retained the ceRNA pairs with Δk Figure A.2+ ATPase gene expression.We analysed the topological features of the dysregulated ceRNA network. All nodes in network followed a power law distribution Figure B, sugges3.6http://www.disgenet.org/web/DisGeNET/), which is a platform that integrates information on gene‐disease associations from several public data sources and the literatures. We mapped 16 disease genes into the network and scored for lncRNA nodes by random walking. To obtain the significances for scores of lncRNAs, we performed 3000 time permutations and only reserved the lncRNAs with the P‐value < .05. As a result, we found three significant lncRNAs , which were highly correlated to disease status. In Figure 4Heart failure is a common outcome of various cardiovascular diseases, such as cardiac hypertrophy, myocardial infarction and has become a leading cause of death worldwide. Thus, it is essential to discover the effective therapeutic targets for heart failure. During the past years, great efforts have been made to provide the novel insights into the molecular mechanisms for decoding heart failure. But more studies have focused on elucidating the molecular mechanism that involved in protein‐coding genes and miRNAs.In this study, we performed a computational analysis to investigate the function of lncRNAs in heart failure. We constructed a heart failure‐related lncRNA‐mRNA network through integrating Ago CLIP‐seq supported miRNA‐target interactions and differentially expressed information of genes/lncRNAs. First, we performed network topological analysis and identified several lncRNAs with important topological structures. After performing bidirectional hierarchical clustering, two modules that contained more lncRNA‐mRNA interaction clusters were identified from the network. We found the two modules were enriched in pathways that were highly related to heart failure, suggesting lncRNA might exert their regulatory functions by implicating in pathways. Moreover, based on the novel ceRNA mechanism, we calculated the Pearson correlation coefficients for all lncRNA‐mRNA interactions in normal and disease status, respectively. We identified the dysregulated ceRNA interactions in normal or disease status, which could play a “switch” role in pathological processes.2+ ATPase gene expression.Importantly, we investigated the dysregulated ceRNA interactions that might play “switch” role in heart failure. Recent studies have demonstrated that ceRNAs exerted functions in certain disease or physiological status. Paci et al constructed normal and cancer networks of miRNA mediated lncRNA‐mRNA interactions, using breast cancer expression data.However, our study also had some limitations. First, we used a probe re‐annotation pipeline to identify heart failure‐related functional lncRNAs, which has been widely used by many bioinformatics studies. However, we had to admit this pipeline filtered out many lncRNAs that miss matched to the probe sequences. Recently, because of the insufficient open data resource, we could not obtain more valid data. Like the microarray platform GPL570, only a few studies and biological samples focused on heart failure. This probe re‐annotation pipeline could indeed help researchers re‐analyse the microarray data to get both mRNA expression data and lncRNA expression data. In future, with the next‐generation sequencing data increasing, we would add more heart failure‐related lncRNA data information for more detailed analysis. Second, we integrated gene expression and miRNA‐target interactions to identify dysregulated ceRNA interactions. If a more accurate algorithm was proposed, our results will be more stable. Third, as a research in the bioinformatics field, this work aimed at verifying accuracy and reliability of the lncRNA‐mRNA network, the lncRNA‐associated functional modules or the diagnostic potentiality of the biomarker lncRNAs often through the way of statistical significances and scientific literature validation. In future studies, if conditions allowed, we would add experimental design to perform more detailed mechanism research and further validate our conclusions. However, to validate the expression of risk lncRNAs in heart failure, we further downloaded another six heart failure‐related gene expression data sets from GEO database and performed the up/down‐regulated analysis for lncRNAs. Results demonstrated that most of the 19 lncRNAs showed a similar expression trend in at least three datasets Table , indicatIn summary, we performed a network analysis to identify functional lncRNAs by multiple dimensions in heart failure. First, we identified the lncRNAs with the central topological features. Second, bidirectional hierarchical clustering identified some lncRNAs functioned in modules. Third, we identified the lncRNA‐associated dysregulated ceRNA interactions. And we performed a random walk algorithm to identify more heart failure‐related lncRNAs. Additionally, all these lncRNAs showed a strong diagnosis power for heart failure. These results will help us to understand the mechanisms of lncRNAs in heart failure and provide novel lncRNAs as candidate diagnostic biomarkers or potential therapeutic targets.The authors declare that they have no conflicts of interest to disclose. Click here for additional data file. Click here for additional data file. Click here for additional data file.


The Nanostring nCounter gene expression assay uses molecular barcodes and single molecule imaging to detect and count hundreds of unique transcripts in a single reaction. These counts need to be normalized to adjust for the amount of sample, variations in assay efficiency and other factors. Most users adopt the normalization approach described in the nSolver analysis software, which involves background correction based on the observed values of negative control probes, a within-sample normalization using the observed values of positive control probes and normalization across samples using reference (housekeeping) genes. Here we present a new normalization method, Removing Unwanted Variation-III (RUV-III), which makes vital use of technical replicates and suitable control genes. We also propose an approach using pseudo-replicates when technical replicates are not available. The effectiveness of RUV-III is illustrated on four different datasets. We also offer suggestions on the design and analysis of studies involving this technology. The Nanostring nCounter gene expression platform has become widely used for research and clinical applications due to its ability to directly measure a broad range of mRNA expression levels without cDNA synthesis and amplification steps . HoweverThere will always be random errors and possibly also systematic errors in any measurement process. Systematic errors are also called bias, and arise when the measurement process tends to measure something other than what was intended, e.g. . In manyThis does not mean that we cannot find generally good normalization methods, but that we should focus more on assessing the effectiveness of any normalization with each dataset and goal. How do we do this? In this paper not only do we present our novel normalization for nCounter gene expression data, but in discussing its possible value, we also illustrate a number of methods we know for assessing normalizations. They range from statistical summaries such as principal component analysis (PCA) plots , relativThe normalization method presented here, RUV-III, is a novel extension of previously reported methods, which exploit positive and negative control genes and technical replicates ,11. Hereaverage log count of all eight NEG transcripts, of all six POS transcripts and of all of the HK genes, together with library size in one nCounter assay, and plotting these three averages and the log for that assay, all suitably offset vertically to minimize overlap, in the order in which the assays were run, or a related order. Trends, temporal clustering, or other patterns in these plots can reveal issues that normalization should address.These are obtained by taking the Here we depict the location of these replicates in time, and in relation to known batches, using lines connecting replicate pairs, triples, etc. When connecting lines are viewed together with Average and RLE plots (see below), we can get a sense of how well differences between replicates can help remove batch effects. From seeing where there are no connecting lines, we can also get a sense of how suitably defined pseudo-replicates (see below) might help.A common representation of the results of studies like those discussed in this paper is the collection of boxplots of the log(count)s from each assay, lined up in the order in which the assays were run. Again this permits trends, temporal clustering or other patterns to be visible, and it allows outliers to stand out. The RLE plot is a more sensitive modification of this representation, as it removes the variation between mean gene counts. Instead of making a boxplot of all the log(count)s in each assay, the quantities going into the boxplots are relative to the gene-wise median counts. More fully, instead of log(count), we use log(count)—median[log(count)], where for each transcript (gene), the median over all the assays is taken and its log subtracted. Equivalently, the log(count) for a given transcript, is median corrected, where we use the median rather than the mean to reduce the influence of extremes. For a fuller discussion of RLE plots, see .The principal components  of the sample × transcript array of log(count)s are the linear combinations of the transcript measurements having the largest, second largest, third largest,… variation, standardized to be of unit length and orthogonal to the preceding components. Our PCA plots are of the second versus the first principal component (PC) of the sample x transcript array. The calculations are done on mean-corrected transcript log counts, using the R code adopted from the EDAseq R package .1/count2)] of the counts from the two assays. These log ratios can then be pooled across all transcripts in the assay, for a given pair of replicates, and across all technical replicate pairs in a study, and a boxplot created out of them. This we call a TRA boxplot. The boxplot, in particular its inter-quartile range, summarizes the average closeness of technical replicates. It provides a simple visual metric for assessing and comparing normalizations. If, as in RUV-III and some other normalizations, explicit use is made of technical replicates, then the contribution of a given replicate pair to the TRA plot must be calculated after normalization without that pair being used, i.e. with them left undeclared as replicates. All TRA plots calculated after RUV-III in this paper were formed this way.If two assays 1 and 2 are technical replicates, that is, independent assays of two samples of cells from the same tissue sample or subject, then for any transcript (gene) we can form the log ratio log[ for that transcript, where FC is the fold change and p denotes the associated (unadjusted) P-value for the test of the null hypothesis of no change. These plots provide a convenient visualization of the efficacy of the study and the location of controls; ideally negative controls should be near , while positive controls should be well away from 0 on both axes.A volcano plot is a summary of the results of a two-condition differential expression analysis, where each point corresponds to a transcript, the vertical axis being—P-values will be uniformly distributed over , while those transcripts (genes) that are DE will have P-values close to zero. As a result, we usually expect that the histogram of P-values from a DE study should look uniform, with a possible spike near zero. The presence of unwanted variation will frequently destroy this expected shape. If the number of transcripts (genes) is just a few hundred, and they were pre-selected to exhibit the DE under study, then the expected shape may not result, even in the absence of unwanted variation, and so extra care in interpretation is needed in this case.In a two-condition differential expression study with hundreds or thousands of transcripts (genes), the majority will in general not be differentially expressed (DE), in which case their (unadjusted) not expected to change much across the samples being analyzed. Second, we need to emphasize that our approach to negative controls is pragmatic: if using a given gene in RUV-III as one of the set of negative control genes helps, as indicated by various measures, then whether or not it is an ideal negative control gene is moot: it helped. This does not rule out the fact that we may be able to do better by replacing that gene with a different gene designated a negative control. Third, we point out that theoretical analyses not presented here show that it is not the extent to which individual genes designated as negative controls are ideal or less than ideal negative controls that drives the success or otherwise of RUV-III; that is a property of the full set of negative controls. We will frequently get very good results using the entire set of genes being studied as negative controls, even when many genes are changing. (That this is not unreasonable follow from the theory just mentioned.) Fourth, it is usually the case that using more genes as negative control genes is better than fewer. That is a matter of stability, but as stated in the introduction, there is a bias-variance trade-off here: using too many genes as negative controls may be counter-productive. You must look and see. Fifth, endogenous genes generally make more suitable negative controls than spike-ins. The reason here is the obvious one, namely, that endogenous genes have shared the complete sample experience of the other genes, whereas spike-ins can only reflect unwanted variation in the process from the point at which they were added onward. The best source of endogenous negative control genes are ones that were found to be stable in previous studies similar to the one being analyzed. Given that Nanostring expression assays are frequently designed following initial, genome-wide microarray or RNA-seq assays, it should be straightforward to include 20–50 endogenous likely negative controls. We believe that the results in this paper show that their presence for use in normalizing the data will definitely justify the extra expense. Suitable generic sources of negative controls include those genes that are traditionally regarded as HK genes . Gender related genes, such as those on the Y chromosome, can double as negative and positive controls in a gender DE. Having said this, it can at times be quite challenging to find suitable endogenous negative controls in Nanostring studies, especially where there are few ‘HK’ genes, and where no prior studies are available. Sometimes it becomes a matter of trial and error, of doing the best you can.First, we need to reiterate that in our usage, a negative control gene is one that is take residuals from the replicate expression measurements and estimate one aspect of the unwanted variation;take the results of (i) together with the expression values of the negative controls, and estimate another aspect of the unwanted variation; andcombine the results from (i), (ii) into an estimate of the unwanted variation and subtract that from the data.https://cran.r-project.org/package=ruv), respectively.The dimension (k) of the unwanted variation also needs to be determined. Full details of the method and R code are given in the Supplementary Method (supplementary file) and the ruv R package . Informally, the three steps that make up RUV-III are the following:To use RUV-III, we need to fix the number k of (linear) dimensions of unwanted variation to be removed. As with the set of negative controls, and the replicates (or pseudo-replicates), there are no clear rules for determining k. Our general approach is to repeat the analysis with different values of When a study has several batches, it is highly desirable to have replicates placed so that the unwanted variation between any pair of batches can be captured, directly or indirectly, via differences of expression values between technical replicates. For example, if there are three batches, and (for instance) there are samples with technical replicates present in all three, then all gene-wise batch–batch differences can be estimated using differences of these technical replicate samples.  But we need not have samples with technical replicates in all three batches: some which span batches 1 and 2, and others which span batches 2 and 3 will suffice, for the difference between batches 1 and 3 can be estimated by the sum of a 1–2 and a 2–3 difference. Again how well it can be estimated is a different matter. This observation generalizes in a natural way, which we do not spell out at the moment. Roughly, the best results will come when different batches can be ‘connected’ by a short chain of replicate pairs. To see this point, suppose that we have two batches, and that all the technical replicates lie wholly within batch 1, or wholly within batch 2, with no replicate pair linking batches 1 and 2. How then can we adjust all gene expression measurements for any bias in the batches? The answer is that we can’t do so directly for all genes, as we could when the two batches were ‘connected’, we can only get a direct measure of the 1–2 difference for our negative control genes. When the two batches are not connected by a replicate pair, we may do a poor job removing batch differences using RUV-III.pseudo-replicates. This is precisely what we did with the dendritic cell example (see below).What can be done in this case? Suppose that there is a pair of samples with one in batch 1 and the other in batch 2, and that we believe that the batch differences between these two samples of most genes are greater than the biological differences. We might have reason to believe that the batch 1–2 difference is really large, for example, by looking at the RLE plot. If we designate this pair of samples a replicate pair, and use RUV-III, we may find ourselves coming out ahead. That is, we may be able to use the standard battery of graphical diagnostics and positive controls to convince ourselves that the resulting estimate of the batch 1–2 difference, while imperfect, was near enough to the unwanted variation we sought to remove, to permit a successful application of RUV-III. We would call the pair of samples so used P-value histograms and comparisons with as much prior information as possible, including positive controls. Normalizing using RUV-III with different sets of negative controls, or making use of pseudo-replicates, if the natural controls or replicates available seem inadequate, is a reasonable approach, which can ‘rescue’ data that is otherwise fatally compromised by unwanted variation.In the preceding discussion, we have suggested that there is no such thing as a perfect negative control, but rather, plausible candidates should be tried, and their value assessed. And we said that, up to a point, the more the better. We have also said that to get RUV-III working well, we would like replicates suitably spanning the major batches or batch-like groups of samples. For example, if there are visible trends in the average or RLE plots, then we would like replicates spanning the trend, not all at one end, or all in the middle. When there are samples in excess, and clearly major pockets of unwanted variation, then creating pseudo-replicates may help. All of which brings up a point that cannot be emphasized too much: Any normalization should be critically assessed using all the tools that can be brought to the task: RLE, PCA, TRA, volcano plots, To evaluate the performance of the RUV-III method on Nanostring gene expression data, we examined 4 studies comprising one in-house and three different published datasets that had technical replicate samples. The details of each study are given separately below.Our in-house Nanostring dataset was part of a study of the expression of DNA repair genes in lung adenocarcinoma (LUAD). The data was generated using 15 nCounter cartridges, with the majority of the normal samples being run in a 15th cartridge a year after the rest of the experiment . For morWe began our analysis with an examination of the Average Plot  of the raw nCounter data . The spik = 6 . In the RLE plot of the raw data  plotted vertically against the log(fold-change) horizontally for each gene counted. More such plots can be found in et al., far more than in the RUV-III normalized data.In order to determine whether the batch differences visible in Figure s Figure  are dispet al. . The RUV-III normalization largely removed the technical variation between the T-cell conditions, but the time effects remained in the data due to not having technical replicate across 2012 and 2013. Enlarging the number of negative control genes from 9 to 100 produced little change  genes with lowest variance in the microarray data as a set of negative controls and all technical replicates to normalize the Nanostring data using RUV-III , Escherichia coli bacterial lipopolysaccharide for 5 h (LPS 5 h), influenza virus for 10 h (FLU 10 h) and interferon beta (INFβ) for 6.5 h. We refer to the supplementary file for fuller details, particularly The final dataset we examined is from the study by Lee et al.  of four We began our analysis with average and RLE plots of the raw and Lee-normalized data , and we In Figure k = 10. The resulting RLE plot is in A differential expression analysis comparing the U-samples with the rest of the samples showed striking heterogeneity of Lee-normalized gene expression data under the UNS condition . By defiWhen we normalized using RUV-III with the pseudo-replicates spanning the U and ULVI sample usage types, the resulting PCA plots split into two clusters on PC1 corresponding to gender for three of the four conditions Figure . It seemNormalization is key to deriving meaningful biological results from complex gene expression datasets, especially when the data is compromised by unwanted variation. A normalization method for Nanostring gene expression data must be able to adjust for sample loading differences, variations in assay efficiency, the use of different Nanostring CodeSet batches and other factors.Our goal in this paper is to compare our new way of normalizing Nanostring data based upon the use of technical replicates with other normalizations in the literature. In order to do so, we made use of our own data on lung adenocarcinoma samples and three published Nanostring datasets. Our comparisons are based on statistical summaries, biological positive controls and concordance with previous results.RRM1, ERCC1) correlation close to that of the RUV-III normalization, but RLE plots for these options include many boxes far from centered (blue), while with RUV-III they are all centered  in the T-cell study by Ye together . This suzed data .Although there were plenty of technical replicates, these cannot bridge conditions, and so we had to rely more heavily on negative control genes for RUV-III. We had great trouble finding suitable ones using just the Nanostring data, but the nine genes exhibiting least variation across conditions in the microarray data worked well, and replacing 9 by 100 was almost as good.There are some quite noticeable differences between the log fold changes obtained using the Ye normalization and the microarray data, whereas the agreement between the RUV-III normalized data and the microarray data is much better Figure . We concet al. resembled that just discussed. We felt that a good normalization of all the data should lead to PCA plots clearly separating the four conditions, and that PCA plots within condition should have two clear clusters corresponding to males and females, ideally defined by PC1. The Lee-normalization did not fully achieve this . All computational source code required for reproducing all results presented in this paper are available in https://github.com/RMolania/NanostringNormalization.RUV-III is implemented in the open-source R package ruv, with source code freely available through the CRAN (https://gdac.broadinstitute.org), data version 2016/01/28; Expression data for none-small cell lung cancer Affymetrix microarray data by Hou et al. (et al. (et al. (et al.(et al. (et al. (The publicly available datasets used in this study are available through the following addresses: The TCGA lung cancer adenocarcinoma RNA-SeqV2 in the Broad GDAC Firehose (u et al. , GEO GSE (et al. , GEO GSE (et al. , GEO GSE. (et al., GEO GSE.(et al.  GEO GSE6 (et al. , GEO GSEgkz433_Supplemental_FilesClick here for additional data file.


Detection of differentially expressed genes is a common task in single-cell RNA-seq (scRNA-seq) studies. Various methods based on both bulk-cell and single-cell approaches are in current use. Due to the unique distributional characteristics of single-cell data, it is important to compare these methods with rigorous statistical assessments. In this study, we assess the reproducibility of 9 tools for differential expression analysis in scRNA-seq data. These tools include four methods originally designed for scRNA-seq data, three popular methods originally developed for bulk-cell RNA-seq data but have been applied in scRNA-seq analysis, and two general statistical tests. Instead of comparing the performance across all genes, we compare the methods in terms of the rediscovery rates (RDRs) of top-ranked genes, separately for highly and lowly expressed genes. Three real and one simulated scRNA-seq data sets are used for the comparisons. The results indicate that some widely used methods, such as edgeR and monocle, have worse RDR performances compared to the other methods, especially for the top-ranked genes. For highly expressed genes, many bulk-cell–based methods can perform similarly to the methods designed for scRNA-seq data. But for the lowly expressed genes performance varies substantially; edgeR and monocle are too liberal and have poor control of false positives, while DESeq2 is too conservative and consequently loses sensitivity compared to the other methods. BPSC, Limma, DEsingle, MAST, t-test and Wilcoxon have similar performances in the real data sets. Overall, the scRNA-seq based method BPSC performs well against the other methods, particularly when there is a sufficient number of cells. Traditional gene expression profiling with high-throughput RNA-sequencing technology measures the aggregated expression levels of genes from a collection of millions of cells. Such bulk-cell RNA-sequencing cannot capture cellular heterogeneity since there is no cell-specific information . Single-Differential expression (DE) analysis to discover quantitative changes between different groups or conditions plays an important role for understanding the molecular basis of phenotypic variation. However, due to the unique characteristics of scRNA-seq data, it is not immediately obvious that we can just use standard methods developed for bulk-cell data. A particular challenge is dealing with the large number of low (or zero) read counts in the scRNA-seq data. A previous study  has showThese bulk-cell−based methods are methodologically sophisticated, and they have been used for scRNA-seq data, but evaluation of their applicability to scRNA-seq data is still uncommon and different studies have reported opposite results. For example, authors in a recent study  comparedTo compare the DE methods, previous studies have used conventional statistics such as type-I error rate, false discovery rate (FDR) and receiver operating characteristic (ROC) curve. Notably, these metrics are applied to the full collection of genes. Reproducibility is also an important metric, although it is sometimes calculated differently in the different studies. For example, a recent study  assessesIn this study, we compare the performance of nine DE methods, including both bulk-cell and single-cell–based approaches as well as general statistical tests not specifically designed for RNA-seq data. We focus on the reproducibility of the methods in terms of rediscovery rate (RDR)  of top-rTo get realistic distributional characteristics and capture some diversity in single-data data, we utilize three real scRNA-seq data sets; in addition, we use simulated data from the beta-Poisson model (BPSC), which has been suggested for scRNA-seq data in a recent study . BecauseWe compare nine methods for detecting differentially expressed isoforms, including edgeR , DESeq2 To get realistic distributional characteristics, the following three real scRNA-seq data sets are used as the basis for simulations. Breast-cancer cell line MDA-MB-231 data set  has two Conquer project (Mouse embryonic stem cells (mESCs) belong to two groups from different culture conditions, 94 cells in group 1 and 174 cells in group 2; see the Materials and Methods section for details. The expression estimates of 112,593 isoforms are provided by the  project .Neuronal progenitor cells (NPCs) also form two groups, one from the patient and the other from a healthy donor , 360 celIn addition, we also simulate single-cell data based on the beta-Poisson model . The varIn each experiment, the comparison focuses on the DE analysis of two predefined groups of cells. Briefly, an equal number of samples is randomly selected from the two groups in the original data set to generate the training set. For each sampled cell from a real data set, all isoforms are taken together; this preserves the statistical dependencies between the isoforms. For the validation set, a different set of samples from both groups is selected. The selection of training and validation sets is repeated 50 times to average out the effect of random selection. Note that the training and validation sets are always disjoint. The nine DE methods are then applied to the training and validation sets separately.p < 0.05). This fraction is also known as the FPR. To highlight the effects of the dropout events, which tend to produce low expression and zero inflation, we split the isoforms into two groups based on the expression level: highly expressed isoforms and lowly expressed isoforms. The former refers to the isoforms with an estimated expression above 1 transcripts-per-million (TPM) in more than 25% of the cells, and the remaining isoforms are assigned to the latter. This threshold was also suggested in a recent comparative study of DE methods in scRNA-seq . Thus, the null data sets are expected to have no true DE isoforms, and the p-value distribution of each method is expected to be uniform. Theoretically, the p-values should follow a uniform distribution if the null hypothesis is true . The unicRNA-seq .Results in For the lowly expressed isoforms, DESeq2 is also the most conservative method, p < 0.05) in the validation set. The RDR is calculated based on the top 5%, 10%, 20% DE and all isoforms in the training set.The RDR is the proportion of the top-ranking DE isoforms in the training set that is found to be significant . In To capture the true distributional characteristics of real data, three real scRNA-seq data sets are used for the evaluation of the nine DE methods. The first data set (MDA-MB-231) includes 160 single cells from a triple-negative breast cancer cell line, half of which are treated with metformin. The cells are captured using the Fluidigm C1 system and sequenced on Illumina HiSeq 2500 machines for 80 control and 80 treated cells separately. Then we use Cufflinks  to estimConquer data set  is collected from a public scRNA-seq data (GSE60749-GPL13112) in the data set , which pdata set . Then thg Salmon . This daThe third real data set (NPCs) is a subset of GSE102934 data from the NCBI Gene Expression Omnibus . This daThe simulated data set for isoform expression of single cells is generated by the beta-Poisson model . In partThe mean and variance of the model can be written asand1 to create mean differences between groups. A more detailed description of the model can be referred to in the original study . Specifically, the parameter λNine DE methods included in this study are categorized into four groups based on different statistical models. These nine methods are selected to cover most statistical models used in recent DE analysis. Regarding other DE methods that are not included in this study, they use similar approach comparing to the nine selected methods. For instance, D3E  utilizesijy ~ NB, where µij is the mean and φi is the dispersion parameter for isoform i and sample j. Reliable estimation of the dispersion parameter φi for each isoform is crucial for detecting DE isoforms. Differences in the estimation of φi explain the main differences between edgeR and DESeq2.The read counts of an isoform from the technical replicates (repeated sequencing runs of the same sample) are usually modeled to follow a Poisson law . Howeveri in average log count-per-million (logCPM). For DE testing, edgeR allows the user to select among different hypothesis tests including quasi-likelihood F-test (edgeRQLF) for bulk-cell RNA-seq data and likelihood ratio test (edgeRLFT) for scRNA-seq data as suggested by the developer. However, a recent study  is used in edgeR  to estimnt study  shows thDESeq2  uses a sH0), and finally uses the likelihood ratio test for testing H0.DEsingle  has anotBPSC  is an anLimma  method iMonocle  is a tooY is a response variable, and ix‘s are predictor variables. The function g is a link function, typically the log function, and if‘s are nonparametric functions, such as cubic splines or other smoothing functions. Gene expression level across cells is modeled by a Tobit model; with some approximations, monocle’s GAM is thuswhere t is the assigned pseudo-time of a cell and s is a cubic smoothing function with (by default) three effective degrees of freedom. ϵ is the error term that is normally distributed with a mean of zero. The DE test is performed with a x2-approximation of the likelihood ratio test.where ψMAST  uses a hT-test  is a genWilcoxon rank sum test   1, 2, 3, and 4 using the simulated dataset. Results in the The nine methods compared in this study are selected to cover most statistical models used in recent DE analysis. Although some DE methods are not included in this study, they use similar approach to those we included. For instance, D3E  utilizesThe main strengths of our comparison method include (i) the use of three real scRNA-seq data sets in order to capture the true distributional characteristics and the diversity of single-cell data; (ii) the use of the RDR metric for top-rank genes. This is consistent with the data analysis process of identifying the list of interesting genes. In some cases we show that considering the full collection of genes will lead to misleading comparisons; (iii) Separate results of highly and lowly expressed genes, as these two groups have distinct distributions and the methods vary more in their performances for lowly expressed genes. In summary, performances of DE methods do vary, so we need to pay attention in choosing the method to use, and, at least for highly expressed genes, some methods designed for bulk-cell RNA-seq analysis do not necessarily perform worse than those specifically designed for scRNA-seq data. Finally, as shown the figures, the number of lowly expressed genes is not trivial, so our results also highlight the need for further development of methods to deal with these genes.There are large differences in the performance of methods for detecting DE in single-cell RNA-seq data. This is driven partly by the expression level of genes. For highly expressed genes, many bulk-cell–based DE methods perform well against single-cell–based methods. But, for lowly expressed genes, the performance of the methods varies, so a careful check of the gene expression level should be made before choosing a DE method in analyses. This is to ensure that the chosen method is appropriate for your data. We found edgeR and monocle to have poor control of false-positives on lowly expressed genes, so we do not recommend these two methods for such genes. DESeq2 tends to be too conservative, so it sacrifices sensitivity for higher specificity. According to the simulation results, BPSC performs well against the other methods, particularly when there is a sufficient number of cells. RDR for top-rank genes is a useful metric for assessing performance of DE methods, sometimes giving different results compared to analysis of the full set of genes. We suggest to be considered in choosing DE methods to use, performances of DE methods in scRNA-seq data strongly depend on the expression level of genes.https://github.com/Tianmou/scRNAseq-DE-comparison.The raw data of two data sets mESC (GSE60749-GPL13112) and NPCs (GSE102934) are published by the original studies and publicly available from NCBI Gene Expression Omnibus repository. The gene expression data of these data sets, MDA-MB-231 data set, simulated data sets and the two supplementary data sets can be found at: YP and TM designed the study. TM, TV, and YP performed the analysis and wrote the manuscript. WD performed the acquisition of MDA-MB-231 data. FG performed a part of simulation studies. All authors read and approved the final manuscript.This work was partially supported by funding from the Swedish Cancer Fonden, the Swedish Research Council (VR) and the Swedish Foundation for Strategic Research (SSF).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Glioblastoma (GB) is the most common and aggressive tumor of the brain. Genotype-based approaches and independent analyses of the transcriptome or the proteome have led to progress in understanding the underlying biology of GB. Joint transcriptome and proteome profiling may reveal new biological insights, and identify pathogenic mechanisms or therapeutic targets for GB therapy. We present a comparison of transcriptome and proteome data from five GB biopsies (TZ) vs their corresponding peritumoral brain zone (PBZ). Omic analyses were performed using RNA microarray chips and the isotope-coded protein label method (ICPL).As described in other cancers, we found a poor correlation between transcriptome and proteome data in GB. We observed only two commonly deregulated mRNAs/proteins (neurofilament light polypeptide and synapsin 1) and 12 altered biological processes; they are related to cell communication, synaptic transmission and nervous system processes. This poor correlation may be a consequence of the techniques used to produce the omic profiles, the intrinsic properties of mRNA and proteins and/or of cancer- or GB-specific phenomena. Of interest, the analysis of the transcription factor binding sites present upstream from the open reading frames of all altered proteins identified by ICPL method shows a common binding site for the topoisomerase I and p53-binding protein TOPORS. Its expression was observed in 7/11 TZ samples and not in PBZ. Some findings suggest that TOPORS may function as a tumor suppressor; its implication in gliomagenesis should be examined in future studies.In this study, we showed a low correlation between transcriptome and proteome data for GB samples as described in other cancer tissues. We observed that NEFL, SYN1 and 12 biological processes were deregulated in both the transcriptome and proteome data. It will be important to analyze more specifically these processes and these two proteins to allow the identification of new theranostic markers or potential therapeutic targets for GB.The online version of this article (10.1186/s12867-018-0115-6) contains supplementary material, which is available to authorized users. MGMT has also been identified through omic analyses: this feature predicts sensitivity to temozolomide, an alkylating agent that is the current standard treatment for GB patients  for 2 h at room temperature. Horseradish peroxidase-conjugated secondary antibodies were then used and visualized with an enhanced chemoluminesence (ECL) reagent and the LAS4000 digital imaging system (Fisher Scientific).TZ and PBZ samples (n = 11) were lysed in RIPA buffer containing a protease inhibitor cocktail and PMSF  at 4 °C for 30 min. The lysates were clarified by centrifugation at 14,000The transcriptome pooled analysis with 41,000 probes of the five TZ vs PBZ samples identified 478 mRNAs differentially expressed between TZ samples and their corresponding PBZ samples. A total of 437 non-redundant genes were identified; 101 genes were over-expressed in the florid TZ, and 300 genes were under-expressed  confirmed the transcriptomic results; in most cases, an under-expression of NEFL and SYN1 proteins was observed in the TZ (9/11 for NEFL and 10/11 for SYN1)  between transcriptomic and proteomic data  and 12 biological processes; they are related to cell communication, synaptic transmission and nervous system processes. These findings are consistent with previous reports , 16, 38.proteome , 40. We nt in GB , 26. FurAlthough NEFL, SYN1 and twelve biological processes were in common between the two omic analyses, they did not show the same deregulation : transcriptome analysis indicated that they are under-expressed in TZ whereas proteome analysis indicated that they are over-expressed. Mismatches of this type between such pairs of datasets has already been described , 25, 26;The low correlation between transcriptome and proteome data is not new and not restricted to cancer tissue –33. For Of interest, the search of TFBSs upstream from the open reading frames encoding the deregulated proteins that may explain the modification of their level of expression revealed a common binding site for TOPORS. We observed through Western blot analysis an expression of TOPORS in 7/11 TZ samples while no expression was evidenced in PBZ. The consultation of the Human Protein Atlas is consistent with our data where a higher expression of TOPORS was observed through immunohistochemistry analysis in 5/11 glioma specimens relative to brain tissue . TOPORS In this study, we showed a low correlation between transcriptome and proteome data for GB samples as described in other cancer tissues. We recognize that the number of studied samples was only five and that only a single sample per patient was used indicating that the results should be considered observational at this time. Future multi-omics studies must be performed on large cohort of patients and on spatially distinct tumor fragments per patient to consider the inter- and intra-heterogeneity of GBs. This may lead to an interpretation accuracy of the overall transcriptomic and proteomic data. We observed that NEFL, SYN1 and 12 biological processes were deregulated in both the transcriptome and proteome data. It will be important to analyze more specifically these processes and these two proteins to allow the identification of new theranostic markers or potential therapeutic targets for GB. Furthermore, a more detailed study of TOPORS which its TFBS was present upstream from the open reading frames of all proteins altered between TZ and PBZ may be promising.Additional file 1: Figure S1. Graphical abstract.Additional file 2: Table S1. Differentially expressed probes between TZ and PBZ pooled samples transcriptomic analysis.Additional file 3: Table S2. List of altered biological processes in pooled GB transcriptomic and proteomic profiles.Additional file 4: Table S3. List of identified TFBS located upstream of deregulated proteins’ coding region.