Artificial Dilution Series (ADS). Our approach uses existing annotation data to generate a series of annotation sets with different levels of correctness . We calculate the evaluation metric being tested for each annotation set in the series, allowing us to identify whether it can separate different signal levels. Finally, we contrast these results with several false positive annotation sets, which are designed to expose systematic weaknesses in GO assessment. We compared 37 evaluation metrics for GO annotation using ADS and identified drastic differences between metrics. We show that some metrics struggle to differentiate between different signal levels, while others give erroneously high scores to the false positive data sets. Based on our findings, we provide guidelines on which evaluation metrics perform well with the Gene Ontology and propose improvements to several well-known evaluation metrics. In general, we argue that evaluation metrics should be tested for their performance and we provide software for this purpose (https://bitbucket.org/plyusnin/ads/). ADS is applicable to other areas of science where the evaluation of prediction results is non-trivial.Automated protein annotation using the Gene Ontology (GO) plays an important role in the biosciences. Evaluation has always been considered central to developing novel annotation methods, but little attention has been paid to the evaluation metrics themselves. Evaluation metrics define how well an annotation method performs and allows for them to be ranked against one another. Unfortunately, most of these metrics were adopted from the machine learning literature without establishing whether they were appropriate for GO annotations. We propose a novel approach for comparing GO evaluation metrics called  In the biosciences, predictive methods are becoming increasingly necessary as novel sequences are generated at an ever-increasing rate. The volume of sequence data necessitates Automated Function Prediction (AFP) as manual curation is often impossible. Unfortunately, selecting the best AFP method is complicated by researchers using different evaluation metrics. Furthermore, many commonly-used metrics can give misleading results. We argue that the use of poor metrics in AFP evaluation is a result of the lack of methods to benchmark the metrics themselves. We propose an approach called Artificial Dilution Series (ADS). ADS uses existing data sets to generate multiple artificial AFP results, where each result has a controlled error rate. We use ADS to understand whether different metrics can distinguish between results with known quantities of error. Our results highlight dramatic differences in performance between evaluation metrics. The biosciences generate sequences at a faster rate than their structure, function or interactions can be experimentally determined. This has created a demand for computational methods that can automatically link novel sequences to their biological function , resulti“the primary performance metric … for CAFA is unsatisfactory. … by this measure, a null ‘prediction method’ outperforms most methods.” Kahanda et al. , a sigmoid function could be used to scale the score values. Next, Ppos and Pneg are merged to form an AP set, APS = Ppos ∪ Pneg. What we now have is a set of GO predictions with A) a set of errors, represented by Pneg, that can be separated by setting a threshold on sc values and B) a set of errors embedded into Ppos, that cannot be separated with a threshold (ADS noise level). Furthermore, A stays constant for all AP sets, whereas B is adjusted with ϵ. Finally, the predictions are extended to include ancestor nodes in the GO hierarchy for a given metric. In some cases, such as semantic similarity-based metrics, the metric will already account for dependencies between GO terms   and variation across ADS signal levels that evaluation metric should be able to distinguish (point 5).tsignal in the signal model to move to nearby terms in the ontology. First, real-life GO prediction tools often predict nearby terms as these tend to be correlated with tcorr. Second, omitting these changes would favour metrics that focus on tcorr in the evaluation and ignore the ontology structure during evaluation, which would bias our analysis results.We have two reasons for allowing tnoise is clearly separated from tcorr in the ontology and b) prediction scores for erroneous predictions, scnoise, come from the same distribution as correct predictions. This means that a threshold on the sc scores is unable to separate signal and noise predictions, ensuring that noise cannot be excluded from the evaluation. An EvM should consider our noise predictions as negative cases and score the data set accordingly. Furthermore, generating noise by permuting the GO classes from T keeps many other variables constant, such as average class size and average distance to the root node of predicted terms. The comparison of evaluation metrics is therefore not biased towards metrics that consider unrelated variables.The noise model, embedded in the Positive Set, aims to represent the worst type of error a classifier could make as a) sc corresponds to lower prediction quality. Note that most metrics take this into consideration by using varying thresholds on the sc values. If this noise was excluded, there would be no benefit in using a threshold on AP set results as an EvM could simply average over all prediction results.The noise that is added with lower prediction scores is motivated by trends in classifier results, where lower T. This gives us an output matrix:Metric is a given evaluation metric, columns, j = , correspond to different ADS signal levels, rows i =  represents the different repetitions with the same ADS signal level and APSij is the ith AP set obtained at signal level j. A good metric should have low variance within one ADS signal level (within Output columns) and a clear separation between the different ADS signal levels (between Output columns). We test this in two ways: A) visually by plotting Output columns as separate boxplots at different ADS noise levels and B) numerically by calculating the rank correlation, RC, between Output matrix and ADS signal levels. Boxplots give an overview of the performance of the tested metric. RC monitors how closely EvM is able to predict the correct ranking of the ADS signal levels. One could also consider the linear correlation, but that would favour linear correlation over the correct rank.EvMs are tested by scoring the AP sets and comparing the results to the truth set, False Positive Sets (FP sets). We included the following FP sets:In addition to AP sets, we also generated several prediction sets where all genes were annotated with the same set of non-informative GO terms or with randomly chosen GO terms. We wanted to be able to identify evaluation metrics that give high scores to prediction sets containing no signal. We refer to these sets as all positive set.Each gene has the same large set of the smallest GO terms. We call this the sc is taken from the term frequency in the database. We call this the naïve set.Each gene has the same small set of the largest GO terms, where random set.Each gene has a set of randomly selected GO terms. We call this the The motivation for the all positive set is to test the sensitivity of the evaluation metric to false positive predictions (low precision). Note that a set where each gene has all GO terms would be impractical to analyse. Therefore, we decided to select a large set of the smallest GO terms, each having many ancestral GO terms. The random set shows the performance when both precision and recall are very low and should therefore always have a low score. The naïve set tests if the evaluation metric can be misled by bias in the base rates to consider the results meaningful. The naïve set uses the same list of GO classes, ranked in decreasing order of size, for every analysed gene. The naïve set was originally proposed by Clark and Radivojak  and it hThere is a parameter that needs to be considered for FP set generation: the number of GO terms reported per gene. We selected this to be quite high (800 GO terms). This emphasises the potential inability of the metric to separate the noise signal in the FP sets from the true signal in predictions output by classifiers. We used three false positive sets:Naïve-800: all genes assigned to the 800 most frequent GO terms. Here larger GO classes always have a better score.Small-800: all genes assigned to the 800 least frequent GO terms. Here smaller GO classes always have a better score.Random-800: all genes assigned by a random sample of 800 GO terms (sampling without replacement).ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/UniProt/, 01.2019).GO frequencies for FP set sampling was estimated using the UniProt GO annotation , i.e. as a horizontal line at the lowest region of the plot  Finding the approximate point where the horizontal FP set line crosses the curve created by the medians of each signal level (see 2) choosing the corresponding signal level from the x-axis as the FP score for the FP set in question and 3) selecting the maximum over all FP sets as the final FP Score (referred to later as FPS). Note that in step 2 we limit the signal between zero and one. In step 3 we focus on the worst result over all tested FP sets. A good EvM should not be sensitive towards any of the FP sets, making the worst signal (= highest FP score) the best measure for the insensitivity towards these biases.In our numerical analysis, we summarise FP sets by evel see , 2) choorank-based metrics, GO semantic similarity-based metrics and group-based metrics.We selected representatives from various types of EvMs commonly used in AFP evaluation  and area under precision-recall curve (AUC-PR). Both strategies capture the amount of false positive and false negative errors at different threshold levels. We evaluated three semantic similarity-based metrics: Lin , Resnik Smin . We alsoMost of the metrics compared here, have been used actively in AFP evaluation  score.The semantic similarity metrics we tested return a matrix, Mg needs to be summarised into a single score for each gene at each tested threshold. As there is no clear recommendations for this, we tested six alternatives:When semantic similarity is used as an evaluation metric, AScorr and SpredMean of matrix. This is the overall similarity between all classes in BMg for classes in Scorr.Mean of column maxima. This is the average of best hits in CMg for classes in Spred.Mean of row maxima. This is the average of best hits in DMean of B and C.EMinimum of B and C.FMean of concatenated row and column maxima.Scorr , we evaluate the predicted set of GO terms against the true set separately for each gene and then summarise these values with the mean over all genes. In Term Centric evaluation (TC) we compare the set of genes assigned to a given GO term in predictions against the true set separately for each GO term and take the mean over all GO terms. In UnStructured evaluation (US) we compare predictions as a single set of gene–GO pairs, disregarding any grouping by shared genes or GO terms. In total we tested 37 metrics . These data sets represent three alternatives for evaluating an AFP tool : k = 2, 3, 4. RC and FPS scores for these runs are listed in k = 2), k = 4) and k = 3). We also tested thnoise = 0.8 with k = 3. Metric scores for all AP and FP sets are available in We tested all metrics with three values of http://ekhidna2.biocenter.helsinki.fi/ADS/).First, we investigate the stability of results across parameter settings. Second, we show differences within groups of EvMs with the CAFA data set. We then highlight differences between data sets and identify the best performing methods overall. All results were generated using scripts available from our web page , Resnik with summary method D and A, and Lin with summary method D to see if ADS can find performance differences. Visualisations for these metrics are shown in We compared the following evaluation metrics: US AUC-ROC, Fmax (RC = 0.982) and Smin (RC = 0.985), and next best separation is for US AUC-ROC. Three of the semantic similarity measures performed poorly: they have unstable distributions in the obtained RC scores. Resnik in particular, is shown to have the worst separation . Our results show that it is impossible to infer the original signal levels using these particular semantic similarity measures.Results with CAFA data show drastic differences between evaluation metrics. We see a clear separation across ADS signal levels in the boxplots for signal = 1) and last  boxplots do not fully reveal how good the evaluation metric is. For Lin score and Resnik score, for example, two extreme boxplots might be acceptable, assuming clear separation. However, the intermediate levels reveal considerable variability.It could be argued that comparing random data and data with a strong signal should be sufficient for metric evaluation. Our results, however, show that in most plots the first  and area under precision-recall curve (AUC-PR) with the CAFA data set. AUC-ROC has been used extensively in AFP evaluation, whereas AUC-PR has not. We look at the unstructured (US), gene centric (GC) and term centric (TC) versions of both AUC methods.signal = 1 , each coupled with six different semantic summation methods, A-F see . The ressignal = 0. The only methods that perform well in both ADS and FP tests are E and F. Furthermore, the intentionally flawed methods, B and C, were found to be weak by either ADS or FP tests.This comparison shows the power of ADS. Our first observation is that the choice of summation method has a bigger impact than which semantic measure is used. Methods A and C fail at separating signal levels in ADS and methods B and D fail at ranking FP sets next to the Our second observation is that Lin similarities exhibit marginally better performance than the other metrics tested, as shown in RC values. While these results might suggest that Lin similarity summarised with methods E or F performs the best, this is dependent on data set (see below).Smin calculates a distance between them  and therefore set tRC = 0.95. For FP values, we found that the top performing metrics would be selected with tFPS =  and set tFPS = 0.16. There is a trade-off when setting these thresholds, as different values can emphasise either RC or FPS. However, both tests should be monitored for metric selection. In conclusion, a good metric should have as little red shading as possible and high performance in both RC and FP tests. Our supplementary table Furthermore, we wanted to highlight the consistent high performance of metrics by setting a threshold on The upper part of tFPS. However, ic SimGIC2 mostly outperforms other tested SimGIC functions. Overall, based on these results, ic SimGIC2 is our strongest overall recommendation.Finally, ic SimGIC and ic SimGIC2 shows quite good performance across the tests, with ic SimGIC2 often outperforming ic SimGIC. We mark also ic2 SimGIC2 as a potential recommended method, although it fails in FP test with UniProt data. Note that ic2 SimGIC2 goes only slightly over the selected threshold (0.18) and in other data sets it stays below Smin1 [ic weights to emphasise small GO classes. However, we still point out that FP results for Smin are not bad, except for Uniprot data, and it has the strongest RC results in the table. Indeed, Smin was one of the best performing metrics for MouseFunc data. Therefore, we included it in our potential recommendations.We are surprised to see that the ic2 Smin1  has an eWe were also surprised by FP signals with TC AUC-PR, as term centric analysis should be insensitive to FP signal. This is explained by our calculation method for AUC-PR. Even when there is no variance among prediction scores across all the genes, we allow precision-recall curve to return a single point when calculating the area. While this is in agreement with how to calculate AUC-PR , it can Smin) with some data sets.In conclusion, our results suggest that ic SimGIC2 as the most stable, well-performing metric, despite being beaten by other potential recommendations  is a novel method to systematically assess how reliably different evaluation metrics cope with assessing GO classifiers. ADS is a valuable new tool for developing and complementing the existing evaluation methodology. Using ADS we have revealed drastic differences between popular evaluation metrics and metrics that give inflated scores to false positive annotations.We demonstrate ADS by testing a large number of evaluation metrics including many variations. We were able to show that:One should only use term centric AUC-ROC and AUC-PR methods and avoid their gene centric and unstructured versions.Our novel summary methods, E and F, both outperform existing semantic similarity summary methods.The simGIC metric shows consistently high performance across all data sets.Other well performing metrics, besides simGIC2, were TC AUC-PR, TC AUC-ROC, Lin score with method E and Resnik score with method E. These even outperformed ic simGIC2 on some data sets, which suggests that the selection of the best evaluation metric is data-dependent. Note that our analysis scripts and ADS are freely available, allowing testing with any GO data set.One motivation for the ADS project is the development of better evaluation metrics for GO classifier comparisons. We tested, for example, the following variations on existing metrics:We modified the summary methods for semantic similarities.We tested Area Under Precision Recall curve (AUC-PR).We tested the unstructured variant of the SimGIC function.Here 2, 3 and some of the methods tested in 1 improved performance. All of these modifications were simple to implement using our existing evaluation code. Furthermore, we have distributed a standalone C++ program that calculates all of the evaluation metrics tested here, allowing them to be used in practice.Our results included metrics that were expected to exhibit poor performance. These were included as negative controls in our analysis to prove that ADS works as expected. Indeed, we recognised that US AUC-ROC and GC AUC-ROC show very bad performance with FP tests. This is in agreement with the results by Ferri et al.  with AUCEvMnaive, would allow “cheating” in method comparison. For example, when the classifier lacks predictions for a gene, G, it could improve its results simply by returning a set of naïve predictions for G (i.e. the base rate for each GO class). A better alternative for the end user would be to not make any predictions. Also, a bad EvMnaive would evaluate the correct GO term as an equally good prediction as any of its parents. This would promote predicting large unspecific GO terms for all genes, although the small terms contain more biological information. Likewise, a metric penalising false positives weakly would promote methods that predict as many terms as possible and a metric penalising false negatives weakly would promote predicting as few GO terms as possible. Note that these artefacts are often hard to observe without the kinds of comparisons we have presented.The importance of a good evaluation metric is best demonstrated with flawed metrics. A metric with sensitivity to our naïve FP set, Our results are dependent on the signal and noise model, as these are used to create the correct and wrong predictions. Although these models were motivated in the Methods section, we emphasise that this is only the first version of ADS. Our plan is to extend these models and use them in addition to our current models. ADS analysis can be easily extended in this manner. We propose the ADS approach in general, without limiting it to specific signal and noise models.Furthermore, we use a real dataset as a starting point, thus maintaining its challenging features. Also, there are no real classifier result sets available, to our knowledge, where the correct ordering of the classifiers is known. So one is bound to use some kind of simulation in this type of analysis. Here the controlled percentage of noise makes the correct ordering of the results known. We point out that the only similar evaluation, extensive work by Ferri et al. , uses fuWe envision several application scenarios for ADS. The first, is a general user, who would like an overall answer to the question: which metrics are optimal? For this case a set of metrics that have stably high performance on very different datasets is needed. In this work we give our recommendations with tests on three data sets. Further testing on a larger spectrum of datasets and with altered noise models is planned for follow-up articles. The second scenario is a specialised user who wants to optimise metrics for the type of data he or she is testing. Here the researcher would perform the ADS analysis using the selected dataset(s). This ensures the applicability of the selected group of evaluation metrics on the used dataset(s). Finally, we expect that ADS would be useful for the developers of evaluation metrics, as it allows thorough testing of novel metrics.ADS is designed to compare evaluation metrics that are used in GO term prediction for proteins, however, similar problems with evaluation metrics occur in other areas of the biosciences. Complex hierarchical classifications are used, for example, for classifying disease genes. Also the use of ontologies in the biosciences is growing , creatinThe motivation for the ADS project came from our own GO classifier work , 7, wherScience needs numerous predictive methods, making the comparison and evaluation of these methods an important challenge. The central component of these comparisons is the Evaluation Metric. In the case of Automated Function Predictors the selection of the evaluation metric is a challenging task, potentially favouring poorly performing methods. In this article we have described the Artificial Dilution Series (ADS) which is, to our knowledge, the first method for testing classifier evaluation metrics using real data sets and controlled levels of embedded signal. ADS allows for simple testing of evaluation metrics to see how easily they can separate different signal levels. Furthermore, we test with different False Positive Sets representing data sets that the evaluation metric might mistakenly consider meaningful. Our results show clear performance differences between evaluation metrics. We also tested several modifications to existing evaluation metrics, some of which improved their performance. This work provides a platform for selecting evaluation metrics for specific Gene Ontology data sets and will simplify the development of novel evaluation metrics. The same principles could be applied to other problem domains, inside and outside of the biosciences.S1 FigThis scatter plot represents EvM performances on MouseFunc data set. Labels and abbreviations, used here, are explained in (PDF)Click here for additional data file.S2 FigThis scatter plot represents EvM performances on Uniprot data set. Labels and abbreviations, used here, are explained in (PDF)Click here for additional data file.S1 TextPseudocode for generation of ADS.(PDF)Click here for additional data file.S2 TextText shows an example on how the signal and noise is defined in ADS. Text also shows labels, definitions and discussion of tested metrics. Finally, we discuss the summary methods for semantic similarities and motivate our novel summary methods.(PDF)Click here for additional data file.S1 Filek = 2.This compressed archive file represents the evaluation metric scores for all generated AP and FP sets, obtained with (GZ)Click here for additional data file.S2 Filek = 3. Our further analysis focuses on this data.This compressed archive file represents the evaluation metric scores for all generated AP and FP sets, obtained with (GZ)Click here for additional data file.S3 Filek = 4.This compressed archive file represents the evaluation metric scores for all generated AP and FP sets, obtained with (GZ)Click here for additional data file.S1 TableWe represent a summary table of all the compared EvMs. Table shows the used abbreviation, core function, used data summary method, used threshold function over the classifier prediction score, used IC weighting and summary method for semantic similarities. In addition we mark the EvMs that have been popular in AFP evaluation, ones that have some novelty and ones that we consider to be simple. We also mark EvMs that we expect to perform badly as negative controls. More detailed description of these EvMs is in suppl. text (PDF)Click here for additional data file.S2 Tablek and thjacc. Table A shows results for different k values and table B shows results for two values of thjacc. We selected in each comparison one data set, created results for all EvMs using two parameter values and compared the obtained results using Pearson and Spearmann rank correlation. FPS and RC results are evaluated here separately. Tested parameters show very little effect on the results.Here we evaluate the stability of the results when we vary parameter (PDF)Click here for additional data file.S3 TableTable explains the size and density of the three used data sets . Data sets are explained in main text. Table shows skewed distribution of GO terms per gene by representing selected quantiles from this distribution. This is shown separately for each data set with and without the propagation to parent terms. We also show the same size distribution for the GO terms, again showing the skewness of the size distribution.(XLSX)Click here for additional data file.S4 TableRC and FPS scores for tested metrics using k = 2. Table is tab-delimited text file.(TAB)Click here for additional data file.S5 TableRC and FPS scores for tested metrics using k = 4. Table is tab-delimited text file.(TAB)Click here for additional data file.S6 TableRC and FPS scores for tested metrics using k = 3. Table is an excel file that shows similar results as (XLSX)Click here for additional data file.


Researchers today are generating unprecedented amounts of biological data. One trend in current biological research is integrated analysis with multi-platform data. Effective integration of multi-platform data into the solution of a single or multi-task classification problem; however, is critical and challenging. In this study, we proposed HetEnc, a novel deep learning-based approach, for information domain separation.HetEnc includes both an unsupervised feature representation module and a supervised neural network module to handle multi-platform gene expression datasets. It first constructs three different encoding networks to represent the original gene expression data using high-level abstracted features. A six-layer fully-connected feed-forward neural network is then trained using these abstracted features for each targeted endpoint. We applied HetEnc to the SEQC neuroblastoma dataset to demonstrate that it outperforms other machine learning approaches. Although we used multi-platform data in feature abstraction and model training, HetEnc does not need multi-platform data for prediction, enabling a broader application of the trained model by reducing the cost of gene expression profiling for new samples to a single platform. Thus, HetEnc provides a new solution to integrated gene expression analysis, accelerating modern biological research. The use of integrated analysis with multi-platform gene expression data in current biological research is increasing –4. In geThese well-established and publicly available resources have provided a huge opportunity for developing integrative analysis approaches to gain more comprehensive insights. A particular interest is to build predictive models that integrate multi-platform data for enhanced performance. However, handling multi-platform data effectively is quite challenging. The difficulties mostly come from the inability to utilize the complicated, close linkages among features from different platforms efficiently. Several reviews have been conducted in integrative models , 8. PopuDeep learning, one of the most promising methods in current machine learning, has been implemented in a variety of research fields, including object recognition, keyword triggering, language translation, and others . It has HetEnc, a deep learning approach for integrated gene expression analysis, which integrates different platforms of genomics features on the same cohort of subjects. HetEnc is designed as two sequential modules. In the first module of feature representation, it utilizes the multi-platform information in an unsupervised fashion to generate a high-level abstracted feature set, also known as intermediate features. In the second module of predictive modeling, a deep feed-forward neural network is constructed using the intermediate features as input, to train the model for each targeted endpoint.Here we propose The SEQC neuroblastoma dataset includes 498 neuroblastoma patients’ gene expression profiles measured both by Microarray and RNA-seq. The training dataset consisted of 249 samples, and the other 249 samples were in the validation/test dataset. We used the sample distribution as defined in SEQC project .The expression profile for both Microarray and RNS-seq analyses currently are publicly available in the National Center for Biotechnology Information (NCBI) GEO database. The Microarray data (GEO accession: GSE49710) was generated using customized 4x44k oligonucleotide microarrays (Agilent Technologies) and extracted via Agilent’s Feature Extraction software (Ver. 9.5.1). The RNA-seq sequencing data (GEO accession: GSE62564) was performed on the Hi-Seq 2000 platform (Illumina). Detailed sample preparation and data pre-processing has been described elsewhere [.We investigated three clinical endpoints from the neuroblastoma dataset, including favorable prognosis (FAV), overall survival (OS_All), and high-risk patient survival (OS_HR). FAV is a binary label for patients belonging to a favorable subgroup that is event-free  without chemotherapy for at least 1000 days, or those belonging to an unfavorable subgroup that died from disease despite chemotherapy. OS is the occurrence of death from disease, and high risk (HR) only includes patients belonging to a high-risk subgroup (with stage 4 disease > 18 months at diagnosis or with MYCN-amplified tumors at any age and stage). Based on previous experience, these three endpoints have different levels of predictability: FAV is usually easy to predict and has a high predictive performance in all modeling algorithms, whereas OS_HR is very difficult to predict no matter which modeling algorithm is applied. As reported, the predictive difficulty of OS_All falls between FAV and OS_HR. Since not all clinical endpoints were available, the FAV and OS_HR study did not include all 498 samples. A detailed description of these three endpoints and their previous predicting performance are summarized in Table 1All SEQC neuroblastoma datasets were already pre-processed when downloaded from the GEO database. While there were several available data pre-processing pipelines for RNA-seq data, we chose the dataset pre-processed by Su, et al. (2014), to focus on the subset of 10,042 genes that were mapped one-to-one between Microarray and RNA-seq. Therefore, the final data matrix of train and test datasets were  and , for both Microarray and RNA-seq platforms, meaning that the notation of  is 249 samples with 10,042 (gene expression) features for each sample.Unsupervised Learning is a topic of interest in today’s deep learning community. One typical unsupervised learning algorithm is autoencoder (AE), which is designed to compress high-dimension data into low-dimension features. A typical AE is composed of two connected networks: an encoding network and a decoding network. The encoding network tries to compress the input data into low-dimensional features, which made up the bottleneck (layer); the decoding network, in reverse, tries to reconstruct the original input data from the low-dimensional features. In a combination of the encoding and decoding networks, the AE is much like a regular Multilayer Perceptron (MLP), where the major difference is that the input and expected output of this MLP are the same. In other words, the learning process of this AE tries to reconstruct the input data with minimal information loss.A novel aspect of this study is that we not only used the regular AE for one platform; we also designed other two kind of representation networks. The first network is named as CombNet, which first combined two different gene expression data together, treated them as the same type of data that could be represented by single autoencoder Fig. a. ParticIn the predictive modeling step, we applied a fully-connected neural network with feed-forward architecture. A fully-connected neural network is an artificial neural network format with at least three layers: one input layer, one output layer and one hidden layer. Fully-connected means linkages exist among nodes between two adjacent layers; however, there are no linkages between nodes in the same layer. Usually, there is more than one hidden layer in a deep neural network architecture, and HetEnc used four of these in the modeling step. Feed-forward means the network did not have a connection forming a cycle, unlike Boltzmann machine and recurrent neural networks.1 = Wa0 + b) or non-linear . Usually, for a non-linear function, an activation function can be added to a linear basis, making the whole function non-linear , where f is the non-linear activation function and z1 is the output of a linear function).The linkages between nodes in two adjacent layers could be either linear  gradient descent in backpropagation and the loss function for both datasets was https://github.com/seldas/HetEnc_Code.The source code for the whole HetEnc is available at: In this study, we compared HetEnc models to three kinds of machine learning algorithms.p-value (< 0.05) and log2 fold-change (> 1.5). Parameter K in KNN ranged in ; kernel used in SVM is ‘rbf’; and the other parameters are set as default. In training process, each model was trained based on randomly selected 70% of training data and its performance was evaluated on the remaining 30% of training data. The training process repeated 500 times to retrieve an overall cross-validation modeling performance. The model was then tested on the other 249 testing samples. In this study, we only used the model testing performance for comparison, which was averaged among 500 models. This part of the experiment was performed in R, with packages of ‘class’ for KNN, ‘pamr’ for NSC, and ‘kernlab’ for SVM. More detailed description of these three models are published elsewhere.In our previous study, three different types of predictive models, as K-Nearest Neighbors (KNN), Support Vector Machine (SVM) and Nearest shrunken centroids (NSC), were constructed by using the exact same pre-processed dataset. In general, gene features were pre-filtered by their XGBoost’ and ‘SKLearn’.Besides KNN, SVM and NSC, we also compared HetEcn to two more commonly-used machine learning models, as Random Forest and XGBoost, using the same processed datasets. For Random Forest, we tuned the number of trees from 10 to 200, and observed a saturated performance when trees = 100. For XGBoost, tree-based models were selected as default. The other parameters are used as default. The training process of Random Forest was repeated 100 times, for each time the whole training dataset was used to train the model; and the model was then evaluated on the other 249 testing samples. Similarly, we only used the model testing performance for comparison, which was the average AUC among 100 repeats. Since XGBoost performance will not change when different random seed was set, we only ran XGBoost once. This part of the experiment was performed in Python, with modules ‘We also compared HetEnc to the best predictive model that developed by various attendees that submitted to the consortium during the MAQC/SEQC projects. Note that these best models were not restricted to any data normalization, feature selection or modeling algorithms, and their performance was only evaluated by the testing samples, which were blinded to them when training their models. The best models were selected as using the best model of a single attendee, which included 6 microarray models and 54 RNA-seq models. The final performance of SEQC models were their average AUCs.HetEnc is inspired by the domain separation network  developeThe entire HetEnc architecture is composed of two modules. The first feature representation module is the key module, which is designed to extract the gene expression representation into different subspaces via different representing or encoding networks. The first module involves three distinct encoding networks; Autoencoder (AE), CombNet and CrossNet Fig. a, for exIn the feature representation step, the most differences between three networks  are the definition of input and output data, which could be the same or different platform. In all, there are four different combinations of microarray and RNA-seq, as shown in Fig. p, 4096, 3072, 2048, 1024, 2048, 3072, 4096, p, respectively, where p was the number of input (and reconstructed) gene features. The first four layers are the encoding network and the last four layers are the decoding network. The compressed feature set, therefore, is from the bottleneck  layer, with 1024 nodes. We used the hyperbolic tangent (Tanh) activation function for all AE hidden layers, and the sigmoid (logistic) activation function for the output layer. For denoising, we also added one drop-out layer between sets of two layers, and the drop-out ratio was set to 0.2.The hyper-parameters of AE are described as follows: in total, nine layers were formed, with the number of nodes, In the modeling step, the six-layer feed-forward deep neural network (6-DNN) is depicted in Fig. We evaluated our model on the SEQC neuroblastoma dataset. In total, six predictive models were trained for endpoints FAV, OS_All and OS_HR, and two data platforms , respectively. Because the first step (feature representation) is unsupervised, the encoding networks  generated by the first step would be shared between three endpoints in the modeling step.We first applied Principle Component Analysis on the intermediate features generated by three encoding networks. Latent features from each encoding network will be analyzed both independently and combined as HetEnc features, as shown in Fig. , the pair-wise correlation of PC2 from AE between Microarray and RNA-seq is not significant (r2 = 0.289). On the other side, there are high linear correlation of PC2 from CombNet and CrossNet between RNA-seq and Microarray, as r2 reached 0.921 and 0.89 respectively. Further, when combined latent features from AE, CombNet and CrossNet together as HetEnc, the linear correlation between two platforms became higher (r2 = 0.949). This PCA and pair-wise PC2 correlation analysis result indicated that CombNet and CrossNet could represent platform-independent features from the raw dataset.To further reveal the platform-independent variance in PC2, a correlation analysis was performed on PC2 between Microarray and RNA-seq of the same sample. As shown in Fig. The predictive models were first constructed and evaluated by five-fold cross-validation within the training dataset. In the five-fold cross-validation, we randomly separated the entire training dataset into five subgroups; where in each run, four subgroups of samples were used to train the model, and the remaining one was used as a testing set for evaluating the model’s performance. Each subgroup was tested once. By picking different random seeds, we repeated the five-fold cross-validation 20 times; therefore, a total of 100 (5*20) sub-models were built for each endpoint.After evaluating the model via cross-validation, we trained one model using all 249 training samples for each endpoint. The model was finally evaluated on the test dataset with the other 249 samples. Similarly, we set different random seeds to run the whole modeling process 100 times to retrieve an average predicting performance. We measured the model’s performance for three endpoints . As shown in Table We compared our model performance to three machine learning models - support vector machine (SVM), nearest shrunken centroids (NSC) and k-nearest neighbors (KNN) -using the same training/testing sample distribution and data preprocessing . A detailed modeling process of SVM, NSC and KNN can be found elsewhere . Furtherp < 0.01) than the best fine-tuned predictive models from the SEQC community.The performances (AUC) of HetEnc, KNN, NSC, SVM and the best model from SEQC/MAQC for three Endpoints  with two platforms (RNA-seq and Microarray) was shown in Table By developing HetEnc, the underlying hypothesis is that we assume the gene expression profiling value is determined by two factors: platform-independent factor and platform-related factor; where the platform-independent factor is mostly attributed to the sample itself, and the platform-related factor is specific to the platform used to measure the expression value. Thus, the main goal of HetEnc is to separate the information from these two factors, to reduce the noise (component) introduced by the platform and emphasize the platform-independent factor.As demonstrated in this study, HetEnc outperformed previously-reported machine learning models overall, achieving a significantly better predictive performance. Three aspects accounted for its performance: (1) By using exact the same dataset , HetEnc showed significantly better predicting performance than such machine-learning algorithms as support vector machine (SVM), nearest shrunken centroids (NSC) and k-nearest neighbors (KNN). We observed this superior performance from both “head-to-head” comparative analysis and previously-published results. (2) With no restrictions on data pre-processing and modeling strategies, HetEnc still performed better than the best models developed by other groups in the SEQC project. (3) Performance differences between cross-validation and external testing are relatively small in developed HetEnc models, indicating that the HetEnc model can be applied more generally to new test sets.For high-dimensionality biological data analysis, feature selection and feature representation are two most applied strategies for dimension reduction purpose. Feature selection methods aim to extract a subset of meaningful biological entities  from the raw data. Common feature selection methods are based on the feature itself , its coefficients with other features , or coefficient with the endpoint . One advantage of using feature selection is about the model interpretability, as discovered biomarkers like genes or proteins could be intuitively interpreted with biological functions. However, if the relationship between the biomarkers and endpoint could be simply interpreted (such as gender-related genes and gender prediction), usually a linear/logistic regression model will be good enough to capture the relationship. On the contrast, many studies have demonstrated that the advanced modeling algorithms such as Random Forest and SVM could significantly improve the predictive performance especially for “moderate and difficult to predict” biological endpoints, indicating the relationships between biomarkers and the endpoint could be too complicated to be interpreted in a simple format. That is why we developed a feature representation method as HetEnc to learn latent features of raw inputs from multi-platforms via deep encoding networks. Still, for “easy to predict” endpoint, we believe using simple model such as KNN might be OK due to its higher biological relevance and interpretability, without losing too much performance. On the other hand, for “unpredictable” endpoints such as OS_HR in this study, any modeling algorithm would not have significant differences.Moreover, unlike other integrated modeling which combined features from different platforms either horizontally or hierarchically , one uniAdditionally, these two steps are quite independent of each other, which enables HetEnc to maximize its data utilization capability. For example, many datasets have unlabeled samples; although these samples cannot be used in supervised learning , they could still be used in the feature representation step to further fine-tune the model parameters. Further, the modulization of HetEnc also enables the new deep learning technologies and data format to be easily embedded. For example, HetEnc could also be applied to data with spatial information, such as changing the basis architecture in the feature representation step from autoencoder to convolutional autoencoder, recurrent autoencoder, etc.In current study, we developed the HetEnc architecture by combining AE, CombNet and Cross Net. To determine the synergic effect among these three encoding networks, we evaluated the model based on only one single encoding network. As a cross-validation result on OS_All endpoint, models with only AE, CombNet or CrossNet yield 0.808, 0.801 and 0.811 in AUC respectively, where the model with a combination of three performed better  could reflect the similarity between two words. Similarly, in HetEnc the real value of latent features (generated by encoding networks) may not have explicit biological meaning however the distance of latent features between two samples may reflect their similarity/closeness in phenotype. In this study, we applied Principle Component Analysis (PCA) to visualize these latent features from three different encoding networks as their 2D projection. As the result shown in Fig. As we are entering the big data era, more and more researches would include multi-type data that generated for a consensus decision making, where HetEnc is one promising way to bridge different types of data by identifying their common and specific features. In this study, we restricted HetEncs to multi-platform gene expression data analysis due to the resource we now have in hand. However, the concept of HetEnc may not only suitable for the platform-specific information, but also apply to find a common information between different sites/labs, different time/dose, or different organ/tissues. Further experiments are needed to see whether the concept of HetEnc may also apply to other multi-type data researches.In this study, we only evaluated HetEnc on a two-platform dataset, and we did not discuss multi-platform (> = 3) data analysis. Multi-platform data analysis would be more complicated, due to data sparsity and feature mapping issues, especially finding/developing a mapping system among all platforms would be difficult. Nevertheless, HetEnc holds the potential to expand to multi-platforms. A possible way for HetEnc to work with multi-platform datasets is to design more comprehensive CombNet and CrossNet models. For example, the CombNet could combine multiple platforms, and the CrossNet could represent the two-way translation between every two platforms . Another strategy is to design multiple CombNet and CrossNet models. There may be one specific CombNet and CrossNet model for each platform pair, thus combining the features from all representation models, this strategy may allow to avoid developing CombNet and CrossNet models for two rarely related platforms due to lack of mapping system. Still, further investigation and optimization is essentially needed to generalize HetEnc to multi-platform data analysis, as well as to sparse datasets for which including samples that not have multi-platform data.


Analyzing the human transcriptome is crucial in advancing precision medicine, and the plethora of over half a million human microarray samples in the Gene Expression Omnibus (GEO) has enabled us to better characterize biological processes at the molecular level. However, transcriptomic analysis is challenging because the data is inherently noisy and high-dimensional. Gene set analysis is currently widely used to alleviate the issue of high dimensionality, but the user-defined choice of gene sets can introduce biasness in results.In this paper, we advocate the use of a fixed set of transcriptomic modules for such analysis. We apply independent component analysis to the large collection of microarray data in GEO in order to discover reproducible transcriptomic modules that can be used as features for machine learning. We evaluate the usability of these modules across six studies, and demonstrate (1) their usage as features for sample classification, and also their robustness in dealing with small training sets, (2) their regularization of data when clustering samples and (3) the biological relevancy of differentially expressed features.p < 0.01). The models also had higher accuracy and negative predictive value (p < 0.01) for small data sets (less than 30 samples). Additionally, we observed a reduction in batch effects when data is clustered in the FC-space. Finally, we found that differentially expressed FCs mapped to GO terms that were also identified via traditional gene-based approaches.We identified 139 reproducible transcriptomic modules, which we term fundamental components (FCs). In studies with less than 50 samples, FC-space classification model outperformed their gene-space counterparts, with higher sensitivity  contains supplementary material, which is available to authorized users. Gene set enrichment analysis (GSEA) i are the i-th component (columns) of the “best solution” and the permuted source matrix from the k-th run respectively.Where We resampled the full compendium randomly without replacement to obtain 50 similar-sized pseudo-representative compendiums. Whitening was performed as described previously, but we selected the same number of components as the original solution to facilitate comparison between the models. For each of the 50 resampled compendiums, we ran ICA ten times, and chose the solution with the highest negentropy score as the solution for that resampled compendium. We compared the 50 chosen solutions to the “best solution” from the representative compendium using the same methodology as per the previous section.For each component, we defined the sets of genes with loadings that were three standard deviations above or below the mean as the up or down modules respectively for the component. Collectively, we term the union of both set of genes as active genes for the component. As per Engreitz et al. , we perfp-value of less than 0.01 were retained.The percentage overlap between gene signatures from the H collection of MSigDB  and the All 353 normal human samples from GSE3526, coming from 65 different tissue types derived from ten post-mortem donors, were downloaded from GEO, processed and projected into FC space. To obtain representative samples from the 22 nervous system tissues, we calculated the pairwise distances within each tissue type and selected the medoid . Clustering of the 22 samples was then performed. The set of all samples from GSE3526 were also used as a compendium to annotate queries with their most similar tissue origin.GSE7307 (Human Body Index) contains 677 samples from 90 tissue types, some of which were from diseased patients. We downloaded only the healthy samples, and processed them as per GSE3526. We compared the tissues types that were common to both GSE3526 and GSE7307 using Pearson correlation coefficient. To provide robust estimates that were not affected by outlying samples in the tissue types, we reported the median and the standard deviation of the correlations for each “GSE/tissue”-“GSE/tissue” pair. We also included the sub-compendium (“Human Tissue Compendium”), containing both 353 samples from GSE3526 and 504 samples from GSE7307, in our R package so that users can also use it to annotate their query samples with the most probable tissue types.Qgene into the corresponding FC space is done via the matrix multiplicationFor all evaluation datasets, the raw CEL files were downloaded from GEO. Each GSE was processed independently by running RMA on the set of samples, followed by technical bias correction as per the earlier section “Data Collection”. Projection of a dataset A unitary vector space based on the FCs loadings can also be defined by normalizing each component to have unit length, which we provide as an option in our R package. For all analysis in this paper however, projection onto FC space was done using the original gene loadings in the calculated FCs.p-values [Wherever t-test was used, Benjamini-Hochberg correction was performed on the p-values , with N p-value threshold of 0.05. The union of the three sets of differentially expressed FCs was then used as the signature to cluster the sample types. We performed hierarchical clustering based on the FC values in the signature, using average linkage. To identify FCs that were specific to the differences between RAPBM and RASFM, we focused on the DE FCs that were unique to the pair , and report the corresponding GO enrichment annotations.For GSE71370, the meta-data available in GEO was used to annotate the samples under three categories: synovial fluid from rheumatoid arthritis (RA) patients (RASFM), peripheral blood from RA patients (RAPBM) and peripheral blood from healthy patients (HCPBM). Gene expression data were projected into FC space. For each of the three pair-wise comparisons between categories, unpaired t-tests were performed across the FCs, with a BH-corrected kernlab [GSE13159 contains data from the Microarray Innovations in Leukemia (MILE) study program, consisting of eighteen different categories of leukemia (including a control group). The class labels of the individual sample were obtained from the Data Supplement accompanying the original publication . After pkernlab  was usedSamples from classes C3 , 122 samples) and C8 , 237 samples) in GSE13159 were defined as the positive and negative groups respectively. For a given simulation run, we randomly chose 22 C3 and 37 C8 samples as the held-out test set. The remaining data in the two groups (100 C3 and 200 C8) were then subsampled at 5 10%, 20%, 40%, 60% and 80% to produce corresponding training sets for training SVM classifiers . For each subsampling percentage, we repeated the sampling 200 times. For a particular sampling, we calculate the negative predictive value (NPV), positive predictive value , sensitivity , specificity and accuracy as follows:Where x is the model based on feature under consideration (FCs or genes).p-value that was less than 0.05, we defined them to be significantly different. When there is perfect agreement between the pair, the test statistic is undefined; in this case, we simply note that the two models are identical. For a given run, we compute the percentage of the 200 samplings where the two models were observed to be different.In addition, for each sampling, we also performed McNemar's test with continuity correction on the classification results from the two models (FC vs gene space). If the calculated test statistic for the pair of models had a The average across the 200 sampling for a given subsampling percentage were then recorded as the respective statistic for that run. A total of ten independent simulation runs were performed, and the mean and standard deviation for the statistics were reported across the ten runs. For comparison, we also calculated the above statistics using the full remaining data (i.e. subsampling percentage is 100%) at each run. Repeats were not performed for this case.Affymetrix HGU-133 Plus 2.0 samples from the MicroArray Quality Control (MAQC) project were downloaded from GEO (GSE5350) and processed. The 120 samples were clustered in both FC space and full gene space, and cophenetic correlation between the trees was computed. For visualization purposes, a tanglegram  using boCi is the i-th cluster. The Gini impurity for each of the two clusters was calculated asWhere fj is the fraction of samples in the i-th cluster that are from the j-th mega-class.Where limma [GSE15434 contains a total of 251 AML samples, coming from three different centers in Germany: Dresden (DRE), Munich (MUC) and Ulm (ULM), with 78, 96 and 77 samples respectively. Approximately half of the samples contained mutations in the NPM1 gene. We identified differentially expressed (DE) functional components (FCs) and genes between the NPM1-mutated and NPM1-wild type groups using the R package limma  at a falFor GSE66533, the rhabdomyosarcoma samples were separated into two main groups (33 PAX3-FOX01 Fusion-Positive and 25 Fusion-Negative samples) based on descriptions obtained from Supplementary 1 of the paper by Sun et al. . Gene exgo-basic ontology and the default goslim_generic subset.For E-MTAB-3162, the raw CEL files was downloaded from ArrayExpress  and procTo avoid overrepresentation of any biological phenotype in the training data, clustering of microarray samples was performed on the full compendium  to obtain a representative compendium. The height cutoff of the clustering tree was determined to be 0.3 based on k-nearest neighbor plots for k = 4 and 5  or the On average, each of the 100 independent runs took 2461 iterations to reach the convergence requirement. The final negentropy of the ICA solutions ranged from 0.2417 to 0.2443, with a median of 0.2442. Run 39’s solution yielded the highest negentropy and was thus used as the “best solution” in the rest of the paper. The 139 columns of the canonized S matrix are the independent components obtained from ICA, and we refer to them as functional components (FCs). Each FC had zero mean and unit standard deviation.The derived FCs were well correlated between all 100 runs Fig. , and forTo gain better understanding of the FCs, we identified the key gene contributors to each of them. The elements of each component are the gene loadings, and can be interpreted as the level of contribution of a gene to the component’s score. For a given FC, we consider the set of genes whose absolute loading is three standard deviations above the mean as active genes. Apart from FC 1, which only had 28 active genes, the number of active genes in the other FCs ranged from 103 to 494, with a median of 382. Amongst the 20,089 genes, 12,978 genes were found to be active in at least one FC. The majority of the genes were active in only up to three FCs . A total of 689 unique GO codes were obtained across the 139 FCs, a 66% increase compared to the 415 unique GO codes obtained from the corresponding 139 leading principal components. This suggests that there is more biological signal in the FCs than components obtained via PCA, in line with current literature [The percentage gene overlap between active genes in the FCs and the respective gene signatures in the H collection of MSigDB were calculated, and only the statistically significant pairs are shown in Fig. Twenty of our FCs do not have any significant gene overlap with the signatures in the H collection. Of these, fourteen of them  also do not have any GO annotations. The lack of GO annotations for these FCs does not necessarily indicate a lack of biological significance; for instance, the active genes in FC 1 are clearly markers for sex-specific features  with each other.About a third of the samples from GSE3526 were from 22 tissue types belonging to the nervous system, and we performed clustering of the representatives from these tissues . The median Pearson correlation coefficients (MPC) between tissues from GSE3526 and GSE7307 are shown in Fig. To demonstrate the applicability of our FCs as features for use in machine learning algorithms, we apply our FCs to two different studies (rheumatoid arthritis and leukemia). Additionally, we performed subsampling of the leukemia study to compare how model performances in FC space and full gene space are affected in low-sample settings.GSE71370 contains three sample types: peripheral blood from rheumatoid arthritis (RA) patients (RAPBM), peripheral blood from healthy patients (HCPBM), and synovial fluid from RA patients (RASFM). Using the standard Affymetrix chip definition file (CDF), we found 6636 DE genes between RASFM and HCPBM, and zero DE genes between RAPMB and HCPMB.There are zero differentially expressed (DE) FCs between RAPBM and HCPBM, 72 DE FCs between RAPBM and RASFM, and 89 DE FCs between RASFM and HCPBM. 61 FCs were common in the latter two sets, resulting in a combined signature of 100 FCs for clustering. Figure GSE13159 contains patient samples from 18 different classes of leukemia. Table The random forest we built indicated that FC 18, 39 and 54 are the three most important variables  for subsampling fractions of up to 10% (both p < 0.01). The FC-based model was also able to provide balanced predictions at very low subsample fraction (5%), whereas the gene-based model breaks down and predicts all negatives, resulting in undefined PPV  to demonstrate that DE FCs can provide similar insight to a disease as the conventional DE gene approach.p-values were BH-corrected). The Fusion-Positive samples generally had higher FC scores in FCs 59, 75, 82, 96 and 112, and lower scores in 66, 86, 98, 106 and 134 as compared to the Fusion-Negative samples , followed by “endometrium” (top hit for five samples), “deltoid muscle” (top hit for four samples) and “synovial_membrane” (top hit for one sample).The dengue viral study contained Day 0 and Day 4 patients, and the two time points were previously reported to have different gene expression profiles due to the dynamics of viral response. Six FCs were differentially expressed between in Day 0 and Day 4 dengue patients Fig. , which cp-value (FC 79) was also found to be involved in inflammatory and immune responses.The use of gene set enrichment analysis (GSEA) is ubiquitous today in transcriptomic analysis, and the growing number of gene sets (signatures) in the MSigDB repository enables a better characterization of biological processes. There is inherent subjectivity in GSEA, however, due to the users’ choice of signatures. Liberzon et al.  suggest Although we retained only the leading 139 components of the whitened data, ICA convergence on the tall matrix took close to 2000 iterations to reach the preset threshold. We used multiple repeats and also subsampling of the full compendium to evaluate the results, and found that our FCs were well-converged and reproducible. Many of the FCs also had GO annotations for distinct pathways and biological processes, and a few FCs corresponded directly with some of the gene signatures in MSigDB’s H collection Fig. . We noteDespite the huge reduction in feature space, our FCs are able to capture critical biological features of a dataset and performed well when used in subsequent classification task. In the rheumatoid arthritis study GSE71370), Rajasekhar et al. noted in their paper  that the370, RajaAn additional benefit of using the FC scores as features is that it is easy to train tree-based models that are more interpretable than SVM. Using the same leukemia dataset, we trained a random forest, which would have been challenging to do if the full gene space  was used instead. We identified FC18, FC 39 and FC54 as the top three most important features Fig.  in separFC-based models are also robust in low-sample count settings. By subsampling two related leukemia classes from GSE13159 at various percentage levels, we obtained estimates of how well a classification model would perform if trained only on limited data. We note that the FC-space models generally perform better than the corresponding full gene-space models when the subsampling percentage is low, displaying higher NPV, sensitivity and accuracy Fig. . AlthougThe use of FCs can also help in denoising the transcriptomic data by emphasizing changes in the active genes. Clustering of MAQC data in both gene and FC space were observed to be highly correlated, although the latter displayed slightly higher cluster purity. We also considered a set of samples from AML patients collected across three different study centers in Germany (GSE15434) to study how the batch effects Fig.  are affeThe FCs we have found are also biologically relevant, and the traditional workflow for identifying DE genes can be done analogously to identify DE FCs. In GSE71370, we compared rheumatoid arthritis synovial fluid samples (RASFM) indirectly with HCPBM  and identified eleven DE FCs, which were associated with 75 unique GO annotations. Many of them were immediately recognizable as markers of inflammation and apoptosis. A few also had GO annotations that corresponded to lipopolysaccharides (e.g. GO:0071222), which have been reported  to be anThe compact representation of the full gene values enables a quick scan across GEO datasets to search for similar samples (“neighbors”) based on gene expressions. We demonstrate this search process using the samples in GSE66533, obtaining 43 “neighbors” for the Fusion-Negative group and 38 “neighbors” for the Fusion-Positive group. We note, however, that the rhabdomyosarcoma samples are highly correlated between the two groups (mean Pearson correlation: 0.970). This means that the “nearest” sample to a member of the Fusion-Positive group is also likely to be “near” members of the Fusion-Negative group. Of the “neighbors” we obtained, 30 were common to both groups. Although leiomyosarcome samples seem to be exclusively found in Table R package (humanFC) that contains the FC loadings and other functions for users to perform simple analysis. The package can be downloaded at https://simtk.org/projects/humanfcOur FC methodology negates the need for selection of genes to be used in classifier models, allowing it to be easily implemented in smaller studies where data is scarce. Its superior performance in classification tasks at sample sizes common to clinical studies (less than 50 samples) suggest that it is beneficial to perform analysis in FC space. We have written up an Additional file 1:Text S1. Axis labels for Figures in Paper. Table S1. MeSH Annotations of Representative Compendium Samples. Figure S1. KNN-kneeplots from Full Compendium. Figure S2. Tanglegram of FC and Gene space dendrograms (MAQC data). Figure S3. Heatmap of all samples in nervous system (GSE3536). Figure S4. Clustering of GSE71370 samples using 6636 DE genes. Figure S5. Reanalysis of GSE7538 . Figure S6. Number of unique GO terms versus number of leading principal components from PCA. Table S2. Full GO annotations for DE FCs in E-MTAB-3162. Table S3. Full GO annotations for DE FCs in GSE71370. (ZIP 1335 kb)


Combining genomic data sets from multiple studies is advantageous to increase statistical power in studies where logistical considerations restrict sample size or require the sequential generation of data. However, significant technical heterogeneity is commonly observed across multiple batches of data that are generated from different processing or reagent batches, experimenters, protocols, or profiling platforms. These so-called batch effects often confound true biological relationships in the data, reducing the power benefits of combining multiple batches, and may even lead to spurious results in some combined studies. Therefore there is significant need for effective methods and software tools that account for batch effects in high-throughput genomic studies.Here we contribute multiple methods and software tools for improved combination and analysis of data from multiple batches. In particular, we provide batch effect solutions for cases where the severity of the batch effects is not extreme, and for cases where one high-quality batch can serve as a reference, such as the training set in a biomarker study. We illustrate our approaches and software in both simulated and real data scenarios.We demonstrate the value of these new contributions compared to currently established approaches in the specified batch correction situations.The online version of this article (10.1186/s12859-018-2263-6) contains supplementary material, which is available to authorized users. In the past two decades, owing to the advent of novel high-throughput techniques, tens of thousands of genome profiling experiments have been performed , 2. ThesOne illustrative example of cross-platform (and within-platform) heterogeneity can be found in The Cancer Genome Atlas (TCGA) . ProfiliMany prior studies have clearly established the need for batch effect correction , 6. To aHowever, despite these useful existing approaches for data cleaning and combination, there are still significant gaps that need to be addressed for certain data integration scenarios. For example, the ComBat approach removes batch effects impacting both the means and variances of each gene across the batches. However, in some cases, the data might require a less (or more) extreme batch adjustment. Below, we present higher order moment-based metrics and visualizations for evaluating the extent to which batch effects impact the data. Then, at least in the case of less severe batch effects, we propose a simplified empirical Bayes approach for batch adjustment.Another limitation of the current ComBat model is that it adjusts the data for each gene to match an overall, or common cross-batch mean, estimated using samples from all batches. While this approach is advantageous for cases with small sample size or where the batches are of similar caliber and size, this is not the best solution when one batch is of superior quality or can be considered a natural ’reference’. In addition, the current ComBat approach suffers from sample ’set bias’ , meaningAlthough these alternative models for batch only represent alternative formulations of the original ComBat modeling approach, their implementation will have significant downstream impacts on certain batch combination scenarios. Below, we detail these modifications and demonstrate their utility and increased efficacy on real data examples.We present several approaches for improved diagnostics and batch effect adjustment for certain batch adjustment situations. We focus on developing models based on the ComBat empirical Bayes batch adjustment approach, although similar methods and models can be applied to other existing approaches. One set of diagnostic procedures attempts to characterize the distributional  differences across batches. We present a solution for the cases where adjusting only the mean of the batch effect is sufficient for harmonizing the data across batches. In addition, we present an approach that allows the user to select a reference batch, or a batch that is left static after batch adjustment, and to which all the other batches are adjusted. This approach makes sense in situations where one batch or dataset is of better quality or less variable. In addition, this approach will be particularly helpful for biomarker studies, where one dataset is used for training a fixed biomarker, then the fixed biomarker is applied on multiple different batches or datasets, even at different times. This approach avoids the negative impacts of test set bias in the generation of the biomarker signatures. Below we describe the methodological developments for these cases.Yijg) for gene g, batch i, and sample j (within batch i) follows the model: ComBat  is a fleαg is the overall gene expression. Xij is a known design matrix for sample conditions, and βg is the vector of regression coefficients corresponding to Xij. γig and δig represent the additive and multiplicative batch effects of batch i for gene g, which affect the mean and variance of gene expressions within batch i, respectively. The error terms, εijg, are assumed to follow a normal distribution with expected value of zero and variance γig and δig) and uses an empirical Bayes procedure to estimate these parameters [where rameters . This prThe ComBat model described above robustly estimates both the mean and the variance of each batch using empirical Bayes shrinkage, then adjusts the data according to these estimates. However, in some cases, adjusting only the mean of the batches may be sufficient for further analysis. In other scenarios (see examples in “” below), adjustment of the mean and variance is not sufficient, and thus the adjustment of higher order moments is needed. Here, we present multiple diagnostics for interrogating the shape of the distribution of batches to determine how batch effect should be adjusted.Zijg, as follows: As with ComBat, we assume that in the presence of batch effect, the mean and variance (as well as higher order moments) of gene expression demonstrate systematic differences across batches on a standardized scale . Thus, wZijg, originate from a distribution with mean γig, variance ηig, and kurtosis ϕig. In addition, consistent with the ComBat assumptions, we assume that each of these moments originates from a common distribution (henceforth denoted the hyper-distribution), namely that the γig are drawn from a distribution with mean γi that is common across all genes. Similar assumptions of exchangeability across genes are made about the variance, skewness, and kurtosis.After standardization, we assume the standardized data, We apply two tests of significance to individually test for significant differences in these moments across batches. In both of the tests, we estimate and conduct the test on the hyper-moments (i.e. moments of the hyper-distribution) across batches. The first test estimates the hyper-moments within each sample, whereas the other test estimates the hyper-moments within each gene. The first, sample-based test is more robust for small sample size, whereas the second, gene-based test is more robust and sensitive in larger sample size. Finally, for quantile-normalized data , the samrobust F-test (described below) to compare the moment estimates across batches. For example, for the mean, the sample-wise test first estimates the mean gene expression of each sample, namely The first test is a sample-level test that estimates the hyper-moments by summarizing the moments of gene expression within each sample, and then conducts a standard or ng is the total number of genes. We then conduct an F-test on the Zijg are estimated across genes within each sample (using standard estimation approaches for these moments) and then tested for significant differences across batches in the same way as the mean. Overall this is not specifically testing the moments of the assumed ComBat model hyper-distribution, but rather the marginal distribution of the data and hyper-distributions of the data.where The second test is a gene-level test that estimates the hyper-moments within each gene, using samples in each batch separately, and then conducts a robust F-test comparing the moment estimates across batches. For example, for the mean, the gene-wise test first estimates the mean of each gene across samples within a batch, namely ni is the total number of samples in batch i. We then conduct a test of significance on the Zijg are estimated across samples within each batch for each gene (using standard estimation approaches for these moments) and then tested for significant differences across batches in the same way as the mean. Unlike the sample-wise test, this test more specifically follows the ComBat hierarchical model assumption, by first estimating the parameters drawn from the hyper-distribution.where P-value problem [P-value quickly drops to zero under a very large sample size. The P-value problem influences tests that are sensitive to the sample size, including the F-test used in this study. To address this problem and better interpret the results of the two tests above, we applied a robust F-test. The robust F-test is modified from standard F test, by adding a variance inflation factor in the F statistics, which accounts for the influence of the sample size in P-values. Details of the robust F-test are documented in Additional file In hypothesis testing, a large enough sample size may cause the  problem : small eThe robust F-test is especially useful for the gene-level test, as the total degrees of freedom in this test is in fact the amount of moment estimates, which equals to the number of genes times the number of batches. This value can easily become very large in genomic studies. We used both the robust and the non-robust versions of F statistics in the sample- and gene-level tests, and evaluated and compared their performances in diagnosing the degree of batch effect in our example data.The current ComBat model adjusts for effects in both the mean and the variance across batches. However, for some datasets, after testing for the moments of the batch effect, it may be determined that differences are only present in the mean across batches. Other datasets may be expected to have variance differences across batches for non-technical reasons, such as in a study combining a in vitro perturbation experiment (low variance) with patient samples (high variance). For cases in which that batch differences are only present in the mean, we have modified the current ComBat model to only adjust the mean batch effect. Specifically, we modify the ComBat model  as folloand then use the same approach for standardization and shrinkage as described previously with the exception of not estimating and adjusting for variance differences across batches.Many batch adjustment approaches, including ComBat, are dependent on the datasets in hand for their batch adjustments. In other words, if additional samples or batches of data are added, the batch adjustments and adjusted data would be different. We present a reference-based batch adjustment approach that uses one batch as the baseline for the batch adjustment. The reference batch is not changed and the other batches are adjusted to the mean and variance of the reference. Thus, as long as the reference batch does not change, the adjustments and adjusted data would be the same, regardless of the batches of data that are included in the dataset. This also allows batches of data to be adjusted at different times without impacting the results. This approach will be advantageous to data generating consortiums where data arrive sequentially in small batches. It will also be important for applications in personalized medicine where biomarkers need to be established and validated prior to the collection of patient data. For our reference-based version of ComBat, we will assume a model slightly different than the model  presenteXij and βrg are the design matrix and regression coefficients as described before, but αrg is the average gene expression in the chosen reference batch (r). Furthermore, γrig and δrig represent the additive and multiplicative batch differences between the reference batch and batch i for gene g. The error terms, εijg, are assumed to follow a normal distribution with expected value of zero and a reference batch variance γrig and δrig will be obtained as in the current ComBat approach.where The models presented here have been integrated into the ComBat function available in the ’sva’ Bioconductor package (version 3.26.0) , 19. MorN, and an ’after’ distribution: N. The rest of the genes are drawn from a N distribution in all 6 samples, representing genes that do not respond to the pathway perturbation. Batch 2 consists of a 200  by 600 (sample) matrix, and represents a large and highly variable patient data set. The 600 patients are divided equally into 6 subgroups with different levels of pathway activation between groups; signature genes are drawn from a N distribution, where μ=0.5,0.7,0.9,1.1,1.3, and 1.5 for the six subgroups. The control genes are drawn from a N distribution. We set up these simulation studies based on the design of real signature profiling studies [https://github.com/zhangyuqing/meanonly_reference_combat.We generated simulated data to represent a case where we (1) derive a gene expression signature of a biological pathway or drug perturbation, and 2) profile the signature into another batch of data to predict pathway activity (or drug efficacy). The study consists of two experimental batches which are designed as follows: batch 1 is given by a 200 (gene) by 6 (sample) matrix of expression data, where the columns contain three replicate samples before pathway activation and three after activation (i.e. overexpressing key pathway driving genes). Among the 200 genes, the first 100 represent ’signature genes’ that are differentially expressed (before vs. after) based on a ’before’ Gaussian distribution:  studies , and sel profile We used a previously published bladder cancer microarray dataset, which aims to measure gene expression in superficial transitional cell carcinoma (sTCC) in the presence and absence of carcinoma in situ (CIS) . This daThis study was designed to investigate whether exposing mammalian cells to nitric oxide (NO) stabilizes mRNAs . Human lThe growth factor receptor network (GFRN) contributes to breast cancer progression and drug response. This RNA-Seq dataset is designed to develop gene signatures for several GFRN pathways: AKT, BAD, HER2, IGF1R, RAF1, KRAS, and EGFR. We used recombinant adenoviruses to express these genes in case samples and produce green fluorescent protein (GFP) in control samples, using replicates of human mammary epithelial cells (HMECs). RNA-Seq data are collected from these HMECs overexpressing GFRN genes and GFP controls . This daThis dataset contains microarray measurements from histologically normal bronchial epithelium cells collected during bronchoscopy from non-smokers, former smokers, and current smokers. Samples are selected from various studies, which are divided into three batches A , but has P-values above 0.33 for variance differences for both tests. Since the bladder cancer dataset only exhibits batch effects in the mean and not in the variance, mean-only adjustment is more suitable for this dataset. In the nitric oxide dataset, however, mean/variance ComBat is required to remove the difference in batch variances detected by the gene-wise test . All four datasets show certain levels of significant differences in skewness and/or kurtosis even after the mean/variance ComBat is used, which suggests that adjustment for higher order moments may be required, which is beyond the scope of this paper.We introduced sample- and gene-wise tests to detect significant differences in the moments of batch effect distributions. We applied these tests to four different datasets Table  and obseWe modified the current mean/variance ComBat into a mean-only version of ComBat, which allows users to only adjust the batch effects in mean. It is recommended for cases where milder batch effects are expected (i.e. there is no need to adjust the variance). For example, we have shown in Table P<0.0001), the mean estimates in the original data are significantly different across batches. In particular, Fig. We applied both the mean-only and mean/variance ComBat on the bladder cancer dataset to compare their performances. We compared batch mean (from Eq. ) and varmentclass2pt{minimmentclass2pt{minimmentclass2pt{minimmentclass2pt{minimFrom the gene-wise perspective, we found that the mean/variance ComBat overcorrects the data by shrinking the variance of batch 3 and 4 when pooling information from all batches  in the nitric oxide dataset. In this dataset, we found that mean-only ComBat cannot completely remove the significant difference in gene-wise variance estimates across batches  no batch effect, 2) only differences in the mean, and 3) both mean and variance batch effects. We applied both the mean-only and the mean-variance ComBat on each dataset. Then in both adjusted and unadjusted data, we performed differential expression analysis, and calculated the type I error rate and statistical power of our detection. We observed that using the ComBat model corresponding to the type of batch effect in the data is able to gain more power of detection, at the same cost of type I error rate increase  biological pathway or drug perturbation that is profiled into another batch of data to predict pathway activity (or drug efficacy). Descriptions of these simulated datasets are detailed in the Dataset descriptions - Pathway simulation section. We used the two versions of ComBat  to combine the two batches and to enable the prediction of the activity strength of the pathway from batch 1 into the batch 2 samples. Batch 1 was selected as reference for the reference-batch ComBat. Pathway activation levels are added in both versions ComBat as covariates. Results of not using activation levels as covariates is shown in Additional file The original and reference-batch ComBat yield very different results in the two batches Fig. . The oriTo quantify the impact of batch correction on batch 1, we use a k-means clustering approach to attempt to identify the biomarker gene set (the first 100 genes are the signature genes and the subsequent 100 genes are unaffected by the perturbation). We treat the gene expression of each sample as high-dimensional vectors . We used k-means clustering to divide these vectors into two groups for batch 1 alone, batch 2 alone, and batches 1 and 2 combined, with both ComBat adjustments . We compared the clustering assignment of genes with the signature/non-signature separation, and calculated the accuracy as the maximum percentage of correctly classified genes in either way of labeling the two clusters as signatures and non-signatures. We evaluated how using original and reference-batch ComBat affects this accuracy.In batch 1 without adjustment, all genes are correctly separated into signature and non-signature. However, this separation is confounded when batch 2 is combined with batch 1, as only 58.5% of the genes are correctly separated in the combined dataset. When using original ComBat, because the variance of batch 1 is artificially increased, the accuracy in batch 1 alone drops from 100 to 54.5%, and only 64.5% of the genes maintain their correct signature/non-signature labels after combining batch 2 with batch 1. In contrast, reference-batch ComBat keeps the cluster assignment in the adjusted batch 1 100% correct, because batch 1 stays intact as the reference, and 91% of the genes retain their correct labels in the combined dataset after adjustment. Thus reference ComBat improves the ability to identify biomarker genes across multiple studies compared to no adjustment and standard ComBat.We also considered a real signature study using ASSIGN , a pathwWe used ASSIGN to develop a 50-gene signature from the EGFR samples in the training set . We firsWe further compared the correlations of pathway predictions with the following validation datasets: (1) EGFR protein expression data (cell line and TCGA), and 2) EGFR inhibitor drug response (cell lines). As shown in Table  EGFR inhCombining multiple genomic datasets is beneficial for boosting statistical power of studies, especially in cases where data are generated in small batches necessitated by high experimental cost or difficulties in collecting samples. In addition, combining batches of data from similar experiments from different labs also provides opportunities for increased power for the detection of biological differences, as well as providing ways for testing/validating biomarkers generated in one batch of data. The presence of batch effects due to technical heterogeneity and batch effects due to different profiling platforms, protocols, or other factors can often confound the biological signals in data, which reduces the benefit of combining datasets. Despite the many previously developed techniques for batch adjustment, there are still situations where new methods need to be developed to appropriately or more effectively remove batch effects.We introduced new models and tools for addressing batch correction in several scenarios based on the ComBat model. For example, many methods focus on adjusting the means and variances of batches, but some datasets require less (or more) stringent adjustments. We proposed two significance testing approaches, based on the batch effect moment distributions, to diagnose the degree of adjustment required. We visualized different degrees of batch effect detected by these tests in four experimental datasets. In the bladder cancer data where mean is significantly different across batches but not the variance, our proposed mean-only ComBat successfully removes the batch differences. In all four datasets, we presented evidence that adjusting both mean and variance is not sufficient to remove all batch effect, which calls for a method that performs a more severe batch adjustment.For datasets where a less severe adjustment targeting the mean is sufficient, adjusting the variance may lead to unnecessary costs in downstream analysis. The original ComBat model pools all samples to estimate both the mean and the variance batch effects, which introduces a correlation structure between the samples. Such correlations may cause issues in further analysis, such as inflating type I error rate in differential expression detection, if we do not account for them properly. Therefore, for the datasets with only mean batch effect, our proposed mean-only ComBat is able to avoid the cost of estimating the variances, and is more beneficial than the mean-variance version.It is important to highlight that a thorough evaluation of the degree of batch effect is necessary before applying any version of ComBat. We presented simulation results to demonstrate that using the ComBat model corresponding to the type of batch effect in the data is able to achieve more statistical power at the same cost of type I error rate in differential expression analysis. Therefore, we strongly suggest evaluating whether there is differences in mean and variances between batches with the moment-based significance tests, and selecting the appropriate version of ComBat based on the type of batch effect in the data.Also, we noticed that the gene-wise and sample-wise tests yielded different results in a few of our datasets. To resolve a conflicting result from the two tests, we recommend visualizing the batch distributions as we did in this study, and decide the level of adjustment required based on all available information. BatchQC offers visualizations from various aspects including boxplot of gene expressions, dendrogram of clustering, and scatter plot of principal components, which can all assist in diagnosing the degree of batch effect.In addition, we illustrated the benefits of selecting a reference batch in batch correction, in situations when one batch is high quality and less variable, and when biomarkers need to be developed from one study, fixed and validated on another study. Particularly in the situation where the goal is to generate fixed biomarkers, including an extra batch of data to those in hand can strongly affect the results, an issue described as ‘set bias’. In these situations, analysis need to be re-run in order to process the new batch of data, which can cause the biomarker genes to be largely different.Our reference-batch ComBat is proven more successful in retrieving biological signals in signature profiling examples, where one batch shows a clear signal of biological conditions. We demonstrated that reference-batch ComBat resolves the ‘set bias’ caused by the original version of ComBat in adding data sequentially, and yields better prediction of pathway activities and drug effects. Although these approaches are only alternative expressions of the ComBat model, their implementation has critical impact in real batch correction scenarios.We proposed diagnostic tools and improved models based on ComBat to evaluate and address batch effects in certain batch adjustment situations. The significance tests for batch differences can be used to determine the degree of batch effects to be adjusted. We purposed mean-only ComBat for the situation where a less severe adjustment is preferred. The reference-batch ComBat is able to leave one batch unchanged, which is especially useful for generating a fixed biomarker for further clinical use. We have shown in both simulations and real data that these proposed methods provide better solutions to batch effects than the existing ComBat model in their corresponding batch correction scenarios.Additional file 1Supplementary materials (PDF 1612 kb)Additional file 2Mean and variance of gene expression distributions estimated from the EGFR signature and the TCGA breast cancer patient datasets. In TCGA, we used proteomics data of the patients, and binned the EGFR protein expression into 6 gradually increasing levels, partitioning all patients into 6 equal-sized groups. Mean and variances are estimated within each group. Up- and down-regulated genes are both EGFR signature genes derived by ASSIGN. The design and parameters for our simulation studies resemble the real estimates in these tables. Batch 1 represents the EGFR signature dataset with small gene variances, and a clear separation between the two condition groups in the expression of up-regulated genes. Batch 2 resembles the TCGA patient data with much larger variances than Batch 1. (XLSX 10 kb)Additional file 3P-values for applying robust and non-robust F tests on the four experimental datasets. (XLSX 12 kb)Comparison of