Microbiome samples harvested from urban environments can be informative in predicting the geographic location of unknown samples. The idea that different cities may have geographically disparate microbial signatures can be utilized to predict the geographical location based on city-specific microbiome samples. We implemented this idea first; by utilizing standard bioinformatics procedures to pre-process the raw metagenomics samples provided by the CAMDA organizers. We trained several component classifiers and a robust ensemble classifier with data generated from taxonomy-dependent and taxonomy-free approaches. Also, we implemented class weighting and an optimal oversampling technique to overcome the class imbalance in the primary data. In each instance, we observed that the component classifiers performed differently, whereas the ensemble classifier consistently yielded optimal performance. Finally, we predicted the source cities of mystery samples provided by the organizers. Our results highlight the unreliability of restricting the classification of metagenomic samples to source origins to a single classification algorithm. By combining several component classifiers via the ensemble approach, we obtained classification results that were as good as the best-performing component classifier. This was followed by classification to city labels using the random forest. Zhu et al.  paired-end whole genome shotgun (WGS) metagenomics data of 1,065 samples from 23 cities across 17 countries and (2) metadata comprising of the biome, weather, and location information for these cities.Bogota, Krakow, Marseille, Naples, and Vienna. The primary dataset corresponds to the “training set,” while the mystery samples make up the “test set” for validating the classification models built during the analysis phase of the study. Furthermore, WGS data for 121 “mystery” samples from 10 different cities were subsequently provided by the organizers. Among the 121 “mystery” samples, 56 samples originated from cities that were not represented in the primary dataset. Cities in the mystery data which were not represented in the primary data include To prepare WGS data for downstream machine learning analysis, we construct a standard pipeline to carry out necessary bioinformatics procedures. First, we obtain the raw paired-end reads from the host server, and then, we perform some quality control and filtering to obtain good quality reads. After the quality control, we perform metagenomic taxonomic profiling of the sequencing reads. At the termination of the bioinformatics procedures, the final output is a relative abundance table. All the bioinformatics procedures carried out in this phase of our analysis were performed using the University of Florida HiperGator supercomputer.The first procedure in the bioinformatics procedure entails the quality assessment of the WGS data using FastQC Andrews,  and MultBowtie  of all genomes known to contain a given k-mer. The LCA approach adopted by the Kraken system usually results in the underestimation of the number of reads which are directly classified at lower taxonomic levels ), of dimension 1, 047 × 4, 770 where the rows represent samples from different cities, and the columns represent taxa. Here, the number of features is greater than the number of samples. It is usually desirable to reduce the feature space and thereby get a set of informative features that explain the variation in the samples. Several techniques exist for performing feature selection. For example, relevant features can be selected based on the k top-ranking importance scores returned by a fitted random forests model. To maximize classification accuracy, k can be chosen from a set of a predetermined number of features via cross-validation. The final classification model can then be trained with this set of k features.After the initial preprocessing of the species-level abundance table, we get a data matrix For our analysis, we focus on the hierarchical feature engineering (HFE) method . There is an over-representation of certain classes in the primary data. For instance, Ilorin and New York City have 98 and 96 samples, respectively, whereas there are only 19 samples from Lisbon. Fitting machine learning models with an apparent class imbalance in the training data is likely to make the models to be biased toward the majority classes. In such a scenario, the fitted classifier is likely to predict the label of a random sample to be a majority class. Learning from imbalanced data is a popular topic in machine learning research, and the literature is rife with several techniques for handling such a problem. Amongst these techniques include down-sampling, up-sampling, hybrid methods, and class weighting. Down-sampling involves randomly removing samples from the majority classes until class frequencies are roughly balanced, while up-sampling involves randomly replicating instances in the minority classes to achieve the same size as the majority class. In practice, the utilization of these sampling techniques usually comes at a cost. For instance, down-sampling the majority classes results in potential loss of relevant information. While oversampling the minority classes is likely to introduce more bias into the fitted model.In the primary data, there are Hybrid methods such as SMOTE (and its variants), AdaSyn, DSRBF, ProWsyn, and many others oversamples the minority classes by creating synthetic samples from those classes. Kovács  gives anc = 1, …, C where nc the number of samples from city c.Moreover, class weights can be used to train several classifiers. In this sense, the algorithm puts more weight on the minority classes, thereby imposing a heavier cost when an error is made in the classification of labels from these classes. We define the class weights as In a general sense, the implementation of these techniques does not improve the overall fit of the model, and they do not increase the information in the training data , random forests, and neural networks have been widely used for the prediction of the origin of a given metagenomic sample. For instance, Casimiro-Soriguer et al.  used decIn principle, no single classification algorithm performs optimally on all types of data. In particular, the performance of the classification algorithm is likely to depend on the techniques used during the bioinformatics preprocessing and taxonomic profiling of the metagenomics data. The application of these individual techniques is likely to yield variable competing results, and notably, the classification accuracy from these methods is likely to depend on the structure of the available data. A similar observation was noted while discussing the results of a classification competition based on proteomics data Hand, .In this study, we adopt a method for combining a variety of classification algorithms in addition to dimension reduction techniques (if necessary) rather than employing stand-alone classification algorithms for metagenomics classification problems. Datta et al.  developen samples and p taxa, Xnxp and the n-vector of classification labels, y. First, the data is partitioned into training and test sets. Set N, the number of independent bootstrap samples to draw from the training set. Also, select M candidate classification algorithms and K performance measures. Then the following iterative steps are performed for j = 1, 2, ..., NNext, we outline the steps presented by Datta et al.  for the jth bootstrap sample. The bootstrap sample is obtained by simple random sampling with replacement and the sampling is performed such that all classes in the primary data are represented in the bootstrap sample. Samples that are not included in the bootstrap sample are called out-of-bag (OOB) samples.Obtain the M classification models on the jth bootstrap sample.Train all M classification models predict the class labels for the OOB set.Using the trained K performance measures.Based on the true values of the OOB set, and the predicted labels, compute the K ordered lists, L1, L2, ..., LK, each of size M. Using weighted rank aggregation, the ordered lists are aggregated to determine the best single performing classifier denoted as Perform weighted rank aggregation: the performance measures used in step (iv) ranks the classifiers according to their performance under such measure, thereby producing The ensemble is a set of To avoid overfitting the classifiers, the performance of each classifier is evaluated based on their prediction of the OOB samples. This approach is similar to the cross-validation procedure, and the misclassification error from the OOB samples approximates the testing set error.xp1x, prediction using the ensemble classifier is based on the following procedures.Given a new sample xp1x. Let ŷ1, ..., ŷN denote the class predictions from the N models in the ensemble.Each classifier, N predicted classes.The final prediction is obtained by majority voting, that is, the most frequent label among the For the analysis of the species abundance data obtained after bioinformatics pre-processing of the WGS data, we performed three distinct analyses for both the abundance table with a complete feature space and the table with a reduced feature space. We aimed to classify the species abundances of each metagenomic sample to a source city that is present in the primary data set. Our analyses focused on training standard classification algorithms, classification algorithms with class weights, and implementing an optimal over-sampling technique when training the classifiers. The implementation of the last two techniques was to handle the problem of class imbalance. Here, the candidate classifiers considered for the analyses include the random forest (RF) Breiman, , supportFurthermore, for each analytical method presented, we constructed the ensemble classifier discussed in section 2.6 which comprises the chosen candidate classifiers. Note, the ensemble classifier may not be restricted to only distinct individual classifiers. Classifiers with different tuning parameter combinations can also be considered as candidate classifiers. Moreover, hyperparameters of the candidate classifiers can be tuned to attain optimal performance. However, tuning hyperparameters will increase the time it takes to construct the ensemble classifier.P0 is the relative observed agreement among classifiers , PE is the probability that agreement is due to chance, K is the number of classes, and Recalli is the recall for class i. For classification problems with imbalance data, these measures are usually preferred over say, the classification accuracy. Since accuracy is only marginally impacted by rare classes in the data, the classification accuracy is often not an appropriate measure to employ for imbalance learning  dimension reduction technique applied in the analysis, we compare the classification results shown in nc is the number of samples in class c. For the oversampling procedure, we employed the GaussNoise oversampling technique. For each classifier in either Furthermore, to assess the impact of the methods used to handle the class imbalance problem, we compare the classification results across the standard models, the models with class weighting, and models for which an oversampling procedure is implemented. For the classifiers trained with class weights, the weights were computed as Now, we turn our attention to the performance of each classification algorithm. Across the different analyses conducted and for the performance measures utilized, the ensemble classifier gives classification results as good as the best performing candidate classifier. Also, we observe the variation of classification performance among the candidate classifiers. The MLP, PLS+RF, RF, PLS+XGB, and XGB classifiers were mostly the best-performing candidate classifiers. Also, the RPart, PLS+RPart, and the SVM give moderate classification performance. For the analysis of the data with complete feature space, the classifiers trained with integrated PLS terms give very good classification performance. On the contrary, classifiers trained with integrated PCA terms were the worst performing classifiers. Moreover, from the last column of the subtables of Stockholm, Barcelona, Berlin, Denver, Doha, Hong Kong, Seoul, Ilorin, Kuala Lumpur, London, New York City, São Paulo, Santiago, Sendai, San Francisco, Taipei, and Tokyo. The average precision value for these cities was at least 85%. On the other hand, the classifier gave only moderate prediction result for Kyiv, Lisbon, Singapore, and Zurich. For the prediction Lisbon, which is a minority class in this study, the average precision value substantially increases from 66 to 73% when the class weighting procedure is employed as against the standard fit. However, we do not see the same effect when the over-sampling technique is applied. This empirical finding may be due to oversampling employed, further investigation will be done elsewhere to understand the impact of oversamplers in the analysis of microbiome data. Further, we observe the classifier does not perform well in distinguishing samples from Kyiv and Zurich, this pattern also persists for Ilorin and Offa. Geographically, Ilorin and Offa are two urban centers that are about 59.6 km apart in the same territory in Nigeria, whereas Kyiv and Zurich are European cities that are over 2,000 km apart. Also, we noticed a reverse misclassification between Singapore and Kyiv, these cities lie on different continents.For the prediction of the source origins, the classifiers give a varying performance for predicting the cities in the primary data. To elaborate on this point, we describe the prediction performance of the ensemble classifier that was trained on data with complete feature space. Bradyrhizobium which is a soil bacteria and is also found in the roots and stems of plants  distance between the mystery sample and the samples in the primary data is obtained as the prediction for the mystery sample. In this projection approach, for each mystery sample, we compute the KL distance between the mystery sample and all samples in the primary data. The city in the primary data that minimizes the KL distance is selected as the prediction of the source origin for the mystery sample. Following this approach, we obtained a classification accuracy of 52%. For each mystery sample, we also considered the source cities corresponding to the least three KL distances, we noticed that the true source origin falls in the set of the three cities 66% of the time.The results discussed in section 3.1 were based on the analysis of the abundance data obtained after taxonomic profiling of the WGS data with the kraken2-bracken system. Typically, the taxonomic profiling step involves classification or mapping of sequence reads to their taxon of origin using a well-curated reference database of microbial genomes. For instance, kraken2 . For each sample obtained after the bioinformatics pre-processing step described in section 2.2, we utilize the KMC 3 , and it ignores k-mers containing “N” symbol (ambiguous nucleotides). We set k = 31, and we filtered out k-mers occurring less than 20 times (this threshold was set to 50 for samples gotten from Offa and Ilorin). For each city, we merged the k-mer frequency vectors generated for all samples. We observed a significant variation in the total number of unique k-mers obtained for the samples in each of the 23 cities in the primary set. For instance, there were a total of 6M, 47M, 268M, and 39M unique k-mers for the samples obtained from Denver, Hong Kong, Offa, and Zurich, respectively. We observed the samples in the mystery set generated a total of just 17M unique k-mers. k-mers corresponding to each of the 23 cities in the primary set and the mystery set.In this section, we describe the design of a microbial fingerprint by the direct modeling of DNA fragments obtained from metagenomics samples using an ensemble classifier. The DNA fragments are derived by breaking down the sequence reads into short sequences composed of k-mer frequency vectors from all available samples, we obtain a large sparse matrix with 923M unique k-mers. We performed some pre-processing on the matrix. First, we filter the data matrix to retain only k-mers that are present in at least 10% of the samples. A total of 298,644 k-mers were retained after filtering. Secondly, we obtain normalized counts for each k-mer corresponding to each sample.After combining the k-mer frequencies as feature vectors, we train the classifiers with class weights and without class weights. First, we describe the results for the analysis of the primary data. k-mer frequency vectors gives a comparative performance as to when trained with taxonomic profiles. In contrast, there is greater diversity among the classifiers that make up the ensemble for predicting the abundance data, whereas the XGB and the RF (in particular) dominate the ensemble for the k-mer counts data. Also, we observe that the utilization of feature reduction terms (PLS and PCA) when training the classifiers based on the k-mer frequency counts does not yield better classification results.To fit the ensemble classifier, we utilize the same settings discussed in sections 2.7 and 3.1. Using the normalized k-mer approach, the ensemble classifier that comprises unweighted classifiers reached a classification accuracy of 35%, while the ensemble classifier based on weighted classifiers attains 33% accuracy. Some potential reasons could explain the discrepancy in the classification performance for samples in the primary set and the samples. First, with the diversity of samples in the mystery set (121 samples from 10 different cities), one would expect the total number of unique k-mers obtained from the mystery set to be greater than the total number of k-mers obtained from any of the 23 cities in the primary set. However, from k-mers from the mystery set was lesser than the amount returned from 15 cities. Secondly, the discrepancy may be due to the batch effects for the primary and mystery data, and there may have been a difference in the sequencing procedures utilized for each batch. If there was a difference in the techniques used to generate the raw WGS data, the taxonomy-free approach appears not to be agnostic to the different techniques. Note, we implemented a uniform bioinformatics pipeline for the pre-processing of the raw WGS data. Lastly, when training the classifier with k-mer frequency vectors, it is difficult to know if the classification is driven by noise or pure biological relevance. For instance, we observed little or no changes in the classification performance when we discard k-mers that were not present in more than 1 to 30% of the total number of samples in the primary data. Besides, when feature reduction is implemented using t-SNE, PLS, or PCA, we observed the classifiers give poor results. In contrast, when we train the classifiers with an abundance matrix, classification is driven by taxonomy profiles that have biological meaningfulness.For predicting the samples in the mystery set, we restrict the analysis to samples whose source origins were present in the primary data. Overall, the classification accuracy for the mystery set using this approach was much worse than the accuracy for the primary dataset and the taxonomy-dependent approach. For the k-mer counts. It considers .fastq files as the input and processes the data in two major steps. The first step computes an n × p k-mer count matrix, where n is the number of samples and p is the number of distinct k-mers among all the samples. Based on the k-mer count matrix in the first step, it computes a n × n distance matrix. Table 1 of Benoit et al. . The city label that corresponds to the minimum distance from mystery sample Sj is regarded as the predicted city label for that mystery sample. We utilized several different distance measures to implement the described method. The accuracy for the 18 distance measures that we used, varied from 0 to 28.57%. We observed that the qualitative distance measures i.e., based on presence-absence data had slightly higher accuracy than quantitative distance measures, i.e., based on abundance data. The 28.57% prediction accuracy for mystery samples was obtained by using qualitative Kulczynski distance measure. The average distance in the above method was computed using leave-one-out cross-validation to mitigate overfitting.We attempted yet another taxonomy-free approach called Simka , the analyst may prefer to adopt the analysis based on data with reduced feature space since such case will have a lesser computational burden.To address the problem of class imbalance, we trained the classifiers with class weights, and we also adopted an oversampling technique. We observed that there was no substantial improvement in the overall classification performance when these methods were employed. In a microbiota study, Knights et al.  utilizedThe ordination plots in Also, for any classification problem, it is natural for the analyst to consider a variety of classifiers. This usually depends on the objective of the scientific study under investigation. In practice, after carrying out standard pre-processing, the analyst initiates the classification analysis by evaluating the performance of simple models in classifying the abundance data to known class labels before experimenting with other advanced models. Based on user-specific performance metrics, the ensemble classifier offers the analyst the opportunity to evaluate the performance of different candidate classifiers on the chosen dataset in one shot. The ensemble classifier adapts to various bioinformatics procedures performed during pre-processing and taxonomic profiling of raw metagenomics data and will outperform or will adequately compete with any stand-alone classifier in its ensemble.k-fold cross-validation and a weighted rank aggregation procedure. In this case, each candidate classifier in a user-defined set of classification algorithms is used to predict the hold-out set, and the performance measures utilized for evaluating the model are averaged across the k folds. Then, based on the resulting performance measures, a weighted rank aggregation procedure is used to rank the candidate classifiers, and the classifier with the top-most rank is chosen as the best performing classifier.Moreover, before fitting the ensemble classifier, the analyst does not know the candidate classifier that will give the best classification performance. Also, it has been shown that the ensemble classifier performs better than direct techniques such as the greedy algorithm (Datta et al., Furthermore, the ensemble classifier itself is an adaptive classification algorithm that excels in its flexibility in harnessing the strengths of the individual classifiers to yield better classification performance. However, we emphasize that an ensemble classifier will also depend on the set of candidate classifiers in its ensemble. To improve the performance of pre-selected candidate classifiers, the analyst may choose to tune hyperparameters of such classifiers since the default parameter settings in most software may not be ideal for the problem under consideration. In addition to adapting to the nature of the training data, techniques for handling class imbalance and dimension reduction can easily be incorporated into the construction of the ensemble classifier.The apparent drawback of the ensemble classifier is the computational time required to train this classifier. The computing time is influenced by three main factors considered in the construction of the ensemble classifier - the number of candidate classifiers, the number of bootstrap samples for which each candidate classifier is trained, and the performance measures which are used in the weighted rank aggregation step of the construction procedure. Furthermore, the complexity of the candidate classifiers will also impact the computing times. For instance, more compute time will be required to train the MLP classifier than it will take to train the RPart classifier. If the analyst using the ensemble classifier chooses to tune the hyperparameters of the candidate classifiers, this will add some complexity to the process.Nonetheless, the compute time can be reduced substantially by running the procedures described in section 2.6 in parallel on a computing cluster. For instance, consider the scenario where we utilized nine candidate classifiers (namely the candidate classifiers presented in Bradyrhizobium and Pseudomonas are amongst the top taxa responsible for differentiating between the city locations. This type of classification analysis can be a useful tool in identifying species that vary across different types of environments, climatic conditions, and surface types. The information from such analysis can be further synthesized in the fields such as forensics and disease epidemiology for identifying harmful pathogens.The classification methods described in this paper can help identify several microbial species linked to specific geographic locations. For instance, several species from genus k-mer approach for the direct modeling of DNA fragments obtained from metagenomics samples. For samples in the primary data set, classification results based on data generated from the k-mer approach were as good as the results from the analysis of taxonomic abundance data. However, the k-mer approach performs poorly in identifying the source origins of samples in the mystery set.While the identification of organisms is instrumental in metagenomics forensics, a good portion of sequenced reads remains unclassified when standard reference databases for taxonomic profiling. Sequenced reads which remain unclassified are likely to represent novel taxa, and ignoring such genetic data when constructing a fingerprint may impose substantial limitations. In this study, we utilized a http://www.camda.info/.Publicly available datasets were analyzed in this study. This data can be found at: SD and SG conceived and designed the study. SA-S and AS contributed equally in the analysis of the data and writing the draft. All authors contributed to editing the final manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Transcriptome sequencing has been broadly available in clinical studies. However, it remains a challenge to utilize these data effectively for clinical applications due to the high dimension of the data and the highly correlated expression between individual genes.n = 2976), GSE81538 (n = 405), and GSE163882 (n = 222) datasets, we created AIOs for the subjects and designed CNN models to classify biomarker Ki67 and Nottingham histologic grade (NHG).We proposed a method to transform RNA sequencing data into artificial image objects (AIOs) and applied convolutional neural network (CNN) algorithms to classify these AIOs. With the AIO technique, we considered each gene as a pixel in an image and its expression level as pixel intensity. Using the GSE96058 (With fivefold cross-validation, we accomplished a classification accuracy and AUC of 0.821 ± 0.023 and 0.891 ± 0.021 for Ki67 status. For NHG, the weighted average of categorical accuracy was 0.820 ± 0.012, and the weighted average of AUC was 0.931 ± 0.006. With GSE96058 as training data and GSE81538 as testing data, the accuracy and AUC for Ki67 were 0.826 ± 0.037 and 0.883 ± 0.016, and that for NHG were 0.764 ± 0.052 and 0.882 ± 0.012, respectively. These results were 10% better than the results reported in the original studies. For Ki67, the calls generated from our models had a better power for prediction of survival as compared to the calls from trained pathologists in survival analyses.We demonstrated that RNA sequencing data could be transformed into AIOs and be used to classify Ki67 status and NHG with CNN algorithms. The AIO method could handle high-dimensional data with highly correlated variables, and there was no need for variable selection. With the AIO technique, a data-driven, consistent, and automation-ready model could be developed to classify biomarkers with RNA sequencing data and provide more efficient care for cancer patients.The online version contains supplementary material available at 10.1186/s13058-021-01474-z. Breast cancer is a complex disease; early detection and evaluation of the tumor are critical for prognosis and long-term survival. Once a tumor is detected, histopathologic analyses with estrogen receptor, progesterone receptor, human epidermal growth factor receptor2, and evaluation of Nottingham histologic grade (NHG) will be performed. More recently, assessment of the proliferation antigen Ki67 is increasingly recommended , 2. ThesIn recent years, technologies for transcriptome sequencing have become stable and matured, and their applications in clinics are steadily increasing. Some researchers use RNA sequencing to discover new biomarkers; others use it to evaluate existing biomarkers. Although the results vary, the assessment of many markers is comparable to that of histopathologic evaluation. As more and more RNA sequencing data are accumulated, data-driven and machine learning (ML)-based approaches have been explored to discover and classify biomarkers –7. Most The arise of modern computation power and ML algorithms provides an opportunity to address these issues. Convolutional neural network (CNN) is such an algorithm that has been used very successfully in computer vision and image classification , 10. Rechttps://www.ncbi.nlm.nih.gov/geo/), GSE81538 (n = 405), GSE96058 (n = 2976) [n = 222). The GEO datasets provided pathological assessments for the samples in the datasets and RNA sequencing procedures, which were described previously by the original authors [We obtained three RNA sequencing datasets from the NCBI GEO database ( = 2976)  and GSE1 authors . GSE8153− or 0; patients with more than 20% of cells stained with Ki67 antibody were assigned as Ki67+ or 1. Table In this paper, we used the clinical information for Ki67 and NHG to create outcome measures or labels for our model training and prediction. For the Ki67 label, we used the pathologists’ consensus percentage of tumor cells with Ki67 staining to create a binary label. Patients with 20% or less cells stained with Ki67 antibody were assigned as Ki67The AIO technique was based on the concept that considered each element in a dataset, such as a single nucleotide variation in genome wide association study, a gene/transcript in RNA sequencing data, or a CpG locus in methylation study, as a pixel in a digital image so that we could use a collection of elements to create an AIO. With these AIOs, we could apply advanced AI and ML algorithms to analyze and classify them. In this study, we applied this technique to transcriptome sequencing data. When the genes/transcripts were selected, we rescaled the expression levels to a range between 0 and 255 for each gene/transcript. For a given patient, the rescaled expression level would be the pixel intensity of the AIO. From the shared genes among the 3 datasets used in this study, we used the first 16,384 genes (sorted by chromosome number and gene transcription start position) to create an AIO for each of the patients in the datasets. The 16,384 genes could be configured into two different image objects. One was a 128 × 128 (high × wide) pixel grayscale AIO; the other was a 64 × 64 × 4 pixel (high × wide × channel) pseudo-color AIO. The processes to transform gene expression data into grayscale AIOs are shown in Fig. www.tensorflow.org/) [https://keras.io/api/) and the CNN architecture [In this study, we used the TensorFlow (ow.org/) , 21, Keritecture , 23 to cFor the Ki67 binary phenotype, we reported binary accuracy , precision , recall or sensitivity , F1 score  , and the area under the curve (AUC) of the receiver operating characteristic (ROC) for the training process as defined in the scikit-learn package . For thehttps://github.com/therneau/survival) and “survminer” (https://rpkgs.datanovia.com/survminer/index.html), and the results were plotted with R package “ggplot2” (https://ggplot2.tidyverse.org). We compared the predictive performance of Ki67 and NHG status from the pathologist’s calls with that of the calls predicted from our models. The p values reported were not corrected for multiple comparisons.Survival analyses were conducted with R packages “survival” (https://github.com/mdsamchen/AIO_scripts.git). Figure With the combined GSE96058 and GSE81538 dataset, we tested multiple CNN models to select model hyperparameters, such as the number of convolutional layers, kernel size, regularizer sizes, learning rate, optimizers, and number of fully connected layers. We found that a CNN architecture Fig.  with sixSimilar to the Ki67 analyses, we performed five-fold cross validation for NHG as well. Figure In these analyses, we used the portion of the GSE96058 dataset that had Ki67 and NHG status as training samples to classify those patients in the GSE81538 and GSE163882 datasets. The purpose was to evaluate the generalizability of the model using independent training and testing datasets. Based on the results from five-fold cross validation, we made slightly adjustments of the hyperparameters and did 5 or more runs on the GSE81538 and GSE163882 datasets. As shown in Table R = 0.99), but that between GSE96058 and GSE163882 was moderate (R = 0.79), Additional file The significantly worse performance of NHG on the GSE163882 dataset surprised us. Therefore, we examined the dataset more carefully. We looked at the correlation of gene expressions, a technical measure of data replicability, among the three datasets used in this study, and we found that the correlation between GSE96058 and GSE81538 was good . Since genes were arranged on chromosomes linearly, we could treat gene expression as an one-dimensional data such that we could use an one-dimension convolution CNN architecture 1D-CNN) to classify the biomarkers. The results were summarized in Additional file D-CNN to p value < 0.0001 vs p value = 0.014). We conducted similar analyses for the NHG. Since the number of subjects with missing NHG status was very small (n = 59) compared to the subjects with pathologist assigned NHG status (n = 2917), we only saw a trend  for the model produced calls  compared to that of GSE81538 (n = 405). Both GSE96058 and GSE81538 were produced by a Swedish team [R = 0.79). As a comparison, the correlation of gene expressions between GSE96058 and GSE81538 was very good (R = 0.99) , the model produced calls showed a trend. Should we have a larger sample size, the model produced calls would produce significant results. These results demonstrated that once implemented, these models would improve the productivity and consistency in clinical applications.We evaluated the predictive power of the calls produced from our CNN models by comparing it to that of the consensus calls from trained pathologists. With similar sample size (see Table In the literature, there was a report that proposed a different method, DeepInsght, to transform non-image data into images for CNN classification . Both ouIn this article, we reported the application of a technique to transform genomic data into AIOs and adopted CNN algorithms for their classification. Applying the technique to Ki67 and NHG, biomarkers that had substantial inconsistent assessments among trained pathologists, we demonstrated that our CNN models achieved classification accuracies better than similar models reported in the literature. Furthermore, with survival analyses, we showed that the calls generated by the models had a better predictive power than the consensus calls made by trained pathologists. These results illustrated the utility of the AIO technique in biomarker classification. With the demonstration of the principle, the AIO technique could have broad applications in clinical and genomics studies, facilitating more effective care of cancer patients.Additional file 1. Supplementary information.


Zea mays L.) samples from seven countries using a high resolution non-targeted metabolomics approach. Multivariate data analysis and univariate statistics were used to identify promising marker features related to geographical origin. Classification using only 20 selected markers with the Random Forest algorithm led to 90.5% correctly classified samples with 100 times repeated 10-fold cross-validation. The selected markers were assigned to the class of triglycerides, diglycerides and phospholipids. The marker set was further evaluated for its ability to separate between one sample class and the rest of the dataset, yielding accuracies above 89%. This demonstrates the high potential of the non-polar metabolome to authenticate the geographic origin of grain maize samples. Furthermore, this suggests that focusing on only a few lipids with high potential for grain maize authentication could be a promising approach for later transfer of the method to routine analysis.Safety along the food and feed supply chain is an emerging topic and closely linked to the ability to analytical trace the geographical origin of food or feed. In this study, ultra-performance liquid chromatography coupled with electrospray ionization quadrupole-time-of-flight mass spectrometry was used to trace back the geographical origin of 151 grain maize ( The traceability of goods, especially food and feed, is an emerging topic, as supply chains are more and more complex in a globalized market. Today the verification is mainly done by checking shipping documents, which may be falsified. Methods such as the blockchain strategy, which in principle make it possible to trace the entire supply chain back to its origin, are being developed, but are not immune to manipulation from the real world (from the field) into the digital world (the blockchain). In short, if the grower provides false information, this information is passed on without further verification. Blockchain technology simply ensures that these data, whether correct or incorrect, can no longer be manipulated within the chain. The only way to verify this is by using experimental analytical methods . TherefoTraditional analytical methods of cereal-based food or feed focus on specific quality parameters such as moisture, protein, oil and starch content ,4 but alDue to the huge amount of data, data analysis in non-targeted metabolomics studies is often performed using multivariate data analysis, which reduces the high dimensional data and facilitates the observation of differences ,8,9,18. Zea mays, L.) is an important cereal that is widely used as feed material, especially for poultry, but also as food for humans and in the renewable energy and bioethanol production +, [M + K]+, [M + NH4]+, [M − H2O + H]+, [M − CO2 + H]+, [M − NH3 + H]+.Mass calibration and feature detection for each analysis file were done in DataAnalysis 4.1 software . Parameters passed to the function “Find Molecular Features” were: signal-to-noise threshold of 5, correlation coefficient of 0.7, minimum compound length of 8 and a smoothing width of 2. Adducts considered were [M + H]m/z-value and retention time in “buckets”. Parameters for collecting consensus features were: ΔRT = 20 s and Δm/z = 0.02 Da. A time alignment was conducted and features, which appear in the bucket table, were set to be detectable in 50% of all sample analyses. Missing values of features were replaced by the lowest detected level in all samples. Consensus peak heights were then exported as csv-files and reformatted in Microsoft Excel 2010 for subsequent multivariate data analysis in R version 3.6.3 +-peak for diglycerides (DG) resp. triglycerides (TG). At this stage of the analysis, mainly the [M + H]+ resp. [M + NH4]+ adduct peaks were used for integration, since sodium or potassium ions were only present in small, rather non-reproducible amounts due to contamination + of DGs or TGs [4]+), but also as sodium adducts ([M + Na]+), following literature + adduct of TG (56:2)) or m/z 868.7 ([M + NH4]+ adduct of TG (52:6)) and phospholipids  or m/z 716.5 ([M + H]+ adduct of PE (34:2)), but also DGs ).In addition, a CVA was conducted to analyze the sample set especially for differences between the different sample classes . The CVA scores plot is shown in p-value of ≤0.01);Show significance in one-way ANOVA-test regarding the geographical origin  that were prominent in the grain maize samples see  was selep-value ≤ 0.01 in a one-way ANOVA-test with respect to the geographical origin (significance level 0.01) and a RSD in QC samples ranging from 2.6–9.3% (median: 5.4%). The maximum number of country pairs that each marker substance could be distinguished  was 16 (achieved by four markers) and the median was estimated to 12.5. From the country perspective, the French, Peruvian, and Hungarian samples had relatively many metabolites with significantly different metabolite values to distinguish them from the other countries; in contrast, the Slovakian samples were the most difficult to distinguish from the other countries. These results were also reflected in the RF classification results  between French and Hungarian samples: all PE markers, PC (36:5), PC (36:3), DG (34:2), TG (58:3), TG (58:2) and TG (58:1). In relation, high levels of marker substances were observed for: Phospholipids in samples from Spain and Hungary, DGs in Spanish samples and TGs in Peruvian samples. The high concentrations of TGs in samples from Peru could be related to the particular growing region, located on the one hand nearby the equator and on the other hand in a coastal region with proximity to the Andes.It was notable that peak areas differed between growth origins, but no systematic differences of all considered markers were observed in one sample origin. The detected values of some classes of marker in certain countries were striking: the low levels of all phospholipid markers in Ukrainian samples and the low levels of most DG marker substances in French and Hungarian samples. These samples were not used as feed but as raw material for starch production and were homogenized in the frozen state with a slightly different protocol using a knife mill and dry ice. With the exception of DGs, the peak areas for all other metabolites analyzed were in a comparable range to the other samples used as feed material. Since milling with the addition of dry ice is also known as to be gentler on the metabolite-state , the lowReasons for the different regulation of lipid classes could be that TGs are storage lipids, while DGs are playing an intermediate role in the lipid biosynthesis pathway . In contThere are some approaches in literature to show the relationship between genetics and oil content of maize plants . Laurie General variabilities in the fatty acid profile measured by gas chromatography in maize and wheat DDGS were also used to differentiate geographic origin by Tres et al. . RegardiPhospholipid and acylglyceride biosynthesis are closely linked. Differences in lipid biosynthesis may be caused by exogenous factors as well as by a different genetic background of plants . This mam/z-values with values deposited in metabolite databases . Further neutral loss of one fatty acid chain leads to a fragment ion [RCOOCH2COCH3 + H]+ [4 − (RCOOH + NH3)]+ as well as the appearance of [RCO]+ ions + . The preO]+ ions . IdentifFourteen of the marker substances belong to the class of acylglycerides, nine were TGs and five were DGs. Six substances belong to the substance class of phospholipids (three each were PCs resp. PEs). Most of the identified lipids consist only of even-numbered fatty acid chains. Since acyl coenzyme A is subsequently formed by repeatedly adding acetyl coenzyme A to the growing acyl chain, the fatty acid residues tend to have an even number of carbon atoms . HoweverThe PCA scores plot  shows thp-value ≤ 0.01, for further statistical results, see MANOVA analysis showed that the differences between the geographical origins were significant , a comparably accurate result was obtained, although the number of features used was greatly reduced by a factor of about 50 (only 20 out of 1022).Looking at the classification performance at the country-level see , the claSpecifically for the samples from the USA, a greater diversity could also be observed in the sample point cloud in multivariate data analysis (as well as by RF results). This greater variance could be explained by the fact that the samples from the USA were distributed over six states in the southern of the U.S., covering an east-west distance (Texas-Georgia) of approx. 1400 km and a north-south distance of approx. 900 km. Compared to samples of other geographical origins such as Spanish, Slovakian but also Peruvian samples, the U.S. samples were geographically more widely distributed. Comparing the classification accuracies with those obtained in the literature, comparable results were achieved: Tres et al. who authenticated maize DDGS samples from two regions in China and the USA by means of the fatty acid profile, achieved comparable accuracies per sample class by partial least squares discriminant analysis classification models .In addition, it must be clarified that the occurrence of different concentrations of marker substances does not consider national borders, but is based on many different influencing factors, e.g., soil composition, climate conditions: solar radiation, water ability, but also usage of fertilizers and pesticides etc. . These iThe ability of the selected marker set was further analyzed by calculating binary one vs. rest RF classification models for each geographical origin included in the sample set. These models had to classify the samples to be either from one specific country or from the other countries . Thereby, classification accuracies of above 89% could be achieved see .The separation of the Hungarian samples from all other samples in the dataset did perform well (above 99% accuracy), while the separation of samples from the USA did perform worse with only 89% accuracy. The classification accuracy was especially lower for samples from Slovakia and the USA. This was also already observed in case of the CC on country level see , where sWith regard to separability using specific marker substances, by looking at the variable importance in RF classification, it was found that different markers were important, depending on which sample class was to be distinguished see . Each maWith respect to lipid class, some distinct features were observed in the ranking of variable importance in the classification models: Marker substances belonging to the lipid class of phospholipids were particularly important in the classification models of samples from the Ukraine or the USA. DGs were potent marker substances for differentiating samples from Hungary from all other sample classes. In case of Spanish or French samples DGs were also mainly powerful for separation between samples from other countries. TGs were especially able to separate samples from Peru. Samples from Slovakia were most powerfully separated by a combination of phospholipids and TGs but also DGs, from all other samples in the dataset. These results indicate that especially the combined usage of marker substances from four lipid classes present in grain maize samples was an advantage to obtain high classification accuracy for all included sample classes.The ability of analytical geographical origin verification based on the non-polar metabolome of grain maize samples by classification was investigated. For this purpose, 151 grain maize samples originating from seven countries worldwide were analyzed using UHPLC coupled to a high resolution qToF-analyzer. Lookup in metabolomics-related databases revealed that the non-polar metabolome of grain maize consisted mainly of acylglycerides and phospholipids. By multivariate data exploration, differences between the origin-specific sample groups could be observed and 20 high-potential marker substances were discovered, which were at the same time significant, reliable and high abundant. Those marker substances were further verified by LC-MS/MS and belonged to lipid species of PCs, PEs, DGs and TGs. Classification of all samples using peak areas of the identified 20 marker substances was found to provide a classification ability of 90.5% by RF algorithm. Classification of 20 different grain maize varieties grown at the same location did not exhibit lower classification accuracies than samples from other geographical origins. Additionally, the binary differentiation of samples from every country in the dataset from all other samples could be conducted by RF classification with high accuracies (89–99%), strengthening the high potential of the identified lipid marker substances for later usage in an authentication method.However, as the differences in metabolite levels might be affected by factors which may vary from one year to another, it further has to be taken into account that the long-time stability of the metabolite profile of grain maize should be accessed in future studies. Also, it might be useful to extend the classification model in the future with samples originating from China. China was as of 2019/20 the biggest producing country of maize  in the world despite most of its corn production is yet used in the country itself .


We apply BatchBench to eight methods, highlighting their methodological differences and assess their performance and computational requirements through a compendium of well-studied datasets. This systematic comparison guides users in the choice of batch correction tool, and the pipeline makes it easy to evaluate other datasets.As the cost of single-cell RNA-seq experiments has decreased, an increasing number of datasets are now available. Combining newly generated and publicly accessible datasets is challenging due to non-biological signals, commonly known as batch effects. Although there are several computational methods available that can remove batch effects, evaluating which method performs best is not straightforward. Here, we present BatchBench ( Single-cell RNA sequencing (scRNA-seq) technologies have made it possible to address biological questions that were not accessible using bulk RNA sequencing , e.g. idk-nearest neighbor graph constructed from the expression matrices and consequently its output is restricted to downstream analyses where only the cell label can be used.Fortunately, with appropriate experimental design it is possible to remove a portion of the batch effects computationally, and recently there has been a large degree of interest in developing such methods for scRNA-seq. We group the methods into three categories depending on what space they operate on with respect to the expression matrix Figure . The exphttps://github.com/cellgeni/batchbench under the MIT Licence.As the choice of batch correction method may impact the downstream analyses, the decision of which one to use can be consequential. To decide what method to use, most researchers rely on benchmarking studies. Traditionally such comparisons are carried out using a compendium of relevant datasets. The downside of this approach is that methods published after the benchmark was carried out are not included and that the comparison may not have featured datasets that contain all the relevant features required to evaluate the methods. To overcome these issues we have developed BatchBench Figure , a flexiBy default, BatchBench evaluates batch correction methods based on two different entropy metrics. The normalized Shannon entropy is used to quantify how well batches are aligned while preserving the separation of different cell populations. However, the entropy measures do not provide a complete picture of how the batch correction impacts downstream analyses. Therefore, BatchBench has a modular design to allow users to incorporate additional metrics, and we provide two examples of such metrics - unsupervised clustering and identification of marker genes. Five different unsupervised clustering methods are applied to the merged cells to afford the user a better understanding of how the different methods affect this step which is often central to the analysis. We also compare cell-type specific marker genes to understand how different batch correction methods affect the expression levels.GSE84133) (GSE85241) (E-MTAB-5061) (We consider three published pancreas datasets: Baron (SE84133) , Muraro SE85241)  and SegeAB-5061)  generateFor Figure Figshare site and merged by tissue, generating 37 organ datasets. From these, 18 datasets containing more than one batch and with a reasonable proportion of cells across batches were selected. Through further preprocessing we removed cells expressing <250 genes, genes expressed in <50 cells, cell types representing <1% of total cell population in a tissue, and batches containing <5% of the total number of cells in a tissue .The data was downloaded from the paper's i representing a cell . Each cell is connected to its k = 30 nearest neighbors as defined by the similarity of expression profiles calculated using the Euclidean distance. Using the graph, we calculate for each cell i the probability that a neighbor has cell type c, icP, as well as the probability that a neighbor comes from batch b, ibP. From these joint probabilities we can calculate cell type and batch entropies. We report the average value across all cells divided by the theoretical maximum to ensure a value in the interval . For the datasets considered in this study, the results are robust with respect to the choice of k .Uniform Manifold Approximation and Projection (UMAP) is computed through the ap-learn . For theThe filtered Tabula Muris dataset was sampled using uniform selection and no replacement to 1, 2, 5, 10, 20 and 50% of its cells. Resulting in objects of: 4168 genes and 608, 1217, 3041, 6083, 12 166 and 30 414 cells. The initial proportion of the batches  was maintained through the different subsets.f of cells sampled with uniform probability from the sequence ; (ii) a value d representing the dispersion of the effect to be simulated sampled with uniform probability from the sequence , where n is the number of batches to simulate. For each of the 10 cell types in the input data we add count values by drawing values from a normal distribution with a standard deviation d. The artificial batch effect is only applied to those genes expressed in >f of the cells. If a gene is assigned a negative value, then it is replaced by 0. The result is a simulated data set of 1001 cells and 4168 genes which is appended to the input data set. We followed this approach to simulate data sets with 2, 3, 5, 10, 20 and 50 equally sized batches.We work with a reduced version of the Tabula Muris atlas object. We first removed all the Smart-seq2 cells and then retained only the 10 largest cell types. From this 1001 cells are randomly sampled to serve as input to the artificial batch generation. All 4168 initial genes are considered. We base our simulation of batch effects on a normal distribution. For each batch to be simulated, we define: (i) a fraction We rank genes in descending order by their coefficient of variation establishing five fractions of features: 0.05, 0.1, 0.2, 0.5 and 1.0 . Feature selection is performed as a first step in each clustering algorithm script prior to any processing of the input data.k to the number of cell populations of each dataset. If the dataset had >5000 cells we enable sc3_run_svm to speed up the processing. RaceID uses Euclidean distances based on the Pearson correlation distance. All three RaceID clustering options  are implemented in our clustering step. The other clustering algorithms can be applied to all batch correction methods in our study. Louvain and Leiden methods were implemented with the Seurat function FindClusters, with other parameters set to their default values. We also implemented standard hierarchical clustering using Ward's agglomeration method with the hclust function from the stats package.The merged samples were clustered using five different clustering algorithms: SC3  from theAs a pre-processing step after feature selection and prior to SC3, RaceID and hierarchical clustering, all cells and features with zero variance are removed. Moreover, in the case of RaceID clustering, negative values in the expression matrix that may result from the batch effect removal step are set to to zero.arandi and the vi.dist functions from the mclust package respectively.To assess the similarity of each corrected output clustering annotation with the provided ground truth Adjusted Rand Index and Variation of Information are computed with the To compare the results across datasets we select the best similarity metric value across all feature selection fractions considered Figure . Since vFindMarkers function from the Seurat package which restricts the comparison to methods that output a normalised count matrix. For a gene to be considered as a marker, we require that the absolute value of the log fold-change >2, and that the gene is expressed in at least half of the cells in each population. We use the default Wilcoxon Rank Sum test to find genes that are significantly different  between the merged dataset, and in each of the individual batches.To obtain marker genes we use the To compare the overlap of the sets of marker genes identified across batches and the merged data we used the multiple site generalized Jaccard index . We resthttps://github.com/cellgeni/batchbench) requires a SingleCellExperiment (R based) or AnnData object (python based). The input object is then converted into its counterpart in the other language. This object must contain: log-normalized counts, and the batch and cell type annotation of their cells as Batch and cell_type1 respectively, in the object metadata. The workflow performs an initial QC step where cells, genes, batches or cell types can be filtered according to user-defined parameters. Cells not assigned to any batch or cell type are excluded in this step also. Each dataset is then sent in parallel as input to each of the batch effect correction tools, after which rds and h5ad objects containing the output are saved and made available for the user. Each of the batch corrected outputs serves as input for a series of downstream analyses: (i) UMAP coordinates are computed and saved as a csv file for visualization of the different batch corrections, (ii) entropy computation and saved as csv file, (iii) clustering analysis, (iv) marker gene analysis and any module optionally added by the user.As an input, BatchBench   and TabuNext, we investigated another mouse cell atlas, Tabula Muris  and our To determine how the number of cells in each sample influences batch correction performance and running times we considered the Tabula Muris dataset, and downsampled it to 1, 5, 10, 20 and 50% of the original 60 828 cells (Methods). Across all subsets, the input objects contain 64% of 10X cells and 36% of FACS-sorted Smart-Seq2 cells. Note that this batch correction task is more challenging than the one in Figure The number of cells has a strong impact on performance and it becomes more difficult to align the two batches with increasing cell numbers. All methods except Scanorama, Harmony and Seurat reduce the batch entropy by >50% as the number of cells increases from 608 to 60 828 Figure . UnfortuThe main goal of the investigation involving different numbers of cells is to learn how the computational resource requirements change as this is an important factor when choosing a method. Considering the time required to perform the integration, we found substantial differences as ComBat, Limma, Harmony and BBKNN have more or less constant run times as the number of cells grow. By contrast, mnnCorrect and fastMNN grow exponentially, with the former being the slowest method in our study. Seurat initially has a stable runtime before it starts to grow exponentially Figure . For allAs sequencing costs decrease, the number of different samples that can be processed will increase. Thus, we also evaluated how well each method handles an increasing number of batches. For this study we considered subsets of the Tabula Muris 10X dataset with 4168 genes and 18 347 cells. As the batches created by subsampling this dataset are entirely artificial, we added small batch-specific random counts to each gene to ensure that there are differences that require correction (Methods). In our simulations, cell types are well separated whereas the batches are more overlapping.We fixed the batch size to 1001 cells and we created datasets including 2, 3, 5, 10, 20 and 50 and batches, introducing small artificial batch effects. Cell type entropies are maintained low with the number of batches for all methods, highlighting the capacity of our batch simulating procedure to not mix distinct cell populations as batches are included. Regarding batch entropy Figure , BBKNN, A key advantage of the entropy measures is that they can easily be calculated for any dataset containing discrete cell state clusters and that they are easy to interpret. However, they only evaluate the mixing of the cells as represented by the nearest neighbor graph, and they do not directly assess how the batch correction will impact downstream analyses based on the corrected data. To understand how specific aspects of the analysis are affected, tailored benchmarks are required. BatchBench allows users to add customized modules to evaluate the aspect they find most relevant. Here, we consider two common types of analyses, unsupervised clustering and identification of marker genes.To evaluate the effect on unsupervised clustering, we apply four published methods, Leiden , LouvainOur analysis of the MCA suggested small differences in cell type entropy, but large differences in how well the batches were mixed Figure . By contFor the Tabula Muris we observe a similar pattern with large differences in ARI between tissues and relatively small differences across methods. Compared to MCA, we observe an improvement in similarity values for RaceID, SC3 and hierarchical clustering algorithms, whereas Leiden and Louvain algorithms show worse performance. Closer inspection reveals that the Leiden and Louvain methods perform poorly for datasets with a small number of clusters . SurprisFor the pancreas dataset, hierarchical clustering together with RaceID and SC3 algorithms tend to have a higher ARI. Inclusion of all features in the clustering tend to yield better similarity results . We alsoThe main objective of batch correction methods is to ensure that cells with similar expression profiles end up near each other. The most widely used metrics, e.g. mixing entropies or inverse Simpson index ,19,29, aWe have developed BatchBench, a customizable pipeline for comparing scRNA-seq batch correction methods. We have assessed the performance of eight popular batch correction methods based on entropy measurements across three datasets, suffering from donor and platform effects. Our results highlight Seurat as the top performer as it correctly merges batches while maintaining the separation of distinct cell populations. Harmony also shows very good results in pancreas and MCA but surprisingly fails in correcting the Tabula Muris batch effects. Scanorama and fastMNN can be considered consistent good performers. Regarding BBKNN, we note that the entropies are not suitable for evaluating its performance as the method operates by identifying nearest neighbours in each of the provided batches  and adjuA key insight from our study is that the entropy measures do not fully reflect how the choice of batch correction method will impact downstream computational analyses. We applied five different unsupervised clustering methods to the merged datasets, and the results are not as clear as for the entropy analyses. No single method emerges as the best performer, and in some cases the best results were obtained using the uncorrected data. This result highlights the importance of using benchmarks that are more closely linked to the analysis that will be carried out for the merged dataset.Our attempt to identify marker genes from the corrected dataset demonstrates the difficulty of using the merged expression matrix for downstream analyses. As none of the methods considered in our study performed adequately in this benchmark, we highlight this as an area where improvements are required. Since marker genes are not preserved, we stress the importance for users to monitor how expression levels change. Any analysis based on the expression levels, e.g. identification of marker genes or differentially expressed genes, will need to be verified to ensure that the result was not distorted due to the alterations introduced by the batch correction method. An important limitation of our marker gene analysis is that it only quantifies consistency as there is not yet an established ground truth for what marker genes are represented for the cell types in our study. We tried to use marker gene lists from the literature as represented by the CellMarker database , but we et al. (Benchmark studies are important as they help guide researchers in their choice of methods. They are also helpful for developers as they can highlight limitations of existing methods and provide guidance as to where improvements are needed. One shortcoming of traditional benchmarks, however, is that they are static in nature and that they only consider the datasets that the authors of the benchmark study had chosen to include. A related issue is that the metrics used to evaluate methods may not be relevant to all datasets and research questions. Along with a similar study by Leucken et al. , BatchBegkab004_Supplemental_FileClick here for additional data file.


High-throughput omics data are becoming more and more popular in various areas of science. Given that many publicly available datasets address the same questions, researchers have applied meta-analysis to synthesize multiple datasets to achieve more reliable results for model estimation and prediction. Due to the high dimensionality of omics data, it is also desirable to incorporate variable selection into meta-analysis. Existing meta-analyzing variable selection methods are often sensitive to the presence of outliers, and may lead to missed detections of relevant covariates, especially for lasso-type penalties. In this paper, we develop a robust variable selection algorithm for meta-analyzing high-dimensional datasets based on logistic regression. We first search an outlier-free subset from each dataset by borrowing information across the datasets with repeatedly use of the least trimmed squared estimates for the logistic model and together with a hierarchical bi-level variable selection technique. We then refine a reweighting step to further improve the efficiency after obtaining a reliable non-outlier subset. Simulation studies and real data analysis show that our new method can provide more reliable results than the existing meta-analysis methods in the presence of outliers. With the advances in biological sciences, omics data have been playing an important role in many different fields of research. A typical example of such data includes gene expression data targeting for the identification of important genes that are related to disease status or clinical outcomes  in (2.2) treats βj1, …, βMj as a group for each covariate j, and conducts a group-type variable selection. In addition, the L1 norm used inside the square root penalty can perform a study-specific variable selection that shrinks the small coefficients to zero and keeps only the large coefficients  andFor the minimization problems of (2.2) and (2.4) with tuning parametera) ifis a solution of (2.4), thenwithis a solution of (2.2); and (b) ifis a solution of (2.2), then there exists a solutionof (2.4) such thatkj, the parameter αj controls the sparsity of the jth group βj1, …, βMj, and γkj reflects the sparsity within the jth group. If αj is shrunk to zero, all βkj, k = 1, …, M in the jth group will be zero. Since the objective function (2.4) only has two lasso penalties without interaction, Zhou and Zhu  recursively, we can obtain the approximate clean subset for the kth study in the tth iteration, denoted as Hk,t. Consequently, the approximate clean subset for all studies in the tth iteration is given as Accordingly, let Driessen  and AlfoDriessen ; that isTHEOREM 2. For any given initial setby recursively applying (2.5), we haveQ1 ≥ Q2 ≥ ⋯ ≥ QtM with C-step). Note that the selected subset after convergence of the C-step is related to the initial subset; to alleviate this problem, we perform this searching procedure with several different initial subsets as in Alfons et al.  = ϕ(x), ψ = ϕ(−x), and ϕ(x) is given in Definition A1 of the x, y) provides a robust loss measure for binary classification, which assigns a nearly zero score to the points with correct classification and a high score to the points with misclassification. Hence, the best clean subset for the kth study indicates the lowest classification loss among all those identified clean subsets for this study. Finally, the best clean set for the M studies is given as where ψ, denoted as Let kth study asSearch the approximate clean subset for the M studies is where Repeat Step 2 on s = 1, …, 500 and k = 1, …, M, letFor k, 1, …, πk, 10} ⊂ {1, …, 500} with ek,πk, 1 ≤ … ≤ ek,πk, 10. The best 10 sets among s = 1, …, 10.Search a subset of indexes such that {πs = 1, …, 10, respectively, and repeat Steps 2–3 for a total of t times until convergence such that 2 is the Euclidean norm and ϵ is a pre-specified small constant. The converged approximate clean subset and the coefficient estimate for all M studies are denoted as Let s = 1, …, 10 and k = 1, …, M, letFor M studies is given as β is The best clean subset for all C-steps, the algorithm for minimizing (2.4) may not stable. For this, we may restrict that α1 = ⋯ = αp = 1.Finally, we observe that in the first several rki is a standardized statistic and is approximately normally distributed, the weights for the observations are given asNote that the LTS-type estimator only uses a subset of data and may suffer from a low efficiency. To further improve the model estimation, Kurnaz et al.  proposedwhere Φ is the cumulative distribution function of the standard normal distribution. Throughout this paper, we select δ = 0.0125 as in Alfons et al.  and KurnConsequently, the robust estimator for meta-analyzing multiple studies is given askth study. Obviously, when the data do not have outliers or only have a small proportion of outliers, the reweighing procedure uses more observations and hence may improve the estimation accuracy.where Finally, to give more insights on the algorithms in sections 2.2.1 and 2.2.2, we present a flow chart of the two-step procedure for meta-analyzing multiple studies in h1, …, hM before searching the approximate clean subset hk ≈ nk/2, and vice versa. In practice, if we do not have prior knowledge for the number of outliers, we recommend to use hk ≈ 0.75nk as adopted in Rousseeuw and Driessen  and (2.7) can be rewritten as (2.4), and hence we only need to select the tuning parameter in (2.4). Various data-driven techniques have been well developed in the literature including, for example, the cross-validation, the generalized cross-validation, and the Bayesian information criterion (BIC) procedures. We adopt the BIC to select the tuning parameter as recommended in Alfons et al. . Specifiwhere In this section, we conduct simulations to evaluate the numerical performance of our new robust lasso-type meta-analysis method (RL-meta) and compare it with some existing methods. Specifically, we consider the state-of-the-art methods from Li et al. , Alfons F1 score (F1), and the root mean squared error (RMSE), whereLet TP, FP, and FN be the number of true positives, false positives, and false negatives, respectively. Then to evaluate the performance of these methods, we consider four criteria including Precision = TP/(TP+FP), Recall = TP/(TP+FN), the F1 all range from 0 to 1, which measure the accuracy of variable selection with a larger value being preferred. As an addition, RMSE measures the loss of coefficient estimation, for which a small value is favorable.Note that Precision, Recall, and M studies and each has n observations. The covariate vector N for k = 1, …, M and i = 1, …, n, where Ip is the identity matrix. Then the response variables are generated as yki = 1 if xkiβk + εki > 0, otherwise, yki = 0, where ki is the independent noise sampled from N. To access the performance of our RL-meta under different levels of heterogeneity, we let βkj = zkjbkj for k = 1, …, M and j = 1, …, 10 and βkj = 0 for j = 11, …, p, where zkj is randomly sampled from a Bernoulli distribution with the success rate π0 and bkj is randomly sampled from N. This means that only the first 10 covariates may be possibly related to the response variable in each study, it is informative with probability π0 and uninformative with probability 1 − π0. We consider π0 = 0.2, 0.5, or 0.9 to represent three levels of heterogeneity between the studies. In addition, we also consider  or  as a low or large dimension, respectively, and the numbers of studies as M = 2 or 8.In the first simulation, we consider clean data with no outliers. Specifically, we generate M = 2 studies (see p = 50). Figure A1 shows that RL-each and L-each can provide an accurate variable selection, whereas the estimates for nonzero coefficients tend to be smaller than the true coefficients, especially when the dimension is larger than the sample size. This phenomenon was also observed in Zhao and Yu  for dies see , 3. To sdies see , S2. Wheo and Yu , where tF1 in most settings. RL-meta is nearly as good as L-meta and is much better than L-each and RL-each. For example, when the dimension and the number of informative covariates tend to large, L-each and RL-each both exhibit an inflated RMSE, whereas RL-meta and L-meat still keep a low RMSE. This verifies that borrowing information across the studies can improve the estimation accuracy, especially when the dimension is large and the sample size is small.m0 observations from each study and add outliers to those informative covariates. More specifically, for the observations with yki = 0 (or yki = 1), the informative covariates are replaced by values randomly sampled from N. To avoid high-leverage points, those observations are assigned an opposite class label. That is, yki = 1 − δ(xkiβk > 0), where δ(·) is an indicator function. Throughout this section, we fix m0 = 10, and the other parameter are the same as those in section 3.1.In the second simulation, we consider contamination data with outliers. We randomly select M = 2 studies under contamination data. To save space, we also move the plots of L-each and RL-each to M = 2 and π = 0.2), we note that RL-each has a smaller RMSE than RL-meta, which exhibits a good coefficient estimation. One possible reason is that when the number of studies and informative covariates is very small, there no or little information can be borrowed to improve the estimation accuracy. As the number of studies and/or the number of informative variable tend to large, our RL-meta consistently has the best performance among the three methods including L-meta, RL-each, and L-each. This again verifies that borrowing information across similar studies can significantly improve parameter estimation and the accuracy of variable selection . The first data are the gene expression signature of cigarette smoking (GSE10072), which contains the gene expression levels of 49 normal and 58 tumor tissues from 28 current smokers, 26 former smokers, and 20 never smokers, and each sample has 22,283 genes. The second data are the early stage non-small-cell lung cancer (GSE19188), which contains the gene expression levels of 65 adjacent normal and 91 tumor tissues, and each sample has 54,675 genes. The third data are the non-smoking female lung cancer in Taiwan (GSE19804), which contains the gene expression levels of 60 normal and 60 tumor tissues, and each sample has 54,675 genes. Consequently, there are 13,515 common genes shared between these three datasets. We map the probes of the raw data to gene names by annotation packages in Bioconductor. Also as per Hui et al. . We select the 1,000 genes with the largest values of Tj for j = 1, …, 13, 515, and then perform the variable selection for the three datasets based on RL-meta, L-meta, RL-each, and L-each, respectively.In this section, we consider three publicly available lung cancer datasets from GEO (i et al.  and Alfoi et al. , if multTo further compare the performance of the four methods, we also consider to create outliers for the three datasets. Specifically, we select the first eight samples from each of the three datasets, and then add a number 10 to the expression levels of those informative genes. In order to generate outliers instead of leverage points, we assign the labels of those samples to their opposite class. C-steps with different starting subsets, and hence the number of iterations is considerably higher than the classical lasso-based methods.In this paper, we propose a robust method for meta-analyzing multiple studies with high-dimensional data. Our method is based on a two-step procedure including a search step for a clean subset from each study and a reweighting scheme to improve the estimation efficiency. In particular, we incorporate the bi-level variable selection technique into both of the two steps. Our new robust method has the capacity to capture the sparsity at both the study and group levels so as to better integrating the structural information that can enhance the parameter estimation and variable selection. Simulation studies demonstrate that, in the presence of outliers, our proposed method can provide better parameter estimation and also identify informative covariates more accurately than the existing strategies, especially when the dimension is large. We also provide a comparison of computational cost for RL-meta, RL-each, L-meta, and L-each in Table A1. We note that RL-meta and RL-each suffer from a heavy computational burden. The main reason is that the two robust methods need to perform In addition, the lasso-based variable selection methods usually suffer from a low power when some covariates are highly correlated. As an alternative, Zou and Hastie  and TibsWe leave this problem for further theoretical and numerical studies.Finally, we note that Bayesian meta-analysis is also a popular approach for the integration of multiple studies. Recently, Cai et al.  proposedhttps://www.ncbi.nlm.nih.gov/gds/.Publicly available datasets were analyzed in this study. This data can be downloaded from the link: ZH developed the study and performed the simulation studies and the real data analysis. YZ and TT initiated the study and provided helpful discussions. All authors contributed to the article and approved the final version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


A major challenge in analyzing cancer patient transcriptomes is that the tumors are inherently heterogeneous and evolving. We analyzed 214 bulk RNA samples of a longitudinal, prospective ovarian cancer cohort and found that the sample composition changes systematically due to chemotherapy and between the anatomical sites, preventing direct comparison of treatment-naive and treated samples.in situ hybridization experiments.To overcome this, we developed PRISM, a latent statistical framework to simultaneously extract the sample composition and cell-type-specific whole-transcriptome profiles adapted to each individual sample. Our results indicate that the PRISM-derived composition-free transcriptomic profiles and signatures derived from them predict the patient response better than the composite raw bulk data. We validated our findings in independent ovarian cancer and melanoma cohorts, and verified that PRISM accurately estimates the composition and cell-type-specific expression through whole-genome sequencing and RNA https://bitbucket.org/anthakki/prism.Bioinformatics online. Precision oncology aims to identify targetable alterations based on molecular profiling of tumors . As cancAlleviating the sample composition issue by discarding low tumor content samples , 2015 caTo counter this, we present PRISM , which is a statistical latent variable framework for RNA-seq data. Compared with the existing methods, PRISM is unique in that it estimates both the composition and the constituent expression profiles simultaneously in individual bulk samples, a combination which was previously unmet. This is achieved by exploiting a single-cell reference, which is subject to the statistical model rather than being treated as ground truth, which allows PRISM to form adaptive profiles even for unmatched data. These estimates provide personalized expression profiles that are unbiased to changes in the sample composition, enabling tracking the tumor progression in individual patients.in situ hybridization (RNA-ish) experiments, we confirmed the improved survival prediction in other cohorts and cancer types by using The Cancer Genome Atlas (TCGA) data.We applied PRISM on 214 bulk RNA-seq samples that were longitudinally collected from homogeneously treated high-grade serous ovarian cancer (HGSOC) patients. HGSOC is the most common subtype of epithelial ovarian cancer (EOC) with only 43% five-year survival rate . It is oThe patient cohort consists of patients treated for ovarian or primary peritoneal HGSOC at Turku University Hospital between September 2010 and October 2018. All patients participating in the study gave written informed consent. The study and the use of all clinical material have been approved by The Ethics Committee of the Hospital District of Southwest Finland (ETMK) under decision number EMTK: 145/1801/2015.http://www.project-hercules.eu/).We acquired 214 bulk RNA sequencing samples from 61 of the patients. Of these, 120 are primary (before chemotherapy), 60 interval (after chemotherapy) and 20 relapsed tumors (after being diagnosed as recurring). The samples are from primary ovarian tumors and various sites of intra-abdominal solid metastases and ascites fluid, as detailed in the analysis. Immediately after surgery, the HGSOC tumor specimens from our cohort were incubated overnight in a mixture of collagenase and hyaluronidase  to obtain single cell suspensions. Specimens were processed with a modified Fluidigm C1 protocol  or the sspan sufficient expression state-space for PRISM to be able to adapt to unmatched profiles in the bulk samples.We acquired 6312 single cell profiles from 8 samples (from 7 patients and from various tissues) using the Chromium platform, and 347 cells from 8 samples (8 patients) using the Fluidigm single-cell sequencing platform. The latter were used for comparison purposes only. The single-cell samples were all matched to the bulk RNA samples but most bulk RNA samples remain unmatched. The single-cell reference need not to be matched to the bulk samples, but it needs to Bulk RNA sequencing reads were preprocessed using the SePIA  pipelineWFDC2, PAX8, MUC16, EPCAM, KRT18; stromal cell markers: COL1A2, FGFR1, DCN; immune cell markers: CD14, CD79A, FCER1G, PTPRC, NKG7, CD3D, CD8A.For the single-cell sequencing data, the raw base call (BCL) files were processed, including demultiplexing, alignment, barcode assignment and UMI quantification, with CellRanger (version 2.1.1) pipelines. The reference index was built upon the GRCh38.d1.vd1 reference genome with GENCODE v25 annotation. Single-cell transcriptomes were clustered using a shared nearest neighbor (SNN) modularity optimization based clustering algorithm implemented in Seurat (version 2.3.4) . PCA wasm genes, k cell types, and n samples, and λ. This approach allows capturing both biological and technical noise and accommodates either overdispersion (as commonly observed) and underdispersion (which improves stability under systematic errors) with respect to Poisson noise. The posterior of the observed We assume that latent cell-type-specific RNA counts ithm see . Unlike ithm see , we are r, with different composition but equal expression profiles. For each bulk sample, we assume that a cell type (and bulk specific) expression profiles i runs over the genes, l over the cell types, j over the single-cell profiles and r over the bulk replicates . As ilT can vary, the decomposition will weigh in the genes that are informative in discriminating the cell types. The cell-type-specific contributions The model can be exploited for decomposing bulk data by considering a joint model on the bulk α of unperturbed genes, and finding an unperturbed common subprofile jG for each single-cell sample (Supplementary Algorithm S2). Here, In mixtures, the scale factors are naturally estimated as part of the deconvolution process. Meanwhile, in pure (single-component) samples, the scale factors can be estimated by considering a fraction In the decomposition, the composition Formalin-fixed paraffin embedded (FFPE) tissue sections were analyzed using the RNAscope Multiplex Fluorescent Reagent Kit version 2 . We used catalog probes  for the For fluorescence quantification, we used CellProfiler (version 3.1.8)  for segmhttps://bitbucket.org/anthakki/prism/.PRISM employs a latent statistical model for composite (a mixture of multiple phenotypes) RNA-seq data, which accounts biological heterogeneity, compositional heterogeneity and sampling noise. The estimated model can be exploited for decomposing bulk RNA-seq data, finding sample specific scale factors or clustering RNA-seq data. An overview of PRISM is shown in We first studied how the composition of HGSOC bulk samples varies over the treatment phase, the anatomical location and the treatment response cf. . Figure  samples . This is therapy . The resP-value Specifically, the fraction of cancer cells and fibroblasts vary significantly between the treatment-naive and interval samples, even when accounting for anatomical sites  scores  for the P-value et al.  data. Supplementary Figure S6a shows the correlation with ASCAT  purity en cancer  bulk RNAn cancer  bulk RNAn cancer  and compn cancer , 2015, An cancer  , C1R, COL1A2, NAALADL2 (fibroblast), RNASE6, GPR34 and C3AR1 (immune). The genes were selected to have high expression in the specific cell type , for TRIM29 and PARD6B the trend was opposite, and for NAALADL2 was inconclusive. The trend of TRIM29 and PARD6B opposite to the general trend is due to the seven patients being a counterexample with respect to the general population: also the PRISM-derived expression the opposite pattern  profiles, the data might lack power for the comparison of specific infrequent subtypes of stromal and immune cells separately. These points may warrant further investigation, but we expect that the issues are mitigated in the future as single-cell cataloging efforts move forward.Precision oncology approach calls for methods that can exploit general statistical patterns in a cohort of a heterogeneous disease, but operate reliably at the individual patient level regardless of the evolving disease state, and adapt to the specifics of that patient, to which PRISM is a response regarding whole-transcriptome analysis of bulk samples. We believe PRISM has direct applications in analyzing transcriptomic data from other diseases that stem from heterogeneous causes and sampling setting, such as other cancer types, and that analysis methods for other genomic domains can benefit from the insights of our approach.btab178_Supplementary_DataClick here for additional data file.