Normalization of RNA-seq data aims at identifying biological expression differentiation between samples by removing the effects of unwanted confounding factors. Explicitly or implicitly, the justification of normalization requires a set of housekeeping genes. However, the existence of housekeeping genes common for a very large collection of samples, especially under a wide range of conditions, is questionable.github.com/hippo-yf/MUREN and on the conda platform: anaconda.org/hippo-yf/r-muren.We propose to carry out pairwise normalization with respect to multiple references, selected from representative samples. Then the pairwise intermediates are integrated based on a linear model that adjusts the reference effects. Motivated by the notion of housekeeping genes and their statistical counterparts, we adopt the robust least trimmed squares regression in pairwise normalization. The proposed method (MUREN) is compared with other existing tools on some standard data sets. The goodness of normalization emphasizes on preserving possible asymmetric differentiation, whose biological significance is exemplified by a single cell data of cell cycle. MUREN is implemented as an R package. The code under license GPL-3 is available on the github platform: MUREN performs the RNA-seq normalization using a two-step statistical regression induced from a general principle. We propose that the densities of pairwise differentiations are used to evaluate the goodness of normalization. MUREN adjusts the mode of differentiation toward zero while preserving the skewness due to biological asymmetric differentiation. Moreover, by robustly integrating pre-normalized counts with respect to multiple references, MUREN is immune to individual outlier samples.The online version contains supplementary material available at 10.1186/s12859-021-04288-0. The RNA sequencing (RNA-seq) technology has been the primary mean to explore the transcriptome in the past decade. Like the microarray technique, it can profile mRNA and non-coding RNA  transcriAccuracy is key to the transcript quantification. Despite that RNA-seq avoids the biases due to dye effects and hybridization in the microarray technology , 7, otheOne of the most widely used normalization methods is Reads per Kilobase per Million mapped reads (RPKM),  and its M-values (TMM), [The assumption of the constant total RNA contents or transcripts is unrealistic in some situations . Some scs (TMM), ), used is (TMM), , 15, estOther methods have their own assumptions. Quantile method , widely Biologists prefer housekeeping genes  in normaThe invariant gene set is a statistical counterpart to the housekeeping gene set . In the The idea of normalizing pairwise samples with respect to multi-references followed by integrating them via removing the reference effects was initially proposed in the microarray setting . We founCrucial for normalization is the evaluation of its goodness. We claim that the goodness includes not only the reduction of bias and variation, but also the preservation of skewness of expression differentiation. The claim is supported by our biological interpretation and statistical analysis of expression skewness, which is exemplified by a single cell data of cell cycle.We propose a two-step normalization procedure for RNA-seq data: pairwise normalization and integration. The introduction of the reference factor allows us to carry out robust normalization with respect to multiple references. The method emphasizes on robustness by adopting least trimmed squares (LTS) and least absolute deviations (LAD) in the two steps respectively. The general scheme of the proposed normalization method is shown in Fig. Suppose we have two RNA sequencing samples: one reference and one target. Denote the read counts of each transcript indexed by Consider a system with Our goal is to reconstruct the input The blind inversion scheme  leads usWithout loss of generality, we further assume that.is a monotone (increasing) function.Then Assumption R1 becomesThe next minimization result is the mathematical basis for the regression-based normalization.Suppose Assumption R2 is valid for some functionThen it is the minimizer ofwhich equalsThis proposition motivates us to estimate g by minimizing the sum of squaresFinally, we consider the more practical situations. Suppose a portion g by minimizingSince We parameterize The normalized abundance/count of Now we consider the parameter estimation of the regression model . Given aThe LTS estimate is regression, scale, and affine equivariant. The breakdown point of entclass1pt{minimaurbances .At the value of the LTS estimate The trimmed averageThat is, the average of the log ratios of the undifferentiated transcript set between a pair of samples is zero after normalization.The above describe Part C of the MUREN workflow shown in Fig. Suppose the RNA-seq samples are indexed by There are some ways to select references. Biologically, we can select one or several samples under each experimental condition as references and align every target sample to the reference set. In this strategy, the experiment design of biology provides certain prior knowledge. Statistically, we can get hints from some exploratory data analysis. For examples, the hierarchical clustering arranges samples by some measure of distance/dissimilarity. Heuristically, we can select the samples on different branches as references. Last, it is straightforward to select all samples as references if sample size is relatively small, and select a random subset of samples as references if the sample size is large. In the examples shown in this report, slight differences were observed across different sets of references.Next, we describe the model in Part D of the MUREN workflow as shown in Fig. Suppose that a collection of ples see . DiffereTo understand the rationale of the model , we consThe model  is identSimilar to the results in the three-factor model in , we can nditions .The general LAD can be formulated as a linear programming (LP) problem and thus be solved by the simplex or the interior point algorithm , 26. ForIn the integration step, one specific model of form  is assumto model , we get\i.e.The models of different transcripts become identical if we reparametrize Consider the following two optimization problems, M1:M2:Ifsolves M2 thensolves M1, whereThe proof is essentially substitution of the corresponding variables. Then the integrated (log) count in the al model  the paraIn the double parameter formulation, if we take LS estimate ; control rats were maintained without drugs. At the same time, all RNA samples were spiked in with the External RNA Controls Consortium [It (GSE47792 ) comes fnsortium  mixed seUsing dataset A we evaluate the accuracy of different methods by comparing their estimated ratios of ERCC sequences with the known control fold changes/ratios. Hereafter ratios and fold changes are used interchangeably. Then we show that the common methods are applicable to the cases of regular transcription profiles yet less effective in the cases of asymmetrically regulated transcription profiles (ART). In the latter, the patterns of up- and down-regulated transcripts between certain pair of samples are different. In statistical words, the expression differentiation is skewed. ART can be visualized by an asymmetric density plot and be summarized by the statistical measure—skewness of log ratios. We propose a guiding criterion of normalization: recover true (log) ratios while preserving the log ratios’ skewness due to its biological context.This  is from MUREN has two available forms: the single-parameter (MUREN-sp) and the two-parameter (MUREN-dp). Other than MUREN, our evaluation and comparison also include Raw (Raw counts), CPM, Q (Quantile), RLE, RUV, TMM, TPM, and UQ (Upper Quartile ).M-value) is defined asA-value) is defined asNotice that throughout the article, the log ratio  symmetric, namely the distribution of transcript-wise log ratios is (nearly) symmetric, the up- and down-regulated transcripts are comparable. In this case, we cannot see much difference between MUREN, TMM, RLE, and UQ as shown in Fig. In the following evaluation, we perturbed the THI data by truncating the transcriptome at the right tail as follows: first, summarize the counts by the medians of the three replicates respectively for the control and treatment samples; second, sort the transcripts in the ascending order by the ratios of the summarized treatment and control counts; finally, remove the top 15% transcripts from all samples. The truncated transcriptome is more asymmetric than the original one is.y-axis limit. Even though the mode of RUV is not influenced, the shape of log ratios’ density is distinctively changed. See Additional file As shown in Fig. M–A plots of ERCC sequences with selected methods applied to the truncated transcriptome. The results coincide with those in Fig. Back to the ERCC sequences, Fig. Part E concerns the evaluation of normalization. According to Proposition M–A plot offers a rather comprehensive diagnosis of the normalization goodness.Another informative feature of the density is its shape. Later we will offer, by a typical example, a biological interpretation of the skewness of transcriptomic differentiation. Thus, we recommend the normalization should not change the overall skewness or modality of log ratios’ distribution. Too flexible methods, usually nonlinear methods, tend to change the shape. The log ratios’ density together with the The shape of log ratios’ distribution, characterized by such as unimodality/multimodality and skewness, is a biological signature of transcriptomic differentiation. We propose that the aim of normalization has two folds: first, improve the accuracy of the log ratios; second, preserve the overall shape of log ratios’ density. Normalization, for example, should neither turn a positively skewed distribution to a negatively distributed one, nor turn a unimodal distribution to a multimodal one.Hereafter, we quantify the skewness by the empirical measure We have explained the necessity of normalization with multiple references in the theoretical setting. In practice, the results using single reference may not have much difference with those using multiple references, provided the differentiation is relatively small and the data quality is high. But we cannot rule out the possibility that an outlier sample in the dataset due to contamination or errors in the sequencing process would be taken as the reference. Normalization with multiple references is not influenced by individual outlier reference sample while the normalization with a single reference is influenced severely. Indeed, this is confirmed by simulations in which some samples were artificially disturbed by increasing or decreasing the counts randomly, see Additional file When we compare transcriptomes of two samples, positive/negative skewness of log ratios are characterized by a heavy right/left tail. This implies that certain biological processes are significantly up- or down-regulated from one sample to the other, Next, we show such an example of cell cycle transitions using single cell RNA-seq data.Dataset B is from a single cell RNA-seq experiment spanning over different cell cycle phases: G1, S, G2, and M. G1 is the first growth phase, and rates of RNA transcription and protein synthesis are high; S is the DNA replication phase, in which most other biosynthesis turns lower; G2 is the growth phase preparing the cell for mitosis; the relatively short M phase undergoes cell division. We normalize the counts of each cell with its total counts. Using the tool implemented in the R package scran , 35 to ap-values. The conclusions agree with the changes of activities along the cell cycle phases.If we compare the expression profiles of G1 versus S phases, the distribution of the log ratios is expected to be positively skewed because general biological processes involved in growth are more active in G1 phase than in S phase. Similarly, the distribution between S and G2/M phases is expected to be positively skewed; the distribution between G2/M and G1 phases is expected to be negatively skewed. Indeed, as shown in Fig. To investigate skewness at the bulk level, we merge the counts of cells in the same phase into pseudo bulk RNA-seq counts and normalize them with MUREN-sp. The diagnostic density plots along with the skewness are shown in Fig. M–A plots. The mode and skewness of the density are important indicators of normalization goodness.In this report, we address the issue—goodness of normalization in two aspects: (1) improve the accuracy of normalization; (2) preserve the skewness of differentiation. Specifically, we check the density plots of expression differentiation along with the The undifferentiated transcripts set between a pair of samples is consistent with the notion of housekeeping genes. With appropriate normalization, as we have shown, the average of the log ratios of the undifferentiated transcripts set is zero. Compared with trimmed average, mode can be visualized for diagnosis. If the symmetric assumption about the undifferentiated transcripts set approximately holds, then the mode of pairwise expression differentiation should be near zero, see Fig. The ability of preserving the asymmetrical differentiation or skewness of data varies across different normalization methods as shown in the examples in Fig. This proposed approach does not depend on a parametric model models such as Poisson distributions or negative-binomial distributions on the read counts. The method is applicable to any dataset as long as the assumption that more than 50% genes are both undifferentiated and are not subject to distortion between a sample and a reference is valid.The MUREN implemented in R package is ready for daily normalization of RNA-seq data. MUREN has an efficient implementation and is integrated with a parallel R package. For the THI data (6 samples), it takes less than half a minute with single thread on a generic desktop computer. For large datasets, the parallel implementation can be specified by one line of code.c, see Fig. 2(0 + 1) = 0, which means zero observed count is still zero after transformation. Second, the fold change of low counts is vulnerable and radical. The offset 1, indeed, shrinks the fold change to zero. Consider two raw counts 4 and 0, the fold change is infinite which is unreliable. Actually, we cannot determine the fold change accurately in this situation. Hence, a shrinkage of the fold change to zero is reasonable. When the offset is 1, log2(4 + 1) − log2(0 + 1) = 2.3; when the offset is 0.0001, log2(4 + 0.0001) − log2(0 + 0.0001) = 15.3.At the beginning of normalization, we log-transform the raw counts plus an offset MUREN performs the RNA-seq normalization using a two-step statistical regression induced from a general principle. We propose that the densities of pairwise differentiations are used to evaluate the goodness of normalization. MUREN adjusts the mode of differentiation toward zero while preserves the skewness due to biological asymmetric differentiation. Moreover, by robustly integrating pre-normalized counts with respect to multiple references, MUREN is immune to outlier samples.Additional file 1. Supplementary_file.pdf.


Canine lymphoma, the most common hematological cancer in dogs, shares many molecular and clinical characteristics with human Non-Hodgkin lymphoma (NHL). The standard treatment for canine lymphoma is “CHOP” multiagent chemotherapy protocol consisting of Cyclophosphamide, Doxorubicin (Hydroxydaunorubicin), Vincristine (Oncovin™), and Prednisone. Approximately 70–85% of patients treated with CHOP achieve clinical remission. However, duration of remission varies and the majority of dogs eventually relapse. To identify possible biomarkers for patients failing to achieve remission, we performed RNA-Seq analysis on 25 cases of canine lymphoma obtained prior the start of their CHOP therapy regime and assessed gene expression associated with patient progression free survival (PFS).The data consists of (1) raw RNA-Seq reads in 75 bp fastq format from fine needle aspirate samples of enlarged lymph nodes from canine patients with naturally occurring lymphoma; (2) Fragments Per Kilobase Million (FPKM) values for each sample; (3) raw transcript counts for each sample; (4) anonymized patient details including PFS; (5) heat map of gene expression and (6) Cox proportional hazard analysis showing significantly expressed genes. These data may be useful for comparative analysis of gene expression in human NHL and analysis of gene expression associated with disease outcome in canine lymphoma. Canine lymphoma is the most common haematopoietic neoplasm in dogs  and is sThe standard treatment is standard in both canine lymphoma and NHL, consisting of a multi-agent chemotherapy protocol of cyclophosphamide, doxorubicin, vincristine, and prednisone (CHOP), often with the addition of rituximab in humans . There iDogs with naturally occurring lymphoma, diagnosed by cytology or histology at the Mona Campbell Centre for Animal Cancer, University of Guelph, who had received no prior treatment other than a single injection of prednisone were eligible for this study. No breed, sex or age restrictions were in place, but dogs with other concurrent neoplasms or prior neoplasms including lymphoma were excluded. Lymphomas were immunophenotyped by flow cytometry . Dogs weTumor samples were collected by fine needle aspirate and expressed into sterile collection vials containing 1.0 ml of RNAprotect Cell Reagent (Qiagen). Poly A-RNA was isolated from the tissue samples using QIAGEN RNeasy isolation kit and quantified using a Nanodrop spectrophotometer. The RNA integrity number (RIN) was calculated for each sample using Bioanalyzer analysis. Twenty-five samples with RIN above 9 were selected for sequencing, including 16 immunophenotyped as B-cell lymphoma, three immunophenotyped as T-cell lymphoma, and six patients with missing immunophenotype. The selected samples were sequenced using the Illumina NextSeq platform by the London Regional Genomics Centre, London, ON, Canada. The sequence data were returned in single read 75 bp fastq format. Data Set 1 contains the RNA sequencing results for this publication, which have been deposited in NCBI’s Gene Expression Omnibus and are accessible through GEO Series accession number GSE130874 .FastQC [Hisat2 [StringTie [Ballgown [p-values < 0.05 . While a file 5 ), we werThis study is limited by the small sample size, and technical issues resulting in batch effects further reducing the statistical power. The lack of complete immunophenotype information for all cases, and the lack of technical replicates (repeat sequencing) are also limitations. The selection of cases to sequence was initially influenced by client decision to treat their dogs with CHOP, and client consent to participate in this study. Recovery of sufficient total RNA of suitable quality from available cases and finances limited the RNA-Seq analysis to 25 samples. All or any of the above may have had unintended and unknown influence on the data obtained.


Through the experiments, it was observed that the two tree-based methods, DT and RF, showed similar trends in results and the remaining three methods, MLP, SVM, and DT, showed similar trends. DT and RF generally showed worse performance than other methods except for one dataset. This suggests that, for the effective classification of microarray data, selecting a classification algorithm that is suitable for data traits is crucial to ensure optimum performance.Microarrays are applications of electrical engineering and technology in biology that allow simultaneous measurement of expression of numerous genes, and they can be used to analyze specific diseases. This study undertakes classification analyses of various microarrays to compare the performances of classification algorithms over different data traits. The datasets were classified into test and control groups based on five utilized machine learning methods, including MultiLayer Perceptron (MLP), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and  Microarrays have been developed by combining modern mechanical and electrical engineering technologies with the existing knowledge in molecular biology. While the traditional methods allowed researchers to measure the expression of a small number of genes at a time, the introduction of microarrays enabled the expression analysis of tens of thousands of genes in a single experiment. This led to the development of experimental techniques that were capable of generating a large volume of genomic information from a single cell . Since vMicroarray-based analysis methods can be broadly classified into five categories: analysis of differentially expressed genes, analysis of differentially expressed gene pairs, functional annotation, clustering analysis, and classification analysis ,3,4,5. IMeanwhile, in recent years, classification analysis using RNA-seq data has frequently been conducted ,8,9,10. The microarray data used in this study included datasets of samples categorized on the following bases. First, the datasets representing the presence or absence of a specific disease were included. Second, the datasets containing either of two similar diseases were selected for this study. Third, the datasets containing samples obtained from people with occupations entailing exposure to harmful environments, such as fine dust, and those obtained from people with other occupations, which are for the control group, were included. Furthermore, two types of data, i.e., miRNA data and RNA data, were analyzed. The application of machine learning algorithms to this variety of microarray datasets can provide a better understanding of the performance of machine learning with microarray data. The machine learning algorithms used for classifying microarray data were implemented in Python, followed by cross-validation to calculate the accuracy as a metric of algorithmic performance.The rest of the paper is organized as follows: Microarrays are highly dense arrays of DNA molecules with known nucleotide sequences aggregated on a small slide . MicroarThe miRNA and RNA datasets were acquired from the Gene Expression Omnibus repository at the National Center for Biotechnology Information (GEO-NCBI) . Among tDataset 01, titled “MicroRNA profiling of chronic lung disease for the Lung Genomics Research Consortium”, provided miRNA data on patients with COPD and interstitial lung disease (ILD), which are two similar pulmonary diseases. COPD is a lung disease caused by repeated exposure to a noxious agent resulting in irreversible airflow limitation and ILD is a loosely defined group of diseases characterized by changes in the interstitium of the lung, causing pulmonary restriction and impaired gas exchange .Dataset 02, titled “The site and nature of terminal bronchiolar destruction in chronic obstructive pulmonary disease (COPD)”, provided RNA data of patients with COPD besides that of healthy subjects . Dataset 03, titled “Genome-wide screening of transcriptional modulation in non-smoking female lung cancer in Taiwan”, provided RNA data of non-smoking females with lung cancer and those without lung cancer . Dataset 04 was titled “Differences in monocyte miRNA profiles between patients with coronary artery disease and healthy controls”. This dataset included miRNA data from healthy subjects and patients with CVD, which is one of the primary causes of death in humans . Dataset 05 was titled “Transcriptomic changes in the nasal epithelium associated with diesel engine exhaust exposure”. Diesel engine exhaust (DEE) is one of the primary causes of air pollution worldwide, which can affect the human body. This RNA dataset was collected from those working in diesel engine factories, along with those working without DEE exposure as the control group, for a detailed analysis of genes affected by air pollution . Dataset 06, titled “Expression of viral and human microRNAs in blood in the Beijing Truck Driver Air Pollution Study”, provided miRNA data collected from the blood samples of truck drivers exposed to air pollution, with that of office workers as the control group. This dataset was constructed to investigate the association between air pollution exposure and clinical outcomes . Since the sample traits listed in the six datasets not only included occupation but also BMIs and smoking status, we were able to conduct classification experiments on various features, such as truck driver/office worker, obese/non-obese, and smoking/non-smoking. All datasets used for the experiments are listed in Feature scaling is a kind of data preprocessing to improve the performance of machine learning algorithms . NormaliThe equation for normalization used in this study is given as follows:If there are no negative values in the dataset, the values are scaled to a range between 0 and 1, and if there are negative values, the values are scaled to a range between −1 and 1. As the microarray datasets used in this study did not contain any negative values, all of them were scaled to a range .Machine learning is a subfield of artificial intelligence in which algorithms are developed to allow systems to train based on a given dataset and execute activities that are not specified in the code  Machine MLP is a layered neural network with one or more hidden layers between the input and output layers . The netSome optimizers that enhance and stabilize the learning rates of MLP include stochastic gradient descent, momentum, nesterovated gradient, and adaptive moment estimation (Adam). Adam was selected for this study, owing to its high computational efficiency, low memory requirements, and scalability in large datasets . The defN-dimensional space into several subsets using the (N-1)-dimensional hyperplane of the maximum margin [C, specifying the degree of error permitted, and γ, specifying the curvature of the boundary.SVM is a machine learning algorithm proposed by Vapnik . It is am margin . The algm margin . The SVMIn this study, the RBF -SVM was used for its specialization in nonlinear factors. The RBF kernel function is expressed as follows :(3)K. The dataset included 319 subjects, of which 183 had ILD and 136 had COPD ,47.C and C and RF and DT, using tree structures, showed lower accuracy than the others. RF had a lower accuracy (73.3%), and all five classification algorithms classified the patients with COPD from those with ILD with more than 70% accuracy.Dataset 02 included microarray data from 77 patients with COPD and 40 healthy subjects .C value of 0.001 and a While RF had the lowest accuracy (68.3%), the algorithms generally had accuracies over 80%, thus classifying the subjects with a disease satisfactorily.Dataset 03 provided miRNA data for healthy subjects and patients with lung cancer . This daC and γ values displayed on a color scale. The highest accuracy was achieved with a C value of 0.001 and γ values of 0.001, 0.01, or 0.1. The lowest accuracy was 73%, suggesting that all algorithms classified between the two groups well.Dataset 04 provided the miRNA data of 40 males with premature CVD and 40 healthy males of the same age, measured using microarrays ,50,51C value of 0.001 and a C and γ values displayed on a color scale. The highest performance was observed when the C value was 0.001, and the γ value was 0.01.Dataset 05 provided RNA data of factory workers exposed to DEE and those not exposed to DEE ,52. The C and γ values displayed on a color scale. The highest performance was achieved when the C and γ values were both 0.001. k plotted as a graph. KNN achieved the highest accuracy, with a k value of five. MLP and KNN showed high accuracies in this experiment. However, the tree-structure algorithms, random forest, and decision tree failed to classify this data well with an accuracy of less than 50%.Dataset 06 provided miRNA data on truck drivers with high exposure to air pollution, as well as office workers with relatively low exposure to air pollution . All parThis dataset listed occupations, BMIs, and information on whether the subjects smoked or not as traits of each sample. This enabled us to conduct further classification experiments, such as truck driver/office worker, obese/non-obese, and smoking/non-smoking. Using this dataset, we classified the differences between truck drivers and office workers.C and C and k plotted as a graph. KNN achieved the highest accuracy with a k value of 24. The decision tree correctly classified less than half of the dataset.In this experiment, KNN showed the highest accuracy among the five classification algorithms. In the next experiment, the algorithms were applied to classify the same dataset by obesity (obese/non-obese). In this experiment, subjects with a BMI of 25 or higher were classified as obese. The WHO  uses the cutoff point of BMI for defining obesity as 30 , howeverC and γ values displayed on a color scale. The highest accuracy was achieved with a C value of 0.001 and a γ value of 0.001 or 0.01. k plotted as a graph. The best performance was recorded with a k value of 3, and other algorithms also performed the classification adequately.The next experiment also used the same dataset to classify the subjects based on whether or not they smoked.C and γ values displayed on a color scale. The highest accuracy was achieved when the C and γ values were both 0.001. k represented as a graph. KNN and DT classified the data adequately, with KNN achieving the highest accuracy, with a k value of eight.Experiments were conducted to classify microarray data into two groups using machine learning. In the first experiment, wherein incidences of COPD and ILD were classified, all five algorithms distinguished two groups with an accuracy of more than 70%. The second experiment aimed to classify the data regarding the differences between patients with COPD and healthy subjects. In this experiment, SVM showed the highest accuracy (99%) in contrast to RF (random forest), which showed the lowest accuracy (68.3%). In the third experiment, classification was carried out to differentiate between lung cancer patients and healthy people, and the DT (decision tree) and SVM models showed high accuracies. In the fourth experiment, subjects in the dataset were classified based on the difference between patients with CVD and control groups. SVM showed higher accuracy (77%) than the other algorithms. The fifth experiment was aimed at differentiating between factory workers with and without exposure to DEE. In this case, KNN classified the dataset optimally and was able to distinguish between the two groups with an accuracy of 90%, while RF and DT displayed low accuracies. The sixth experiment was performed to classify the individuals by occupation, obesity, and smoking habits. When classifying by occupation, all the algorithms showed low accuracy. KNN showed good accuracy of 80.3% in the classification based on obesity. However, all algorithms were also inadequate in classifying the same dataset based on smoking and displayed a mean accuracy of 61.5%.C values than the heatmaps for other datasets where the performance of SVM was poorer.Furthermore, the heatmaps for Datasets 02 and 03, which were classified by SVM with substantially high performances, displayed clearer boundaries for Among the dataset used, Datasets 01, 04, and 06 are the miRNA datasets and others are the mRNA datasets. In this paper, microarray data with various traits were classified into two groups using various representative machine learning methods, MLP, SVM, DT, RF, and KNN. In the experiments, diverse classification criteria were applied, such as classification between two similar diseases, classification between people with and without diseases, and classification between two occupational groups. The accuracies by the five machine learning methods with these various datasets were compared. The results suggest that the best-performing machine learning model varies across datasets. However, it was observed that the tree-based methods, DT and RF, showed similar trends in results and the remaining methods, MLP, SVM, and KNN, showed similar trends. DT and RF generally showed worse performance than other methods except for one dataset.Although only microarray data were dealt with in this paper, the methodology of this study is similarly applicable to RNA-seq data, which is known to be more sensitive in detecting differential expression and offers increased dynamic range. So, further study using the latest RNA-seq with the methods in this paper is needed. It will be interesting to verify whether similar results are derived in experiments using RNA-seq data as experiments on microarray data.


Advancements in sequencing have led to the proliferation of multi-omic profiles of human cells under different conditions and perturbations. In addition, many databases have amassed information about pathways and gene “signatures”—patterns of gene expression associated with specific cellular and phenotypic contexts. An important current challenge in systems biology is to leverage such knowledge about gene coordination to maximize the predictive power and generalization of models applied to high-throughput datasets. However, few such integrative approaches exist that also provide interpretable results quantifying the importance of individual genes and pathways to model accuracy. We introduce AKLIMATE, a first kernel-based stacked learner that seamlessly incorporates multi-omics feature data with prior information in the form of pathways for either regression or classification tasks. AKLIMATE uses a novel multiple-kernel learning framework where individual kernels capture the prediction propensities recorded in random forests, each built from a specific pathway gene set that integrates all omics data for its member genes. AKLIMATE has comparable or improved performance relative to state-of-the-art methods on diverse phenotype learning tasks, including predicting microsatellite instability in endometrial and colorectal cancer, survival in breast cancer, and cell line response to gene knockdowns. We show how AKLIMATE is able to connect feature data across data platforms through their common pathways to identify examples of several known and novel contributors of cancer and synthetic lethality. We describe a new method that incorporates multimodal molecular measurements with gene pathway information for classification and regression prediction tasks. The method combines powerful machine learning methodologies such as ensemble, kernel and stacked learning. A key new contribution is the creation of empirical kernels from pairwise similarities of sample predictions and sample paths along the trees of a random forest specific to a particular gene module. We demonstrate the method performs as well as, or better than, top approaches on three very different cancer genomics applications. PLOS Computational Biology Methods paper.This is a The drop in sequencing cost has made it common for biological experiments to generate multi-omic profiles under a variety of conditions and perturbations. For example, the Cancer Genome Atlas (TCGA) contains thousands of patient samples with simultaneous copy number, mutation, methylation, mRNA, miRNA and protein measurements . The anaIn this study, we introduce the Algorithm for Kernel Learning with Integrative Modules of Approximating Tree Ensembles (AKLIMATE)—a novel approach that combines heterogeneous data with prior knowledge in the form of gene sets. AKLIMATE can evaluate the predictive power of individual features (e.g. genes) as well as feature sets (e.g. pathways). It harnesses the advantages of Random Forests , Multiple Kernel Learning , and stacked learning (improved accuracy) while avoiding many of their shortcomings. AKLIMATE relies on three major computational insights. First, it leverages biological knowledge and multiple data types by building a separate RF model for each distinct gene set , with the ability to incorporate heavily overlapped feature groups without the undesirable side effects of other approaches  knockdown viability in cancer cell lines. We benchmarked AKLIMATE against comparable methods that have performed well in recent Dialogue on Reverse Engineering and Assessment Methods (DREAM) challenges. We chose both classification and regression tasks as well as various levels of data availability—a single data type, multiple data types (including inferred data), or multiple data types with clinical information.We first tested AKLIMATE on predicting microsatellite instability in the colon and rectum adenocarcinoma (COADREAD) and uterine corpus endometrial carcinoma (UCEC) TCGA cohorts. Microsatellite instability (MSI) arises as a result of defects in the mismatch repair machinery of the cell. Tumors with MSI (often accompanied by higher mutation rates) represent a clinically relevant disease subtype that is associated with better prognosis. MSI is also an immunotherapy indicator as such tumors produce more neoantigens. MSI can be predicted with high accuracy from expression alone , providiWe used expression data  and MSI annotations as described in  for the ζκ, ηκ), which were set to  in the dense models  =  in the sparse ones). Finally, the single-task models were trained separately on the UCEC and COADREAD cohorts, while the multitask versions learned MSI status on the two TCGA cohorts jointly, with each cohort representing a separate task. All train/test splits were matched across methods. All BMKL models used 196 PID gene sets; model parameters, kernel computations and data filtering steps matched the setup in λ1∑i=1M‖αα* is the solution to The explicit form of AKLIMATE’s optimization problem is:ribed in . The optThe most common approach to kernel matrix evaluation is to specify an explicit data dependency model for the kernel function—e.g. polynomial, Gaussian, ANOVA or graph kernels . HoweverM is the number of trees in the forest, T is the number of leaves in a tree, θm is a variable capturing the random selection of training set samples and features in the process of constructing the mth tree, Rt is the tth leaf of the mth tree, and I(⋅) is the indicator function. The finite approximation of M < ∞; K3 kernel) is positive definite [Defining a kernel through a RF is not a new idea—in fact the concept was introduced at the same time as RFs . In partdefinite . Kernelsdefinite  and 73]K=K1 kernel). The reasoning for the second one is that earlier node splits in a tree generally separate more distinct sample groups, while later splits tend to fine tune the decision boundary, highlighting more subtle differences. Thus, irrespective of the predicted labels, two samples that end up in different tree depths—one early- and one late-split leaf—should be considered less similar than samples landing in two late-split leaves (K2 kernel). Incorporating these three patterns, AKLIMATE’s RF kernel matrix is computed using the following steps:AKLIMATE’s RF kernel extends the original definition  by incorpi =  is a vector of predictions for data point i from the M trees in the RF. pi always has continuous entries—either the actual predictions in a regression setting, or the probabilities of class membership for classification problems. The ‖pi − pj‖2 distances are divided by the scaling constant σ = maxi,j‖pi − pj‖2 so that they map to the  range. Exponentiation of the negative distances converts them to similarities.Calculate similarity over predictions across RF trees:ti =  is a vector of the leaf indices of sample i across the RF. Tree nodes are indexed starting from the base node and then sequentially across each depth level of the tree. The scaling constant σ is set in the same manner as in K1.Compute similarity over leaf node indices across RF trees:K3 on a similar scale to K1 and K2.Calculate similarity over leaf node co-occurrence using the finite approximation of K1,K2 and K3:Calculate the RF kernel as the geometric mean of the element-wise product of K1, K2, and K3 are positive definite, so too is K. K1 and K2 are Gaussian kernels over K3 is the exponent of a positive definite kernel, i.e. a positive definite kernel as well [K is the Hadamard product of three positive definite kernels raised to a positive power—both of these operations preserve positive definiteness [Importantly, since the components niteness . K3 is t as well . Finallyniteness .Δ = {Δ1, …, ΔS} then the stacked generalization proceeds as follows [Stacked learning is a generalization of ensemble learning in which the stacked model (meta-learner) uses the prediction output of its components (base learners) as training data for the computation of the final predictions , 74. Wha follows , 75:V folds of roughly equal size—h1, …, hV . Note that each hv defines a set of indices that select a subset of the samples in X.Randomly split the level-zero data into s base learner, train V models, collectively denoted as For each ΔV sets of predictions from z of length N. The N × S matrix Z of such vectors for all S base learners becomes the feature matrix for level-one training.Concatenate the Z with hyperparameter tuning if necessary, again using Y as the labels.Train a meta-learner Ψ on s base learner on the full level-zero training set X.The base learners used in the final stacked model are created using all training samples, so one final  training round is needed. To this end, train each Δs: s = 1, …, S}, Ψ). To predict on a new data point Xnew, compute a 1 × S vector Znew = {Δs(Xnew): s = 1, …, S} and use Ψ(Znew) as the stacked model’s prediction.Form the stacked model using the base-learners trained on the full data set and the meta-learner to obtain  , 59. Sim1, …, ΔS} are RFs, each of which is used to produce an associated kernel. The meta-learner Ψ is an elastic-net regularized MKL that can be interpreted as the kernel learning counterpart of linear regression. Before describing the algorithm, we first discuss how the level-one RF kernels are constructed using OOB samples.To our knowledge, AKLIMATE is the first instance of a kernel-based stacked learner. The base learners {ΔAKLIMATE uses RF kernels as the level-one training data. Normally, level-one data is generated with cross-validation to help the meta-learner avoid overfitting. Analogously, AKLIMATE utilizes out-of-bag (OOB) samples to generate the components of the RF kernels. For each pair of samples in the RF, the kernel similarity matrix is calculated using only those trees for which both of the samples have been withheld, with the following procedure:M trees, define OOB(m) as the set of samples that are OOB in the mth tree. Let IOOB be the tree-level indexing function recording when both samples i and j are simultaneously OOB in a given RF; i.e.:For a RF with Compute the first constituent kernel matrix:Compute the second constituent kernel matrix:Compute the third constituent kernel matrix:Calculate the combined kernel matrix:σ calculations as in Eqs (with the same notation and s in Eqs –10)..σ calcul1, λ2) that require tuning. This is done by generating a random set of  pairs and ranking them based on V-fold (default V = 5) cross-validation fit with OOB RF kernels as input. To improve generalization, we use a simplified version of the overfit correction procedure in [AKLIMATE’s MKL meta-learner has two elastic-net hyperparameters  are supplied, each containing a list of features Ps such that the complete set is Δ*. In addition, a user-supplied parameter G determines the number of top feature sets (pathways) to incorporate for each sample during the base learning step. We use G = 5 in practice.We next describe AKLIMATE’s learning algorithm. The method takes training data τ is stored so that the top L models (by τ) can be selected that correctly predict an individual sample. A final set of relevant RFs Δ* is obtained by taking the union over all sample-specific top L models. The helper function RFKERNELOOB creates a kernel from a given RF utilizing the out-of-bag approach described previously. MKL uses these level-one kernels to tune the elastic net hyperparameters λ1 and λ2 based on cross-validation performance. A final meta-learner Ψ* is then trained by running MKL with kernels constructed from the full RFs (RFKERNEL helper function) and with the determined elastic net hyperparameters.Formally, AKLIMATE operates according to the pseudocode in Algorithm 1. First AKLIMATE trains a separate RF for each feature set using data from all modalities relevant to the features in the set. Model accuracy Algorithm 1: AKLIMATEX, Y), C, P, GInput:  RFs predicting sample n correctly endKoob ← {}forRFr ∈ Δ* doKoob ← Koob ∪ RFKERNELOOB endKfull ← {}forRFr ∈ Δ* doKfull ← Kfull ∪ RFKERNEL endΔ* in Algorithm 1 can filter the full collection of feature sets down to a subgroup of relevant sets two or more orders of magnitude smaller in size. This makes it possible to incorporate appreciably more feature sets than standard MKL algorithms which require kernel evaluation for all feature sets. When the number of such sets is in the thousands, MKL can be computationally very slow, even for data sets of small sample size. However, usually only a small proportion of the feature set collection is truly explanatory for a given prediction task. Thus, filtering out the non-relevant parts of the compendium does not impact MKL accuracy yet drastically improves computation time.AKLIMATE’s selection step for the best RFs Δ* discovery process for a classification task. However, if Y is continuous (i.e. a regression problem), the predictions and labels are not directly comparable for equality. One approach is to take the squared error of the prediction-label differences and use that metric to re-rank RFs for each sample. In our experience, this leads to the selection of suboptimal RFs due to overfitting. Instead, AKLIMATE uses a more robust scheme that shows better results in practice—the vector of predictions for each sample across all RFs q is a user-specified quantile of the empirical distribution of absolute prediction errors q = 0.05). This setup prioritizes RFs that perform near-optimally on individual data points and optimally when the training set is considered as a whole.Algorithm 1 demonstrates the wFS, i as its member and their associated RF was selected among the set of best RFs Δ*, while m is an RF-specific feature importance score computed from the RF associated with the kth such feature set.Feature set weights earner Ψ*. Featurem is by averaging the improvement in the splitting criterion over all nodes that used feature i as a splitting variable. For the often used Gini impurity measure, this involves computing the mean difference in impurity before and after each split, with larger mean impurity decreases indicative of more important variables [The simplest way to compute ariables . While fariables .m rules have been proposed [i is randomly permuted and the permuted variable is used in the calculation of OOB predictions; the difference in error rate between the permuted and non-permuted OOB predictions is taken as a measure of feature i’s importance. Permutation-based importance is robust and generally performs on par with more complex m rules. Its biggest drawback is the higher computational cost. In cases where compute time is the main constraint, we recommend the actual impurity reduction (AIR) metric [Many alternative proposed –82. For proposed —the vect) metric , which i) metric . It is sS1 Text(PDF)Click here for additional data file.S1 TableAKLIMATE weights were averaged over 50 train/test splits. The table lists the 20 most relevant feature sets, out of 1836 feature sets with a non-zero weight in at least one train/test split. Weights were normalized to sum to 1. Aliases for the top 10 feature sets are included in brackets—they provide a more descriptive name for the underlying biological function, based on information gathered from the source publication. The aliases are used in (PDF)Click here for additional data file.S1 FigΔ*. Ensemble superlearner component RFs—learning a regularized linear regression on predictions from component RFs; ensemble average component RFs—taking the average of the predictions of component RFs; top component RF—only using predictions from the top ranked RF. (A) UCEC TCGA MSI prediction. (B) METABRIC Breast Cancer survival prediction. (C) Achilles shRNA knockdown prediction. P-values for paired Wilcoxon signed rank test.To ensure fair comparison, the component RF set for each prediction task was restricted to AKLIMATE’s best RFs (EPS)Click here for additional data file.S2 FigTop 10 predictive feature sets and top 50 predictive features from 50 stratified 75% train/25% test splits are shown. Organized as (EPS)Click here for additional data file.S3 FigAUC computed for 50 75%/25% stratified train/test splits. P-values for paired Wilcoxon signed rank test. Methods as in (EPS)Click here for additional data file.S4 FigMethods include 3 versions of AKLIMATE with different combinations of input data types and some high-performing competitive methods (see main text). Point-and-whiskers plot displays mean and standard error of a method’s test accuracy during cross-validation. Mean and standard error for FSMKL and BCC taken from .(EPS)Click here for additional data file.S5 Fig(EPS)Click here for additional data file.S6 FigSamples were assigned to BRCA subtypes using the PAM50 gene expression signature .(EPS)Click here for additional data file.S7 FigA) Measured by Spearman correlation. B) Measured by the number of times an algorithm produced the best Spearman correlation on a prediction task. C) Measured by Pearson correlation. D) Measured by the number of times an algorithm produced the best Pearson correlation on a prediction task.(EPS)Click here for additional data file.S8 FigRows correspond to individual gene knockdowns; columns represent different methods. To highlight differential performance within each task, rows are centered by subtracting the median. Lower centered scores represent lower RMSE (better performance).(EPS)Click here for additional data file.S9 FigColumns correspond to the relative contributions of each input data type across prediction tasks. Relative contributions were computed by summing the model-assigned feature importance scores within individual data types. Columns are normalized to sum to 1.(EPS)Click here for additional data file.S10 FigOrganized as (EPS)Click here for additional data file.S11 FigFeature and feature set weights averaged over 10 matched stratified 80%/20% train/test splits. Organized as (EPS)Click here for additional data file.S12 FigResults averaged over 10 matched stratified 80%/20% train/test splits.(EPS)Click here for additional data file.S13 FigResults averaged over 10 matched stratified 80%/20% train/test splits.(EPS)Click here for additional data file.


Alzheimer's Disease (AD) is one of the most common causes of dementia, mostly affecting the elderly population. Currently, there is no proper diagnostic tool or method available for the detection of AD. The present study used two distinct data sets of AD genes,which could be potential biomarkers in the diagnosis. The differentially expressed genes (DEGs) curated from both datasets were used for machine learning classification, tissue expression annotation and co-expression analysis. Further, CNPY3, GPR84, HIST1H2AB, HIST1H2AE,IFNAR1, LMO3, MYO18A, N4BP2L1, PML, SLC4A4, ST8SIA4, TLE1 and N4BP2L1 were identified as highly significant DEGs and exhibited co-expression with other query genes. Moreover, a tissue expression study found that these genes are also expressed in the brain tissue.In addition to the earlier studies for marker gene identification, we have considered a different set of machine learning classifiers to improve the accuracy rate from the analysis. Amongst all the six classification algorithms, J48 emerged as the best classifier,which could be used for differentiating healthy and diseased samples. SMO/SVM and Logit Boost further followed J48 to achieve the classification accuracy. Alzheimer's Disease (AD), a cognitive, neurological disorder characterized by progressive dementia, commonly causes dementia. Pathologically, AD is marked by degeneration of myelinated axons of nerve cells, the presence of neuritic plaques, and neurofibrillarytangles (NFT) . AD evolTwo different Gene expression datasets analyzed on the HG-U133_Plus_2 platform were retrieved from NCBI's GEO database (https://www.ncbi.nlm.nih.gov/geo/). The first dataset (Accession ID: GDS2795) was collected from samples of Neurofibrillary tangles bearingentorhinal cortex (Diseased) and Non-neurofibrillary tangles bearing entorhinal cortex . The second dataset (Accession ID: GDS4136) had samples from Hippocampal sections (CA1) tissue blocks containing gray and white matters. These samples wereclassified as Control, Incipient, Moderate and Severe. Only Control/Normal and Severe samples were selected for further analysis.The normal and diseased samples from both the datasets were downloaded in CEL format and analyzed in R (4.0.3) utilizing Bioconductor packages. The probe intensities were normalized using RMA package and DEGs were obtained using limma package of Bioconductor.The p-value for the two datasets was set to 0.01 and 0.001, respectively, to obtain DEGs top-hits.Machine learning classifier and cross-validation of DEGs were performed in Weka . Weka is open-source software that helps in data preprocessing and implementing several Machine Learning algorithms to solve real-worlddata complexities by clustering, classification and other techniques. The DEGs obtained from both the datasets were taken and their transformed expression values respective to each sample type were fed to six different classifiers. For classification, samples werecategorized into two classes i.e. Normal and Diseased. Twenty folds cross-validation were set with each classifier. The classifiers used were Bayes Net, Naive Bayes, SMO/SVM, Logit Boost, Decision Table and J48. An online functional annotation tool DAVID was used for identifying the expression of DEGs in their respective tissues. Further, DEGs were fed to Gene Mania® online tool to identify their co-expression and evaluate its association with other genes with thehelp of co-expression GRN.A total of 39 genes and 12 genes were obtained from GDS2795 and GDS4136, respectively, data presented in Among all the classifiers, only six classifiers showed the highest accuracy with 90%, later followed by 85% accuracy. The True Positive Rate (TP Rate) and False Positive Rate (FP Rate) for these classifiers varied from 0.9 to 0.8 and 0.0 to 0.2, respectively.The accuracy percentage of SMO/SVM, Log it Boost and J48 was 90%, whereas the accuracy percentage of Naïve Bayes, Bayes net and Decision Table was 85%. Table 3 representing the classification results from all these classifiers.The co-expression GRN of the DEGs was obtained from both the data sets shown in Two different datasets of gene involved in AD progression were used in identifying DEGs. Different p-values, e.g., 0.01 and 0.001, were used to generate top genes with higher differential expression. We have identified a total of 51 DEGs, 39 and 12 from GDS2795and GDS4136, respectively. Further, validation was performed for assessing the involvement of DEGs in AD. These genes were subjected to the online annotation tool DAVID. The genes obtained from the annotation tool were GPR84, LMO3, N4BP2L1, ST8SIA4, PSITPTE22, CNPY3,CDK6, DTNA, GNAL, HIST1H2AB, HIST1H2AE, LOC339047, ZNF337, LOC100132540, IFNAR1, LRRC8A, METTL14, MYO18A, NLGN1, LOC652346, PML, SEMA6A, TLE1, LOC645382, SLC4A4 and ZNF518A, and these 26 genes are expressed in the brain. Further, the next hypothesis was to identifywhether these genes could be implemented to classify a sample as Diseased or Normal. For attaining this objective, the training data set was prepared by using 26 genes and twenty-fold cross-validation was utilized. As a result, classifiers such as SMO/SVM, J48 andLog it Boost achieved 90% accuracy, while Naïve Bayes, Bayes Net and Decision Table attained only 85% accuracy. Since machine-learning classifiers have been widely used for sample classification ,29. In oIn the present study, an integrated approach of bioinformatics data analysis and machine learning classification was used. We have identified 13 DEGs that could be utilized in distinguishing AD and Normal samples. Therefore, these 13 genes could be used as potential gene set as biomarkers to identify AD. Moreover, only six machine-learning classifiers qualified for further analysis and J48 emerged as the bestclassifier amongst all the classifiers. The accuracy of J48 was 90% and TPR was found to be 0.9 and 0.00, respectively. Other classifiers such as SMO/SVM and Log it boost showed an accuracy of 90% and attained TPR and FPR 0.9 and 0.0, respectively. Therefore, theresults from this study also signify the highest accuracy result from J48 algorithm amongst the set of six considered classifiers applied on the same data. This accuracy of J48 for sample classification may need further validation by using it to datasets from abroader range of AD samples and other diseases.


When a condition causes gene expression changes in a mixed tissue sample containing different types of cells, the changes can originate from either altered cell type composition or altered expression in some of the cell types. For example, in the context of type 1 diabetes it is an open debate if the pancreatic beta cells are dead  or if they have, at least initially, just stopped insulin production  . ComputaThere are many publicly available tools for computational deconvolution , and then’ (e.g. ) aims ton’ (e.g. ) estimats’ (e.g. ) do bothAs mentioned above, the cell type-specific expression profiles might be altered by some factors  and, in such case, the assumption of the matrix  ISOpure  is relat ISOpure  provides ISOpure  offers t ISOpure . NotablyIdentifying csDEGs is a more relaxed goal than fully personalized the task , 25–27, the task , 28–30, Here we evaluate nine methods for identifying csDEGs: tools for the analysis of heterogeneous tissues (TOAST) , cell tyWe utilized three semi-simulated datasets addressed as GSE60424, EMTAB9221 and GSE124742 in these tests. The datasets were constructed by first generating csGEPs with realistic individual variation and DEGs between two sample groups, 100 samples each. For the evaluation purposes, these csGEPs were then combined into bulk samples by calculating their sum weighted by the cell type proportions. As gold standard csDEGs, we considered DEGs identified using the csGEPs  To evaluate the accuracy of the estimated csDEGs, we considered the overlap between the estimated csDEGs and the gold standard csDEGs. Among the tested methods, LRCDE detected over 5000 csDEGs with FDR Because of the large variation in the number of significant detections, we compared the top most significant findings to the gold standard. The size of the evaluated top list was defined as the number of detections in the gold standard. For dataset GSE60424, methods TOAST, CellDMC and TCA had the best performance. For dataset EMTAB9221, CARseq and TCA were the most accurate ones. For dataset GSE124742, TOAST, Rodeo, qprog, and CellDMC showed the best performance. Overall, LRCDE and DESeq2 provided less accurate estimates for csDEGs than the other methods, but otherwise differences between the methods’ performances were moderate as shown in Relative performance between the methods is not the only conclusion that can be drawn from the accuracies reported in Although most of the methods were fast to run (maximum few minutes), CARseq and especially expression deconvolution methods Rodeo and qprog were more time consuming. For reference, the running times with dataset EMTAB9221 for different methods wereTOAST: 3.18 s,csSAM: 3.43 min,LRCDE: 4.51 s,CARseq: 1.37 h,Rodeo: 1.74 min,qprog: 13.21 s,CellDMC: 28.83 s,TCA: 2.12 min andDESeq2: 2.27 min.P-value estimation (1000 in this study) to get the full time required for detecting csDEGs. Despite making the approaches slow, the P-value estimation step can not be neglected as demonstrated in Section 3 of Supplementary text.Notably, as described in section ‘Tested methods’, the running times of expression deconvolution methods Rodeo and qprog need to be multiplied by the number of random samplings for Next, we investigated how the individual variation in csGEPs over samples affects the accuracy of detecting csDEGs. We set the coefficient of variation, i.e. ratio of standard deviation and mean, to a fixed level by controlling the standard deviation of the gene when constructing the csGEPs for the bulk data. Coefficients of variation of 0.1, 0.5, 1, 1.5, 2 and 2.5 were tested in dataset EMTAB9221, and they were the same for all genes. To investigate the effect of individual variation in only one cell type, we also generated data where only the coefficients of variation within monocytes were controlled while other cell types had their original, measured standard deviations varying from one gene to another. As this test was computationally intensive, only methods TOAST, csSAM, CellDMC and TCA were considered.Our results show Figure  that indOur observations this far highlight the cell type proportions and internal variation over samples as important factors affecting the accuracy of the results. In addition, we tested multiple other potentially important factors, but according to our observations, their impact was minor compared to cell type proportion and individual variation. These include imbalanced sample groups (Section 5 in Supplementary text), definition of the present cell types (Section 6 in Supplementary text), and noise from very rare cell types (Section 7 in Supplementary text). Although the input matrix Although the end user does not have direct knowledge about the sample heterogeneity of underlying entclass1pt{minimaAs the bulk data to be analyzed can contain samples, whose csGEPs drastically differ from those of the other samples, we tested how sensitive the different methods are to such outlier samples. For this purpose, one outlier case sample and five outlier control samples were added into each dataset and the accuracy of the detected csDEGs was evaluated.Rodeo is designed to be robust against few outlier samples and indeed with such samples it had the best performance in datasets GSE60424 and GSE124742. In dataset EMTAB9221, TCA and Rodeo were the top performers. The relative performance of the other tested methods mostly did not change despite the accuracies overall decreasing, except for CARseq, which clearly suffered from the outliers in dataset EMTAB9221 and csSAM, which tolerated them fairly well in dataset GSE124742 Figure .Additionally, we tested how well the methods tolerate different numbers of outlier samples. To evaluate this, we added 1—8 outlier samples to EMTAB9221. As shown in Figure As pathway-level findings can be more robust than gene-level results, we tested the reproducibility of the pathway-level results using the estimated csDEGs. TOAST and CellDMC were excluded from these tests as they do not provide fold changes (or csGEPs for both sample groups separately) required for SPIA pathway analysis, whose benefits have been shown in earlier comparisons , 37. HerNone of the estimated csDEG lists generated many false positive findings, i.e. when there were only few, if any, gold standard findings, the number of findings from different methods were also low. The precision and recall values for cell types and datasets with more than 10 gold standard findings are listed in Based on these observations, it seems that if many pathway findings can be detected from the estimated csDEGs, both pathways and the csDEGs are likely accurate. However, the opposite is not necessarily true: few or no pathway findings can indicate either inaccurate csDEGs (e.g. csSAM estimates for NK cells (GSE60424)), accurate csDEGs but no pathway findings as there is not much to find (e.g. csSAM findings for beta cells (GSE124742)), or accurate csDEGs but no pathway findings even if there would be significant pathways (e.g. csSAM findings for neutrophils (EMTAB9221)).Not a single method can be claimed as the best one, but TOAST, CARseq, CellDMC and TCA had the highest accuracies. In case of over four outlier samples with altered csGEPs, Rodeo has more reliable performance. In the original publication introducing CARseq, it was validated with raw counts, which might explain why its performance compared to the other tested method was better in datasets EMTAB9221 and GSE124742, which are based on raw counts, than in GSE60424, which is based on normalized csGEPs. TCA is an accurate and fairly robust method, but it threw errors in some occasions. The observation that it performed well in sample deviation tests with small coefficients of variation but threw errors with larger variation suggests that the method does not tolerate noisy data with large heterogeneity between the samples. All methods excluding LRCDE tend to detect too few rather than too many csDEGs if FDR cutoff 0.05 is used, but the detections are mostly correct (i.e. high precision). Our results show that accurate csDEGs can be achieved if (i) the individual variation is moderate, which can be evaluated using residuals, and (ii) cell type proportion is not low. Other tested factors such as imbalanced sample groups, definition of cell types present in the samples, and minor noise from rare cell types had weaker influence on the accuracy. Pathway analysis from the estimated csDEGs can serve two purposes. Firstly, if there are fairly many (e.g. more than five) significant pathway detections, the findings are likely accurate and conclusions about underlying biological processes can be drawn as in any pathway analysis. Secondly, sufficiently many pathway findings indicate that the estimated csDEGs used as an input for the pathway analysis are accurate. However, in case there are no pathway findings, no conclusions can be drawn from that.Notably, when estimating the accuracy of the results from the information the end user can access, we used residuals relative to bulk expression to make our conclusions robust against the overall magnitude of the bulk expression values. Also, in order to prevent possible issues caused by different depth of the measured transcriptomes, only 10 000 genes with the highest median expression in the bulk data were used to calculate the median residual. With these precautions, our cutoffs for the median residual should generalize well for other studies. However, for instance, samples from tissues with very few expressed genes might not follow the rough pattern described here.Estimating the level of individual variation within cell types is not the only application for the residuals. It is not obvious whether the bulk data to be analyzed contains outliers, because a sample with a heavily altered cell type composition likely has a bulk expression profile clearly distinct from the rest of the samples. However, it is still not an outlier in a sense that it causes difficulties for the tested methods, unless the csGEPs are also altered. The residuals contain the variation originating from csGEPs instead of We are unsure why results from dataset GSE60424 were less accurate than those from the other datasets despite having the lowest coefficients of variation (Section 4 in Supplementary text) over the samples for the different cell types. The dominating cell type (neutrophils) has higher internal variation than the other cell types present in the samples, but the same is true for alpha cells in dataset GSE124742, so that is not likely the key explanation. Also, the dataset has more cell types than the other test datasets (six versus four), but reducing them by combining T cells or considering NK cells as noise did not increase the accuracy. Another possible reason is that the randomly generated csDEGs in GSE60424 are weaker than those based on real observed differences within cell types in the other two datasets (section ‘Test data’). However, the number of detected gold standard csDEGs was typically similar to the other datasets, which weakens this explanation. Finally, one explanation could be the very dominating role of neutrophils, which leaves the proportions of the other cell types quite low (Section 2 in Supplementary text).The main limitation of this study is that all our test data are semi-simulated, because cell type-specific datasets with enough sample donors were not publicly available, yet sample size has been shown to affect the accuracy of the estimated csDEGs and expression deconvolution , 27, 38.Besides the methods tested here, there are few tools to detect csDEGs that were excluded from this study for various reasons. For example, BSEG-sc (utilizes csSAM) , CellCODAs expression deconvolution requires more samples than available in publicly available datasets with separated cell types, we constructed semi-simulated data using three datasets, GSE60424, EMTAB9221 and GSE124742, with 200 samples. In all cases, the bulk data was constructed by first generating cell type proportion matrix Dataset GSE60424 is based on RNAseq data from individuals with different autoimmune diseases available in Gene Expression Omnibus (GEO) with the same accession id [Dataset EMTAB9221 was built from single cell RNAseq data E-MTAM-9221  availablAlso dataset GSE124742 was constructed from single cell RNAseq data and the original data is available in GEO under the same accession id . It contTo evaluate the accuracy of the results, we used csDEGs detected by ROTS  from theTo test the effect of individual variation in When evaluating how the residuals can be utilized to evaluate the accuracy of the results, csSAM and data from individual variation test was used. Data with all cell types’ internal variation controlled and data with only monocytes’ internal variation controlled were both used. Method csSAM provides the residuals readily as an output, but they can be also calculated manually by subtracting re-constructed bulk To evaluate the effect of outlier samples, we generated one case sample and five control samples with altered csGEPs into each dataset. The cell type proportions for these samples are from the same distribution as the rest of the samples. For each cell type and data set, the csGEPs for the outliers are generated from normal distribution using randomly either lower or upper limit for an outlier as a mean. The limits for an outlier were set according to interquartile range (IQR) definition, i.e. the lower limit is As pathway detections can be more robust than gene-level ones, we tested their reproducibility as well. The pathway analysis was done with signaling pathway impact analysis (SPIA)  as its iWe tested nine methods, all implemented into R packages: TOAST (version 1.0.0), csSAM (version 1.4), LRCDE (version 1.0.1.0000), CARseq (version 0.0.0.9007), Rodeo (version 1.0), qprog (CellMix version 1.6.2), CellDMC (EpiDISH version 2.2.2), TCA (version 1.2.1) and DESeq2 (version 1.26.0). A cell type proportion matrix t-test is utilized for the task. Although csSAM is an old and widely cited tool for this particular problem, LRCDE contains also a function to estimate cell type proportions using a signature matrix. CARseq [Among the nine tested methods, four have been designed to detect csDEGs. TOAST  is a ver. CARseq  is a novMethods Rodeo and qrpog have been originally developed for expression deconvolution, but we tested here their utility to identify csDEGs. These two were selected into this study due to their accurate performance , but anyEstimate Calculate (absolute) difference between For P value estimates as P values for all genes in all cell types. Correct P values into FDR values, if wanted.Calculate Rodeo  is basedCellDCM  and TCA DESeq2  is a divTOAST, CARseq, CellDMC and TCA have favourable performance among the tested methods, if the data does not contain many outliers.Methods designed for methylation data perform well also on RNAseq data.The most important factors affecting the accuracy of the estimated cell type-specific differentially expressed genes are (i) abundance of the cell type and (ii) individual heterogeneity between the samples.Investigating residuals can reveal outliers and whether the data is too heterogeneous for this type of analysis.SupplementaryText_bbab433Click here for additional data file.


One unifying challenge when classifying biological samples with mass spectrometry data is overcoming the obstacle of sample-to-sample variability so that differences between groups, such as between a healthy set and a disease set, can be identified. Similarly, when the same sample is re-analyzed under identical conditions, instrument signals can fluctuate by more than 10%. This signal inconsistency imposes difficulties in identifying subtle differences across a set of samples, and it weakens the mass spectrometrist’s ability to effectively leverage data in domains as diverse as proteomics, metabolomics, glycomics, and imaging. We selected challenging data sets in the fields of glycomics, mass spectrometry imaging, and bacterial typing to study the problem of within-group signal variability and adapted a 30 year old statistical approach to address the problem. The solution, “local-balanced model,” relies on using balanced subsets of training data to classify test samples. This analysis strategy was assessed on ESI-MS data of IgG-based glycopeptides and MALDI-MS imaging data of endogenous lipids, and MALDI-MS data of bacterial proteins. Two preliminary examples on non-mass spectrometry data sets are also included to show the potential generality of the method outside the field of MS analysis. We demonstrate that this approach is superior to simple normalization methods, generalizable to multiple mass spectrometry domains, and potentially appropriate in fields as diverse as physics and satellite imaging. In some cases, improvements in classification can be dramatic, with accuracy escalating from 60% with normalization alone to over 90% with the additional development described herein.  Because of the power and potential of machine learning, mass spectrometry experts are increasingly interested in using the tools in this field to analyze their data. For example, using DESI-MS data and various supervised machine learning techniques, researchers identified the ethnicity, age, and gender of individuals based on fingerprints . In anotIn addition to using existing machine learning methods, the field of mass spectrometry can benefit by identifying knowledge gaps and solutions where additional math tools could improve analyses. We recently designed a novel supervised machine learning tool, the Aristotle Classifier; this classifier was built with mass spectrometry data in mind, and its unique scoring algorithm has shown promise for various data analysis problems using mass spectrometry data, including: biomarker discovery , bacteriIn the work described herein, we seek to improve upon the capabilities of the Aristotle Classifier and address another data science problem that mass spectrometrists frequently face: How can one best classify samples with noisy data, where the variation due to the target difference is hard to detect? In mass spectrometry, the signal differences between two groups, a healthy group and a diseased group, for example, can be overshadowed by biological variability, variance in the instrument response, or unintended changes introduced by sample-preparation steps. To date the problem of signal variability, with respect to instrumentation, has been addressed most often by focusing on normalization methods. In depth studies of the best normalization practices for various sub-disciplines of mass spectrometry have been conducted, and guidelines are available for the fields of metabolomics , proteomThe local balanced model is most similar to the existing strategy of combining of a k-Nearest Neighbors (kNN) classifier with a second classifier, such as Support Vector Machine (SVM) or Naïve Bayes. The most relevant combination of kNN and SVM, where kNN is used to define the training data for a local SVM model, has been around since at least 2006 , 15. A mlocal-balanced model paradigm, the number of samples in each class are specified as a tuneable parameter. Therefore, a model may contain equal number of labeled samples from each class, even though the Euclidian distances for the most different sample in one class may be substantially farther away from the test sample than the least-nearest neighbor included from the other class. The local-balanced model concept, therefore, is unique from prior work in the field of machine learning, and the general adaptation of a local model is new in the field of mass spectrometry.The two innovations described herein, then, are to apply a local model to mass spectrometry data and to improve its classification accuracy by balancing the subsets used for training. While local model strategies are known inside the field of machine learning, and have been around for about 30 years, they have never been applied to mass spectrometry data. Furthermore, in all the examples we have studied where kNN and another classifier are combined, no effort is made to balance the number of samples from each class in the training group. Rather, the N nearest training samples, relative to the sample to be classified, are used. By contrast, in the We successfully applied the local- balanced model to five diverse data sets using two different classifiers. Three of the data sets encompass exclusively mass spectrometry data; the other two are non-mass spectrometry examples where variations due to instrument response are also potentially embedded in the output data. In almost all cases, using this approach does better than existing machine learning methods that do not include a local-balanced model. In some cases, the improvements are spectacular. We also show examples where using a local-balanced model does not substantially improve the accuracy of the base classifier. Including these counterexamples is meant to be instructive for readers, so they can better understand the circumstances in which a local balanced model can improve the overall accuracy of a given model. The local-balanced model, which was developed specifically for mass spectrometry data, but shown here to be applicable to multiple types of instrument-generated data, has the potential to offer scientists in diverse fields the ability to classify challenging data sets with highly enhanced accuracy.Unless otherwise stated below, the following procedures apply to all classifications conducted herein. All machine learning was done in R Studio, using R version 3.5.1. SVM was conducted using the e1071 package. The base Aristotle Classifier results were obtained using the previously published algorithm [When the local-balanced model, provided in ESM, is used for classification, certain input data are required. The data set to be classified, in the orientation required for the method being implemented, is input, along with the data set dimensions. The identities of the training samples for the two groups being classified (“Low” and “Hi”) need to be specified, along with the desired samples to be tested. Aside from this basic input information, the only parameters optimized in these studies were the sizes of the training sets for the local-balanced model. The training set size is a manually adjustable value that appears in the lines of code defining “LowTrain” and HiTrain”, approximately line 40. Naturally, researchers could choose to additionally optimize the models, by modifying the hyperparameters in SVM, for example. Since the goal of these studies was not to achieve the best-possible outcome, but rather to compare a standard set of conditions – with and without a local-balanced model – we did not adjust any hyperparameters beyond those specified herein.All data sets were processed so that the samples were in rows and the features were in columns. Before all SVM classifications, data sets were scaled using the embedded “scale” command in R unless otherwise noted. The test sample was never included as part of the training set: for base SVM classifications, a leave-one-out strategy was employed, and during testing with the local-balanced model, the test sample is automatically excluded from the training set in the code provided. The accuracy for all SVM classifications was determined without considering probabilities, as this method generally produced optimal results on these data sets. The assigned class identities (either Group 1 or Group 2) were compared to the correct class identity for each sample, and the overall accuracy was calculated by determining the percentage of the correctly assigned samples. The AUC (area under the receiver-operating curve) was determined using the probabilities for each sample; it was calculated using the package, pROC .All data sets were processed so that the features appeared in rows and the samples appeared in columns. For all calculations, the test samples were always excluded from the training set. The class identity was determined by the sign of the final Result, as described previously . When thLC-MS data of 42 IgG1 samples, where 21 samples had a native glycosylation profile and 21 samples had a modified profile, were available from a previous experiment , and theFor classification with the Aristotle Classifier (AC), the data set of  percents were used. When the local-balanced model was incorporated with AC, the training sets included seven samples from each class. For classification with SVM, the training sets included nine samples from each class; these values were determined through optimization.m/z 210.3 to m/z 994.5. The mass spectral data were acquired in the negative ion mode, and, based on the mass range and the ionization mode, the features are assumed to represent endogenous metabolites, mainly lipids, from cancer spheroids. The difference between the two classes is that one class represents pixels from a spheroid that had undergone chemotherapy treatment with cetuximab, while the samples from the other class originate from a spheroid that had not undergone treatment. More information about data acquisition can be found in reference 25.This data set is described in a prior report . It contFor classification of the MALDI-Imaging data, the MS signal of each sample had already been normalized so that the total ion counts for each sample summed to 1.0 . For theThe Satellite data set contains a total of 6435 samples and 36 features. In this case, the samples correspond to different geographic locations, and the features correspond to different light readings. The data were acquired from the UCI Machine Learning Repository . Six claFor the train/test experiment on the Satellite data set, the last 965 samples were reserved for a test set, leaving the first 5470 samples to be used for training. Classification was performed on the test data with both base models (SVM and Aristotle Classifier), using only the first 5470 samples in model training. Classification was also performed on the test data using both local-balanced models. During the train/test experiments, the group sizes were re-optimized since the size and composition of the training set was changed. As before, optimization proceeded by maximizing the AUC on the training data. The optimal group size for both SVM and the Aristotle Classifier were 10 samples for the minority class and 18 samples for the majority class.The “Hill-Valley” data set was also acquired from the UCI Machine Learning Repository . This daFor this data set, the data were scaled using the embedded function in R studio, and the scaled version was used for all classifications with both classifiers. For SVM, the local-balanced model performed optimally with 90 samples from each class; for AC, the optimal number of samples per class was 8. In addition to the classification on the original set of features, a second local-balanced model classification was performed using the Aristotle Classifier on a larger feature set. It included the original set of 100 features, along with all possible ratios of these features. The code to automatically generate the expanded feature set, with a non-redundant list of all the possible ratio pairs, is provided in ESM. The file is labeled: Ratio.Builder.txt. Due to the larger feature set size, k was set to 100 for this classification.For the train/test experiment on the Hill-Valley data set, the last 100 samples from each group were reserved for a test set, leaving the first 1012 samples to be used for training. Classification was performed on the test data with both base models (SVM and Aristotle Classifier), and both local-balanced models were also used to classify the test data. For the SVM classification, the data set containing 100 features per sample was used. For the Aristotle Classifier, the expanded feature set, of 5050 features per sample, was used. During the train/test experiments, the group sizes were re-optimized since the size and composition of the training set was changed. As before, optimization proceeded by maximizing the AUC on the training data. The optimal group size for SVM was 100 (for both groups). The optimal group size for the Aristotle Classifier was 8, also for both groups.The final data set studied herein is the ”MicroMass” data set. It has been described previously , and is In addition to the 571-sample training set, a separate test group is available for this data set. The test set contains an additional 360 samples; not all the test samples could be used in this study because some of them were mixtures of two different Gram types. Therefore, prior to analysis, the test group was partitioned so that all the samples in the group were either 100% Gram negative or 100% Gram positive: All test samples that were from a single genus/species were used along with all samples that contain a mixture of different bacteria from the same Gram type. Thus, a total of 248 test samples were used to evaluate model performance after all optimization was completed on the training data. Again, 130 training samples were used for each group in the local-balanced model.The data in Poor classification indeed results when these data are subjected to various classifiers . When the Aristotle Classifier is used to classify the samples, a disappointing result is obtained. The receiver-operating curve (ROC) is in Since our ultimate driver in studying this sample set was to use it as a testbed to improve upon the classification tools available for mass spectrometrists, we next set out to identify an analysis approach that would offer better classification accuracy than what had been achieved in After sample selection, a local-balanced model can be generated for each sample to be classified using any desired classifier. Two different classifiers were selected for the experiments herein: a Support Vector Machine (SVM) classifier was chosen because it is well known for its general utility for biomarker studies, where feature space is often greater than the number of collected samples . The AriIn applying the local-balanced model strategy, researchers must decide how many labeled samples should be included for model training. There is no one best answer to this question; optimal results vary from one data set to the next. The number of samples selected is thus a tuneable parameter. In the work described herein, we optimized the sample numbers to achieve an optimal area under the receiver-operating curve (AUC), as this is a well-recognized predictor of model performance . It woulNext, we applied the local-balanced model concept to other data sets, with the goal of learning more about its utility and general applicability. The second sample set contained MALDI imaging mass spectrometry data that had been recently classified using the Aristotle Classifier . This saAs shown in After these studies, we endeavored to determine the possibility of using of the local-balanced model concept to improve classification accuracy of data outside the domain of mass spectrometry. Could this novel implementation of the local model concept, using balanced numbers of samples in the training sets, provide a general benefit to the machine learning field compared to the approaches that are currently implemented for classification problems? To answer this question, we describe results from two example data sets that are difficult to classify accurately using conventional methods. The first example contains satellite imaging data, where the goal is to predict the presence of a particular feature based on a numerical rendering of the light readings received from the satellite . This prIn this case, the base SVM classifier  was unable to correctly assign a single target sample to the target class. Rather, the model that is generated is the dreaded “majority state classifier”, where every example gets assigned to the majority class. While one may note that this model results in an accuracy of 90%, since 90% of the species are in the majority class, such a model is completely useless, as it has no ability to discriminate between the target and non-target examples. When a local-balanced model is incorporated with SVM, much better performance is realized. Two thirds of the minority class examples are correctly predicted, and the AUC is substantially improved. The accuracy is also increased, although an increase in accuracy of about 5%, as shown in Interestingly, SVM with local-balanced model was not the best performing approach for classifying these data. The local-balanced model with the Aristotle Classifier had a slight edge. Its AUC was higher (0.967 vs 0.957); its G-mean was higher (0.83 vs 0.80) and its F-measure was higher (0.73 vs 0.72). While these results are better than either attempt with SVM, the data in While the local-balanced model was successful at improving classification performance for both the AC and SVM classifiers, one might like to know how these results compare to the large array of other possible ways of classifying these data. For example, decision tree-based methods are often used in cases where a large class imbalance exists , and theThe next data set in this study, “Hill-Valley,” is renowned for its classification difficulty. In this case, 1212 examples are available. For each example, a simulated instrument signal is generated that either goes up or down over the course of 100 measurements. Instrument response for 100 data points (100 features) are provided, which, when plotted along an x axis, either generate a “hill” or a “valley” because a graph of the feature values either have a recognizable peak or trough. An example of a “hill” and a “valley” from this data set is shown in and three features would be unwieldy: approximately 250,000 features. Therefore, we wrote a short script, provided in ESM, that accepts any input matrix and automatically generates a new output matrix that contains ratios of all the pairs of features in the original input matrix. Using this output matrix– containing the original 1212 samples, with each sample now containing about 5000 features – a local-balanced model was again built with the Aristotle Classifier. Since this new feature set addresses the relationships between the different features, it is useful for accurately solving this challenging classification problem. The AUC rockets up to 0.99 and the accuracy climbs to over 93%, as shown in The performance of the SVM classifier on this data set is quite similar to that of the Aristotle Classifier, with the caveat that the SVM classifier already accounts for the feature relationships, so an enlarged feature set containing ratios of the peaks is not needed. The base SVM method is not that effective at assigning the data, but incorporating a local-balanced model with SVM shows a dazzling performance enhancement. Accuracy jumps from 60% to almost 94% and the AUC rises from 0.80 to 0.98. This performance dramatically exceeds all benchmarks that we could identify in the published literature.The performance enhancements provided by the local-balanced model are dramatic in the last two data sets, and skeptics may wonder whether this is due to overfitting the data. To address this question, we changed the way the classification was done on the Satellite and Hill-Valley data sets. In the new experimental design (for both data sets), the model was built using only the first ~85% of the data, and the last ~15% of the data was reserved as a test set. The optimal group sizes were re-determined with the new training data, using a leave-one-out validation, and the optimized model was then applied to the test data. In both the Hill-Valley and the Satellite data sets, no drop in performance was seen between the train and test data with either classifier. Indeed, the AUC was >0.97 for both classifiers on both data sets. see . This exIn considering the experiments presented here in aggregate, one final question remains. Why did the local-balanced model offer significant improvements for the SVM classifier on the two non-mass spectrometry data sets, but little or no improvement on the two examples containing mass spectrometry data? We hypothesized that the answer to this question lie in the data sets selected for study. The first mass spectrometry data set was rather small, so removing training samples offered little benefit. In the second case using MS data, the base model already performed optimally, so no improvement could be made. By contrast, in both the two non-mass spectrometry data sets, many more training samples were present, and the base model performed poorly. We hypothesized that if a mass spectrometry data set with more training examples were studied, and the base SVM model had room for improvement, the local-balanced model may show a performance enhancement over a basic SVM model. To test this hypothesis, we studied one final dataset, “MicroMass”, which contains MALDI-MS data for 571 different bacterial samples.Does the local-balanced model offer a better classification on this data set? To answer this question, one must first define the classification problem to address. The data set contains MS abundances for 1300 peaks from a total of 20 different species of bacteria from 8 different genera. Furthermore, about 30% of the training samples are from Gram positive bacteria, while the remaining 70% are from Gram negative bacteria. Since the goal was to test the hypothesis that the local-balanced model would outperform the base SVM model when enough training samples (from both classes) are available, the classification problem that would best achieve that goal is determining whether each sample can be classified correctly as Gram negative bacteria vs. Gram positive.But was this increase in accuracy due to overtraining or due to a truly better model? Fortunately, this question is readily answerable with this data set because the investigators who collected the data also generated a second independent test set, which provided an additional 248 unseen samples to study. These samples had been generated using the same methods as the training set. Using the model built on the training data, the test samples were classified. The base model misclassified 17 samples, or about 7% of the 248 samples. The AUC for the test set, using the base classifier, was 0.964. This AUC is slightly lower than that for the base SVM model of the training data. The local-balanced model significantly outperformed the base model, misclassifying only 3 of the 248 samples. The error rate was about 1%, and the AUC for this test set was 0.9998. See . This exThis example also provides users with more information about the circumstances under which this approach offers a benefit: the method, which removes samples from the training set, is one that has a higher likelihood of outperforming the base model when the training set is adequately sized. “Adequately sized” likely varies from one case to another, but based on this principle, we expect that the approach may be more useful in MS-imaging applications, high-throughput (high sample) problems, like bacterial typing, and large-scale clinical studies; by contrast, pilot biomarker studies, where the sample sets often contain less than 50 samples, would probably benefit less from this approach, particularly when coupled with an SVM classifier.We studied the problem of inconsistent instrument response in mass spectrometry data and developed a useful solution: the local-balanced model. This method is a small tweak on an old machine learning strategy, the local model, although that strategy has never before been implemented in the mass spectrometry field. Using the local-balanced model, we observed an impressive improvement in supervised classification performance with two different classifiers and multiple data sets. The approach is simple, accomplished with just a few lines of code that are provided in ESM, and it adjusts for inconsistencies introduced by instrumental variation. We demonstrated the approach’s effectiveness on three examples in the field of mass spectrometry and two additional examples outside the field, where instrumental data are used for machine learning. In several cases, the incorporation of a local-balanced model resulted in classifications with AUC’s of >0.95. This performance exceeds existing benchmarks and offers mass spectrometrists and other analytical scientists a new tool to generate better predictions from their existing data.We fully concede that using a local-balanced model will not offer improvements in every classification problem; we demonstrated the point herein already. If the classifier is performing very well on its own, as was the case with SVM and the mass spectrometry imaging data, the local-balanced model will likely not make it better. Furthermore, if the data set already has a small number of samples in it, then subsetting the training samples may not be useful in generating an improved model. By contrast, the classification problems that will most benefit from incorporating a local-balanced model are those that are currently hard to solve: cases where experimental variation impacts samples differently across a sample set and cases where the characteristics defining a given class label are nonuniform. These types of problems are ideally suited for machine learning with a local-balanced model. Therefore, we expect this innovation to improve classification in multiple domains, including MS analysis for large clinical sample sets, imaging applications, and areas beyond these domains, to other scientific applications that rely on instrumental data, and more broadly, to any domain where training a model with a relatively homogenous, sub-population of samples would provide a superior answer to a classification question.1686935_Supp_Info41686935_Supp_Info31686935_Supp_Info21686935_Supp_Info1