Single-cell RNA sequencing (scRNA-seq) can be used to characterize variation in gene expression levels at high resolution. However, the sources of experimental noise in scRNA-seq are not yet well understood. We investigated the technical variation associated with sample processing using the single-cell Fluidigm C1 platform. To do so, we processed three C1 replicates from three human induced pluripotent stem cell (iPSC) lines. We added unique molecular identifiers (UMIs) to all samples, to account for amplification bias. We found that the major source of variation in the gene expression data was driven by genotype, but we also observed substantial variation between the technical replicates. We observed that the conversion of reads to molecules using the UMIs was impacted by both biological and technical variation, indicating that UMI counts are not an unbiased estimator of gene expression levels. Based on our results, we suggest a framework for effective scRNA-seq studies. Single-cell genomic technologies can be used to study the regulation of gene expression at unprecedented resolution13468Yet, there are a number of outstanding challenges that arose in parallel with the application of single cell technologyTo better account for technical variability in scRNA-seq experiments, it has become common to add spike-in RNA standards of known abundance to the endogenous samples11111214To address challenges related to the efficiency and uniformity with which mRNA molecules are amplified and sequenced in single cells, unique molecule identifiers (UMIs) were introduced to single cell sample processing16171871215There are a few common platforms in use for scRNA-seq. The automated C1 microfluidic platform (Fluidigm), while more expensive per sample, has been shown to confer several advantages over platforms that make use of droplets to capture single cells33To address this gap, we collected scRNA-seq data from induced pluripotent stem cell (iPSC) lines of three Yoruba individuals (abbreviation: YRI) using C1 microfluidic plates. Specifically, we performed three independent C1 collections per each individual to disentangle batch effects from the biological covariate of interest, which, in this case, is the difference between individuals. Both ERCC spike-in controls and UMIs were included in our sample processing. With these data, we were able to elucidate technical variability both within and between C1 batches and thus provide a deep characterization of cell-to-cell variation in gene expression levels across individuals.We collected single cell RNA-seq (scRNA-seq) data from three YRI iPSC lines using the Fluidigm C1 microfluidic system followed by sequencing. We added ERCC spike-in controls to each sample, and used 5-bp random sequence UMIs to allow for the direct quantification of mRNA molecule numbers. For each of the YRI lines, we performed three independent C1 collections; each replicate was accompanied by processing of a matching bulk sample using the same reagents. This study design  allows uIn what follows, we describe data as originating from different samples when we refer to data from distinct wells of each C1 collection. Generally, each sample corresponds to a single cell. In turn, we describe data as originating from different replicates when we refer to all samples from a given C1 collection, and from different individuals when we refer to data from all samples and replicates of a given genetically distinct iPSC line.We obtained an average of 6.3 +/− 2.1 million sequencing reads per sample (range 0.4–11.2 million reads). We processed the sequencing reads using a standard alignment approach (see Methods) and performed multiple quality control analyses. As a first step, we estimated the proportion of ERCC spike-in reads from each sample. We found that, across samples, sequencing reads from practically all samples of the second replicate of individual NA19098 included unusually high ERCC content compared to all other samples and replicates . We concWe next examined the assumption that data from each sample correspond to data from a single cell. After the cell sorting was complete, but before the processing of the samples, we performed visual inspection of the C1 microfluidic plates. Based on that visual inspection, we flagged 21 samples that did not contain any cell, and 54 samples that contained more than one cell . Visual inspection of the C1 microfluidic plate is an important quality control step, but it is not infallible. We therefore filtered data from the remaining samples based on the number of total mapped reads, the percentage of unmapped reads, the percentage of ERCC spike-in reads, and the number of genes detected . We chosOur final quality check focused on the different properties of sequencing read and molecule count data. We considered data from the 564 high quality samples and compared gene specific counts of sequencing read and molecules. We found that while gene-specific reads and molecule counts are exceptionally highly correlated when we considered the ERCC spike-in data  per biological condition. In that case, it is impossible to distinguish between biological and technical effects associated with the independent capturing and sequencing of each C1 replicate. We designed our study with multiple technical replicates per biological condition  in order to directly and explicitly estimate the batch effect associated with independent C1 preparations .P < 0.001, P < 0.001).As a first step in exploring batch effects, we examined the gene expression profiles across all single cells that passed our quality checks (as reported above) using raw molecule counts (without standardization). Using principal component analysis (PCA) for visualization, we observed – as expected - that the major source of variation in data from single cells is the individual origin of the sample. Specifically, we found that the proportion of variance due to individual was larger (median: 8%) than variance due to C1 batch . However, total ERCC molecule-counts are also quite different across individuals, when variation between batches is taken into account  to be consistent, regardless of the individual assignment. Indeed, we observed that total ERCC molecule-counts are significantly different between C1 batches . However, the proportion of variance explained by the individual was significantly larger than the variance due to C1 batch  and endogenous genes  across C1 replicates from the same individual. Moreover, the difference in read-to-molecule conversion efficiency across the three individuals was significant not only for endogenous genes , we reasoned that the spike-ins allow us to account for a subset of technical differences between samples, for example, those that arise from differences in RNA concentration (We next incorporated a standardization step by computing log transformed counts-per-million (cpm) to remove the effect of different sequencing depths, as is the common practice for the analysis of bulk RNA-seq data  and b. Wntration .Finally, to account for the technical batch effect, we modeled between-sample correlations in gene expression within C1 replicates (see Methods). Our approach is similar in principle to limma, which was initially developed for adjusting within-replicate correlations in microarray dataOnce we removed the unwanted technical variability, we focused on analyzing biological variation in gene expression between single cells. Our goal was to identify inter-individual differences in the amount of variation in gene expression levels across single cells, or in other words, to identify differences between individuals in the amount of regulatory noiseP < 10−5) in gene-specific dropout rates between individuals in more than 10%  of expressed endogenous genes. As expected, the expression levels, and the estimated variation in expression levels across cells, are both associated with gene-specific dropout rates  for each gene, by either including  or excluP < 0.0001, KLF4 and DPPA2  is significantly different between individuals. For the purpose of this analysis, we only included data from cells in which the gene was detected as expressed. Based on permutations , we clasnd DPPA2 .Our nested study design allowed us to explicitly estimate technical batch effects associated with single cell sample processing on the C1 platform. We found previously unreported technical sources of variation associated with the C1 sample processing and the use of UMIs, including the property of batch-specific read-to-molecule conversion efficiency. As we used a well-replicated nested study design, we were able to model, estimate, and account for the batch while maintaining individual differences in gene expression levels. We believe that our observations indicate that future studies should avoid confounding C1 batch and individual source of single cell samples. Instead, we recommend a balanced study design consisting of multiple individuals within a C1 plate and multiple C1 replicates μl of C1 Suspension Reagent was added to a 70-μl aliquot of ~17,500 cells. Five μl of this cell mix were loaded onto 10–17 μm C1 Single-Cell Auto Prep IFC microfluidic chip (Fluidigm), and the chip was then processed on a C1 instrument using the cell-loading script according to the manufacturer’s instructions. Using the standard staining script, the iPSCs were stained with StainAlive TRA-1-60 Antibody . The capture efficiency and TRA-1-60 staining were then inspected using the EVOS FL Cell Imaging System (Thermo Fisher) (Single cell loading and capture were performed following the Fluidigm protocol (PN 100-7168). Briefly, 30  Fisher) .® ERCC Spike-In Mix1 (Life Technologies) were added to the lysis buffer, and the template-switching RNA oligos which contain the UMI (5-bp random sequence) were included in the reverse transcription mix2035μl C1 Harvesting Reagent and quantified using the DNA High Sensitivity LabChip . The average yield of samples per C1 plate ranged from 1.26–1.88 ng per microliter .Immediately after imaging, reverse transcription and cDNA amplification were performed in the C1 system using the SMARTer PCR cDNA Synthesis kit (Clontech) and the Advantage 2 PCR kit (Clontech) according to the instructions in the Fluidigm user manual with minor changes to incorporate UMI labelingcroliter . A bulk For sequencing library preparation, tagmentation and isolation of 5′ fragments were performed according to the UMI protocolThe scRNA-seq libraries generated from the 96 single cell samples of each C1 chip were pooled and then sequenced in three lanes on an Illumina HiSeq 2500 instrument using the PCR primer (C1-P1-PCR-2: Bio-GAATGATACGGCGACCACCGAT) as the read 1 primer and the Tn5 adapter (C1-Tn5-U: PHO-CTGTCTCTTATACACATCTGACGC) as the index read primer following the UMI protocolThe master mixes, one mix with all the bulk samples and nine mixes corresponding to the three replicates for the three individuals, were sequenced across four flowcells using a design aimed to minimize the introduction of technical batch effects . Single-http://www.bioinformatics.babraham.ac.uk/projects/fastqc) and observed a decrease in base quality at the 3′ end of the reads. Thus we removed low quality bases from the 3′ end using sickle with default settingsTo assess read quality, we ran FastQC  with at least one observed read. Using the high quality single cells, we further removed genes with low expression levels for downstream analyses. We removed all genes with a mean log batches . We also® ERCC Spike-In Mix1 (Life Technologies) in our setup. Therefore, our estimation of the total spiked-in molecule number was 16,831 per sample. Since the relative concentrations of the ERCC genes were provided by the manufacturer, we were able to calculate the molecule number of each ERCC gene added to each sample. We observed that the levels of ERCC spike-ins strongly correlated with the input quantities  and fit a linear mixed model with the fixed effect of individual αi and the random effect of batch bij:Individual effect and batch effect between the single cell samples were evaluated in a series of analyses that examine the potential sources of technical variation on gene expression measurements. These analyses took into consideration that in our study design, sources of variation between single cell samples naturally fall into a hierarchy of individuals and C1 batches. In these sample-level analyses, the variation introduced at both the individual-level and the batch-level was modeled in a nested framework that allows random noise between C1 batches within individuals. Specifically, for each cell sample in individual bij of batch follows a normal distribution with mean zero and variance εijk describes residual variation in the sample measurement. To test the statistical significance of individual effect , we performed a likelihood ratio test (LRT) to compare the above full model and the reduced model that excludes α1. To test if there was a batch effect  was modeled as a function of individual effect and batch effect, separately for the ERCC spike-ins and for the endogenous genes.Total molecule count  and the independent variables were individual and batch, both modeled as random effects.To assess the relative contributions of individual and technical variation, we analyzed per-gene expression profiles and computed variance component estimates for the effects of individual and C1 batch . The goaThe variance parameters of individual effect and batch effect were estimated using a maximum penalized likelihood approachThe estimated variance components were used to compute the sum of squared deviations for individual and batch effects. The proportion of variance due to each effect is equal to the relative contribution of the sum of squared deviations for each effect compared to the total sum of squared deviations per gene. Finally, we compared the estimated proportions of variance due to the individual effect and the batch effect, across genes, using a non-parametric one-way analysis of variance .et al.et al.2 counts per million (cpm). This standardization was performed using only the endogenous gene molecules and not the ERCC molecules. Third, we corrected for cell-to-cell technical noise using the ERCC spike-in controls. For each single cell, we fit a Poisson generalized linear model (GLM) with the log2 expected ERCC molecule counts as the independent variable, and the observed ERCC molecule counts as the dependent variable, using the standard log link function. Next we used the slope and intercept of the Poisson GLM regression line to transform the log2 cpm for the endogenous genes in that cell. This is analogous to the standard curves used for qPCR measurements, but taking into account that lower concentration ERCC genes will have higher variance from Poisson sampling. Fourth, we removed technical noise between the eight batches (three replicates each for NA19101 and NA19239 and two replicates for NA19098). We fit a linear mixed model with a fixed effect for individual and a random effect for the eight batches and removed the variation captured by the random effect (see the next section for a detailed explanation).We transformed the single cell molecule counts in multiple steps . First, 2 cpm.For the bulk samples, we used read counts even though the reads contained UMIs. Because these samples contained RNA molecules from ~10,000 cells, we could not assume that the 1,024 UMIs were sufficient for tagging such a large number of molecules. We standardized the read counts to log2 gene expression levels for cell-to-cell correlation within each C1 plate. The algorithm mimics a method that was initially developed for adjusting within-replicate correlation in microarray datag, cells that belong to the same batch j are correlated, for batches j = 1, …, 8. We also assume that the cell-to-cell gene expression variation due to C1 batch effect is independent of biological variation between individuals.Our last normalization step adjusted the transformed logg that includes a fixed effect of individual and a random effect for within-batch variation attributed to cell-to-cell correlation in each C1 plate:We fit a linear mixed model for each gene yg,ijk denotes log2 counts-per-million (cpm) of gene g in individual i, replicate j, and cell k; i = NA19098, NA19101, NA19239, j = 1, …, ni with ni the number of replicates in individual i, k = 1, …, nij with nij the number of cells in individual i replicate j. μg denotes the mean gene expression level across cells, αg,i quantifies the individual effect on mean gene expression, bg,ij models the replicate effect on mean expression level  as a function of mean RNA abundance across C1 replicates, we found that the CV of molecule counts among endogenous genes and ERCC spike-in controls suggested similar expression variability patterns. Both endogenous and ERCC spike-in control CV patterns approximately followed an over-dispersed Poisson distribution , which i1146g:To identify differential noise genes across individuals, we computed median absolute deviation (MAD) - a robust and distribution-free dissimilarity measure for gene MADg suggest a large deviation from the median of the adjusted CV values. We identified genes with significant inter-individual differences using a permutation-based approach. Specifically, for each gene, we computed empirical P-values based on 300,000 permutations. In each permutation, the sample of origin labels were shuffled between cells. Because the number of permutations in our analysis was smaller than the maximum possible number of permutations, we computed the empirical P-values as b is the number of permuted MAD values greater than the observed MAD value, and m is the number of permutations. Adding 1 to b avoided an empirical P-value of zeroLarge values of We used ConsensusPATHDBhttps://github.com/statgen/verifyBamID) developed by Jun et al.We were able to successfully determine the correct identity of each single cell sample by examining the SNPs present in their RNA sequencing reads. Specifically, we used the method verifyBamID (http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE77288). The code and processed data are available at https://github.com/jdblischak/singleCellSeq. The results of our analyses are viewable at https://jdblischak.github.io/singleCellSeq/analysis.The data have been deposited in NCBI’s Gene Expression OmnibusHow to cite this article: Tung, P.-Y. et al. Batch effects and the effective design of single-cell gene expression studies. Sci. Rep.7, 39921; doi: 10.1038/srep39921 (2017).Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


High-Throughput technologies provide genomic and trascriptomic data that are suitable for biomarker detection for classification purposes. However, the high dimension of the output of such technologies and the characteristics of the data sets analysed represent an issue for the classification task. Here we present a new feature selection method based on three steps to detect class-specific biomarkers in case of high-dimensional data sets. The first step detects the differentially expressed genes according to the experimental conditions tested in the experimental design, the second step filters out the features with low discriminative power and the third step detects the class-specific features and defines the final biomarker as the union of the class-specific features.http://github.com/mbeccuti/PGS.The proposed procedure is tested on two microarray datasets, one characterized by a strong imbalance between the size of classes and the other one where the size of classes is perfectly balanced. We show that, using the proposed feature selection procedure, the classification performances of a Support Vector Machine on the imbalanced data set reach a 82% whereas other methods do not exceed 73%. Furthermore, in case of perfectly balanced dataset, the classification performances are comparable with other methods. Finally, the Gene Ontology enrichments performed on the signatures selected with the proposed pipeline, confirm the biological relevance of our methodology. The download of the package with the implementation of Peculiar Genes Selection, ‘PGS’, is available for R users at:  The high potential of HT technologies lies in the quantity of information obtained by one experiment which may increase the possibility of discovering unknown mechanisms underpinning the differences in the biological conditions of interest. For this reason, one of the goals of HT data analysis is the detection of biomarkers for purposes –3. Despilarge p small n problem, where p indicates the number of available predictors—i.e. the thousands of genes assessed—and n indicates the number of conditions tested in the experiment—i.e. the number of samples. In the last years, several machine-learning approaches have been proposed to deal with the risk of overfitting deriving from the large p small n problem, leading to a vaste literature about classification methods and features selection/extraction [The problem of high-dimensionality of the output is known as the traction –8.signature.Features selection and features extraction methods reduce the data dimensionality by removing noise and non-informative data and by storing the information needed for the classification purpose in a subset of features called feature selection methods shrink the information related to sample classification in the (sub-)optimal subset by removing irrelevant or redundant variables without altering the original representation of the features. Feature extraction methods creates new predictors as combination of the features [The difference of the two approaches relies in how the signature is created from original data. Indeed, features .Once the signature is selected, a classification method is applied to test it. Even though both feature extraction and feature selection are good solutions for the large p small n problem, feature selection allows extraction of meaningful biological rules from the classifier without altering the original characteristics of the selected features and it is less computationally expensive to perform .Another important aspect to consider in classification tasks is the possibility that the data set analyzed is characterized by imbalance among the size of classes. The issue of class imbalance and how to derive good biomarkers from such datasets has received a great deal of interest from the research community and has been the focus of several recent studies –13. ReceThe imbalance problem frequently occurs in the new discipline of systems vaccinology , 16 wherHT studies are commonly used also in cancer-related researches where the capability to distinguish between cancerous and noncancerous tissues or to classify different type of cancer is indeed an inestimable help in medical and biological domain.Microarrays are a commonly used HT technology which measure the expression levels of large numbers of genes simultaneously or to genotype multiple regions of a genome extracted from a relatively small human samples. The necessity of tools to extract significant information from microarray experiments leads to a vaste amount of softwares and packages available –19. HowePeculiar Genes Selection (PGS), to identify predictive biomarkers that are robust to class imbalance and improve the Support Vector Machine (SVM) classification performances in case of imbalanced data sets. The PGS procedure is also applied to a balanced dataset to confirm that it perform well also when the size of the classes is the same. The biomarkers generated by this procedures are therefore a suitable compromise between maximum overall accuracy and correct classification within the underrepresented class.In this study we present a new features selection method, called The novelty of PGS relies on the simple and fast computation of a binary matrix, created by fitting a logistic regression model using single gene expression as predictor of the class label of samples. Grids of parameters for the SVM  were explored to maximize one of the three metrics related to classification tasks: overall accuracy, specificity and sensitivity. Since these three metrics are strongly inversely related, the parameters maximizing one do not maximize the other two.We then compare the classification results obtained on two public microarray data sets with those obtained using one of the available packages for microarray data analysis: the Classification for MicroArrays package, CMA, available under the Bioconductor distribution for R software  and withWe applied our pipeline on two public sets of data available at NCBI GEO database: a vaccination-related, (GEO accession code: GSE48024) , and a cThis dataset is the result of two studies conducted on two different cohorts of patients. The first cohort, consisting of 119 adult male subjects vaccinated with the 2008–2009 inactivated trivalent influenza vaccine . A second cohort, included 128 adult females that received the 2009–2010 trivalent influenza vaccine .−) and after vaccination on days 1, 3, and 14 whereas the antibody titers measurements were available before vaccination and on days 14 and 28 after vaccination.i, i ∈ { H1N1, H3N2, FluB } was detected as the highest baseline level at which a subject was able to reach a 4 fold-increase in his antibody titers.Since the proposed pipeline is specific for binary classification problems, the selected subjects were classified into two classes: ‘High responders’ and ‘Low responders’. The classification is usually based on the 4 fold-increase of the antibody titers following the vaccination. However, because of the well known baseline effect , 24, conThe cancer-related data set consists in 60 paired samples of tumor and adjacent normal lung tissue coming from a study conducted in Taiwan on non-smokers females aged 50 to 70 years old. Three tumors are represented in the data set: Adenocarcinoma, Bronchioloalveolar and Squamous carcinoma .The proposed methodology is a new feature selection procedure called Peculiar Genes Selection (PGS), that detects genes characterizing the class of the samples they belong to. The classification accuracy performances of the gene signature derived from the application of PGS are evaluated using a SVM classifier exploring grids of parameters to maximize the desired metric: overall accuracy and specificity.In the proposed pipeline the feature selection is performed in three steps:Identification of Differentially Expressed Genes (DEGs)Identification of good predictorsSelection of the peculiar good predictors for each classStep 1 allows the detection of J differentially expressed genes under two conditions of interest leading to a significative data dimensionality reduction [eduction .Step 2 is based on the computation of a regression model in which single variable levels are used to predict the class label of each subject [ subject .n + m = N, n being the number of subjects belonging to class 0 (‘C0’) and m the number of subjects belonging to class 1(‘C1’). Let also J be the number of DEGs detected in step 1. We indicate with Xj = {xj1, …, xjN}, j ∈ {1, …, J} the expression of the j-th DEG across all subjects and with Let N be the number of subjects with Xj. pi is the predicted probability of success for subject i, β0 the intercept of the model, βj the fitted parameter and Xji the expression of the j-th gene of subject i.PGS computes J logistic regressions to predict the probability of each subject to be a success, in other words to belong to class ‘1’, given the expression of the j-th independent variable p of predicted probabilities of ‘success’, where each component is the pi calculated in τ stands for pre-selected threshold value that can varies in case of strong imbalance between classes size.The logistic-regression fit lead to J N-dimensional vectors predictive power (pp) and it is defined as follows:The comparison of pp form a distribution of predictive power values, describing the ability of the DEGs of classifying the samples. The P, with P ≤ J, good predictors are chosen among the DEGs which pp >= q, with q being a quantile of the predictive powers distribution, describing the minimal number of subjects a DEG needs to correctly classify to be considered a good predictor.The J values of Step 3 consists in the analysis of the binary matrix P binary vectors i by each predictor.pp = 1, meaning that n columns filled with 0 and the remaining m columns filled with 1.In an unrealistic situation where all the DEGs have n column of n samples belonging to class 0. It detects the features that assign the correct class label to the majority of the samples and simultaneously detects the most mislclassified samples. In fact, the basic idea is that the peculiar genes are those capable to correctly classify a subject when the majority of genes misclassifies them.To select the peculiar predictors for ‘C0’, denoted by n columns of Kk, k ∈ {0, 1} indicates the chosen quantile of the misclassification distribution to detect the most misclassified subjects in the two classes:The criterion used to identify the most misclassified subjects is based on the analysis of the peculiar genes are now easily detectable as the genes  as classifier . SVMs arfeature space where the classes are linearly separable by using the ‘kernel’ technique [SVMs separate a given set of binary labeled training data finding the equation of the hyperplane that maximizes the distance between the two classes. In case of noisy/sparse data the linear separation of the two classes is not always possible in the input space. In this case SVMs can perform a non-linear mapping of data in a so called echnique .n labeled data points: S = {, …, }, where yi ∈ {0, 1} and let xi → ϕ(xi) by the relation K = ϕ(xi) ⋅ ϕ(xj). Classical choices for kernel functions areσ, d, a, r are kernel parameters to be tuned.Let S be a sample of c to account penalties for misclassification. This pipeline uses SVM as classifier and explores a grid of parameters in order to detect the setting which allows the best classification performances on the selected metric.SVMs give the possibility to chose a constant ktraining and validation sets [k-fold cross-validation and usually uses the 80% of the original data for training the model and the 20% of the remaining samples to check the accuracy of the model on new inputs [In order to assess the accuracy of the model, it is common practice to split the original data set into ion sets , 36. Thiw inputs . All theTo evaluate the proposed approach, we applied the PGS on the two public microarray data sets described in Materials and Methods and we compared the classification performances to those obtained using MRMD software and the CMA package.− and day 28 against at least one of the three antigens were labeled as ‘high responders’while the others were labeled as ‘low responders’.The proposed pipeline is specific for binary classification tasks. As described in Materials and Methods section, the binary classification of the samples was based on the 4-fold increase in the antibody titers levels: subjects reaching a 4-fold increase between day 1T such that Ti > 256, i ∈ { H1N1, H3N2, FluB } at the baseline, never exceed a 4-fold increase. Therefore, to avoid misclassification due to pre-existing immunity, all subjects with a baseline titer Ti > 256 were excluded from subsequent analysis.According to the classification criterion, 49 of the remaining samples were labeled as low responders and 144 as high responders. With such a relatively small sample size, it appears evident the imbalance between the two class sizes: the number of high responders is around three times the number of low responders. This situation is not surprising: the vaccination is expected to elicit a good response in the majority of the people.Step 1), was performed using limma package [PGS, as described in Materials and Methods, consists of three steps. The identification of DEGs, , we applied the logistic regression model using fold-change of the gene expression between day 0 and day 1 as independent variable. To predict the class label of each subject we then applied the criterium presented in τ = 0.6. The fit of the model allowed us to compute, for each gene, the proportion of subjects correctly classified, which is referred to as predictive power (pp). We defined good predictors the genes whose pp belonged to the 5th percentile of the pp distribution of all the DEGs as showed in For the detection of the peculiar genes selection (Step 3), required the analysis of the binary matrix obtained in Step 2. Computational experiments showed that by setting K0 = 165 and K1 = 20, we identified 2 highly frequently misclassified ‘low responders’and 5 frequently misclassified ‘high responders’. The algorithm detected 11 peculiar genes belonging to The S, performed with Erichr [S, confirming the biological relevance of PGS.Tables h Erichr , 39, fouTo conclude the comparison of the feature selection procedures, we compared the lists of genes belonging to the different signatures. To further investigate the properties of the genes shared among the different sigantures, we detected 43 genes selected by at least 5 different feature selection procedures. The GO perfermed on the list of the most selected genes showed results reported in We performed classification experiments using the feature selected as described in the previous paragraph to build a classification model with SVMs using a 10-fold cross-validation. i.e. of the form Kij = d, where d = 5 is the degree and c0 the initial coefficient. The associated cost is 0.1, where the cost parameter for an SVM represents the tolerance of misclassification within each training example. When the parameter is small it means that the SVM looked for a larger-margin separating hyperplane allowing misclassification. To maximize the overall accuracy, the weight vector, the parameter accounting for the imbalance in the SVM has equal components: the same weight is given to both classes. Interestingly the situation is completely different when we want to detect all the subjects belonging to the underrepresented class: in this case, the imbalance in the classes size is reflected by the weight vector that shows the same strong imbalance between its two components.The 10 fold-cross validation showed that the model reached its best performance maximizing the overall accuracy when the kernel of the SVM is a polynomial; -In Cancer data set description we pointed out that this data set is composed by paired-samples: each subject indeed, provided two samples of tissue, one normal and one tumor. This important remark has two main consequences on the subsequent analysis:no risk of misclassification in labeling the samples with 0 or 1perfect balance between the size of the two classes, with 60 samples in each.Step 1) we used the limma setting as contrast the class labels of the samples. We detect 10901 DEGs starting from a chip containing 21655 genes. To detect the good predictors, as described in Step 2, we applied the logistic regression model using the gene expression of each sample to predict its class. For this dataset the τ used to predict the class labels of the samples was τ = 0.5. We obtained the pp distribution and, we defined good predictors the genes whose pp belonged to the 99th percentile of the pp distribution of all the DEGs as showed in To detect DEGs in this case study (Step 3 of our procedure) required the analysis of the binary matrix obtained in Step 2. Computational experiments showed that by setting K0 = 10 and K1 = 80 this analysis led to the detection of 2 highly frequently misclassified ‘low responders’ and 6 highly frequently misclassified ‘high responders’. The algorithm, found 4 peculiar genes belonging to peculiar genes belonging to For this data set, the peculiar gene selection . The results of GO performed on the oncogenic database are showed in i.e.w0 = w1 = 1).We applied PGS method to the cancer data set and compared the results with those obtained by using the CMA package with all the feature selection methods available. For this data set no parameter tuning was necessary: the optimal results were obtained by using the default settings of the SVM classifier, choosing a polynomial kernel: the default degree is 3, the initial coefficient is 1. The optimal cost here is, again, 0.1. The perfect balance between the two classes size lead to the logical choice of a weight vector whose components are equal Click here for additional data file.S2 File(XLSX)Click here for additional data file.S3 File(XLSX)Click here for additional data file.


In the present study, we introduce a novel semi-supervised method called the semi-supervised maximum discriminative local margin (semiMM) for gene selection in expression data. The semiMM is a “filter” approach that exploits local structure, variance, and mutual information. We first constructed a local nearest neighbour graph and divided this information into within-class and between-class local nearest neighbour graphs by weighing the edge between the two data points. The semiMM aims to discover the most discriminative features for classification via maximizing the local margin between the within-class and between-class data, the variance of all data, and the mutual information of features with class labels. Experiments on five publicly available gene expression datasets revealed the effectiveness of the proposed method compared to three state-of-the-art feature selection algorithms. Only a few genes can cause disease4. The gene expression levels of these disease-causing genes greatly vary between positive and negative samples6. Therefore, the classification of tumour tissue or other diseases by analysing differential expression data and identifying disease-causing genes is attractive and practically meaningful9. Curse-of-dimensionality may occur during the classification phase due to a property of gene expression data that states that a small sample size has high dimensionality10. Various dimension-reduction methods have been developed to avoid this phenomenon.Currently, the expression level of hundreds of thousands of genes can be successfully monitored with popular high-throughput technology in a single experiment. This technology is widely used in the post-genomic era for related disease research12, such as variance criteria13, maximum local margin criteria, mutual information criteria16, and correlation criteria17. Feature selection methods contain wrapper18 and filter methods21. Compared to wrapper methods, filter methods are efficient and simple due to their classifier-independent feature selection. In addition, manually labelling a positive or negative sample is both time- and labour-consuming. Thus, gene expression data lack labelled samples but have abundant unlabelled samples. Current studies have attempt to uncover the most discriminative information from all samples. Although some supervised and unsupervised feature selection methods can perform well, utilizing the information of both labelled and unlabelled data can enhance their performance24. This point was verified in a series of different environmental settings of the Fisher criterion28. These three studies showed that the local manifold structure is useful for selecting more informative genes, and the discriminative power can be increased in a local semi-supervised manner.Feature selection is a dimension-reduction technique that evaluates features using proper optimization criteria29, Laplacian score (LS)30, and mutual information technique, we proposed a novel semi-supervised gene selection method called the semi-supervised maximum discriminative local margin (semiMM). The semiMM can be used for tumour classification or analysis of differential gene expression levels. This method aims to maximize the local margin between within-class and between-class data and simultaneously discover the most closely related class features. The features were evaluated according to their contribution to the local margin that preserves power and class discriminative capability. Specifically, the maximum local margin is designed to maintain the consistency in local geometrical structure of the same class and the separability of different classes. In maximizing the mutual information between classes and features, the relationship between class labels and features is considered to achieve increased discriminative power.Motivated by the maximum margin projection (MMP)The present review is structured as follows. Work-related dimensionality reduction methods are briefly reviewed in Section 2. The proposed semiMM algorithm is introduced in Section 3. Experiments on five publicly available gene expression datasets are presented in Section 4. Finally, the conclusions from the present study and suggestions for future work are discussed in Section 5.30 and MMP29, which are related to the proposed semiMM.In this section, we present a brief review of two dimensionality reduction methods, namely, the LSm denotes the number of samples, and n denotes the number of genes, which is the dimensionality number. n dimensional column vector that denotes the rth gene in the gene expression data, where fri indicates the rth gene in the ith sample. The matrix is presented by boldface and capital letters, whereas the vectors are denoted by boldface and lowercase letters.In the present study, matrix 31. The former indicates that if two points are close to each other in a high-density region, then the corresponding projecting outputs should also be close. The latter assumes that the points in the same cluster tend to be in the same class. MMP obeys these two rules and aims to capture both the geometrical and discriminating structures of the local data manifold with both labelled and unlabelled data.The MMP is a semi-supervised learning method for dimensionality reduction. This semi-supervised learning method has two assumptions: smoothness and clusterG with a binary weight to depict the geometry of the underlying local manifold. G is divided into two subgraphs, that is, the within-class graph Gw and between-class graph Gb, to discover the discriminating information of the data manifold. N(xi) denotes the k nearest neighbours of arbitrary data point xi and is naturally composed of Nb(xi) and Nw(xi). If the samples are neighbours and have different class labels, then they belong to set Nb(xi); otherwise, the remaining neighbours are placed into Nw(xi). Wb and Ww are the weight matrices of Gb and Gw, respectively, with the following definitions:The MMP constructs a k nearest neighbour graph 32.Semi-supervised graph embedding is similar to locality sensitive discriminant analysis (LSDF), a semi-supervised feature selection algorithm proposed inGw and between-class graph Gb:a is a projection vector of projection matrix A and A ∈ Rd×n. By performing some algebraic steps and imposing a constraint, aTXDwXTa = 1, the objective functions ,,Gw and bEquivalent toγ is the generalized eigenvalue. This linear transformation can optimally and simultaneously preserve the local neighbourhood and discriminatory information.Thus, the optimization problem is:30. This method was developed due to the observation that two data points close to each other are potentially in the same class. The LS selects features with more locality preserving power as evaluated by Eq. . Thus, the semiMM constructs between-class and within-class neighbour graphs to simultaneously characterize the local manifold of the dataset with all samples and the discriminative information from the labelled samples. Moreover, the semiMM also considers the variance of features and the mutual information between the classes and features. This method aims to maximize the local margin between within-class and between-class data and simultaneously discover the most related class features.Based on the two basic assumptions about semi-supervised learning mentioned in Section 2, two data points from the same neighbourhood potentially belong to the same class (and vice versa) with the name of the local preserving power. A “good” feature possesses more local preserving power and is most discriminative in clarifying the data.NMI. A larger difference indicates more mutual information and vice versa. Maximizing the mutual information between features and class labels might enhance the discriminative capability. A reasonable criterion of the semiMM is to minimize the objective function given as follows:semiMMr denotes the score of the λth feature evaluated by the proposed semiMM.Therefore, the within-class and between-class information should be simultaneously minimized and maximized, respectively, to ensure a maximum local margin. In addition, a good feature for gene selection should be genes differentially expressed for samples with different class labels. This difference in gene expression level can be characterized by the mutual information between features and class labels, denoted by S = Ww − Wb, the objective function can be rewritten as Eq.  and FL (1) . The dataset contains a total of 77 samples, the DLBCL and FL sample ratio is 58:19, and the total number of genes is 5469.Prostate_Tumor: This dataset is also a two-class dataset; the two sub-categories are tumour samples, Tumour (0), and normal samples, Normal (1). The dataset contains a total of 102 samples, the Tumour and Normal sample ratio is 50:52, and the total number of genes is 10,509.Leukemia2: This dataset is a three-class dataset, and the three subclasses are AML (0), ALL (1) and MLL (2). The dataset contains 72 samples, and the total number of genes is 11,225.SRBCT: This dataset is a four-class dataset, and the four subclasses are EWS (0), RMS (1), BL (2), and NB (3). The dataset contains a total of 83 samples, including EWS, RMS, BL and NB at a sample ratio of 29:25:11:18, and the total number of genes is 2308.Lung_Cancer: This dataset is a five-class dataset, with four subclasses, including Adeno (0), Normal (1), Squamous (2), COID (3) and SMCL (4). This dataset contains 203 samples, and the total number of genes is 12,600.The methods presented in this article were evaluated using five tumour datasets and compared to other methods. The following is a brief introduction to the five datasets used in this article.In this experiment, we first pre-processed the five gene expression datasets to obtain the prepared data: initial data for feature selection and split data for classification. The optimal values of parameters were selected in the proposed method. Three state-of-the-art feature selection methods were selected for comparison to better understand the proposed method. The experiments were conducted, and the outputs were recorded and analysed.In this experiment, we set up a semi-supervised setting to simulate the “small sample, high dimension” problem. In a semi-supervised setting for a filtered feature selection, both labelled and unlabelled samples must be used during the calculation of the score of each feature, and the feature selection method is used to rank the features. Here, we selected different numbers of samples per subclass of a gene expression dataset, in which the labelled data with stratified random sampling is denoted by L. The values of L are 2, 4, and 6. Thus, the number of labelled samples in a certain gene expression dataset is the product of L and the number of classes. The remaining data in the dataset are regarded as unlabelled data. The obtained data were termed initial data for convenience.During classification, we divided each gene expression dataset into a training and testing set with a ratio of 6:4 through stratified random sampling. We conducted the classification with different numbers of genes ranging from 5 to 300 with a step of 5. Considering the intrinsic characteristics of semi-supervised feature selection, we repeated the experiment 10 times at each step and recorded the average prediction accuracy for evaluation.37. The LS selects the features with more locality preserving power25.The LS is an unsupervised feature selection method. In this method, a nearest neighbour graph is constructed to model the local geometric structure32.As a supervised feature selection method, the Fisher score seeks features according to their discriminating powerAs a semi-supervised feature selection algorithm, the LSDF utilizes both labelled and unlabelled data and determines the discriminative structure and geometrical structure of the data. Features that can maximize the margin between the within-class and between-class graphs are selected by the LSDF.In the proposed semiMM, the tuning parameter lambda can be searched from the grid {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. The number of nearest neighbours, k, is empirically assumed to be 5 because the k nearest neighbours are adopted to model the local manifold structure of the data. The weight in the whole experiment is determined by binary similarity, and the alpha is set as 100, similar to LSDF. By conducting many experiments to select the proper value of parameters in the proposed algorithm, we determined that the proposed method can robustly detect changes of the parameters, whereas the LSDF is sensitive to k and alpha. Thus, k and alpha are set at 5 and 100, respectively, to ensure that the LSDF can still perform well in the experiments, and a better comparison between LSDF and the proposed semiMM can be obtained. In this experiment, the top 300 genes were selected as the feature subset for classification, and each gene was normalized to achieve zero mean and unit variance for further assessment.32, a larger AUC indicates better performance. Thus, the AUC score is applied to assess the prediction performance of classification to properly evaluate FPs and FNs for cancer classification.In this evaluation framework, five evaluation metrics, including accuracy, precision, recall, f-score, and area under the receiver operating characteristic curve (AUC), were used to assess the performance. These metrics were determined by the following equations:The proposed semiMM can manage both binary classification and multi-classification datasets. The gene expression datasets used in the present study include two binary classification datasets and three multi-classification datasets. To perform the multi-classification experiment, we devised a one-against-rest approach for each class and thus constructed c binary classifiers, where c denotes the number of classes in each dataset. The average results over the c binary classifiers are shown as the final result of multi-classification.In this subsection, classification is performed via SVM on the training set with a chosen feature subset (the top 300 genes) in the five gene expression datasets to evaluate the performance of the proposed semiMM method and compare it with three other methods. Figure All filter methods achieve high average prediction accuracy with an increased number of selected genes in most cases. Figures The semiMM method performs best and converges fastest to the optimal point when less than 100 genes are selected. This finding might indicate that the proposed semiMM method has better ability to utilize the label information than LSDF, i.e., the semiMM has more discriminating power than the LSDF method. Roughly speaking, the semiMM and LSDF show stable performance with varying values of L because the shapes of their curves are almost unchanged. This finding can be explained by the semi-supervised properties of the semiMM and LSDF; these methods simultaneously select features from both labelled and unlabelled samples.The multiclass classification performance of three publicly available datasets is shown in Figs Therefore, the semiMM performs well irrespective of the dataset itself, whereas its competitors are sensitive to the dataset. The semiMM is effective for tackling “small sample” problems. The good and stable performance of this method is due to its simple and efficient idea to discover both geometrical and discriminating information with labelled and unlabelled samples together. Although no method outperforms the other three algorithms in all circumstances, with regard to robustness of the dataset and good prediction accuracy, and the proposed semiMM is a good choice for gene selection with small and limited numbers of labelled samples.Considering that all four methods show a stable and promising performance when the number of selected genes is 150, we listed the corresponding classification results with different values of L in Tables In the present study, we introduced a novel semi-supervised method called the semiMM that is based on spectral graph and mutual information theories and is used for gene selection. The semiMM method is a “filter” approach that simultaneously exploits local structure, variance, and mutual information. In the first step, we constructed a local nearest neighbour graph and subsequently divided this information into within-class and between-class local nearest neighbour graphs by weighing the edge between two data points. This method aims to discover the most discriminative features for classification by maximizing the local margin between within-class and between-class data, the variance of all data, and the mutual information of features with class labels.In contrast to three state-of-the-art methods, i.e., the Fisher score, LS, and LSDF methods, the experimental results show that the semiMM method perfectly balances the use of both labelled and unlabelled samples. Regardless of whether the dataset is binary-class or multiclass, the proposed semiMM can always achieve a good performance. The performance of the semiMM is comparable to that of the Fisher score and even outperforms the Fisher score when the number of labelled samples equals 2, and the number of selected genes is less than 50. Both the Fisher score and semiMM are superior to the LS and LSDF in most cases.The following issues should be addressed in future research:No theoretical selection is established for the controlling parameter lambda, which tunes the weight between the first and second terms of the present criterion.The semiMM considers only the discriminating information of class labels as features and between-labels. If this method can delete these redundant features, then a compact feature subset that is maximally discriminative and minimally redundant can be obtained.The second term, which is the mutual information between class label and features, can be time-consuming when dealing with datasets with many subclasses. This factor makes the proposed semiMM method less competitive for multi-classification problems with limited time.The analysis of single cell data has become a hot topic at present, and it is very interesting to extend the semiMM method to be used in the analysis of single cell data.


We considered the prediction of cancer classes (e.g. subtypes) using patient gene expression profiles that contain both systematic and condition-specific biases when compared with the training reference dataset. The conventional normalization-based approaches cannot guarantee that the gene signatures in the reference and prediction datasets always have the same distribution for all different conditions as the class-specific gene signatures change with the condition. Therefore, the trained classifier would work well under one condition but not under another.To address the problem of current normalization approaches, we propose a novel algorithm called CrossLink (CL). CL recognizes that there is no universal, condition-independent normalization mapping of signatures. In contrast, it exploits the fact that the signature is unique to its associated class under any condition and thus employs an unsupervised clustering algorithm to discover this unique signature.We assessed the performance of CL for cross-condition predictions of PAM50 subtypes of breast cancer by using a simulated dataset modeled after TCGA BRCA tumor samples with a cross-validation scheme, and datasets with known and unknown PAM50 classification. CL achieved prediction accuracy >73 %, highest among other methods we evaluated. We also applied the algorithm to a set of breast cancer tumors derived from Arabic population to assign a PAM50 classification to each tumor based on their gene expression profiles.A novel algorithm CrossLink for cross-condition prediction of cancer classes was proposed. In all test datasets, CL showed robust and consistent improvement in prediction performance over other state-of-the-art normalization and classification algorithms. The rapid development of high-throughput technologies including microarray and high-throughput sequencing have significantly advanced our understanding of disease including cancer . TorrentAs exciting as these new discoveries are, their translation from laboratories to real clinical practice remains a challenge. Overcoming systematic and condition-specific biases presented in expression data as a result of different technological platforms, varying experimental/measurement conditions, and heterogeneities in the patient age, gender and race continues to be an issue yet to be completely addressed. Although improved standards in uniform experimental and clinical protocols have and will help reduce the systematic biases, eliminating biases specific to experimental/clinical conditions, patient individuals, technology/platforms would be more effective dealt with by using computational algorithms. The well-known Microarray Quality Control project (MAQC) spearheaded the algorithm development in this front and demonstrated that through careful algorithm-based normalization, consistently differentially expressed genes can be reproduced in data produced from different platforms . Since tIn this paper, we consider the problem of predicting cancer classes (e.g. subtypes) based on patient gene expression profiles. Particularly, a reference expression dataset is assumed available, where the true cancer class labels for each sample are known. However, compared with the reference dataset, the prediction dataset is generated using a different platform, from patient samples of, for instance, different races, and collected under a different condition. That is, we assume that the prediction dataset contains both systematic and condition-specific biases. Currently, the mainstream practice to this prediction starts by first normalizing the reference and prediction dataset so that both can follow the same desired characteristics (e.g. distribution). Then, a classifier is trained using the normalized reference dataset, which would produce a set of signature genes, accompanied also by their associated class-specific expression signatures . This geTo address the problems of current normalization based approaches, we propose a novel algorithm called CrossLink (CL). The CL algorithm represents a complete departure from the current normalization-classification paradigm. CL only assumes that each cancer class is associated with a set of signature genes, which are independent of the conditions. However, CL recognizes that although for a specific condition, the signature genes should define a unique, cancer class-specific gene expression signature but this signature changes under a new condition. Moreover, the change in the signature is condition-specific and there is no universal, condition-independent normalization mapping of signatures. As a result, unlike existing normalization-based algorithms, CL does not attempt to explore a mapping of the signatures across different conditions; in contrast, it exploits the fact that the signature is unique to its associated class under any condition and thus employs an unsupervised clustering algorithm to discover these unique signatures Fig. .The rest of the paper is organized as follows: In Methods, the workflow of CL is discussed in details. In Results, we demonstrate the improved, robust performance of CL using both simulated and real data. The concluding remarks are drawn in Conclusion.Suppose that we are given a reference dataset that measures global gene expression of a set of known cancer classes . The problem that CL addresses is to predict the cancer classes for a set of new expression data samples collected under a different condition. The workflow of the CL algorithm can be divided into two steps: signature gene set identification and class prediction. For the first step, the goal is to identify the signature gene sets for each cancer classes from the reference dataset. For the second step, the signature gene sets are used to predict the class labels for the prediction dataset. The details of these two steps will be discussed next. Notice that before implementing CL, gene entries of data samples from reference and prediction datasets need to be mapped into a set of common reference entries. By default, CL uses the common Gene Symbol as the reference entries. A data entry is removed from all samples if no entry in any samples can be mapped.N classes and the reference dataset contains M genes. A one-way ANOVA model is then proposed for each gene expressionk = 1 … N, Gi is the gene expressions of the ith gene in all samples, gi is the ambient expression of this gene while αk represents the unique effect of the kth class on the gene expression, and ε ~ N denotes the zero-mean noise. The ANOVA analysis identifies these signature genes for each class by testing the hypothesisAs commonly defined, the signature gene set of a cancer class include genes that show uniquely differential expression in that class. Analysis of Variance (ANOVA) solves such problem. Suppose that there are against the alternative hypothesisN signature gene sets will be obtained.A multiple comparison procedure such as Benjamini-Hochberg is applied to calculate the False Discovery Rates (FDRs) for each gene and the genes that are tested significant (FDR>0.05) for a class are determined as the candidate gene of the signature gene set of that class. An additional filtering step is followed to reduce the possible false positive signature genes. The filtering imposes three expression conditions on every candidate signature genes, first, the candidate signature gene should have the largest absolute average expression in the class it test significant for; second, a lower limit is introduced on the difference of average expression between the class it tested significantly for and the rest; and third, a lower limit is introduced on the absolute average expression of the class it is testing. The leave-one-out cross-validation was applied to determine the limits that yield the best classification outcomes. Only those candidate genes that satisfy all three conditions will be retained to form the final signature gene sets. In the end, N signature gene sets independently. For each of the clustering results, the cluster with a higher absolute average expression value is determined target class, whereas the other cluster is determined as the non-target class. Now that each sample can be assigned with a target class label for any of the N classes, a sample can be associated with multiple class labels. To resolve this ambiguity, a multiple call adjustment procedure is proposed. Specifically, for each class j that is assigned to a sample i, a confidence score Si,j is calculated as: j = 1, …, k. Since for each class specific signature gene set, the clustering algorithm separates the dataset into two clusters: class target clustering and class non-target cluster. We further assume that the class target cluster can be modeled by a normal distribution N1 and the non-target cluster by another normal distribution N2. The first criteria p1 calculate the probability of samples in the class j’s gene set under the hypothesis that the non-target cluster distribution is true. This probability essentially measures the distance from the sample to the center of the class non-target cluster center. The second criteria p2 calculate the probability of μj,t − μj,nt under the hypothesis that the difference of two means follows a student t distribution sp is the pooled variance of two normal distribution N1 and N2, and degree of freedom is n1 + n2 − 2, where n1 and n2 are the sample size of class target cluster and class non-target cluster respectfully. This probability is essentially a two-sample t test, measures the distance between the class target cluster and class non-target cluster. It is obvious that both two criteria are maximized to yield a higher confidence level. A confidence score is determined by multiplying these two criteria together. The class with the highest confidence will be assigned to the sampleOnce the signature gene sets are determined for each class, the next step is to predict the class labels for a new set of data samples. As opposed to using a supervised approach that performs the prediction with a classifier trained on the  training data set, CL employs a novel unsupervised solution. Since we assume that each signature gene set possesses a unique expression signature for the corresponding class, it would be natural to expect that the class-specific gene set can separate the dataset into two groups: one that belongs to the target class that the gene set is associated with and the other one that contains samples from other classes. Therefore, CL employs the K-means clustering  to groupN represents the size of the prediction dataset. Since different dataset certainly have different PAM50 class label rates, this difference in the class label rates could yield an accidentally equal ISEP. Also, because the conditional probability of each PAM50 class does not equal to each other, although the ISEPs of two experiments may differ, they could infer the exact same classification accuracy. Because of these reasons, we want to point out that the ISEPs for two datasets should not be compared.We investigated CL performance for cross experiment predictions of PAM50 subtypes (See Results for details). However, the true PAM50 subtypes are rarely available for most of the prediction datasets and when this is the case, direct evaluation of the CL performance is infeasible. In contrast, the pathological biomarker assessments of the estrogen receptor (ER) progesterone receptor (PR) are much more accessible for most of the patient samples. Particularly, in a recent study, over 800 breast cancer patients were genetically profiled and their PAM50 subtypes tested by a novel RT-qPCR approach that is independent of microarray platform and their ER and PR status were recorded . This sthttp://compgenomics.utsa.edu/CrossLink/CL_R.zip.All algorithms are designed and implemented under Matlab R2013a. Function ‘anova1’ is used in the signature selection; function ‘kmeans’ is used in the classification procedure. The designed algorithm is also implemented with R (version 3.1.1). The R code and an example demonstrating the whole pipeline are provided to show how to extract signatures from a reference dataset and how they can be used to classify independent cross-condition samples. The package can be downloaded from The study was approved by the Institutional Review Board of Weill Cornell Medicine-Qatar and the Hamad Medical Corporation’s Ethics Committee, Doha. All subjects signed informed consent documents for participation in this study. The diagnosis of cancer was confirmed by histopathologic analyses. Expression of ER, PR and Her2 was revealed by immunohistochemistry. Human breast cancer tumor samples and non-malignant healthy breast tissues were collected, immediately placed in RNAlater solution and frozen at -80 °C until further use. RNeasy Minikit (Qiagen) was used to extract and purify RNA from these breast tissue samples. The GeneChip Human Genome U133A 2.0 Array (Affymetrix) was used to explore the differentially expressed genes according to manufacturer’s instructions.This section is separated into three parts: (1) the ability of CL for PAM50 classification is first demonstrated in several scenarios; (2) the application of CL on Cancer2000 classification is then demonstrated; (3) a Qatar breast cancer patients’ Microarray data analysis is conducted.PAM50 breast cancer intrinsic subtype is a gene expression based classification system that includes five breast cancer subtypes: Luminal A (LumA), Luminal B (LumB), Her2 enriched (Her2), Basal and Normal-breast like  . It has ij according to the following model:We first evaluated CL on a simulated dataset, where the true class labels for the test samples are known. Breast Cancer Patient Microarray dataset (BRCA) from The Cancer Genome Atlas (TCGA)  was usedGij is the gene expression of the ith gene in the jth sample of the prediction dataset, αi ~ N is the experimental bias for gene i and is constant across all the samples, and εij ~ N models the sample-specific noise. Notice that the experimental biases are different for different genes. These gene-specific biases simulates the varying influence of a different experimental condition on the expression of different genes. In this experiment, we investigated the robustness of CL prediction against experimental bias, where we let σ2 equal to 0.5 and σ12 ranged from 0 to 7.where The prediction performance of CL and seven state-of-the-art cross- platform normalization algorithms are shown in Fig. We carried out next an evaluation of CL performance on a real dataset: the UNC breast cancer patient dataset. In this case, the PAM50 subtypes are available and the prediction performance can be directly evaluated. The data samples were collected from Gene Expression Omnibus (GEO) under the data entry GSE2740 . Out of In this process, the impact of different threshold (see Methods for details) was also investigated Table . We can This signature gene set yielded a leave one out classification rate of 80 % for the BRCA dataset. In addition, this gene set was pooled together and compared with the well-known PAM50 signature gene set Fig. . SpecifiWe then proceeded to test CL on additional datasets. This time, the true PAM50 labels were not available and we applied the proposed ISEP instead to direct assess the prediction performance. Before we proceeded to prediction, we first evaluated the relationship between the ISEP accuracy and the accuracy based on true PAM50 labels. The better the ISEP represents the true performance, the more correlated the ISEP and the true accuracy should be. ISEPs corresponding to different PAM50 classification accuracy based on the reference dataset (TCGA-BRCA) were calculated. The result shows that ISEP strongly correlated with PAM50 classification accuracy with a correlation coefficient of 0.96 Fig. . The ISENext, CL was applied to a dataset that includes 28 breast cancer patients, whose stroma and epithelium cells were profiled with Affymetrix U133A 2.0 GeneChips (GSE10797). Only 20 samples with both ER and PR information were selected in order to calculate the ISEP accuracy. TCGA-BRCA was still used as the reference dataset and this time there was also a difference in platforms in addition to the apparent experimental differences. As a comparison, the original PAM50 classifier (R code)  was alsoRecently, over 2000 breast cancer patients (cancer2000) were profiled and a classification including 10 novel breast cancer subtypes were reported based on the integrative study of microarray gene expression, copy number variation as well as gene mutation information . These nCancer 2000 contains two parts, where first part is a discovery dataset that includes 997 breast cancer patients samples and the second part includes 5 additional validation sets including another over 900 breast cancer patient samples. For each patient, the expression levels of 48,803 genes were measured . Here weWe then used CL to predict the Cancer2000 subtypes for TCGA-BRCA dataset. A set of 10 signature gene sets was first obtained on the reference Cancer2000 dataset Table  and the t test and adjusted P value set to 0.05, 116 genes showed significantly differential expression between Qatar breast cancer patients and Qatar normal breast tissue samples. We also aimed to find the genes uniquely expressed only in Aerobic species by comparing QNRF dataset with another set of breast cancer population. For comparison, dataset GSE22035 was downloaded from GEO. This dataset contains 43 Caucasian species samples. It has the same microarray platform as the QNRF dataset. Both datasets went through the same pre-process procedure and additional round of normalization was done on two datasets together. Note that this analysis was not performed on all the genes but only on the differential expressed genes detected previously. All seven cross platform normalization algorithms and quantile normalization were performed in order to detect common differential expressed gene unique to QNRF dataset. However, among all the cross-platform normalization algorithms, no common gene is reported. With Quantile normalization, 9 genes were reported but for DisTran and MRS, another set of 6 genes were reported. Although we cannot provide a consistent list of genes that differential expressed across all normalization algorithm, this 15 gene together could be our primary target of interest in future study for breast cancer in Qatar population. The PAM50 classification and Cancer 2000 classification were also reported by CL procedure (Table First we aimed to find genes differential expressed in Qatar breast cancer patient compare to the control sample. With two sample In this paper, we proposed a novel algorithm CrossLink for cross-condition prediction of cancer classes. Unlike other normalization-based method, CL employs an unsupervised algorithm, which aims at identifying unique class-specific signatures patterns. CL was applied for cross-condition prediction of the PAM50 and Cancer2000 subtypes. In all tested datasets, CL showed robust and consistent improvement in prediction performance over other state-of-the-art normalization algorithms.Despite its advantages, CL has limitations. First, CL is better fitted for datasets of large sample size, because CL needs to perform an unsupervised learning. It cannot be applied to individual samples separately as what a classifier would do. By the same reasoning, CL would fail when there are samples from only a single class.Our future work includes to three directions. First, the result of the CL indicates that instead of choosing a common signature set for all subtypes classification, subtype specific signatures can lead to better robustness and accuracy for subtypes classification. Further investigation is needed to discover the biological insight of those signatures. By doing so, the subtype related function could be also discovered. Second, CL shows great potential for subtype classification in cross-condition breast cancer subtype classification. This ability could be further extended into other cancer genomic classification problems when condition specific bias presented. Third, the unique design of CL allows it bypassing the condition specific bias to achieve a robust classification accuracy. This advantage can be further extended to handle bias between different technical platforms, for example, between microarray and RNA-seq data.


In genetic data modeling, the use of a limited number of samples for modeling and predicting, especially well below the attribute number, is difficult due to the enormous number of genes detected by a sequencing platform. In addition, many studies commonly use machine learning methods to evaluate genetic datasets to identify potential disease-related genes and drug targets, but to the best of our knowledge, the information associated with the selected gene set was not thoroughly elucidated in previous studies. To identify a relatively stable scheme for modeling limited samples in the gene datasets and reveal the information that they contain, the present study first evaluated the performance of a series of modeling approaches for predicting clinical endpoints of cancer and later integrated the results using various voting protocols. As a result, we proposed a relatively stable scheme that used a set of methods with an ensemble algorithm. Our findings indicated that the ensemble methodologies are more reliable for predicting cancer prognoses than single machine learning algorithms as well as for gene function evaluating. The ensemble methodologies provide a more complete coverage of relevant genes, which can facilitate the exploration of cancer mechanisms and the identification of potential drug targets. With the development of genetic sequencing technology, genetic information could be recorded as gene expression data. Data mining, such as using machine learning methods, is commonly used to reveal latent correlations between diseases and gene expression. Supervised or semisupervised machine learning algorithms have been proposed to predict the clinical outcomes of cancers –4 in theTo address this problem, the present study thoroughly investigated the performance of single models among different datasets and proposed a strategy to combine multiple predictive models as well as the datasets into a final ensemble for clinical prediction. Compared with a single machine learning algorithm, an ensemble scheme could not only perform more reliably when predicting clinical endpoints but could also provide broader coverage of disease-related genes, which will be beneficial for further downstream analysis in such applications as the identification of potential drugs.The workflow of this study is listed in https://gdc.cancer.gov/developers/gdc-application-programming-interface-api). The genomic data were from the official normalized microRNA and RNA sequence expression data because it was restricted by multiplatform coverage and accessibility; a portion of the cancer data was excluded from this study. For example, the number of available samples of neuroblastoma in GDC is 1127 (this number might change if the database is updated), but only approximately half of these samples have associated RNA sequence data, and no microRNA sequence data are available. Finally, the freely accessible data for breast invasive carcinoma (BRCA), ovarian serous cystadenocarcinoma (OV), and kidney renal clear cell carcinoma (KIRC) were downloaded and used for modeling. The clinical information was downloaded in the XML format. The relationships of samples from different platforms  were identified by the official MetaData file in the JSON format. The detailed distribution can be found in All data were downloaded from the NCI's Genomic Data Commons (GDC)  by usingSince we wanted to make the results from two sequencing platforms  comparable and able for voting at last, the selection of samples was determined by the integration of different platforms. Only samples that contained clinical information and expression data for both mRNA and microRNA were retained for subsequent analysis.The sample label was determined by the clinical information. For OV and KIRC, the label was determined by the survival time. A sample was deemed positive if the recorded survival time was larger than one year and if the patient was still alive . Similarly, a sample was deemed negative if the associated patient was dead and if the recorded survival time was less than one year. The sample label for BRCA was determined by the estrogen receptor (ER) status (negative or positive), which was also recorded in the clinical XML file. Therefore, if the required information for a sample could not be found in the clinical data or its status did not satisfy the criteria , it was excluded.t-test was used to subsequently reduce the mRNA and microRNA numbers. Only genes that had significant expression with a p value less than 0.05 were retained in a dataset. The ratio of positive to negative samples was kept in an appropriate range to reduce classification bias. In this study, the range of this ratio was 0.5 to 2. For example, if a dataset contained 22 positive and 50 negative samples, 6 negative samples were randomly removed to adjust the ratio so it fell within the required range. The eliminated datasets were divided into two parts for cross-validation and independent tests in a 4 : 1 ratio. The scale of the datasets is listed in The sample number was reduced by the clinical information. Student's functions class, most of the methods use a functional solution for modeling the data, such as LibSVM [meta class, the methods use resampling and voting for classification and regression, and the methods in the other classes are considered model units, such as AdaBoost [Bagging [bayes class are from the probability and graph theory, and most of them, including NaïveBayes [BayesNetwork [BayesianLogisticRegression [rules class use rules (such as decision table) for classification [lazy class are instance-based and could be optimized for better efficiency using a lazy algorithm [trees class are based on the classification and regression tree algorithm, but the way they are carried out is different. Many other mechanisms are integrated into the trees such as resampling used in a random forest [misc class contains methods for which it is difficult to assign to another class. Only two methods fell into this class in this study, namely, the VFI, an ensemble method based on a voting protocol [HyperPipes, based on an algorithm that finds similarities among attributes. Considering running time, comparability, and reducing the risk of overfitting, only default parameters are used for modeling.49 modeling methods in WEKA  version. The difs LibSVM  and logis LibSVM , and in AdaBoost  and Bagg[Bagging . The metïveBayes , BayesNesNetwork , and Baygression , are senfication . The metlgorithm . Most ofm forest . Finallyprotocol , and theOneR, ReliefF, InfoGain, SymmetricalUncert, and GainRatio. OneR is executed by using the OneR classifier, which is based on measuring the error between the attributes and the response values [ReliefF uses a resampling mechanism for evaluating the attributes [H” in the formula is the information entropy (Shannon entropy) [GainRatio could be considered as a normalization of InfoGain. However, the information entropies of all of the attributes, such as the expression of mRNAs and microRNAs, are different, so both methods are used in this study. By using the ranking mechanism in WEKA, in every subdataset for cross-validation, the attributes can be ranked, and the top 10, 30, 50, and 100 ranked attributes are selected for cross-validation and modeling. Note that since the number of microRNAs in OV is limited , the numbers of the attributes in the subdatasets are 10, 30, 50, and 83. Totally, we investigated a total of 5 variable selection methods × 4 subdatasets × 49 modeling methods = 980 predictive models.Because the sample number was limited, fewer genes should be considered to avoid overfitting. In this study, five variable selection methods were used for dimensional reduction: e values . ReliefFtributes . The othentropy)  and “claAfter generating hundreds of models, it is possible to combine their predictions. As previously mentioned, the prediction performances are ranged in the datasets. To integrate the ranged predictions and find a stable modeling method for genetic datasets, we used a voting protocol in this study to identify the datasets.InfoGain, SymmetricalUncert, and GainRatio. The weights in the information theory methods were modified to 1/3 when voting the predictions from the subdatasets of the OV-miRNA group according to the prediction distribution. More details may be found in “the coverage and reliability of selected genes” and “the distributions of the predictions.”All of the weights were the same, except in the information theory methods such as Three voting schemes were used to arrive at a comprehensive conclusion. All of the methods were first used for voting. Then, some of the methods which performed poorly for all datasets were eliminated. Finally, mRNA and miRNA datasets were combined for voting.Since there were many predictions, box plots were used to reflect the stability of the different classes. The quartiles Q1 and Q3, the interquartile range (IQR), and the whiskers  in the box plot are discussed. Two types of box plots were used to present the results in different angles, one based on the WEKA classes of modeling methods and another based on the scale of the subdatasets. Because the ratio of positive to negative samples was biased, the Matthews correlation coefficient (MCC) was used as the criterion for the plots. The MCC is one of the criteria used to evaluate the prediction performance, and the associated formula isThe prediction performance was ascertained in two ways: by the modeling method class defined by WEKA see  and by tmeta class and trees class methods performed better than those in other classes for the two types of BRCA genomics data  that contained more attributes had a better distribution . However, the peak was sometimes found in a smaller dataset. In more detail, the two BRCA groups Figures  and 5 sebayes class in the OV-miRNA group. In more detail, similar to the box plots, the voting performance based on the BRCA datasets was similar and near the upper whisker except for misc. The range was larger in the other datasets . Different classes in turn, including bayes, functions, meta, lazy, misc, and trees, showed the top three best prediction performance. The rules class always ranked lower in the overall voting test, but the bayes class showed good voting performance even though its distribution, as indicated by the box plot, was not stable.The performance of all of the voted methods is listed in the column “Vote from all methods” of Tables The filtering rule was based on the distribution of the prediction results. Values that fell out of the range indicated by the whiskers in a box plot were considered to be outliers. Similarly, in our study, a method with an MCC below the lower whisker was considered as an outlier. There were 5 value selection methods, and each generated 4 subdatasets. If 6 datasets (3 cancers × 2 sequencing techniques) were considered, a modeling method was used: 5 variable selection methods × 4 subdatasets × 6 datasets = 120 times. Therefore, if a method was designated as an outlier more than 6 times (5% of 120), it was not considered in this voting test.meta. It was evident that ZeroR had the most counts, and it was accordingly eliminated. Since only default parameters were used, LibSVM and SPegasos, which could be considered an optimized SVM, were designated as outliers with high counts. Similarly, the BayesianLogisticRegression method required parameter optimization and was thus eliminated by many counts. No methods in the meta class were eliminated, which indicated that methods based on resampling and an ensemble mechanism were stable and could address varied datasets even they were not always the best.The details of eliminated methods are listed in As mentioned in the Preprocessing, the datasets for one cancer comprised the same samples. Therefore, voting from both mRNA and miRNA datasets is possible. The prediction performance is in the last row of Tables In this section, the modeling results will be discussed. Therefore, the discussion is comprehensive, and the gene selection and the modeling results are discussed separately.Five variable selection methods were used in this study, and different mRNA/miRNA datasets were separately generated. If the subdatasets generated are similar, combining multiple variable selection methods is worthless. Therefore, it is necessary to analyze the contents of the subdatasets and determine the similarity of the datasets. Moreover, the importance of the mRNA was evaluated by using a 3rd party database.The coverage fraction is the number of the genes which are used more than once of all genes, and accordingly, the formula isAccording to Tables To determine if the shared mRNA is important, the Human Protein Atlas database  was usedThe records in the Human Protein Atlas will be updated, and more mRNAs will become available. Therefore, the fraction of hits and the reliability will change accordingly and the shared mRNAs that are currently not recorded in this database might be worthy of study. In addition, the reliabilities of the genes were similar whether a gene was independent or shared, and this indicates that the selected genes were representative for a prognosis but might have a redundant function; therefore, the shared genes are not significantly different from the independent genes in reliability.functions class and lazy class, the MCC from cross-validation had a relatively better predicting performance  than independent test. According to the whiskers, most of the classes had small differences, but there was still a lot of the modules that had a large difference which could be reflected by the outliers. Thus, it is still risky to get the unreliable prediction if we only use the modules which have good performance in cross-validation for predicting. However, since most of the methods were reliable, combining the methods together becomes useful and necessary to reduce the risk.The results from cross-validation could be used as the reference to evaluate the reliability by comparing the results with independent test. As an empirical conclusion, in most cases, the results from cross-validation could be better than an independent test due to various reasons such as the overfitting and batch effects, but if the difference is not too large, the modules could be identified as reliable. Reflected in the figures, except a few classes such as the A balanced ratio of positive samples to negative samples is an important factor for prediction. The BCRA datasets had the largest sample numbers; therefore, the IQRs were the smallest, which indicates that the MCCs were concentrated toward the median. The boxes become wider for the KIRC, OV-mRNA, and OV-miRNA datasets. The sample number should be guaranteed before modeling if a stable prediction is to be obtained. However, sometimes, many reasons such as cost, privacy, and difficulty limit the sample number, so it is insufficient to confirm the prediction stability. Such predictions should be considered very carefully because overfitting may have occurred.t-test, fold-change, or FDR. In the analysis, the genes are independent from each other when calculating the correlations and thus the validity of the identified genes is only affected by the sample number. If the sample number is too small (less than 10), the t-test result is not reliable.A basic way to avoid overfitting is to reduce the attribute number for modeling, and that is why 4 subdatasets  were used for modeling. As shown in the Modeling Results, especially in Figures meta class are relatively stable for prediction, which is reasonable because the methods in the meta class use other methods for ensemble learning, so they are not sensitive to different dataset distributions as other types of methods are. The methods in the trees class were similar but had a more varied performance than the meta class methods because the classification and regression tree algorithms can be as simple as REPTree or as complex as random forest. Therefore, the boxes of the trees class usually had a larger IQR than the meta class. However, the misc class performed best for two datasets, but only two methods were contained in this class. Based on the algorithm, the methods in the misc class were much different and thus could have a much more variable performance for different datasets. Except for the rules class, the methods in the bayes class also had a relatively inferior performance, which might have been caused by the low sample number, because probability-based methods are sensitive on a modeling scale. The performance of methods in the functions class was more skewed; the boxes usually had a good upper whisker but a poor lower whisker. One reason for these might be that the algorithms in this class are also different from those in the misc class. There were 6 methods in the functions class, and thus the prediction performance varied widely. Another reason might be parameter optimization; less parameter optimization would affect all of the methods, but the methods in the functions class might be the most affected because they are much more sensitive to the parameters when only default parameters were used.Limited sample and attribute numbers make a prediction sensitive to the datasets. The various distributions in Figures The high similarity of the gene data used would lead to a similar prediction performance, but when the similarity is lower than 50%, as reflected by the coverage in According to the coverage of mRNA and microRNA used and shown in Tables As previously mentioned, the predictions using different datasets differed, meaning that we cannot determine which method is best for all datasets. The separate use of different modeling methods will result in a loss of information, so using ensemble methods to integrate the modeling results is necessary.bayes class is much lower. However, the result is anomalous in that the bayes class showed a good ensemble classification performance (it was ranked in top three) for 5 datasets. On the other hand, the meta and trees classes were not as good as the bayes class even though they had relatively similar distributions. One reason is many ensemble algorithms are contained in those two classes, so that the voting had already been accomplished, so the prediction results were concentrated near the median or the average. It is not difficult to see that in most cases, the voting results of the two classes were near the average. Another reason is that only default parameters were used in the entire test. Many ensemble methods must optimize the submethods for ensemble learning, and the resampling methods also must be optimized.A sufficiently large fraction of coverage in the data space is necessary for good voting performance, and the average accuracy must not be too low. Therefore, if only the box plots are considered, the expected performance of the bayes class was negatively affected by the filter. The MCCs based on the bayes class were not larger than prior to voting. The rules and functions classes were benefited by the filter, and the MCCs were improved for most of the datasets. The trees class was slightly benefited in the BRCA-miRNA group but was not globally affected as the meta class was. The other classes, including the overall voting, were affected positively or negatively by different datasets. The biased effects could indicate that the methods were sensitive to the datasets; the prediction performance of a method changed greatly when the dataset changed. On the other hand, the overall performance was not affected too greatly. One reason was the meta class methods, which showed a stable prediction ability, were not eliminated, and thus, the overall results remained stable. Another reason might be that the performance effects were polarized. For example, the bayes class that showed the best comprehensive performance was negatively affected but the rules class was benefited. Therefore, the overall differences from the two voting mechanisms could be less.As a comparison, the voting performance from the eliminated methods was similar. However, the voting results were not always better than the original results. The The biased effects mean that a filter might not be that necessary if there is no prior knowledge. However, the voting by combining the mRNA and miRNA datasets could produce better performance if the sample size is sufficiently large, as shown in bayes class show a good ensemble performance, even though the individual methods are not as stable as those in the meta or trees classes. The meta and trees classes already contain many ensemble methods; therefore, their performance is stable but, again, not good for ensemble twice. Therefore, using the methods in the bayes class as a group and one of the algorithms in the meta class might be a practical approach for a dataset without sufficient prior knowledge. If prior knowledge exists for a cancer, the methods and datasets used can be more specific. For example, this study indicates that using miRNA as an attribute for modeling the OV data could yield a better result than using mRNA, if we knew that at first, some of the negative effects could be avoided. We hope that the scheme can facilitate related studies of genetic data modeling and elucidate important genes to enhance the reliability of the final model.The purpose of this study was to discover a reliable way to predict unknown data to reduce the risk of error prediction when not enough samples were used for modeling. The distribution of the modeling performance indicated that the best methods were different for different datasets; therefore, the methods were integrated using a voting protocol. Finally, we proposed a better way to model different gene expression datasets. In conclusion, no prior knowledge exists; a comparison of the prediction results for three cancers indicates that the methods in the