We developed a classifier using RNA sequencing data that identifies the usual interstitial pneumonia (UIP) pattern for the diagnosis of idiopathic pulmonary fibrosis. We addressed significant challenges, including limited sample size, biological and technical sample heterogeneity, and reagent and assay batch effects.We identified inter- and intra-patient heterogeneity, particularly within the non-UIP group. The models classified UIP on transbronchial biopsy samples with a receiver-operating characteristic area under the curve of ~ 0.9 in cross-validation. Using in silico mixed samples in training, we prospectively defined a decision boundary to optimize specificity at ≥85%. The penalized logistic regression model showed greater reproducibility across technical replicates and was chosen as the final model. The final model showed sensitivity of 70% and specificity of 88% in the test set.We demonstrated that the suggested methodologies appropriately addressed challenges of the sample size, disease heterogeneity and technical batch effects and developed a highly accurate and robust classifier leveraging RNA sequencing for the classification of UIP.The online version of this article (10.1186/s12864-018-4467-6) contains supplementary material, which is available to authorized users. Interstitial lung disease (ILD) consists of a group of diseases affecting the pulmonary interstitium with similar clinical presentation; idiopathic pulmonary fibrosis (IPF) is the most common ILD with the worst prognosis. By definition, the cause of IPF is unknown; this can make an accurate, confident and timely diagnosis challenging. An accurate diagnosis for IPF requires multidisciplinary evaluation of clinical, radiologic and histopathologic features , 2, withtransbronchial biopsy samples (TBBs) collected via bronchoscopy, a less invasive procedure compared to surgery. Several studies have revealed that genomic information in transcriptomic data is indicative of phenotypic variation in cancer and other chronic disease [To build this new diagnostic tool, we used exome-enriched RNA sequencing data from  disease , 5. Comp disease . More im disease . Here weMachine learning methods have been applied extensively to solve biomedical problems, and have deepened our understanding of diseases such as breast cancer  and glioInter- and intra-patient heterogeneity adds significant complexity to classification. The non-UIP category is not one disease, but a collection of heterogeneous diseases with a wide range of phenotypes, all of which are encountered in the clinic. This, coupled with the small sample size, resulted in limited numbers of samples in each non-UIP disease category. Another unique feature of this study is heterogeneity within a patient: histopathology features are not uniform across the entire lung. Not surprisingly, genomic signatures also vary depending on the location of the biopsy sample . To bettThus far, we have described challenges specific to machine learning. However, since the classification models we developed serve as the foundation for a diagnostic test to be used clinically, there are two additional requirements we need to address. First, for cost-effectiveness, only one sequencing run per patient is commercially viable and the independent test set needs to reflect this reality. This necessitated analytically bridging individual samples used in the training set to pooled samples in the test set. Secondly, we need to ensure that the final locked classifier not only performs well on the independent test set, but will maintain this performance on all incoming future samples. Therefore, developing a classifier that is highly robust to foreseeable batch effects in the future becomes critically important.In the following sections, we illustrate some of the challenges with quantitative analysis, describe practical solutions to overcome those challenges, show evidence of improvement, and discuss limitations of these approaches as well as directions of our future work Fig. .Fig. 1AnPatients under medical evaluation for ILD that were 18 years of age or older and were undergoing a planned, clinically indicated lung biopsy procedure to obtain a histopathology diagnosis were eligible for enrollment in a multi-center sample collection study  . PatientDuring study accrual, 201 BRAVE patients were prospectively divided into a group of 113 considered for use in training (enrolled December 2012 to July 2015) and 88 used in validation (enrolled August 2014 through May 2016). The training group ultimately yielded 90 patients with usable RNA sequence data and reference labels derived from pathology (described below) that were used to train and cross-validate the models. The independent test group yielded 49 patients that met prospectively defined inclusion criteria related to sample handling, sample adequacy, and the determination of reference truth labels. All clinical information related to the test set, including reference labels and associated pathology, were blinded to the algorithm development team until after the classifier parameters were finalized and locked. The test set was prospectively scored by an independent third party who was not involved in algorithm development, sequencing data generation or reference label generation.Histopathology diagnoses were determined centrally by a consensus of three expert pathologists using biopsies and slides collected specifically for pathology, following processes described previously , 12. TheUp to five TBB samples were sampled from each patient and typically, two upper lobe and three lower lobe samples were collected. TBB samples for molecular testing were placed into a nucleic acid preservative  and stored at 4 °C for up to 18 days, prior to and during shipment to the development laboratory, followed by frozen storage. Total RNA was extracted , quantitated , pooled by patient where appropriate, and 15 ng input into the TruSeq RNA Access Library Prep procedure , which enriches for the coding transcriptome using multiple rounds of amplification and hybridization to probes specific to exonic sequences. Libraries which met in-process yield criteria were sequenced on NextSeq 500 instruments (2 × 75 bp paired-end reads) using the High Output kit . Raw sequencing (FASTQ) files were aligned to the Human Reference assembly 37 (Genome Reference Consortium) using the STAR RNA-seq aligner software . Raw reasizefactor estimated with the median-of-ratio method and transformed by parametric variance-stabilizing transformation (VST), asymptotically equal to the logarithm to base 2 . To mimFor algorithm training and development, RNA sequence data was generated separately for each of 354 individual TBB samples from 90 patients. Eight additional TBB samples (‘sentinels’) were replicated in each of eight processing runs, from total RNA through to sequence data, to monitor potential batch effects. For independent validation, total RNA extracted from a minimum of three and a maximum of five TBBs per patient were mixed by equal mass within each patient prior to library preparation and sequencing. Each patient in the training set thus contributes up to five sequence samples to training, whereas each patient in the test set is represented by a single sequenced sample, analogous to the planned testing of clinical samples.p-value < 0.05 after multiple-testing adjustment and fold change > 2 was used to select differentially expressed genes. Within the training set, we performed pairwise differential analyses between all non-UIP and UIP samples, and between UIP samples and each non-UIP pathology subtype with more than 10 samples available, including bronchiolitis (N = 10), hypersensitivity pneumonitis (HP) (N = 13), nonspecific interstitial pneumonia (NSIP) (N = 12), organizing pneumonia (OP) (N = 23), respiratory bronchiolitis (RB) (N = 16), and sarcoidosis (N = 11). Principal component analysis (PCA) plots of all the training samples were generated using differentially expressed genes identified above.We first explored whether differentially expressed genes found using a standard pipeline  could be2 values of samples for six representative patients were computed using their VST gene expression, and a heatmap of the correlation matrix with patient order preserved was plotted to visualize intra- and inter-patient heterogeneity in gene expression. The 6 patients were selected to represent the full spectrum of within patient heterogeneity including two non-UIP and two UIP patients with the same or similar pathology subtypes between upper and lower lobes, as well as one UIP and one non-UIP patient each having different pathology subtypes, one being UIP and the other being non-UIP, in upper versus lower lobes. The heatmap was generated using the gplots R package.The correlation rWe summarize the development and evaluation of a classifier in Fig. First, features that are not biologically meaningful or less informative were removed due to low expression level without variation among samples. We filtered genes annotated in Ensembl as pseudogenes, ribosomal RNAs, individual exons in T-cell receptor or Immunoglobulin genes, and excluded low expressed genes with raw counts expression level < 5 for the entire training set or expressed with count > 0 for less than 5% of samples in the training set.gij is the gene expression of sample j and batch i, μ is the average gene expression for the entire set, sampleij is a fixed effect of biologically different samples, and batchi is the batch-specific random effect. The total variation was used to identify highly variable genes; the top 5% of genes by this measure were excluded  is the index set of individual sample i that belongs to patient p. The frozen variance stabilizing transformation (VST) in the training set was applied to The classifiers were trained and optimized on individual TBB samples to maximize sampling diversity and the information content available during the feature selection and weighting process. However, for patients in the test set and in commercial setting, ideally multiple TBB samples would be pooled at the post-extraction stage, as RNA, and processed as pooled RNA in a single reaction through library prep, sequencing and classification . To evalAs the test is intended to recognize and call a reference label defined by pathology , we defiTo minimize overfitting during training and evaluation, we stratified each cross-validation fold such that all data from a single patient was either included or held out from a given fold. Hyper-parameter tuning was performed within each cross-validation split in a nested-cross validation manner . We chosBest practices for a fully independent validation require that all classifier parameters, including the test decision boundary, are prospectively defined. This therefore must be done using only the training set data. Since the test set would be composed of TBBs pooled at the patient level, an in silico mixing model was used to simulate the distribution of patient-level pooled scores within the training set. We simulated within-patient mixtures 100 times at each LOPO CV-fold, with gene-level technical variability added to the VST gene expression. The gene-level technical variability was estimated using the mixed effect model  and calculate the area under the curve (AUC). The binary classification predictions were compared against the histopathology reference labels to calculate sensitivity and specificity.p-value 0.27). Three patients in the training set and one patient in the test set have potential within patient disease heterogeneity: one lobe was labeled as one of non-UIP subtypes , while the other lobe was labeled as a UIP pattern, driving the final patient-level label to a designation of UIP.Table The non-UIP group includes a diversity of heterogeneous subtypes commonly encountered in clinical practice. Due to the small sample size, several subtypes have only one or two patients. Three subtypes—amyloid or light chain deposition, exogenous lipid pneumonia, and organizing alveolar hemorrhage—are present in the test set, which do not exist in the training set.Heterogeneity in samples from the same patient was observed in both histopathologic diagnosis and gene expression. Three such patients with pathology diagnoses spanning UIP and non-UIP groups posed a computational challenge for patient-level diagnostic classification. The correlation matrix of samples from six patients also revealed prominent intra- and inter-patient variability in expression profiles : 55 were up-regulated and 96 were down-regulated genes in UIP  between UIP samples and each non-UIP pathology subtype with a sample size greater than 10 . The average r-squared value based on expression level of 26,268 genes for the pairs of in silico mixed and in vitro pooled samples was 0.99 (standard deviation (SD) = 0.003), which indicated that the simulated expression level of in silico mixed samples was well-matched with that of in vitro pooled samples, considering the average r-squared values were 0.98 (SD = 0.008) for technical replicates and 0.94 (SD = 0.04) for biological replicates.The classification scores of in silico and in vitro mixed samples by two candidate classifiers, the ensemble and penalized logistic regression models (described below) were also compared in a scatterplot Fig. . The numWe evaluated multiple methods of feature selection and machine learning algorithms on our training set of 354 TBB samples from 90 patients. As an initial attempt, individual and ensemble models were evaluated separately based on 5-fold CV and cross-validated AUC (cvAUC), as estimated using the mean of the empirical AUC of each fold. Overall, the linear models such as the penalized regression model (cvAUC = 0.89) outperformed non-linear tree-based models such as random forest (cvAUC = 0.83) and gradient boosting (cvAUC = 0.84). The cvAUC of a neural network classifier was under 0.8. The best performance was achieved by (1) the ensemble model of SVMs with linear and radial kernels, and (2) penalized logistic regression; both of which had cvAUC = 0.89. However, with the heterogeneity among diseases and the small sample size, CV performance on all models was found to vary significantly depending on the split.In LOPO CV, we evaluated the patient-level performance by using 100 replicates of in silico mixed samples for each patient within LOPO CV folds. The computed classification scores of individual samples and averaged scores of in silico mixed samples are shown in Fig. The estimated score variability was 0.46 and 0.22 for the ensemble model and the penalized logistic regression model, respectively . The siA successful validation that meets the required clinical performance Fig.  is only Limited sample size and high heterogeneity within the non-UIP class are two major classification challenges we faced in this project and which commonly exist in clinical studies. In addition, a successful commercial product needs to perform economically and consistently for all future incoming samples, which requires the underlying classification model to be applicable to pooled samples and highly robust against assay variability. We demonstrated that it is feasible to achieve highly accurate and robust classification despite these difficulties and we described practical solutions to each of these challenges. The methodologies have proven to be successful in this study and could be applicable to other clinical scenarios facing similar difficulties.Additional file 1:Supplementary document including a description of score variability simulation and supplementary Figures S1 to S6. (PDF 3639 kb)


As large-scale studies of gene expression with multiple sources of biological and technical variation become widely adopted, characterizing these drivers of variation becomes essential to understanding disease biology and regulatory genetics.We describe a statistical and visualization framework, variancePartition, to prioritize drivers of variation based on a genome-wide summary, and identify genes that deviate from the genome-wide trend. Using a linear mixed model, variancePartition quantifies variation in each expression trait attributable to differences in disease status, sex, cell or tissue type, ancestry, genetic background, experimental stimulus, or technical variables. Analysis of four large-scale transcriptome profiling datasets illustrates that variancePartition recovers striking patterns of biological and technical variation that are reproducible across multiple datasets.http://bioconductor.org/packages/variancePartition.Our open source software, variancePartition, enables rapid interpretation of complex gene expression studies as well as other high-throughput genomics assays. variancePartition is available from Bioconductor: The online version of this article (doi:10.1186/s12859-016-1323-z) contains supplementary material, which is available to authorized users. High-throughput genomics assays have revolutionized our understanding of the molecular etiology of human disease, molecular biology of cell lineage and genetic regulation of gene expression. Transcriptome profiling in particular has been widely applied to detect variation in transcript levels attributable to differences in disease state, cell type or regulatory genetics. As transcriptome profiling studies have expanded in size and scope, they have grown increasingly complex and consider multiple sources of biological and technical variation. Recent studies have simultaneously considered multiple dimensions of variation to understand the impact of cell type , tissue The fundamental challenge in the analysis of complex datasets is to quantify and interpret the contribution of multiple sources of variation. Indeed the most pressing questions concern the relationship between these sources of variation. How does cell or tissue type affect the genetic regulation of gene expression, and does it vary by ancestry , 2? WhatStandard computational workflows employ principal components analysis  and hierThe linear mixed model is uniquely suited to interpreting drivers of variation in complex gene expression studies. Yet the lack of a convenient workflow and scalable implementation for analysis and visualization have prevented wider application of this rich statistical framework. Applying this analysis framework to gene expression data currently requires particular expertise in regression modeling, computational statistics, the R programming language and data visualization. Even then, the time required to implement the analysis is often prohibitive.As gene expression datasets become more complex, the analysis and interpretation of the data is becoming the rate-limiting step. We have developed the variancePartition software and workflow to facilitate rapid analysis and improve interpretation of complex gene expression datasets. The software and workflow enables any analyst to perform a sophisticated analysis and visualize the results in hours using a few lines of R code. variancePartition leverages the power of the linear mixed model –31 to joHere we apply variancePartition to four well-characterized gene expression studies to demonstrate how the workflow facilitates interpretation of drivers of expression variation in complex study designs with multiple dimensions of variation. We illustrate how variancePartition enables rapid interpretation of the drivers of expression variation in these complex datasets.The variancePartition R package implements a computational workflow Fig.  that is http://bioconductor.org/packages/variancePartition and is compatible with Bioconductor ≥v3.2 for R ≥v3.2.The variancePartition software is implemented in R and is optimized for genome-wide analysis of large-scale transcriptome profiling datasets. variancePartition uses the packages lme4  foreach R2 value from a simple linear model, variancePartition extends FVE to applications with complex study designs with multiple variables of interest. The linear mixed model framework of variancePartition allows multiple dimensions of variation to be considered jointly in a single model and accommodates discrete variables with a large number of categories. This analysis has a similar motivation as the standard ANOVA method. Yet the linear mixed model framework has several statistical and practical advantages that make it more accurate and generally applicable to complex study designs with multiple dimensions of variation . While the concept of FVE is widely applied to univariate regression by reporting the Each gene is analyzed separately using the linear mixed model –31 1\doy is the expression of a single gene across all samples, Xj is the matrix of jth fixed effect with coefficients βj,Zk is the matrix corresponding to the kth random effect with coefficients αk drawn from a normal distribution with variance ε, is drawn from a normal distribution with variance where kelihood  as simulmentclass2pt{minimpost hoc calculation Variance terms for the fixed effects are computed using the The total variance is jth fixed effect is so that the fraction of variance explained by the kth random effect is by the and the residual variance is In the standard application of variancePartition, these fractions sum to 1 and are always positive by definition. Moreover, the fraction of variation is also interpretable in terms of intra-class correlation, a metric used to assess biological and technical reproducibility , 36. EacThe formulation of the linear mixed model is very general and includes as special cases models where only fixed effects or only random effects are used. When only fixed effects are used, this model corresponds to a fixed effects analysis of variance (ANOVA) where parameters can be estimated with ordinary least squares. When random effects are specified, the variance terms can be estimated with maximum likelihood or restricted maximum likelihood (REML) . Since RWe focus on the most general case (i.e. mixed models) that includes both fixed and random effects. In this case parameters can be estimated with maximum likelihood. Maximum likelihood estimates are used exclusively in the main text and are the default in the variancePartition software when random effects are specified because this method performs best in simulations.The fixed effects ANOVA model has been widely applied for decades to decompose variance into multiple components of variation . Yet thiThe linear mixed model used by variancePartition has three distinct advantages compared to ANOVA. First, by placing a Gaussian prior on variables modeled as random effects, the linear mixed model more accurately estimates the fraction of variance explained. Even as the number of categories in a discrete variable increases, the linear mixed model still produces accurate estimates because the prior shrinks the estimate for each category towards the zero. Conversely, the fixed effects ANOVA is fit with a linear regression model using ordinary least squares. This method is known to suffer from overfitting and over-estimates the variance fractions for variables with many categories. These properties are well established , 38, 39 Second, the linear mixed model can decompose variance into multiple components in situations where the fixed effects ANOVA cannot be applied because the design matrix is degenerate (i.e. singular). This situation is very common for the types of question of relevant to complex gene expression studies. For example, sex and ancestry are invariant properties of an individual, so jointly analyzing variation across these 3 dimensions of variation involves a degenerate design matrix. In cases like these, the linear mixed model can accurately estimate the desired variance fractions . Consider the simplest example of the with only an intercept term, one random effect corresponding to individual, and an error term. In this case ICC corresponds to the correlation between two samples from the same individual. This value is equal to the fraction of variance explained by individual. For example, consider the correlation between samples from the same individual: The correlation between samples from different individuals is: This interpretation in terms of fraction of variation explained (FVE) naturally generalizes to multiple variance components . See AddThe linear mixed model underlying variancePartition allows the effect of one variable to depend on the value of another variable. Statistically, this is called a varying coefficient model . This anUncertainty in the measurement of RNA-seq data can be modeled with observation-level precision weights that model the relationship between expression magnitude and sampling variance . variancwi denote the inverse of the variance of the observation yi for the ith observation. The precisions can be used to re-weight the samples in a regression to account for the variation in the uncertainty about each observation. Weighting by the precision upweights samples with low measurement error and down weights samples with high measurement error. Denoting the vector of precision weights for a single gene across all samples as w, the model is fit by weighting the residual variance from equation  project  evaluateAs expected, variation across the 4 RNA samples is the major axis of variation, explaining a median of 87.5% of variation in expression Fig. a. But tAnalysis and visualization with variancePartition succinctly illustrates that while variation across laboratories and library constructions is not negligible, it is small compared with the magnitude of biological variation for the large majority of genes. Moreover, variation across flowcells and lanes is very small in this dataset. Thus variancePartition illustrates that RNA-seq data is highly reproducible genome-wide with a small subset of genes showing large technical artifacts.However, there are notable deviations from these genome-wide trends. First, there are a set of transcripts that show little variation between the 4 RNA samples and, in fact, these correspond to spike-in synthetic RNA added to each sample at a standardized concentration to act as controls having equal abundance in all experiments . As expe+CD16 − monocytes and CD4 + T-cells on the Affymetrix Human Gene 1.0 ST Array platform in order to characterize the role of cell type in genetic regulation of gene expression [% of the variation for 4,591 genes. The observation that batch and cell type are the strongest drivers of variation is largely consistent with results from principal components analysis (PCA)  project assayed gene expression in CD14 Meanwhile, sex drives expression variation in a small number of genes, while the age of each individual has a negligible effect. We note that despite the large batch effect observed in this dataset, the biological variation across cell type, individual and sex are still large enough to make meaningful conclusions about cell-specific regulatory genetics when this technical effect is accounted for .Moreover, variancePartition identifies genes that vary along different aspects of the study design Fig. c, and vBy leveraging the flexibility of the linear mixed model, variancePartition can quantify the variation across individuals within each cell type. Since the variance is analyzed within multiple subsets of the data and each sample is only in a single subset, the total variation explained no longer sums to 1 as it does for standard application of variancePartition. Yet the results allow ranking of dimensions of variation based on genome-wide contribution to variance and enables analysis of gene-level results  and monocytes median 16.4%) is substantially larger than when the two cell types were combined   and hierAnalysis of publicly available gene expression studies demonstrate that variancePartition recovers striking patterns of biological and technical variation that are reproducible across multiple datasets. At a genome-wide level, expression variation across individuals and cell types is large enough to overcome the technical variation of transcriptome profiling. Yet at the gene-level there is substantial deviation from the genome-wide trend due to a range of biological and technical factors. By quantifying the variance attributable to each aspect of the study design, variancePartition facilitates the interpretation of these gene-level effects in the context of additional information. We demonstrate reproducible findings that cross-individual variation is driven by cis-eQTL’s and technical variation across laboratories associated with GC content. Moreover, variation across individuals and the relationship to cis-eQTL’s depend on the cell or tissue type.The variancePartition workflow and implementation makes the rich linear mixed model framework easily applicable for interpreting drivers of variation in complex gene expression data. variancePartition provides a general statistical and visualization framework for studying drivers of variation in RNA-seq datasets in many types of high-throughput genomic assays including RNA-seq , protein quantification, metabolite quantification, metagenomic assays, methylation arrays and epigenomic sequencing assays. Although we have focused here on large-scale studies, variancePartition analysis has given valuable insight into RNA-seq datasets with as few as 20 samples. The variancePartition software is an open source R package and is freely available on Bioconductor. The software can easily be applied to RNA-seq quantifications from featureCounts , HTSeq , kallistProject name: variancePartitionProject home page:http://bioconductor.org/packages/variancePartitionOperating systems: Linux, Mac OS X, WindowsProgramming language: R ≥ v3.2Other requirements: Bioconductor ≥ v3.2License: GPL-2


Detecting disease-associated genomic outcomes is one of the key steps in precision medicine research. Cutting-edge high-throughput technologies enable researchers to unbiasedly test if genomic outcomes are associated with disease of interest. However, these technologies also include the challenges associated with the analysis of genome-wide data. Two big challenges are (1) how to reduce the effects of technical noise; and (2) how to handle the curse of dimensionality . To tackle these challenges, we propose a constrained mixture of Bayesian hierarchical models (MBHM) for detecting disease-associated genomic outcomes for data obtained from paired/matched designs. Paired/matched designs can effectively reduce effects of confounding factors. MBHM does not involve multiple testing, hence does not have the problem of the curse of dimensionality. It also could borrow information across genes so that it can be used for whole genome data with small sample sizes. We propose to develop Bayesian statistical models to identify genomic outcomes associated with complex human diseases, such as cancer and other chronic diseases, that are causing significant burden to patients, families, societies and countries. Identifying disease-associated genomic outcomes could not only help discover the underlying molecular mechanisms of complex human diseases, but also help explain the inter-individual variation of response to drug treatments. It is the first step toward precision medicine that takes into account individual variability in genes, environment, and lifestyle for each person in delivering treatment and prevention measures. Messenger RNA (mRNA) could reflect the effects of both genetic and environmental factors on complex human diseases. By comparing the mRNA abundance between diseased subjects and normal subjects, researchers can identify potential disease-associated genes.Cutting-edge DNA microarray technology has been developed to simultaneously measure the intensities of mRNAs for tens of thousands of genes in the human genome. This whole-genome approach, unlike the candidate gene approach, could unbiasedly evaluate the associations of tens of thousands of genes to the disease of interest.When analyzing whole-genome gene expression data, researchers face two big challenges: the effects of noise  in the microarray data and the curse of dimensionality  is much larger than the number of observations (samples)).The noise  could either mask the true gene differential expression or create false detection of gene differential expression. Several effective noise-reduction methods, such as quantile normalization  and surrPaired/matched designs can also reduce the effects of noise. Paired designs are common in intervention studies, such as clinical trials. Matched designs are common in observational studies, such as matched case-control studies. Both are designed to reduce the effects of potential inter-individual variations by providing a homogeneous environment  for comparing measurements under different conditions. Paired /matched designs are commonly used in gene microarray studies.limma and samr from the Bioconductor project ))limma an project  and 7]))limma anMBHMs)  would be much greater than the number of “variables” . Therefore, probe clustering based on MBHMs does not have the curse-of-dimensionality problem. In addition, unlike probe-specific tests that have several parameters per probe, probe clustering based on MBHMs has only a few parameters per cluster and could borrow information across probes to estimate model parameters. Hence it could produce more accurate estimates of model parameters and could work well for datasets with small sample sizes. This property is particularly useful for genomic data that usually have small sample sizes due to high cost of obtaining genome-wide data. Probe clustering based on MBHMs is a special type of model-based clustering that has a known number of clusters (2 or 3 clusters) and imposes special restrictions on the structure of mean vectors and covariance matrices for each cluster , where c2 = Φ−1(0.05) and c1 = Φ−1(0.95) are the 5-th and 95-th percentile of the standard normal distributions, respectively. Hence, if a probe is from cluster 1 (cluster of OE probes), we expect that c1. In other words, we require the probability that makes a mistake that Ideally, we should require τg for all probes in cluster 1 should satisfy the inequality in τg  should satisfy the inequality in c1 = Φ−1(0.95).It would be too stringent to require that c2 = Φ−1(0.05).Similarly, for probes in cluster 2 (cluster of UE probes) we requireψ = , where μ1 = exp(δ1), k1 = Φ(ξ1), α1 = exp(λ1), β1 = exp(ν1), μ2 = −exp(δ2), k2 = Φ(ξ2), α2 = exp(λ2), β2 = exp(ν2), α3 = exp(λ3), β3 = exp(ν3), andTo make sure the parameters satisfy the constraints in numerical optimization, we re-parameterized the parameters by f1(dg|ψ), f2(dg|ψ) and f3(dg|ψ) as the marginal densities of the 3 clusters, respectively. The formulae for these 3 marginal densities are shown in Section B in π =  as the cluster proportions. We impose a symmetric Dirichlet D(b) prior on π with concentration parameters b =  =  to stabilize the estimate of π. We would like to choose the value for b so that the mixture proportions are most likely to be equal (π1 = π2 = π3 = 1/3). Any value b > 1 would satisfy this condition since the mode of D(b) is 1/3, which does not depend on b. Following ) over parameters π and ψ. We repeat these two steps until the difference of the parameters π and ψ between two consecutive iterations is small or the number of iterations exceeds the allowed maximum number. Details about the marginal distributions and the EM algorithm are shown in Sections B and C in g will be classified to cluster c if the posterior probability The EM algorithm is used to estimate parameters π1f1(dg|ψ), a blue line for π2f2(dg|ψ), a black line for π3f3(dg|ψ) and brown line for the summation of these three weighted density functions. However, since dg is a vector of multiple dimensions, it would be very difficult to visualize π1f1(dg|ψ), π2f2(dg|ψ) and π3f3(dg|ψ). To provide a rough plot for these weighted density functions, we set dg to be one dimension, that is, it only contains information from one sample, to approximate the actual weighted densities.We intend to plot density functions in one plot with a red line for GSE43292 [GSE43292  is from GSE24742 [GSE24742  is from GSE6631 [The study associated with GSE6631  aimed tohttps://www.ncbi.nlm.nih.gov/geo/ and performed quality checking before further analysis. First we checked if there were any missing values, duplicated samples or duplicated subjects. We used lumiT in Bioconductor package lumi to test if the original dataset had been log2 transformed; if not, a log2 transformation was performed. We found that GSE24742 and GSE6631 were not log2 transformed, while GSE43292 had already been log2 transformed. To check the existence of outliers, for each array we calculated its 0-th, 5-th, 25-th, 50-th, 75-th, 95-th, and 100-th percentiles of expression levels and viewed them across all arrays. We also obtained the principal components (PCA) of gene expression data and plotted the first component against the second component. Please refer to Figs A, B and C in We downloaded datasets from 2 difference of expression levels within a pair of samples were generated from the eLNNpaired model. We used the model parameters estimated from GSE43292 as the true values of the model parameters. That is: μ1 = 0.441, k1 = 0.118, α1 = 1.718, β1 = 0.029, μ2 = −0.442, k2 = 0.079, α2 = 1.766, β2 = 0.034, α3 = 2.138, β3 = 0.131, π1 = 0.086, π2 = 0.071, and π3 = 1−π1−π2 = 0.843. We considered two scenarios to evaluate the effect of sample size on the performance of probe detection. In the first scenario (denoted by G30), we generated 100 datasets using this model, each of which has 1000 genes and 30 pairs of samples. In the second scenario (denoted by G100), we generated 100 datasets using this model, each of which has 1000 genes and 100 pairs of samples.We conducted two sets of simulation studies. In the first set of the simulation studies, log2 difference of expression levels within a pair of samples using three simple normal distributions, separately. For over-expressed gene probes, we assumed that the log2 differences follow 2 differences follow 2 differences follow μ1 = −μ2 = 2 and σ1 = σ2 = 1, σ3 = 2. In addition, we set the proportion of the over-expressed and under-expressed gene probes as 5 percent respectively. We considered 2 scenarios. In the first scenario (denoted by S30), we generated 100 datasets using this model, each of which has 1000 genes and 30 pairs of samples. In the second scenario (denoted by S100), we generated 100 datasets using this model, each of which has 1000 genes and 100 pairs of samples.In the second set of the simulation studies, we generated logMBHM models could handle data from paired/matched designs. We identified two regularized regression models that could handle data from paired/matched designs [eLNNpaired with several existing hypothesis-based gene selection methods that can handle data from paired/matched design: linear models for microarray data (limma) [gt) [samr) [lmPerGene) [To best of our knowledge, no existing  designs , 16. How (limma) , global ma) [gt) , 18, sig) [samr) , and linPerGene) . Using tLimma and samr are essentially paired t-tests with an adjustment for the variance of the mean within-pair difference of gene-expression levels. Limma uses probe-specific adjustment based on an empirical Bayesian approach, while samr used a fixed constant as adjustment. Gt and lmPerGene are linear regression approaches, in which the outcome variable measures the within-pair difference of gene expression levels and a non-zero intercept indicates differential gene expression. A positive (negative) test statistic indicates that the gene probe is over (under)-expressed.limma, gt, samr, and lmPerGene, a gene is detected as differentially expressed if the FDR-adjusted p-value is <0.05. For samr, FDR-adjusted p-values were based on 100 permutations.For eLNNpaired. The five agreement indices are Rand index (Rand), Hubert and Arabie’s adjusted Rand index (HA), Morey and Agresti’s adjusted Rand index (MA), Fowlkes and Mallows’s index (FM), and Jaccard index (Jaccard) [HA and MA correct for chance agreement and were recommended by [Five agreement indices and four error rates are used to evaluate the performance of Jaccard) . HA and ended by . For perThe four error rates are false positive rate (FPR), false negative rate (FNR), false discovery rate (FDR), and false non-discovery rate (FNDR). FPR is the percentage of detected DE probes among truly NE probes. FNR is the percentage of detected NE probes among truly DE probes. FDR is the percentage of truly NE probes among detected DE probes. FNDR is the percentage of truly DE probes among detected NE probes.For real data sets in which true gene cluster membership is unknown, we applied the Random Forest classification algorithm  to predieLNNpaired and to compare its performance with limma, gt, samr, and lmPerGene.We used both real datasets and simulated datasets to evaluate the performance of We downloaded from Gene Expression Omnibus (GEO) three gene expression datasets , all of which used paired designs to collect samples and have been preprocessed by their submitters to ensure good quality of the data. We performed further quality checking to clean the data.  and under-expressed (UE) genes. For non-differentially expressed gene probes, we denote them by NE. We then used cross table to compare the 3-cluster partitions obtained by the 5 methods  by ble c.f. .eLNNpaired are in accordance with limma, gt, samr, and lmPerGene; all gene probes, except for 42 gene probes, classified as UE by eLNNpaired are in accordance with the other 4 methods. The 1,817 gene probes that were classified as UE by eLNNpaired were claimed as OE or UE by limma. The 1,663 gene probes that were classified as UE by eLNNpaired were claimed as OE or UE by gt. The 3,611 gene probes that were classified as UE by eLNNpaired were claimed as OE or UE by samr. The 2,321 gene probes that were classified as UE by eLNNpaired were claimed as OE or UE by lmPerGene.For GSE6631, all gene probes classified as OE by eLNNpaired are similar to those of limma, gt, samr, and lmPerGene.We compared the prediction power of the DE probes obtained by the five probe-detection methods to predict disease statuses of subjects by using the Random Forest algorithm. ROC curves and precision-recall curves are shown in Figs J and K in eLNNpaired model parameters for the 3 GEO datasets are shown in The estimates of the eLNNpaired by two sets of simulation studies. In the first set, the simulated data were generated from the eLNNpaired model. The model parameters estimated from GSE43292 were used as the true parameter values. In the second set, the simulated data were not generated from eLNNpaired model. For each set, we generated 100 simulated datasets, each of which contained expression levels of 1000 gene probes for n pairs of samples. To evaluate the effect of sample size, we investigated two scenarios: n = 30 pairs and n = 100 pairs for each set of simulation studies. We denoted the first set of simulation as G30 and G100 respectively for n = 30 and n = 100. Similarly, we denoted the second set as S30 and S100 respectively for n = 30 and n = 100.In this section, we evaluated the performance of eLNNpaired performed better than the other 4 methods in terms of agreement indices, FDR and FPR and (2) eLNNpaired had similar FNDR and FNR to limma, gt, samr, and lmPerGene.Tables MBHM methods to analyze genomic data collected from paired/matched designs. The proposed model does not involve hypothesis testing; hence it does not have the problem of the curse of dimensionality. The proposed model can also borrow information across genes to estimate hyper-parameters, which makes it useful for data with small sample sizes.In this paper, we aimed to extend existing limma, gt, samr, and lmPerGene in terms of agreement indices in the simulation studies.The performance of the proposed model in detecting DE gene probes worked better than the existing hypothesis-based methods limma, gt, samr, and lmPerGene, although the proposed model detected much fewer DE probes than the other four methods.Both simulation studies and real data analyses showed that the proposed model had similar error rates and prediction accuracy to MBHM methods is that it introduces constraints on the model hyper-parameters to reduce false discoveries. More stringent constraints could result in fewer positives and a reduction in false discoveries. One possible benefit of the constraint setting is to make it adaptive to different datasets. Specifically, we can first set constraints empirically, then compare the derived results with what limma provides. If a gene is classified by limma as over-expressed but by our model as under-expressed or the other way round, we assume that limma is correct and tighten our constraints by a small amount. If no such genes are discovered, we loosen our constraints in the same manner. Under these new constraints, we run our model again. We repeat this procedure until we reach a critical point where we find as many positives as possible, while also avoiding false discoveries to a large extent.One advantage of the proposed model over the existing limma. This can be reduced by setting the constraints stronger. For instance, we can set c2 = Φ−1(0.025) instead of Φ−1(0.05), and we will get a new cross table with less number of false UE  method) to estimate model hyper-parameters.To simplify the model building and parameter estimation, we assume that the within-pair logeLNNpaired algorithm to an R package Click here for additional data file.S2 File(GZ)Click here for additional data file.


LINCS L1000 is a high-throughput technology that allows gene expression measurement in a large number of assays. However, to fit the measurements of ~1000 genes in the ~500 color channels of LINCS L1000, every two landmark genes are designed to share a single channel. Thus, a deconvolution step is required to infer the expression values of each gene. Any errors in this step can be propagated adversely to the downstream analyses.l1kdeconv based on a new outlier detection method and an aggregate Gaussian mixture model (AGMM). Upon the remove of outliers and the borrowing information among similar samples, l1kdeconv showed more stable and better performance than methods commonly used in LINCS L1000 data deconvolution.We presented a LINCS L1000 data peak calling R package l1kdeconv package achieved more stable results than the commonly used LINCS L1000 data deconvolution methods.Based on the benchmark using both simulated data and real data, the  The NIH Common Fund’s Library of Integrated Network-based Cellular Signatures (LINCS) is a rich collection of gene expression data from a variety of human cell lines perturbed with a large battery of drugs and small molecules , 12. So k-medians clustering algorithm which partitions all the data into two distinct components, and the median expression values are assigned as the expression value of the two gene of a color. Liu et al. [k-medians or naïve GMM [This peak deconvolution step is necessary because each bead color is associated with two genes rather than one. For each color, deconvolution is supposed to yield a distribution that generally consists of two peaks. The default approach provided by LINCS is the u et al. , 9 propoaïve GMM . Thus, il1kdeconv, that implements a new peak-calling algorithm based on an aggregate Gaussian mixture model (AGMM) along with an outlier detection method for improving the robustness of AGMM. AGMM is based on the Gaussian mixture model and borrows information from the samples of the same condition to improve the peaking calling accuracy.Here, we presented an R package, Gaussian mixture models based on clustering analysis methods in general are sensitive to outliers . To imprdy_thr (0.0001 as the default), the corresponding regions are recognized as data free gaps. Using these data free gaps, the beads are split into different clusters, as shown in Fig. clustersize_thr (3 as the default), the beads in it will be identified as outliers. See Fig. Given the bead intensities of a color, we first estimated the kernel density of the bead intensities using the R function density. When the estimated density is less than the threshold To accurately detect the expression values of a pair of genes, we introduced a novel peak calling method to borrow the information of the samples at the same condition . This method is an extension of naïve GMM with a constraint that the order of the gene expression values of two genes of the same color is consistent across these similar samples.X of the beads measuring any given gene follow a Gaussian distributionμ and σ2 as the mean and variance of bead intensities of the gene. Since each bead of a color can come from any of the two genes of the color, a two-component Gaussian mixture model was used to estimate the mean intensity of each gene. Due to the limited number of beads of each color, in order to have robust estimates of the intensity variance of each gene of a color, a common variance σ2 was assumed. Therefore, the probability density function of the bead intensities for a given color in a sample is.λ is the proportion of the beads of the first gene and 1 − λ is the proportion of the beads of the second gene, μ1and μ2 are the mean intensities of the first and second genes respectively.After the outliers were removed in the outlier-detection step, the two peaks of the pair of genes of a given color can be computed in the following way. We assumed the color intensities Because the two genes assigned to a color were selected in a way such that their intensity difference was maximized , 9, thism similar samples, we define the aggregate intensity vectors of a given color as {x1, x2,  … , xm}, where each i-th sample and ni is the number of beads. The log-likelihood of the AGMM is:λ is the proportion of the beads of the gene with a higher gene expression and 1 − λ is the proportion of the beads of the gene with a lower gene expression, μ1 = {μ1 , 1,  … , μm1 , } and μ2 = {μ2 , 1,  … , μm2 , } are the vectors of the two mean intensities in the m samples with the given color, and σ is the standard deviation across these m samples with the given color. The maximum likelihood method can be used to estimate the parameters  by optimizing the log-likelihood function of the AGMM. To enforce the differences between μ1and μ2 of the same colors across similar samples having the same sign, we reparametrize the log-likelihood as follows so that box constraint optimization can be used:The detailed method was described as below using an aggregate Gaussian mixture model (AGMM). Supposing there are δ and μ are defined asl1kdeconv package contains a novel peak calling algorithm for LINCS L1000 data, using aggregate Gaussian mixture model, an extension of the naïve GMM. Rather than fitting the bead intensities of each color in a single sample using two-component Gaussian mixture model separately, AGMM calls the two peaks of each color across similar samples simultaneously. In addition, the outlier detection method included in l1kdeconv can also enhance the clustering performance of AGMM. Figure The k-means is used to compute the rough estimates of the mean intensities, which are used as initial values for the optimization. To make the package user-friendly, detailed help pages and running examples are provided in the package.The parameters of AGMM can be estimated using the R function optim by the method of maximum likelihood. The L-BFGS-B method , 13 is ul1kdeconv, a simulated dataset was created with the key characteristics of LINCS L1000 data using a hierarchical model as described below. First, the empirical true expression values of each pair of genes were extracted by k-medians using the LINCS L1000 data of the A375 cell line treated with luciferase. Since the differences and averages of the empirical true expression values approximately followed a Gamma distribution and a Gaussian distribution respectively, these two distributions were used to resample the differences and the averages of the mean expression values of the two genes of any color. These differences and averages were then converted to the mean expression values of each pair of genes. The bead intensities were simulated from a two-component Gaussian mixture model with these mean expression values as the center of each Gaussian component. Then outliers were added to each color using a uniform distribution, and the number of outliers was generated from a Poisson distribution. Ten samples were created so that AGMM can borrow information across them. In each sample, there were 500 colors. In each color, there were roughly 60 beads and the ratio of the number of the beads for each gene in a color was 2:1.To make a realistic evaluation of the performance of k-medians, naïve GMM and AGMM with outlier detection in the l1kdeconv package. To have a fair comparison, we compared our AGMM with k-medians and naïve GMM that are also used in the standard LINCS L1000 deconvolution method. Figure k-medians (a) and naïve GMM (b). The Pearson Correlation Coefficients (PCC) are 0.76, 0.89 and 0.97 for k-medians, naïve GMM and AGMM with outlier detection, respectively. Then a test of significance for the difference between the two correlations based on dependent groups is conducted. The Williams’s Test [k-medians and naïve GMM with the p-value less than 1 × 10−20.Using the simulated data, a series of comparisons were conducted among s’s Test , 5 betwek-medians and naïve GMM, respectively. In addition, AGMM without outlier detection results in a reduction of 6%, indicating the importance of the outlier detection as a filtering step before calling AGMM.Comparisons were made among these methods with respected to the true prediction which is defined a prediction whose difference from the real value is less than 0.05. Table k-medians and naïve GMM lack this ability, thus they produced many flipping cases.To further benchmark the accuracy of the three methods, we compared whether the order of the predicted two peaks were flipped. Figure k-medians and naïve GMM. To demonstrate the performance improvement of l1kdeconv on real data, we used a dataset of A375 cells treated with luciferase from LINCS L1000 Phase II data (GSE70138). Eleven replicates were used to borrow information across these replicates. Unlike calling the peaks of each replicate separately by k-medians and naïve GMM, AGMM filters the outlier of each sample, and then deconvolutes them in one group. Figure k-medians (a and c) and naïve GMM (b) were unable to identify the two peaks correctly as the order of the two peaks are flipped. However, AGMM with outlier detection can identify the two peaks correctly because the information in the easy cases of d - i can be used to help with the deconvolution of the difficult ones . In the cases d, e and f, although the results from k-medians and naïve GMM are not flipped, AGMM with the outlier detection can fit the real data better. The peak coordinates predicted by k-medians, naïve GMM and AGMM with outlier detection in Fig. The comparison using the simulated data confirmed that AGMM with outlier detection is more stable and more accurate than l1kdeconv package provides a stable and accurate deconvolution algorithm for LINCS L1000 data. Because the deconvolution is the first step of the analysis of LINCS L1000 data, any significant improvement in this step will have a critical impact on the downstream analyses. The l1kdeconv package has two components --- the outlier detection and aggregated Gaussian mixture model. By filtering the outliers and borrowing information from similar samples, l1kdeconv outperforms k-medians and naïve Gaussian mixture model using the limited number of intensity values.The http://cran.r-project.org.The package also provides detailed help pages, which makes it user friendly. Furthermore, we standardize the distribution, installation and maintenance of this package. It is available on the Comprehensive R Archive Network (CRAN) at


Procedures for conducting qPCR have received significant attention; however, more rigorous approaches to the statistical analysis of qPCR data are needed.Cq) and efficiencies of reactions (E). The Common Base Method keeps all calculations in the logscale as long as possible by working with log10(E) ∙ Cq, which we call the efficiency-weighted Cq value; subsequent statistical analyses are then applied in the logscale. We show how efficiency-weighted Cq values may be analyzed using a simple paired or unpaired experimental design and develop blocking methods to help reduce unexplained variation.Here we develop a mathematical model, termed the Common Base Method, for analysis of qPCR data based on threshold cycle values (The Common Base Method has several advantages. It allows for the incorporation of well-specific efficiencies and multiple reference genes. The method does not necessitate the pairing of samples that must be performed using traditional analysis methods in order to calculate relative expression ratios. Our method is also simple enough to be implemented in any spreadsheet or statistical software without additional scripts or proprietary components. The use of quantitative polymerase chain reaction (qPCR) for diverse applications has increased dramatically –4 since Cq) and the efficiency of the reaction (E). Current methods used to analyze qPCR data utilize at a minimum the Cq values. The Cq values are derived from a logistic curve that plots the growth of a population of amplicons produced through the use of sequence-specific primers , genes g [gene of interest or reference gene], and technical replicates located in wells i, we obtain data pointsE, Cq) =  for each well , we calculate a single value log10(E) ∙ Cq , which we call the efficiency-weighted Cq valuer, sample type t, and gene g, we then calculate Cq value over all n technical replicate wells, i.e.,From each pair of values (w) is a label to denote the use of efficiency-weighting on the Cq values and does not denote exponentiation.Please note that the superscript Cq; r, t, g, i, and R theoretically matches that of Pfaffl ,  [sample type A vs. C], and  [sample type B vs. C]. Notice that the first interval exceeds 1, showing that the gene expression for sample type A is significantly larger than that for B. The second interval includes 1, meaning that the gene expression is not significantly different between sample type A and C. The third interval is completely below 1, showing that the gene expression for sample type B is significantly smaller than that for C.In the following example Table , an expe2,9 = 4.064; P-value = 0.055).The purpose of blocking is to increase sensitivity by reducing unexplained variation . That isSome , 19 havet-test to the data. Similar to the last example5, we are testing whether the mean of differences is different from 0.For experiments with paired samples, each biological replicate of sample type A is directly paired with a replicate of sample type B. Common situations would involve sample replicates of two types harvested from the same organism or geographic location, or the expression of a particular gene before and after some experimental treatment is applied to an individual Fig. , Inset Bt-test statistic of −3.48, 95% confidence interval of , and P-value of 0.040 using SPSS software ,  [sample type A vs. C], and  [sample type B vs. C]. Notice that the first interval exceeds 1, showing that the gene expression for sample type A is significantly larger than that for B. The second interval includes 1, meaning that the gene expression is not significantly different between sample type A and C. The third interval is completely below 1, showing that the gene expression for sample type B is significantly smaller than that for C.The analysis shows that we may reject the null hypothesis .The advantage of the common base method lies in the use of the common base 10 (or any other base of choice) to force all of the data-based calculations into the logscale and the flexibility to incorporate ciencies , averageciencies , or geneciencies , 15. Givciencies , 30–32, We caution against a few potential pitfalls that may arise from improper analysis of qPCR results. First, avoid grouping data values unless there is a biological motive for the pairing of samples, such as the samples are blocked on the same qPCR plate. For example, the work in Table n), while averages calculated for relative expression ratios should be done with geometric means (multiply the items and take an nth root). The different use of means is directly related to the exponential identity axay = ax + y where addition in the exponent corresponds to multiplication at the base.Second, use the appropriate type of mean. Averages calculated in the logscale , then the mean expression ratio is 0.333 with a P-value of 0.005, which shows a significant difference in gene expression. The same data cannot both reject and fail to reject the hypothesis that the relative expression ratio of the sample types is different from 1.Consider the paired sample data from Table t-test from the information in Table t-test statistic, and the confidence interval. The standard deviation and P-value remain the same. Consequently, the test will have the same significance result and, after calculating The interested reader can confirm that our methods are immune to this problem by running a paired Though analysis is conducted using log-transformed n et al.  in findientclass1pt{miniman et al. , Di Stefn et al. , and Nakn et al. . Note thIn this article we have presented a Common Base Method for use in the statistical analysis of relative expression ratios arising from qPCR experiments. The model is presented in Eqs. E calculated in a variety of different manners, whether averaged across plates, genes, or from specific wells.The Common Base Method has advantages over current methods for analyzing qPCR data. The primary advantage is that the model keeps all calculations in the logscale as long as possible. Staying in the logscale allows one to use arithmetic means instead of geometric means and opens up a larger world of parametric statistical tests that cannot be validly applied at the level of the relative expression ratio. Although we contained our examples to two types of experimental designs, unpaired and paired, the Common Base Method can be adapted to include other analyses within the general linear model. The technique of blocking in such experiments can increase power, and the design of the qPCR plate experiment deserves attention. The use of multiple blocking factors is also possible and the appropriate analysis of such experiments warrants future attention. The potential utility of the Common Base Method suggests that there is great value in determining whether relative expression ratios are lognormally distributed in general. Our method also has the flexibility to be adaptable to efficiency values While our use of logscale calculations is not necessarily groundbreaking on its own, we believe that our model presents these concepts in a more accessible manner that will allow easier adaptation for researchers who are not necessarily experts in statistics or bioinformatics. The simplicity of our model and its ability to be quickly calculated in any spreadsheet software is its primary strength.


High-throughput technology allows for genome-wide measurements at different molecular levels for the same patient, e.g. single nucleotide polymorphisms (SNPs) and gene expression. Correspondingly, it might be beneficial to also integrate complementary information from different molecular levels when building multivariable risk prediction models for a clinical endpoint, such as treatment response or survival. Unfortunately, such a high-dimensional modeling task will often be complicated by a limited overlap of molecular measurements at different levels between patients, i.e. measurements from all molecular levels are available only for a smaller proportion of patients.We propose a sequential strategy for building clinical risk prediction models that integrate genome-wide measurements from two molecular levels in a complementary way. To deal with partial overlap, we develop an imputation approach that allows us to use all available data. This approach is investigated in two acute myeloid leukemia applications combining gene expression with either SNP or DNA methylation data. After obtaining a sparse risk prediction signature e.g. from SNP data, an automatically selected set of prognostic SNPs, by componentwise likelihood-based boosting, imputation is performed for the corresponding linear predictor by a linking model that incorporates e.g. gene expression measurements. The imputed linear predictor is then used for adjustment when building a prognostic signature from the gene expression data. For evaluation, we consider stability, as quantified by inclusion frequencies across resampling data sets. Despite an extremely small overlap in the application example with gene expression and SNPs, several genes are seen to be more stably identified when taking the (imputed) linear predictor from the SNP data into account. In the application with gene expression and DNA methylation, prediction performance with respect to survival also indicates that the proposed approach might work well.We consider imputation of linear predictor values to be a feasible and sensible approach for dealing with partial overlap in complementary integrative analysis of molecular measurements at different levels. More generally, these results indicate that a complementary strategy for integrating different molecular levels can result in more stable risk prediction signatures, potentially providing a more reliable insight into the underlying biology.The online version of this article (doi:10.1186/s12859-016-1183-6) contains supplementary material, which is available to authorized users. When molecular measurements are obtained from patients in a clinical cohort by high-throughput techniques, the number of covariates is typically much larger than the number of observations. There are established statistical techniques for linking measurements from a single molecular level to clinical endpoints such as survival. In particular, techniques for regularized estimation of regression models, such as the lasso  or compoIn more recent studies, microarray and sequencing technologies are used to measure a large number of patients’ characteristics at different molecular levels simultaneously. For example, the data that motivated us to develop an integrative analysis strategy comes from a study on acute myeloid leukemia (AML) patients with single nucleotide polymorphism (SNP) microarray measurements  and micrhttps://tcga-data.nci.nih.gov/tcga/). Due to a multitude of reasons , not every type of measurement is available for each individual. As a straightforward complete case analysis would discard much of the available information, partial overlap has to be addressed before an integrative analysis can be performed.Such an integrative analysis is complicated by having a partial overlap, i.e. not all molecular levels are measured for all patients. For example in our AML application, there are some patients with measurements available from both genome-wide measurement platforms, but some patients have measurements from only one single molecular level, either from the microarray-based GEP or from the SNP microarray. An indication that this is a more widespread problem is provided, e.g. by the data available in the Cancer Genome Atlas (TCGA) , and then details the proposed sequential complementary strategy, including a stepwise estimation procedure and an imputation approach. In the results section, the sequential complementary strategy is illustrated for SNP and microarray-based gene expression data in the first AML application and for methylation and gene expression sequencing data in the second AML application. In the subsequent section, the results are discussed and more general concluding remarks are provided.Acute myeloid leukemia (AML), the most common acute leukemia in adults, represents a genetically heterogeneous disease with respect to clinical outcome, such as time to relapse or death. Currently, several clinically relevant molecular markers are known that characterize AML at the molecular level . An inteNPM1) and fms-related tyrosine kinase 3 internal tandem-duplication (FLT3−ITD) genes as well as a cytogenetic risk group factor are available for all AML patients.In the two motivating data examples considered in the following, the time-to-event clinical endpoint of interest is survival , i.e. time to relapse or death from any cause, whatever happens first, in a first application, and overall survival in a second application). In the first application, measurements from two different molecular levels, single nucleotide polymorphism (SNP) information from microarrays  and micrhttps://tcga-data.nci.nih.gov/tcga/).The second application, considers methylation and gene expression sequencing data as well as clinical information from the Cancer Genome Atlas (TCGA). Data sets were not available for all individuals on all platforms because of same availability, assay failure and quality issues. DNA methylation information for 194 patients were generated from the Illumina Infinium HumanMethylation450 BeadChip for bisulfite-treated DNA. There are measurements of methylation status for 485577 CpG sites for each patient. After pre-processing steps, 396065 methylation features remain. Twelve samples were excluded due to missing outcome data. Of these 182 AML patients, 118 AML cases died during their follow-up. In addition to methylation measurements, there are gene expression sequencing data (RNA-seq) for 179 AML cases, comprising 20442 RNA-seq measurements per patient. Ten of 179 AML cases were excluded due to missing outcome information. As mentioned above, death is the event of interest. This event was observed in 108 out of 169 AML cases. Since our focus lies on the integrative analysis of molecular measurements at different levels while taking clinical predictors into account, two clinical covariates  available for all AML cases are investigated. In total, there is an overlap of 166 AML cases with information from both types of molecular levels. The data can be downloaded directly from the TCGA website  and SNP data, and the second considering gene expression sequencing (RNA-seq) and methylation data, to illustrate the general applicability of our proposal. Yet, notation and explanation will focus on the combination of GEP and SNP data (first AML application) in the following, for simplicity.Specifically, we propose a sequential complementary strategy for an integrative analysis of different molecular levels with potentially partial overlap in the biological samples. We first provide two risk prediction models, for the SNP data and the GEP data respectively, followed by an imputation approach for dealing with partial overlap, and finally the technical details of model fitting and a brief discussion of the order in which SNP and GEP data are considered for modeling. A flowchart describing the sequential complementary strategy based on the stepwise procedure and the imputation approach is given in Fig. ti,δi,zi), i=1,…n, where ti is the observed time for an individual, which is given by ti= min, with event time Ti and censoring time Ci. The event indicator δi=I(Ti≤Ci), with indicator function I(.), takes value 1 if its argument is true, and 0 otherwise. The covariate vector zi=′ comprises covariates known at baseline. We assume that it can be partitioned into a vector d clinical covariates, a vector k SNP covariates, and a vector q gene expression covariates. In the following, the vectors Censored time to event data, e.g. as in the time to relapse AML application above, are typically given as triples  likelihood techniques without penalization.The hazard This Cox proportional hazard model for the SNP data can be used for guiding the fit of a Cox proportional hazard model for the microarray-based GEP data for identifying genes whose expression provides prognostic information beyond the SNP data. In particular, we propose to use regularized estimation for fitting a model αGEP for the clinical covariates is different from αSNP in model   2) for tComponentwise likelihood-based boosting starts with estimated parameter vectors equal to zero and builds up estimates using a stepwise algorithm with a potentially large number of boosting steps. To obtain sparse fits, as intended in high-dimensional data, only one element of the parameter vector is updated in each boosting step. To identify the element to be updated, a candidate model is fitted for each covariate in each boosting step, comprising only this covariate and a fixed offset that contains the information from the previous boosting steps. The parameters in these models are estimated by a penalized maximum  likelihood approach, and finally the element to be updated is chosen to be the one that maximizes the  log-likelihood. Componentwise likelihood-based boosting can distinguish between mandatory covariates, which are always included in the model, e.g. established clinical predictors that need to be adjusted for, and optional covariates. While the parameters for the optional covariates are estimated as described above, the parameters for the mandatory covariates are estimated by standard maximum  likelihood techniques. For estimating the parameters in the three regression models , 2), an, an2), aThe proposed strategy integrates data from two molecular levels with potentially small overlap in the biological samples, while taking established clinical predictors into account. The sequential approach is not based on the assumption of direct links between molecular levels as both sources are used in different roles. The sequential order of the different data sources depends on the underlying question of the integrative analysis. Such an order can be based on the biological relations, such as between SNP and microarray-based GEP data or can be swapped, e.g. first a prognostic model for GEP data could be fitted, and the linear predictor from this model could then be used for adjustment in prognostic model building from the SNP microarray data. While this order could be considered to be contrary to biological relations, such analyses could nevertheless be performed. However, the sequential order should be prespecified before starting the integrative analysis, to avoid decision making based on the results.For the first application with GEP and SNP data, the SNP microarray data are analyzed first and the results are used for guiding GEP features that contain information beyond what is already provides by the SNPs in the following, reflecting the biological relation between SNP and GEP data. For the second application, investigating gene expression sequencing (RNA-seq) and methylation data, we consider RNA-seq data first and then analyze the methylation data adjusted for RNA-seq effects, thus potentially better highlighting the signals in the methylation data that are not directly visible in the RNA-seq measurements.In a first application, we illustrate the proposed approach for a complementary integrative analysis of SNP and microarray-based GEP data from AML patients with respect to relapse free survival. As a reference approach we fit a risk prediction model for the GEP data without adjusting for the effects of the SNP data. The model for the reference approach will be estimated again by componentwise likelihood-based boosting. As criteria for comparison, we will consider stability of selected signatures, as quantified by resampling inclusion frequencies, as well as prediction performance.i=1,…,308 and j=1,…,390443. Shrunken estimates, as obtained by the boosting technique, then take the meaning of ’no/small’-dominant and ’no/small’ recessive effects. The predictors age, WBC, NPM1 and FLT3−ITD status as well as a cytogenetic risk group factor are included as mandatory covariates in addition to the optional SNP covariates in componentwise likelihood-based boosting used to obtain sparse fits from the SNP data.For modeling the effects of the SNPs, we use a coding scheme with two dummy covariates per SNP to distinguish between dominant and recessive effects. Minor allele frequencies are coded as eference  for examTwenty-three SNPs are selected for the SNP risk prediction model , i.e. thTo address the partial overlap in the biological samples, imputation of the SNP signature is performed by fitting linking model  for the To judge the model fit on the overlap samples the mean squared error is determined by the bootstrap.632+ technique , which iFor extracting complementary information from the GEP data, the SNP signature is incorporated as a fixed offset into the risk prediction model  using thSome IMAGE  cDNA clones are not linked to genes, but rather represent expressed sequence tags (ESTs), whereas other IMAGE clones represent annotated genes. Note that single genes can be represented by several IMAGE clones, similar to Affymetrix arrays where single genes are often covered by numerous probe sets.Stability quantified by resampling techniques is considered to judge model quality. The underlying idea is to perform model building (including variable selection) in several resampling data sets and to consider for each covariate the proportion of final models derived from the resampled data sets where it has been selected. See for example , 16 for n. The resampling inclusion frequency algorithm is illustrated in Additional file In the following, feature selection of the molecular entities is performed for each of 100 resampling data sets, i.e. the sequential complementary strategy as well as the reference approach are applied. Resampling can be performed with replacement, corresponding to the bootstrap, or without replacement, i.e. subsampling. However, resampling with replacement introduces a bias that will affect approaches that select a tuning parameter, e.g. the number of boosting steps in each resampling data set, which is selected according to prediction performance to avoid overfitting using 10-fold cross-validation. See Reference  for moreIn the first AML application, we consider the inclusion frequency for each gene selected in the original GEP data by the sequential complementary strategy (33 genes compared to 34 IMAGE clones) and by the reference approach (24 genes compared to 27 IMAGE clones) shown in Fig. Specific components of the sequential complementary strategy, such as boosting, can be replaced by other modeling techniques. In particular, by penalized regression approaches, such as lasso . To carrTo investigate whether the sequential complementary strategy provides reasonable prediction performance and to additionally verify that the extracted complementary information from the GEP measurements after adjusting for the effects from the SNP microarray is not only due to noise, the prediction performances of the sequential and reference approach are investigated. For evaluating the prediction performance, the Brier score , i.e. a Figure Figure One might expect that the relative size of overlap has a considerable impact on the performance of the sequential complementary strategy. Therefore, in the first AML application, we systematically decreased the size of overlap from 26 to 10 biological samples. This is done by randomly drawing 15 and 10 biological samples, respectively, from the 26 overlap samples with available measurements from both molecular data in parallel. The remaining biological samples, which are not randomly drawn, are treated as if they would have measurements from the GEP data only. Figure In addition, to examine the effect of the overlap size on the performance of the linking model  used forA swapped order of molecular levels within the sequential complementary strategy is investigated in the first AML application by analyzing first the GEP data and using the corresponding linear predictor for adjustment in a risk prediction model based on the SNP microarray data. Boxplots of the resulting integrated prediction error estimates are shown in Additional file In the second AML application, we consider integration of methylation and gene expression sequencing measurements with respect to overall survival. For the methylation data, we use M-values, which are simply a transformation of the beta value . The latRecently, a multitude of techniques have become available to obtain genome-wide data sets at various molecular levels. Integrative analysis of such molecular data from different platforms might provide additional insight. For example, our motivation for the integrative analysis of various molecular levels came from an application with AML patients, where there is still huge unexplained heterogeneity within the defined molecular subtypes based on clinically relevant molecular marker from single genetic data. Our aim was to develop a strategy for integrating multiple genome-wide data sets in a complementary way, despite a potentially small overlap in the biological samples. This also is a potential alternative to parallel approaches which incorporate all information in once but require availability of all sources for all individuals. In the first AML application, we considered measurements SNP microarrays and microarray-based gene expression (GEP) measurement. Both molecular data sets need to be combined into an integrative data analysis for identifying prognostic signatures. In the second application, we considered integrative analysis for gene expression sequencing (RNA-seq) and methylation measurements.We developed a sequential complementary strategy that is a stepwise procedure based on componentwise likelihood-based boosting. The stepwise procedure analyzes first one molecular level individually and then the results from the first molecular data set are used for guiding the model for another molecular data set. Specifically, this means that the sequential complementary strategy is designed for extracting complementary information from the second molecular data set by adjusting for the effects from the first molecular level in the model for the second data set. The developed strategy is not based on the assumption of direct links between molecular levels. Such a sequential complementary strategy can take the biological hierarchies between the different molecular levels into account, e.g. focusing on GEP features that provide information beyond SNPs, or the order can be swapped. Also, more than two sources could potentially be investigated by considering several linear predictors.To assess the proposed sequential complementary strategy we considered a reference approach, where a prognostic model was fitted based on the GEP data without adjusting for the effects of the SNP microarray data in the first application. Twenty genes were identified by the sequential as well as by the reference approach. In addition to the twenty genes, thirteen genes were identified by the sequential complementary strategy only. This might be an instance where the sequential complementary strategy provides molecular features that were not selected by a risk prediction model considering a single molecular data set only. When considering resampling inclusion frequencies for judging signature stability, genes selected by the sequential complementary strategy in the original GEP data also received larger inclusion frequencies when using this strategy, as compared to the reference approach, further underlining that our strategy results in a structurally different stable set of prognostic genes.DNAL2 [NDFIP2 represents another interesting leukemia relevant candidate gene as it is an important element of the ubiquitin-proteasome system (UPS) that plays an essential role in the homeostasis of cellular protein traffic and degradation, regulating cell fate, together with autophagy and apoptosis. While disruption of UPS is essential for leukemogenesis, recently NDFIP2 was found deregulated and associated with blast counts in AML [NR2C2  coding for a ligand-activated transcription factor that is an important repressor of nuclear receptor signaling pathways such as retinoic acid receptor, which is crucial for the differentiation of myeloid cells. Similarly, the PRKCE  gene represents an interesting candidate belonging to the serine- and threonine-specific protein kinases that phosphorylate a wide variety of protein targets and are known to be involved in diverse cellular signaling pathways also important in the differentiation of myeloid cells.Importantly, in the first application, the list of genes identified from the GEP data after adjusting for the effects from the SNP data was biologically meaningful including both genes previously reported to be associated with outcome in AML such as DNAL2 . Similars in AML . In addiIn the second AML application, improved prediction performance compared to a reference approach indicated that the proposed imputation strategy works well for different overlap sizes, in particular still providing some gain even for a small overlap.The sequential strategy was developed to resolve the problem of the complementary integrative analysis of different molecular data sets with potentially partial overlap in the biological samples. It provided promising results in two applications to data from AML patients. While we explored it in the context of these specific applications, it does not depend on these examples and can also be applied to similar problems. Specific components within the stepwise procedure, such as componentwise boosting, might be replaced by other modeling approaches, e.g. lasso, thus providing a more general strategy. However, such modifications need further research in the future. Naturally, as the results from the AML application have only exemplary character, further research is needed for refining and validating of the proposed approach to arrive at such a more general strategy. However, the present results already indicate that such an approach might be promising.


RNA-sequencing (RNA-Seq) has become a powerful technology to characterize gene expression profiles because it is more accurate and comprehensive than microarrays. Although statistical methods that have been developed for microarray data can be applied to RNA-Seq data, they are not ideal due to the discrete nature of RNA-Seq data. The Poisson distribution and negative binomial distribution are commonly used to model count data. Recently, Witten  proposed a Poisson linear discriminant analysis for RNA-Seq data. The Poisson assumption may not be as appropriate as the negative binomial distribution when biological replicates are available and in the presence of overdispersion . However, it is more complicated to model negative binomial variables because they involve a dispersion parameter that needs to be estimated.In this paper, we propose a negative binomial linear discriminant analysis for RNA-Seq data. By Bayes’ rule, we construct the classifier by fitting a negative binomial model, and propose some plug-in rules to estimate the unknown parameters in the classifier. The relationship between the negative binomial classifier and the Poisson classifier is explored, with a numerical investigation of the impact of dispersion on the discriminant score. Simulation results show the superiority of our proposed method. We also analyze two real RNA-Seq data sets to demonstrate the advantages of our method in real-world applications.http://www.comp.hkbu.edu.hk/~xwan/NBLDA.Ror https://github.com/yangchadam/NBLDAWe have developed a new classifier using the negative binomial model for RNA-seq data classification. Our simulation results show that our proposed classifier has a better performance than existing works. The proposed classifier can serve as an effective tool for classifying RNA-seq data. Based on the comparison results, we have provided some guidelines for scientists to decide which method should be used in the discriminant analysis of RNA-Seq data. R code is available at  RNA-sequencing (RNA-Seq) is a revolutionary technology that uses the capabilities of next-generation sequencing to infer gene expression levels –3. CompaRNA-seq is different to the microarray technology that measures the level of gene expression on a continuous scale. It counts the number of reads that are mapped to one gene and measures the level of gene expression with nonnegative integers. As a result, popular tools that assume a Gaussian distribution in microrray data analysis, such as linear discriminant analysis, may not perform as well as those methods that adopt appropriate discrete distributions for RNA-Seq data. Law et al.  recentlyFor RNA-Seq data, the Poisson distribution and negative binomial distribution are two common distributions considered in the expression detection and classification. Many methods have been proposed to detect differentially expressed genes, including edgeR , 10, DESWe note that Witten  has consIn light of the importance of the dispersion in modelling RNA-Seq data with the negative binomial distribution, some dispersion estimation methods have been proposed recently in the literature. For example, Wu et al.  proposedWe extend Witten’s method  by buildWe further explore the relationship between NBLDA and PLDA. A numerical comparison is conducted to explore how the dispersion changes the discriminant score. The comparison results will provide some guidelines for scientists to decide which method should be used in the discriminant analysis of RNA-Seq data.In this paper, we propose a negative binomial linear discriminant analysis (NBLDA) for RNA-Seq data. The main contributions of this paper are in, but not limited to, the following two aspects: To demonstrate the performance of our proposed method, we conduct several simulation studies under different numbers of genes, sample sizes, and proportions of differentially expressed genes. Simulation results show that the proposed NBLDA outperforms existing methods in many settings. Three real RNA-Seq data sets are also analyzed to demonstrate the advantages of NBLDA. Specifically, we propose the negative binomial classifier, explore the relationship between NBLDA and PLDA, and present the parameter estimation in Section “Xig denote the number of reads mapped to gene g in sample i, i=1,…,n and g=1,…,G. To identify which class a new observation belongs to, Witten . The λg values are i.i.d. random variables from the exponential distribution with rate 0.04. Note that, for the sake of fairness, we have essentially followed the same simulation settings as those in PLDA. For the values of G, n, ϕ and σ, we specify them in Figs. The total number of classes is n test samples and compute the following misclassification rate: To compare these methods, we compute the mean misclassification rates as follows: for each simulation, we generate We run 1,000 simulations, compute its mean, and then obtain the mean misclassification rate. It is worth noting that Witten  discussen=8), NBLDA has a significant improvement over the other approaches.Figure G=20, 30, 50, and 100 for this investigation. From Fig. Figure ϕ=1, 5, 10, 20 and 30 for this investigation. Figure Figure Cervical cancer data [cer data . Two groHapMap data [Map data , 26. A tIn this experiment, we use two real data sets to further compare our methods with the other methods. We note that SVM and KNN are applied on the log-transformed counts of these two real data sets. The reason is that in real data sets, the number of genes is large and their counts may exhibit largely different distributions. In this situation, a few strongly expressed genes with very large counts may dominate those weakly expressed genes, which may decrease the performance of SVM and KNN. However, in our simulation experiment, the number of genes is not big (100 maximum) and the counts of all expressed genes in one particular data set are from the negative binomial distribution with a common dispersion parameter. Such a situation has little chance to happen. Therefore, we directly applied SVM and KNN on the raw counts in the simulation experiment. The details of these two data sets are described as follows: The Cervical cancer data was also used in . It is wFor real biomedical research in which RNA-Seq technology is used, it is common that thousands or tens of thousands of genes are measured simultaneously. We perform a gene selection procedure to screen the informative genes before applying a classification rule to RNA-Seq data. By doing gene selection, we rule out the noise as much as possible so that the variance of the discriminant score is reduced, and consequently we have an increased interpretability.The BSS/WSS method  is a comdkg towards 1. However, this method can not be applied to our discriminant analysis because the dispersion is involved in our discriminant rule. For the negative binomial distribution, edgeR [Witten  proposedn, edgeR , 10 has G genes for further analysis. We then randomly split the sample into two sets: the training set and the test set. The training set is used to construct the classifier and the test set is used to compute the misclassification rate. We repeat the whole procedure 1,000 times and compute the mean misclassification rate for the four methods, NBLDA, PLDA, SVM, and KNN, respectively.We first conduct the gene selection procedure using edgeR (version 3.3) and obtain The comparison results are shown in Fig. Finally, we estimate the medians of the dispersions of these two data sets to check if it also supports our comparison results made in the previous paragraph. The simplest way for estimating the dispersion is to use the method of moments. However, this estimate may not be reliable  when the sample size is small. Landau and Liu  and Yu eIn this paper, we have proposed an NBLDA classifier using the negative binomial model. Our simulation results show that our proposed NBLDA has a better performance than PLDA in the presence of moderate or high dispersions. When there is little dispersion in the data, NBLDA is also comparable to PLDA. We have further explored the relationship between NBLDA and PLDA, and investigated the impact of dispersion on the discriminant score of NBLDA by conducting a numerical comparison. It is worth noting that even for a small dispersion, the two discriminant scores can be rather different. This suggests that for real RNA-Seq data with moderate or high dispersion, NBLDA may be a more appropriate method than PLDA. Note that the true dispersions are unlikely to be known in practice. Therefore, we propose to first estimate the average dispersion using some novel estimation methods in the recent literature. Second, if the estimated average dispersion is small, we use PLDA; and otherwise we use NBLDA.We note that the independence assumption in both Witten’s method and our method is very restrictive. For real gene expression data sets, it may not be realistic to assume that all genes are independent of each other. In our future study, we would like to incorporate the network information of pathways or gene sets to further improve the performance of classification. The clustering of sequencing data is also an important issue in biomedical research. Hence, another possible future work is to extend the proposed clustering method  to folloNext generation sequencing technology has been widely applied in biomedical research and RNA-Seq begins to replace the microarray technology gradually in recent years. Since RNA-Seq data are nonnegative integers, differing from that of microarray data, it is necessary to develop methods that are well suited for RNA-Seq data. Two discrete distributions, the Poisson distribution and negative binomial distribution, are commonly used in the literature to model RNA-Seq data. Compared to the Poisson distribution, the negative binomial distribution allows its variance to exceed its mean and is more suitable for the situations when biological replicates are available. Nevertheless, the negative binomial model is more complicated than the Poisson model as the additional dispersion parameter also needs to be estimated. In this paper, we have developed a new classifier using the negative binomial model for RNA-seq data classification. Our simulation results show that our proposed classifier has a better performance than existing works. To conclude, our proposed classifier can serve as an effective tool for classifying RNA-seq data.