Clinical trials increasingly employ medical imaging data in conjunction with supervised classifiers, where the latter require large amounts of training data to accurately model the system. Yet, a classifier selected at the start of the trial based on smaller and more accessible datasets may yield inaccurate and unstable classification performance. In this paper, we aim to address two common concerns in classifier selection for clinical trials: (1) predicting expected classifier performance for large datasets based on error rates calculated from smaller datasets and (2) the selection of appropriate classifiers based on expected performance for larger datasets. We present a framework for comparative evaluation of classifiers using only limited amounts of training data by using random repeated sampling (RRS) in conjunction with a cross-validation sampling strategy. Extrapolated error rates are subsequently validated via comparison with leave-one-out cross-validation performed on a larger dataset. The ability to predict error rates as dataset size increases is demonstrated on both synthetic data as well as three different computational imaging tasks: detecting cancerous image regions in prostate histopathology, differentiating high and low grade cancer in breast histopathology, and detecting cancerous metavoxels in prostate magnetic resonance spectroscopy. For each task, the relationships between 3 distinct classifiers  are explored. Further quantitative evaluation in terms of interquartile range (IQR) suggests that our approach consistently yields error rates with lower variability  than a traditional RRS approach  that does not employ cross-validation sampling for all three datasets. A growing amount of clinical research employs computerized classification of medical imaging data to develop quantitative and reproducible decision support tools –3. A keyAnother issue in utilizing computerized image analysis for clinical research is the need to select the best classifier at the onset of a large-scale clinical trial . The selIn this paper, we aim to overcome the major constraints on classifier selection in clinical trials that employ medical imaging data, namely (1) the selection of an optimal classifier using only a small subset of the full cohort and (2) the prediction of long-term performance in a clinical trial as data becomes available sequentially over time.Given a small pilot dataset, can we predict the error rates associated with a classifier assuming that a larger data cohort will become available in the future?Will the relative performance between multiple classifiers hold true as data cohorts grow larger?To this end, we aim to address crucial questions that arise early in the development in a classification system, namely:a priori prediction of classifier performance using only a small dataset.Traditional power calculations aim to determine confidence in an error estimate using repeated classification experiments , but do K-fold cross-validation framework to ensure that all samples are used for both classifier training and testing  that characterize the behavior of error rate as a function of training set size.Due to the heterogeneity present in biomedical imaging data, we extend the RRS-based approach originally used to model gene microarray data  by incor testing . First, Application of the RRS model to patient-level medical imaging data, where each patient or image is described by a single set of features, is relatively well-understood. Yet disease classification in radiological data (e.g. MRI) occurs at the pixel-level, in which each patient has pixels from both classes (e.g. diseased and non-diseased states) and each pixel is characterized by a set of features . In thisk-nearest neighbor (kNN) classifier [This paper focuses on comparing the performance of three exemplar classifiers: (1) the non-parametric assifier , (2) theassifier , and (3)assifier , 18. Allassifier , (2) graassifier , and (3)assifier .The novel contributions of this work include (1) more stable learning curves due to the incorporation of cross-validation into the RRS scheme, (2) a comparison of performance across multiple classifiers as dataset size increases, and (3) enabling a power analysis of classifiers operating on the pixel/voxel level (as opposed to patient/sample level), which cannot be currently done via standard sample power calculators.Methods section presents a description of the methodology used in this work. Experimental Design includes a description of the datasets and experimental parameters used for evaluation. Results and Discussion are subsequently presented for all experiments, followed by Concluding Remarks.The remainder of the paper is organized as follows. First, the x ∈ 𝓓 is denoted by yt ∈ {ω1, ω2}. A set of training set sizes N = {n1, n2, …, nN}, where 1 ≤ n ≤ |𝓝| and |·| denotes set cardinality.For all experiments, a dataset 𝓓 is divided into independent training 𝓝 ⊂ 𝓓 and a testing 𝓣 ⊂ 𝓓 pools, where 𝓝∩𝓣 = ∅. The class label of a sample n ∈ N, a total of T1 subsets 𝓢(n) ⊂ 𝓝, each containing n samples, are created by randomly sampling the training pool 𝓝. For each n ∈ N and i ∈ {1, 2, …, T1}, the subset Si(n) ∈ 𝓢 is used to train a corresponding classifier Hi(n). Each Hi(n) is evaluated on the entire testing set 𝓣 to produce an error rate ei(n). The mean error rate for each n ∈ N is calculated asThe estimation of classifier performance first requires the construction of multiple classifiers trained on repeated subsampling of the limited dataset. For each training set size Permutation tests are a well-established, non-parametric approach for implicitly determining the null distribution of a test statistic and are primarily employed in situations involving small training sets that contain insufficient data to make assumptions about the underlying data distribution , 20. In Si(n) is compared against the performance of randomly labeled training data. For each Si(n) ∈ 𝓢(n), a total of T2 subsets n samples, are created. For each n ∈ N, i ∈ {1, 2, …, T1}, and j ∈ {1, 2, …, T2}, the subset yr ∈ {ω1, ω2} and used to train a corresponding classifier n, a p-valueθ(z) = 1 if z ≥ 0 and 0 otherwise. Pn is calculated as the fraction of randomly-labeled classifiers Pn < 0.05, i.e. there is a statistically significant difference between M = {n:n ∈ N, Pn < 0.05} includes only those n ∈ N that have passed the significance test.To ensure the statistical significance of the mean error rates K-fold cross-validation strategy. In this paper, the dataset 𝓓 is partitioned into K = 4 pools in which one pool is used for evaluation while the remaining K − 1 pools are used for training to produce mean error rates k ∈ {1, 2, …, K}. The pools are then rotated and the subsampling and permutation tests are repeated until all pools have been evaluated exactly once. This process is repeated over R cross-validation trials, yielding mean error rates r ∈ {1, 2, …, R}. For all training set sizes that have passed the significance test, i.e. ∀n ∈ M, learning curves are generated from a comprehensive mean error ratek ∈ {1, 2, …, K} and iterations r ∈ {1, 2, …, R}.The selection of training 𝓝 and testing 𝓣 pools from the limited dataset 𝓓 is governed by a n, a is the learning rate, and α is the decay rate. The Bayes error rate b is defined as the lowest possible error given an infinite amount of training data [a, α, and b are calculated by solving the constrained non-linear minimization problema, α, b ≥ 0.The power-law model  describeror rate  for traiing data . The modN at the patient-level. Definition of the K training and testing pools as well as creation of each subsampled training set Si(n) ∈ 𝓢 are also performed at the patient-level. Training of the corresponding classifier Hi(n), however, is performed at the pixel-level by aggregating pixels for all patients in Si(n). A similar aggregation is done for all patients in the testing pool 𝓣. By ensuring that all pixels from a given patient remain together, we are able to perform pixel-level calculations while avoiding the sampling bias that occurs when pixels from a single patient span both training and testing sets.The methodology presented in this work can be extended to such pixel- or voxel-level data by first selecting training set sizes T1 = 50 subsampling trials, T2 = 50 permutation trials, and R = 10 independent trials of K = 4 fold cross-validation. In addition, all experiments employ the k-nearest neighbor (kNN), naive Bayes (NB), and Support Vector Machine (SVM) classifiers. A more detailed description of each classifier is presented in Our methodology is evaluated on a synthetic dataset and 3 actual classification tasks traditionally affected by limitations in the availability of imaging data . All expThe three different datasets used in this study were retrospectively acquired from independent patient cohorts, where the data was initially acquired under written informed consent at each collecting institution. All 3 datasets comprised de-identified medical image data and provided to the authors through the IRB protocol # E09-481 titled “Computer-Aided Diagnosis of Cancer” and approved by the Rutgers University Office of Research and Sponsored Programs. Further written informed consent was waived by the IRB board, as all data was being analyzed retrospectively, after de-identification.1 comprises anonymized hematoxylin and eosin (H & E) stained needle-core biopsies of prostate tissue digitized at 20x optical magnification on a whole-slide digital scanner. Regions corresponding to PCa were manually delineated by a pathologist and used as ground truth. Slides were divided into non-overlapping 30 × 30-pixel tissue regions and converted to a grayscale representation. A total of 927 features including first-order statistical, Haralick co-occurrence [N = {25, 30, 35, 40, 45, 50, 55}, was used to extrapolate error rates  a triage mechanism to help pathologists spend less time analyzing samples without cancer and (2) an initial step for decision support systems that aim to quantify disease aggressiveness via automated Gleason grading . Datasetcurrence , and stecurrence   or high (mBR > 7) grade. First, boundaries of 30–40 representative epithelial nuclei were manually segmented in each image region , grade is routinely used to characterize tumor differentiation in breast cancer (BCa) histopathology ; yet, ite region . Using te region . A singl regions .3 comprises 34 anonymized 1.5 Tesla T2-weighted MRI and MRS studies obtained prior to radical prostatectomy, where the ground truth was defined (as cancer and benign metavoxels) via visual inspection of MRI and MRS by an expert radiologist [N = {2, 4, 6, 8, 10, 12}, followed by LOO cross-validation on a larger dataset of 34 patients (Magnetic resonance spectroscopy (MRS), a metabolic non-imaging modality that obtains the metabolic concentrations of specific molecular markers and biochemicals in the prostate, has previously been shown to supplement magnetic resonance imaging (MRI) in the detection of PCa , 26. Theiologist  Fig 4)..3 compripatients .1 and corresponding experimental parameters from Experiment 1. However, since traditional RRS does not use cross-validation, a total of This experiment compares the results of Experiment 1 with the traditional RRS approach, using both dataset 𝓓N = {25, 30, 35, 40, 45, 50, 55} in conjunction with a kNN classifier. Validation is subsequently performed on a larger dataset containing 500 samples . This suggests that the classifier yielding the best results for the smaller dataset may not necessarily be the optimal classifier as the dataset increases in size.Learning curves from kNN and NB classifiers yield predicted error rates similar to their LOO cross-validation errors (0.1552 for both classifiers) as shown in Fig ncreases . However1, the LOO error for both the NB and SVM classifiers  fall within the range of the predicted error rates  predict long-term error rates when only small pilot studies may be available and (2) select the classifier that will yield the lowest error rates when large datasets are available in the future. Predicting classifier performance from small datasets is difficult because the resulting classifiers often produce unstable decisions and yield high error rates. In these scenarios, traditional RRS approaches have previously been used to extrapolate classifier performance (e.g. for gene expression data). Due to the heterogeneity present in biomedical imaging data, we employ an extension of RRS in this work that uses cross-validation sampling to ensure that all samples are used for both training and testing the classifiers. In addition, we apply RRS to voxel-level studies where data from both classes is found within each patient study, a concept that has previously been unexplored in this regard. Evaluation was performed on three classification tasks, including cancer detection in prostate histopathology, cancer grading in breast histopathology, and cancer detection in prostate MRS.We demonstrated the ability to calculate error rates with relatively low variance from three distinct classifiers . A direct comparison of the learning curves showed that the more robust NB and SVM classifiers yielded lower error rates than the kNN classifier for both small and large datasets. A limitation of this work is that all datasets comprise an equal number of samples from each class in order to reduce classifier bias from a machine learning standpoint. However, future work will focus on application to imbalanced datasets where class distribution is based on the overall population . In addition, we will incorporate additional improvements to the RRS method (e.g. subsampling of testing set as in RIDT) while maintaining a robust cross-validation sampling strategy. Additional directions for future research include analyzing the effect of (a) noisy data on different classifiers  and (b) S1 AppendixkNN), Naive Bayes (NB), and Support Vector Machine (SVM). For ease of the reader we provide a metholodogical summary for each of these classifiers with appropriate descriptions and equations.Each experiment in this paper employs three classifiers: k-nearest neighbor ((PDF)Click here for additional data file.


The grade of a cancer is a measure of the cancer's malignancy level, and the stage of a cancer refers to the size and the extent that the cancer has spread. Here we present a computational method for prediction of gene signatures and blood/urine protein markers for breast cancer grades and stages based on RNA-seq data, which are retrieved from the TCGA breast cancer dataset and cover 111 pairs of disease and matching adjacent noncancerous tissues with pathologists-assigned stages and grades. By applying a differential expression and an SVM-based classification approach, we found that 324 and 227 genes in cancer have their expression levels consistently up-regulated vs. their matching controls in a grade- and stage-dependent manner, respectively. By using these genes, we predicted a 9-gene panel as a gene signature for distinguishing poorly differentiated from moderately and well differentiated breast cancers, and a 19-gene panel as a gene signature for discriminating between the moderately and well differentiated breast cancers. Similarly, a 30-gene panel and a 21-gene panel are predicted as gene signatures for distinguishing advanced stage (stages III-IV) from early stage (stages I-II) cancer samples and for distinguishing stage II from stage I samples, respectively. We expect these gene panels can be used as gene-expression signatures for cancer grade and stage classification. In addition, of the 324 grade-dependent genes, 188 and 66 encode proteins that are predicted to be blood-secretory and urine-excretory, respectively; and of the 227 stage-dependent genes, 123 and 51 encode proteins predicted to be blood-secretory and urine-excretory, respectively. We anticipate that some combinations of these blood and urine proteins could serve as markers for monitoring breast cancer at specific grades and stages through blood and urine tests. Breast cancer is a major threat to women's health, accounting for 22.9% of cancer cases in women . AccordiCancer stages are used to reflect the size of a cancer tumor and its extent of invasion. It has been traditionally determined by cancer pathologists based on tumor size, nodal spread and metastasis . In the Cancer grading is a measure of the malignancy and aggressiveness independent of stage. Unlike staging, cancer grading has been predominantly done through visual inspection of the cell morphology and tissue structure , generalversus low risks of recurrence [A number of computational studies have been published on cancer staging and grading prediction based on transcriptomic data. For example, Cui et al have reported a 198-gene and a 10-gene panel for grading and staging prediction of gastric cancers, respectively [currence , 12. Hencurrence , 14, theIn this study, we conducted a computational analysis tissue-based gene-expression data to identify possible gene signatures and blood/urine proteins markers for breast cancer grading and staging prediction. The following represents the unique contributions by this study, to the best of our knowledge: (1) RNA-seq-based gene-expression signatures for breast cancer grading and staging prediction; and (2) predicted potential marker proteins for cancer staging and grading that can be measured by using blood and urine samples. Clearly, this work represents only a pilot study for prediction of blood and urine marker proteins for breast cancer grading and staging. We expect that follow-up studies will demonstrate the feasibility of the predicted signature genes and protein markers.Gene-expression data of 111 paired of breast cancer and adjacent control tissue samples were retrieved from the TCGA database , where eCOPA, GATA3, HDGF, LUM, SPINT2, STAT1, AEBP1, CALR, TRPS1, EPRS, ARL6IP1, EVL, RAD21, PKM2, CD9, NPNT, CLTC, CDH1, NAT1, SH3BGRL}, has been identified, which can distinguish breast cancer from all other cancers, achieving a 94.3% average level of discrimination, and breast cancer from control samples with 99.9% accuracy. For detailed information of comparisons, we refer the reader to To predict gene signatures specific to breast cancer, we have searched for gene combinations among the 853 up-regulated genes in breast cancer, whose expression pattern can best distinguish breast cancer from other cancers and breast cancer from the control samples, using a support vector machine based feature elimination approach, named SVM-RFE see . A 20-geOut of the 111 cancer samples used in this study, 11, 40 and 28 are well, moderately and poorly differentiated, respectively, with the remaining having no grade information provided. 3,881 genes are found to be differentially expressed between the WD and the matching control tissues, with 1,817 genes being up-regulated. 8,022 genes are differentially expressed between the MD tissues and matching controls, with 3,585 up-regulated; and 8,066 genes are differentially expressed between the PD tissues and the controls, with 3,469 up-regulated. We noted that there is a clear trend that the number of differentially expressed genes increases as the grade going from more to less differentiated, as shown in DLGAP5, KIF2C, ZMYND10, and VAV3), with their overexpression levels positively or negatively correlate with the breast-cancer grades. It is noteworthy that DLGAP5 has been found that its silencing suppresses tumorigenicity and inhibits cellular proliferation in cancer cells [KIF2C has been reported that its overexpression involves in breast carcinogenesis [ZMYND10 is a tumor suppressor gene in neuroblastoma [VAV3 has been reported to serve as an oncogene and its overexpression is associated with poor prognosis of a breast cancer [We have checked if some genes have their expression-level changes correlate with the cancer grades. Significant correlation between the level of up-regulation and the three cancer grades WD-MD-PD has been identified of 324 up-regulated genes, as detailed in er cells . KIF2C hogenesis . ZMYND10blastoma . VAV3 hat cancer .A function enrichment analysis of the 324 genes has been conducted by using a hypergeometric test against 2,801 pathways covering the GO terms, canonical pathways from Msigdb  and our FGD3, CENPI, AURKB, DEPDC1B, FAM83D, NCAPH, TNFRSF18, FCGR1A, DEPDC1), has been identified, whose expression pattern can distinguish the PD group from the MD and WD groups with 96.3% classification accuracy (94.5% sensitivity and 97.3% specificity). Similarly, a 19-gene combination, , has been identified that could distinguish the MD group from the WD group with 94.2% classification accuracy (95.0% sensitivity and 92.2% specificity).We have then searched among the 324 genes to find combinations among them whose expression patterns can distinguish among different cancer grades. A 9-gene combination, , 47 in stage II (T2), 19 in stage III (T3) and 1 in stage IV (T4) with the remaining not having such stage information. Considering that stage IV has only one sample, we combined samples in stages III and IV into one stage T3-4. We have observed: 5,358 genes are differentially expressed in T1 samples BHLHE40, HSD17B6, CACNA1A, HDAC8), each with either positive or negative correlation with the stage progression. Among them, BHLHE40 has been reported to correlate with the increased malignancy potential and invasiveness in breast cancer [HSD17B6 is known to be a key enzyme that can catalyze the conversion of 3α-diol to DHT in prostate cancer [CACNA1A is predicted to be a tumor suppressor gene in lung cancer [HDAC8 has been reported to link to the dysregulated expression or interaction with transcription factors critical to tumorigenesis [We have also checked if some genes may have their expression-level changes correlate with cancer stages. Overall, 227 up-regulated genes are found to have their expression levels correlate with cancer progression from T1 through T3-4, as detailed in e cancer . CACNA1Ag cancer . HDAC8 higenesis .Pathway enrichment analysis has also been carried out on the 227 stage-dependent genes. 59 pathways have been found to be significantly enriched by these genes, including carbohydrates metabolism, ion metabolism and homeostasis, mRNA metabolism, apoptosis, ER stress, ABC transporters, protein binding, response to acidosis plus a few signaling pathways. A few of the enriched pathways are listed in OR6K3, RMND5B, LPCAT3, FRRS1, LOC728554, RFX5, JAKMIP1, CLGN, NDST3, GPR6, RIPK3, C2CD4A, PCDHA8, LENEP, CGA, GABRD, DLX1, GPR39, C1orf227, KLF1, ANXA10, EIF3C, UQCRQ, MAPKAPK3, SH3BP5L, TCTEX1D2, TCL1A, IFT122, RAET1L and ABCC13. Similarly we identified a 21-gene combination, , whose expression pattern can distinguish the T2 tissues from the T1 tissues, with 98.0% classification accuracy (99.7% sensitivity and 91.3% specificity).We have searched among the 227 up-regulated genes to find combinations among them whose expression patterns can distinguish among different cancer stages. Using an analysis similar to that for grade-dependent genes, we have identified a 30-gene combination whose expression pattern can best distinguish advanced stage (T3-4) from early stage (T1+T2) breast cancers, with an 99.9% classification accuracy (99.5% sensitivity and 100% specificity). The 30 genes are Five breast cancer microarray datasets, with either grade or stage information, in the GEO database are analyzed to validate the gene markers predicted in the previous sections, to demonstrate that the predicted markers are stable on different datasets collected by different groups. The detailed information of the datasets are given in For the predicted gene markers, we have examined their expression patterns with respect to stages and grades, respectively, in the test microarray datasets by using the Mann Whitney test with significance level 0.05 as the cutoff. Considering that there could be an intrinsic difference between RNA-seq and microarray data, we have first tested the stability of the marker genes predicted using the RNA-seq data on the matching microarray data over the same set of tissue samples, which are made available in the TCGA database. 81.3% (234/288) of the grade-dependent marker genes and 44.7% (83/183) stage-dependent marker genes passed the above statistical test, suggesting the level of the intrinsic difference between RNA-seq and microarray data based predictions.We then examined the predicted grade and stage-dependent marker genes against three microarray data sets retrieved from the GEO database with grade and stage information available, respectively. We noted that 92.8% (284/306) of the grade-dependent genes passed the statistical test in at least one of the three data sets with grade information and the average validation rate across the three datasets is 76.4%. Meanwhile, 42.7% (89/208) of the stage dependent marker genes passed the test in at least one out of three datasets with stage information and the average validation rate across the three data sets is 21.67%. Detailed statistics of the validations are listed in We have also assessed the discerning power of each predicted gene combination for breast cancer staging and grading on the microarray datasets. On the TCGA microarray dataset (matching the RNA-seq data), the 9-gene combination for distinguishing the PD group from the MD and WD groups achieves 87.08% prediction accuracy with 79.83% sensitivity and 89.63% specificity; the 19-gene combination for MD versus WD classification achieves 83.13% prediction accuracy with 91.83% sensitivity and 50.00% specificity. For cancer staging, the 21-gene combination for classification between the T1 and T2 groups and the T3-4 group achieves 76.38% prediction accuracy with 48.00% sensitivity and 85.95% specificity; and the 18-gene combination for T1 versus T2 classification achieves 80.20% prediction accuracy with 89.33% sensitivity and 54.00% specificity. Detailed statistics are shown in On the other three microarray datasets, the gene combination for distinguishing the PD group from the MD and WD groups achieves 77.04% prediction accuracy with 80.03% sensitivity and 72.03% specificity, and the gene combination for discerning the MD group from the WD group achieves 71.73% prediction accuracy with 77.72% sensitivity and 56.67% specificity. Similarly, the gene combination for distinguishing stages T1 and T2 from T3-4 achieves 59.34% accuracy with 32.72% sensitivity and 63.13% specificity; and the gene combination for T1 and T2 discrimination achieves 69.60% accuracy with 81.72% sensitivity and 23.80% specificity on average.It is worth noting that a partial reason for the reduced performance level of our predicted markers on the microarray data is that some selected genes based on the RNA-seq data are missed in the microarray data and some genes expression levels may be not accurately reflected in microarray data due to the intrinsic limitations of the technique .We have also conducted SVM-RFE-based classifier training directly using microarray data and the 324 grade and 227 stage-dependent genes identified based on RNA-seq data. Across the board, gene combinations were identified to achieve better than 90% prediction accuracy with at least 90% sensitivity and 90% specificity for each of the prediction tasks discussed above. Detailed performance statistics by these predicted gene combinations are given in versus control tissues, which are up-regulated in at most three other cancer types out of the 12 we have examined as controls  [CA15-3 [C-telopeptide of collagen type I [In a very similar fashion, we have predicted blood biomarkers for highly  (MIB-1)  and CA15 [CA15-3  being bln type I , which cWe have also examined if some combinations of the 188 genes’ protein products may have high distinguishing power among breast cancers of different grades. A 19-gene combination is identified that can best distinguish the PD group from the MD and WD groups with a 93.6% classification accuracy based on gene-expression data. Similarly, a 17-gene combination is predicted to have strong discriminating power between the MD group and the WD group, with a 93.3% classification accuracy.Similarly, we have examined if some combinations of the 66 urine excretory genes’ proteins have high distinguishing power among breast cancers of different grades. A 15-gene combination has been found to well distinguish the PD group from the MD and WD groups, with a 92.8% classification accuracy based on gene-expression data. And a 19-gene combination is predicted to distinguish the MD group from the WD group, with a 83.8% classification accuracy. The detailed gene lists of these blood and urine gene combinations are given in Using a similar prediction procedure to that in the above section, a 23-gene combination is predicted to be the best distinguisher between the T3-4 and the T2-T1 group, with a 99.4% classification accuracy based on gene-expression data. And a 23-gene combination is predicted to have the best discerning power to distinguish between the T2 group and the T1 group, with a 98.1% classification accuracy.Similarly for urine secretory gene panels, a 19-gene combination is predicted to best distinguish the T3-4 group from the T2-T1 groups, with a 87.6% classification accuracy. And a 25-gene combination is predicted to best distinguish the T2 from the T1 group, with a 97.2% classification accuracy. The detailed gene lists of these stage biomarkers are given in Reliable prediction of a cancer’s grade and stage is very important as it can provide useful information for cancer mechanism studies as well as for selection of the most appropriate treatment plans. In this study, we used our in-house computational approaches to have predicted reliable gene signatures and protein markers for breast cancer detection and their grades and stages by using 111 pairs of breast cancer and matching control tissues. In order to identify the most reliable markers, we specifically applied the non-parametric Mann Whitney test with relatively lower sensitivity but higher specificity compared to other parametric tests for differential gene expressions . In addiWe noted that among the predicted marker genes, breast cancer specific markers, especially the 20 identified by SVM-RFE, tend to be cancer associated genes such as oncogene CLTC and tumor suppressor genes CDH1 and GATA3 while PKM2, STAT1, EPRS, HDGF, LUM, SPINT2, TRPS1, EVL, RAD21, NPNT, NAT1, whose gene expression level changes have been reported as breast cancer associated –49. AmonBy using our in-house software, we predicted the possible blood and urine protein markers based on the predicted uniquely over-expressed genes in breast cancer. Such information could provide useful targets for guided search for protein markers in blood and/or urine for breast cancer detection and/or classification through blood or urine tests. We fully expect follow-up studies will demonstrate the feasibility of the predicted signature genes and protein markers.RNA-seq data of breast cancer and 12 other cancer types, namely bladder carcinoma, colon adenocarcinoma, head and neck squamous cell carcinoma, kidney chromophobe, kidney renal clear cell carcinoma, kidney renal papillary cell carcinoma, liver hepatocellular carcinoma, lung adenocarcinoma, lung squamous cell carcinoma, prostate adenocarcinoma, rectum adenocarcinoma and thyroid carcinoma were downloaded from the TCGA database , all of In addition to the RNA-seq data, the matching microarray data collected on the same breast cancer and control samples are also retrieved from TCGA. In addition, five microarray data sets collected on independent collections of tissue samples are retrieved from the GEO database and analyzed to validate our RNA-seq data based marker predictions. The detailed information of all these data is listed in versus control samples. Specifically, our null hypothesis H0 is that a gene is not differentially expressed in cancer versus the control samples; rejection of this hypothesis means that the gene is differentially expressed in cancer versus the controls. Let Ci and Ni be a gene’s expression level in the ith− cancer and the matching control tissues, i = 1,…,N and N being the number of paired samples. We calculate |Ci − Ni| and sgn(Ci − Ni), with Ci − Ni| = 0 in our gene-expression data analyses. Let Nr be the remaining sample size, and sort the Nr tissue pairs in the increasing order of the |Ni − Ci| values. We then give each pair a rank, numbered contiguously, consistent with the relative positions in the sorted list of paired tissues, i.e., with the first pair of tissues having rank 1 and tissues with the same |Ni − Ci| value having the same rank, overall represented by Ri. We calculate the test statistic W using Nr ≥ 10, a z-score is calculated as z > zcritical, the null hypothesis H0 is rejected. For Nr < 10, W is compared to a critical value from a reference table. If H0 is rejected, i.e. we consider the gene as differentially expressed.Our previous study has revealed that the normalized microarray or RNAseq gene expression profile through multiple cancer samples may follow mixed distributions with multiple peaks due to possible intra-tumor heterogeneity or disease sub-types as shown in versus the matching control be larger than 2.0. For breast cancer grades and stages, significantly differential expression is determined by FDR < 0.05 and the fold-change > 1.5.For each cancer type, significantly differential expression is determined by the False Discovery Rate (FDR) < 0.05  and the Spearman correlation coefficient  was usedPathway enrichment is assessed using a hypergeometric test against 2,801 gene sets covering pathways in the KEGG , Biocartw‖2, which satisfies the conditions yi(w∙xi + b) ≥ 1, where xi is a feature vector, yi is 1 or -1, representing the class to which the point xi belongs. Gene signatures of each training set were selected by using the recursive feature elimination procedure [A support vector machine (SVM)-based recursive feature elimination approach was applied to predict gene signatures for each breast cancer grade as well as stage. A linear SVM was used for training our classifier , 61. It rocedure . The overocedure .All up-regulated genes in breast cancer were analyzed for predicting if their protein products are blood-secretory, using a program developed by our lab , and uriThe basic idea of each algorithm is as follows. Human proteins known to be blood secretory (urine excretory), according to the published data, are selected to form a positive training set and proteins, known to be not blood secretory (urine excretory), are selected to form a negative training set. A list of features related to protein sequence and structures were examined and those found to have discerning power between the positive and the negative training data were selected. A (SVM)-based classifier for blood secretory (urine excretory) proteins. Both programs have been systematically assessed against large datasets, having achieved high-level prediction accuracy in both cases.S1 Table(XLSX)Click here for additional data file.S2 TableIn the table, the column p value and sign represent the p value of the differential expression and up (“+”) or down (“-”) regulation between the two labeled classes, respectively. The TCGA RNAseq data analysis results and microarray validation results are colored in green and yellow respectively while “IS” represents insignificant. Blank elements in the validation columns suggest the genes are non-differentially expressed in the RNAseq data.(XLSX)Click here for additional data file.S3 TableNon-cell proliferation associated pathways are yellow highlighted.(XLSX)Click here for additional data file.S4 TableIn the table, the column p value and sign represent the p value of the differential expression and up (“+”) or down (“-”) regulation between the two labeled classes, respectively. The TCGA RNAseq data analysis results and microarray validation results are colored in green and yellow respectively while “IS” represents insignificant. Blank elements in the validation columns suggest the genes are non-differentially expressed in the RNAseq data.(XLSX)Click here for additional data file.S5 Table(XLSX)Click here for additional data file.S6 Table(XLSX)Click here for additional data file.S7 Table(XLSX)Click here for additional data file.S8 Table(XLSX)Click here for additional data file.S9 Table(XLSX)Click here for additional data file.S10 Table(XLSX)Click here for additional data file.S1 FigIn each figure, the gene expression prolife  of TCGA breast cancer samples and normal breast samples are colored by blue and pink, respectively. The expression profile cancer samples are fitted by mixed Gaussain distributions. The red, blue and green curves represent the density function of the fitted mixed Gaussain distributions (weighted by sample size).(PDF)Click here for additional data file.


Alternative Splicing (AS) as a post-transcription regulation mechanism is an important application of RNA-seq studies in eukaryotes. A number of software and computational methods have been developed for detecting AS. Most of the methods, however, are designed and tested on animal data, such as human and mouse. Plants genes differ from those of animals in many ways, e.g., the average intron size and preferred AS types. These differences may require different computational approaches and raise questions about their effectiveness on plant data. The goal of this paper is to benchmark existing computational differential splicing (or transcription) detection methods so that biologists can choose the most suitable tools to accomplish their goals.This study compares the eight popular public available software packages for differential splicing analysis using both simulated and real Arabidopsis thaliana RNA-seq data. All software are freely available. The study examines the effect of varying AS ratio, read depth, dispersion pattern, AS types, sample sizes and the influence of annotation. Using a real data, the study looks at the consistences between the packages and verifies a subset of the detected AS events using PCR studies.No single method performs the best in all situations. The accuracy of annotation has a major impact on which method should be chosen for AS analysis. DEXSeq performs well in the simulated data when the AS signal is relative strong and annotation is accurate. Cufflinks achieve a better tradeoff between precision and recall and turns out to be the best one when incomplete annotation is provided. Some methods perform inconsistently for different AS types. Complex AS events that combine several simple AS events impose problems for most methods, especially for MATS. MATS stands out in the analysis of real RNA-seq data when all the AS events being evaluated are simple AS events.The online version of this article (doi:10.1186/s12859-014-0364-4) contains supplementary material, which is available to authorized users. Alternative splicing (AS) is a post-transcriptional regulation mechanism that allows a single gene to produce multiple mRNA transcripts. Some of the roles of AS include regulating gene expression in response to environmental stimuli and developmental changes -3. In ad′ splice site (A3SS), alternative 5′ splice site (A5SS), mutually exclusive exons (MXE), and intron retention (IR) [%), but the least common type in plants (5%) [%) but the least prevalent type in humans [Although there is no consensus classification of AS types, the five standard types are skipped exon (SE), alternative 3ion (IR) . Animalsnts (5%) . Intron n humans ,8. This n humans . Also, An humans .Some evidence also suggests that plants and animals may regulate AS in different ways. For examples, plants possess nearly double the number of SR proteins as compared to nonphotosynthetic organisms . SR stanThe advent of RNA-seq has increased the observed frequency of AS in plants from 30% -14 in thEffectiveness: the method should detect differential AS across samples. Note that this is not necessarily equivalent to isoform quantification problem as changes in the absolute isoform expression do not necessarily imply differential alternative splicing .Biological replicates: the selected method should be able to take advantage of biological replicates in the RNA-seq data sets.Software engineering: the method has to be implemented as a usable and robust program so that a scientist with limited computational skills can run the program regardless of understanding the theory behind it.This work benchmarks eight popular methods for differential AS according to the three criteria given below: effectiveness, biological replicates and software engineering. https://github.com/ruolin/ASmethodsBenchmarking. The whole pipeline is documented, interested readers can repeat the study and test the results with their preferred differential AS detection tools.For example, under these criteria, some methods are ruled out for inclusion in this study. E.g., SpliceTrap  only quaMethods for detecting AS may be categorized into two quantification schemas: count-based models and isoform resolution models Figure . These tThe count-based models are based on the methods used to quantify transcripts with single isoforms. The number of reads falling on a transcript , like RPKM (Reads Per Kilobase per Millions of reads mapped), is used as an estimate for abundance . Count-bi in group j and variance of SeqGSEA  and DSGsLike SeqGSEA and DSGseq, DEXSeq  transforThe rDiff  package ψ, which is the proportion of the reads that exclusively support one outcome of the events to reads that exclusively support another outcome of the identified events. The exon inclusion level is always between 0 and 1. Then, the posterior probability of the difference of exon inclusion level across two samples which is larger than a user-defined cutoff, denoted p(|ψ1−ψ2|>c | data), is calculated. MATS reports which AS event is significant rather than which gene is alternatively spliced. MATS differs from other count-based model methods in that it uses Bayesian approaches. It is also the only method that does not assume independence of two biological conditions. A bivariate uniform prior is used to model the dependence. Information across genes is borrowed in the process of estimating the common prior. Although the method in MATS’s original paper is only designed for a two sample comparison, the latest version of MATS (3.0+) accepts multiple replicates. However, it is unclear how the program models biological variability.MATS  first reLike DEXSeq and DSGseq, SplicingCompass  uses a uq is usually done by maximizing a likelihood function L. Maximizing this likelihood function is equivalent to maximizing the likelihood of selecting a read or fragment from a transcript [Isoform resolution models  are mulanscript . Isoformanscript .Cufflinks  and DiffRigorously speaking, DiffSplice is not “These differential AS detection methods were first evaluated using simulated data with known ground truth, where we could control the level of differential splicing across conditions and other factors that may affect detection. The NB distributions were used to simulate read counts on genes. The mean and dispersion parameters for the NB distributions were estimated from heat shock data . The 588In the simulation study, we evaluated the robustness of the methods by varying the degree of differential splicing, read depths, sample sizes and dispersion setting in different conditions. We set High, Medium and Low levels for AS ratio, two dispersion patterns and three levels of read depth . In addition, we have compared the computational time required for running the analysis  curves and the Area Under the Curve (AUC) metric. The ROC curve depicts the true-positive rate (TPR) of a method for different false-positive rates (FPR) by varying the threshold for given scores. TPR is defined as the proportion of the events that are known to be differentially spliced that test as positives. Similarly the FPR is the proportion of the events that are known not to have differential splicing that test as positives. As almost all AS detection software packages tightly control FPR, we restricted the ROC curves to the range of 0−0.2 Figures  and 4. TPadj=0.05 cutoff were used as a additional set of evaluation metrics . Recall is equivalent to TPR while precision is the proportion of the events that test as positives that are actually true discoveries. Precision is also known as 1−falsediscoveryrate. Evaluating on precision examines whether the methods are able to control the FDR at the claimed 0.05 level. DSGseq does not return p-values and was excluded from this evaluation and SeqGSEA did not report any gene under FDR=0.05 when the sample size was 3. The results of other seven methods under the measurement of recall and precision or FDR at Padj=0.05 are summarized in Table As ROC curves and AUC measure the discrimination power between non-differentially spliced (non-DS) gene and differentially spliced (DS) gene over an interval, scientists are often interested in the discrimination power at a single cutoff point. Therefore the recall and precision at a For the real data, we first compared the results obtained by the different methods in terms of absolute number of significant AS gene calling, the overlap of results across software and the concordance of gene rankings. We further compared these results to a list of experimentally validated genes that are known to be alternatively spliced in response to ambient temperature changes. Finally we carried out an semi-RT-PCR study and compared the results of the computational methods using RNA-seq to the results from RT-PCR.PALT (Percentage of ALTernative isoform) to control the level of differential splicing in our simulation. PALT, whose range is from 0−1, simply represents the relative abundances of alternative isoforms for given genes. For multi-transcript genes, we randomly chose one transcript as an alternative isoform while the rest of isoforms remained as standard isoforms across conditions. For each of given genes, all standard isoforms have relative abundances which summed to 1−PALT. The PALT for 2000 true AS genes was set to 0.2 in the control group and 0.4, 0.6, 0.8 in the three treatment groups, corresponding to low, medium and high AS ratio levels. We investigated the effect of varying the AS ratio level under two dispersion patterns. As a result we carried out 6 simulation studies and denoted them in the format of Since the difference required between two isoform compositions to be biologically significant enough to call as differential splicing is an open question, we defined a parameter PALT changed from 0.8 to 0.4, the difference between the isoform compositions under the two simulated conditions became smaller. All methods lost their discrimination power as the signal of differential splicing became weaker. The results from simulation studies with the same dispersion pattern were similar and are shown in the  have more than one AS type. This made testing the performance in terms of the effect of different AS types difficult. Also as some methods, e.g. MATS and DiffSplice, test on individual events or local regions while others work on the gene level, the previous comparisons were not based on common ground. To overcome these problems, we picked out 1755 genes that have exactly two transcripts and a single splicing event from the 5885 genes. We then reevaluated all methods on these 1755 genes in the baseline simulation study. This equated the detection on a gene level to the detection on a splicing event. We classified these 1755 genes into three new sets by their splicing event types which include exon skipping, intron retention and alternative donor/acceptor sites Figure . We treaPadj=0.05 threshold in all simple AS events  in a sample size of 8. However the ROC curves and AUC statistics for SeqGSEA were almost the same for the different sample sizes . We computed the Spearman rank correlations between all pairs of the eight methods and visualized it using a heat map Figure . The corSince there have been studies that have linked some genes to alternatively spliced variants in response to heat stress, we came up with a list of six experimentally validated AS genes based on a search of the literature. AT1G01060 encodes LHY, a transcription factor involved in regulation of circadian rhythm. An A3SS event, encoding a 3-nt difference, has been found to occur as the ambient temperature changes . This alFDR=0.05, MATS identified all seven genes and successfully located the actual genomic regions. DEXseq found two of them (SR1/SR34 and SR30) and Cufflinks reported one (FLM). None of the other methods were able to find these genes. For LHY and SR45, the A3SS events encompass a range of nt differences from a few to tens. MATS’s success in finding these events can probably be attributed to the exclusive use of junction reads. The small differences were easily overlooked by other methods that take into account of reads on full exonic regions. The junction reads that uniquely supported the A3SS events tend to be overwhelmed by the non-junction reads along the long exon , we used semi-quantitative PCR to characterize heat induced splicing changes in seven genes that were annotated in TAIR 10 as being alternatively spliced. These seven genes thus provided a useful positive control for estimating the accuracy of the splicing analysis methods described here. These seven genes are AT1G77180, AT1G01490, AT2G02390, AT2G26670, AT3G19720, AT5G26780, AT1G09140. At the cutoff In this paper, we have evaluated and compared eight methods for alternative/differential splicing analysis of RNA-seq data. The major observations for the AS methods are summarized in Table From the perspective of AUC statistics, DEXSeq and DSGseq performed well in the simulation studies when the annotation is accurate and complete. DEXSeq was slightly better when two groups of samples were simulated using different dispersion parameters while DSGseq excelled when the same dispersion parameter is used. DSGseq is also more robust to changes in the AS ratio than DEXSeq. The drawback of DSGseq is that it does not calculate p-value. Both methods belongs to count based models. However, like other methods which depend on gene models, they performance was largely impaired when incomplete annotation was used. This may impose problems when working on non-model species or simply any species that are not well annotated. Cufflinks and DiffSplice are capable of assembling reads into transcripts and are thereby able to detect novel AS events. Only Cufflinks can take advantage of established gene models and is not fully dependent on the prior knowledge. These attributes render Cufflinks the best combination of accuracy and robustness against incomplete annotation. Therefore it is recommended for non-model species. On the other hand, Cufflinks achieves a better tradeoff between precision and recall. It also performs the best in an median read coverage of 60. The change of AS ratio affected methods’ discrimination power as well as the ability to control FDR. The rankings, however, were relatively stable as AS ratio changed, indicating that most methods is generally good enough to analyze real RNA-seq experiments where the splicing ratio might vary from gene to gene.MATS uses a Bayesian framework to calculate the probability of a gene being alternatively spliced. Although MATS did not exhibit good performance under the evaluation of ROC curves and AUC, it was the best method under our comparison with respect to controlling the FDR at a proposed level. MATS excels in the precision of its results, which is very important for most biologists. The reason MATS had low recall and AUC is that MATS was only designed for detecting simple AS events. Therefore it was not satisfactory when the simulation included complex AS events. When only genes with simple AS events were involved, both recall and AUC improved dramatically for MATS. The superb performance of MATS in real data is boosted by the fact that all the 6 validated AS genes from the literature as well as for the 7 PCR validated AS genes are simple AS genes. rDiff-parametric also had a low FDR, however, but it appears to be due to its use of BF correction. In the analysis of heat shock RNA-seq data, MATS turned out to be the method that was the most consistent with the established experimental evidence as well as our PCR validations. The drawback of the MATS is that it is highly dependent on the goodness of annotation but it would be recommended for validating known AS events.Large sample size (8 samples per condition) did not affect the discriminating power under ROC and/or AUC evaluation, but did improve several methods’ recall at the cost of decrease in precision. The several methods include Cufflinks, DEXSeq, SplicingCompass and especially SeqGSEA. SeqGSEA uses a permutation based approach to calculate p-values for genes being alternative spliced. It is likely that the p-values are scaled in accordance with sample size and we may expect a optimal sample size around 5 or 6 for using SeqGSEA. The sets of significantly alternatively spliced genes at given FDR threshold (FDR = 0.05) varied considerably between methods for the analysis of heat shock data. SeqGSEA and DSGseq had the highest correlations of the gene ranking scores due to using the same test statistics.All of the selected methods in this paper allow users to specify certain parameters. We have mostly used the default parameters as this is how most users apply these software packages. The detailed command lines and parameter choices used in the baseline simulation study are given in the Additional file For MATS, we used the mapping results instead of fastq files as the program input. Starting with MATS (3.0+), the program outputs two types of results: analysis based on both exon body reads as well as junctions reads and analysis based on junction reads alone. For all the comparisons, we used the latter but we showed in the Additional file For Cufflinks, we first assembled each sample individually using Cufflinks and then merged the resultant transcripts with annotation using Cuffmerge. The merged transcripts was used in Cuffdiff to perform the analysis of differential splicing. We used the fragment bias correction option in Cufflinks. In the analysis of heat shock data, the minimum number of replicates were set to 2 because one of the conditions has only two samples.SeqGSEA integrates analysis regarding differential gene expression (DE) with analysis regarding differential splicing (DS). We only performed the latter and calculated the DS permutation p-values for 1000 iterations.In the heat shock experiment , RNA wasWe generated Arabidopsis RNAseq data using Flux Simulator  with exaWe created a custom simulation pipeline  was deliberately chosen to produce more reads that cross exon-exon junctions. The generated synthetic reads were then mapped against the latest Arabidopsis genome TAIR 10 using the GMAP and GSNAP packages (version 2013-05-09) . To maxiPadj. Rdiff use Bonferroni correction while SplicingCompass, MATS, DEXSeq, SeqGSEA and Cufflinks-Cuffdiff use Benjamini-Hochberg correction. DiffSplice and DSGseq do not provide p-values, and so we used their test statistics as the ranking scores: square root of JSD for DiffSplice and NB statistics for DSGseq (see the We defined ranking scores for each method directly from the output. This score is a direct reflection of significance or evidence for alternative splicing across two conditions. For the six methods that provide adjusted p-values after multiple testing correction, we defined the score as 1− see the .Additional file 1Supplementary material. Contains supplementary figures referred to in the text. Here we also illustrate the simulation pipeline, and we compare the distribution and dispersion between simulated data and real data. The file also contains sample command lines or R code and computational time requirements for running each program. Finally the direct read alignments for the seven experimentally validated AS genes are shown in this file.Additional file 2Results of eight selected programs on real data. TXT file contains the result of each program run on the heat stress data set.


Determining differentially expressed genes (DEGs) between biological samples is the key to understand how genotype gives rise to phenotype. RNA-seq and microarray are two main technologies for profiling gene expression levels. However, considerable discrepancy has been found between DEGs detected using the two technologies. Integration data across these two platforms has the potential to improve the power and reliability of DEG detection.We propose a rank-based semi-parametric model to determine DEGs using information across different sources and apply it to the integration of RNA-seq and microarray data. By incorporating both the significance of differential expression and the consistency across platforms, our method effectively detects DEGs with moderate but consistent signals. We demonstrate the effectiveness of our method using simulation studies, MAQC/SEQC data and a synthetic microRNA dataset.Our integration method is not only robust to noise and heterogeneity in the data, but also adaptive to the structure of data. In our simulations and real data studies, our approach shows a higher discriminate power and identifies more biologically relevant DEGs than eBayes, DEseq and some commonly used meta-analysis methods.The online version of this article (doi:10.1186/s12859-015-0847-y) contains supplementary material, which is available to authorized users. Detection of differentially expressed genes (DEGs) between biological samples is the key to understand how genotype gives rise to phenotype. With the rapid accumulation of consortium studies (e.g. ENCODE ) and pubDespite these well-known benefits, combining gene expression data from different sources involves many intricate issues. For example, if data are collected from different platforms, the scales of measurements on individual studies may not be comparable. Even though many normalization methods have been developed, normalization across platforms still remains a challenge –10. FurtA convenient way to handle heterogeneity and noise in the data is to use rank-based approach, since rank is robust to outliers and is always comparable across platforms. It has been shown that ranking fold changes of differential gene expression produces better agreement of DEG lists across labs and platforms than using p-values in microarray gene expression studies . RecentlIn this work, we develop a rank-based semi-parametric statistical method to integrate gene expression profiles from different sources. Our model emphasizes the biological intuition that true signals from samples measuring the same biological mechanism should have concordant signals. It builds in a strong preference for concordance of differential directionality and significance across sources, such that the genes that have moderate but consistent signals across studies can be effectively detected. Unlike meta-analysis methods based on fixed rules, our method explicitly models the structure of the data through a copula mixture model, making it both adaptive to the data and robust to noise.We illustrate our method in the integration of microarray and RNA-seq data for DEG detection. Microarray has been the major experimental platform for gene expression study since mid-1990’s , 16. DesWe apply our method to microarray and RNA-seq data from the Microarray Quality Control (MAQC)  and SequOur goal is to develop a data integration method that is both robust to noise and heterogeneity in the data, and adaptive to the structure of data. We develop our method based on a copula mixture model in Li et al (2011) . This moThough this method was originally proposed for assessing reproducibility, it can also be viewed as a semi-parametric aggregation method that combines rank lists from different studies. The signals that are consistent across studies are weighed more favorably than those with similar significance but inconsistent across studies. However, this model only clusters the entries into two groups, with the top - ranked ones as interesting signals and the bottom - ranked ones as noise; whereas, for gene expression studies, DEGs reside on both ends of the rank lists, and both ends would be of interest.Here we extend this model to the context of gene expression studies. We assume that the sample consists of non-DE, up-regulated, and down-regulated DE genes. We use the level of differential expression (e.g. fold change) as our data, such that the up-regulated, down-regulated, and non-DE genes are concentrated on the top, bottom, and middle part of the rank lists, respectively. For simplicity of discussion, we focus on the case of two studies in what follows, and provides the extension to the case with more studies in Additional file i on two studies are , we assume that Xj = , j = 1, 2 is an independent and identically distributed sample with CDF Fj, where Fj is unknown and can vary across studies. Let Ki denote whether the ith gene is non-DE (Ki = 0), up-regulated (Ki = 1) or down-regulated (Ki = 2), and let π0, π1 and π2 = 1 − π0 − π1 denote the corresponding proportions.Suppose the level of differential expression for gene Ki = k, the dependence across studies for the genes in the kth component is induced by a latent bivariate Gaussian random variable, zk = |Ki = k. The correlation coefficient between the two studies ρk is positive for z1 and z2, and 0 for z0. Though the marginal distribution of observed differential expression level, Fj, may be different across studies, it is natural to assume zi,1|Ki = k and zi,2|Ki = k have the same marginal distributions, as different studies are assumed to measure the same underlying biological process. To reflect up- and down-regulation, we assume z1 has a higher mean than z0, and z0 has a higher mean than z2. Finally, as the scales of the marginal distributions are unknown, only the difference in means between two latent variables and the ratio of their variances can be identified, but not their actual means and variances. Thus, we set z0 to have mean 0 and variance 1. Putting the above together, the model that generates the dependence structure isμ0 = 0, σ02 = 1, ρ0 = 0, μ1 > 0 > μ2, 1 > ρk > 0 for k = 1, 2. Let xi,j areBecause differentially expressed genes are expected to be concordant in both the direction of differentiation and the level of significance across studies, we expect the differential expression level of a gene to be positively correlated across studies for DEGs but not for non-DEGs. To model this dependence structure, we assume that, given θ =  and , where F1 and F2 will be substituted by the empirical distributions if they are unknown. Thus it is scale-free. The corresponding mixture likelihood for the data ishk is the bivariate normal density function with parameters μk, σk2 and ρk. The parameters θ can be estimated by maximizing the mixture likelihood using an estimation procedure similar to Li et al 32], respt-statistics and their associated p-values. To compare our method with commonly-used meta-analysis methods, we combine the p-values from DEseq and eBayes using Fisher’s method and Stouffer’s method, and combine the fold changes using RankProd and our method.To identify DEGs from a single-platform, we use DEseq for RNA-seq  and eBayIn an attempt to simulate realistic data, we estimated the parameters for simulation from a real dataset in Marioni et al , which cHere we provide a brief description of the simulation procedure. A detailed description and parameter settings can be found in Additional file Here we evaluate the performance in several scenarios, in particular, the scenarios when two platforms have similar versus different data quality, when data quality is high versus low, and when the proportions of up- and down-regulated genes are equal versus unequal. For each scenario, three replicates are simulated under two conditions for each platform, with 10,000 genes for each replicate. Simulated expression levels then undergo the standard pre-processing procedure . The EST enrichment score reflects the specificity of gene expression in a tissue and is expected to be correlated with the differential expression levels measured in the experimental data [To evaluate the functional relevance of identified DE genes, we obtained the EST (Expressed sequence tag) enrichment score of brain preferentially expressed genes, which consists of 118 genes specifically expressed in brain, from the TiGER database  among all methods of comparison .As true fold changes are known, the sensitivity and specificity of the identification of DEGs at various thresholds can be evaluated. Here we consider classification of DE and non-DE genes at two levels of stringency. One treats the genes with identical expression levels in both samples as true non-DEGs, and the rest as true DEGs; and the other treats the genes with log 2 fold change less than ±0.5 as true non-DEGs, and the rest as true DEGs. As shown in the ROC curve


We designed a custom nCounter® codeset based on elements from multiple published fresh frozen tissue microarray-based prognostic gene signatures for colon cancer, and we used this platform to systematically compare gene expression data from FFPE with matched microarray array data from frozen tissues. Our results show moderate correlation of gene expression between two platforms and discovery of a small subset of genes as candidate biomarkers for colon cancer prognosis that are detectable and quantifiable in FFPE tissue sections.The prognosis of colorectal cancer (CRC) stage II and III patients remains a challenge due to the difficulties of finding robust biomarkers suitable for testing clinical samples. The majority of published gene signatures of CRC have been generated on fresh frozen colorectal tissues. Because collection of frozen tissue is not practical for routine surgical pathology practice, a clinical test that improves prognostic capabilities beyond standard pathological staging of colon cancer will need to be designed for formalin-fixed paraffin-embedded (FFPE) tissues. The NanoString nCounter The identification of colorectal cancer (CRC) patients who either more or less likely to benefit from adjuvant systemic chemotherapy after surgical resection poses a major unmet need in providing safe and effective care. The current practice may result in under-treatment of some high-risk stage II patients, and potential over-treatment of low-risk stage II and stage III patients . The cor® system has been applied to quantify gene expression of various gene signatures for multiple tissue types including FFPE samples [The NanoString nCounter samples –13. We hHuman tissues used for microarray analysis were collected and annotated according to established protocols and approved by the appropriate Institutional Review Boards (IRB) at Vanderbilt University (VUMC). All tissues were collected over the time period from 1999–2011. Tumor stage was assessed by American Joint Commission on Cancer guidelines. Written informed consent was obtained from all patients prior to inclusion in the studies. Quality assessment slides were obtained to verify the diagnosis and the amount of cellular material for each sample. De-identified human tumor tissues for immunohistochemistry were obtained with VUMC IRB approval. For microarray studies, representative sections of fresh tissue specimens were flash frozen in liquid nitrogen within 20 minutes of resection and stored at −80°C until the RNA isolation step. The median (minimum-maximum) frozen tissue storage time from date of resection to RNA extraction are 455 (35–2858) days. RNA was purified from tissue sections containing >80% epithelial tumor tissue using RNeasy  according to manufacturer’s instructions. Samples were hybridized to Affymetrix arrays Human Genome U133 Plus 2.0 GeneChip Expression Arrays, Santa Clara, CA), which included approximately 39,000 human genes. The samples included four healthy control patient tissues, 12 stage I, 17 stage II, 20 stage III and 15 stage IV CRC patient tissues. The detailed sample information including suvival data can be found in For nCounter studies, archived tumor FFPE samples matched to frozen tissues described above were identified and processed by the Vanderbilt Translational Pathology and Imaging Core facility. Tissues were fixed in 4% paraformaldehyde, embedded in paraffin and stored at room temperature until RNA extraction. The median (minimum-maximum) FFPE tissue storage time from date of resection to RNA extraction are 1021 (364–3483) days. The first 20μm of the tissue was discarded before cutting sections for RNA extraction. Tissue sections were mounted on uncharged glass slides and one of the tissue section slides was stained with H&E for quality control from each tissue block. RNA was purified from 5μm thick tissue sections containing greater than 80% tumor using High Pure FFPE RNA Micro Kit (Roche) according to manufacturer’s instructions. A minimum of 4 sections per sample were required. The mRNA hybridization, detection and scanning were followed the protocol and performed by NanoString Technologies.® assay  for quantitative assessment of expression of 414 gene elements. This multiplexed assay can detect expression of up to 800 transcripts at very low mRNA concentrations (0.1fM/1 copy per cell) [We designed a custom nCounterer cell) . Table 1er cell) , as weller cell) –24 in thAffy [SVA [NanoStringNorm [Limma package in Bioconductor was used to identify differentially expressed probe sets between the tumor and normal tissues in both Affymetrix and nCounter platforms [survival.Affymetrix microarray data were normalized using the Robust MultiChip Averaging (RMA) algorithm as implemented in the Bioconductor package Affy . The batffy [SVA . NanoStrringNorm . For grolatforms . For surIn this study, in order to make direct comparison between Affymetrix HGU133plus 2 and nCounter gene expression data, we first found the best-matched Affymetrix microarray probes to NanoString nCounter probes. Toward this end, we used gene symbol or transcript ID to link NanoString nCounter IDs and Affymetrix Probe IDs. If the gene had only one unique probe, then we used that Probe ID as the match for NanoString. Otherwise, if the gene had multiple probes, then we picked the one with the largest Smith–Waterman alignment score, which was calculated by Smith-Waterman algorithm to identify homologous regions between sequences by searching for optimal local alignments, between NanoString nCounter and Affymetrix Probe sequences as the match for nCounter . Both miRNA Integrity Numbers (RIN) values were obtained using an Agilent Bioanalyzer instrument for both frozen and FFPE derived RNA samples. The instrument software generates a RIN score based on its entire electropherogram. RIN values range from 1–10, with 1 being totally degraded RNA and 10 being high quality (intact) RNA. For microarray studies, a cut-off of RIN = 7.0 was used. RIN values ranged from 7–10, with a median of 8.5 and mean of 8.4. For nCounter studies, RIN values ranged from 0–8, with a median of 2.4 and mean of 2.5. In addition, to be acceptable for analysis for nCounter, ≥20% of the RNA sample had to contain fragments of at least 300 base pairs in length. The details of the results were included in After pre-processing and normalization, log2 transformed data from 414 genes of both microarray and nCounter were compared. The distributions of signal intensities for all samples are displayed with boxplots in To examine the reproducibility of the nCounter platform, six FFPE samples were selected to generate two more technical replicates for each sample. For each pair of FFPE sample quantified by nCounter and fresh frozen sample quantified by microarray, the Pearson correlation was calculated based on normalized, log2 transformed gene expression values of the 414 genes. The mean correlation was 0.501 with 95% bootstrap confidence interval , and the minimum and maximum correlation were 0.337 and 0.591 respectively. The heatmap in For each of 414 genes, the Pearson correlation was also calculated between two tissue types. We plotted the histogram of gene-wise correlation coefficients in Bioconductor package LIMMA. Using nominal p-value 0.05 as the cutoff, 76 genes were identified as significantly differentially expressed genes in both platforms. Among the 76 genes, 40 and 26 genes were overexpressed and downregulated in cancer samples. There were 30 and 60 significant genes discovered in only one platform by nCounter and microarray respectively, and 248 genes were not significant by either platform. The complete gene testing lists including coefficient, nominal p-value and false discovery rate can be found in After preprocessing and normalization, to detect differentially expressed genes between 64 tumor and 4 normal samples in nCounter or microarray platforms separately, we used regularized t-test as implemented in the We also examined the association of gene expression with overall survival outcomes of 64 patients using a Cox proportional hazard model. Using a nominal p-value 0.05 as the threshold, there were 6 significantly associated genes including PPL, CCND1, EXT2, S100A11, USP9X, and PHLDA1 by both platforms, 25 significant genes by nCounter, and 24 significant genes by microarray. PPL, CCND1, S100A11, and PHLDA1 were associated with higher hazard, and EXT2 and USP9X were associated with better prognosis. The majority of the genes (359 genes) were not significantly associated with overall survival outcomes by either platform due to relatively small sample size. The complete gene testing lists including coefficient, nominal p-value and false discovery rate or this survival data analysis are in The quality of mRNA transcript quantification in FFPE samples is crucial for translational cancer research. In this study, our comparisons were based on the measurements of NanoString’s nCounter platform using FFPE-derived samples and Affymetrix’s Human Genome U133 Plus 2.0 GeneChip Expression Arrays on patient matched fresh frozen tissue. Ideally, mRNA quantification using the same platform on matched FFPE and frozen tissues would make a more direct comparison. However, in order to determine whether gene signatures developed using fresh frozen tissues are applicable for clinical samples, comparisons between the two platforms using matched, but differentially archived tissues, is required.The nCounter-based mRNA profiling using FFPE technical replicates showed high reproducibility of the platform similar to the previous study . HoweverFor tumor and normal group comparisons, 166 of the 414 genes were identified as significantly differentially expressed genes by either platform and that of these 166 genes, 76 genes (46%) were expressed concordantly between the two platforms. Demonstrating an association with survival outcomes is more challenging, only 11% genes (6 genes) among significantly associated genes (55 genes) by either platform were concordant.in silico comparisons with other published studies. The degree of RNA degradation is another challenging issue facing transcriptome profiling strategies using FFPE-derived samples. For example, a recent study on hepatocellular carcinoma demonstrated that mRNA transcript measurements achieved using the nCounter system for archived sectioned FFPE samples compared poorly with freshly cut FFPE tissues [Multiple factors can affect these experimental results. The nCounter platform, by its very nature, represents a custom build of candidate gene elements as a biomarker, thus the selection of genes in an nCounter codeset inherently limits  tissues . This isArchived FFPE preserved tumor tissue remains the most available tissue specimen source for large-scale development and validation of prognostic and predictive gene signatures for colon cancer. Identification of subgroups of archival tumor tissues producing higher quality RNA samples for expression profiling is one approach to improving biomarker development from these samples. Improvements in technology such as new FFPE RNA extraction protocols, new FFPE RNA amplification and labeling methods, and modified data processing workflows can also help to side-step issues of RNA degradation and enhance the transcriptome data quality from RNA-Seq or microarray –36. In tS1 Fig(PDF)Click here for additional data file.S2 Fig(PDF)Click here for additional data file.S1 Table(CSV)Click here for additional data file.S2 Table(CSV)Click here for additional data file.S3 Table(CSV)Click here for additional data file.S4 Table(CSV)Click here for additional data file.S5 Table(CSV)Click here for additional data file.S6 Table(CSV)Click here for additional data file.S7 Table(CSV)Click here for additional data file.S8 Table(CSV)Click here for additional data file.


ChIP-seq has become a routine method for interrogating the genome-wide distribution of various histone modifications. An important experimental goal is to compare the ChIP-seq profiles between an experimental sample and a reference sample, and to identify regions that show differential enrichment. However, comparative analysis of samples remains challenging for histone modifications with broad domains, such as heterochromatin-associated H3K27me3, as most ChIP-seq algorithms are designed to detect well defined peak-like features.To address this limitation we introduce histoneHMM, a powerful bivariate Hidden Markov Model for the differential analysis of histone modifications with broad genomic footprints. histoneHMM aggregates short-reads over larger regions and takes the resulting bivariate read counts as inputs for an unsupervised classification procedure, requiring no further tuning parameters. histoneHMM outputs probabilistic classifications of genomic regions as being either modified in both samples, unmodified in both samples or differentially modified between samples. We extensively tested histoneHMM in the context of two broad repressive marks, H3K27me3 and H3K9me3, and evaluated region calls with follow up qPCR as well as RNA-seq data. Our results show that histoneHMM outperforms competing methods in detecting functionally relevant differentially modified regions.http://histonehmm.molgen.mpg.de.histoneHMM is a fast algorithm written in C++ and compiled as an R package. It runs in the popular R computing environment and thus seamlessly integrates with the extensive bioinformatic tool sets available through Bioconductor. This makeshistoneHMM an attractive choice for the differential analysis of ChIP-seq data. Software is available from The online version of this article (doi:10.1186/s12859-015-0491-6) contains supplementary material, which is available to authorized users. Post-translational modifications of histones, such as methylation, acetylation, phosphorylation or ubiquitination have central roles in genome function  and in tGenome-wide measurements of various histone modifications can be readily obtained using ChIP-seq technologies, which combine immmunoprecipation techniques with next generation sequencing . AlthougSeveral methods have been developed to facilitate comparisons of ChIP-seq samples for peak-like features ,10. HoweTo address these issues we developed histoneHMM, a novel bivariate Hidden Markov Model for the differential analysis of histone modifications with broad genomic footprints. histoneHMM aggregates short-reads over larger regions and takes the resulting bivariate read counts as inputs for an unsupervised classification procedure, requiring no further tuning parameters. histoneHMM outputs probabilistic classifications of genomic regions as being either modified in both samples, unmodified in both samples or differentially modified between samples.We extensively evaluate the performance of histoneHMM in the context of ChIP-seq data of two broad repressive histone marks, H3K27me3 and H3K9me3 from rat, mouse and human cell lines. Using several biological criteria and follow-up experimental validation, we show that histoneHMM outperforms competing methods in calling differentially modified regions between samples.histoneHMM is a fast algorithm written in C++ and compiled as an R package. It runs in the popular R computing environment and thus seamlessly integrates with the extensive bioinformatic tool sets available through Bioconductor. This makes histoneHMM an attractive choice for the differential analysis of ChIP-seq data.We analyzed ChIP-seq data collected from the left ventricle of the heart of two different inbred rat strains, Spontaneously Hypertensive Rat (SHR/Ola) and Brown Norway (BN-Lx/Cub). SHR is a classical animal model for hypertension which is extensively used in studies of cardiovascular disease. The bioAll of the analyzed histone marks and especially H3K27me3 and H3K9m3 are known to have large genomic footprints that can extend up to several thousands basepairs in length ,13. To eGenome-wide, histoneHMM detected 24.96 Mb (0.9% of the rat genome) as being differentially modified between the two strains for H3K27me3, and 121.89 Mb as differentially modified between male and female mice for H3K9me3 (4.6% of the mouse genome) , we performed additional functional validation of differential calls using RNA-seq data from age-matched animals Table .P=3.36×10−6, Fisher’s exact test, Figure We employed DESeq  to identP= 4.79·10−7). These were mainly genes from the MHC class I complex which is a key part of the innate immune response. Interestingly, all of the differential MHC genes are located in blood pressure quantitative trait loci (QTL) that were previously identified using either crosses derived from these two strains or from closely related strains )histoneHMM is not limited to this bivariate setting but can, in principle, be extended to an arbitrary number of dimensions. This feature makes it possible to compare a large number of samples for the same histone mark, or alternatively, many different histone marks in a single sample (e.g. in a single cell line). This latter application provides a formal method to detect and characterize combinatorial histone states in a genome-wide manner. Extension to histoneHMM to perform such a combinatorial analysis promises to provide a powerful alternative to chromHMM or Segway, two algorithms that have been employed in the ENCODE project.histoneHMM runs in the popular R computing environment and integrates with the extensive bioinformatic tool sets available through Bioconductor. This makes histoneHMM an attractive choice for the differential analysis of ChIP-seq data.Histone modification H3K27me3 was analyzed using ChIP-seq in the left ventricle of the heart from 3 biological replicates of each rat strain BN and SHR . We used the aligned reads (genome assembly hg19) resulting from the uniform processing pipeline of ENCODE and removed duplicated reads using samtools. Since the coverage of H3K27me3 was very low for H1-hESC . Reads were mapped to the BN reference genome rn4 using TopHat v 1.2.0. . Gene exLiver gene expression data for the comparison of female and male mice was obtained from gene expression omnibus (GEO accession GSE48109). This data also comprises differential gene expression results obtained by the authors using edgeR .FDR<0.01.ENCODE RNA-seq data for H1-hESC and K562 cell lines  was obtained from the UCSC ENCODE data center. Here we also used the aligned reads hg19) as proccessed by the ENCODE pipeline. We obtained read counts as measure of gene expression using gene annotation from ENSEMBL release 63. Differential gene expression was determined using the DESeq method p→, where each Ui∼Unif. For two random variables Zx,Zy with joint CDF G and marginal CDFs Gx,Gy the probability integral transformation can be used to obtain a copula G is the CDF of the multivariate Normal distribution and Gx,Gy are the corresponding univariate Normal marginal CDFs. To obtain a CDF for the original random variables X and Y with marginal CDFs F,θa,b) asρ,σx,σy. Φ−1 is the inverse of the univariate standard normal CDF and a of sample A, and b of B, respectively  in both samples and created four subsets of regions for all possible combinations of univariate states . Then for every given subset the read data  was transformed to f and f. Finally Σa,b was estimated by the sample covariance of the transformed data in each subset.The covariance matrix Since we are working with discrete count data we are interested in the probabilitiesWe evaluate this integral using numerical integration techniques .A and B. This HMM has four states, corresponding to the situations where both samples are unmodified, both samples are modified, only sample A is modified or only sample B is modified. The four fixed emission densities are given by the four components of the bivariate mixture . As recommended by the authors we used a threshold of P<10−5 on the P-value.Pepr is the only tool in the comparison that works only when replicates are provided, so we used all available replicates before merging them. The authors recommend a threshold of In this comparison we always used input control data when possible.FDR<0.01.We used macs version 2 with the broad option. The recommended threshold for peak calling was P>0.5 on the latent state probability to call regions.We used the mappability files for human, mouse and rat that were provided on the Zinba website. We used the generalized linear model with just the input count as predictor. For the ROC analysis we used the latent state probability of modification. For the comparison to gene expression we used the threshold FDR<0.01.Sicer was run with a window size of 200 bp and a gap size of 600 bp as recommended for H3K27me3 by the authors. Significant regions were identified using Broadpeak does not output scores and also does not require the specification of a threshold, therefore we just ran Broadpeak with default options and used all predictions that were returned.P>0.5 that we used for histoneHMM to call regions.We used the ‘rseg-diff’ software with ‘-mode 2’ to provide the input control data. We obtained latent state probabilities for modified regions for all bins in the genome. Finally we used the same latent state probability threshold histoneHMM and is freely available from http://histonehmm.molgen.mpg.de.The software was implemented in the R package