Sporadic Amyotrophic Lateral Sclerosis (sALS) is a devastating, complex disease of unknown etiology. We studied this disease with microarray technology to capture as much biological complexity as possible. The Affymetrix-focused BaFL pipeline takes into account problems with probes that arise from physical and biological properties, so we adapted it to handle the long-oligonucleotide probes on our arrays (hence LO-BaFL). The revised method was tested against a validated array experiment and then used in a meta-analysis of peripheral white blood cells from healthy control samples in two experiments. We predicted differentially expressed (DE) genes in our sALS data, combining the results obtained using the TM4 suite of tools with those from the LO-BaFL method. Those predictions were tested using qRT-PCR assays.TARDBP, SKIV2L2, C12orf35, DYNLT1, ACTG1, B2M, and ILKAP. Four of the seven have been previously described in sALS studies, while ACTG1, B2M and ILKAP appear in the context of this disease for the first time. Supplementary material can be accessed at:http://webpages.uncc.edu/~cbaciu/LO-BaFL/supplementary_data.html.LO-BaFL filtering and DE testing accurately predicted previously validated DE genes in a published experiment on coronary artery disease (CAD). Filtering healthy control data from the sALS and CAD studies with LO-BaFL resulted in highly correlated expression levels across many genes. After bioinformatics analysis, twelve genes from the sALS DE gene list were selected for independent testing using qRT-PCR assays. High-quality RNA from six healthy Control and six sALS samples yielded the predicted differential expression for 7 genes: LO-BaFL predicts DE results that are broadly similar to those of other methods. The small healthy control cohort in the sALS study is a reasonable foundation for predicting DE genes. Modifying the BaFL pipeline allowed us to remove noise and systematic errors, improving the power of this study, which had a small sample size. Each bioinformatics approach revealed DE genes not predicted by the other; subsequent PCR assays confirmed seven of twelve candidates, a relatively high success rate. We ara priori, but bioa priori-5 that da priori. Noise ha priori. Determia priori.We had available microarray data previously produced in a study by Mougeot et al.. BrieflyIn addition to applying the BaFL method to our data we performed a parallel analysis with the TM4 package,For rare diseases, such as sALS, it is difficult to obtain the large sample sizes needed to identify factors having moderate effects: in our study we had only 22 samples, 11 from patients with confirmed sALS and 11 from age and gender-matched healthy controls. We hoped to increase these numbers by combining our study with studies from other researchers, both to increase our statistical power and to broaden the effective population for any markers. We were unable to identify another sALS microarray study with publically available raw data. However, we did identify an Agilent 4x44k microarray expression data set from a Coronary Artery Disease (CAD) study whThe effectiveness of a data cleansing method can be assessed in a number of ways, including improvements in the accuracy of subsequent data mining efforts, whether detecting differential expression or sample classification. The moshttp://webpages.uncc.edu/~cbaciu/LO-BaFL/supplementary_data.html. Summarized results are reported below, and implementation details are in the Materials and Methods.For each LO-BaFL step the output is available along with the final results of our analyses, in the Supplementary Material section:Scanning for near-perfect and perfect matches between the 41,000 Agilent probes and HRG36.1 (Human Reference Genome build 36.1), identified 370,139 hits, or an average of ~9 matches per probe. The full list of matches is available on the project Web site/Cleansing process/tera_blast_results/tera_blast_raw.zip. In a gene expression array the impact of a perfect match to a secondary target depends on whether it is an expressed sequence: most cross-hybridization was to unexpressed regions. Perfect matches of probes mapped to protein-coding genes, with a few exceptions (described in (iii)).Using the Kane criteria and OligoArrayAux values, ~8.63 % of the probes had cross-hybridizing partners to expressed regions that would produce mixed measurements ; such probes were flagged for removal,26. The There are 407 probes that no longer map to an exon in HRG36.1, a list is given in ‘probes_info_no_pm_no_crosshyb_not_mapped.csv’ in Supplementary Data/Cleansing Process/loss_of_target_filter; these were flagged for removal. While several of these probes do map properly to transcripts, they must cross exon junctions. They are clearly described and could be returned to the data pool if the isoform were known to be present.Of the remaining probes, ~2.53 % map to targets containing 4 or more SNPs, so they were removed from the list of allowed probes. The file showing the probes with major/minor alleles for each SNP position is in Supplementary Material/Cleansing process/SNP filter/snp_info_probes_gt_4snp.csv. Should samples be SNP qualified, probes can be returned to the allowed list. We include another file, ‘agilent_probe_info_3SNPs.csv’ that describes probes with one to three SNPs.-1 separates probes that are less responsive to changes in target concentration from the majority of probes that are similar in their response to increasing target concentration. Applying this filter resulted in removal of ~21.5 % of the probes, listed in the Supplementary Material file under ‘DeltaG_filter’ tables.For 60-mers under the stated hybridization conditions, setting monomer folding stability to ΔG = −5.2 kcal mol10(I) > 4.5) vs. 1.5 % with this intensity in the probes without this feature. At the other extreme, 16 % of poly(G) probes had log10(I) < 1.1 vs. 7.7 % of the probes with no poly-G stretch. Since these probes have unusual behaviour at both extremes, we examined the final probe list for the frequency of this feature: only 19 poly-G containing probes were still present. Since 8 of these probes show very high signal (log10(I) > 4.5) we removed all of them.There are 4,742 poly-G containing probes in the original array design . Of these ~2.3 % had unusually high intensity mean and log10(I)median for this set of probes yielded values of 3.51 and 3.49 respectively, while for the CAD samples the values were 3.0 and 2.9 respectively. This is ten-fold higher than the value we find for most experiments using Affymetrix scanners  but the instruments are different. Because we lost so many genes from this filter we estimated a second cut-off value, using unexpressed genes having ΔG < −10 kcal mol-1 which results in a background estimate value that is about 10-fold lower (200–300 fluorescent units). In comparison, the TM4 pipeline uses Lowess smoothing to estimate background, shown in FigureEstimating the background and then filtering for measurements that fall below it is experiment specific. To retain a probe it must have a valid measurement in every array in a sample class. In the sALS study ~27,000 probes were removed by this filter; see Supplementary Material/Cleansing Process/Instrument_cutoff/all_samples_gt_instrument_cut-off_log_intens.csv for the list. In our sALS arrays, logAll of the samples in the sALS experiment passed outlier detection tests, including those whose residual samples had poor RNA Integrity Numbers (RIN), which we interpret to mean that the target applied to the microarray had acceptable quality, with degradation occurring during storage. CAD samples were similarly tested: no outliers were detected.In both the CAD and sALS studies, Fisher’s test indicated unequal variances between the healthy control and diseased groups of samples. Testing measurements for the individual probes for the DE genes of interest, across replicates, in the two sample classes, using the Shapiro-Wilks test for normality-30, alsoFor input data to the Wilcoxon test we used measurements from either the 12 sALS samples with the highest quality RNA (6 in each class) or from all 22 samples (11 in each class); the latter resulted in a shorter input list as some genes were eliminated when their probe signal fell below background in a few arrays. After processing measurements with TM4 or LO-BaFL and applying the Wilcoxon test we first used the Bonferroni multiple-test correction, but were left with no significant DE genes in any of the trials. However, applying a FDR with p < 0.05 resulted in a set of significant differentially expressed genes from each method. In the output files the results for TM4 are labelled ‘TM4/W12’ and ‘TM4/W22’, results for LO-BaFL are labelled ‘LO/W12’ and ‘LO/W22’. To verify that differences were not due to differing implementations of the Wilcoxon test we also loaded LO-BaFL filtered data into SAM and selected the Wilcoxon test (results are labelled ‘SAM/W12’ and ‘SAM/W22’). The main caveat in interpreting this comparison is that there must be sufficient observations in the expected classes for each method to be valid.The LO/W22 sALS experiment returned 87 probes as DE and the LO-W12 experiment returned a subset of 60 of those genes. For the CAD study, 386 genes were found to be differentially expressed, and this set included those confirmed by RT-PCR whose probes had not been removed by LO-BaFL filtering. The list with all DE genes for ALS is provided for each analysis in Supplementary Material/Data post filtering/DE_genes_12_samples tables. DE genes for CAD experiment are listed in Supplementary Data/CAD Study/DE_genes_CAD/DE_genes_CAD_data.csv.JUNB) or with TM4 (the gene being DYNLT1). Checking the list of LO-BaFL deprecated probes shows that four of the TM4 DE genes fell below the background estimate value for LO-BaFL, explaining their absence. The final TM4-predicted gene, DYNLT1 did not appear on the LO-BaFL list because it did not meet the p-value criterion.TableACTG1, SKIV2L2, C12 orf35, B2M and DYNLT1 and more modest values for ILKAP and TARDBP. Our experimental results are in good agreement with the recent report by Mougeot et al.[SKIV2L2, C12orf35, DYNLT1 are differentially expressed in PBL samples from patients with ALS vs. normal healthy control subjects. As in previous studies, TARDBP is among the genes with differential expression for ALS[ACTG1, B2M, and ILKAP.Levels of the four reference genes were stable and at the relative expression levels with respect to each other remained unchanged across all samples (data not shown). The summarized results for the DE genes are shown in Tablet et al. showing  for ALS,34, althTARDBP and its cross-hybridizing gene, ILKAP, highlights why using only TM4 is likely to lead to errors as well. In fact using both pipelines and then confirming the predictions with qRT-PCR is the best approach to achieve a complete and accurate assessment of potential biomarkers. Because LO-BAFL explicitly lists excluded probes by relevant filter, it is possible to add back particular probes when additional information warrants, a procedure not possible with TM4.Each analysis method predicted DE of genes excluded by the other, and in each case some of the predictions were verified with the independent assay. LO-BaFL is likely to exclude some genes that are DE because of its high background estimation value and because it removes probes that are only problematic in some populations, leading to false negatives. The example of 2 = 0.81), indicated visually by the flatness of the Lowess smoothing line in red  and those at the lower end of detection  are the same and characteristic of white blood cells. We carried out this analysis because determining differential expression is entirely a function of the baseline data set. In the sALS study both the diseased and reference populations were very small, and we knew that for some of the samples the quality of the remaining RNA was not high. We wanted an independent assessment of the healthy control samples: although the cell types in the sALS and CAD studies are not identical, this was the closest match we could identify. We are aware that subpopulations of cells such as PBMCs and PBLs have been shown to have distinct response signatures[The gene-to-gene correlation for healthy normal controls across the sALS and CAD experiments was quite high , four are close variants of the DE genes, while the remainder of the genes reported as DE in the paper were eliminated from our list based on the criteria already discussed . The file listing DE genes for this experiment as determined by our method is found in the file ‘LO-BaFL_DE_genes_CAD_data.csv’, located in the directory mentioned above.For the CAD study, the list of most significant DE genes determined by LO-BaFL was compared with results reported in. Two of ACTG1, SKIV2L2, C12orf35, B2M, DYNLT1, TARDBP, and ILKAP, all of which were elevated in sALS versus healthy Controls. The four control (non-DE) genes showed their predicted responses as well. These results confirm DE genes obtained in other studies[ACTG1, B2M, ILKAP.We showed that results obtained from a modified probe-filtering method (LO-BaFL) gave comparable performance to a statistical filtering method (TM4), using data obtained from independent experiments that were performed using similar cell types and the same platform with similar healthy control groups, but differing disease states. For the sALS experiment the top predicted DE genes from each method, and two that cross-hybridized with one of the candidates, were tested with qRT-PCR. We confirmed the microarray DE predictions for 7 of the genes:  studies,38,39, aTARDBP. On the Agilent array this probe cross-hybridizes with IlKAP and two other genes. TM4 assigns the signal to TARDBP, and here identifies it as DE, but ILKAP is missing entirely and we show here that it does contribute to the signal and is also differentially expressed. LO-BaFL had eliminated the probe because of cross-hybridization but by combining the results of both methods we could test all candidates with qRT-PCR. Thus we produced better DE data for TARDBP and new data for ILKAP.Employing different methods to obtain DE genes on same data set has the advantage of accounting for errors inherent to each method. The LO-BaFL pipeline identifies and eliminates cross-hybridizing probes that TM4 does not, as illustrated by Comparing normal samples from the CAD experiment with thoFor data storage, data organization, and recording the order and parameters used in the pipeline transformations, we have used DataFATE , a software system based on a relational model that includes a toolset with data import and organization tools for relational database management systems (RDBMS), tools for factor  definition, QT set construction, and storage of data from processing steps. The RDBMS is currently PostgreSQL 8.0.3.. The proTM[Microarray image files and corresponding spot intensity values for the sALS study were provided by JLM and BRB. The microarray experiment used Agilent 4x44K human genome microarrays in a pooTM, produciA major shortcoming of many analysis methods is that they are over-tuned to a particular experiment, so that parameters that yield excellent results in one study give poor results in another. If the LO-BaFL filters are experiment neutral (except for the estimation of background) then LO-BaFL should predict the behaviour of genes from similar samples but different experiments relatively well. We identified one experiment that used human peripheral white blood cells and the same type of Agilent array. Some of the DE genes had been followed up with qRT-PCR assays, so these became our benchmark for determining whether LO-BaFL could produce a comparable DE gene list. Although the disease conditions differed, the healthy Control samples were similar and could provide a check on the quality of the sALS Control samples. The experiment studied coronary artery disease (CAD) from healthy controls (n = 14) and patients with disease (n = 27). The datThe steps in the pipeline described below are summarized in FigureA. In this section the probe-sequence based filters are described.Note that the order of operations is independent for the above filters; some probes fall into multiple categories so summing over the ‘bad’ probes identified per step will give a larger number than the total number of probes removed.Rather than remove flawed measurements from individual files, flawed probes and probes for which there were not consistent above-background measurement in all samples (discussed next) were removed from the array layout file with Aroma, ensuring consistent removal of the related measurements across all files.http://webpages.uncc.edu/~cbaciu/LO-BaFL/supplementary_data.html under Input Files/agilent_fasta or Cleansing Process/tera_blast_results.(i) Re-map the Agilent probes to build 36.1 of the human genome using the accelerated Tera-BLAST algorithm, as implemented on a TimeLogic-Decypher server. (ii) Determine the cross-hybridization potential of probes to other sites in the genome, using the Kane criteria. Briefly(iii) Identify probes that no longer anchor to the reference genome. This information is acquired when a TeraProbe query returns ‘no hit’, and this is stored as an explicit type. We note that our criteria include restricting location to one exon.(iv) Identify SNPs and short indels known to occur in the probe-binding region. Probes were mapped to the human instance of dbSNP, taking + and 0.0 M Mg++. Output was used to define probes that are less responsive to target (high concentrations are required to compete with monomer structure) under experimental conditions. Heterodimers with low stability under specific experimental conditions also do not yield signal[(v) Employ the OligoArrayAux software to deterd signal-51. The (vi) Identify G-runs. It has been shown for Affymetrix arrays that four or more consecutive guanines (G-runs) may lead to unusual probe structures that cause very high signal. We iden(vii) The presence of any member of the transposable elements family, short (SINE), long (LINE) or primate-specific (Alu) repeat elements, can have a great effect on gene expression. Using the TranspoGene database we examiB. Background (Noise Estimation)We were unable to find technical specifications for the linear detection range of the Agilent scanner so we had to estimate the value. This isC. Sample Outlier DetectionThe sALS experiment used a pooled reference design, so we determined the reproducibility of the reference signal across all of the samples. An outlier was defined as any sample whose signal fell outside two standard deviations of the class mean for either the average signal per probe or the average number of probes per array. We compared the signal intensities for both the normal and diseased signal to the mean and variance of each class. Two contrasts were examined:(i) Sample to class comparison: Using only accepted probes with measurements above the threshold in a sample class, determine the mean signal per probe per array and across all arrays in the class. Samples whose probe-signal mean falls more than two standard deviations outside of the array mean are rejected. Determine the number of acceptable probes yielding good measurements per array and across all arrays in the class, and similarly reject any sample whose probe-number mean falls more than two standard deviations from the class mean.10 transformed).(ii) Sample distribution comparison: Determine whether the remaining values fall in a normal distribution, and compare the within- category and between-category distributions. If not normal this would suggest that a log transformation might be advisable ; these values are used by the Significance Analysis of Microarray (SAM) package using the Wilcoxon non-parametric test with permutation (set to 100),32 to id10 transformed measurements of accepted probes since the arrays were found to have similar background and total intensity. These values were submitted to the Wilcoxon non-parametric t-test implemented in R (which does not use permutation) to determine DE genes for both the high RIN and full set of samples (result sets are ‘LO/W12’ and ‘LO/W22’ respectively).The LO-BaFL workflow simply used the logIn the absence of a calibration standard the actual expression levels of genes in the individual samples are not readily available. Thus the wet-lab work had two goals: determine the level of expression that a microarray value yields in a qRT-PCR assay; determine whether either analysis method was accurate in its predictions of the difference in expression levels between genes in normal and diseased samples. TableFrom 2007 through March 2008, the blood samples used for microarray and PCR analysis were collected at Carolinas Neuromuscular/ALS-MDA Center with approval by the IRB at Carolinas Medical Center. Informed consent was obtained from all participants in this study. Detailed information about patient status and early stages of sample processing are provided in. The isoUBE2Z, PGK1, COX4I1, SRRM1. A reference is context specific: these were selected based on an apparently consistent level of expression in the microarray data across sample classes and individuals, over a moderate range of concentrations. For all genes the PCR primers were designed to bridge the position of the array probe where possible (the set cross-hybridizing with the TARDBP probe had to be adjusted) and tested against genomic DNA. The list of test qRT-PCR genes and the assay primers is given in Table4.A properly designed qRT-PCR assay includes calibration genes for the sample and assay. For ourTM Complete Transcriptome RNA Amplification Kit from System Biosciences , according to the manual. After quantification of the yield, and standardization of the concentrations, the cDNA products were qualified by determining whether the four reference gene PCR primers yielded the expected size product on 12 % Acrylamide (native) gels[2, 50 mM stock solution, for Mg++ 3.5 mM final concentration), 2.5 μl/reaction dNTP mixture , 0.5 μl/reaction DNA Taq Polymerase  5U/mL in stock; final concentration of 2U/mL), 2.5 μl/reaction forward and reversed primers  solubilised with DNA Suspension Buffer for a concentration of 100 mM stock; final concentration of 5 mM), 2.0 μl/reaction cDNA as template (100 ng). The GeneAmp® PCR System 9700 from Applied Biosystems  was set up to the following profile: the initial DNA denaturation at 95 °C for 5 min; 30 cycles of denaturation at 94 °C for 30 s, primer annealing at 57 °C for 30 s and extension at 72 °C for 30 s; a final elongation at 72 °C for 4 min and a 4 °C hold.To test primers and conditions for the qRT-PCR assays, we extracted RNA from an anonymous sample of white blood cells, suspended in Triazol and kept at -80 °C, using the AllPrep DNA/RNA Mini Kit from Qiagen , following the manufacturer’s instructions. This RNA was qualified and quantified as above. We then synthesized double-stranded cDNA from the ALS samples that passed the RNA quality/quantity test  and from the control RNA, using the Full Spectrumve) gels. Even whhttp://frodo.wi.mit.edu/primer3/) in combination with NCBI Primer-BLAST (http://www.ncbi.nlm.nih.gov/tools/primer-blast/) to check for specificity. Whenever possible, the primers were designed to bridge the positions occupied by the corresponding Agilent probes, in order to account for sensitivity to transcriptional isoforms. Primers were purchased in dry form from Eurofins mwg|operon and resuspended in DNA Suspension Buffer; concentrations were verified with the Nanodrop1000 spectrophotometer, length and purity were gel-verified using 12 % Acrylamide in 1X TBE buffer[++ concentration in a range of 2.5 – 4.0 μM, the annealing temperature in the interval 55 °C - 60 °C and varying the dNTP mixture concentration from 2.5 μM - 3.5 μM. Where necessary new primers were designed and run again through the QC protocol mentioned above.The primers were designed using Primer3 software  that includes 2X reaction buffer with dNTPs, iTaq DNA Polymerase, 6 mM MgCl2, SYBR Green I, fluorescein and stabilizers according to the Bio-Rad specifications; forward and reversed primer mixture (2.0 μl/well at 5 mM); 5.0 μl/well of template solution which variously contained the standard gene, the unknowns or Accugene water , and 3.0 μl/well Accugene water in final reaction volumes of 20.0 μl/well. Titration series were set up as follows: six 10- fold serial dilutions of the gene product reference and of the samples, in triplicates, with negative controls in all series to identify any cross-contamination problems. The reactions were set up in 96-well clear Multiplate® PCR Plates covered with iCycler iQTM Optical Tape . The instrument employed for these reactions was MyiQ Single-Color Real-Time PCR Detection System from Bio-Rad. We used a 2-step protocol with the following profile: Cycle 1:(1X) Step 1: 95.0 °C for 03:00; Cycle 2:(40X) Step 1: 94.0 °C for 00:15; Step 2: 57.0 °C for 00:30; data collection and real-time analysis enabled; Step 3: 72.0 °C for 00:15; Cycle 3: (1X) Step 1: 95.0 °C for 01:00; Cycle 4:(1X) Step 1: 55.0 °C for 01:00. Data were analysed according to the method of Pfaffl[Assays were set up in parallel, taking PCR reagents from a master mix to amplify the gene product reference at known concentrations against a mass-titration of a sample’s cDNA product. We usedf Pfaffl.The authors declare no conflict of interest.JWW designed the experiments; CB performed the research; J-L.M and BRB provided the PBLs samples, the raw ALS microarray data and contributed to TM4 data analysis; KJT helped with data analysis and figures; CB and JWW wrote the paper. All authors read and approved the final manuscript.Supplementary Material can be freely accessed at the author’s project Website:http://webpages.uncc.edu/~cbaciu/LO-BaFL/supplementary_data.html.Click here for file


Recently, several classifiers that combine primary tumor data, like gene expression data, and secondary data sources, such as protein-protein interaction networks, have been proposed for predicting outcome in breast cancer. In these approaches, new composite features are typically constructed by aggregating the expression levels of several genes. The secondary data sources are employed to guide this aggregation. Although many studies claim that these approaches improve classification performance over single genes classifiers, the gain in performance is difficult to assess. This stems mainly from the fact that different breast cancer data sets and validation procedures are employed to assess the performance. Here we address these issues by employing a large cohort of six breast cancer data sets as benchmark set and by performing an unbiased evaluation of the classification accuracies of the different approaches. Contrary to previous claims, we find that composite feature classifiers do not outperform simple single genes classifiers. We investigate the effect of (1) the number of selected features; (2) the specific gene set from which features are selected; (3) the size of the training set and (4) the heterogeneity of the data set on the performance of composite feature and single genes classifiers. Strikingly, we find that randomization of secondary data sources, which destroys all biological information in these sources, does not result in a deterioration in performance of composite feature classifiers. Finally, we show that when a proper correction for gene set size is performed, the stability of single genes sets is similar to the stability of composite feature sets. Based on these results there is currently no reason to prefer prognostic classifiers based on composite features over single genes classifiers for predicting outcome in breast cancer. Modern high-throughput methods provide the means to observe genome wide changes in gene expression patterns in breast cancer samples. Gene expression signatures have been proposed The underlying idea of these methods is that genes do not act in isolation, and that complex diseases such as cancer are actually caused by the deregulation of complete processes or pathways, representing ‘hallmarks of cancer’ et al.et al.et al.The observed improvement in classification accuracy achieved by the approaches employing secondary data is hard to assess since it is dependent on many factors such as the specific data sets and evaluation protocol employed. To shed more light on this issue we performed an extensive comparison of a simple, single genes based classifier with three of the most popular approaches that include secondary data sources in the construction of the classifier. More specifically, we included the approaches proposed by Chuang et al.Chuang et al.et al.et al., Lee et al. do not exploit the connectivity of the pathway in the construction of the composite features. Lee et al. claim that by using these pathway activities a higher classification performance can be achieved on different cancer types, most notably leukemia, lung and breast cancer. They also report a higher overlap between genes in the top scoring composite features compared to the top scoring single genes.Lee et al.et al. and Chuang et al., Taylor et al. detect aberrations in the correlation structure between a hub and its immediate interactors. As correlation between two genes cannot be assessed for a single sample, Taylor et al. employ the pairwise expression difference between the hub and each of its interactors as features for the classifier. While no claims are made regarding performance improvements, we included this approach in the comparison as it is a recently proposed, novel approach for exploiting secondary data sources to predict outcome in breast cancer.The underlying assumption in the study by Taylor i.e. composite feature construction, the selection of the optimized number of features for classification and the training of the final classifier are all performed on the training data set, while the testing of this trained classifier is performed on a completely separate test set without calibrating the classifier on the test data. See All three studies listed above use their own specific cross-validation (CV) protocol and evaluate their method on different (combinations of) data sets. This makes it hard to assess the improvement over other methods. In this work, we therefore employ an unbiased training and validation protocol and present a comprehensive evaluation of cross data set classification performance and stability on six publicly available breast cancer data sets. Given that these classifiers are intended to predict the unknown outcome of a patient, we suggest a cross-validation procedure that does not assume any knowledge about the samples used for testing. Thus, we strictly separate the training data set from the test data set, et al.To prevent biases associated with a specific secondary data source, we tested the algorithms on different types of secondary data sources.  using single genes with an NMC employing feature extraction methods based on pathway and PPI data. The results are depicted in Lee all composite LOG classifiers perform equally or even worse than the single genes LOG classifier. However, it should be noted that the performance of the LOG classifier is highly variable as a function of the number of included features–see Taylor composite NMCs. Clearly, the LOG classifier as employed here  requires additional regularization to ensure convergence across the whole range of feature values. Also in combination with this classifier, Taylor performed poorly. This together with the high computational burden associated with this method, prompted us to omit Taylor from the remaining analyses.We also provide the classification results for the logistic regression (LOG) classifiers in Based on the results presented in Next we formulated a number of hypotheses that could explain why classifiers employing secondary data sources do not outperform single genes classifiers. These hypotheses relate to (1) the feature selection approach employed; (2) the starting set of genes employed in each of the approaches; (3) the effect of the training set size on performance and (4) the homogeneity of the data set employed. In the following sections we will investigate these hypotheses one by one.et al.Our cross-validation protocol employs individual feature filtering to select an optimized number of features for the classifier. We . We can, however, also follow an alternative scheme where we train on all data sets except the test set, the so-called merged setting. More specifically, in this setting four of the five Affymetrix data sets were merged to form a single training data set and the fifth data set was used as test set. Thus, we receive for each feature selection method five AUC values. This increases the size of the training set, and by comparing the results obtained in this setting with the results from the paired setting, we can investigate the effect of the training set size on the classifier performances.A third possible factor that could explain the lack of performance difference between the composite feature classifiers and the single genes classifier is the size of the training set. Recall that in the cross-validation procedure we train on one data set and then test on another data set. We repeat this procedure for all possible pairs of data sets; excluding, of course, training and testing on the same data set  and on the other hand by the secondary data sources (composite features) vary immensely. Please note that in this specific context we do not employ the Fisher exact test to test a certain hypothesis. We rather interpret the p-values returned by the test as a measure of enrichment or overlap between two sets of marker genes. Thus, the smaller the p-value the more unlikely the overlap. In addition to the Fisher exact test, we calculated the overlap between gene marker sets by employing the Jaccard index and compensated for the size of the population from which the signatures were selected by a random sampling procedure. See Section 7.2 in In this study we evaluated the prediction performance of network and pathway-based features on six breast cancer data sets. In contrast to previous studies we found that none of the classifiers employing composite features derived from secondary data sources can outperform a simple single genes classifier. Moreover, we did not find any evidence that composite features show a higher stability across the six breast cancer cohorts. Our findings suggest that with the feature extraction methods tested in this study, we cannot extract more knowledge from secondary data sources than we find in the expression of single genes.i.e. treating each single sub-network or sub-pathway as a single dimension in the classification space. Possible dependencies between features are not taken into account. Also, exploring the subnetwork search space in a heuristic manner may decrease classification performance. The recent method by Dittrich et al.There are several issues that could potentially contribute to that situation. First, secondary data sources are, to a large degree, generated by high-throughput biological experiments and could thus contain a level of noise that deems them inappropriate for outcome prediction in breast cancer. On the other hand, the search algorithms could be unsuitable to detect biologically meaningful networks. All three feature extraction methods only extract local information without taking into account the full structure of the network or pathway data. This local information is then combined in the classifiers in a rather crude way, namely by simply averaging the expression of the genes associated with a feature that was found to be associated with outcome, et al.et al.et al.et al.et al.i.e. a subset of the poor outcome samples, is an interesting aspect for further evaluation, in particular for breast cancer outcome prediction as it may quite accurately capture the large phenotypic variety of this rather inhomogeneous disease.Other recent algorithms as presented by Ulitsky et al.et al.et al.Lee et al.To find a subnetwork scoring function remains one of the biggest problems when including promising gene sets into a classification framework. Abraham et al.Chuang and Lee, look for highly differentially expressed subnetwork or pathway markers, and these can also be found in the randomized data. Furthermore, overlaying networks or pathways that contain protein level information with mRNA expression data might result in erroneous results. These data sources reflect events on very different molecular levels. While gene expression and protein expression is undoubtedly linked, there are many processes that prevent this from being a trivial one-to-one mapping. Thus, we may measure, for a set of genes, an effect on the mRNA level that leads to differential expression between the two patient classes but this may have little bearing on the relationships between these genes as captured in the PPI graph. In conclusion, our results show that it may not be sufficient to search for sub-networks or sub-pathways that are differentially expressed on average but that complex interactions between entities as well as the more complicated relationships between mRNA levels and the topology of the PPI graph need to be taken into account.In our experiments where we shuffle the genes in the secondary data we showed that features determined on this nonsensical biological data perform equally well in classification than features determined on the real secondary data. This again could possibly be caused by the low quality of the network and pathway data. However, the nature of gene expression patterns in breast cancer, and specifically its association with outcome can also explain these findings. Since many genes are involved in breast cancer and are differentially expressed and associated with outcome, as shown, for example by Ein-Dor et al.et al.bona fide independent validation of the classifier as the training and test sets are not strictly separated. An advantage of the approach employed by Chuang et al.et al. is that differences between platforms can more easily be accommodated. This is less of a problem in our study as five of the employed datasets in our study come from the same Affymetrix platform (U133A) and only one dataset from the Agilent platform. Our different classification results are partially owed to the fact that we used a different cross-validation procedure, which, in our opinion, fits the clinical situation, better. The studies by Chuang Lee. In general, this hints at the fact that the choice of classifier influences the ranking of the feature extraction methods. However, the NMC and 3NN achieved similar performances for all feature extraction methods. Moreover, in accordance with Popovici et al.Also the usage of various types of classifiers in this study contributes to the different findings. Here we tested the NMC, LOG and a 3NN classifiers. While the LOG classifier generally performed much worse than the NMC, it did yield better classification results when employing features extracted with In summary, we introduced a framework to test the use of feature extraction methods with respect to the prediction of their determined features. We used this framework to specifically test the superiority of feature extraction methods based on network and pathway data over classifiers employing single genes. Across six breast cancer cohorts, we showed that the three tested methods do not outperform the single genes classifier nor do they provide more stable gene signatures for breast cancer.An important aspect that hampers progress in the field of network and pathway based classification is the lack of proper evaluation of proposed algorithms. In our opinion this is caused by (1) lack of reproducibility of the results; (2) lack of large and standardized benchmark sets to test proposed algorithms and (3) lack of a standardized, unbiased protocol to assess the performances of proposed methods on the benchmark sets. To overcome these issues, we have created a software pipeline that implements all the classifiers as faithfully as possible and also runs our validation protocol. We have also established a large collection of breast cancer datasets (and this is currently being expanded) on which the algorithms can be tested. Both the datasets and the pipeline are freely available. In the long term, we envision a web service where a classifier can be submitted as a software package. The server will then autonomously evaluate the performance of the classifier using the standardized pipeline on the benchmark set.The microarray data sets used in this work is listed in i.e., removing these ambiguous probes from the expression matrices and re-executing our cross validation procedure for all of the methods, yielded the same performances and overlap between markers. ER status of patients was predicted from the expression values of the gene ESR1. For more detail of the processing of the data see van Vliet et al.We found that probes matching several geneIDs in the Affymetrix datasets were not discarded, instead the smallest geneID was used to map the probe to a certain gene. This could potentially influence the mapping from gene expression to the network or pathway data and thus, influence the classification result and overlap of gene markers. However, classification on the corrected expression data, et al.All feature extraction methods were run on the databases KEGG Homo sapiens (hsa) from the KEGG database, version December 2010. The entries contained information on metabolic pathways, pathways involved in genetic information processing, signal transduction in environmental information processing, cellular processes and pathways active in human disease and drug development. We obtained 215 pathways. The nodes contained in the pathways were matched with the KEGG gene database such that each node carries an Entrez GeneID. In this way we obtained a network composed of 200 pathways containing 4066 nodes and 29972 interactions of which 3110 nodes are also contained in the expression sets.We collected all pathway information available for et al.et al.As second pathway database we used the C2 collection of the MsigDB (version 3) The HPRD (version 9) provides information on protein-protein interactions (PPI) derived from the literature. The HPRD contains 9231 proteins and 35853 interactions. The proteins were mapped to their genes carrying Entrez GeneIDs. There are 7390 genes contained in both the HPRD and the expression sets.C. elegans. The database contains 12643 nodes and 142309 edges. 9453 of the nodes are also present in the six breast cancer studies examined here.The OPHID/I2D database, downloaded in April 2011, contains protein-protein interactions derived from BIND, HPRD and MINT as well as predicted interactions from yeast, mouse and et al.The network curated by Chuang k samples and n genes. Each entry j in sample i. All samples carry a binary class label G is the set of genes in the network and I is the set of interactions between these genes, also called edges in the following. We define a subnetwork as the connected graph i is given asLet et al.N is used once as seed. In each step, an additional gene is picked that leads to a maximal MI improvement. If no improvement is possible, the search stops.Given a network More precisely, the association between the subnetwork activity and the class labels is computed as follows: Given a subnetwork In our study we use PinnacleZ, an implementation of the algorithm provided by the authors. As feature values for classification the subnetwork activity as given in Equation (1) of the determined subnetworks was used.omit its usual gene-wise z-normalization step.Before determining the subnetworks, PinnacleZ performs a z-normalization of the given data set. This is undesirable when looking at subsets of data sets as we do in the five-fold cross-validation. In order to skip the normalization step, we implemented a patch in the PinnacleZ source code. This patch adds a “-z” option that instructs PinnacleZ to et al.One problem when mapping the expression data to the network data is that for some nodes there is no expression data. Chuang et al.j for all samples with class label 1 and j for all samples with class label 0, respectively. Let The algorithm by Lee t value is negative, or in descending order, if the highest absolute t value is positive. Given this order the greedy search iteratively combines genes and calculates their average expression, or pathway activity, across the samples as it is given in Equation (1), i.e.m genes according to the ranking. To evaluate the combined discriminative power of the genes that have been averaged, the t-statistic of the averaged expression is computed as follows:The genes in a pathway are sorted either in ascending order, if the highest absolute Lee can only be executed on predefined gene sets. Those gene sets are normally not provided in a PPI database. Thus, we used the KEGG and MsigDB databases to evaluate this algorithm. In order to decrease the running time the authors executed a pathway ranking by employing the algorithm by Tian et al.et al.As mentioned beforehand, et al.et al.In contrast to the algorithms by Chuang h and an interactor n of this hub is defined as the Pearson correlation between their expression profiles across the k samplesCandidate hubs are the 15% most densely connected proteins in the network data, independent from their expression status. For the following calculations proteins without expression data are excluded. To identify hubs that significantly change their interaction behaviour between the two classes the authors introduce the hub difference and the average hub difference which are based on the Pearson correlation. The Pearson correlation between a hub k samples and h then the average hub difference isLet To extract only those hubs that show a significant average hub difference the value is compared to a distribution of the average hub difference for a permuted dataset, using a p-value cut off of 0.05. This distribution is calculated by 1) randomly permuting the class labels and 2) recalculating the average hub difference and repeating these two steps 1000 times. The significant hubs are ranked by their average hub difference.As feature values in the classifier differences of the expression of the hub and each of its interactors were used. For example, suppose Since the edges attached to a hub are not ranked, all those edges were included in the classifier, given that the hub shows a significant average hub difference. For the cross-validation procedure this means that we can not train the number of features but only a number of feature sets.w of each neighbor to a sample’s score byjth neighbor and In our study we employed a nearest mean classifier (NMC), logistic regression (LOG) and a 3-nearest neighbor classifier (3NN). As distance metric for the NMC we employed the Euclidean distance. More specifically, a sample is projected on the line connecting the two class means, and the Euclidean distance of the projected sample to each class mean is computed. The sample is assigned to the closest class. In addition to the NMC we executed all features extraction methods in combination with the LOG. We found that simple LOG without any regularization parameters cannot be trained properly since for higher numbers of features (approximately 50 features and more) the training step does not converge on the breast cancer data. Moreover, we found that for many features different imfplementations of LOG return different weight vectors. Thus, we used three different implementations  and only accepted the classification result of the R GLM implementation when all three versions converged to the same weight vector. Furthermore, we tested the classification performance of pathway and network based features and single genes in a 3-nearest neighbor classifier (3NN). As distance metric we chose the Euclidean distance. Further, we weighted the contribution In the cross-validation procedure we employed, we rigorously separate the training and test data sets. For details, see Cross-validation procedure.Require: DatasetsFeature extraction methodSecondary data sourceN.Classifier/* Cross-validation on k and 1: fordo2: 3: 4: Train */5: rankedF6: 7: for to ndo8: i features */9: 10: 11: end for12: end for13: /* train CV-optimized classifier on for  to ndo14: 15: end for16: 17: 18: 19: 20: 21: returnAUC22: i.e. the test set is never used in training the classifier.The training phase consists of determining the best performing number of features and training the final classifier with this number of features. The trained classifier is then tested in the test phase. The data sets used in these two phases are completely independent, To determine the optimized number of features in our classifiers we employed five fold cross-validation. In this cross-validation, we first determined all required composite features (if necessary) and their ranking on four splits of the training data set. Then a series of classifiers is trained on the same four splits by gradually adding features according to the ranking. These classifiers are then evaluated on the fifth split of the data set. Since this is done in a five fold cross-validation we obtain for each of the classifiers five different AUC values. The optimized number of features extracted corresponds to the number of features yielding the highest mean performance. Once the best performing number of features is determined, the features are calculated using the whole training data set and the final classifier is trained also using the complete training data set. The classifier is then tested on the test data set. For each method we used all possible pairs of data sets as training and test set respectively. Since we have six data sets available this resulted in 30 AUC values for each method.File S1Supporting Information File. The File contains all supporting Figures, Tables and Texts.(PDF)Click here for additional data file.


Motivation: Alternative splicing and other processes that allow for different transcripts to be derived from the same gene are significant forces in the eukaryotic cell. RNA-Seq is a promising technology for analyzing alternative transcripts, as it does not require prior knowledge of transcript structures or genome sequences. However, analysis of RNA-Seq data in the presence of genes with large numbers of alternative transcripts is currently challenging due to efficiency, identifiability and representation issues.Results: We present RNA-Seq models and associated inference algorithms based on the concept of probabilistic splice graphs, which alleviate these issues. We prove that our models are often identifiable and demonstrate that our inference methods for quantification and differential processing detection are efficient and accurate.Availability: Software implementing our methods is available at http://deweylab.biostat.wisc.edu/psginfer.Contact:cdewey@biostat.wisc.eduSupplementary information:Supplementary data are available at Bioinformatics online. AP events result from the phenomena of alternative splicing and alternative polyadenylation sites, which can give rise to different mRNAs from the same pre-mRNA, and alternative transcription start sites, which can lead to different pre-mRNAs produced from the same gene locus. The alternative transcripts resulting from these processes are important for development and have also been implicated in disease . PSGs build on the concept of splice graphs  detection given RNA-Seq data and fixed gene structures, which may be provided by an existing genome annotation or predicted from the data via transcriptome assembly. Our approach is based on a class of models that were first independently described in the context of expressed sequence tag (EST) analysis  RNA-Seq reads identify only a small fraction of the transcript from which they are derived and (ii) alternative transcripts from the same gene typically share a large amount of sequence.Drosophila gene Dscam is famous for having four alternatively spliced domains that result in over 38 000 possible isoforms  span both domains  parameters of a PSG, given single or paired-end RNA-Seq data. The estimation of PSG parameters is equivalent to the alternative transcript quantification task, with the restriction that transcript abundances must be consistent with the frequencies of dependent or independent processing events. Our algorithm runs in polynomial time even if a gene may generate an exponential number of isoforms, assuming the isoforms may be represented compactly as a PSG. Through experiments on real and simulated data, we assess the accuracy of the estimates from our method and demonstrate its theoretically predicted advantage over a more simplistic junction-read (JR) approach that does not make full use of the data. Our parameter estimation method forms the basis of a couple of DP tests, which we show to have low false-positive (FP) rates using real replicate data and superior true-positive (TP) rates on simulated data.As we will show, our novel application of the PSG modeling framework to RNA-Seq analysis alleviates all three of these issues. First, we present an efficient Expectation-Maximization (EM)-based algorithm for estimating the maximum Second, we provide conditions under which a PSG is provably identifiable given RNA-Seq data, conditions which are less restrictive than those for full-length isoform models. Thus, the biologically motivated assumption of independence between processing events has an important statistical side benefit.de novo assemblers, our method has been implemented in such a way that a reference genome is not required. Thus, our software may be used in conjunction with such assemblers to study alternative splicing in species without a sequenced genome.Lastly, we argue that transcriptome assemblers should report PSGs rather than sets of full-length isoforms, particularly when the data do not provide sufficient information to determine dependencies between processing events. The PSG framework offers an alternative parsimony objective in this context: that of minimizing the number of processing event parameters, rather than the number of isoforms, needed to explain the data. The fact that many transcriptome assemblers already build splice graphs as intermediate data structures makes PSGs a natural fit for these assemblers. To facilitate integration of our method with One of the first descriptions of PSGs appears in Non-probabilistic splice graphs have been used in the context of RNA-Seq as well. For example, For the task of gene annotation, The model of RNA-Seq read generation that we use is similar to the models assumed by many other methods for transcript quantification  that happens not to model some true dependencies, the individual edge weights of the PSG may be accurate, but the full-length isoform frequencies implied by the PSG will likely be off.In this article, we assume that we are provided with a fixed PSG structure for a gene and focus on inference tasks given RNA-Seq data. PSG structures may be constructed using known gene annotations or inferred from RNA-Seq data by other methods. In this section, we describe methods for two tasks: (i) estimating the parameters of a PSG, and (ii) determining whether a gene or splice junction is differentially processed between two samples.This section is structured as follows. We first formally define a PSG and a number of useful quantities computed from a PSG. We then describe a model of RNA-Seq data given a PSG. The identifiability of this model will then be addressed, followed by a description of how the EM algorithm is used to determine MAP parameters. We then provide simple likelihood ratio tests for detecting genes or splice junctions that are differentially processed between two samples. Our model, its associated inference methods and arguments about its identifiability are all independent of the specific form of PSG used (e.g. a line or unfactorized graph).v0, and an end vertex, Mv, where v0 and the only vertex with Mv. Each vertex, iv, of a PSG is associated with a sequence, which we denote by w(s), of a subpath, s, is the product of the weights of its edges:A PSG is a directed acyclic graph (DAG), t, with t is defined as the weight of its path, w(t).An isoform is represented by a path jv is included in a transcript given that iv is in the transcript. We denote this quantity by i, i.e. s is simply the sum of the lengths of the sequences associated with its vertices: iv to be the expected length of the subpath beginning at v0 and ending at iv; analogously, the expected suffix length iv is the expected length of the subpath beginning at iv and ending at Mv. These quantities can be calculated via the recurrences:v0 or the expected prefix length of Mv, Several useful quantities can be computed from a PSG that have not been described previously. These quantities will be important for specifying the RNA-Seq model and efficiently estimating parameters using the EM algorithm. Because a PSG is a DAG, each of these quantities can be described by a recurrence and computed efficiently using dynamic programming. First, we can compute the conditional probability that vertex G, that describes the relative abundances of isoforms of a gene. This model will allow us to estimate the parameters of G given RNA-Seq data. Our model is a PSG generalization of the models that require lists of full-length isoforms  tail at the end of a typical eukaryotic protein-coding transcript. We will often use the notation r is derived from s if there exists some b such that We assume that an RNA-Seq dataset represents nb, at which a read begins, is uniformly distributed across the length of the transcript from which it is derived. Thus,We assume that the position nt, is proportional to the product of the relative frequency of the transcript, i, we place a Dirichlet prior with parameters We assume that the probability of generating a read from a specific transcript, Supplementary Material, we state some general conditions under which a PSG is guaranteed to be identifiable. These conditions are generalizations of those for the identifiability of full-length isoform models  in the splice graph and to compute MAP estimates of the edge parameters using the number of reads that map to each edge. We will refer to this more simplistic strategy as the JR approach. Although this method is simplistic and ignores much of the data, it is statistically consistent under our RNA-Seq model, with JR estimates converging to true parameter values as the sample size becomes large. In addition, it is relatively robust to violations of the assumptions made in our RNA-Seq model, such as the assumption that the probability of a read being generated by a transcript is proportional to that transcript’s length. And while it is sensitive to non-uniformities around individual junctions, it is not affected by biases in the read distributions elsewhere in the transcripts and the estimates at one junction do not affect those at other more distant junctions.An article published during the revisions of this manuscript demonstrated that a JR approach can provide more accurate estimates than more sophisticated methods that use reads mapping to exon bodies . The advantage of EM over JR is most pronounced for read set sizes s-cup), as EM gains information from reads that map to the body of that exon. Simulations with paired-end reads gave similar results, with both methods benefitting from paired-end data .For our simulations, we used protein-coding Drosophila RNA-Seq data to assess the impact of the model assumptions used by EM. Here we assumed that the more robust JR estimates are accurate when a large number of reads are available. We used RNA-Seq data from the Drosophila cell line CME-W1-Cl.8+ (SRA accession SRS002587) . We also examined how often the estimates at each vertex agreed in terms of which AP event following that vertex was most likely. EM and JR agreed with respect to this measure on 84 and 81% of the vertices for single and paired-end estimates, respectively. The single-end and paired-end estimates agreed with each other on 95 and 93% of vertices for EM and JR, respectively. These results indicate that the EM estimates are highly accurate on average, assuming that the JR estimates are close to the truth. This suggests that the model assumptions used by EM are reasonable, at least on this dataset. The differences observed between the estimates of the same method on single and paired-end data show that many of the discrepancies between the methods arise simply because they draw information from different subsets of the data. The remaining discrepancies may be the result of highly biased read distributions or incorrectly annotated gene structures.The line PSG parameters for each of these genes were estimated using JR and EM. For each vertex with outdegree Supplementary Figure S5, the estimates from the line, first-order exon and unfactorized PSGs are all similar in terms of their distances from the reference JR estimates. Thus, the independence assumptions made by the line PSGs do not appear to affect the accuracy of the AP frequency estimates for this dataset. This experiment also shows that AP frequency estimates from factorized PSGs are comparable with those estimated from full-length isoform models, as the unfactorized PSGs are equivalent to such models.We further explored the validity of the independence assumptions made by the line PSGs in these experiments by additionally computing estimates of the AP event frequencies with first-order exon and unfactorized PSGs for these genes. As is shown in We additionally assessed the convergence rates of JR and EM through bootstrapping experiments on these data. For each of the selected 88 genes, we generated a series of bootstrapped read samples from its full read set. We generated read samples of sizes ranging from 10 to 5000, with 100 samples per read set size. On each read sample we estimated the line PSG parameters with both JR and EM and calculated the distance between each method’s estimates and its estimates on the full read set. Mean distances were computed for each read set size and a further mean was taken over all vertices . As expeDrosophila samples .Supplementary Table S1), which are much smaller than those for the untrimmed set. We believe this occurs because the trimming of reads reduces read coverage levels, which are used by FDM.The FDM method performed similarly to our PSG DP method, generally reporting fewer DP genes between replicates than between non-replicates. However, for all sample sets in Cuffdiff, on the other hand, had odd behavior in these experiments, as it called large numbers of genes as DP between replicates and a small number as DP between non-replicates. After discussing these results with the authors of Cuffdiff, we believe that they are explained by the combination of Cuffdiff accurately estimating low global processing variability between the replicate samples but then having highly noisy estimates of isoform frequencies for a subset of genes, particularly those that have multiple highly similar isoforms. When Cuffdiff is given a pair of non-replicate samples, it estimates higher levels of global variability and thus the effect of noisy isoform frequency estimates is diminished. This explanation fits with the general trend observed in the results in which the lower the variability between the samples, the larger the number of genes predicted as DP by Cuffdiff. Although Cuffdiff was originally used in an experiment without replicates (Supplementary Table S2). In this dataset, four samples were simulated with two biological replicates for each of two conditions (A and B). Each sample was simulated with separate parameter settings estimated from one of the four real human ENCODE samples (e.g. the simulation parameters for A Rep 1 were estimated from HUVEC Rep 1). Within replicates of the same condition, the relative isoform frequencies were set to be identical. We note that the gene-level abundances were different across all samples, even if the relative isoform frequencies were the same. Between conditions, 10% of multi-isoform genes expressed in both conditions were set to be DP, with isoform frequencies randomly shuffled between the two conditions for these genes.The performance of the three methods on the simulated dataset was similar to that on the real datasets .Because we knew the true set of DP genes between the two conditions, we were able to measure the recall and precision of the methods on pairs of samples from different conditions, in addition to the number of FPs measured on pairs of samples from the same condition. The PSG DP method exhibited good FDR control with a precision of 0.93–0.95 and had the highest recalls (0.54–0.60) of the three methods. Although Cuffdiff displayed the same high FP behavior between pairs of replicate samples, it had reasonable precision (0.88–0.98) between non-replicate samples but with lower recall (0.11–0.13). FDM had moderate recall (0.24–0.39) but poor precision (0.40–0.51). Precision-recall curves constructed by varying the Supplementary Table S3). FDM, on the other hand, did not predict any DP genes with this test, which was the result of the highly conservative permutation test it uses when multiple replicates are provided. However, in terms of its precision-recall curve performance, FDM improved slightly at the high precision end of the curve, as compared with its performance with no replication . Despite only taking into account pairs of samples, the average accuracy of the PSG method between non-replicate pairs was superior to the accuracy of both Cuffdiff and FDM when these methods were provided with multiple biological replicates .Because Cuffdiff and FDM are able to take into account multiple biological replicates per condition, we additionally performed an A versus B DP test in which all replicates were provided to the methods at the same time. Cuffdiff’s recall improved noticeably with this test, although with a slight decrease in precision . Even with more comparable alignment inputs, the PSG method outperformed FDM and Cuffdiff on these data.One implementation detail that sets our PSG methods apart from Cuffdiff and FDM is that our PSG methods align RNA-Seq reads to transcript subsequences, whereas Cuffdiff and FDM accept genomic alignments. Although we used standard alignment options for our genomic alignments using TopHat  of isoforms, a compact PSG model of the gene can enable polynomial-time inference of the model parameters. Although factorized PSGs only indirectly model the abundances of individual isoforms, they enable an efficient alternative for quantification.Drosophila gene Dscam, which is capable of producing tens of thousands of isoforms  of Dscam for use with the quantification-only mode of Cufflinks  identifying the exonic sequences, splice junctions, transcription start sites and polyadenylation sites of a gene, and (ii) determining the statistical dependencies between AP events. The first task is currently an active area of RNA-Seq research involving splice junction detection methods [e.g. In addition to the learning of PSG structures, our future work includes several extensions to our current model. First, we plan to modify the model to include sequencing biases, which have been demonstrated to significantly impact quantification from RNA-Seq data (


Genome-wide expression signatures are emerging as potential marker for overall survival and disease recurrence risk as evidenced by recent commercialization of gene expression based biomarkers in breast cancer. Similar predictions have recently been carried out using genome-wide copy number alterations and microRNAs. Existing software packages for microarray data analysis provide functions to define expression-based survival gene signatures. However, there is no software that can perform survival analysis using SNP array data or draw survival curves interactively for expression-based sample clusters.We have developed the survival analysis module in the dChip software that performs survival analysis across the genome for gene expression and copy number microarray data. Built on the current dChip software's microarray analysis functions such as chromosome display and clustering, the new survival functions include interactive exploring of Kaplan-Meier (K-M) plots using expression or copy number data, computing survival p-values from the log-rank test and Cox models, and using permutation to identify significant chromosome regions associated with survival.http://dchip-surv.chenglilab.org.The dChip survival module provides user-friendly way to perform survival analysis and visualize the results in the context of genes and cytobands. It requires no coding expertise and only minimal learning curve for thousands of existing dChip users. The implementation in Visual C++ also enables fast computation. The software and demonstration data are freely available at  In cancer clinical practice, predicting patient survival based on traditional tumor staging systems using clinical, histopathological and molecular markers remains an integral component in the treatment decision for patients. For example, patients with advanced disease and poor survival prognosis are subjected to more aggressive treatments. However, this conventional approach is non-specific and has limited success in the cancer treatment. Many patients have recurrence despite having aggressive therapy based on survival risk score ,2.With high-throughput cancer genomics data, we and others have reported using genome-wide expression signatures to predict survival risk, and these signatures are now increasingly being used in treatment decision for several cancer types -6. Survihttp://www.dchip.org) [However, there is no specific utility that can perform survival analysis using SNP array data or draw survival curves interactively for expression-based sample clusters. We have developed the widely-used dChip software that can efficiently process and derive gene expression and copy number data from microarray datasets (hip.org) ,15, and hip.org) . Thus, tHere, we describe the survival analysis module in the dChip software that performs survival analysis across the genome for gene expression and copy number microarray data. The new survival functions include interactive exploring of Kaplan-Meier (K-M) plots using both expression and copy number data, computing survival p-values from the log-rank test and Cox models, and using permutation to assess the survival significance of copy numbers genome-wide. Researchers can also compare survival curves between sample clustering groups derived from expression data. The dChip survival module enables user-friendly, interactive survival analysis and visualization of microarray data in the context of genes and cytobands. It requires no need for coding and minimal learning curve for existing dChip users. The implementation in Visual C++ also enables fast computation for processing large data sets from studies such as the Cancer Genome Atlas (TCGA).The survival analysis functions are implemented in dChip using Visual C++ and optimized for fast computation. The computed log-rank test and Cox model statistics and p-values are confirmed using R code. Figure Here we will use two example data sets to demonstrate the functions: 1) performing survival analysis using SNP data, and 2) drawing K-M plots using expression-based sample clustering groups. For the first dataset , we willWe will use a 192-sample microarray dataset processed on Affymetrix SNP 500 K microarray platform  to illusAnalysis > Open group", we use the "Tools > Array List File" menu function to create an array list to order samples by the values of a particular sample variable. Array list files are also useful when doing survival analysis on a subset of samples, leaving other samples out of the downstream analysis.We use the dChip software to normalize all the arrays of 192 myeloma samples and additional 10 normal blood samples to compute model-based signal values. Normalization and model-based signals are calculated for each of the two sub-arrays and combined. The median genotype call rates are 96.77% and 97.35% respectively for the 250 K Nsp and Sty sub-arrays. A tab-delimited sample information file is prepared with columns specifying survival outcome including survival time and event indicator . Numerical columns are marked in the column header such as "Survival(numeric)", and will be standardized and displayed above the samples in the clustering or chromosome data views. After loading the normalized data into dChip using the menu function "Analysis > Chromosome" to specify analysis parameters for copy number and LOH analysis  or a SNP from the regions with significant survival scores. Then, we use the "View > K-M plot" function to draw a K-M plot using the copy number data of the SNP to form Deletion, Gain, and "No change" groups as above  for an individual SNP, we first click on either a SNP nearby a gene of interest  blocks, and randomly switch the order of these blocks while preserving the order of SNPs within each block. In this way, the SNPs and their copy numbers in a sample are randomly relocated in blocks to new positions in the genome, while only minimally perturbing the dependence of the copy number data of neighboring SNPs. The same permutation applies to all samples using a different random partition for each sample. The survival score at each SNP locus can then be computed for the permuted data set, and the l scores . The max blocks, Unsupervised hierarchical clustering is frequently used to discover novel sample sub-groups from microarray data or inspect the expression-based samples clusters in relation to clinical variables. In particular, it is interesting to know whether sample clusters correspond to differential survival outcomes.Analysis > ANOVA & Correlation" menu to specify the survival variable and event indicator, similar to Figure Analysis > Clustering & Enrichment" menu to cluster samples using a variation-filtered gene list. Interested in whether sample clusters correspond to different survival outcomes, we can click to select a main sample cluster branch (in blue color) and use Control-click to select and color additional sample clusters  tool by Corradi et al.  a usefulIn summary, the dChip survival module addresses the frequent need of many researchers to integrate survival data analysis under a single microarray analysis package with minimal learning curve, fast computing, and no requirement of programming skills. We will add more survival functions as well as extend them to microRNA and RNA-seq data in future dChip versions.Project name: dChip survival analysis modulehttp://dchip-surv.chenglilab.org/Project home page: Operating system(s): Windows 2000 or afterProgramming language: Visual C++ 2005Source code: Available on request.License: The software is freely available.CL designed and implemented the software module, S Amin and CL wrote the manuscript, S Amin, PKS and AY contributed to the data analysis and manuscript preparation, S Adamia, SM, HA and NCM designed and performed the microarray experiments.All authors read and approved the final draft.Figure S1Click here for fileFigure S2Click here for file


Current prognostic gene signatures for breast cancer mainly reflect proliferation status and have limited value in triple-negative (TNBC) cancers. The identification of prognostic signatures from TNBC cohorts was limited in the past due to small sample sizes.We assembled all currently publically available TNBC gene expression datasets generated on Affymetrix gene chips. Inter-laboratory variation was minimized by filtering methods for both samples and genes. Supervised analysis was performed to identify prognostic signatures from 394 cases which were subsequently tested on an independent validation cohort (n = 261 cases).Using two distinct false discovery rate thresholds, 25% and <3.5%, a larger (n = 264 probesets) and a smaller (n = 26 probesets) prognostic gene sets were identified and used as prognostic predictors. Most of these genes were positively associated with poor prognosis and correlated to metagenes for inflammation and angiogenesis. No correlation to other previously published prognostic signatures  was observed. In multivariate analyses in the validation cohort the two signatures showed hazard ratios of 4.03  and 4.08 , respectively. The 10-year event-free survival was 70% for the good risk and 20% for the high risk group. The 26-gene signatures had modest predictive value (AUC = 0.588) to predict response to neoadjuvant chemotherapy, however, the combination of a B-cell metagene with the prognostic signatures increased its response predictive value. We identified a 264-gene prognostic signature for TNBC which is unrelated to previously known prognostic signatures. Breast cancer represents a heterogeneous disease and the currently most relevant clinical classification is based on the expression of the estrogen receptor (ER), progesteron receptor (PgR), as well as the human epidermal growth factor receptor 2 (HER2) In previous studies we demonstrated that analysis of a cohort of only TNBC allows the identification of different molecular phenotypes within this subtype of breast cancer http://www.r-project.org/) and SPSS version 17.0  were used for all analyses. Chi square test was applied to assess associations between categorical parameters. All reported P values are two sided and P≤0.05 was considered significant. An R script of the analyses is available as The REMARK recommendations for tumor marker studies affy package 2-transformed, median-centered, and the expression values of all the probesets from the U133A array were multiplied by a scale factor S so that the magnitude  equals one. Within this large breast cancer dataset, 579 triple negative breast cancers (TNBC) were identified based on the expression of ER, PgR, and HER2 from microarray C for each of the 579 arrays to identify the most comparable samples. This metric C is derived from the sum of the squared differences of the mean (μ) within a specific dataset and among all datasets, respectively, normalized by the standard deviation (σ) calculated for all genes (g) on the array:To generate a homogeneous dataset for the identification of genes with prognostic power among TNBC we used (i) only one array platform (Affymetrix U133 gene chips) and (ii) included only samples defined as triple negative based on the mRNA expression levels of ER, PgR, and HER2 as previously described C≤0.03), corresponding to 394 samples in total, were used as the discovery cohort  samr was applied to the finding cohort of 297 TNBC samples with known follow up to train the predictor. Delta values of 0.3 and 0.5 with median false discovery rates of 25% and <3.5%, respectively, were used to select prognostic probesets and a compound covariate predictor was developed that used the SAM-Score as a weight for each corresponding probeset. For Kaplan-Meier analysis we split the cases into quantiles of prediction scores and plotted survival curves by quartiles and also for the highest quartile versus all the rest.We applied a supervised classification method using all 22,283 probesets on the Affymetrix microarrays to identify a prognostic gene expression signature. The Informative probesets obtained by SAM analysis were checked for dataset bias . To assess dataset bias, we used Kruskal Wallis statistic comparing the expression of each probeset with the primary dataset vector across the 394 TNBC. Each probeset was then tagged with that Kruskal Wallis value throughout all analyses . Cutoffsunclassified”.To determine if the genes (i.e probesets) from the prognostic signature correspond to or serve as surrogates for previously described molecular subtypes within the TNBC group, we calculated the correlation between each of the genes from the prognostic gene lists and 16 previously established metagenes that represent different cell populations and different molecular variants of TNBC. These metagenes included the intrinsic genes of the basal-molecular class genefu R-package The correlation of the newly identified prognostic signatures with seven previously published prognostic signatures was analyzed by calculating the Pearson correlation coefficient between signature scores in the finding cohort of TNBC. The following prognostic signatures were included in this analysis: Recurrence score Follow-up data was available for 297 of the 394 TNBC samples from the finding cohort, and for 105 of the 261 samples from the validation cohort . All surA cohort of TNBC treated with neoadjuvant chemotherapy was assembled for which gene expression data from pre-treatment biopsies were available. Samples from biopsies which were microdissected were excluded. For 191 samples from seven datasets information on pathological complete remission (pCR) was available see . ReceiveCox score option of Significance Analysis of Microarrays (SAM) samr was applied to the finding cohort (n = 297 samples with follow up). A delta value of 0.3 resulted in 264 prognostic probesets (235 probesets associated with poor prognosis and 29 probesets associated with good prognosis). The median false discovery rate (FDR) when using this delta value was 25%. A more stringent delta of 0.5 resulted in 26 probesets associated with poor prognosis with a median false discovery rate <3.5% . In order to examine if our TNBC-derived prognostic signatures correspond to previously described metagenes that were used to subdivide TNBC, we calculated the correlation between the our signature scores and 16 different previously published TNBC-related metagenes g cohort  panel A.is group  panel B.is group . A similunclassified” , Histone-related , Molecular-Apocrine–related , Proliferation–related , and IL-8/inflammation–related    panel A.VEGF-, IL-8-, Molecular apocrine-, Claudin-CD24-, and Histone-metagenes in a separate cluster  using a supervised discovery method. The smaller signature based on probe sets with the lowest false discovery rate represent a subset of the larger signature. We validated the independent prognostic value of both signatures in a separate validation cohort both using the signatures as continuous scores  which were inversely associated with a poor prognosis and therefore we refer to it as good prognosis genes. Twenty one of these (72.4%) were correlated with immune cell metagenes which is consistent with several other publications which have shown that lymphocyte infiltration of TNBC is associated with an improved prognosis Our study has several limitations. The definition of TNBC was based on gene expression data which is not the standard definition used in the clinic. This definition holds the promise that samples erroneously characterized as receptor-negative by immunohistochemistry do not introduce noise into our analysis but discrepancies to cohorts defined by immunohistochemistry can occur. We found agreement of ER status between immunohistochemistry and gene expression data for 444 (84.4%) of 526 samples . For PgR status we found agreement for 407 (87.5%) of 465 samples , and for HER2 agreement for 347 (94.3%) of 368 samples . Agreement for the status of all three receptors was found for 276 (76.2%) of 362 samples . Regarding histological grading the proportion of grade 3 tumors is 73.5% and 74.1% in finding and validation cohort, respectively . These nOur analysis involved pooling of several datasets to increase sample size and power for discovery and validation. This strategy bears the risk of a confounding effect through systematic technical differences that exist between individual datasets During the generation of this report Lehmann et al. In our previous study In summary, in this paper we demonstrated that the use of a homogenous TNBC dataset allowed us to identify prognostic gene signatures that are unrelated to previously published general breast cancer prognostic signatures. The composition of the signature suggests that IL-8 mediated inflammation and VEGF related signaling herald very poor prognosis in TNBC and immune infiltration predicts better outcome. These observations could also suggest potential novel therapeutic strategies for these patients as e.g. inhibiting IL-8 signalling Figure S1Selection of the TNBC finding cohort from multiple datasets based on dataset comparibility. Triple negative breast cancers  from 28 datasets were sorted by dataset according to a dataset comparability metric . Shown are the full array data of normalized Affymetrix U133A microarrays. The 15 most comparable datasets encompassing n = 394 TNBC samples were subsequently used as a finding cohort and the remaining 13 datasets (n = 185 TNBC samples) withhold as validation cohort.(PDF)Click here for additional data file.Figure S2Analysis of a potential dataset bias among probesets of the prognostic signatures from the supervised analysis. A) The standard Kruskal-Wallis rank test was used to analyze the dependence of each individual probesets' expression on the vector of the 15 different datasets in the finding cohort of n = 394 samples. The distribution of the rank sum statistics for all 22,283 probesets from the U133A array is shown. Two dotted vertical lines mark the used cutoff values of 75 (yellow) and 150 (red). B) Distribution of the Kruskal-Wallis rank sum statistics for the 235 probesets identified by SAM as associated with poor prognosis. Used cutoffs are represented by dotted vertical lines as in (A). C) Distribution of the Kruskal-Wallis rank sum statistics for th 29 probesets identified by SAM as associated with good prognosis. Used cutoffs are represented by dotted vertical lines as in (A).(PDF)Click here for additional data file.Figure S3Kaplan Meier analysis of quartiles according to the prognostic signature scores in the finding and validation cohorts. A) The 394 TNBC samples from the finding cohort were stratified according to quartiles of expression of the 264-probeset signature score. Kaplan Meier analysis of event free survival of 297 samples with follow up information is shown. B) The 261 TNBC samples from the validation cohort were stratified according to quartiles of expression of the 264-probeset signature score. Kaplan Meier analysis of event free survival of 105 samples with follow up information is shown. C) The same analysis as in (A) was performed using the 26-probeset signature. D) The same analysis as in (B) was performed using the 26-probeset signature.(PDF)Click here for additional data file.Figure S4Correlation of the prognostic signatures with metagenes for molecular phenotypes in triple negative breast cancer. A) The continous score of the 264-probeset signature was correlated with the expression of 16 metagenes for molecular phenotypes in the 394 TNBC samples from the finding cohort. Shown is the result from hierarchical average linkage clustering based on absolute Pearson correlation. The signature score clustered together with VEGF, Histone, and IL-8 metagenes. B) The same analysis as in (A) was performed in the validation cohort of 261 independent TNBC samples. In this analysis the signature score clustered together with Stroma, Hemoglobin, VEGF, and IL-8 metagenes. Of note, however, Stroma and Hemoglobin metagenes are associated with a high dataset bias (see Supplementary (PDF)Click here for additional data file.Figure S5Analysis of dataset bias of metagenes and the prognostic signatures. A) The dependence of earch probeset from the U133A array on the dataset vector was analyzed using the standard Kruskal-Wallis rank test in the finding cohort of 394 samples (see Suppl. (PDF)Click here for additional data file.Figure S6Correlation of individual markers from the prognostic signatures with known metagenes in triple negative breast cancer. From the 264 Affymetrix probsets of the supervised prognostic signature, 235 probesets were associated with poor prognosis  and 29 with good prognosis . A) The 235 individual probesets associated with poor prognosis  were analyzed for their correlation with the expression of 16 metagenes  for molecular phenotypes in the 394 TNBC samples from the finding cohort. 116 probesets displaying a Pearson correlation above a cutoff 0.2 are sorted  on the left according to the assigned metagene while 60 probesets remained unclassified. B) The 29 individual probesets associated with good prognosis were analyzed as in (A) and 21 assigned to metagenes (cutoff 0.2) are sorted horizontically on the left while 8 remained unclassified. C) The same analysis as in (A) was performed using the more stringent cutoff 0.3 for assignment to a metagene resulting in 118 probesets correlated to metagenes from the list of 235 probesets associated with poor prognosis. D) The same analysis as in (B) was perfomed using the more stringent cutoff 0.3 resulting in 18 of the 29 good prognosis probesets assigned to metagenes. All individual correlation values are given in Supplementary (PDF)Click here for additional data file.Figure S7Relationship of the 26 probeset signature to the 16 metagenes and seven known prognostic signatures in TNBC. The 394 TNBC samples were analyzed for the expression of 16 metagenes and seven previously published prognostic signatures as described in (PDF)Click here for additional data file.Figure S8Stability analysis of the prognostic signatures from the supervised analysis. The 264 Affymetrix probsets of the supervised prognostic signature were filtered according to their dataset bias measured through Kruskal-Wallis statistic and different stringency from SAM analysis as given in the Table below the graphs. The resulting probeset lists of 252, 24, 181, and 16 probesets were used for prognostic signature generation as the original 264 probeset list. In upper panels A, B, C, and D the correlation of the four alternative signatures to the 264-probeset signature is shown by scatter plot analysis. The lower panels display the results from the Kaplan-Meier analyses of the validation cohort of 261 TNBC (105 samples with follow up information). In addition P-Values of multivariate Cox regression analysis of the validation cohort using continous signature scores are given in the table below.(PDF)Click here for additional data file.Figure S9Expression of the 264-probeset and 26-probeset signature scores in seven different TNBC subtypes according to Lehmann et al. A) Box plots comparing the expression of the 264-probeset signature in the seven different TNBC subtypes according to Lehmann et al.  separately for our finding and validation cohorts. No clear correlation of the expression of the signature with any of the subtypes was observed. The seven subtypes have been ordered according the expression of the signature in the finding cohort. Highest expression was observed in the “basal-like 2” (BL2) and “luminal androgen receptor” (LAR) subtypes of the finding cohort. However this effect was not reproduced in the validation cohort. B) The same analysis as in (A) was performed for the expression of the 26-probeset signature. The observed result was similar in that no reproducible correlation of the signature with any subtypes was detected.(PDF)Click here for additional data file.Table S1Summary of Affymetrix microarray datasets used in this study.(PDF)Click here for additional data file.Table S2List of 355 Affymetrix probesets used for metagene calculation.(PDF)Click here for additional data file.Table S3Details of probesets from the supervised signatures.(XLS)Click here for additional data file.Table S4A) Multivariate Cox regression of continous 264-probeset signature and standard parameters for event free survival in the finding cohort B) Multivariate Cox regression of continous 26-probeset signature and standard parameters for event free survival in the finding cohort.(PDF)Click here for additional data file.Table S5Pre-therapeutic samples from neoadjuvant treated TNBC.(PDF)Click here for additional data file.Table S6Clinical parameters of TNBC according to expression of the 264-probeset signature.(PDF)Click here for additional data file.Table S7Histological grade among samples in the finding and validation cohort.(PDF)Click here for additional data file.Table S8Distribution of intrinsic molecular subtypes according to expression of the 264-probeset signature in TNBC.(PDF)Click here for additional data file.Table S9264 probeset supervised prognostic signature for TNBC from SAM.(PDF)Click here for additional data file.Data S1R script of the analyses.(R)Click here for additional data file.Data S2R.Data file (contains 11 data objects used in the R script from (7z)Click here for additional data file.