The results demonstrate the usefulness of the multisensor system for analysis of wine.This work presents a multiparametric system capable of characterizing and classifying white wines according to the grape variety and geographical origin. Besides, it quantifies specific parameters of interest for quality control in wine. The system, known as a hybrid electronic tongue, consists of an array of electrochemical microsensors—six ISFET based sensors, a conductivity sensor, a redox potential sensor and two amperometric electrodes, a gold microelectrode and a microelectrode for sensing electrochemical oxygen demand—and a miniaturized optofluidic system. The test sample set comprised eighteen Catalan monovarietal white wines from four different grape varieties, two Croatian monovarietal white wines and seven bi- and trivarietal mixtures prepared from the Catalan varieties. Different chemometric tools were used to characterize ( However, it is in the food quality control and safety where the applicability of these biomimetic systems has been explored more . Wine, wRecently, a review has covered the research done in the field of electronic and bioelectronic tongues for the analysis of wine samples . HoweverIn our previous work , the app2.2.1.−2 and 10−4 M were prepared. In the case of those sensitive to cations , the corresponding chloride salts were considered. For the Cl− and NO3− ions, solutions of NaCl and KNO3, respectively, were prepared. A solution containing 0.1 M KNO3 was used to activate the amperometric gold (Au) sensor. In order to calibrate the conductivity sensor, two different standard solutions from Crison , with nominal values of 1,413 and 147 μS/cm, were utilized. Two standard redox solutions from Panreac , with values of 468 mV and 220 mV, were used to calibrate the oxido-reduction potential (ORP) sensor. Finally, for the electrochemical oxygen demand (EOD) electrode, a solution containing 0.1 M NaOH and a glucose stock solution of 25 g/L were used. For the ISFET measurements, a reference solution containing an average concentration of the main species present in wine was prepared. The composition of this solution has been reported elsewhere [All reagents used were of high purity, analytical grade or equivalent. All solutions were prepared with de-ionized water. For ISFET calibration, solutions with ionic salts with concentrations between 10lsewhere .2.2.The set of samples consisted of 18 monovarietal (100%) white wines based on the following grape varieties: Macabeu , Parellada , Chardonnay  and Xarel·lo , all provided by the Catalan Institute of Vineyard and Wine (INCAVI). The index “07” indicates that the samples correspond to the 2007 vintage. All the samples have the same geographical origin (Penedes), except the Macabeu samples 11/07 and 12/07, which came from the Garraf zone. The winemaking process for all these samples is the same. It begins with the reception of the grapes in the winery. Processes of crushing, pressing and the addition of sulfites to the grape juice are performed successively. Then, a first clarification in order to eliminate the suspended solids and the alcoholic fermentation that transform the grape juice into wine are done. Once the wine is obtained, a second stage of clarification followed by stabilization and filtration steps and a second addition of sulfites are performed, before bottling the final product.Samples were provided by INCAVI in 750 mL bottles. Aliquots of 25 mL were collected from each wine sample in bottles protected against light and put into the refrigerator at 4 °C for a maximum of one week. Two hours before the analysis, they were removed from the fridge to equilibrate at room temperature. During the analysis, the aliquots were successively opened and analyzed to prevent the possible changes of composition. Besides, two monovarietal white wines from Croatia based on Grasěvina and Zelenac grape varieties were included in the model and seven different wine mixtures were prepared from the Catalan samples, thus giving bivarietal and trivarietal wines, whose composition is shown in Certain chemical parameters in these wines were determined by the INCAVI using standard methods. These methods are dictated by the European Union  and are 2.3.+, K+, Ca2+, Cl− and NO3− ions. Polymeric membranes were based on photocurable polymers with commercial ionophores from Fluka . The ionophores used in each case were: 4-tert-butylcalix[+, valinomycin (Ionophore I) for K+, N,N,N′,N′-tetracyclohexyl-3-oxapentanediamide  for Ca2+, tridodecylmethylammonium chloride for Cl− and tetraoctylammonium nitrate for NO3−. All these ionophores are selective to the principal ion, but are not specific and they present a certain degree of cross-response to other ions in solution. Membrane composition, preparation and characterization has been presented elsewhere [3COOLi solution in its outer chamber was employed for all the potentiometric measurements. These were performed with the aid of a laboratory constructed data-acquisition system, which was connected to a computer through the RS-232 board. The readings were done employing specially designed software programmed with VisualBasic .A set of six ISFET sensors were fabricated using standard microelectronic technology . One ISFlsewhere ,23. An OA flow cell  for sensing electrochemical oxygen demand developed by our group . In bothOptical measurements were done using an MIR configuration fabricated in poly(dimethylsiloxane) (PDMS). The photonic lab on a chip comprises microlenses, self-alignment structures, microfluidic channels and air mirrors for highly sensitive absorbance measurements . The lig2.4.Sample analysis was carried out under batch conditions, except for the ISFET set, which was performed under flow conditions. A complete analysis of a wine sample took around 15 min. No replications of each sample were done following the INCAVI protocol and also to get a rapid analysis and prevent changes of the wine sample. All the sensors that form the array were characterized before the analysis. The response characteristics are reported in , includiFor the ISFET set, the output signals corresponded to the relative measurements of each sensor with respect to the reference solution, which was checked periodically. This is a common strategy to correct the possible drift of the sensors.2SO4 6.0 M and de-ionized water. Calibration of the conductivity sensor was carried out using two standard solutions . For ORP sensor evaluation, a test with standard redox solutions of 220 and 468 mV (at 25 °C) versus the Ag/AgCl reference electrode was performed. Once the good behaviour of the sensors was confirmed, they were immersed in the wine sample and the signals were recorded every 30 seconds during 3 minutes.The 4-electrode sensor was firstly chemically cleaned successively with ethanol 96%, H3 where the electrode was cycled from +0.8 to −2.2 V for at least 20 times. Then the electrode was immersed in the sample under studied and two cyclic voltammograms (CV) from +1.6 to −0.5 V at a scan rate of 0.1 V/s were run: the first one to stabilize the signal and the second one to obtain the information. Finally, the electrode was activated after measuring each wine samples by running five CV in 0.1 M KNO3.The Au microelectrode was firstly chemically cleaned as before, followed by an electrochemical activation carried out in 0.1 M KNORegarding the electrochemical oxygen demand sensor (EOD) based on a CPE, previous studies in our laboratories have demonstrated the response of this electrode towards glucose and some other sugars. For this reason, analyses were performed using glucose as internal standard. Using the chronoamperometric mode and setting the potential at +600 mV, the CPE was immersed in NaOH 0.1 M. Then 50 μL of glucose 25 g/L was added. When the response became stable, 100 μL of the studied wine was added. In this case, the output signals used to construct the model were the ratio of the wine signal to the glucose signal.In case of the photonic lab on a chip, measurements were carried out by filling the system with the sample to be analyzed. In order to obtain the spectra in absorbance units, de-ionized (DI) water was used as reference. Measurements were taken prior and after each wine to determine signal drifts due to non-specific adsorption at the walls. Throughout the experiments, the DI water reached the same value between the experimental errors, confirming the reversibility of the photonic lab on a chip. Finally, for statistical purposes, for each wine type, the average of 10 consecutive scans was considered.2.5.Obtained data were treated using different multivariate methods. Principal Component Analysis (PCA) was used as the first technique to evaluate the characterization power of the system for monovarietal and mixture samples. The Soft Independent Modeling Class Analogy (SIMCA) method was also utilized to achieve a good classification model for the wine mixtures. The Partial Least Squares (PLS) regression was employed to perform the quantification of different parameters of the samples . In thisFor all these methods, the original values were previously autoscaled—all the variables were centered and set to a standard deviation equal to 1—to avoid variables from having a different influence on the model. Besides, all the obtained models were centered. For the PCA and SIMCA analyses, the cross-validation technique was used in order to ensure the performance of the generated model. In typical cross-validation, the training and validation sets of samples cross-over in successive iterations such that each sample has a chance of being validated against. Contrary, the test-set validation technique was used for the PLS regressions. In this case, a fixed calibration and prediction sets of samples are chosen. To control all these parameters and to perform the analyses, the Unscrambler v.9.1 informatics package  was used.3.3.1.Once all the samples were passed through the sensors, a data matrix was constructed with the different variables to be used as the input of the chemometric tool. These variables are fixed for ISFET sensors, conductivity, ORP and EOD sensors but the value of current and the absorbance depends on the specific response of the sensor to the samples. For this reason the election of the variables is an important step in a multivariable analysis. In other words, the goodness of the models generated by the algorithms depends essentially on the success of this choice. In the present study, the input data was composed by 16 variables as shown in The input data of the six ISFETs correspond to the relative signal in mV of each ISFET with respect to the reference solution. It is also the case of the conductivity, ORP and EOD sensors, which give an absolute signal as a result. For the measurements with the Au microelectrode, cyclic voltammograms (CV) were carried out. In As an example of the values obtained by the different sensors, the response of three representative variables for the 20 monovarietal wines is represented in For the pH ISFET  similar 3.2.In order to prove the capability of the system to differentiate monovarietal samples from different geographical origin, a PCA was performed using the 20 monovarietal wines: the 18 Catalan samples and the two Croatian samples. These Croatian wines are based on two characteristic grape varieties from the Slavonia zone: Grasěvina White and Zelenac. In + and Ca2+ have also importance in the first PC, while the PC 2 is composed basically by two variables of the Au microelectrode (intensities at +1.31 and +0.65 V). However, the three optical variables have a discrete contribution both to the first component (between 0.21 and 0.27) and to the second component (between 0.27 and 0.30). In other words, the electrochemical variables have a higher weight for the monovarietal white wine classification than the optical ones, and could be removed of this first model.It is interesting to analyze which is the weight of each original variable in the constructed model. We can observe in The next study performed with this system was the characterization of the 18 Catalan wine samples. A new PCA was performed using the raw data obtained from the selected variables. In In order to compare the capability of our system to differentiate Catalan wine samples with the data from standard methods, a PCA using the chemical and optical data provided by the INCAVI was performed. The obtained scores plot along the first two components is shown in 3.3.Once the model using raw data from the sensor variables was constructed, the values obtained for the seven mixtures were interpolated in the PCA. The situation of these samples inside the model is shown in 3.4.Next, a PLS was realized in order to assess if the system was able to quantify some quality parameters of the samples already analyzed with standard methods. For the regression, the calibration set was formed by 14 samples: four Macabeu samples , four Parellada samples , three Chardonnay samples  and three Xarel·lo samples . The rest of the samples were used as the prediction data set. The relative standard errors of the calibration set (RSEC) and the prediction set (RSEP) were calculated in order to compare the precision of the different PLS regressions. The RSEC values were between 1.0 and 5.0% for all the determinations and the RSEP values were between 2.2 and 7.6%, without excluding any value.The data of optical and chemical parameters for the prediction set obtained with our system and with standard methods are shown in 4.This work demonstrates the feasibility of using the proposed hybrid electronic tongue to analyze monovarietal white wines and their bi- and trivarietal mixtures. The high complexity of the wine matrix makes particularly interesting the combination of electrochemical and optical variables in order to obtain the maximum sample information. Then, this obtained data is treated with powerful chemometric tool to extract useful chemical information.The qualitative results using PCA technique confirm that this system is capable of distinguishing the samples according to the grape origin and even to the geographical origin. Using the SIMCA classification technique, the ability of the system to differentiate between the monovarietal samples and their mixtures has been demonstrated with a probability of 75%. In addition, the application of the PLS technique to the collected data, permits one to quantify some chemical and optical parameters with relative errors smaller than 10% in general, obtaining especially good results for the prediction of VAD, pH, magnesium and tonality. Even though the electrochemical variables seem to be more significant for these analyses, the optical ones cannot be undervalued because they are essential for some applications, like the correct differentiation of the Catalan wine set and the quantification of optical parameters.In conclusion, good results are obtained both for discrimination and quantification methodologies that confirm the viability of the multisensor system. Ongoing experiments are focused on processing a higher number of samples, from different varieties, vintage years and origin to improve the reliability of the system. Moreover, an important effort will be performed to analyze qualitatively bi- and trivarietal samples and even to quantify the proportion of each grape variety presents in the wine mixture. Besides the integration of sensors in the same substrate in order to obtain a compact system that could be applied to flow measurements will permit more automated and feasible analysis.


Small sample sizes used in previous studies result in a lack of overlap between the reported gene signatures for prediction of chemotherapy response. Although morphologic features, especially tumor nuclear morphology, are important for cancer grading, little research has been reported on quantitatively correlating cellular morphology with chemotherapy response, especially in a large data set. In this study, we have used a large population of patients to identify molecular and morphologic signatures associated with chemotherapy response in serous ovarian carcinoma.P = 0.04) or an external validation set (P<0.0001). In parallel, we present the production of a tumor nuclear image profile generated from 253 sample slides by characterizing patients with nuclear features  in incremental bins, and we identify a morphologic signature that demonstrates a strong association with chemotherapy response in serous ovarian carcinoma.A gene expression model that predicts response to chemotherapy is developed and validated using a large-scale data set consisting of 493 samples from The Cancer Genome Atlas (TCGA) and 244 samples from an Australian report. An identified 227-gene signature achieves an overall predictive accuracy of greater than 85% with a sensitivity of approximately 95% and specificity of approximately 70%. The gene signature significantly distinguishes between patients with unfavorable versus favorable prognosis, when applied to either an independent data set (A gene signature discovered on a large data set provides robustness in accurately predicting chemotherapy response in serous ovarian carcinoma. The combination of the molecular and morphologic signatures yields a new understanding of potential mechanisms involved in drug resistance. Ovarian carcinoma (OvCa) remains a leading cause of mortality from gynecologic cancer, with approximately 21,880 new cases and 13,850 deaths estimated in the United States in 2010 Understanding the molecular mechanisms for chemoresistance has been the subject of intense research. Various genomic methodologies The Cancer Genome Atlas (TCGA), a project of the National Cancer Institute and the National Human Genome Research Institute, generates a comprehensive catalog of genomic abnormalities with large-scale data sets that include cancers with the highest mortality rates including serous OvCa. In addition, the TCGA effort has led to the accumulation of a large set of tumor images in the repository. It is recognized that cell morphologies are intimately linked to multiple cell functions, such as cell growth, apoptosis, differentiation, and migration In this study, we leverage the full scope of the TCGA database with a large population of patients, including gene expression and tumor images, to identify the molecular and morphologic signatures associated with chemotherapeutic response in OvCa. Integration of the genomic and morphologic dimensions of OvCa will yield potential insights into mechanism of drug resistance and facilitate identification of novel system-level events for alternate therapeutic interventions.To gain insight into the potential mechanisms underlying the differential response of OvCa to chemotherapy, we perform an integrated analysis of gene expression and tumor nuclear image profiles. The 232-sample set that has both gene expression and image data is used to identify a gene expression pattern that could predict clinical outcome. The 227 genes most weighted in achieving the prediction are identified, of which 154 (67.8%) were downregulated and 73 (32.2%) are upregulated in the chemoresistant group . The genP = 0.04, HR [95%CI] = 0.43 [0.19–0.97]). The clinicopathologic characteristics of patients in these two groups are summarized in A validation of the gene signature is performed on an independent set of 261 samples from TCGA. Based on the score cutoff from P<0.0001, HR [95%CI] = 0.36 [0.22–0.56]) where the median PFS of Group 2 (12.0 months) is almost 4 times shorter than that of Group 1 (50.0 months). The clinicopathologic characteristics of the patients in these two groups either with high scores or with low scores in both validation sets are detailed in Robustness and scalability of the gene signature are next evaluated by using the Australian data set which is based on a different microarray platform. We use it to validate whether the discovered genes are associated with patient outcome. Overlap analysis reveals 198 among the 227 genes in this data set . Using tThese results not only validate the predictive performance of the gene signature but also suggest its strong association with tumor prognosis, which is most likely contributable from chemotherapy response.FDR≤2%, P = 0.001, HR [95%CI] = 1.99 [1.32–3.01]) and poorer PFS . Cox proportional hazard analysis demonstrates that the two subgroups, split on the basis of Std_Ar_Bin2 feature values, have significantly different OS and PFS patterns, after controlling for age, stage, and grade  in the Low Std_Ar_Bin2 group (P<0.005) are depicted in Both genomic and morphologic features are associated with chemotherapy, suggesting that these two types of signatures are strongly associated with each other. With the patients split into the two groups based on the Std_Ar_Bin2 feature values, as described above, we carry out a supervised analysis of the gene expression data and find that five of the signature genes are significantly lower , but with a larger variability (Std_Ro_Bin8) and a larger aspect ratio (Mean_AR_Bin8). Such morphologic differences likely result from the active response of the cells to their environment and heightened cellular metabolism, that is contributable from different molecular regulations . This isIn conclusion, a gene signature discovered on a large data set provides robustness in accurately predicting chemotherapy response in serous OvCa. Meanwhile, we propose a novel approach for tumor nuclear image profile generation by characterizing patients with nuclear features  in incremental bins, and we demonstrate that the tumor nuclear image profile exhibits a strong association with chemotherapy response. This imaging approach is capable of accounting for cell heterogeneity and improving the discriminating power. The integrated approach herein, using gene expression profile that predicts chemotherapy response coupled with the morphologic features to stratify patients to the most appropriate treatment regimen, represents an important step toward the goal of personalized cancer treatment by identifying the area where novel drugs can be developed. Although our observations suggest that the tumor image profile is capable of defining prognosis and yielding mechanistic insights into the process of chemoresistance, one limitation of this study is the lack of validation of the image analysis due to unavailability of the independent image sets especially in a large population. This issue should be addressed in the future in order to determine the ultimate value of this technique in clinical practice. Besides, the resolution dependence of the morphologic features in separate bins has not been systematically investigated yet in this study and deserves attention in the follow-up studies. Future work also consists of inclusion of more possible morphologic features and verification of the gene-feature relation identified in this study.Two hundred fifty three OvCa patients in the TCGA database with explicit platinum status f is the weighting factor, xf is the expression value and μ represents the expression mean for each class. A sample with a score greater than a cutoff is assigned to the chemosensitive group, and a sample with a score less than or equal to the cutoff is assigned to the chemoresistant group. The predictive accuracy, based on a cutoff score determined by receiver operating characteristic (ROC) curve, is assessed. The gene signature is validated on an independent sample set from TCGA and an external data set Expression data are prescreened to remove genes with trivial variation across the samples and low median expression levels, resulting in 14,084 genes in the analysis. The gene signature identified through a supervised method 2 and a circularity of greater than 0.3 are selected for further analysis. The nucleus profile consists of a set of numbers that describe the nucleus’s characteristics, including size, location, and shapes that are automatically measured using the ImageJ Plugins, which is widely used in research 2 per sample are produced which is almost 80 times more than the amount used in other studies An average of 10 high-resolution tumor images  per sample at the different views of the tissue blocks are first selected by a pathologist from hematoxylin- and eosin-stained ScanScope virtual slides, to account for the spatial heterogeneity of the tumor tissues. Next, we automatically identify and measure the nuclei in each image by using a cell-image analysis software  To generate the tumor image profile, we first split the nuclei into 10 evenly spaced bins based on the nuclear size, and then calculate the average value and standard deviation (SD) of these parameters  for each nucleus in each bin as well as for all the nuclei in an image. In addition, the number of nuclei in each bin  and its percentage  are defined as image features and calculated accordingly. The compactness is defined as an image feature to qualify the spatial distribution of nuclei within the tumor tissue. As a result, the tumor image profile consists of 153 morphologic features including 66 means, 66 SDs, 10 percentages, 10 counts, and 1 compactness . DefinitFDR≤2% and then validated in the test set. Feature data exhibit a normal distribution after median centered and log transformed.The 253 TCGA samples with the calculated image profile (described above) are randomly divided into a 130-sample training set and a 123-sample validation set in an approximate 1∶1 ratio. The training set consists of 90 chemosensitive patients (69.2%) and 40 chemoresistant patients (30.8%), while the validation set contains 82 chemosensitive patients (66.7%) and 41 chemoresistant patients (33.3%). The distribution of chemosensitive and chemoresistant patients in both the training and validation sets is selected to reflect clinical chemosensitive rates of approximately 70% OS and PFS curves are generated by the Kaplan-Meier method, and the statistical significance of survival differences is determined with the log-rank test. Survival analysis is performed and an ROC curve is generated using SPSS  and GraphPad Prism . Normality of feature values is verified via a Jarque-Bera test Figure S1Overall predictive accuracy of patients in the training set as a function of the gene expression fold change cutoff. The arrow indicates the fold-change cutoff used in the study that gives rise to the highest predictive accuracy.(PDF)Click here for additional data file.Figure S2P = 1.5 E-04). (B) The same nuclear parameter in Bin 9 (Mean_Ro_Bin9) shows a significant decrease in the chemoresistant group (P = 0.0015). (C) The average roundness of the nucleus in an entire sample shows no significant difference between groups  (P = 0.56).(A) The average roundness of nuclei in Bin 8 (Mean_Ro_Bin8) is significantly higher in the chemoresistant group ((PDF)Click here for additional data file.Figure S3Overall survival (OS) and progression-free survival (PFS) curves of the 253 patients used for tissue nuclear image profile generation, among which 172 patients were sensitive to chemotherapy and 81 of them were chemoresistant.(PDF)Click here for additional data file.Figure S4Flow chart for nucleus parametric profile generation.(PDF)Click here for additional data file.Figure S5Detailed version of (PDF)Click here for additional data file.Table S1P<0.05, as identified by parametric t-test).The 227 genes are differentially expressed between chemoresistant and chemosensitive patients ((XLS)Click here for additional data file.Table S2Clinicopathologic characteristics of patients in the TCGA and Australia Validation data sets.(PDF)Click here for additional data file.Table S3Cox proportional hazard analysis of progression-free survival of OvCa patients in the TCGA and Australian validation data sets in relation to predictive sub-groups (the group with high predictive scores versus the group with low predictive scores).(PDF)Click here for additional data file.Table S4FDR≤2%).The 15 morphologic features are differentially varied between chemoresistant and chemosensitive patients with serous OvCa ((PDF)Click here for additional data file.Table S5Image features defined for tumor nuclear image profile generation.(PDF)Click here for additional data file.Table S6Cox proportional hazard analysis of overall and progression-free survival of OvCa patients in relation to the morphological feature value (Std_Ar_Bin2).(PDF)Click here for additional data file.Table S7Significantly expressed gene signatures associated with each morphologic feature.(PDF)Click here for additional data file.


The maturing of gene expression microarray technology and interest in the use of microarray-based applications for clinical and diagnostic applications calls for quantitative measures of quality. This manuscript presents a retrospective study characterizing several approaches to assess technical performance of microarray data measured on the Affymetrix GeneChip platform, including whole-array metrics and information from a standard mixture of external spike-in and endogenous internal controls. Spike-in controls were found to carry the same information about technical performance as whole-array metrics and endogenous "housekeeping" genes. These results support the use of spike-in controls as general tools for performance assessment across time, experimenters and array batches, suggesting that they have potential for comparison of microarray data generated across species using different technologies.A layered PCA modeling methodology that uses data from a number of classes of controls  was used for the assessment of microarray data quality. The controls provide information on multiple stages of the experimental protocol . External spike-in, hybridization and RNA labeling controls provide information related to both assay and hybridization performance whereas internal endogenous controls provide quality information on the biological sample. We find that the variance of the data generated from the external and internal controls carries critical information about technical performance; the PCA dissection of this variance is consistent with whole-array quality assessment based on a number of quality assurance/quality control (QA/QC) metrics.These results provide support for the use of both external and internal RNA control data to assess the technical quality of microarray experiments. The observed consistency amongst the information carried by internal and external controls and whole-array quality measures offers promise for rationally-designed control standards for routine performance monitoring of multiplexed measurement platforms. Expression profiling using DNA microarrays is increasingly being used for clinical and diagnostic applications and in support of regulatory decision-making. These applications require the technology to be robust and reliable and that the data be well characterized . The quaData quality can be assessed at a number of stages within the microarray experiment  . Few unihttp://www.fda.gov/nctr/science/centers/toxicoinformatics/maqc/ and the ERCC are working to develop reference data sets, reference RNAs, and standard external controls intended for use in the evaluation of microarray performance [A number of initiatives including the Microarray Quality Control (MAQC) project of the FDA formance -9. The EThe MAQC project has examined the use of a diverse set of external controls for a number of platforms , noted tin vitro transcription that are added at a particular stage in the generation of the labeled sample transcriptome extract, at a known concentration  Microarray Centre. This data is stored in, and was accessed, via the Centre's Microarray data Mining Resource (MiMiR) database ,26. ThesBacillus subtilis transcripts   were spiked into the hybridization cocktail according to the manufacturer's instructions. They are used to align the grid and assess the efficiency of hybridization, washing and staining.The polyA+ controls are a cocktail of 5 polyA-tailed ) Figure . These cExtensive whole-array quality assurance metrics and BioConductor-based summary statistics -30 relat2 probe RMA intensities before and after normalization as well as the RMA model fit residuals, relative log2 expression (RLE) and normalized unscaled standard error (NUSE) plots for the identification of outlier arrays within an experiment dataset. In addition, RNA degradation plots show the log2 mean intensity by probe pair position (5' end to 3' end) for each array and are used to identify samples that may have been subject to degradation. The third section, Probe Metrics, are obtained from BioConductor MAS 5.0-based statistical algorithms and are used to assess both RNA assay and hybridization performance. These include measures of scanner variability , summarized exogenous control intensities with respect to their spike-in concentration levels, correlation measures between exogenous polyA+ controls and raw signal values, and 3'/5' ratio measures for both exogenous and endogenous controls to assess the efficiency of labeling and/or sample RNA integrity. The fourth and last section provides a simplified PCA scores plot generated from the complete set of probes  to identify gross outliers within the experimental dataset as a whole. A recent review of these metrics as they relate to the quality assessment of microarray data after statistical processing is provided by Brettschneider et al. [The Microarray Centre QA report metrics are based on .CEL file signal intensity data from GeneChip arrays and include summary statistics of all the hybridizations within a particular experiment generated using the BioConductor (BioC Release 1.9) open source software. This report provides quality assessment metrics based on: 1) Diagnostic Plots, 2) Probe-level Robust Multichip Average (RMA) Model Estimates, 3) Probe Metrics and 4) Principal Component Analysis. The first two sections include summaries of logr et al. Data from 525 hybridizations representing 22 publicly-available experiments generated over a five-year period at the CSC/IC Microarray Centre on multiple types of GeneChips were analyzed as part of this study and included human , rat  and mouse  microarrays. A single, exemplary experiment containing data from 137 Rat Genome RAE230A arrays is highlighted for this manuscript. This included data generated on different days over a 10-month period, with different experimenters, array batches, and QC measures from the whole-array QC report. This example was analyzed using PCA and the results compared to the QC and factor information available within the MiMiR database.PCA was conducted using only data from the control-based probesets  probeset signals). There are four groups, or classes, of controls, external and internal to the biological sample (exogenous and endogenous). The external controls were either polyA+ RNAs spiked into the sample before amplification and labeling or prelabeled hybridization controls spiked into the sample prior to hybridization. The internal controls are those suggested by Affymetrix as a measure of RNA degradation, and report on relatively invariant 'housekeeping' genes. Microarray probesets for the same external controls are present on all Affymetrix GeneChip arrays; probesets for the endogenous controls are organism-specific and are common to all arrays of such type .Probeset data from the individual hybridizations on RAE230A arrays  are described in this manuscript. In brief, this experiment is a comparison of gene expression profiles of peritoneal fat of 6-week rats from 30 recombinant inbred (RI) strains derived from the spontaneously hypertensive rat (SHR/Ola) and Brown Norway congenic carrying polydactylyl-luxate syndrome (BN-Lx) strains. A single hybridization (HFB2003080611Aaa) was missing annotation for experimental QC and was thus omitted from the data analysis. A summarized version of the annotation QC information pertaining to the individual hybridizations used in this experimental dataset is provided in Additional File 2 intensities for the background probesets associated with the individual hybridization was determined and then subtracted from the probesets of interest . This 'brightness-scaled" normalization approach was employed to support control data aggregation across multiple array types can be generated on a similar scale can thus directly compared and permits the identification of sample-associated variability. This 75th percentile normalization was carried out for several datasets that were generated across multiple array types (data not shown) when aliquots of the same samples were hybridized to arrays of the same or different type (e.g. RAE230A and RAE230B). The 75th percentile normalization was the default data analysis method for our investigations.Measures representing expression were generated from the raw data using the RMA "Affy" package (Bioconductor 1.8 release) within the R environment (v 2.6.0). The data was preprocessed using background correction and quantile normalization to the global median . A hybriThe mean and standard deviation (SD) of the RMA values were calculated for all probesets within an experiment conducted on a single array type, comparable to other informatic methods for generating probeset-level precision metrics ,31-33. Ahttp://www.affymetrix.com/support/technical/mask_files.affx). Each PCA data subset was organized into a single data block structure with dimensions of Nrows × Kcolumns that correspond to N samples (hybridizations) and K variables (probesets)  within a MATLAB v. 7.5.0.342 (R2007b)  computational environment. Each experimental dataset was separated into four subsets representing the: 1) spike-in hybridization controls, 2) spike-in polyA+ controls, 3) internal RNA degradation controls (Affymetrix-designated) and 4) endogenous or normalization control genes The optimal number of components to include in the PCA model was determined by the minimum of both the root mean square error of calibration (RMSEC) and of cross-validation (RMSECV) employing a venetian blinds algorithm for which the dataset were split according to their size (here 10 splits for 137 hybridizations). Datasets that contain duplicate hybridizations were subject to replicate sample trapping as the presence of related samples in test and training sets may lead to skewed cross-validation results. Here, an additional cross-validation using a random subset scheme was employed and checked for consistency with the venetian blinds approach. A summary of the PCA models including the cumulative % variance captured for each model is provided in Table 8. The greatest difference observed is for probesets with intensities less than 26, for which the data resembles a "non-normalized" pattern.In this evaluation of internal and external controls for assessing microarray performance, it is assumed that these controls act in a manner similar to and consistent with endogenous transcripts in the biological sample when all are assayed with gene expression microarrays. To provide an initial quality assessment of the probeset-specific performance, the variance behavior of the individual probesets of the controls was examined in relation to average signal level across the entire experiment. Similar approaches have been employed to illustrate relationships between probeset signal level and precision metrics in microarray data ,31-33. TThe different classes of controls are distinct in terms of the overall variability (SD) across their inherent RMA intensities; this observed difference among the control groups can be used as a screening tool to identify high-quality experimental datasets from the lower-quality or more "noisy" datasets . The expDap, Thr, Phe and Lys polyA+ controls . Greater variability, likely attributable to differences in processivity during cRNA labeling, is generally observed for the 5' probesets (denoted with "5"), followed by a moderate level of variability for the probesets that target the middle of the transcript (denoted with "m"). As provided by the quality metrics in the Microarray Centre Quality Assessment (QA) report [The mean/SD plots also reveal the relative precision of individual probesets within a control group relative to other probesets in the experimental dataset. A few of the internal RNA degradation probesets are considerably more variable than both the average background signal and the internal endogenous genes. As shown in Figure ) report , the majThe mean/SD dispersion plots provide an overview of quality through an assessment of probeset-specific performance within the experimental dataset but do not definitively identify particular samples that may be outliers within the experimental dataset. Samples that contribute the greatest amount of variance to the experiment can be resolved through a PCA of the spike-in controls and can be used to identify problems with the discrete sample preparation steps . Likewise, PCA models of the internal controls can be utilized to verify sample RNA integrity or to account for other sample degradation issues.2 probe intensities as well as relatively high average array background values and RawQ noise values, the latter of which is a measure of pixel-to-pixel variation among the probesets that is used to calculate the array background [In an effort to identify individual arrays that may be problematic, PCA was employed to explore the variability within the spike-in hybridization control dataset. PCA score plots for the first three principal components (PCs) of the hybridization control data subset of the rat CWTA dataset are shown in Figure ckground . Notably2 values can be used to identify samples that are outliers and that might possess relatively high leverage along the principal axis of the model, analogous to the end points of a linear regression model. The Q residuals in Supplemental Figure S1(a) highlight hybridization B22, which has also been flagged as potential outlier by the NUSE plot. Hotelling T2 values consistently highlight hybridizations F64, E60, I90, F68 for which scanner QC measures have been denoted as problematic .The Q residuals of the PCA model  between the polyA+ spike in concentration and raw signal value observed for hybridization NNC2004020512Aaa (sample J111) as reported in the MiMiR QA report. Correlation values of R2 > 0.95 are expected for typical samples. Outliers such as these are easily identified through an examination of the relative RMA intensities; as an example, the relative RMA intensities for this extreme polyA+ control outlier are shown in Table 2 concentrations for the polyA+ spike-in controls.A cocktail of RNA controls with artificial polyA+ tails are spiked into each RNA sample over a range of concentrations Table  to monitelation R = 0.4498Phe and Lys polyA+ controls are substantially above the Affymetrix-defined tolerance ratio of 3, which is usually indicative of either insufficient labeling efficiency or poor sample quality. For example, the hybridizations denoted as G73, G74, G75, G82 and G77 had 3'/5' ratios for the relatively high concentration Phe polyA+ control of 30.32, 18.91, 11.10, 6.70 and 6.82, respectively.The PCA model for the polyA+ controls comprises of 4 PCs. The first PC captures the largest variance (76.8%) and primarily separates hybridization J111 from the other 136 hybridizations within the experimental dataset  ). This result is consistent with the obvious difference in RMA intensity; the log2 probe intensities for these four polyA+ controls for hybridization J111 were several orders of magnitude lower when compared to the other hybridizations in the experiment. In contrast, the log2 intensities for the Trp polyA+ control probesets  were relatively small relative to the overall experiment (median z-score of 0.7). Consistent with the observed intensity data, these probesets have a low contribution to the loadings for the PC 1. In addition, the probeset loading pattern of 5'-middle - 3' trend as observed for the higher concentration controls . This is observed for the group of flagged hybridizations for elevated 3'/5' ratios for GAPDH and/or β-Actin controls  that are separated in PC 1 and represent 68% of the model variance. Likewise, the major variables that contribute to the loadings for PC 1 correspond to the 5'-end and middle-segments of the Affymetrix GAPDH and β-Actin probesets  are separated on PC 2. Notably, hybridizations B20 and D46 are partially separated from the other hybridizations on PC 3 (≈ 7%), the former of which has a slight indication of cRNA degradation (3'/5' ratio of 3.16 for β-Actin) but it is unclear how D46 (hybridization ID NNC2003070706Aaa) is different from the others with regards to the Affymetrix cRNA degradation internal controls. In all, the PC 1 × PC 2 × PC 3 scores profile as illustrated in Figure The PCA model results for the Affymetrix-designated RNA degradation internal control data Figure  illustraLys, Phe, Dap and Thr). This exemplifies the utility of controls that probe data quality at multiple stages in data generation  capture only 53% of the total model variance, with the remainder dispersed among subsequent PCs Figure . The PC n Figure .Different types of controls provide distinct levels of data quality information that can be readily resolved through principal component analysis. A layered PCA modeling of the four classes of controls  is valuable for evaluating data quality at a number of stages within the experiment . The variance at each stage, whether spike-in or internally present, provides complementary information on data quality to those provided by the QA/QC metrics.This work supports the use of both external and internal control data to assess the technical quality of microarray experiments. In the results presented here, using a layered PCA approach, we find that both the external and internal controls carry with them the critical information about technical performance that is consistent with whole-array quality assessment. This information is obtained for every sample generated using spike-in controls and permits assessment of technical performance for each array. This study is thus a key element in our efforts to develop control methods, materials and designs that support the use of genome-scale data with confidence. Furthermore, these results validate the proposal to use such controls with large data sets generated on multiple platforms or with other multiplexed technology applications.cRNA: copy RNA; CSC/IC: Clinical Sciences Centre/Imperial College; ERCC: External RNA Controls Consortium; MAQC: MicroArray Quality Control; MiMiR: Microarray data Mining Resource; NUSE: Normalized Unscaled Standard Error; PCA: Principal Component Analysis; polyA+: polyadenylated; QA/QC: Quality Assurance/Quality Control; RawQ: Noise attributed to both the scanner and sample quality; RMA: Robust Multichip Average; RMSEC: Root Mean Square Error of Calibration; RMSECV: Root Mean Square Error of Cross Validation; SD: Standard Deviation.The authors declare that they have no competing interest.HCC and LG designed and implemented the spike-in controls for use in the MiMiR experimental protocols in addition to the experimental quality control assessment metrics. HCC, LG and MLS conceived of this study's approach to the analysis of these data and participated in the study design and coordination. DLD developed a prototype for data analysis and display. KAL created the datasets, performed the data preprocessing and chemometric analysis. All authors were involved in the data interpretation. KAL and MLS drafted the manuscript and all authors read and approved the final manuscript.2 plots and loadings plots for the spike-in hybridization and polyA+ control data and the internal cRNA degradation control data subsets in Supplemental Figures S1, S2, and S3 respectivelycontains the PCA model results include both diagnostic Q/Hotelling T. Additionally, two supplemental tables are provided to aid in the data interpretation within the manuscript. These include Table S1 that provides condensed annotation information for the single Rat experiment and Table S2 that lists the probe set identifiers for spike-in hybridization and polyA+ controls together with the internal Affymetrix cRNA degradation (RNAd) and endogenous controls for the RAE230A array.Click here for filecontains a copy of the Microarray Centre Quality Assessment of Affymetrix Data for EXP_CWTA_0103_01 comprising 138 hybridizations on Rat Expression Set 230A Arrays. Hybridization HFB2003080611Aaa listed in the QA report was excluded from the PCA dataset as the full annotation information was not available at the time of this study.Click here for file


High throughput parallel sequencing, RNA-Seq, has recently emerged as an appealing alternative to microarray in identifying differentially expressed genes (DEG) between biological groups. However, there still exists considerable discrepancy on gene expression measurements and DEG results between the two platforms. The objective of this study was to compare parallel paired-end RNA-Seq and microarray data generated on 5-azadeoxy-cytidine (5-Aza) treated HT-29 colon cancer cells with an additional simulation study.We first performed general correlation analysis comparing gene expression profiles on both platforms. An Errors-In-Variables (EIV) regression model was subsequently applied to assess proportional and fixed biases between the two technologies. Then several existing algorithms, designed for DEG identification in RNA-Seq and microarray data, were applied to compare the cross-platform overlaps with respect to DEG lists, which were further validated using qRT-PCR assays on selected genes. Functional analyses were subsequently conducted using Ingenuity Pathway Analysis (IPA).SPARC gene suppression after treating HT-29 cells with 5-Aza. Thirty-three IPA canonical pathways were identified by both microarray and RNA-Seq data, 152 pathways by RNA-Seq data only, and none by microarray data only.Pearson and Spearman correlation coefficients between the RNA-Seq and microarray data each exceeded 0.80, with 66%~68% overlap of genes on both platforms. The EIV regression model indicated the existence of both fixed and proportional biases between the two platforms. The DESeq and baySeq algorithms (RNA-Seq) and the SAM and eBayes algorithms (microarray) achieved the highest cross-platform overlap rate in DEG results from both experimental and simulated datasets. DESeq method exhibited a better control on the false discovery rate than baySeq on the simulated dataset although it performed slightly inferior to baySeq in the sensitivity test. RNA-Seq and qRT-PCR, but not microarray data, confirmed the expected reversal of These results suggest that RNA-Seq has advantages over microarray in identification of DEGs with the most consistent results generated from DESeq and SAM methods. The EIV regression model reveals both fixed and proportional biases between RNA-Seq and microarray. This may explain in part the lower cross-platform overlap in DEG lists compared to those in detectable genes. In recent years, RNA-Seq emerged as an appealing alternative to classical microarrays in measuring global genomic expressions ,2. The RSPARC (EMBL: ENSG00000113140) gene promoter and reverse suppression of SPARC mRNA expression in HT-29 cells .Assessment of error in the microarray data was conducted using the model described previously by Rocke and Durbin  in whichescribed  as shownHere y is the probe expression intensity, α denotes the mean of background noise, c stands for Aexpress  was usedε; and the transformed data could then be used to build an empirical distribution of u. The true expression levels of simulated genes were sampled from this empirical distribution in such a way that: a histogram of true data was generated using 500 bins at first step; a simulated gene was then assinged to a bin based on the frequency with a small turbulance added to its value. Uniform distribution (ranging the length of the bin) was assumed to the turbulance term to differentiate genes in the same group. As a result, the transformed expression level of a gene at a certain x% quantile of a given sample is equal to the same x% quantile of g(y) in terms of distribution + a small turbulance. The turbulance added same effect of variation to every gene because of the variance stablizing transformation.By averaging across the φ was also generated from a gamma distribution described before [The RNA-Seq data were fitted in a negative binomial model as described by Kvam . The mead before . In pracPre-defined significant DEGs were randomly sampled so that the log fold changes of these preset DEGs were generated from a mixed normal distribution where the probabilities of being up and down regulated were both equal. Moreover, in our analysis, the absolute expectation of log fold changes and standard errors for up and down regulated genes were set to be the same.bona fida DEGs were generated with their fold changes following a log-normal distribution with 95% of the 1000 genes having their fold changes above the given level. We implemented a total of seven algorithms in our study, namely T test, SAM and eBayes on microarray data and baySeq, DESeq, SAMseq, NOISeq for RNA-Seq data.In practice, we generated 10 sets of simulated data in practice with 5 replicates included in both treatment and control group and 1000 random selected genes were preset to be differently expressed in different levels using a method of 95% minimum fold change such that the 1000 preset In this work, the sensitivity and false discovery rates (FDR) were firstly evaluated for each DEG method under the 95% minimum fold change of 2 for preset DEGs and FDR cutoff of 0.05. For NOISeq method, a q = 0.8 (recommended by NOISeq author) criterion was used due to the absence of FDR control in this method. We further evaluated the sensitivity and false positive rate of each DEG algorithm by varying the differential significance levels of the preset 1000 genes using 95% minimum fold change method. Specifically, a range of values from 0.5 to 4 by an increment of 0.5 were used to generate the simulated DEGs.SPARC and GAPDH (EMBL: ENSG00000111640), and the corresponding commercial Taqman assays are listed in Additional file GAPDH - Ct for the test gene) were calculated for each RNA sample. The Student t-test was used to analyze whether there was a significant difference between the mean ΔCt for the control vs. the 5 μM 5-Aza treated HT-29 groups, with a threshold significance level of 0.05. The fold change in gene expression was calculated as 2-ΔΔCt (ΔΔCt = ΔCt of 5-Aza group - ΔCt of control group).Reverse transcription was performed on 5 μg aliquots of each of the six RNA samples using the SuperScript III First-Strand Synthesis System for RT-PCR  according to the manufacturer's protocol. QRT-PCR was initially performed on serial dilutions of the cDNA for each Taqman assay kit using Taqman assay kits  in order to confirm that each of the assays were conducted in the linear range and the slopes of the threshold cycle Ct when plotted against the dilution were the same for all of the assays. Thirteen genes were selected by the majority vote of platform specific DEG detection methods and are categorized into 3 groups, which are: 1) commonly identified on RNA-Seq and microarray datasets; 2) RNA-Seq data only and 3) microarray data only. This list of genes, with the additional ® Inc, Redwood city, CA) on up-regulated DEGs (5 µM vs. 0 µM 5-Aza) and down-regulated DEGs respectively. 5 DEG lists were generated by the SAM, eBayes, Cuffdiff, DESeq and baySeq algorithms. Significantly enriched canonical pathways were selected based on the p value cutoff of 0.05 and included gene number > 3 [Based on the results of the simulation, we performed IPA analysis  indicated a strong correlation between the two platforms  of the RNA-Seq specific genes did not have corresponding probe sets on the array. The overlap rates of the genes detected by both RNA-Seq and microarray datasets for the 0 μM, 5 μM and 10 μM 5-Aza HT-29 cultures, respectively, ranged between 66.8-68.6% Figure . We furts Figure . This res Figure . The hisAn Errors-In-Variables regression model was built to investigate the consistency between normalized microarray gene abundances and the normalized FPKM genomic intensities from RNA-Seq platform with both measurements in log2 scale. Using the maximum likelihood estimation of the EIV model, we obtained a linear relationship of the gene expression profiles between RNA-Seq and microarray for each experimental group  and five RNA-Seq algorithms  were applied to the experimental HT-29 microarray and RNA data, respectively . Cuffdiff detected 15↑ 485↓, SAMSeq detected 0↑ 626↓, DESeq detected 43↑ 389↓, baySeq detected 58↑ 424↓, and NOISeq detected 95↑ 123↓ using the same RNA-Seq data . With the exception of the T-test and NOISeq, the remaining six algorithms detected an overall down-regulation in gene expression when the concentration of 5-Aza was increased from 5 μM to 10 μM. This could reflect toxic effects of 5-Aza at the higher 10 μM concentration.The cross-platform overlap rates between the DEG lists generated by each of the three microarray algorithms with DEG lists generated by each of the five RNA-Seq algorithms are summarized in Table See Methods). Intra-microarray platform comparisons revealed that the T-test generated DEG list overlapped poorly with both the SAM and the eBayes generated DEG lists. However, SAM and eBayes DEG lists achieved 95% overlap with each other. Intra-RNA-Seq platform comparisons revealed that baySeq and DESeq DEG lists achieved 75.7% overlap with each other, while the overlap percentages ranged between 46% and 54% for the remaining RNA-Seq algorithms. The highest cross-platform overlap percentages (48-50%) were observed between the SAM and eBayes microarray DEG lists and the baySeq and DESeq RNA-Seq DEG lists. Not surprisingly, the T-test DEG list overlapped poorly with the results of all the RNA-Seq algorithms.Simulated datasets were generated from independent parallel RNA-Seq and microarray datasets generated from kidney tissue . In thisSee Methods), the baySeq method continues to outperform other algorithms while DESeq, slightly inferior to baySeq, has been generally yielding good results, too ; The baySeq method, albeit more sensitive in calling true positives, has relatively poorer performance in controlling FDR and this drawback becomes more remarkable at higher fold change end . The specificity of each method was also evaluated and all of them were well above 99.9%.The sensitivity and the false discovery rate of each method were also calculated in ten simulated runs for the sake of accuracy evaluation. Based on the same significance level (95% minimum fold change method: FC cutoff ≤ 2 and FDR ≤ 0.05), we found that baySeq produced the highest sensitivity (52.6%) from RNA-Seq while SAM achieves the best sensitivity (50.4%) among microarray methods Figure . On the o Figure . On the o Figure , Right; d Figure , Left. TSPARC gene expression was previously reported to be undetectable in control HT-29 cells but detectable in 4 μM 5-Aza treated HT-29 cells using a qualitative gel-based RT-PCR method [SPARC gene . DESeq constantly results in the lowest FDR among all the RNA-Seq algorithms evaluated in the simulation experiments, indicating its superior reliability. The NOISeq showed a very poor performance on FDR evaluation curve particularly with lower 95% minimal fold change thresholds , reflecting the fact that NOISeq's DEG discerning power by comparing noise distribution against a true signal was seriously compromised when the 'true difference' is less remarkable. In practice, it is of theoretical importance to weigh more on preventing false positives than false negatives; we thus favor DESeq over Bayseq in RNA-Seq analysis as the former method controls FDR better than the latter in higher differential significance level .With respect to FDR evaluation, however, baySeq resulted in more false positive calls than most of the other RNA-Seq algorithms except for NOISeq, especially when the 95% minimum fold changes of true positive genes are higher  the existence of both fixed and proportional biases detected by the errors-in-variable (EIV) regression model, and (2) discrepancies in DEG identification have been discovered in our study. We further confirmed that the DEG discrepancies are mostly related to the different algorithms used for both platforms. Among all the DEG algorithms surveyed in this study, the largest cross-platform overlaps were observed between the DEG lists generated by two RNA-Seq algorithms, baySeq and DESeq, and the DEG lists generated by two microarray algorithms, SAM and eBayes, from the HT-29 experimental dataset. The simulation studies, which did not include evaluation of Cuffdiff, indicate that the the DESeq algorithm outperformed the other RNA-Seq algorithms, based upon the combined considerations of sensitivity and false discovery rate. DESeq also demonstrated the highest overlap rate with the DEG list generated by SAM from the microarray data. Overall, the nonparametric based DEG methods such as SAMSeq or NOISeq exhibited suboptimal performance compared to their parametric counterparts, partly due to the limited number of replicates. QRT-PCR validated a higher percentage of the DEGs identified by both platforms and RNA-Seq only, than the DEGs identified by microarray only. Finally, while there were common IPA canonical pathways identified by both microarray and RNA-Seq data, a large number of additional canonical pathways were identified by RNA-Seq data alone. No additional canonical pathways were identified by microarray data alone.5-Aza: 5-azadeoxy-cytidine; EIV: Errors-In-Variables; DEG: differentially expressed genes; FDR: false discovery rate; FC: fold change; qRT-PCR: quantitative reverse transcriptase polymerase chain reaction.W.R.M. has participated in Illumina sponsored meetings over the past four years and received travel reimbursement and an honorarium for presenting at these events. Illumina had no role in decisions relating to the study/work to be published, data collection and analysis of data and the decision to publish.W.R.M. has participated in Pacific Biosciences sponsored meetings over the past three years and received travel reimbursement for presenting at these events.W.R.M. is a founder and shareholder of Orion Genomics, which focuses on plant genomics and cancer epigenetics.The other authors declare that they have no competing interests.Drs. Jennie Williams, Eric Antinoiou, Paula Denoya and W. Richard McCombie performed the experiment; Dr. Xiao Xu designed the study, analyzed the data and wrote the initial draft of the paper; Yuanhao Zhang also designed the study and performed EIV and simulation analysis, Drs. Song Wu and Wei Zhu participated in study design and provided statistical advices, Ellen Li funded and designed the study. Dr. Nicholas Davidson performed the biological functional analyses. All authors participated in revising the final draft of the manuscript.Table of ABI Taqman Assay Kit IDs used in qRT-PCR assays. The list of 13 genes selected by majority vote on platform specific DEG methods plus GAPDH and SPARC are included in this table.Click here for fileTable of DEGs for each algorithm. In this file, each gene that was significant in at least one algorithm is shown with corresponding fold change. NA under an algorithm indicates the gene is not included in the DEG list generated by that algorithm. Up-regulated and Down-regulated genes are in separate sheets. Log2 scaled normalized gene expression intensities are shown for each gene on both microarray and RNA-Seq platform.Click here for fileTable of significant pathway for both up and down regulated DEGs. Pathways that were at least significant in one algorithm are shown in this file. If a pathway is marked as significant in an algorithm, the corresponding DEGs detected by that algorithm are included; otherwise the cell is left blank. Both up-regulated and down-regulated pathways are shown in this file, while down-regulated pathways are bolded.Click here for file


Validating statistical analysis methods for RNA sequencing (RNA-seq) experiments is a complex task. Researchers often find themselves having to decide between competing models or assessing the reliability of results obtained with a designated analysis program. Computer simulation has been the most frequently used procedure to verify the adequacy of a model. However, datasets generated by simulations depend on the parameterization and the assumptions of the selected model. Moreover, such datasets may constitute a partial representation of reality as the complexity or RNA-seq data is hard to mimic. We present the use of plasmode datasets to complement the evaluation of statistical models for RNA-seq data. A plasmode is a dataset obtained from experimental data but for which come truth is known. Using a set of simulated scenarios of technical and biological replicates, and public available datasets, we illustrate how to design algorithms to construct plasmodes under different experimental conditions. We contrast results from two types of methods for RNA-seq: (1) models based on negative binomial distribution (edgeR and DESeq), and (2) Gaussian models applied after transformation of data (MAANOVA). Results emphasize the fact that deciding what method to use may be experiment-specific due to the unknown distributions of expression levels. Plasmodes may contribute to choose which method to apply by using a similar pre-existing dataset. The promising results obtained from this approach, emphasize the need of promoting and improving systematic data sharing across the research community to facilitate plasmode building. Although we illustrate the use of plasmode for comparing differential expression analysis models, the flexibility of plasmode construction allows comparing upstream analysis, as normalization procedures or alignment pipelines, as well. RNA sequencing (RNA-seq) technology is being rapidly adopted as the platform of choice for high-throughput gene expression analysis . An RNA-seq experimental design must consider treatment and block structures, and combine them according to the principles of a well-planned design: randomization, blocking, and replication  applying the model to a novel dataset, (2) deriving analytical proofs, (3) using simulations, (4) comparing to a gold-standard measure, or (5) constructing plasmodes. In (1) the true status of nature is unknown, therefore this method must only be accepted as an illustration and not as evidence to support a model. However, any of the last four options, or a combination of them, could be used to demonstrate adequacy of a model. Obtaining a mathematical demonstration (2), may be impossible for some models , μi is the group mean, eij is a sample-specific residual. The transcript sub-index (g) was omitted for convenience.Equation [1] represents the generalized linear model used to generate the simulated datasets:T by 2nr with a known proportion (p0) of differentially expressed transcripts and known group effects (μi). Treatment is represented in this matrix by nr columns, but with only n independent  replicates. While this simulation is not based on the negative binomial distribution, it is still an over-dispersed Poisson process commonly used to simulate RNA-seq counts . But subsequently, the mean treatment differences  are zeroed out if the transcripts are assigned to S0. For transcripts assigned to S1, mean differences are kept unchanged; consequently S1 includes a whole distribution of treatment effects from very small to large according to the distribution of estimated from the experimental data.We set n = 3, 5, 10) and three levels of technical replication . The proportion of differentially expressed transcripts was set to 0.1.We simulated nine scenarios by combining three levels of biological replication  unrelated CEPH (Center d' Etudes du Polimorphisme Humain) grandparents  on 21 lanes of three Illumina GAIIx flowcells and they analyzed the RNA-seq reads with a simple one-way classification (strain) model. We performed descriptive analysis of gene expression data and found that not only strain but also the experiment number (flowcell) explained a large amount of the variation  to build a plasmode with some genes under alternative hypothesis but any other statistical software can be used with the only requirement of defining a sufficient small q-value threshold. After genes are selected no model is used at any time. Similar to the previous section plasmodes are constructed reshuffling data, but in this case and effect estimated from real data is added to selected genes. Again, we expect that this procedure yields plasmodes with identical distribution to real data for non-differentially expressed genes and with comparable effect sizes for differentially expressed genes.We generated 10 plasmodes executing step 4–7 with To illustrate the use of simulated datasets and plasmodes we compared three R  and transcript by transcript F-test (F1). Significance was assessed using 100 sample permutations , while the FPR was calculated as the proportion of false positives (FP) over the total number of transcripts simulated with no differential expression (S0). See Table To compare performances of derived tests in terms of power and type I error rates, we generated receiver operator characteristic (ROC) curves by computing true positive rate (TPR) and false positive rate (FPR) at given significance thresholds. The TPR was calculated as the proportion of true positives (TP) over the total number of simulated differentially expressed transcripts S, while tp-values were compared by quantile-to-quantile plots and histograms.Finally, distributions of Analyses were performed at the Michigan State University High Performance Computing Center facilities using R (version 2.15.1), edgeR (version 3.0.8.4.6), limma (version 3.14.4), DESeq (version 1.10.1) and MAANOVA (version 1.28.0).Figure p-values to the uniform  distribution expected under null hypothesis . In the case with 3 biological replicates and 1 technical replicate Figure , edgeR ap-values to the uniform distribution for each of the plasmode datasets analyzed with the different models. All the models presented large dispersions with some cases being close to the expected values and some being far apart. In particular, edgeR results tend to be above the identity line which means that observed p-values are smaller than expected. On the contrary, DESeq and both MAANOVA tests tend to have a more conservative behavior as they presented larger observed p-values than expected. See also Figure Q-Q plot in Figure p-value distributions  combined with two methods to compute the p-values (tabulated F or permutation). The choice of the best combination varies for each dataset: In the simulation study, either Fs or F1 using permutation provide a p-value distribution closed to a uniform distribution while none of the F-tests using tabulated values provide a reasonable distribution  or technical issues  may arise only from inspecting protocol details and applying explorative statistical analyses. For instance, descriptive analysis of the Cheung dataset and visualization of samples using multidimensional scaling analysis Figure  suggestep-values tend to be closer to the nominal significance levels. Admittedly, after adjusting for type I error rates, power was similar for edgeR and DESeq and higher than that from MAANOVA  those directly tailored to count data by using negative binomial distributions  or (2) a Gaussian model after transformation (MAANOVA). We found that edgeR and DESeq incur in inflated type I error rates for small significance levels Figures , 6B, 7B A Figure . HoweverThese results emphasize the fact that RNA-seq data are complex and to decide what method to use may be experiment-specific due to the unknown distributions of expression levels. Plasmode may contribute to decide which method to choose by using a similar pre-existing dataset and comparing results. It is critical to select a dataset that has a complete description of the experimental design and detailed protocols of how the data were obtained. Using this information, it is possible to design proper null and alternative datasets. For example, it was easy to find a set of differentially expressed genes in the mouse dataset that studied two inbred lines. Contrarily, in the human dataset, it was not possible to explain the variation in expression only as a consequence of gender effects. The human subjects came from an outbred population and factors such as age, weight, or other characteristics could have explained differences in gene expression. Granted, any of the mentioned effects could have been included in the model if the information was available. The promising results obtained from this approach, emphasize the need of promoting and improving systematic data sharing across the research community to facilitate plasmode building.p-values.”Finally, the flexibility of plasmode construction allows comparing model tuning selection for downstream analysis but also upstream analysis, as normalization procedures or alignment pipelines, could be contrasted. Future uses of plasmodes could be: comparison of alignment programs for a given statistical analysis model or even exploring interaction of statistical model and read processing protocols to find optimal combined pipelines for data processing “from reads-to-The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


One of the most important applications of microarray data is the class prediction of biological samples. For this purpose, statistical tests have often been applied to identify the differentially expressed genes (DEGs), followed by the employment of the state-of-the-art learning machines including the Support Vector Machines (SVM) in particular. The SVM is a typical sample-based classifier whose performance comes down to how discriminant samples are. However, DEGs identified by statistical tests are not guaranteed to result in a training dataset composed of discriminant samples. To tackle this problem, a novel gene ranking method namely the Kernel Matrix Gene Selection (KMGS) is proposed. The rationale of the method, which roots in the fundamental ideas of the SVM algorithm, is described. The notion of ''the separability of a sample'' which is estimated by performing  Microarray data has been applied to the class prediction of different samples, from which the disease diagnosis and prognosis can benefit. A microarray dataset usually contains thousand of genes and a relatively much smaller number of samples  Empirically, wrappers and embedded methods have been observed to be more accurate than filters Meanwhile, although the choice of the classifier also presents a wide diversity, SVMs have been widely recognized for its generalization abilities In summary, a widely-accepted scheme for the analysis of microarray data has been ''identification of DEGs by statistical tests followed by sample classification using SVMs''. The justification is that the prediction accuracy of various classifiers including SVMs, depends on how discriminant the features are. However, SVMs belong to the family of sample-based classifiers whose generalization performance comes down to, more precisely, how discriminant the samples are. DEGs identified by statistical tests cannot guaranteed to establish a set of discriminant samples for SVMs. Consequently, it cannot be promised the highest degree of accuracy for sample classification. This problem necessitates the development of gene selection algorithms that are more consistent with the fundamental ideas of SVMs. It is naturally desired that, the proposed methods can bypass the computationally-expensive training procedure of SVMs, which is required by the SVM-RFE algorithm Given a binary classification problem with the training data set of: As depicted in The condition of optimality requires that the vector Each constant For nonlinear problems where the training data are not separable in the input space, a function, denoted as On a test sample According to the sign of the As mentioned previously, each training sample is eventually assigned a Lagrangian multiplier Using 1. Depending on its class label Consider a sample We use the inner product to measure the similarity level between vectors. Denoting the number of the positive training samples as Likewise, denoting the number of the negative training samples as As a result, we can express, mathematically, the expectation that a positive sample bears more resemblance to class ''+'' than to class ''And a negative training sample 2. The class resemblance of a boundary sample to its supposed class is not as striking as those samples with 3. (a) exactly on the hyperplane of (b) in the region between the hyperplanes of (c) in the region between the hyperplanes of A training sample from group (a), is a boundary sample but its class label can be correctly restored by the SVM solution. As with positive samples whose For a training sample from group (b), the SVM classifier would not have been able to correctly restore its class label if it weren't for the introduction of the slack variables. Our interpretation is that, the class resemblance of this sample to its supposed class is so ambiguous that the SVM has difficulties in acknowledging its actual class membership.For a training sample from group (c), the SVM classifier is simply unable to correctly restore its class label. It is very likely that the class resemblance of this sample to its supposed class in fact contradicts its given class label.In mathematical terms, we reckon that a positive training sample of either group (b) or group (c) satisfies: The hidden information for In summary, the resultant value of Luckily, our analysis above implies that that the function of In the SVM algorithm, the vector of We justify the introduction of standard deviations in the denominator by considering two positive training samples in the feature space. The first sample is assumed to have come from a region of denser population than the second one. We reckon that, compared with the second sample, the first sample is more typical a representative of class ''+'' and is believed to be more similar to class ''+''. The positive sample from a denser population is expected to exhibit a lower deviation of the elements The values of Meanwhile, the similarity between each sample and itself is supposed to be Since the separability of each sample has an impact on the the class separability of a problem, we propose to use the sum of each sample's separability score as an estimate of the separability of the problem.A word about the formulation of In the case of linear kernels, take a positive sample In the case of Gaussian RBF kernels, in order to have Since each gene subset introduces a classification problem represented by the set of training samples, the gene subset thus corresponds to an estimate of the separability of the problem. Consequently, DEGs can be identified as those resulting in ''easier problems'' of high separability. This is the essential idea of our kernel matrix induced gene selection methods, which has been illustrated in Given a microarray dataset of Hence the function The An alternative to the KMGS which proceeds in a greedy manner is also developed, which is namely the Kernel Matrix Sequential Forward Selection (KMSFS) algorithm. The algorithm starts with an empty set of selected DEGs. At each iteration, the algorithm identifies a single DEG which is then appended to the set. We now describe how the KMSFS algorithm proceeds between two consecutive iterations.Given a microarray dataset of Each gene from the remaining The pseudo codes for the KMGS and KMSFS algorithms are given respectively in The proposed methods have noticeable merits:Filter methods identify discriminant features, making them suitable for feature-based classifiers whose normal vector is the linear combination of features. However, A majority of wrapper and embedded methods are based on the assumption that most microarray datasets pose linear problems. However, we reckon that, the problem presented by a set of DEGs can hardly be a linear one when the the set size is as small as only one or two.But the generalization to nonlinear cases have been challenging for various wrappers and embedded methods. For example, the SVM-RFE Advantageously, our algorithms can be directly applied to nonlinear cases by opting for Gaussian RBF kernels. The Gaussian RBF 3. The output of The dataset contains, in total, 136 samples of two types which respectively have 77 and 59 cases. Each sample includes expression values of 12600 genes.The dataset contains the expression values of 2000 genes from 62 tissues, of which 22 are normal and 40 are cancerous.The dataset was collected from 72 patients. 47 of them were diagnosed with acute acute lymphoblastic leukemia (ALL) and 25 with acute myeloid leukemia (AML). Expression values of 7129 genes were measured.Both the prostate dataset and the colon dataset were normalized using the following procedure. A microarray dataset with For the leukemia dataset, we applied the pre-processing procedure proposed by Dudoit et al. Various gene selection algorithms are evaluated and compared by the error rate of SVMs. The simplest technique for error estimation is the holdout method which splits the dataset into a training set and a test set. The gene selection algorithm is performed on the training set and sample classification on the test set. However, the holdout method has been highly discouraged for microarray datasets which usually contain a small number of samples. In contrast to researchers who applied gene selection to the entire training set and employed The B.632+ estimator involves resampling, with replacement, of the original dataset. From a dataset of The flow chart for evaluating a gene selection algorithm using the B.632+ technique has been given by For the methods of KMSFS and KMGS, both the linear kernel and the Gaussian RBF kernel were tested. This resulted in altogether four algorithms which are referred to as Gaussian KMSFS, Gaussian KMGS, linear KMSFS and linear KMGS. They were compared with two wrapper methods: the leave-one-out calculation sequential forward selection (LOOSFS) which improved the least-squares bound measure The Leave-One-Out Cross-Validation(LOOCV) error has been generally used for measuring the generalization abilities of SVMs and Least-Squares SVMs (LS-SVMs). The LOOSFS method thus identifies as DEGs those genes which result in a LS-SVM classifier with the minimal Leave-One-Out Cross-Validation(LOOCV) error rate. The beauty of the algorithm consists in the efficient and exact computation of the LOOCV error. To address the ''ties problem'' in which multiple gene subsets achieve the same LOOCV error rate, a further selection criterion is imposed which favors the gene subset with minimal empirical risk.The starting point of the GLGS method is also the employment of the exact formulation of the LOOCV error for LS-SVMs. The method then utilizes the gradient descent algorithm to seek a diagonal matrix which eventually minimizes the LOOCV error. Genes are ranked according to the absolute values of the diagonal elements of the diagonal matrix.With Cho's method and Yang's methods, genes are individually ranked. We use the following notations for their descriptions. Each micorarray dataset with The score, denoted as The between-class variation with respect to gene In order to estimate within-class variations in terms of gene Two metrics for measuring within-class variations, denoted as compactEventually, two score functions which decide the ranking of gene We refer to these two score functions respectively as Yang's method 1 and Yang's method 2.Cho's method and Yang's two methods identify as DEGs those genes whose associated We employed grid search and Friedman rank sum tests combined with Holm correction to tune the parameters for different algorithms.Among the total 10 gene selection algorithm, both Gaussian KMSFS and Gaussian KMGS require the parameter The Friedman rank sum test is a non-parametric alternative to ANOVA with repeated measures. The test statistic for the Friedman test is a Chi-square with We take the algorithm of Gaussian KMSFS as an example to explain how to apply Friedman test for the discovery of optimal values on Friedman rank sum test was used to detect statistical differences among these 77 groups. The test was based on 100 sets of ranks, with each set corresponding to an individual classification problem. The performances of different parameter settings analyzed are ranked separately for each problem. If we rejected the null-hypothesis stating that all the 77 settings led to equal performance in mean ranking, we employed the Holm post-hoc analysis to identify which setting was significantly better than the rest.All the gene selection methods were coded in Matlab. The linear SVM was implemented using LIBSVM Using Friedman tests with Holm correction, optimal settings on GLGS and linear KMSFS shared the optimal setting of With a minimal error rate of As shown by The error rate of linear KMSFS remained noticeably lower than the filter methods when the number of DEGs fell between 10 and 60. When the number of DEGs grew larger, Gaussian KMGS showed better than performance than the four filter methods.The performance of Gaussian KMGS and linear KMSFS remained competitive to those of the filter methods, respectively between the value ranges of These comparisons lead to the conclusion that linear KMSFS and Gaussian KMGS are the two best methods for the prostate dataset.For Gaussian KMGS, Gaussian KMSFS and LOOSFS, the optimal settings on For the other 3 wrapper methods which are respectively linear KMGS, linear KMSFS, GLGS and the 4 filter methods which are respectively Fisher's ratio, Cho's method and the two methods of Yang's, their optimal parameter settings were found to be With a minimal error rate of It is interesting to note that Gaussian KMGS, with only 10 DEGs, reached the lowest B.632+ error rate which was approximately 0.10. Also the lowest B.632+ error rate of LOOSFS, which was 0.11 was lower than that reported in Gaussian KMSFS remained better than the 4 filter methods whose performances were comparable between each other. Meanwhile, the error rates of Gaussian KMGS were also lower than those of the filter methods, particularly for a smaller number of selected DEGs.In conclusion, Gaussian KMSFS and Gaussian KMGS have proved to be the best methods for the colon dataset.For Gaussian KMGS, Gaussian KMSFS and LOOSFS, the optimal setting on the parameter pair of For GLGS, the optimal setting was found to be However, Gaussian KMGS, linear KMGS and linear KMSFS failed to perform as well as LOOSFS. We reckon it might be attributable to the preprocessing procedure which resulted in the removal of over 86% of the original 7029 genes, although this viewpoint has to be confirmed with more experiments.We regarded the LOOSFS and the Gaussian KMSFS as the two best gene selection algorithms for the leukemia dataset.Due to the employment of B.632+ error estimation technique, each gene selection algorithm was applied to the 200 sets of bootstrap samples as well as the original training set. For each of these 201 sample sets, we selected a sequence of 100 DEGs. This resulted in altogether 201 sets each of which contained 100 DEGs.We calculated the frequency with which each of the 2000 genes was selected into the 201 sets of DEGs and drew the heatmaps of 50 DEGs that were selected most frequently. For the algorithms of Gaussian KMGS, Gaussian KMSFS and LOOSFS, the outcome of gene selection procedures is influenced by the value setting on the parameter Heatmaps for the ten gene selection methods, were shown by It can be seen from The second best gene selection algorithm for the colon data is Gaussian KMGS whose selected DEGs have been presented by The above facts suggest that, although filter methods selected genes whose values, in general, differ significantly between opposing classes, our kernel induced algorithms seemed not to hold it as the selection criterion. Instead, our methods endeavored to select genes that could establish a set of ''discriminant samples'' for SVMs. This possibly accounts for their superiority in terms of B.632+ error rates on the colon dataset.For linear KMGS and linear KMSFS, their B.632+ error rates are affected by value settings on Using the prostate dataset and the colon dataset, we employed the Friedman rank sum test with Holm correction to study the sensitivity of sample classification to value settings on We kept We first analyzed the sensitivity of sample selection for Gaussian KMGS. At the row with Since the optimal value setting for For the other rows for which We can conclude that B.632+ error rates are insensitive to choices of At the first row for Gaussian KMSFS, we can see that For the remaining three rows whose Thus, for Gaussian KMSFS, B.632+ error rates are sensitive to choice of Similar analysis can be performed for the colon dataset, whose scores for various Regarding Gaussian KMGS, we can see that B.632+ error rates are insensitive to choices of In terms of Gaussian KMSFS, we can see that sample classification is sensitive to We then kept For Gaussian KMGS, we can see that B.632+ error rates are sensitive to choices of For Gaussian KMSFS, when In term of both linear KMGS and linear KMSFS, B.632+ error rates remain sensitive to Statistical tests select genes whose expression values differ significantly between the two opposing classes, i.e., the discriminant genes. Samples-based learning machines including SVMs favor genes which results in a set of discriminant samples. The discriminant genes can not be guaranteed to result in a set of discriminant samples. We have shown that the genes leading to discriminant training samples can be detected by applying statistical tests to the kernel matrix.In addition to the competitive performance demonstrated on the three public microarray datasets, the proposed kernel matrix induced gene selection algorithms offer extra advantages:Generality. Our methods are considered applicable to any kernel classifiers, not just SVMs.Flexibility. For the implementation of our methods, users can opt for any mercer kernel which can be linear, Gaussian RBF, sigmoid, or polynomials. However, depending on the specific kernel, properly-designed preprocessing steps may be required. For examples, Gaussian RBF kernels require the tuning of the width parameter. For linear kernels, strategies are required to ensure the diagonal elements of the kernel matrix, each of which suggests the similarity between a sample and itself, to be uniformly one.It is also worth attention that microarray datasets have usually been assumed to present linear problems. However, it is unlikely to be true in the case that the number of DEGs is as few as one. Interestingly, linear problems can be solved by SVMs with nonlinear kernels, while nonlinear problems are hardly solvable with SVMs using linear kernels. A potential solution to the nonlinear problem posed by a small number of DEGs could be the application of nonlinear SVM classifiers. But our method suggested a successful alternative which is the use of the nonlinear Gaussian RBF kernel for the identification of DEGs. We reckon that this effort of instilling ''nonlinearity'' into the identification of DEGs has contributed to the better empirical performance of our methods.Additional File S1The three microarray datasets  in MATLAB format which were used in this work.(ZIP)Click here for additional data file.