Microarray technology allows researchers to simultaneously monitor changes in the expression ratios (ERs) of hundreds of genes and has thereby revolutionized most of biology. Although this technique has the potential of elucidating early stages in an organism's phenotypic response to complex ecological interactions, to date, it has not been fully incorporated into ecological research. This is partially due to a lack of simple procedures of handling and analyzing the expression ratio (ER) data produced from microarrays.Nicotiana attenuata, using procedures that are commonly used in ecologic research. Each gene is represented by two independently labeled PCR products and each product was arrayed in quadruplicate. We present a robust method of normalizing and analyzing ERs based on arbitrary thresholds and statistical criteria, and characterize a "norm of reaction" of ERs for 6 genes  with different ERs as determined across all analyzed arrays to provide a biologically-informed alternative to the use of arbitrary expression ratios in determining significance of expression. These gene-specific ERs and their variance (gene CV) were used to calculate array-based variances (array CV), which, in turn, were used to study the effects of array age, probe cDNA quantity and quality, and quality of spotted PCR products as estimates of technical variation. Cluster analysis and a Principal Component Analysis (PCA) were used to reveal associations among the transcriptional "imprints" of arrays hybridized with cDNA probes derived from mRNA from N. attenuata plants variously elicited and attacked by different herbivore species and from three congeners: N. quadrivalis, N. longiflora and N. clevelandii. Additionally, the PCA revealed the contribution of individual gene ERs to the associations among arrays.We describe an analysis of the sources of variation in ERs from 73 hybridized cDNA microarrays, each with 234 herbivory-elicited genes from the model ecological expression system, While the costs of 'boutique' array fabrication are rapidly declining, familiar methods for the analysis of the data they create are still missing. The case history illustrated here demonstrates the ease with which this powerful technology can be adapted to ecological research. The 'genomics revolution' has provided the information needed to analyze how a genome responds to the environment in the formation of the "transcriptome", the portion of the genome that is transcribed. Microarrays, which offer the ability to analyze the expression ratios (ERs) of thousands of genes simultaneously, represent one of many new tools produced by this effort. However, not all biological disciplines have benefited equally from this technology, and array technology has not been widely adopted by the ecological community for a number of reasons. The large genome-wide arrays are only available for select model organisms, which may not be appropriate for many ecological questions. Moreover, the complexity of their analysis and the costs of the available commercial software solutions prohibit their adoption by small research groups and constrain the number of biological experiments that can be conducted even by large, better-funded groups. 'Boutique' arrays – on which a smaller fraction of the transcriptome, typically representing a selection (hundreds) of genes specific to a class of genetic elements or response types – are readily created for a non-model organism at costs that are affordable for small research groups. However, the problems remain of how best to normalize and analyze array data. A large number of software solutions are available but no clear best solution has emerged -5.A recent review has examined the types of arrays as well as the ecological and evolutionary questions that can be addressed with microarrays . Here weNicotiana attenuata ; 2) both ER1 and ER2 were significantly different from 1 as evaluated by t-tests to control for ER-variance and ER-sample size. An arbitrary threshold was utilized for two reasons: first, to account for normalization errors, and second, to account for the fact that replicate data did not result from repeated hybridizations with the same RNAs but from repeated probe spotting.Because the arrays included both up- and down-regulated genes, the calculation of a microarray-specific normalization factor provided a valuable alternative to the use of external reference controls, which may or may not be influenced by the elicitation conditions ,20-22. TTo evaluate our criteria, we hybridized three arrays with the same cDNA pools and found that 210 of 234 genes (84%) had the same regulation identified by the criteria described above. Of the 30 genes that did not show consistent regulation between the two repeated hybridizations, 24 had the same direction in mean ER but did not meet the statistical requirements for a significant change. These 3 replicate arrays were located together in both the cluster analysis , the error structure on a non-log scale is presented (Fig. A cluster analysis of 35 arrays was performed based on Ward's method and the squared euclidian distance ,24. To eTo test for differences between the groups of different PCR qualities, Kruskal-Wallis ANOVA on Ranks was used. Test statistics and cluster analyses were performed with SPSS 11.0, PCA was carried out with the Canoco 4.5 package .MH carried out the analysis of the microarray data, KG supervised the molecular work. ITB conceived of and coordinated the project, and ITB and MH wrote the manuscript.


Quantitative proteomics is an emerging field that encompasses multiplexed measurement of many known proteins in groups of experimental samples in order to identify differences between groups. Antibody arrays are a novel technology that is increasingly being used for quantitative proteomics studies due to highly multiplexed content, scalability, matrix flexibility and economy of sample consumption. Key applications of antibody arrays in quantitative proteomics studies are identification of novel diagnostic assays, biomarker discovery in trials of new drugs, and validation of qualitative proteomics discoveries. These applications require performance benchmarking, standardization and specification.Six dual-antibody, sandwich immunoassay arrays that measure 170 serum or plasma proteins were developed and experimental procedures refined in more than thirty quantitative proteomics studies. This report provides detailed information and specification for manufacture, qualification, assay automation, performance, assay validation and data processing for antibody arrays in large scale quantitative proteomics studies.The present report describes development of first generation standards for antibody arrays in quantitative proteomics. Specifically, it describes the requirements of a comprehensive validation program to identify and minimize antibody cross reaction under highly multiplexed conditions; provides the rationale for the application of standardized statistical approaches to manage the data output of highly replicated assays; defines design requirements for controls to normalize sample replicate measurements; emphasizes the importance of stringent quality control testing of reagents and antibody microarrays; recommends the use of real-time monitors to evaluate sensitivity, dynamic range and platform precision; and presents survey procedures to reveal the significance of biomarker findings. Traditional immunoassay platforms have very limited multiplexing capability and high sample volume requirement. The development and application of high throughput, multiplex immunoassays that measure hundreds of known proteins in complex biological matrices, is becoming a significant tool for quantitative proteomics studies, diagnostic discovery and biomarker-assisted drug development [reviewed in -4]. Two Unique, general considerations in assembling multiplexed immunoassays include: Requirements for elimination of assay cross-reactivity; configuration of multi-analyte sensitivities; achievement of dynamic ranges appropriate for biological relevance when performed in diverse matrices and biological states; and optimization of reagent manufacturing and chip production to achieve acceptable reproducibility. In contrast to traditional monoplex enzyme-linked immunoassays, generally agreed specifications and standards for antibody microarrays have not yet been formulated. A number of recent articles have started to examine certain of these issues ,6,7.Microarray immunoassays performed on planar glass slides and employing signal enhancement with rolling circle amplification (RCA), have been developed by several groups and have demonstrated usefulness in measurements of temporal and dose-dependent changes in a variety of immunological model systems and human diseases ,2,8-16; -16; 8-16I. Analytes in an applied sample bind to capture antibodies immobilized on a silanized glass surface.II. Applied secondary biotinylated detector antibodies bind to captured analytes, creating a highly specific immune complex.III. Biotinylated detector antibodies bound to the immune complex are detected with a universal anti-biotin antibody. The latter is conjugated to primer oligonucleotides that are pre-annealed to a complementary circular oligonucleotide.IV. DNA polymerase extends the 3' ends of primers around the circles, resulting in long, single stranded RCA products that remain attached to the complex.The RCA product, composed of tandem DNA repeats complementary to the circle sequence, is detected by hybridization with cyanine 5 (Cy5)-labeled complementary oligonucleotides.The present report describes initial development of standardized operating procedures, quality controls and standards for microarray immunoassays performed on planar glass slides using signal enhancement with RCA. These metrics have been tested for use in generation of data with adequate sensitivity, reproducibility and assay performance for biomarker discovery -14,16, P, P16, PaTo demonstrate the feasibility of using a multiplex immunoassay system to measure protein levels in complex biological matrices, the performance of dual-antibody, sandwich immunoassay arrays performed on planar glass slides with RCA signal enhancement was evaluated for specificity, sensitivity, reproducibility and accuracy using standardized titrations, spiked biological matrices and clinical samples. Array performance was evaluated based on ability to: measure analytes across a broad dynamic range at sufficiently low coefficients of variation (CVs); detect proteins at levels requisite to capture biologically relevant expression differences; confirm reliability of methods to normalize data to minimize platform imprecision and demonstrate the utility of generating standard curves to convert analyte MFI (mean fluorescence intensity) data into mass unit information.An advantage of arrays is the ability to measure each analyte multiple times, enhancing precision. Capture antibody spots were printed in quadruplicate on planar glass slides providing redundancy of individual analyte measurements. Data redaction was applied to raw immunoassay data to improve data quality by eliminating outlier data points. Outliers were identified by employing two subsequent statistical approaches in a step-wise manner.First, the Bland-Altman plot was used. Bland-Altman plots are often used in DNA microarray analysis to identify differences and/or replicate outliers. This involves plotting the difference between the logarithm of intensities of two replicates (M) versus the average of logarithm of intensities (A) for each analyte within an individual array . Thus, there will be 6 MvA plots for each data set to reflect the 170 analytes positioned across 6 arrays. Each MvA plot will contain 3*Ns*Na points, where 3 reflects the number of possible unique pair wise combinations of the three replicates, Ns represents the number of samples and Na defines the number of analytes measured on a given array. An example of an MvA plot produced in a project comprising 150 clinical serum samples for Array 4 with 37 analytes is shown in Figure 2) is examined for each plot. Each plot contains Na data points, where Na reflects the number of analytes. Plots with R2 values <0.95 are examined to identify the cause of the poor correlation. We have identified two major sources of poor correlation: incorrect positioning of the capture grid during image quantitation and general aberrations in image spot quality. Assigning the specific source of low correlation is accomplished by tracing back to the image data. In the case of grid misplacement, suspect data images are re-quantified. Poor correlations due to aberrant spot morphology/intensity are manually examined and removed from data set. If the R2 value does not improve as a result of outlier removal, the replicate is redacted from the data set. The sample is considered passed if there are two replicates with R2 >= 0.95. The pass rate is defined as the number of passed samples divided by the total number of samples. A run of an array is considered to be passed if 85% of the samples have two or more passed replicates.The second step of data reduction involves a linear correlation analysis. Pair-wise correlation analysis is done between all replicates of individual sample. Figure 2 values < 0.95. This process reduces throughput of data redaction, particularly on large data sets. Table In our experience, applying the MvA statistical approach first, followed by the linear correlation analysis is an efficient process to identify outliers without compromising data throughput. Since, MvA plots can be generated quickly, it allows for relatively fast redaction of significant outliers using an objective semi-automated approach. In contrast, the sample correlation analysis is considerably more labor intensive and currently requires manual investigation of each scatter plot with RIn general, for data sets with more than 40 samples, outlier removal only demonstrated small improvements in reducing average CVs across all samples. The most significant impact of outlier removal is on improving reproducibility across the three replicates of the individual samples. In our experience, outlier removal has been shown to reduce individual sample replicate CVs by 2–3 fold. This effect is directly related to improving sample correlation pass rates by by 10–20%.Many systematic factors can modify spot intensity during the process of measurement. Normalization is the process of reducing the effects of systematic variation on spot intensity. Normalization in DNA microarrays typically involves adjusting distributional summaries of data  from each chip to common reference values. For example, one assumption could be that the average signal from each protein chip should be the same, as with DNA microarrays and the difference between replicate values is due to systematic variability in the measurement process. Unfortunately, the nature of protein antibody microarrays, configured with a multiplex of individual capture and detector antibodies, is more specialized and differentiated than that of a DNA microarray. Use of a single reference factor derived from a global value is not sufficiently refined to take into account the difference in platform configuration. In the current report, the organization of protein microarrays allows the measurement of up to 16 samples per slide (chip). This is very different from DNA microarrays where one chip represents the total collection of measured values for an individual sample.To accommodate the differences inherent to the platform, we have applied a normalization strategy based on the three major sources of technical variability observed in our system. The first type of variability relates to spot-to-spot differences observed between quadruplicate spots of the individual analytes printed within a sample well. The second level of variability can be described as the difference in measurements between wells within the same slide. The third element of variability represents the variability observed between sample wells compared across different slides. We found that slide-to-slide variability is the largest source of variation accounting for more than 70% of the total measurement imprecision (data not shown). Thus the goal of normalization is to reduce the imprecision of slide-to-slide measurement error since this represents the major source of platform variability.Normalization is performed using a system of standard controls to reduce the effect of slide-to-slide variability. A series of four standard control samples  are run in 4 wells of each slide. Each control sample represents a cocktail of the full repertoire of analytes for the given array tittered at 4 specific concentrations. The standards have been optimized at concentrations  to capture measurements across the linear range of detection for each analyte. The global average of total analyte signal for the four prepared controls is calculated across all slides run in a batch. An adjustment factor is created for each slide that reflects the difference between global intensity average for all slides and the individual intensity average based on the controls from the individual slide. The averaged pixel intensity of each spot on the slide is scaled by the adjustment factor.As an example, the average value of the adjustment factor was evaluated across a batch of 33 slides and found to have a value of 1.33+/- 0.47. The primary benefit of normalization was in reducing the replicate sample CVs. Figure A 15-point series of standardized titrations containing recombinant proteins diluted in buffer were used to evaluate platform precision. This assessment was used in the quality control of each slide lot prior to release, as well as within each client project to verify run-time analyte performance. Six replicates for each point were run in the quality control testing of each slide lot and six replicates of each point were run within each client study to generate standard curves. CVs were evaluated for each concentration of analyte across six slides. Average CVs were calculated for each analyte. Statistical summaries of CV distribution across all array 2 analytes using the standardized 15-point standard titration series are shown in Table A variance decomposition analysis was performed to reveal the extent to which platform error influenced the ability to identify biomarkers. The variance component assigned to platform error was typically found to be an order of magnitude lower than the average inter-individual variation. Figure The left panel of Figure A performance assessment of individual analytes was conducted to determine the utility of each analyte across multiple projects covering diverse disease areas. Each analyte was evaluated according to the percentage of clinical samples that fell within (W), below (B) or above (A) the linear range of detection. Tables ,7,8,9,10The development of an antibody array featuring 25–40 novel immunoassays requires extensive validation related to the comprehensive assessment of antibody cross reactivity, definition of analyte minimal detection limits (MDL) and establishing robust assay performance. Each antibody array must be validated for use with several matrices, since the latter may have different ambient analyte levels  or cross-reactivity profiles.Analyte sensitivity was assessed to identify analytes lacking adequate performance for retention on an array. Additional experiments were performed to determine the endogenous levels of each analyte. For analytes without previously reported biological values, the "0 × n" assays indicated the approximate ambient analyte level. Testing across multiple biological matrices was required, since different matrices affected the detection of analyte specific signals. The "0 × n" experiments also revealed the level of non-specific background which was influenced by the total concentration of antibody load in the detector mix. In our experience, certain plasma matrices were also more likely to generate high background when compared to matched serum samples. The impact of high generalized background is a reduced sample pass rate. When background was observed, the total detector antibody concentration could often be reduced to minimize background noise. Ultimately, a balance between reduction in background and enhancement of sensitivity was required to achieve maximal analyte performance in a mutiplex configuration.The results of the 1 × (n-1) assays identified analytes that demonstrated cross-reaction between the captured analyte and the complex detector mix prepared without the cognate detector antibody. Binding between the spiked analyte and the cognate capture that generated signal, indicated a cross-reaction to one or more non-cognate detector antibodies contained within the complex mix. In cases where non-cognate detector signal was observed, an additional series of experiments were run with the corresponding analyte tested against each of the individual detectors to identify the cross-reacting detector antibody. Since cross reaction is an additive process, the outcome of the cross reaction assessment allowed for adjustments to be made to achieve a balance between maximizing content with multiplexed array specificity. The 0 × (n-1) assays were run to provide a baseline of MFI values to compare to the results obtained in the 1 × (n-1). In addition, the 0 × (n-1) experiments also served to screen the various biological matrices for cross-reactivity with endogenous proteins.2 values obtained between the two conditions provided a measurement of the accuracy of the multiplexed system.Serum MDLs were typically found to be higher than buffer MDLs due to the presence of endogenous analyte, potential analyte-binding proteins present in the biological matrix and other possible matrix-related interferences. The assay conditions used to stress test the system under conditions of high analyte load were designed to identify cross-reaction thresholds for each of the individual analytes. MFI cut off values were used to identify significant increases in non-cognate signal that warranted removal of a feature from the array. The results provide a certain utility in predicting array performance under conditions where sample analyte concentrations exceed reported biological levels. Examples might include patient samples tested under diseased states, elevated analytes produced in stimulated cell culture supernatants or in samples exhibiting a strong drug response. The final validation involved measuring the accuracy of the multiplex assay when challenged with a high concentration of analyte. Figure Thirty years of widespread use of conventional, monoplex immunoassays has established firm benchmarks for performance in protein measurement. In the present paper, we have examined several, unique but general considerations in assembling multiplexed immunoassays with performance similar to conventional monoplex immunoassays. These include development of a comprehensive validation program to identify and minimize antibody cross reaction under highly multiplexed conditions; application of standardized statistical approaches for data handling for highly replicated assays; inclusion of standardized samples in each run to normalize sample replicate measurements; quality control of reagents and antibody microarrays; implementation of real-time monitors to evaluate sensitivity, dynamic range and platform precision; and initial procedures for identification of specific, significant immunoassay results in biomarker discovery projects involving clinical samples. Each of these will be discussed briefly.An array validation program represents the foundation of tests required to establish robust assay performance in a multiplexed environment. The most significant component in array validation is the comprehensive evaluation of cross reactivity. The vast majority of the ~5000 commonly available antibody pairs available today have not previously been evaluated for cross reactivity in a multiplexed environment. Therefore, the recommended program should include procedures that identify analytes demonstrating cross reactivity with immobilized capture antibodies as well as cross reaction that might manifest between the secondary detector antibody with a non-cognate analyte or non-specific binding to an immobilized capture agent.The performance of analytes in a multiplexed configuration should be benchmarked against the baseline, monoplex performance. This multiplexed immunoassay comparison with baseline performance, together with minimal standards for multiplexed cross-reactivity, permits determination of the practical, optimal number of array elements that can be successfully combined. In our experience, using dual-antibody, sandwich immunoassays, planar glass slides and RCA signal amplification, protein micorarrays can generally accommodate multiplexing of 25–35 analytes without an appreciable drop in individual analyte sensitivity or performance. Specifically, we have described development of six different dual-antibody sandwich immunoassay arrays, each containing 25–37 sandwich immunoassays. Since cross reactivity is an additive process, the ultimate goal is to achieve a balance between maximal multiplexing and monoplex-like performance. With exhaustive selection for antibodies without cross-reactivity in multiplexed format, it is possible to multiplex 50 sandwich immunoassays. However, this exercise is very expensive. In our experience, suspension arrays, alternative microarray surface substrates and attachment chemistries do not offer significant advantages in multiplexing while maintaining performance. We have not evaluated the impact of novel, affinity ligands on multiplexing.Additional array validation for cross-reactivity should include "stress-testing" under high analyte load to reflect conditions where analytes may be significantly over-expressed. In our experience, levels of induction of proteins in common biological matrices can be very large following drug administration or in disease states, and may induce cross-reactivity that is not observed in testing within normal biological analyte levels.Finally, array validation should be performed across all common sample matrices to examine effects on assay performance associated with endogenous analyte, matrix specific analyte binding proteins or other matrix-specific inhibitors. Absence of cross-reactivity for an immunoassay in one matrix does not always imply absence of cross-reactivity in others. The matrices for which the antibody microarrays described herein have been validated include isotonic buffers, serum, citrate plasma, heparin plasma, EDTA plasma, cell culture supernatants, amniotic fluid, sputum, and exhaled breath condensates. Several of the arrays have also been validated for use with ex vivo treated whole blood. EDTA plasma and ex vivo treated whole blood had higher levels of background signal and lower sample pass rates than other matrices.2> 0.95 range were eliminated. In order for data from sample replicates to pass and be admitted into the final data set, the overall sample replicate-to-replicate correlation for the 25–37 analytes of the array was required to have an R2> 0.95. Experience in multiplexed immunoassay measurements in samples across more than 30 research projects indicated the R2 value >0.95 to be routinely achievable and associated with high quality replicate data. In each project, the data lost through these two sequential redaction procedures was typically less than 5% of the total original data. An additional quality metric to assess the overall run performance was that at least two of the three replicates must have passed for 85% of the total samples. Runs falling short of this metric were failed and subject to repeat. The typical run fail rate was less than 3%.A significant advantage of array-based immunoassays is the ability to measure each analyte in a sample many times. Removal of outlier replicates is obligatory for microarray assays due to signal-related and morphology-based artefacts typically associated with dispersing small volumes of material on a solid substrate in a microarray format. Application of standardized statistical approaches for data redaction is superior to manual inspection and removal of outliers since operator-dependent subjectivity is minimized and throughput is greatly increased. The data redaction procedures described herein employed two, separate steps: Bland-Altman plots and linear correlation analysis. Bland-Altman plots were employed first and identified 99% confidence intervals for all collected data points. This enabled rapid identification and elimination of the outlying 1% of the data with minimal human intervention. This was determined to be an objective, reproducible redaction procedure that greatly reduced time and effort associated with the subsequent, second data redaction step of linear correlation analysis. Linear correlation analysis required performance of 3 replicate assays on each sample, and manual inspection of the series of 3 scatter plots generated from pair-wise correlations of these 3 sample replicates. Individual replicate points for each specific analyte that fell outside the RWithin-run controls were employed to account for the effects of systematic variation in replicate measurements. Variation was identified at three levels based on the unique configuration of the 16 sample well microarray chip. The lowest level of variability was observed between the quadruplicate spots of an individual analyte measured within a single sample well. The next level of variation was described as the difference between replicate analyte values measured in different wells located within the same slide. The highest level of variation was associated with measurements taken from a single sample applied to multiple wells positioned across different slides within a run. Since slide-to-slide variation demonstrated the highest system variation, a series of four controls were designed to minimize the impact on sample replicate measurements. The four controls contained all analytes for that array at four concentrations spread across the dynamic range. The four controls were run on every slide within a project and used to generate a global average of total analyte signal. Based on the global average, each individual slide was assigned an adjustment factor to compensate for the slide specific intensity bias. The analyte signal from each individual slide could then be scaled by the adjustment factor to normalize the intensity values between the sample replicates positioned across different slides. In addition, it is possible to use a blocking experimental design, intentionally positioning sample replicates across different slides and different slide locations to eliminate the potential for a slide-specific or location-specific intensity bias. An example of the latter might have been the well at the corner of a slide. Replicate measurements in conjunction with a mechanism to normalize systemic variation results in the production of high quality data required for maximal sensitivity in the identification of significant differences between samples in multiplexed immunoassays. An additional benefit of inclusion of standardized controls run across all slides of every project is the ability to standardize data, for example in mass units, and enable data comparisons between runs, between days and between projects. Such comparisons are necessary when projects constitute large numbers of samples or when it is desired to create a relational database of assay results. Our platform described herein, for example, can perform triplicate measurements on up to 200 samples in a single run.Quality control of approximately 1200 individual reagents is necessary in order to provide consistent performance of 170 immunoassays on the array platform described herein. These reagents, unfortunately, have widely different shelf life and storage conditions. Stringent quality control procedures specifying performance metrics associated with these reagents were required to achieve reproducible array performance across hundreds of slide lots and reagent sets. Each new lot of a given component was benchmarked to an earlier lot to verify performance. Analyte intensity, dose-response curve, LLD/ULD absolute values, dynamic range and background signal were evaluated in fuctional tests performed on all assay components. Historical performance was monitored by comparing running averages obtained from earlier lots to prevent performance change over time. These procedures were made practical by assembling cocktails of reagents for each step in an assay, dispensing these in single-use aliquots, establishing optimal storage conditions and shelf life, and performing regular  quality checks on aliquots. Implementation of such procedures required use of a laboratory information management system.The utility of integrating real-time platform performance monitors cannot be understated. Given the complex nature and potential instability of biological reagents associated with a multiplex antibody array, it is critical have a program in place to evaluate performance beyond the quality control release. Real time monitors measure performance of controls under conditions identical to the test samples and reflect a second level verification of assay performance. Our test system employed a series of monitors to capture precision metrics that would create a flag to review the data if the specifications were not met. The requirements included mean coefficients of variation of assay values for controls be less than 15% and for sample replicates be less than 25% for project samples run within a batch. Failure to achieve these metrics indicated a problem related to the performance of the manufactured slides and/or reagents or a technical failure associated with sample handling or assay execution. 15-point standardized titrations were also performed on 6 slides in every run in order to captured detail related to analyte dynamic range, LLQ/ULQ values, dose response behaviour, and background signal that provided a comprehensive assessment of real time platform performance. The detail of the performance assessment was included in final reports for each project to verify data quality and generate confidence in the data generated from a highly complex assay.Evaluating data generated from multiplexed immunoassays for utility in systematic identification of significant differences between samples, or "biomarker discovery", is an important step in understanding the true platform performance. One of the procedures that revealed the sensitivity of the platform for biomarker discovery was variance decomposition analysis for each project. Variance decomposition analysis examines the magnitude of individual components of platform variation and how they compare to analyte variation between samples or individuals. In our experience the platform error of the system described herein was generally an order of magnitude lower than the heterogeneity observed between samples or individuals of the same test group. The utility of this test is in revealing the extent to which platform error impacts the ability to discover moderate expression level differences between samples that are reflective of biological change. Platforms with lower precision will have less sensitivity for detection of relevant differences between samples and will discovery only a subset of the markers that would have been identified with a more precise system.Finally, a global performance assessment should be performed across multiple projects covering diverse disease areas to gain a solid understanding of the platform utility. An evaluation of this type can be used to identify assays that will not identify differences in expression between samples because they are not sufficiently sensitive, unable to generate sufficient dynamic range given the window of expression, or reveal high endogenous abundance producing assay saturation artefacts. In addition, specific assays that have appropriate sensitivity and dynamic range may be constitutively expressed and therefore poor biomarker candidate analytes for certain disease or treatment effect studies. This analysis may be used to direct efforts to continue to optimize the survey platform in order to generate the highest value in identifying biomarkers using a quantitative proteomic approach.Protein microarrays offer the ability to simultaneously survey multiple protein markers in an effort to develop expression profile changes across multiple protein analytes for potential use in diagnosis, prognosis, and measurement of therapeutic efficacy. The current report details certain minimal standards, use of which was found to be necessary to generate the requisite specificity, sensitivity and reproducibility to discover biomarkers. Results revealed that a multiplex system could be operated with high analyte specificity, adequate detection sensitivity and sufficiently broad dynamic range to capture expression differences across diverse disease and therapeutic areas.Raw soda-lime glass slides (1" × 3") prepared with a Teflon mask configured to provide 16 individual sample wells and an etched barcode for traceability were subjected to visual inspection to identify imperfections that might translate into printing and/or scanning artifacts. Slides with scratches, surface contamination or defects in the applied Teflon mask were identified through a visual examination using a long wavelength inspection lamp equipped with a 532 nm filter. The inspection also failed slides that did not meet stringent dimensional specifications, required for downstream printing and automated assay conditions.Slides passing the visual inspection were silanized with 3-cyanopropyltriethoxysilane according to procedures previously described . MeasureCapture antibodies prepared as previously described  were priEach printed array contained 256 spots representing 64 individual elements printed in quadruplicate. Each array contained 25–40 capture antibodies spread across production chips 1–6, generating a panel of 170 survey analytes. The balance of the elements was reserved for internal assay controls. Each printed array contained multiple copies of an element called BLANK, containing the components used in capture antibody preparation. Blanks were used to survey non-specific sample background within each well. Other printed controls included a series of biotinylated mouse IgG calibration standards to monitor RCA signal amplification and a third control that acted as a monitor for spot contamination resulting from carry-over between sequentially printed features.Preparations of standardized multiplex analyte titration series were manufactured using recombinant analytes diluted in buffer that covered the range from 12 pg/mL up to 81 ng/mL at 14 discrete points along with two zero analyte buffer blanks. These titration points were distributed among the sixteen available wells on a slide . Manual immersion washes were substituted with pipette delivered solutions finely tuned to control pipette tip aspiration and delivery position above printed slide wells and to carefully control liquid application and aspiration speeds to minimize disruption to the assembled immunosandwich complex. Incubation times were increased from 30 to 45 minutes for two of the assay steps  and the number and volume of washes between steps increased from 2 to 4–5 and from 20 uL to 30 uL respectively. A Tecan LS200 unit was used to scan the slides. Microarray images were quantified using image capture software (ImageGrabber) developed in-house.Frozen serum samples from over 800 clinical patients were thawed, centrifuged to remove particulate matter and mixed with 0.25 mg/ml Heteroblock (Omega), 0.25 mg/ml IIR (Bioreclamation) and 0.1% Tween-20 prior to the assay. Twenty microliters of serum was applied to each well.2(Rep1/Rep2) versus the average of the log intensities (A=log2((Rep1*Rep2))/2) for each of the replicates across all analytes. Patterns were visualized using fitted curves from robust local regression with applied visual cues to identify a 99% confidence interval. All outliers in the MvA plot outside of the interval  were automatically removed from analysis. The MvA scatter plots also allowed the user to highlight subsets of points on the plot and investigate patterns of intensity differences observed between replicate values. In cases where redaction of an entire replicate  was too stringent, individual spots could be removed using an in-house developed software tool (Terminator) to visually inspect aberrant data points. Data redaction using this method was performed on a limited basis to remove individual spot outliers with poor circularity, non-uniform pixel intensity or missing spots.Data points producing outlier events as a result of missing spots, spots with poor morphology, or printed features demonstrating high pixel outliers were removed using a combination of automated and manual methods. MvA plots, were generated by plotting the difference of the log intensities (M=log2) value cutoff. An ideal microarray, when compared to its identical replicate, would have a R2 value of 1. Any comparison producing values lower than the cutoff would result in at least one failed replicate. Individual sample correlations were generated by plotting analyte MFI values  from each replicate against the other replicates individually covering all combinations of replicates over the 25–37 analytes within the array. The R2 values obtained for the three plots were manually reviewed to identify failed sample replicates. Only sample replicates with R2 values >0.95 for replicates run within a day or R2 values >0.90 for replicates run across multiple days passed the correlation QC. A summary of the overall sample replicate pass rate monitored the number of failed replicates observed across each of the individual arrays. Project performance specifications required that >85% of all study samples had at least 2 reported replicates.As a quantitative QC measure, data review included a sample replicate correlation assessment with a predefined correlation coefficient  scale to generate within-slide titration curves. Linear regression coefficients (slope and intercept) were calculated between individual titration curves from each slide to generate an "average" titration curve. Calculated slope and intercept were used to transform averaged analyte values for each sample well. Data normalization was performed on the data set after outlier removal.A standardized precision assessment was performed on each run to monitor assay performance with respect to; within well variation (based on mean coefficient of variation (CV) observed between quadruplicate printed spots for all features across all sample wells of a project) and between-slide variation . The mean and median CVs with standard deviations were also metrics included in the precision assessment. The precision assessment was performed as a quality control using the 15-point titration calibrators to qualify new slide lots and generate quality metrics for each client project.Mean fluorescent intensity (MFI) values, on a logarithmic scale, from the 3 replicate measurements of the 15 point standard titration series were used to generate precision profiles to define the upper and lower limit of quantitation  within a predefined concentration CV . To do tThe VARCOMP procedure of SAS (SAS Institute), was used to obtain estimates of the variance components in a mixed model. The fixed effect variable represents variance observed in different groups in the study, for example groups of healthy versus diseased individuals. Random effects were represented by unique sample identifiers nested within levels of a fixed variable. This component of variance represented within-group differences associated with patient-to-patient variability or disease heterogeneity. The residual variance represented the platform error.Assay sensitivity was determined in two series of experiments. Initial testing to identify analyte cross-reactivity was performed in a configuration where all printed capture antibodies are surveyed in a "1 × n" format, representing a single recombinant analyte tested against all (n) detectors across multiple matrices . Capture antibodies that revealed binding to non-cognate antigens were removed or replaced with suitable alternatives. Analytes that demonstrated low signal across all matrices were removed. If signals were low in buffer, a comparison was made with signals obtained in serum or plasma to determine if the endogenous analyte level was detectable and determine if depressed signals were due to analyte instability in a non-biological matrix. Assessment of analyte endogenous level was performed using a "0 × n" format where unadulterated serum and plasma  are assayed with the full complement of detector antibodies for a given array.Two conditions were examined to study potential cross-reactivity between the complex detector antibody mixture and the immobilized capture analyte. The first condition included a 1 × (n-1) format in which 1 analyte was tested in the presence of all detectors minus the detector antibody specific to the added analyte (n-1). In the case of an array containing 40 printed features, 40 unique detector antibody cocktails are prepared containing 39 of the detector antibodies found in the complex mix, with each mix containing all but one of the 40 corresponding detectors. The 40 individual reaction mixes are added to specific arrays after the arrays were incubated with the antigen corresponding to the missing detector. The second condition represented the 0 × (n-1) format where no analyte was added in the presence of all detectors minus the detector specific to the analyte under examination. In each case the analytes were spiked in buffer, serum, and plasma  at a fixed analyte concentration of 50 ng/ml.Single analyte titrations were prepared in buffer, serum and plasma  to assign a minimum detection limit (MDL) for each analyte based on a 95% confidence interval above background. The format of the experiments included an n × n design, where all analytes were run in the presence of all detectors. Then, using a 1 × n format, where only one antigen was added to an assay containing all detectors (n), each analyte was tested at 0X, 10X, 50X, and 100X MDL across the same test matrices to identify non-cognate cross-reaction under high analyte load. Additional antigen titration experiments were run to compare the performance achieved in the presence of a single antigen (1 × n) to one in which all analytes were present (n × n).MFI: Mean Fluorescence Intensity.RCA: Rolling Circle Amplification.CV: Coefficient of Variation2: correlation coefficientRLLQ: lower limit of quantitationULQ: upper limit of quantitationMDL: minimal detection limitsSD: standard deviationL.T. Perlee, J. Christiansen, B. Grimwade, S. Lejnine, V. Tchernev and M. Sorette were employees of Molecular Staging, Inc. and D.D. Patel and S.F. Kingsmore have received consulting fees.RD oversaw the manufacturing of reagents, JC established standardized quality control testing procedures, MS implemented assay automation and oversaw all clinical testing. BG was responsible for data curation, SL performed statistical analysis. MM and WS contributed to array develoment. SFK, VTT and DDP were involved in study design, LTP in project execution and LTP, SL, JC and SFK in manuscript preparation.


An important goal of DNA microarray research is to develop tools to diagnose cancer more accurately based on the genetic profile of a tumor. There are several existing techniques in the literature for performing this type of diagnosis. Unfortunately, most of these techniques assume that different subtypes of cancer are already known to exist. Their utility is limited when such subtypes have not been previously identified. Although methods for identifying such subtypes exist, these methods do not work well for all datasets. It would be desirable to develop a procedure to find such subtypes that is applicable in a wide variety of circumstances. Even if no information is known about possible subtypes of a certain form of cancer, clinical information about the patients, such as their survival time, is often available. In this study, we develop some procedures that utilize both the gene expression data and the clinical data to identify subtypes of cancer and use this knowledge to diagnose future patients. These procedures were successfully applied to several publicly available datasets. We present diagnostic procedures that accurately predict the survival of future patients based on the gene expression profile and survival times of previous patients. This has the potential to be a powerful tool for diagnosing and treating cancer. Procedures that utilize both gene expression data and clinical data to identify subtypes of cancer can provide more accurate prognoses When a patient is diagnosed with cancer, various clinical parameters are used to assess the patient's risk profile. However, patients with a similar prognosis frequently respond very differently to the same treatment. This may occur because two apparently similar tumors are actually completely different diseases at the molecular level .The main example discussed in this paper concerns diffuse large B-cell lymphoma (DLBCL). This is the most common type of lymphoma in adults, and it can be treated by chemotherapy in only approximately 40% of patients . SeveralIf different subtypes of cancer are known to exist, there are a variety of existing techniques that can be used to identify which subtype is present in a given patient . HoweverThere are two main approaches in the literature to identify such subtypes. One approach uses unsupervised learning techniques, such as hierarchical clustering, to identify patient subgroups. This type of procedure is called “unsupervised” since it does not use any of the clinical information about the patient. The subgroups are identified using only the gene expression data.  For an overview of unsupervised learning techniques, see Hierarchical clustering  has succThe second approach to identifying subtypes of cancer is based exclusively on the clinical data. For example, patients can be assigned to a “low-risk” or a “high-risk” subgroup based on whether they were still alive or whether their tumor had metastasized after a certain amount of time. This approach has also been used successfully to develop procedures to diagnose patients .However, by dividing the patients into subgroups based on their survival times, the resulting subgroups may not be biologically meaningful. Suppose, for example, that there are two tumor cell types. Suppose further that patients with cell type 2 live slightly longer than patients with cell type 1 but that there is considerable overlap between the two groups . Assume To overcome these difficulties, we propose a novel procedure that combines both the gene expression data and the clinical data to identify cancer subtypes. The crux of the idea is to use the clinical data to identify a list of genes that correlate with the clinical variable of interest and then apply unsupervised clustering techniques to this subset of the genes.For instance, in many studies, the survival times of the patients are known even though no tumor subtypes have been identified . We can Once such a list of significant genes is compiled, there are several methods we can use to identify clinical subgroups. We can apply clustering techniques to identify subgroups of patients with similar expression profiles. Once such subgroups are identified, we can apply existing supervised learning techniques to classify future patients into the appropriate subgroup. In this study, we will use the “nearest shrunken centroids” procedure of Sometimes, however, a continuous predictor of survival is desired. We also describe a supervised version of principal components analysis that can be used to calculate a continuous risk score for a given patient and identify subtypes of cancer. The resulting predictor performs very well when applied to several published datasets.These two methods will produce satisfactory results in most datasets. However, we will describe some variations of these methods that can sometimes improve their performance. When we cluster a dataset using only a subset of the genes, it is important that we choose the correct subset of genes. Choosing the genes with the largest Cox scores is generally a good strategy, but this procedure sometimes selects some spurious genes. We will show that one can use partial least squares (PLS) to compute a “corrected” Cox score. Selecting the genes with the largest “corrected” Cox scores can produce better clusters than selecting genes with largest raw Cox scores. Additionally, we will describe two other continuous predictors of survival that we will call β˜ and γ^. For some problems, they are better predictors than the continuous predictor based on supervised principal components . These mThere are also related methods for predicting the survival of cancer patients using gene expression data. Moreover, in many applications, we would like to identify which genes are the best predictors of survival. These genes could be analyzed in the laboratory to attempt to discover how they influence survival. They could also be used to develop a diagnostic test based on immunostaining or reverse transcriptase PCR. For these applications, it is important to have a predictor of survival that is based on a small subset of the genes. This is another important advantage of our methods over existing methods.p genes for each patient. (Note that 𝓃 ≫ p.) We assume that there are several different types (classes) of cancer, each of which responds differently to treatment, and each of which is distinct at the molecular level. Therefore, given a set of 𝓃 patients with different classes of cancer, we wish to train a classifier that can diagnose which type of cancer a future patient has, given the expression levels of the patient's p genes. We will show that it is possible to identify such subgroups using the semi-supervised learning techniques described in the previous paragraph, and that identification of such subgroups can enable us to predict the clinical outcome of cancer more accurately.Our goal is to identify subtypes of cancer that are both clinically relevant and biologically meaningful. Suppose that we have 𝓃 patients, and we measure the expression level of As noted in the Introduction, we needed to assign each patient to a subgroup before we could apply nearest shrunken centroids. First, we applied an unsupervised 2-means clustering procedure to the DLBCL data of p-value of 0.416. Thus, conventional clustering techniques failed to identify subgroups that differed with respect to their survival times. Subgroups identified using hierarchical clustering also did not differ with respect to survival (data not shown).We compared the survival times of the two subgroups using a log-rank test. The log-rank test statistic was 0.7, with a corresponding p-value of the log-rank test was 0.03.We assigned each patient in the training data to either a “low-risk” or “high-risk” subgroup based on their survival time . A plot of the two resulting survival curves is shown in Recall that when we performed 2-means clustering on the patients in the test data using all 7,399 genes and used a log-rank test to compare the survival times of the patients in the two resulting clusters, the result was not significant. To test our new clustering method, we calculated the Cox scores of all 7,399 genes based on the 160 training observations and ranked the genes from largest to smallest based on their absolute Cox scores. We then clustered the 80 test observations using only the 25 top-scoring genes. This time, the log-rank statistic comparing the survival times of the two clusters was highly significant . This predictor is stronger than the discrete predictor shown in R2 = 0.08, likelihood ratio test statistic = 6.7, 1 d.f., p = 0.00966).Thus far, all of our examples have been based on the DLBCL data of Unfortunately, the expression levels of only 70 genes were available for the 292 patient dataset, making it difficult to test our methodology. However, we were able to apply our supervised principal components method. The expression levels of approximately 25,000 genes were available for the earlier study (consisting of 78 patients). After applying crossvalidation, we selected a model consisting of eight genes, five of which were included among the 70 genes in the larger dataset. Thus, we fit a supervised principal components model using these five genes and applied it to the dataset of 292 patients.R2 statistic for each model. R2 measures the percentage of the variation in survival time that is explained by the model. Thus, when comparing models, one would prefer the model with the larger R2 statistic.) We see that our supervised principal components method produced a stronger predictor of metastasis than the procedure described in The results are shown in R2 calculated using all 292 patients are inflated, since part of the dataset used to validate the model was also used to train the model. We include these results merely to demonstrate the greater predictive power of our methodology. Moreover, we repeated these calculations using only the 234 patients that were not included in the earlier study to ensure that our results were still valid.).We applied the methods discussed above to identify these subgroups  for the simulated dataset. This simulation was repeated ten times. The results are shown in R2 was zero for each iteration. If we had chosen a smaller value of the tuning parameter Δ, the procedure would have performed better, although not significantly better.) The continuous predictor based on supervised principal components performed nearly as well as the methods based on semi-supervised clustering.In the first simulation, we found that the fully supervised and the fully unsupervised methods produced much worse results than the semi-supervised methods. , so a diagnosis based on this procedure is likely to be inaccurate.Supervised clustering methods can overcome these problems. We have seen that if we selected significant genes prior to clustering the data, we could identify clusters that were clinically relevant. We have also seen how knowledge of these clusters could be used to diagnose future patients.This supervised clustering methodology is a useful prognostic tool. It is also easy to interpret. However, it has certain shortcomings as well. Recall our conceptual model shown in X as a predictor of patient survival. The chosen subset contained the genes with the largest Cox scores. This method could also be used to detect cancer subtypes, since the principal components will presumably capture the variation that exists between subtypes. It is also capable of identifying variation within these subtypes, which, as discussed above, cannot be identified using supervised clustering. We showed that this procedure could produce a stronger predictor of survival than the discrete predictor based on supervised clustering.One possible such predictor is our supervised principal components procedure. This procedure used the principal components of a subset of the expression matrix p = 0.05) for all four datasets, which was not true for any of the other methods. Finally, if we consider only discrete predictors of survival, our semi-supervised clustering methods performed better than the other models on at least three of the four datasets.We compared our methods to several previously published methods for predicting survival based on microarray data. In general, our methods performed significantly better than these existing methods. In particular, our supervised principal components method gave the best results on three of the four datasets.  Furthermore, each of our proposed methods was a significant predictor of survival  for a given patient is essential to treat cancer successfully. If the risk of metastasis is high, the cancer must be treated aggressively; if the risk is low, milder forms of treatment can be used. Using DNA microarrays, researchers have successfully identified subtypes of cancer that can be used to assess a patient's risk profile. Our results show that semi-supervised learning methods can identify these subtypes of cancer and predict patient survival better than existing methods. Thus, we believe they can be a powerful tool for diagnosing and treating cancer and other genetic diseases.The nearest shrunken centroids procedure calculates the mean expression of each gene within each class. Then it shrinks these centroids toward the overall mean for that gene by a fixed quantity, Δ. Diagonal linear discriminant analysis (LDA) is then applied to the genes that survive the thresholding. Details are given in We created two classes by cutting the survival times at the median survival time (2.8 y). Any patient who lived longer than 2.8 y was considered to be a “low-risk” patient, and any patient that lived less than 2.8 y was considered to be a “high-risk” patient. In this manner, we assigned a class label to each observation in the training data.Unfortunately, many of the patients' survival times were censored, meaning that the individual left the study before the study was completed. When this occurs, we do not know how long the patient survived; we only know how long the patient remained in the study prior to being lost to follow-up.If an observation is censored, we may not know to which class it belongs. For example, suppose that the median survival time is 2.8 y, but that a patient left the study after 1.7 y. If the patient died in the interval between 1.7 y and 2.8 y, then the patient should be assigned to the “high-risk” group. Otherwise, the patient should be assigned to the “low-risk” group. However, there is no way to determine which possibility is correct.T denote the survival time of this patient. Then, using the Kaplan-Meier curve, we can estimate p(T>50) and p(T>20). Then we can estimate p(T>50|T>20) as follows:Based on the Kaplan-Meier survival curve for all the patients, we can estimate the probability that a censored case survives a specified length of time . For exaand, of course,In this manner, we can estimate the probability that each censored observation belongs to the “low-risk” and “high-risk” classes, respectively.However, it is still unclear how we would train our classifier based on this information. Nearest shrunken centroids is a modified version of LDA. It is described in detail in xi}yi}g represent the number of discrete classes to which the yis may belong.  When we perform LDA when all of the yis are known, the problem is to fit the mixture modelLet {i is a Gaussian density function, and the θis correspond to the mean of the observations in each class. The πis correspond to “prior” probabilities that an observation belongs to class 𝒾.) In this case, we must fit this model on the basis of the classified (uncensored) training data, which we denote by t, and the unclassified (censored) feature vectors x𝒿 , which we denote by t𝓊. ′ denotes the vector of all unknown parameters.) as missing data. It turns out to be very simple in the case of LDA. The E-step is effected here simply by replacing each unobserved indicator variable 𝓏ij by its expectation conditional on xj. That is, 𝓏ij is replaced by the estimate of the posterior probability that the 𝒿th entity with feature vector xj belongs to Gi  . We takei and μi in the M-step are equally simple:The estimates of πandwherei is the posterior probability that the 𝒿th entity with feature vector xj belongs to Gi, or, in other words,In these expressions, τWe continue these imputations until the algorithm converges. In practice, one imputation seems to be sufficient for most problems, since each imputation is computationally intensive, and additional imputations did not seem to change the results significantly.We calculated the Cox scores of each gene based on the 160 training observations, and obtained a list of the most significant genes. Then we performed 2-means clustering on these 160 observations using the genes with the largest absolute Cox scores and obtained two subgroups. We repeated this procedure multiple times with different numbers of genes. For each such clustering, we trained a nearest shrunken centroid classifier to assign future patients to one subgroup or the other and examined the crossvalidation error rate.G of possible values of Γ. (2) Let pmin = 1 and emin = 1. (3) For each Γ in G, do the following: (4) Perform k-means clustering using only those genes with absolute Cox scores greater than Γ. (5) Perform a log-rank test to test the hypothesis that the k clusters have different survival rates. Call the p-value of this test p. (6) If p≥pmin, then return to step 3. (7) Fit a nearest shrunken centroids model based on the clusters obtained in step 3. Calculate the minimum crossvalidation error rate across all values of the shrinkage parameter, and call it e. (8) If e<emin, then let Γbest = Γ, and return to step 3. Otherwise return to step 3 without changing the value of Γbest. The optimal value of Γ is taken to be the value of Γbest when this procedure terminates.The problem of choosing the number of genes on which to perform the clustering is more complicated than it appears. The obvious way to choose the optimal number of genes on which to cluster is to simply minimize the crossvalidation error rate of the nearest shrunken centroids model based on the clustering. This works up to a certain point. It is possible that the clustering procedure will identify a cluster that is unrelated to survival.  Thus, we needed to build a safeguard against this possibility into our procedure. After performing clustering based on a given set of high-scoring genes, we performed a log-rank test to determine if the resulting clusters differed with respect to survival. If they did not, the clustering was discarded without further analysis. An outline of the procedure follows: (1) Choose a set Several comments about this procedure are in order. First, note that we did not recalculate the Cox scores at each fold of the crossvalidation procedure. We calculated them only once, using all of the patients in the dataset. There are several reasons for doing this. Recalculating the Cox scores at each fold would be extremely expensive computationally. Moreover, we found that the Cox score of a given gene varied depending on the number of patients (and which patients) we included in the model. Thus, if a given value of Γ produced a low crossvalidation error rate, there was no guarantee that a model based on the full dataset using this value of Γ would produce good results, since the model based on the full dataset may use a different list of genes. Other studies have found that using the entire dataset to produce a “significant gene list” prior to performing crossvalidation can produce more accurate predictions .G was left unspecified in the procedure. The choice of which (and how many) possible values of Γ to include in G depends on the problem at hand, as well as the computational power available. As a default, we recommend trying 100 evenly spaced values of Γ between the 90th percentile of the Cox scores and the maximum of the Cox scores. However, the optimal Γbest varies greatly from dataset to dataset, so we recommend trying several different forms of G if adequate computing power exists.Also, the set p-value of the log-rank test after performing the original clustering, we insisted not only that the p-value be significant, but also that it be lower than the best p-value obtained thus far. The reasons for this are twofold. First, experience suggests that if a given set of genes produces a good clustering on the training data , then it is likely to produce a good clustering on the test data.  Moreover, this speeds up the algorithm substantially. Calculating the nearest shrunken centroids crossvalidation error rate for a given clustering is the slowest part of the procedure; the time required to perform the clustering and calculate the log-rank statistic is insignificant in comparison. Thus, by only considering clusterings which produce a log-rank statistic with a small p-value, we allow the set G to be much larger than would be feasible otherwise.Furthermore, note that when we calculated the k was unspecified in the procedure. We have experimented with some algorithms to choose the value of k automatically, but without success. If possible, we recommend that the value of k be chosen based on prior biological knowledge.  If this is not possible, we recommend trying several different small values of k and choosing the one that gives the best results. Finally, the number of clusters X be the p×n matrix of expression values, for p genes and n patients. Let xij denote the expression level of the 𝒾th gene in the 𝒿th patient. Assume that each patient has one of two possible underlying tumor types. Without loss of generality, assume that patients 1, …,m have tumor type 1, and that patients m + 1,…,n have tumor type 2. Then assume that the genetic profiles of the two tumor types are distinct from one another, which is equivalent to assuming that the joint distribution of  is different for 1 ≤ 𝒿 ≤ 𝓂 than it is for 𝓂 + 1 ≤ 𝒿 ≤ 𝓃. Thus, if we choose constants {ai}jxija will be different for 1 ≤ 𝒿 ≤ 𝓂 than it is for 𝓂 + 1 ≤ 𝒿 ≤ 𝓃.  Hence, assuming that variation in gene expression accounts for variation in survival, we would expect that TXu1 captures a large percentage of the variation in survival.  For each Γ in G, split the training data into k random partitions . For most problems (and for the rest of this discussion), we can let k = 10. (3) For each crossvalidation fold, take a singular value decomposition of X, leaving out one of the 10 partitions for validation purposes. Use only those genes with absolute Cox scores greater than Γ. (4) Calculate υ^ for the 10% of the data that was withheld, as described above. (5) Fit a Cox proportional hazards model to υ^, and calculate the chi-square statistic for the log-rank test associated with this model. Denote the chi-square statistic for the 𝒾th crossvalidation fold by 𝓌i. (6) Average the 𝓌is over the 10 crossvalidation folds. Call this average 𝓌Γ. (7) If 𝓌Γ is greater than the value of 𝓌Γ∗, then let Γ∗ = Γ and 𝓌Γ∗ = 𝓌Γ. (8) Return to step 2. The set G is left unspecified in the procedure. As a default, we recommend trying 30 evenly spaced values of Γ between the 90th percentile of the Cox scores and the maximum of the Cox scores, although this recommendation is somewhat arbitrary.To choose the optimal value of Γ, we employ the following procedure: (1) Choose a set V (rather than simply taking the first column of V). Suppose we wish to find a predictor based on the first k columns of V. We can perform the following procedure: (1) Let X denote the training data. Take the singular value decomposition of X = TUDV as described above (after selecting an appropriate subset of the genes). (2) Fit a Cox proportional hazards model using the first k columns of V as predictors. (3) Calculate the matrix V^I for the test data using k columns of V^I using the Cox regression coefficients obtained in step 2. Use the resulting sum as a continuous predictor of survival.In some cases, we can improve the predictive power of our model by taking a linear combination of several columns of http://cran.r-project.org/. R source code for the procedures described in this paper are available from the authors upon request .Click here for additional data file.Data S2This file contains R functions for implementing the procedures we have described in our study.(6 KB TXT).Click here for additional data file.Data S3This file contains the R source code that we used to perform the first simulation study in our paper.(31 KB TXT).Click here for additional data file.Data S4This file contains the R source code that we used to perform the second simulation study in our paper.(39 KB TXT).Click here for additional data file.Dataset S1The gene expression data for the breast cancer study of (2.9 MB CSV).Click here for additional data file.Dataset S2The names of each of the 4,751 genes in the study of (74 KB CSV).Click here for additional data file.Dataset S3The clinical data for the study of (1 KB CSV).Click here for additional data file.Dataset S4The gene expression data for the 70 genes in the breast cancer study of (141 KB CSV).Click here for additional data file.Dataset S5The names of the 70 genes in the study of (1 KB CSV).Click here for additional data file.Dataset S6A single column that is 1 if the patient was included in the earlier study (that of (1 KB CSV).Click here for additional data file.Dataset S7The clinical data for the study of (5 KB CSV).Click here for additional data file.Dataset S8The gene expression data for the DLBCL study of (24.38 MB CSV).Click here for additional data file.Dataset S9The clinical data for the study of (2 KB CSV).Click here for additional data file.Dataset S10This is the gene expression data for the lung cancer dataset of (5.39 MB TXT).Click here for additional data file.Dataset S11This is the clinical data for the lung cancer dataset of (1 KB TXT).Click here for additional data file.Dataset S12This is the gene expression data for the AML dataset of (9.96 MB TXT).Click here for additional data file.Dataset S13This is the clinical data for the AML dataset of (1 KB TXT).Click here for additional data file.Figure S1(8.26 MB TIFF).Click here for additional data file.Figure S2(8.33 MB TIFF).Click here for additional data file.Figure S3(8.42 MB TIFF).Click here for additional data file.Protocol S1(28 KB TEX).Click here for additional data file.


Microarray experiments are becoming a powerful tool for clinical diagnosis, as they have the potential to discover gene expression patterns that are characteristic for a particular disease. To date, this problem has received most attention in the context of cancer research, especially in tumor classification. Various feature selection methods and classifier design strategies also have been generally used and compared. However, most published articles on tumor classification have applied a certain technique to a certain dataset, and recently several researchers compared these techniques based on several public datasets. But, it has been verified that differently selected features reflect different aspects of the dataset and some selected features can obtain better solutions on some certain problems. At the same time, faced with a large amount of microarray data with little knowledge, it is difficult to find the intrinsic characteristics using traditional methods. In this paper, we attempt to introduce a combinational feature selection method in conjunction with ensemble neural networks to generally improve the accuracy and robustness of sample classification.We validate our new method on several recent publicly available datasets both with predictive accuracy of testing samples and through cross validation. Compared with the best performance of other current methods, remarkably improved results can be obtained using our new strategy on a wide range of different datasets.Thus, we conclude that our methods can obtain more information in microarray data to get more accurate classification and also can help to extract the latent marker genes of the diseases for better diagnosis and treatment. With the successful completion of the Human Genome Project (HGP), we are entering the post genomic era. Facing mass amounts of data, traditional biological experiments and data analysis techniques encounter great challenges. In this situation, cDNA microarrays and high-density oligonucleotide chips are novel biotechnologies as global (genome-wide or system-wide) experimental approaches that are effectively used in systematical analysis of large-scale genome data. In recent years, with its ability to measure simultaneously the activities and interactions of thousands of genes, microarray promises new insights into the mechanisms of living systems and is attracting more and more interest for solving scientific problems and in industrial applications. Meanwhile, further biological and medical research also promoted the development and application of microarray.Typical issues addressed by microarray experiments include two main aspects: finding co-regulated genes for classification based on different cell-type , stage-sMicroarray experiments usually generate large datasets with expression values for thousands of genes 2000~20 000) but not more than a few dozens samples (20~80). Thus, very accurate classification of tissue samples in such high-dimensional problems is difficult, but often crucial, for successful diagnosis and treatment. Several comprehensively comparative and improved methods have been proposed recently 00~20 000. In this, so we simplify this step and only use all datasets exactly as we found them in their transformed data.The flowchart of our method can be seen in Figure Due to the characteristic of small sample numbers in microarray data, in order to improve the accuracy, robustness and generalization of issue classification, we apply bootstrap mechanism to resample 100 iterations. During each iteration, we input the resample training data into three cooperative and competitive neural networks, and then by averaging their decisions, the neural network set can output their discrimination. From Figure After completing the 100 iterations, we can get 100 individual answers about the problem. In this situation, how to combine these answers into one more precise result is still a problem. Here, we simply use majority voting to combine the result and then give the ultimate solution about this classification problem. As noted above, here we adopt the soft-voting mechanism, that is, we can combine the confidence of each net. All the implementations of our framework were written in Matlab, using the hardware platform of a PC running 2.4 GHz.. We adopted their transformed data format for further research. All datasets we used can be reduced to three categories: binary class with testing samples, binary class without testing samples and multiple class problem. Here we classify samples into binary class with testing samples and without testing samples just according to the reference authors for each dataset. One important reason is that in this way we can easily compare our result with others based on the same training and testing sets. These datasets are shown in Table In this section, simple illustrations of the datasets we used in this paper for exploring the performance of our classification are given. The datasets in our paper have been downloaded from the following website: We use the three datasets below as the example of the first category, for which performance of our classification can be tested using the error ratio of testing samples.The training dataset consists of 38 bone marrow samples (27 ALL and 11 AML), with 7129 probes from 6817 human genes. Also, 34 samples testing data is provided, with 20 ALL and 14 AML.More information and raw data can be found in Golub et al. .The dataset can be reduced to the problem of classification between malignant pleural mesothelioma (MPM) and adenocarcinoma (ADCA) of the lung. The training set contains 32 tissue samples, which consists of 16 MPM and 16 ADCA and the testing samples are constitutive of 15 MPM and 134 ADCA. Each sample is described by 12533 genes. More information about this dataset can be found in Gordon et al. .For the prostate cancer dataset, detailed explanation and raw data is available in Singh et al. . This daAnother three recently popular datasets have been used as the representative of the second category. Using these kinds of datasets, we apply cross-validation to validate our classification performance.This dataset is used for discriminating distinct types of diffuse large B-cell lymphoma (DLBCL) using gene expression data. There are 47 samples, 24 of them are from "germinal canter B-like" group while the rest 23 are form "activated B-like" group and each sample can be described by 4026 genes. More detailed explanation can be found in Alizadeh et al. .. More information can be found in Petricoin et al. [The goal of this significant experiment is to identify proteomic patterns in serum that distinguish ovarian cancer from non-cancer. The proteomic spectra were generated by mass spectroscopy and the dataset provided here is 6-19-02, which includes 91 controls  and 162 ovarian cancers with 15154 molecular mass / charge (M/Z) identities. Here we use the transformed normalization data in The dataset Contains 62 samples collected from colon-cancer patients. Among them, 40 tumor biopsies are from tumors (labelled as "negative") and 22 normal (labelled as "positive") biopsies are from healthy parts of the colons of the same patients. Two thousand out of around 6500 genes were selected based on the confidence in the measured expression levels. Raw data and more information can be found in Alon et al. .Finally, we can generalize our method from binary class to multi-class problems. In this paper, we evaluate the performance using the dataset below.. More information can be seen in Armstrong et al. [This dataset contains training data consisting of 57 leukemia samples  and testing data consisting of 4 ALL, 3 MLL and 8 AML samples. We adopted the transformed data from g et al. .First we primarily focus on the binary class problem. Because most of problems can be reduced to binary class problems, such as diseased vs. normal, survival vs. lethal, two opposite subtypes of some diseases and so on. Finally we generalize our classifier to multi-class application. In this paper, we evaluate the performance of different classification methods using predictive accuracy, which can be defined as:TN1,TN2,…,TNn respectively denote the correct classification numbers of the samples belonging to a corresponding class; totalnum represents total sample numbers.Here, For the first category of the datasets, we evaluate the performance of our classifiers using predictive accuracy of testing samples compared with the best performance of the current available methods. In this paper we use bagging to resample just as Tan and Gilbert , and we For the most popularly used AML-ALL leukemia dataset, to our knowledge, the best classifiers of this dataset can be found in -27, whicIn the same way, we trained our ensemble of neural networks using 32 training sets of lung cancer and then predicted the 149 separate testing sets still with 0 error number. And three (1:2) testing error numbers can be reached using methods by Li et al. , which iFor the third prostate cancer dataset, after training the classifier using 102 training sets, only one wrong classification can happen using our ensemble neural networks to predict the 34 separate testing samples. We did not find a more accurate classification result except for the bagged decision trees in  based onIn order to further validate the effectiveness of predictive accuracy, we also performed the leave-one-out cross-validation (LOOCV) respectively only on the above three training samples. We also obtained the 100% accuracy both on the AML-ALL leukemia dataset and the lung cancer dataset, which are the same results as using individual testing samples. At the same time, 96.08% accuracy can be got based on the prostate cancer dataset, which is a little lower than using individual testing samples. For the purpose of comparison, we also list these results in Table From the results of the above three testing datasets, we can also see that many different classifiers obtain the best results when they concern some certain dataset, but there is still no general best strategy for tumor classification problems based on a wide range of different datasets. Furthermore, from Figure Without separate testing samples, we cannot evaluate the performance of our classifiers with the predictive accuracy of testing samples in the same way as above. Many performance evaluation methods have been proposed, of which various cross validations are most popularly used, such as 3-fold cross validation, 10-fold cross validation, leave-one-out cross-validation (LOOCV), and others. Here, we used the leave-one-out cross validation (LOOCV) to evaluate the performance of ours based on these available datasets. For further comparison with recent published methods based on the same datasets, we also perform 10-fold cross validation just as they used in their research. In Table In the first data column of Table 2 feature selection heuristic and support vector machine (SVM). In our method, only 75 features in total were used, and 99.21% and 98.82% accuracy was obtained respectively by LOOCV and 10-fold cross validation. Thus, we think that our method is comparable to theirs to some extent.For the ovarian dataset we used, Liu et al.  reportedFor the colon tumor dataset, we found that 85.48% predictive accuracy is the best classification result obtained in Dettling et al. , where tFrom the results of the above three datasets, we can see that our method is better, or at least comparable to current other best methods. Also, we need to note that these named best methods can get the best results based on certain datasets but may get worse results based on other datasets ; however, general performance improvement can be obtained using our method.Finally, we generalize our method from binary class to multi-class problems. After minor adjustments to the corresponding parameters of our framework, we obtained 100% classification result of the above-mentioned multi-class dataset – MLL_Leukemia dataset, that is, 4 ALL, 3 MLL, 8 AML of 15 test data can be predicted correctly. Similarly, 100% accuracy also was obtained by Li et al. . In thisIn this paper, we introduce a combinational feature selection and ensemble neural network method for the classification of gene expression data. On a wide range of recently published datasets, our method performs better, or is at least comparable to, the current best methods of our knowledge. As a further test, we randomly selected genes of the same amount as the feature instead of any of the three individual selected features in our research and then used the ensemble neural networks based on these features for classification again. The apparently worse discrimination power can be seen in this strategy. Moreover, we also used the output of a unitary network based on all the same features as the ultimate classification result and the result was also worse than ours. Thus, we believe such remarkable performance improvements of our method are due to the fact that our combinational feature selection mechanism induced more useful information for discrimination, and the ensemble neural network framework improved the stability, robustness and generalization of learning.We performed simple majority voting mechanism to combine the individual networks produced by bagging and got a more accurate solution. The advantage of the ensemble is to reduce the variance, or instability of the neural network, and avoid the error surface of neural network training being trapped into local minima. The ensemble model tends to cancel the noise part as it varies among the ensemble members, and it tends to retain the fitting to the regularities of the data. In this paper, our ensemble neural network model has 100 members; However, further research is needed to determine how many members working together can reach the best performance.In this paper, we focused on classification problem, so we didn't give a detailed analysis about how the importance of each different gene we select and the interaction between them influenced the diseases, which is a very important issue for application and will be researched in our future work.Note that the only drawback of our approach is the problem of increasing computational complexity and the fact that it consumes a little more time than others. However, considering the lost caused by wrong prognosis or diagnosis of disease, we believe that the remarkable improvement in corresponding accuracy deserve these costs.By aggregating various information and ensemble neural networks, we reached a more accurate classification decision based on several datasets. We think that making full use of all available information will more clearly elucidate the latent mechanisms of many diseases. For example, we can combine various imaging techniques, such as CT, MRI, PET and others, which can detect the change of phenotype for the corresponding disease, with microarray data for further research. In this way, we can recognize the nature of various life phenomena both from macro and micro viewpoints. Also, we can retrieve the information of genes that are used in microarray, such as gene functions and gene locations. In this way, we can make use of prior knowledge combined with the microarray data for further research.Feature selection is one of the most important issues in classification, which is a transformation process of observations in order to obtain the best pathway for getting to the optimal solution. At the same time, it can reduce the complexity of the data to make it more comprehensible. It is particularly relevant for microarray datasets with thousands of features because it has been reported that many diseases, especially tumors, have never been caused by a single gene mutation but are the result of a series of gene changes. Such genes are highly relevant to the studied phenomena of diseases. On the other hand, the expression levels of many other genes may be irrelevant to the distinction between tissue classes. We can say that the extraction and selection of features determine the ultimate performance of classifiers. Both for cost and for biological insight, making full use of the most informative genes and finding small feature sets with high classification accuracy are very essential. At the same time, highly informative genes that are part of known biochemical pathways give insights into the processes that underlie the differences between classes, and those of unknown function suggest new research directions.Some classifiers, such as trees, perform automatic feature selection and are relatively insensitive to the variable selection scheme, but most classifiers need to perform feature selection first. So far, various feature selection schemes have been used in microarray data analysis, such as the most popular method of selecting the top-ranked genes based on various different scores  ,22,25,29Just as we alleged above, in such a high-dimension space, finding accurate and significant features (genes) is very essential for classification, for cost savings and for biological insights. However, it has been verified that differently selected features reflect different aspects of the dataset and some selected features can obtain better solutions on certain problems. This is because one feature selection mechanism corresponds to one different artificial hypothesis, but which hypothesis is most near to the true hypothesis on a special problem is unknown to us. Here, we propose combining the above mentioned several feature selection methods to reflect different profiles of samples in order to obtain more useful information for classification and to produce a good approximation to true hypothesis by averaging the different hypotheses. In this paper, we select features using wilcxon's ranksum test  to get tAn Artificial Neural Network (ANN) is an information-processing paradigm that is modeled on biological nervous system, which is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems. In fact, since the basic model was proposed, various improved algorithms and theories have already been successfully applied in many fields. Because neural networks are best at identifying patterns or trends in a large amount of data with little theory, they are well suited for prediction or forecasting needs. That's just the case for microarray data. However, instability and little intrinsic knowledge of neural networks are obstacles to its further generalized application in some specific problems. Here we ensemble multiple networks in an attempt to solve this problem to some extent.Since multi-net systems were introduced by Sharkey in 1996 , the comGenerally speaking, in neural networks ensemble, two problems need to be resolved: how to generate the individual network and how to combine them together. Using bootstrap or boosting resample mechanism to obtain the individual network is the most popular method to solve the first problem. Bootstrap is the most popular resample mechanisam of sampling with replacement, therefore some observations are duplicated and some are omitted. Boosting means to boost a "weak" learning algorithm into a "strong" learning algorithm. Their differences are that using bootstrap can resample uniformly and can get the individual network immediately, but boosting weights every sample in each iteration and must generate the individuals in sequence. Several recent published papers claimed that adaboost-the basic boosting algorithm is not fit to microarray data analysis ,26, and Considering the complexity and noise of microarray data, it's very difficult to get a perfect solution using a unitary neural network based on some certain selected features. At the same time, accurately extracting and selecting the most informative genes is also very difficult using individual available methods. Thus, in this paper we attempt to combine multiple modes of information available from gene expression data using neural networks ensemble to get a better solution. We presented a cooperative and competitive neural network system that each of nets has the same architecture and topology, and each can respectively learn to classify a set of patterns based on partial information of the patterns and then by combining their classification results we can get a more precise result. The detailed framework of combing various features and neural networks ensemble and its implementation methods are discussed below.BL carried out the design of the method and performed the related analysis. QC participated in discussions of algorithms and manuscript preparation. TJ and SM instructed the whole study. All authors read and approved the final manuscript.