Identifying differentially expressed genes (DEG) is a fundamental step in studies that perform genome wide expression profiling. Typically, DEG are identified by univariate approaches such as Significance Analysis of Microarrays (SAM) or Linear Models for Microarray Data (LIMMA) for processing cDNA microarrays, and differential gene expression analysis based on the negative binomial distribution (DESeq) or Empirical analysis of Digital Gene Expression data in R (edgeR) for RNA-seq profiling.Here we present a new geometrical multivariate approach to identify DEG called the Characteristic Direction. We demonstrate that the Characteristic Direction method is significantly more sensitive than existing methods for identifying DEG in the context of transcription factor (TF) and drug perturbation responses over a large number of microarray experiments. We also benchmarked the Characteristic Direction method using synthetic data, as well as RNA-Seq data. A large collection of microarray expression data from TF perturbations (73 experiments) and drug perturbations (130 experiments) extracted from the Gene Expression Omnibus (GEO), as well as an RNA-Seq study that profiled genome-wide gene expression and STAT3 DNA binding in two subtypes of diffuse large B-cell Lymphoma, were used for benchmarking the method using real data. ChIP-Seq data identifying DNA binding sites of the perturbed TFs, as well as known drug targets of the perturbing drugs, were used as prior knowledge silver-standard for validation. In all cases the Characteristic Direction DEG calling method outperformed other methods. We find that when drugs are applied to cells in various contexts, the proteins that interact with the drug-targets are differentially expressed and more of the corresponding genes are discovered by the Characteristic Direction method. In addition, we show that the Characteristic Direction conceptualization can be used to perform improved gene set enrichment analyses when compared with the gene-set enrichment analysis (GSEA) and the hypergeometric test.http://www.maayanlab.net/CD.The application of the Characteristic Direction method may shed new light on relevant biological mechanisms that would have remained undiscovered by the current state-of-the-art DEG methods. The method is freely accessible via various open source code implementations using four popular programming languages: R, Python, MATLAB and Mathematica, all available at:  Genome-wide transcriptional profiling, the parallel measurement of the expression of tens of thousands of genes, is a powerful tool which, for example, aids in the development of clinical biomarkers for disease diagnosis, reveals the heterogeneity of histologically identical cancers, and sheds light on diverse biological mechanisms. After estimating the relative or absolute expression level of all transcripts, the next step is to test statistical hypotheses . TypicalGenes do not function in isolation but are part of a complex regulatory and functional network, and this can be reflected in the significant observed correlations between their expression levels. However, the most widely used methods for identifying DEG are univariate; typically tests are performed gene-by-gene without regarding gene-gene statistical dependencies. The fold-change, an early approach that is not recommended by statisticians but still popular among experimental biologists due to its simplicity, does not take into account the variance which arises from biological and experimental sources, and as such the fold-change measure does not offer any estimate of confidence ,3. BecauT2 test, which is a multivariate generalization of Welsh’s t-test. However, these approaches remain little-used because they are sensitive to the fact that typically microarray or RNA-Seq gene expression profiles have fewer samples than genes. A small sample size compared to the dimensionality of the measured variables brings difficulties to the analysis [There have been a number of attempts to apply multivariate analyses to identify DEG ,9. For eanalysis . A signianalysis ,12,13. Aanalysis -19. In aanalysis ,20.There are currently two main principle technologies to perform whole-genome transcriptional profiling: microarrays and RNA-Seq. The later has a number of advantages such as greater dynamic range, and an ability to measure previously unknown transcripts. The RNA-Seq technology also presents some challenges such as potential non-uniform read coverage and transcript length biases, and recently there has been a flurry of publications approaching these important issues -24. One We propose a new multivariate approach called the Characteristic Direction which is better able to identify DEG than univariate approaches including the methods: fold change, SAM, the Welch’s t test, LIMMA and DESeq. Our approach naturally incorporates a regularization scheme to deal with the problem of dimensionality, and also provides an intuitive geometrical picture of differential expression in terms of a single direction. We show how this geometrical picture reliably characterizes the differential expression and also leads to some natural extensions of the approach such as improved gene-set enrichment analysis. In addition, we take advantage of a neat mathematical trick to make the Characteristic Direction method fast to compute.Previous attempts to validate expression analyses have tended to rely on simulated data due to the general lack of ground truth with which to compare the results when applying the methods to real data. However, simulated data can only contain a simplified reproduction of the rich structure of expression data and so can only provide an incomplete picture of the effectiveness of the method under investigation. We benchmarked our method using simulated data, but we also developed original methods to benchmark DEG calling using real data Figure . FirstlyApart from being able to assess the analysis methods, these results are interesting in themselves as they show that we are able to infer information about TF and drug perturbations from expression data. For example, we find that proteins that are known to interact with the TFs and the drug targets tend to be also within the DEG. Finally, we show how a natural extension of the Characteristic Direction method can be used to perform potentially improved gene-set enrichment analysis. We compared enriched terms for DEG identified in human cancer stem cells and show that the Characteristic Direction enrichment method recovers more relevant Gene Ontology (GO) terms as compared with GO terms recovered by the hypergeometric test, or GSEA.Classification approaches, for example those that predict clinical outcome from gene expression data, are inherently multivariate as they use the structure of the gene expression profiles as a whole in order to distinguish between biological conditions or classes. Our approach is to repurpose linear classification methods in order to characterize differential expression and identify DEG. We use a linear classification scheme, which defines a separating hyper-plane; the orientation of which we show can be interpreted to identify DEG. We also find that the direction normal to the separating hyper-plane provides a simple geometrical conceptualization of the differential expression, which naturally leads to extensions of the approach, such as a new formulation of gene set enrichment analysis.N, in which the expression of p genes is measured, and then let each expression profile sample form a row of the matrix X (a N × p matrix). For generality at this point we shall consider the case where each of the expression samples comes from one of K classes belonging to the set G. In linear discriminant analysis (LDA) the log-ratio of class posteriors P (G|X), is written as follows  is defined by the normal p-vector, in the third term on the right hand side, that we label b,where, p-vector, b, as the direction in expression space that best characterizes the differential expression, and show how the components of this vector can be used to identify differentially expressed genes. However, first we note a few potential issues: the calculation involves the inverse of a very large p × p matrix which is not only expensive to compute but also the elements must be estimated from a relatively small sample-size (p >>N), which means that the matrix is singular and this leads to large variance in the results even when using the generalized inverse.The estimation, from the data, of the terms in this equation is explained in the Additional file The issue of singularity and large variance can be tackled with a regularization procedure, for example, the covariance matrix can be shrunk to the scalar variance as follows,σ2 is the scalar covariance  large number of features (genes) with a relatively small number of samples; 2) significant dependencies between the expression levels of the genes, leading to dimensionality which is much smaller than the number of features. In addition to these properties we require control over the number and identity of genes which are differentially expressed between two datasets. There are a number of ways that datasets with these properties may be generated, but we chose the simplest, with the fewest free parameters. In a nutshell, we use a multi-variate normal distribution distributed throughout a random subspace of the full expression space, the dimension of which reflects the dimension of the dataset. By ensuring that this subspace spans a predefined vector of differentially expressed genes we can perturb the mean of the normal distribution, preserving the covariance matrix, to generate data with pre-defined differentially expressed genes. An explicit description of the algorithm follows:p, the total number of differentially expressed genes, nd, the dimension of the data sets, D, the number of samples in each class, N, and a scale parameter which controls the magnitude of the difference between the “control” and “perturbed” data sets, Δ. First we determine which genes are to be differentially expressed and in which direction – this is done by generating a random unit p-vector with nd non-zero components, corresponding to the differentially expressed genes. We refer to this vector as D isotropic random orthonormal vectors which provide a basis for a random subspace of expression space. This is generated by iteratively generating a random isotopic vector bi at step i, then calculating that part of bi which is parallel to the subspace spanned by the previously generated vectors {bj |j <i},The parameters input into the synthetic data generation algorithm are: the total number of genes on the array, bi, resulting in a new vector which is perpendicular to the previous members of the set; this is normalized before being included in the set and moving on to the next iteration. The result is a set of orthonormal basis vectors for an isotropic subspace of dimension D which also includes the pre-defined vector of differentially expressed genes. For each class: “control” and “perturbed” we next generate random data within this subspace by drawing from a multi-dimensional normal distribution. To do this we must first define the mean and covariance matrices for each class. If, for simplicity, we assume linearity, then we may think of our random subspace as being the Principal Component space, and the data should be uncorrelated in this space, so we set the off diagonal elements of the covariance matrix to be zero and it only remains to determine the variances. We do this in such a way as to reflect a general property of biological expression data where the first principal component captures the most variance, and subsequent principal component capture successively smaller variances. We model this property very simply by setting the variance in the ith, principal component direction to be equal to ei-1)- scores from each experiment. Before proceeding, we describe these two methods in more detail. Expression data from each experiment A, which we label D(r), is examined. If the gene sets Sj contain genes which are neither preferentially significant or insignificant then we expect a uniform distribution andThen the cumulative distribution function of D(r) - r for significant deviations from zero in order to evaluate the various methods. A significant positive value corresponds to the genes in Sj being concentrated at the smaller scaled ranks and therefore having greater significance than a uniform random distribution. The entire process is visualized in Figure Any significant deviation from this indicates that the gene sets are significant in the differential expression analysis, therefore we examine φ, by a premise similar to that behind the Kolmogorov-Smirnov test  - r we also include a right-hand scale to the plots which have the values scaled by φ to give an impression of how the deviation compares to what might be expected from random fluctuations under the null hypothesis of a uniform distribution of rankings. Values >> 1 on this scale indicate significant non-uniformity in the distribution of ranks.Random fluctuations from zero are to be expected and we can estimate the scale for these fluctuations, D(r) - r, for each ranking method and each gene list type .Hence, this method allows us to visually and quantitatively compare the perturbation of cumulative distribution functions from uniform, e Figure a-d. AparNext we collected 130 experiments from GEO  as these are two common sample sizes found in GEO datasets, and we also examined two different values for the dimensionality (10 and 20). The resulting ROC curves show that the characteristic direction outperforms the other methods in recovering the differentially expressed genes from the synthetic data  . The varOne commonly used approach to obtain a picture of the biology from the analysis of differential expression is the evaluation of the DEG for enrichment given previously annotated gene sets. Gene Set Enrichment Analysis (GSEA) mentioned in the introduction, is one of the most popular approaches to accomplish this task. A more basic and widely used approach is to use Welsh’s t test, or SAM, to identify differentially expressed genes and then apply the Hypergeomtric test to examine enrichment of gene sets deriving from various gene-set libraries or the Gene Ontology . We can http://www.maayanlab.net/CD.We have described a new multivariate approach to differential expression which is better able to identify DEG while also addressing the issues associated with the high dimensionality of expression data. The Characteristic Direction approach uses the orientation of the separating hyperplane from a linear classification scheme, linear discriminant analysis, to define a direction which characterizes the differential expression. This results in a simple, highly-regularized characterization which is appropriate for genome-wide expression analysis. We compared the performance of this approach to established univariate approaches, with real and synthetic data. The validation scheme in the context of TF and drug perturbations is in itself valuable for benchmarking both computational and experimental methods. Extracting a large number of control verses perturbation expression datasets from GEO and prioritizing the genes with the various methods, we were able to show that the Characteristic Direction approach prioritizes genes which are associated with the binding sites of the perturbed TF and targets of drugs respectively; and it does so to significantly greater degree than a selection of popular methods. We took advantage of the opportunity to use independent prior knowledge datasets to validate our method. It is established that binding and unbinding of transcription factors to the promoters of genes is, in general, used for gene expression regulation. However, it is also clear that binding to the promoter does not necessarily result in differential expression. This is especially true when considering different cellular contexts. In most cases of our validation scheme the ChIP and array do not come from the same cell lines. However, there is some correlation/overlap between DEG after TF knockdowns and TF putative binding based on ChIP for the same TF in most cases. We do not know the true positives but we know that more overlap is likely due to a more accurate method to identify the DEG. We name this a silver standard for validation as it is not as good as a gold standard but it is good enough to compare DEG calling methods. The fact that we were able to recover genes associated with the binding sites of the perturbed TF is interesting on its own as it reveals a relation between DNA interactions identified by ChIP-Seq experiments and mRNA levels from expression profiling. Similarly, the ability of the method to identify a clear relationship between drug targets and the differential expression of their interactors in a systematic way is also noteworthy because for many drugs we do not know the targeted pathways while differential expression signatures are readily available. For the RNA-Seq validation we used a single study which compares differential binding of a TF, to differential expression, in the context of high-throughput sequencing. We found a stronger apparent relationship between differential binding and differential expression when using the Characteristic Direction approach as compared to the DESeq method. Like all statistical methods, the Characteristic Direction method works best when there are many repeats of the same condition. In principle, the method requires at least two repeats, but at least three repeats are needed for practical applications. The microarray and RNA-seq data used for validation of the method always had at least three repeats for each condition. It is true that in most RNA-seq studies so far investigators do not have that many repeats (1 or 2), but this is likely to change as the cost of such experiments rapidly drops. To make the Characteristic Direction method accessible, we implemented it in Python, R, MATLAB and Mathematica. Readers that are interested in applying the method to their own data should refer to the open source scripts and examples available at: http://www.maayanlab.net/CD.Implementations of the method are provided in Python, R, MATLAB, and Mathematica freely available at: The authors declare that they have no competing interests.AM and NRB designed the study and wrote the paper. NRB performed all analyses, wrote all the equations and implemented the method in Mathematica. KH developed the TF perturbation gene-set library from GEO, ASF developed the drug perturbation gene-set library from GEO, YK processed the data from ENCODE, EYC implemented the method in Python, QD implemented the method in MATLAB and R. All authors read and approved the final manuscript.The Characteristic Direction: A Geometrical Approach to Identify Differentially Expressed Genes.Click here for file


SCNP) for predicting response to standard induction chemotherapy (complete response [CR] or CR with incomplete hematologic recovery [CRi] versus resistant disease [RD]). SCNP-evaluable patients from four SWOG AML trials were randomized between Training  and Validation Analysis Sets . Cell survival, differentiation, and apoptosis pathway signaling were used as potential inputs for DXSCNP. Five DXSCNP classifiers were developed on the SWOG Training set and tested for prediction accuracy in an independent BM verification sample set (N = 24) from ECOG AML trials to select the final classifier, which was a significant predictor of CR/CRi . The selected classifier was then validated in the SWOG BM Validation Set . Importantly, a classifier developed using only clinical and molecular inputs from the same sample set (DXCLINICAL2) lacked prediction accuracy: AUROC = 0.61 (p = 0.18) in the BM Verification Set and 0.53 (p = 0.38) in the BM Validation Set. Notably, the DXSCNP classifier was still significant in predicting response in the BM Validation Analysis Set after controlling for DXCLINICAL2 (p = 0.03), showing that DXSCNP provides information that is independent from that provided by currently used prognostic markers. Taken together, these data show that the proteomic classifier may provide prognostic information relevant to treatment planning beyond genetic mutations and traditional prognostic factors in elderly AML.Single-cell network profiling (SCNP) data generated from multi-parametric flow cytometry analysis of bone marrow (BM) and peripheral blood (PB) samples collected from patients >55 years old with non-M3 AML were used to train and validate a diagnostic classifier (DX In “elderly” AML populations , the complete remission (CR) rate in response to standard-dose cytarabine (Ara-C)-based induction chemotherapy ranges from 35 to 50% depending on the study, while the rate of treatment-related mortality (TRM) ranges from 15–20% , hypometFLT3-ITD mutation status . T. TFLT3-IIn accordance with the Declaration of Helsinki, all patients provided written informed consent for the collection and use of their samples for research purposes. Institutional Review Board approval was obtained from Independent Review Consulting, Inc.  on August 31, 2009. Clinical data were de-identified in compliance with Health Insurance Portability and Accountability Act regulations.The study used cryopreserved pretreatment bone marrow (BM) and peripheral blood (PB) samples collected from two groups of AML patients: patients enrolled in SWOG studies  and patients enrolled in ECOG studies .For patients enrolled in SWOG trials, inclusion criteria were age > 55 years, diagnosis of non-APL AML, and enrollment in one of four SWOG frontline treatment trials using Ara-C-based induction therapy  for 7 days and daunorubicin 30-45mg/m2 for 3 days . Similarly to what is done in clinical practice, patients with unknown cytogenetics risk categories were imputed as intermediate cytogenetics.Cytogenetic risk classification was first determined per the SWOG study from which the samples were obtained for sample randomization (stratification purposes). Cytogenetic risk classification for all samples (SWOG and ECOG) were then assigned using NCCN 2013 guideline criteria for purposes other than randomization (e.g. model building DXCLINICAL1 and DXCLINICAL2. SCNP data for the BM (n = 43) and PB (n = 57) Training Analysis Sets were used to develop the SCNP-based predictive models. Since some patients had two SCNP-assessable samples (BM and PB), a partial overlap existed between the patients in the BM and PB Analysis Sets. However, within each Analysis Set, each patient contributed only one sample  (CLINICAL1 and DXCLINICAL2 which were missing (≤6% for any input except for cytogenetics which was missing in approximately 20%), were imputed as described in SCNP assays for all SWOG patient samples were performed blindly to all clinical data and in a random order as part of a single experiment. Patients with assessable BM and/or PB SCNP results n = 213) were then randomized approximately 1:1 to Training and Validation Sets  . Values SCNP classifier for validation , and the DNA damage response (DDR) were investigated. Apoptotic signaling and DDR pathways were measured after or Ara-C .The term “signaling node” (or simply “node”) refers to a proteomic readout in the presence or absence of a specific modulator at a specific time point after modulation. Modulators included endogenous growth factors , cytokines  and drugs . Several metrics  were designated non-evaluable and excluded from analyses see . ClinicaData were acquired using FACSDiva software  on several Canto II (BD) FACS Canto II flow cytometers. FCS files were gated using WinList  and all data were stored in a MySQL database for access and querying. For details please refer to Specific metrics were developed to describe and quantify the functional changes observed using the SCNP assay. Median fluorescence intensity (MFI) was computed from the fluorescence intensity levels of the cells. Equivalent Number of Reference Fluorophores (ERF), a transformed value of the MFI values, was computed using a calibration line determined by fitting observations of a standardized set of 8-peak rainbow calibration particle beads (RCPs) for all fluorescent channels  to standard values assigned by the manufacturer. ERF was used to standardize, qualify and monitor the instrument during setup, and to calibrate the raw fluorescence intensity readouts on a plate-by-plate basis and to control for instrument variability. ERF values were then used to compute a variety of metrics to measure the biology of functional signaling proteins . AdditioBasal is defined as:Log2Fold Change is defined as:u is the Mann-Whitney U statistic comparing the intensity values for an antibody in the modulated and unmodulated wells that has been scaled to the unit interval  for a given cell population for a sample. Measures the proportion of cells that have higher (Uu > 0.5) or lower (Uu < 0.5) expression of the antibody in the modulated state compared to the unmodulated state. Similar in nature to percentage of cell that have positive (or negative) expression, except that a threshold to determine positivity is not necessary.Ua is the same as the Uu metric except that the auto-fluorescence well is used as the reference instead of the unmodulated well.UThe percent healthy metric was calculated for each sample by using the autofluorescence well and the c-PARP stained well:th percentile value for autofluorescence was used to determine the positive-negative split point.Percentage of leukemic blast cells that is negative for cPARP expression. The 98Standard instrument controls , hemoglobin, platelet count, performance status (0–1 vs. 2–3), FAB class (M0/M1/M2/M7 vs. other), and AML onset (de novo vs. secondary)] for the 74 patients from SWOG trials randomized to the Training Set.CLINICAL1, inputs for DXCLINICAL2 included cytogenetic risk group, percentage of CD34+ cells, and presence of genetic markers for FLT3-ITD (as a continuous variable and as binary) and NPM1 mutations. Percentage of CD34+ cells was calculated from phenotyping well during the SCNP assay because this information was not available for the SWOG samples from the clinical site.In addition to the clinical factors used for DXSCNP modeling separately in the BM (n = 43) and PB (n = 57) Training Analysis Sets and then to assess, by cross applying the classifiers from one tissue type to the other in the training set, if independent classifiers were needed for each tissue type. Logistic regression models were investigated as predictors of response. These models were fit by LASSO using the R Penalized Package , , , ,  in in30] inFLT3 ITD, CEBPα, and NPM1 mutations, which have been incorporated into NCCN guidelines, provide additional prognostic information mostly useful for post-induction treatment planning (i.e. consolidation therapy). More recently, predictive models, such as a web-based application that uses standard clinical and laboratory values  and cytogenetic and molecular risk factors to generate an overall prognostic score, have been shown to have a significant association with induction response ..FLT3 ITDCLINICAL1) or both clinical and molecular parameters (DXCLINICAL2) as classifier inputs. The study design allowed for a descriptive comparison in the same patient population of the different classifiers’ performance. After controlling for clinical and genetic variables, the results supported the independence of the prognostic information provided by the SCNP-based classifier from that of traditional clinical and molecular markers [In the current study the performance of an SCNP-based classifier was assessed in parallel against intra-study developed models which used only clinical  were not available in our data set making it impossible to compare results from the web-based predictor of induction response  to the DOverall, our findings confirm the value of the SCNP-assay classifier, which can assess the functional effects of downstream multiple genetic and epigenetic molecular alterations.SCNP classifier incorporated just two signaling nodes that assessed the functional capacity of the intracellular apoptosis pathway in the total blasts and the proportional reduction of CD34+ cells upon treatment in response to in vitro treatment with AraC and Daunorubicin). Of note, the cell-signaling based classifier developed in a pediatric AML population referenced above [Although the breadth of biology investigated in the training phase of the study included many signaling nodes in multiple pathways believed to be important in the leukemogenesis process and response to chemotherapy , the final locked and validated DXed above  includedSeveral methodological considerations and limitations need to be considered in interpreting these results. First, this study used cryopreserved samples  from biorepositories, rather than fresh samples. While this approach is efficient since it allows for batch analysis of large numbers of samples for which clinical annotations have already been collected, it raises concerns about the applicability of results to clinical settings (in which fresh samples will be used); and about the potential to introduce patient selection bias in the analysis, which could limit the generalizability of the classifier to different patient populations.Previous studies have shown high correlation between SCNP readouts in paired fresh and cryopreserved aliquots of the same AML samples , suggestSCNP in BM sample subsets defined by clinical characteristics is shown in SCNP is somewhat higher for patients with WBC count greater than the median of 19x109/L: 0.86 vs 0.60. The difference in prediction accuracy between subgroups defined by Sex and Age are even less significant. Prospective studies will be required to evaluate the role of SCNP-based predictors in fresh cells from a more representative population of newly diagnosed patients.Regarding potential selection bias, patients were selected on the basis of specimen availability and, as expected, evaluable patients had relatively higher WBC counts and blast percentages when compared to the non-evaluable patients . AccuracThe purpose of our study was to identify and validate a predictive classifier for response to standard induction therapy using as inputs intracellular functional pathway readouts. Despite the heterogeneity of the patient population studied (i.e. samples obtained from patients enrolled on different studies that were conducted over greater than a 10 year span) the SCNP classifier that was verified and validated  was quite robust. These data underscore that the biology identified and measured using SCNP is crucial to AML blast in vivo chemosensitivity and are in alignment with data by Vo and colleagues that demonstrate the relative priming of AML blasts toward apoptosis is associated with clinical outcomes . Of notede novo and secondary AML could have affected model performance characteristics during the Verification and Validation phases, particularly in the PB Validation Analysis Set . When paired (from the same patient) BM and PB samples were grouped by AML onset (de novo vs. secondary), the SCNP classifier scores were concordant between BM and PB in the de novo subset (Pearson R = 0.7). Furthermore, DXSCNP was a reliable . For the 10 patients with secondary AML in the PB Validation Analysis Subset, 5 had outcome of RD and all 5 were predicted incorrectly by DXSCNP as compared to 2 RD donors in the PB training set. These findings are consistent with prior data, which indicated that the underlying biology of secondary AML is different from that of de novo AML [Compared with the BM Verification and Validation Analysis Sets, the BM Training Analysis Set had a lower percentage of patients with secondary AML, which was not a stratification factor during randomization . Although these differences were not statistically significant, likely due to the small sample size, the differences in biology of reliable  predictoSCNP classifier prospectively into the clinical setting to confirm its feasibility and value as a provider of independent and actionable information for consideration prior to initiating therapy. Such validation may allow the development of a taxonomy of response to induction therapy and a more comprehensive understanding of the variegated blast-cell signaling patterns seen in de novo vs. secondary AML. A parallel SCNP study, initially retrospective and latter prospective, is planned to develop classifiers for prediction of disease relapse in patients who achieve CR after induction therapy to guide individualized consolidation treatments.In sum, the results of this study confirm the ability of quantitative SCNP testing using functional flow cytometry to predict induction response in elderly AML patients. The assay provides accurate, independent data on disease biology and has the potential to inform treatment choices by allowing patients to avoid harmful treatment when it is likely futile, while offering the opportunity for those patients to consider enrollment in clinical trials evaluating new targeted and less intensive regimen as first line treatment. Prospective studies in adults and children are planned that move the DxS1 Fig(DOCX)Click here for additional data file.S1 Inputs(DOCX)Click here for additional data file.S1 Methods(DOCX)Click here for additional data file.S2 Methods(DOCX)Click here for additional data file.S3 Methods(DOCX)Click here for additional data file.S4 Methods(DOCX)Click here for additional data file.S5 Methods(DOCX)Click here for additional data file.S1 MiFlowCyt Report(DOCX)Click here for additional data file.S1 Table(DOCX)Click here for additional data file.S2 Table(DOCX)Click here for additional data file.S3 Table(DOCX)Click here for additional data file.


Cervical cancer remains the third most frequently diagnosed and fourth leading cause of cancer death in women worldwide. We sought to develop a micro-RNA signature that was prognostic for disease-free survival, which could potentially allow tailoring of treatment for cervical cancer patients. A candidate prognostic 9-micro-RNA signature set was identified in the training set of 79 frozen specimens. However, three different approaches to validate this signature in an independent cohort of 87 patients with formalin-fixed paraffin-embedded (FFPE) specimens, were unsuccessful. There are several challenges and considerations associated with developing a prognostic micro-RNA signature for cervical cancer, namely: tumour heterogeneity, lack of concordance between frozen and FFPE specimens, and platform selection for global micro-RNA expression profiling in this disease. Our observations provide an important cautionary tale for future miRNA signature studies for cervical cancer, which can also be potentially applicable to miRNA profiling studies involving other types of human malignancies. Micro-RNAs (miRNAs) are a class of small, non-coding RNAs that play important roles in regulating target genes by binding to complementary sequences in mRNA transcripts . Deregulet al., who reported that miR-143 and miR-145 suppressed cell growth, whereas miR-146a promoted cell proliferation in cervical cancer [et al. identified miR-29 to be the most highly enriched HPV-associated miRNA in cervical cancer, which functions by restraining cell cycle progression and inducing apoptosis via YY1 and CDK6 [et al. [Cervical cancer is the third most frequently diagnosed cancer, and the fourth leading cause of cancer mortality in women worldwide, with an estimated 530,000 new cases and 175,000 deaths each year . Althougl cancer . Li et aand CDK6 . More reand CDK6 . A two-m [et al. , and remWritten informed consent was obtained from all human subjects, according to a protocol approved for this study by the University Health Network Research Ethics Board.2 total, 5 doses). Patients were staged using the FIGO  system, with additional clinical information gathered using computed tomography (CT) scans of the abdomen and pelvis and magnetic resonance imaging (MRI) of the pelvis to assess local and lymphatic disease. Pelvic and para-aortic lymph nodes were classified as positive for metastatic disease if the MRI short-axis dimension was >1 cm, and equivocal if it was 8 to 10 mm.Pre-treatment cancer samples were collected from patients with cervical cancer prior to undergoing curative chemo-radiation, consisting of external-beam radiotherapy to the primary cervical tumour and pelvic lymph nodes , combined with weekly cisplatin  compound) for histopathologic examination, then flash-frozen in liquid nitrogen. H&E-stained tissue sections were cut from the OCT-embedded material, and evaluated by a gynecological pathologist (B. Clarke). The total cell content (stroma and tumour cells) was estimated for all tissue samples using a light microscope, and only samples containing at least 70% tumour cells were considered for further analysis. Flash-frozen normal cervix tissues obtained from 11 patients who underwent total hysterectomy for benign causes served as the normal comparators. The clinical data for the patients in the training cohort are found in The validation cohort comprised of diagnostic FFPE blocks collected from 87 cervical patients treated between 1999 and 2007, inclusively. There was no overlap of patients between the training and validation cohorts. All samples contained at least 70% malignant epithelial cells, as determined by a gynecologic pathologist (B. Clarke), or were macro-dissected prior to RNA purification. FFPE normal cervix tissues obtained from 9 patients who underwent hysterectomy for benign causes served as the normal comparators. The clinical data for the patients in the validation cohort are found in For the training cohort specimens, two sections of 50-μm thickness were cut from the OCT-embedded flash-frozen tissues and placed in a nuclease-free microtube. Total RNA was isolated using the Norgen Total RNA Purification Kit (Norgen Biotek), according to the manufacturer’s instructions. Global miRNA expression was measured in both the cervical cancer and normal cervix tissues with the TaqMan Low Density Array (TLDA) Human MicroRNA A Array v2.0 (Applied Biosystems) using the Applied Biosystems 7900HT Real-Time PCR System, as previously described .For the validation cohort specimens, ten sections of 5-μm thickness were cut from the FFPE tissues and placed in a nuclease-free microtube. Total RNA was isolated using the Norgen Total RNA Purification Kit (Norgen Biotek), according to the manufacturer’s instructions. We measured the expression of the 9 miRNAs in our prognostic signature using three methods: 1) Applied Biosystems TLDA Human MicroRNA A Array v2.0 with 300 ng of total RNA per sample; 2) NanoString nCounter Human miRNA Expression Assay v1.6.0 with 200 ng of total RNA per sample; and 3) individual single-well qRT-PCR using Applied Biosystems TaqMan MicroRNA Assays with 10 ng of total RNA per sample, as previously described . The Taq2(2T-CCC)-, where CT represents the threshold cycle, and CC represents the mean threshold cycle of the control genes.To normalize the miRNA expression data from TLDA, the raw miRNA abundances were loaded into the R statistical environment (v2.15.2). Three control genes were utilized for normalization: RNU44, RNU48, and U6. Normalized miRNA abundances were calculated as -logProbe level correction.Code Count Correction = "geo.mean" (geometric mean)Background Correction = "mean.2sd", i.e. mean +/- 2 standard deviations Sample Content Correction = "top.geo.mean" 2 transformationlogTo normalize the miRNA expression data from NanoString, the R package ‘NanoStringNorm’  was utilt > 35) in over 20% of samples. Least Absolute Shrinkage and Selection Operator (LASSO) regression was applied to the normalized TLDA miRNA expression data for the training cohort [The normalized data were filtered to remove miRNAs with low expression . The p-value associated with the two curves was determined using the log-rank test. All statistical analyses were performed with R statistical environment (v2.15.2) using the “Survival” package for survival analysis, and “glmnet” for LASSO analysis. Significance was defined as p-values below 0.05.Publically-available cervical cancer Illumina Small RNA-Seq data from The Cancer Genome Atlas (TCGA) Data Portal was utilized as another independent validation cohort (n = 48). Level three Small RNA-Seq (isoform_expression) data and clinical information for the 48 cervical cancer patients were downloaded from the Broad Firehose (stddata run 2013_06_06). Reads-per-million (RPM) data were log2 transformed and z-score standardized before the 9-miRNA signature equation was applied to calculate risk scores. Patients were dichotomized by the previously established cut-point (median risk score in training cohort) and compared using a log-rank test.P < 0.01) were downregulated in our cervical cancer samples  and validation (n = 87) cohorts are provided in  samples . Of the X represents the normalized expression level of X, and X represents one of the nine miRNAs in the prognostic signature.Using LASSO regression on the normalized TLDA data from the training cohort , we derivs. high risk groups. Our 9-miRNA signature was significantly predictive of DFS for the 79 patients in the training cohort, with a hazard ratio of 9.26 and log-rank p-value of 6.9 x 10-7  was used to dichotomize the low 9 x 10-7 , and wasvs. low risk groups based on their risk score, using the same cut-off point as defined in the training cohort  TLDA (n = 87); 2) NanoString (n = 87); and 3) qRT-PCR . A risk-score was calculated for each of the patients by applying the miRNA TLDA expression values  to our pg cohort . This ang cohort  and the g cohort  datasetsg cohort  and 2C. g cohort . We alsog cohort .et al.  [vs. low risk groups, with 1/3 in the former and 2/3 in the latter categories, which reflected the Hu et al. population when 0 was used as their cut-off. Based on the Kaplan-Meier analysis, the 2-miRNA signature was not significant when applied to the patients in our frozen cohort utilizing the TLDA platform . Using the same analytical methods, again, the 2-miRNA signature could not be corroborated  in nasopharyngeal carcinoma , and miRWe believe that there are three key reasons as to why we could not validate our candidate 9-miRNA prognostic signature for cervical cancer: i) intra-tumour heterogeneity; ii) miRNA expression data from frozen and FFPE samples could not be directly compared in this disease; and iii) the current platforms for global miRNA expression profiling are not sufficiently robust.et al. evaluated intra-tumoural heterogeneity across 11 cervical cancer patients, and demonstrated that multiple biopsies from distinct areas of an individual tumour were necessary to reduce sampling bias, with genes displaying low intra-tumour heterogeneity requiring two to three biopsies, and genes with high intra-tumour heterogeneity requiring more than six biopsies per tumour [Firstly, it is well-established that human tumours are intrinsically heterogeneous. Two types of tumour heterogeneity exist: a) inter-tumour heterogeneity, with differences between tumours arising from different patients; and b) intra-tumour heterogeneity, with differences between distinct sub-populations of cancer cells within a single individual’s tumour. In cervical cancer, intra-tumour heterogeneity has been described on various levels. Many studies have demonstrated significant variations in interstitial fluid pressure , blood pr tumour .To date, intra-tumoural heterogeneity of miRNA expression has only been reported in breast cancer thus far . Howeveret al. analyzed 3 miRNAs  using qRT-PCR, and only calculated the ΔCt values between frozen and matched FFPE samples without performing any correlation tests. Furthermore, several of these published reports have demonstrated that although there were high correlations (> 0.5) between tissue-matched frozen and FFPE samples for the overall panel of miRNAs tested, there were specific individual miRNAs that were poorly correlated between the matched samples [A second reason for the lack of signature validation is the lack of concordance with respect to the tissue preservation method used for the two patient cohorts; specifically, the training and validation sets consisted of frozen and FFPE samples, respectively. This experience would appear to contradict several previous studies that have reported high concordance in the miRNA expression data from tissue-matched frozen and FFPE samples derived from various types of human tissues, including: breast , lung 4, kidney  samples , 45, 47, samples , 49. TheLastly, we believe that the current methods used for global miRNA expression profiling are not sufficiently robust. We used the TLDA and NanoString platforms for global miRNA expression profiling, which are both widely-used for miRNA expression profiling. However, in our own recent analyses, the correlation in miRNA expression levels between these two platforms, even using the same FFPE RNA samples, had only a ρ of 0.65 (manuscript in preparation), underscoring the challenges in the technologies to be able to reliably identify clinically useful biomarkers in this disease. This might also provide one technical explanation for the recent review describing the difficulties in validating miRNAs for human malignancies . Given tOur validation attempt using Small RNA-Seq data from the TCGA data portal was promising, with our 9-miR signature predicting worse outcome for high-risk patients than low-risk patients (p = 0.053). Interestingly, the datasets from the TCGA cohort and our training cohort were both generated from frozen tissues. With the addition of more cervical cancer samples to the TCGA data portal with survival information and miRNA expression data, we could potentially obtain a statistically significant result in a future validation attempt.In conclusion, a prognostic miRNA signature for cervical cancer could not be validated, due to intra-tumoural heterogeneity, incompatibilities between miRNA expression data from frozen and FFPE samples, and insufficiently robust technical platforms for global miRNA expression profiling. We propose that this could potentially be resolved in the near future, with the advancement of technologies for miRNA expression profiling such as deep-sequencing. At present, the TCGA Data Portal contains deep sequencing miRNA expression data with clinical annotation for only 48 cervical cancer patients; when this dataset is expanded to include more samples, it could potentially be utilized as an important resource to identify and corroborate potential miRNA signature sets. Our observations provide an important cautionary tale for future miRNA signature studies for cervical cancer, which can also be potentially applicable to miRNA profiling studies involving other types of human malignancies.S1 TableClinical data for 79 frozen cervix samples.(TXT)Click here for additional data file.S2 TableClinical data for 87 FFPE cervix samples.(XLSX)Click here for additional data file.S3 Table2) of 29 miRNAs that were significantly differentially-expressed in cancer vs. normal cervix samples, in order of increasing P-value.Fold changes (log(PDF)Click here for additional data file.S4 TablemiRNA expression in 79 frozen cervix samples.(XLSX)Click here for additional data file.S5 TablemiRNA expression in 87 FFPE cervix samples.(TXT)Click here for additional data file.S6 TableNanoString data for 87 FFPE cervix samples.(TXT)Click here for additional data file.S7 TablePCR data for 68 FFPE cervix samples.(XLS)Click here for additional data file.S1 FigKaplan-Meier analysis of DFS. A risk score was calculated for each patient in the validation cohort by applying our 9-miRNA signature for DFS to the miRNA expression data generated using A) TLDA, B) NanoString, and C) individual qRT-PCR. The validation cohort was divided into three groups based on risk scores: high-risk, medium-risk, and low-risk. HR; hazard ratio, DFS; disease-free survival, CI; 95% confidence interval.(TIF)Click here for additional data file.S2 Figet al. 2-miR signature. HR; hazard ratio, CI; 95% confidence interval.miRNA expression data from the TCGA miRNASeq cohort (n = 48) was used to test: A) our 9-miR signature, and B) the Hu (PDF)Click here for additional data file.


Despite the individually different molecular alterations in tumors, the malignancy associated biological traits are strikingly similar. Results of a previous study using renal cell carcinoma (RCC) as a model pointed towards cancer-related features, which could be visualized as three groups by microarray based gene expression analysis. In this study, we used a mathematic model to verify the presence of these groups in RCC as well as in other cancer types. We developed an algorithm for gene-expression deviation profiling for analyzing gene expression data of a total of 8397 patients with 13 different cancer types and normal tissues. We revealed three common Cancer Transcriptomic Profiles (CTPs) which recurred in all investigated tumors. Additionally, CTPs remained robust regardless of the functions or numbers of genes analyzed. CTPs may represent common genetic fingerprints, which potentially reflect the closely related biological traits of human cancers. BRAF, EGFR, PAK5, HER2, ALK or hormone receptors [The use of DNA microarray technologies enabled the generation of myriads of data, sustaining further molecular sub-classification of many previously described pathologic phenotypes with significant effects on clinical decision making and prognosis –6. In paeceptors , 10, alleceptors –13, haveeceptors –17.The fact that close to 75% of all genes have already been identified as being potentially cancer relevant , and thevia gene expression analysis. This hypothesis is backed by our recent identification of global gene expression outputs of prognostic relevance in renal cell carcinoma (RCC) [In line with the proposed concept of cancer hallmarks, we assumed that regardless of their highly diverse phenotypic and genotypic appearance, all tumors share common cancer traits, which may potentially be detected ma (RCC) . Based oUnbiased by any cancer-specific marker or classification schemes currently used, we analyzed 55 published studies of gene expression data encompassing a total of 8397 patients with 13 different cancer types by means of gene-expression deviation profiling.Survival data linked to the published dataset GSE19949 was provided by the Cancer Registry Zurich and Zug and approved by the ethics committee of the Canton Zurich (KEK-ZH-Nr. 2013–0629). This data was used for the verification of the prognostic relevance  as propoAll data was retrieved from the GEO repository as published by the authors. An overview of all analyzed datasets is given in 2 gene probe set value, the mean expression value was calculated over all samples of the given data set. This mean value was then subtracted from the log2 expression values of all samples. The calculated difference denoted the individual deviance of the sample from the mean expression value for the respective probe set.The method comprised the following steps: For every normalized logTo those expression values which were close to mean the value 0, to those with significantly higher values than mean the value 1, and to those with significantly lower expression than mean the value -1 were assigned. The threshold for high and low was set to 43% of the standard deviation, which means that all three values  occurred at almost the same frequency. The value 0 was assigned to those deviation values which were located between -0,43 σ and +0,43 σ. Deviation values lower than -0,43 σ were assigned with -1, those higher than +0,43 σ with +1. Additional categories were therefore automatically excluded.The samples were then clustered using well known clustering methods such as k-means or SOM, so that samples with similar  profiles were assigned to the same group A, B, or C. In order to generate the respective CTPs the average values for each probe set and group were calculated.At the end of this procedure, a whole genome CTP profile existed for every group, represented by a vector with a length equal to the number of probe sets of the respective microarray and values ranging between -1 and 1 .a priori [For the RCC data set GSE19949 the assignments to A, B, or C groups were known a priori . Detailede Finetti [de Finetti mappings are efficient means to depict percentage compositions for 3 parameters in an equilateral triangle [de Finetti like mappings can be found in As proposed by  Finetti  ternary triangle , 25. UsiFor each sample of a new data set, the expression profile was calculated as mentioned above. As for individual samples no averaging takes place, the profiles contained only the values -1, 0, or 1. The correlations between the sample profile and the whole genome A-, B- and C-CTPs were calculated. The sample was assigned to the CTP with the highest Pearson correlation Figs .et al., see additional file 12, table S8 [By analyzing gene expression profiles in a RCC patient cohort (GSE19949) we recently identified three subgroups , which were not significantly associated with pathological prognostic parameters such as nuclear differentiation grade and tumor stage (Beleut table S8 ). As surtable S8 .As survival data were meanwhile also available for 89 patients of the first cohort, we could confirm the result obtained from the second tissue microarray patient cohort. As shown in To exclude that those RCC gene-expression signatures are biased due to non-cancerous effects, we first aimed at designing a comprehensive algorithmic Cancer Transcriptomic Profile (CTP) classification model using the RCC gene expression data. Requirements for the model included automated reassignment of GSE19949 RCC tumors to the known CTP groups A, B and C. Additionally, the algorithms should be independent from biases derived from different normalization techniques as well as tissue type artefacts. Finally, the model should consider the entire gene expression profile resulting from the microarray chip.In order to further investigate whether RCCs could be sub-classified by our proposed CTP-model, we utilized the following approach. We classified the GSE19949 tumors into the three known groups by implementing the algorithmic workflow shown in de Finetti like mapping    to betteTo further solve the question whether other clustering technologies could also have been used as a potential starting point for CTP identification, we utilized Self Organizing Maps (SOM)  for clusde Finetti like mapping  but also on other healthy tissues derived from different organs or anatomic body parts as annotated in GSE1133 and GSE2361. In contrast to RCC, healthy controls remained grouped around the center of the  mapping  confirmide Finetti like mappings still illustrate an increased scattering of individual patients as compared to control mappings, thus strengthening the assumption of general cancer specificity rather than RCC specificity of proposed CTPs.To better investigate cancer specificity in general, we also strictly applied identified RCC-CTPs on one dataset of breast cancer  and one −6). The direct comparison of the normal and tumor tissue samples of the renal data set GSE53757  was also significant . We intentionally omitted the SOM clustered data from GSE22541 from this averaging since the independent SOM clustering introduced additional positional scatter. Detailed results for each GSE dataset and corresponding CTPs are listed . Between the two experimental sets, however, the p-values were much higher (0.04–0.98), indicating that the null hypothesis (generated from the same underlying distribution and not by noise) held as expected.Correlating all CTP-A profiles with each other across different virtual cancer types, and similarly for CTP-B and -C, yielded average correlations of less than 0.02. Calculating the same average correlations for the experimental tumor suppressor genes or oncogenes yielded average correlations between 0.7 and 0.76. The probability that these high correlations between CTPs across different cancer types were generated by random is infinitesimally small , -0.5 –-0.6 (AC), and -0.37 –-0.51 (BC). Again, these correlations were stable across the tumor suppressor and oncogenes platform , while the simulation yielded average correlations between 0.008 and 0.21. This differed significantly from the experimental values .In summary, these simulations demonstrate that random effects like mere fluctuations of expression values could not explain the high correlations between CTPs across and within different tumor types.The hallmarks of cancer imply that all cancers follow specific biological concepts which form the basis of malignant behavior. Therefore, we speculated that such biological concepts among cancers relate closely to each other and may be visualized by gene expression patterns. Our hypothesis originated from the detection of three gene expression outputs in renal cell carcinoma (RCC), which appeared to be independent from pathological parameters and correlated with patient outcome . In an aIn this study, we analyzed gene expression data obtained from 8397 samples. Sixteen of 55 datasets accounting for 4177 out of the 8387 tumors were from breast cancer, which currently presents the most frequent tumor type in the GEO repository. We detected three different Cancer Transcriptomic Profiles (CTPs) that are recurring in 13 different cancer types. These CTPs suggest molecular relationships among human cancers.per se not novel, but to our knowledge, its conceptual application to characterize mathematically the 3 CTPs has not been used and published so far. A closer look at the expression profiles of all genes  which differ, for example, between two organs, tissue/cell types or healthy/diseased tissue. As illustrated in vs. 1 (expressed) [Absolute gene expression values with subsequent associated gene enrichment technologies have been used in many gene expression studies. In keeping with this, we first used a similar approach and detected 3 groups in RCC by two-way non-supervised hierarchical clustering . Our subFig A in  clearly pressed) , 29.Furthermore, a binary approach would only distinguish between expression levels which are either low or high in two groups. With the binary approach one focuses only on low (underexpressed) and high (overexpressed) expressed genes but exclude those genes which are more or less equally expressed. Our model includes also genes with no or small expression level changes. A closer look on the 3 CTPs clearly demonstrates that these genes provide similar contributions to the CTP profiles as the high differentially expressed genes. As a result, CTPs yielded with a 2-bin system would introduce substantial distortions compared to our CTPs.These three CTP groups first detected in RCC and confirmed by the discretization method, were transferable to 12 additional cancer types. We developed the algorithm applied to address various challenges encountered in transcription analysis, so that individual expression fluctuations per gene or probe set were neutralized. Subsequent to the standard GCRMA normalization step for Affymetrix gene expression chips , we normAccording to our mathematical approach, CTPs are a reappearing pattern throughout the cancer transcriptomes. Notably, different clustering technologies, such as k-means or SOM , 32, appde Finetti-like visualizations demonstrate, however, that normal samples located closer to the triangle center than tumor samples. At the center, the correlation with any of the CTP was lowest, pointing to a lower CTP differentiation in normal tissue samples. Being limited to the present situation, however, we can only state, as shown in the results part, that the defined calculated differences are highly significant  and are not occurring due to randomness.It is of note that due to the very limited availability of samples from normal (control) tissue published in GEO, it is currently not possible to define the particular threshold between normal and tumor tissue. We noticed that the studies mostly consist of disease sample collections with no corresponding healthy counterparts. Therefore, a healthy/disease threshold could not be yielded, unless by including matched pairs of affected and non-affected tissue samples in a sufficiently large amount. The Finally, CTPs still remained stable with sets of only several hundred genes, whether randomly or non-randomly selected. Therefore, we conclude that i. the expression status of every single gene is important to contribute to a CTP and ii. the expression profile of a minimum set of genes is required to yield a CTP.As the CTPs of our RCC indicated correlation with patient outcome, we calculated Kaplan Meier survival plots using all survival data sets from different cancer types that were available in the GEO database. The results were, however, difficult to interpret (data not shown). Some survival plots for sarcoma, breast and lung cancer were similar to those obtained from RCC. Other survival plots showed no associations or, as in the case of ovarian cancer, even contrasting patterns. This data strongly suggest that the analysis of survival data sets from different patient cohorts  Tables gWe present a novel model that can be applied to identify cancer-specific gene expression profiles. It has been widely accepted that each tumor has developed its own individual molecular landscape. The entirety of molecular events leading to a malignant tumor affects always the same biological concepts described in the hallmarks of cancer. We believe that the CTPs identified by us represent the molecular outputs which exist in all human cancers. However, more in-depth investigations with larger and better defined cancer patient cohorts are needed to support our CTP concept for cancer biology but also as possible additional tool for cancer prognosis.S1 Table(XLSX)Click here for additional data file.S2 Table(XLSX)Click here for additional data file.S3 Table(XLSX)Click here for additional data file.S4 Table(XLSX)Click here for additional data file.S5 Table(XLSX)Click here for additional data file.S1 Text(DOCX)Click here for additional data file.


The rapid advancement of emerging genomics technologies and their application for assessing safety and efficacy of FDA-regulated products require a high standard of reliability and robustness supporting regulatory decision-making in the FDA. To facilitate the regulatory application, the FDA implemented a novel data submission program, Voluntary Genomics Data Submission (VGDS), and also to engage the stakeholders. As part of the endeavor, for the past 10 years, the FDA has led an international consortium of regulatory agencies, academia, pharmaceutical companies, and genomics platform providers, which was named MicroArray Quality Control Consortium (MAQC), to address issues such as reproducibility, precision, specificity/sensitivity, and data interpretation. Three projects have been completed so far assessing these genomics technologies: gene expression microarrays, whole genome genotyping arrays, and whole transcriptome sequencing . The resultant studies provide the basic parameters for fit-for-purpose application of these new data streams in regulatory environments, and the solutions have been made available to the public through peer-reviewed publications. The latest MAQC project is also called the SEquencing Quality Control (SEQC) project focused on next-generation sequencing. Using reference samples with built-in controls, SEQC studies have demonstrated that relative gene expression can be measured accurately and reliably across laboratories and RNA-seq platforms. Besides prediction performance comparable to microarrays in clinical settings and safety assessments, RNA-seq is shown to have better sensitivity for low expression and reveal novel transcriptomic features. Future effort of MAQC will be focused on quality control of whole genome sequencing and targeted sequencing. A decade ago, microarrays were the mainstream genomics technology used by the biomedical and pharmaceutical research communities. Using this technique, a large expanse of microarray data has been generated to support the drug development process. For example, genomics data has been used to evaluate drug safety and efficacy in support of both investigational new drug applications (IND) and new drug applications (NDA). To facilitate the submission of genomics data, the FDA created a novel data submission program known as Voluntary Genomics Data Submission (VGDS) and later extended it to Voluntary eXploratory Data Submission (VXDS) so that other omics data could be included. The idea behind this novel submission program was to facilitate FDA’s communication with the sponsor and to identify the best ways to apply omics data in regulatory application. The results of these efforts have helped to develop the Guidance for Industry on Pharmacogenomics (PGx) Data Submission . The VXDdata repository to keep track of all the data submitted by the sponsors , and 11 of them were published in Nature Biotechnology  consortium is a community-wide effort led by the FDA to address the above mentioned reproducibility concerns about the genomics technologies. It was started about 10 years ago, involving most FDA centers along with the international research community and industry. Its objective was to analyze the technical performance and utility of emerging molecular technologies  for clinical application and safety assessment. Throughout MAQC efforts, there was a consistent emphasis on transparency. The results and conclusions were published in peer-reviewed journals. The data generated during these efforts has been made freely available to the public. Additionally, some biological samples from which the data were generated are also available from commercial vendors. The consortium started in 2005 and by the end of 2014, three projects were completed. During the course of these projects, three different genomics technologies were evaluated. Under the project MAQC 1 and 2, microarrays were evaluated. Various issues related to the genome-wide genotyping arrays were evaluated in the MAQC 2 project . The thichnology -24. The P value cutoff led to increased consistency in differential gene expression analysis and downstream biological interpretation. The reference RNA samples chosen by the consortium have since become standard material widely adopted by the research community and the biotechnology industry for laboratory proficiency testing and development of new genomics technologies. As a natural progression, the MAQC 2 project studied the development and reliability of microarray-based predictive models for a variety of preclinical and clinical endpoints. Over 30,000 models were developed by 36 data analysis teams using numerous model building methods. Performance evaluation through a strictly blind external validation process demonstrated the utility of well-implemented internal cross validation in gauging the model prediction performance. This carefully designed and executed consortium effort with six large clinical and preclinical microarray datasets demonstrated that reliable predictive models can be developed when including sound and unbiased cross-validation techniques in the process. We expect the conclusions from the MAQC 2 project to be applicable to models based on gene expression data from other high-throughput technologies besides microarrays.The MAQC 1 project demonstrated inter- and intraplatform reproducibility of gene expression measurements by microarrays. The comprehensive study design was centered on cross-site cross-platform performance evaluation through the titration of two reference RNA samples. DNA microarray results were compared with the quantitative PCR platforms for gene expression, and high correlation was observed between them. Additionally, external RNA controls for the assessment of microarray performance were also evaluated, along with various microarray data normalization techniques. Importantly, MAQC 1 studies demonstrated that the combination of fold-change ranking and a non-stringent The next-generation sequencing (NGS) technologies were first introduced to the market in 2005 and have since seen tremendous growth in both technology advancements and research adoption. NGS has a wide spectrum of application in biomedical research including but is not limited to genome and exome sequencing, whole transcriptome sequencing , microRNA sequencing, and metagenomics. Some common challenges related to NGS include data storage, transfer, sharing, analysis, and visualization due to the sheer size of NGS datasets, which are referred to as big data challenges. As a tool, the specific application of NGS mainly defines the challenges and issues associated with this technique. Our literature survey on the use of NGS as a tool found that about 50% of the applications are mainly related to the use of NGS to understand genetic variations and their effect on disease and drug response. About a quarter of the applications are related to RNA-seq while the rest of them are split into various areas including microRNA sequencing and metagenomics. Challenges and issues associated with human genome sequencing differs greatly from these associated with microRNA sequencing because the size of the molecular object under investigation varies greatly, i.e., the human genome has 3.2 billion base pairs in contrast to the microRNA size of only 18–25 nucleotides.Relative measurement is more consistent than absolute measurement.We generated large datasets for six reference samples. The samples were sequenced in 11 different laboratories using various platforms   with mulRNA-seq vs microarrays.Among its broad application, RNA-seq has two major applications, first is to determine DEGs by comparing different conditions, e.g., treatment or disease status. The second use is to develop gene expression-based predictive models. However, microarrays have been used for a long time to perform similar tasks. Bioinformatics methods for analyzing and interpreting the results from microarray data have been assessed and established through the first two MAQC projects. In comparison, RNA-seq is a relatively new technology and analysis methods are continuously being developed. Thus, there is a great interest in the community to compare microarrays and RNA-seq to identify the benefits of using RNA-seq over microarrays. To address the comparison, the SEQC project implemented several studies to comprehensively assess the difference and similarity between these two technologies. In one of them, rat livers treated with 15 chemicals and matched controls were profiled with both technologies and the DEGs detected for each chemical were compared between the two technologies. Of note, these chemicals yielded a wide range of treatment effect with a 10-fold difference between the smallest and largest number of DEGs detected. With this design, we could evaluate the concordance in DEG analysis between RNA-seq and microarrays in various levels of treatment effect. We found that the concordance in DEGs between microarray and RNA-seq was positively correlated with the strength of treatment effect. Further analyses indicated that the discordance was mostly due to the difference between two platforms in quantifying the lowly expressed genes. Specifically, for highly expressed genes, we were able to achieve a concordance of about 75% while the concordance was only 35% for lowly expressed genes. Thus, the major difference between microarrays and RNA-seq lies in their accuracy of measuring lowly expressed genes. Further comparison with quantitative PCR indicated that RNA-seq would likely perform better than microarrays for lowly expressed genes .RNA-seq and Gene Discovery.An important potential of RNA-seq is its ability to discover novel, unannotated exon-exon junctions, which is affected by read depth. On increasing the number of reads from 10 million to 10 billion, both known genes and novel junctions were continually detected. Importantly, using quantitative PCR to validate some selected novel junctions, we determined that over 80% of them can be verified but their biological functions are unknown . This obPipeline for RNA-seq.Nature Methods.One of the most asked questions in the research community is which pipeline(s) is to be used for RNA-seq data analysis. To address the question in the context of big data, we evaluated 12 different pipelines in this project. For each pipeline, there are different parameter settings that lead to 278 major permutations covering the common gene modes, various quantification, and normalization methods. The comprehensive assessment was extremely costly in terms of computational time. We identified DEGs and compared the results with quantitative PCR and also evaluated the performance of downstream prediction models. We developed a composite metric including accuracy, precision, sensitivity in detecting lowly expressed genes, specificity in detecting DEGs, and prediction performance to derive the best practice for choosing RNA-seq data analysis pipelines. We observed that the pipeline giving the better estimation of the gene expression likely also gave better performance in predictive modeling. Multiple pipeline components jointly and significantly impacted the quantification of gene expression and downstream prediction performance. The manuscript is currently under review at Legacy microarray data in the RNA-seq era.Microarrays have been widely used in biomedical research and drug development since 1995. Major pharmaceutical companies usually generate thousands of microarrays per year. In this analysis, we tried to address whether RNA-seq-based gene signatures can be applied to microarray data to leverage the investment previously made. We tested three different classifier methods with three gene mapping categories to identify the transferability of microarray information to the RNA-seq data and vice versa. RNA-seq and microarrays were comparable for predictive models. Importantly, signature genes were reciprocally transferable between these two technology platforms. Microarray models can accurately predict RNA-seq-profiled samples. However, RNA-seq was less accurate in predicting microarray-profiled samples, and the performance was affected by modeling algorithms and the gene mapping complexity .Due to rapid advances in NGS technologies, the third phase of the MAQC Project was initiated while the second phase was still under the way. As mentioned above, this phase is also known as the SEQC project with its focus on RNA-seq. Over 180 participants from 73 different organizations across 12 different countries participated in the SEQC project. The project generated over 10 TB of data with over 100 billion reads. On submission of this dataset to the Gene Expression Omnibus (GEO) repository in June 2014, it represented around 6% of the total RNA-seq data in the repository at that time. This rich data provides ample opportunities for RNA-seq data analysis method development. Under this project, four different datasets were generated. The first dataset was generated from six reference samples. These reference samples were sequenced by various laboratories using different RNA-seq platforms such as Illumina HiSeq, Life Technologies SOLiD, and Roche 454. The second dataset was composed of sequencing data for about 500 neuroblastoma samples from pediatric patients. The third dataset was from 100 rat liver samples. The last dataset was a survey of rat transcriptomes using 11 different organs across 4 different developmental stages for both male and female rat. The SEQC project evaluated technical performance, quality control, and cross lab and cross platform reproducibility of RNA-seq. RNA-seq data was also compared with data generated from the same samples by mature microarray technologies. In addition to that, evaluations were made on the use of RNA-seq for clinical applications and safety assessments. The observations from these efforts were published in 10 manuscripts -30. HereNGS technologies have emerged as an important tool for many regulatory activities. Various FDA centers have encountered NGS data in regulatory science research and/or regulatory applications. These include but are not limited to (i) FDA oversight of NGS-based assays for diagnosis and prognosis, (ii) applying NGS in food pathogen identification and outbreak detection, (iii) reviewing NGS data for drug efficacy and safety for both clinical and preclinical assessments, and (iv) NGS as an improved tool for studying immunogenicity of vaccines. More specifically for biological products, NGS data can be utilized in various ways to support their development with one current major use being the identification of microbial contaminations .Building upon the success of the previous MAQC projects, which were fundamental for the development of FDA companion guidance to industry on pharmacogenomics data submission, we are in the process of developing a follow-up project, named SEquencing Quality Control Phase 2 (SEQC2). SEQC2 aims to develop quality control metrics and benchmark bioinformatics approaches for the analysis of the whole genome sequencing and targeted gene sequencing data to achieve best practices, to develop standard analysis protocols, and to apply these newer methods in regulatory settings. The ultimate goal of SEQC2 is the development of standards for using NGS data that will provide the FDA with objective criteria and metrics for data quality assessment that can be applied in regulatory settings and to provide information for precision medicine.In summary, the primary aim of these FDA-led efforts for emerging genomics technologies is to engage the stakeholders and research community for consensus building with respect to the reliable use of genomics data with objective criteria and assessment metrics for data quality and reliability, which can be employed in the FDA for their fit-for-purpose application.


Typically, gene expression biomarkers are being discovered in course of high-throughput experiments, for example, RNAseq or microarray profiling. Analytic pipelines that extract so-called signatures suffer from the "Dimensionality curse": the number of genes expressed exceeds the number of patients we can enroll in the study and use to train the discriminator algorithm. Hence, problems with the reproducibility of gene signatures are more common than not; when the algorithm is executed using a different training set, the resulting diagnostic signature may turn out to be completely different.In this paper we propose an alternative novel approach which takes into account quantifiable expression levels of all genes assayed. In our analysis, the cumulative gene expression pattern of an individual patient is represented as a point in the multidimensional space formed by all gene expression profiles assayed in given system, where the clusters of "normal samples" and "affected samples" and defined. The degree of separation of the given sample from the space occupied by "normal samples" reflects the drift of the sample away from homeostasis in the course of development of the pathophysiological process that underly the disease. The outlined approach was validated using the publicly available glioma dataset deposited in Rembrandt and associated with survival data. Additionally, the applicability of the distance analysis to the classification of non-malignant sampled was tested using psoriatic lesions and non-lesional matched controls as a model.Keywords: biomarkers; clustering; human diseases; RNA The typical application of gene expression signatures for diagnosis and prediction of the course of disease is based upon an oversimplified understanding of the pathology. According to this model, there is a gene or a set of genes  that is "responsible" for a pathophysiological process within certain tissue, or a cell type, that manifests itself on an organismal level as a disease. If we see this gene over- or underexpressed, or observe a set of concerted changes in expression of a set of genes, we can diagnose the disease. We can further use the respective levels of the over- or underexpression to predict the course of the disease.per se, but with a typically limited number of already diagnosed patients we could biopsy and enter into the analysis as a training data set. As shown by simulations in   5] for bp is substantially larger than the sample size n. This property of biological datasets makes the task of distinguishing the noise from the true biological signal quite challenging, and it becomes close to impossible to obtain consistent estimator procedures [Expression levels for individual genes and other variables quantified in high- throughput biological experiments are commonly thought of as dimensions of the space on which we are collecting information. Thus the problem outlined above is known as the "curse of dimensionality" . Brieflyocedures ,10. Hencocedures ,10 or stocedures : startinocedures . Hence, ocedures . MoreoveAnother popular solution to the dimensionality curse is to use a supervised approach that relies either on the pre-selection of the feature-limiting steps or on pre-filtering the data by the strength of an association of each variable with clinical outcome, or associations between variables ,15. UnfoIn this paper we propose an alternative novel approach based on the "distances" in the multidimensional space of gene expression values. As a proof-of-principle, we show that this approach produces surprisingly good results in separation of normal and affected samples both for analysis of human malignancies and for chronic progressive conditions like psoriasis.A result of an expression experiment for a given sample is a (very long) vector that may be represented as a point in a multidimensional space. If we introduce a distance in this space, we can use the standard clustering techniques  to classThere are several ways to define the distance between two points, ssion in ). The naIt takes all components of vectors equally and does not account for any correlations between them. We can normalize the sums in equation (1) on the average vector lengthwhere the averaging denoted by angular brackets is taken over all vectors in the dataset. Then the scaled distance is defined asIn the gene expression experiments the absolute values of the vectors are not relevant since the expressions are arbitrarily normalized. Therefore, using a distance that does not take these values into account is justified. A correlation distance, or Pearson distance, is the one that has this property. It is defined asIf we have enough data to calculate covariance matrix S with the elements obtaining by the averaging over all vectors in the dataset their covariances:them we can use Mahalanobis distanceThis distance takes into account all correlations in the data. The problem with it is that to calculate correlation matrix and its inverse, many data points, i.e. many patients, are required.Therefore, in the calculations below we use, as a rule, Pearson distance (4). Note that since all components of |X| and |Y| are non-negative, this distance is always between 0 and 1.To test for the practical usefulness of the distance-based expression metrics we deployed the following strategy:http://www.ncbi.nlm.nih.gov/geo/.1 The datasets were selected from public MIAME-compliant GEO repository 2 For each subset of the samples within data set  we calculated the coordinate for the center of the space defined by points of all the vectors as the simple arithmetic mean of all the samples in the subset.3 For each point we calculated the distance to the centers of all subsets.r1 vs. the distance to another center r2 were plotted.4 The distances to one center r1, r2 and r3, each plotted to the center of the corresponding subset. In these cases we used barycentric coordinates [i dfrom the vortex i proportional toIn some datasets, three different subsets may be defined, instead of two. In these cases, each point was associated with three distances rdinates  in a equi D= 1 − di. The transformation between these two definitions is trivial:Note that most programs dealing with barycentric coordinates  use a dTo test the hypothesis that the expression profiles of primary tumors and their metastases drift away from the healthy, homeostatic state, the RNAseq dataset with GEO Series accession number GSE46622 described in  was downAccordingly we had three different subsets and three different centers of clusters. These data may be presented either as the points within a 3d cube Figure , or as bIn a practical sense, one may be interested in how far away given tumor sample departed from the center of the space occupied by normal samples. This approach allows one to draw a linear plot Figure . Intereshttp://caintegrator.nci.nih.gov/rembrandt/, which included data on 21 normal samples, 221 glioblastoma multiforme (GBMs), 145 astrocytomas, 66 oligodendrogliomas and 11 tumors of mixed origin. The raw gene expression CEL files from Affymetrix HGU133 Plus 2.0 arrays were normalized using the robust multi-array average (RMA) method [To prove this point, we downloaded data represented in publicly available Repository for Molecular Brain Neoplasia Data (Rembrandt) ) method  with defp = 3.1 × 10−6) or oligodendrogliomas (p = 0.0033). In both astrocytomas and oligodendrogliomas, the observed spreads of the distance values were quite large, possibly reflecting known heterogeneity of these tumors [The plots reflecting the distances to the center of normal samples for all studied groups of samples are shown on Figures e tumors ,23.p = 2 × 10 −9 and R2 = 0.09. This means that the dependence is statistically significant, but the degree of the drift away from the center of the normal space explains only 9% of the survival prognosis, while the remaining 91% is explained by other factors, most likely, particular chromosomal rearrangements and mutations observed within the tumor along with tumor location and other known determinants of glioma prognosis [Interestingly, patients survival were found to be negatively correlated with the distance of the tumor expression profile to the center of normal samples: the farther was the expression profile from the homeostatic center defined by normal samples, the shorter was the survival length for the patient. The corresponding plot is shown on Figure rognosis .N of paired samples = 13) and GSE11903 [N of paired samples = 15). Both datasets were created using Affymetrix Human Genome U133A Array platform. As could be seen on Figures To illustrate an applicability of the gene expression distance analysis to the classification of samples collected from patients with non-malignant chronic disease, we selected psoriasis, an autoimmune disease mainly diagnosed through the visual inspection of the lesion skin by experienced dermatologists. To analyze the properties of gene expression profiles of lesional and non-lesional psoriatic skin samples from the same patient, two datasets were downloaded from GEO, GSE6710   by a large number of reference  samples, as farther away the respective tissue will be from the well maintained, healthy state. In our study, we used publicly available datasets, to develop easily interpretable, composite measure that capable of integrating high-throughput transcriptome profiles into comprehensive, holistic metric describing the molecular homeostasis within given sample.The comprehensive distance measures account for the intrinsic heterogeneity of human tumors that plagues hight-throughput studies involving this type of the biological material  and evenImportantly, proposed composite biomarker may be suitable for a dynamic description of patients' condition. This novel concept allows one to depart from the classical two-bin prediction model (e.g. "bad prognosis/good prognosis") as it produces a continuous prognosis model, where each sample is located in the neighborhood of other samples analyzed post-hoc and associated with known survival. For each sample, this concept quantitatively describes the degree of "the drift" from the standardized phenotype that will reflect the departure of the body from homeostasis. In the concepts, the effects of each personalized intervention could be evaluated by comparing the distance metrics for samples collected before the treatment and at multiple time-points within the interventional treatment course.If proven valid, this concept might be developed into a novel type of integrative tests for the monitoring of the disease progress and the prediction of disease outcomes. The proposed distance analysis has a potential to become versatile in its application as it is equally attributable to gene expression profiles collected both by microarrays and by RNA-seq platforms, as well as, possibly, to proteome and metabolome profiles.There is no doubt that proposed computational approach requires further development and optimization, in particular, other types of correlation-based metrics have to be tested for various kinds of multiparametric datasets that comprise simultaneously measured analytes. Future studies should include an analysis of longitudinal experiments that involve either various time points in course of the therapeutic treatment that ultimately results in the normalization of the pathological condition, or gradual processes detrimental to experimental system, for example, a development of insulin resistance or an ageing.The distance analysis of molecular portraits is robust and versatile in its application as it is equally attributable to gene expression profiles collected by microarrays and by RNA-seq. The distance-based continuous predictive models depart from the classical two-bin prediction model (e.g. "bad prognosis/good prognosis") by placing each sample in the neighborhood of other samples analyzed post-hoc and associated with known survival.The authors declare that they have no competing interests.BV drafted the manuscript, provided insights concerning dimensionality curse and mathematical advice in the course of this study. AB conceived this study and developed the concept of distance metrics. LW and TC analyzed both cancer datasets and contributed to the draft of the manuscript. SB analyzed psoriasis dataset and contributed to the development of the concept of distance metrics.