Urologic cancers include cancers of the bladder, kidney, prostate, and testes, with common molecular features spanning different types. Here, we show that 1954 urologic cancers can be classified into nine major genomic subtypes, on the basis of multidimensional and comprehensive molecular characterization . Tissue dominant effects are first removed computationally in order to define these subtypes, which reveal common processes—reflecting in part tumor microenvironmental influences—driving cellular behavior across tumor lineages. Six of the subtypes feature a mixture of represented cancer types as defined by tissue or cell of origin. Differences in patient survival and in the manifestation of specific pathways—including hypoxia, metabolism, NRF2-ARE, Hippo, and immune checkpoint—can further distinguish the subtypes. Immune checkpoint markers and molecular signatures of macrophages and T cell infiltrates are relatively high within distinct subsets of each cancer type studied. The pan-urologic cancer genomic subtypes would facilitate information sharing involving therapeutic implications between tissue-oriented domains. Urological cancers have disparate tissues and cells of origin but share many molecular features. Here, the authors use multidimensional and comprehensive molecular characterization to classify urological cancers into nine major genomic subtypes, highlighting potential therapeutic targets. Over the last few years, The Cancer Genome Atlas (TCGA) carried out a number of genomic studies each focusing on an individual cancer type6, where classification or subtyping based on multiple molecular data platforms was typically performed. Pan-cancer analysis by TCGA of an initial set of 12 different cancer types  found cancers to segregate largely on the basis of cancer type as defined by tissue of origin, with one notable exception being the “squamous” genomic subtype that spanned multiple cancer types, including lung squamous, head and neck squamous, and a subset of bladder cancers7.Cancers are typically classified based on the tissue site of origin, coupled with observable histologic features, with subsequent therapeutic decisions then following the histologic classification. At the molecular level, cancers associated with a given tissue type may represent a heterogeneous group of diseases. Unsupervised approaches to subtyping human tumors on the basis of molecular profiling data provide a tool in adding to our understanding of the genomic landscape of cancer. Genomic subtypes of cancer as defined by molecular analysis may represent disease subsets being driven by distinct pathways and processes. Data platforms for messenger RNA (mRNA) expression in particular have been used extensively in the molecular classification of cancers8. Urologic oncology is an established specialty within medical practice, as urologic cancers arise within the urinary tract of men and women and reproductive organs of men. Urologists have a key role in the diagnosis and treatment of all of these malignancies. However, in terms of treatment, it is well understood that the different cancer types as defined by tissue of origin would represent quite distinct diseases from each other on the basis of several factors, including histologic appearance, the presence of distinct driver mutations, varying clinical course, and different responses to systemic therapy. Within kidney cancers, the three major types—clear cell, papillary, and chromophobe—are quite distinct from each other at the histologic and molecular levels and are understood to represent truly different cancer types that warrant individual study10. At the same time, common oncogenic processes are thought to underlie cancers of different types12.Urologic cancers include cancers of the bladder, kidney, prostate and testes, all relatively common, with prostate cancer, for example, being the most common cancer in American men13. With the recent conclusion of the data generation phase of TCGA, systematic analyses of the entire TCGA urologic cancer cohort would allow for comparisons and contrasts to be made between the different diseases represented. Many of the molecular differences that exist among urologic cancer types—including bladder, prostate, kidney clear cell, kidney chromophobe, kidney papillary, and testicular—could arise from their respective cells and tissues of origin10. Understanding the mechanisms driving cancer subtypes is challenged by these lineage-specific molecular signals15. Although cancer sample profiles would tend to segregate by associated tissue type, part of these tissue dominant effects can be removed computationally to elucidate common processes driving cellular behavior across tumor lineages.TCGA genomic data sets—which include DNA, RNA, protein, and epigenetic data—provide a major opportunity to develop an integrated picture of commonalities, differences and emergent themes across tumor lineagesIn this study, we define nine pan-cancer molecular-based subtypes, which would transcend tumor lineage across the nearly 2000 urologic cancer cases profiled by TCGA. Data involving multiple molecular profiling platforms are used to both define these subtypes, and to characterize them in terms of associated pathways. Common processes and pathways shared across multiple cancer types include hypoxia, metabolism, NRF2-ARE, Hippo, and immune checkpoint. Subtype-specific patterns as initially observed in TCGA are also observable in an external and independent expression data set of urologic cancers.n = 412 cases); KICH, corresponding to kidney chromophobe (n = 66); KIRC, corresponding to kidney renal clear cell carcinoma (n = 537); KIRP, corresponding to kidney renal papillary cell carcinoma (n = 291); PRAD, corresponding to prostate adenocarcinoma (n = 498); and TGCT, corresponding to testicular germ cell tumors (n = 150).TCGA collected a total of 1954 primary urologic cancer specimens , across cancer types can serve to illustrate our overall approach to molecular subtyping , normalizing the expression within each cancer type would serve to subtract out such differences . Overall differences between cancer types would include any differences that would be specific to the cell- or tissue-of-origin or to the tissue microenvironment. Molecular features that show interesting patterns of variation within two or more cancer types can serve to identify cancer subsets that would span tumor lineage. In an unsupervised clustering analysis, based on the aggregated patterns of 2000 genes most variable across the cancer types, the urologic cancer cases were separated almost exclusively on the basis of cancer type when using mRNA data based on absolute expression  of predominantly BLCA, KICH, and KIRP cases ; a c2 subtype (n = 333) of predominantly PRAD and BLCA cases ; a c3 subtype (n = 106) of predominantly KIRP cases (72%), with additional KIRC, PRAD, and KICH cases ; a c4 subtype (n = 250) of predominantly PRAD and TGCT cases ; a c5 subtype (n = 269) of predominantly KIRC and KIRP cases ; a c6 subtype (n = 268) of KIRC, KIRP, and TGCT cases ; a c7 subtype (n = 202) of predominantly KIRC cases (98%); a c8 subtype (n = 192) of predominantly BLCA cases (99%); and a c9 subtype (n = 115) of PRAD cases (100%).The nine genomic subtypes of TCGA urologic cancers Table  included10—differed by subtype  cases were primarily subdivided between the c1, c3, c5, and c6 subtypes, with c5 associating with Type 2 histology and related “P-e.2” KIRP genomic subtype, and the other subtypes associating with Type 1 histology and related KIRP genomic subtypes. The TCGA BLCA (bladder) cases were primarily subdivided between the c1, c2, and c8 subtypes, with c1 associating with papillary histology and “luminal” expression-based subtype and with c8 associating with a “squamous” pan-cancer subtype7. The TCGA PRAD (prostate) cases were primarily subdivided between the c2, c4, and c9 subtypes, which largely corresponded respectively to the “iCluster2/DNA methylation3”, “iCluster1/DNA methylation2”, and “iCluster3/DNA methylation4” genomic subtypes previously associated with PRAD5. TCGA TGCT (testicular) cases were subdivided between c4 and c6, representing “seminoma” (69 out of 70 TGCT.c4 cases) and “non-seminoma” (62 out of 80 TGCT.c6 cases) histology, respectively.The nine genomic subtypes made across the entire TCGA urologic cancer cohort showed high concordance with other histologic or genomic subtype designations, which had been previously called for a subset of these cases in studies focusing on a specific cancer type17 Fig. . The TCGP < 1E-10 by Log-rank test). Within KIRP cases, c5 (associated with papillary type 2 histology) showed the worst patient survival (similar to that observed for KIRC.c5 cases), c6 showed the best patient outcome , and c3 and c1 showed intermediate survival . Within BLCA cases, c1 (associated with bladder papillary histology) was associated with better outcome, while c2 and c8 were associated with worse outcome . Prostate cancer cases in an external gene expression profiling data set18  were assigned one of the three PRAD-associated subtypes, on the basis of overall similarities in expression patterns; c9 cases showed overall better survival as compared to the other cases . In line with observations made in our previous study10, widespread DNA hypermethylation patterns  indicated differential activation of specific pathways between disease subsets. Previously, aggressive kidney clear cells cancers (associated here with our c5 pan-urologic subtype) demonstrated evidence of a metabolic shift, involving downregulation of genes involved in the TCA cycle, decreased AMPK protein levels, upregulation of the pentose phosphate pathway, and increased acetyl-CoA carboxylase proteinrns Fig. . Other mrns Fig. .Fig. 5DiVHL gene mutations that lead to stabilization of hypoxia-inducible factors such as HIF-1α3. We examined whether hypoxia-associated expression patterns may be relevant across other cancer subsets in addition to clear cell kidney. Analysis of relevant gene transcription signatures22 suggested higher activation of hypoxia, epithelial-mesenchymal transition (EMT), MAP Kinase, NRF2-ARE, and cyclin D1 within the c5, c8, and c9 subtypes in particular , CD247 (CD3), CD274 (PDL1), PDCD1LG2 (PDL2), CTLA4 (CD152), TNFRSF9 (CD137), TNFRSF4 (CD134), and TLR9. Analysis of gene expression signatures from Bindea et al.34 suggested that levels of immune cell infiltrates were also highest in c5, c8, and c9 subtypes  indicated the presence of tumor-associated macrophages and of immune response pathways within disease subsets. In one approach, we examined the top 900 differential mRNAs  roles in the tumor microenvironmentlar Fig. . In thisway Fig. , wherebyn = 31 cases), prostate cancer (n = 83), clear cell renal cell carcinoma (n = 205), and papillary renal cell carcinoma (n = 22) were obtained. Within each cancer type, genes in the expO data set were first normalized, and then each expO tumor profile was classified by pan-urologic genomic subtype as defined by TCGA data  data set (Gene Expression Omnibus accession number GSE2109), mRNA expression profiles of bladder cancer , nor would a common “driver” mutation be found within the different cancer types comprising a pan-urologic subtype. Somatic DNA alterations may serve to help drive microenvironmental-associated phenomenon, e.g., VHL mutations in clear cell kidney promoting expression of hypoxia-inducible genes. However, gene mutation information alone may be insufficient to predict pathway-level alterations and microenvironmental influences as identified through integrated molecular analysis. Across the c4, c5, c8, and c9 pan-urologic subtype in particular, global differences suggested the involvement of macrophages and lymphocytes. Current understanding implicates macrophages in complex and diverse tumor-promoting roles within the tumor microenvironment, and tumor-associated macrophages would represent a potential therapeutic target35. Our results would suggest an intriguing hypothesis, whereby specific subtypes of urologic cancers  would be most responsive to such a therapeutic approach.Commonalities observed within a number of our pan-urologic subtypes would appear to have more to do with microenvironmental influences , than with somatic mutation patterns. While the six major cancer types examined here share the same anatomic region, the tissues and cells composing each organ would be quite different. The different tissue-based cancer types would not necessarily share common etiologic factors or mutational signatures37, Hippo38, metabolism39, and immune checkpoint41. Reprogramming of energy metabolism and evading immune destruction in particular have gained attention in recent years as general hallmarks of cancer12. Interestingly, when diverse cancer types are analyzed in terms of absolute levels of gene expression, kidney clear cell cancers as a group score among the highest in terms of immune cell infiltrates and checkpoint pathway expression40. However, within several other cancer types—including prostate, bladder, and kidney papillary cancers—there is clearly a range in scores observable across a set of cases40. In our study, the changes underlying the immune infiltrates and related pathways involve coordinate expression of large numbers of genes, indicative of the associated processes being systematically at work within distinct subsets of each cancer type analyzed. At what levels would the relative differential patterns observed denote real therapeutic responses in the clinical setting remains to be determined, though previous studies indicate that immunogenicity of tumors cannot be explained by mutation load or neo-antigen load, and that expression-based markers should also be considered in the clinical setting42. The metabolic shift identified here as being associated with poor prognosis kidney cancers represents a Warburg-like metabolic phenotype 3; such aspects of cancer cell metabolism are critical for tumor maintenance in this disease subset, yet are less likely to be relevant in normal cells, representing yet another potential therapeutic target39.The molecular differences represented by our pan-urologic genomic subtypes would point to pathways spanning tumor lineage and having implications for targeted therapy, including NRF2-AREhttp://cancergenome.nih.gov/). Cancer molecular profiling data were generated through informed consent as part of previously published studies6 and analyzed in accordance with each original study’s data use guidelines and restrictions. In total, 1954 urologic cancer cases assayed on at least one molecular profiling platform  were included in the analysis. Somatic mutation calls were obtained from the publicly available “MC3” TCGA MAF file (https://www.synapse.org/#!Synapse:syn7214402). All other molecular, clinical and pathological data are available through the TCGA Data Commons (https://gdc.nci.nih.gov/). For purposes of unsupervised clustering and downstream correlative analyses, with the three expression platforms , values for each gene were centered to s.d. from the median within each cancer type ; for DNA methylation platform, beta values were centered to the median within each cancer type.Multiplatform genomics data sets were generated by TCGA Research Network  first filtered out normal samples from the segmented copy-number data by inspecting the TCGA barcodes and then executed GISTIC (Firehose task version: 140). We calculated copy number-based clusters for the combined pan-urologic cancer cohorts, using ConsensusClusterPlus R-package44 to identify clusters in the data using 1000 iterations, 80% sample resampling from 2 to 15 clusters (k2 to k15) using hierarchical clustering with Ward as the linkage algorithm and Pearson correlation as the similarity metric, with an input log2 tumor:normal data set . Consistent with what was carried out for the other platforms, the k = 7 clustering solution was selected for further investigation. High-level copy gain or copy loss events for individual genes were inferred using Firehose’s “thresholded by genes” results table .DNA from each tumor or germline-derived sample had been previously hybridized by TCGA to Affymetrix SNP 6.0 arrayshttps://www.synapse.org/#!Synapse:syn7214402). This MC3 set is a re-calling of uniform files from all TCGA projects, with variant calling using a standardized set of mutation callers. The BAM files used underwent a standardized local re-alignment to hg19 (Genome Reference Consortium GRCh37), six calling algorithms were applied, and a number of automated filters were applied. Variants called by two or more algorithms were used in the study. For an additional 150 cases not represented in MC3 data set but with exome data featured in Chen et al.10, variants calls from the Chen study were included in the present study. Silent mutations (point mutations that would not result in a change in the amino acid) were not included in the results. Except for FGFR, KIT, and MET genes, mutations in oncogenes  were represented in figures, if the mutations occurred in “hotspot” residues as reported by Chang et al.45; all nonsilent mutations in putative tumor suppressor genes  were represented in figures.Somatic mutation calls were obtained from the publicly-available “MC3” TCGA MAF file . To correct for batch effects between data platforms (HM450 vs. HM27), we used the combat software46.DNA methylation profiles had been previously generated by TCGA using either the Illumina Infinium HumanMethylation450 (HM450) or HumanMethylation27 (HM27) BeadChips 44 was used to identify clusters in the data using 1000 iterations, 80% sample resampling from 2 to 15 clusters (k2 to k15) using hierarchical clustering with Ward as the linkage algorithm and Pearson correlation as the similarity metric. Consistent with what was carried out for the other platforms, the k = 7 clustering solution was selected for further investigation.For the unsupervised clustering analysis, we removed from consideration DNA methylation probes located on X or Y chromosomes and probes not included on the HM27 platform. For unsupervised clustering, we selected the top 2000 probes having the higher average variability (by s.d.) across the six projects , and centered the methylation beta values to the median within each cancer type . ConsensusClusterPlus R-packageCDKN2A; as done previously 10, a beta value of 0.2 or above was considered evidence for epigenetic silencing. For each sample, a total DNA methylation index value was computed as the fraction of methylation probes  having beta value>0.3.The DNA methylation level as interrogated by cg13601799 was used for 3. Expression of coding genes was quantified for 20,531 features based on the gene models defined in the TCGA Gene Annotation File (GAF) (https://tcga-data.nci.nih.gov/tcgafiles/ftp_auth/distro_ftpusers/anonymous/other/GAF/GAF_bundle/outputs/). Expression values within each cancer type  were normalized to s.d. from the median across tumors; other than the analysis in main Fig. k = 2 to k = 15 subtypes, with the stability of the clustering increasing with increasing k. Consistent with what was carried out for the other platforms, the k = 7 clustering solution was selected for further investigation.RNA sequencing data had been previously generated by TCGA18, classifying each external tumor profile by genomic subtype as defined by TCGA data. As a classifier, the top set of 900 mRNAs distinguishing between the nine COCA subtypes in TCGA was used. For the three COCA subtypes with high representation of PRAD cases , the average centroid for each gene was computed, based on the centered TCGA expression data matrix 3, using either the Illumina GAIIx or HiSeq 2000 platforms. To help correct for batch effects between data platforms (GAIIx vs. HiSeq), we used the combat software46. Expression values within each cancer type  were normalized to s.d. from the median across tumors. For unsupervised clustering analysis, we selected the top 500 most variable genes, according to average s.d.  across the six projects . ConsensusClusterPlus R-package44 was used to identify clusters in the data using 1000 iterations, 80% sample resampling from 2 to 15 clusters (k2 to k15) using hierarchical clustering with Ward as the linkage algorithm and Pearson correlation as the similarity metric. Consistent with what was carried out for the other platforms, the k = 7 clustering solution was selected for further investigation.MiRNA sequencing data had been previously generated by TCGA3. Raw data (level 1), SuperCurve nonparameteric model fitting on a single array (level 2), and loading corrected data (level 3) were deposited at the DCC. RPPA data were obtained from The Broad Institute’s Firehose pipeline, which included data for 1570 urologic cancer cases. Expression values within each cancer type  were normalized to s.d. from the median across tumors. For unsupervised clustering , ConsensusClusterPlus R-package44 was used to identify clusters in the data using 1000 iterations, 80% sample resampling from 2 to 15 clusters (k2 to k15) using hierarchical clustering with average linkage algorithm and Pearson correlation as the similarity metric. Consistent with what was carried out for the other platforms, the k = 7 clustering solution was selected for further investigation.RPPA data were previously generated by TCGA47 MAPK signature . For computing RPPA-based pathway score, all proteins levels were first normalized to s.d. from the median within each cancer type.RPPA profiles were also scored for a previously defined16 to define integrated subtypes. For the k = 9 COCA subtype solution, we defined the top differential genes associated with each subtype; we first computed the two-sided t-test for each gene, comparing each subtype with the rest of the tumors, then selected the top 100 genes with the lowest P-value for each subtype.As described above, urologic cancer cases were subtyped according to each of the individual data platforms for DNA methylation, DNA copy alteration, mRNA expression, miRNA expression, and protein expression. Subtypes defined from each platform were coded into a series of indicator variables for each subtype, with the matrix of 1 and 0 s then clustered by a Cluster of Cluster Analysis (COCA)16, as well as outlined here. To computationally infer the infiltration level of specific immune cell types using RNA-seq data  methods examining lymphocyte-specific expression patterns within cancer vs. non-cancer cellular compartments40. For scoring TCGA cancer samples for each of these immune cell signatures, the average of the gene expression values (transformed within each cancer type to s.d. from the median) was used. In addition, samples were scored for expression of Antigen Presentation MHC class I (APM1) genes  and for Antigen Presentation MHC class II (APM2) genes.Tumor expression profiles were scored for gene signatures associated with pathway deregulation essentially as previously describedata Fig. , we used27 were analyzed using a previously utilized approach16, also outlined here. Gene expression profiles from various normal human and mouse tissues27 were obtained from the FANTOM5 data repository (http://fantom.gsc.riken.jp/5/data/); for our study, profiles from fetal or embryonic human specimens were removed from the analysis. The top 900 differential mRNAs by COCA subtype  between the fantom and TCGA profiles was used to represent a summary measure of overall similarity between the two groups . Cell cycle genes were from Whitfield et al.48. Gene targets of p53 were from ref49. Gene transcription signature scores of NRF2-ARE pathway where generated as described22, on the basis of four different signatures: “Malhotra” signatures50, which combined expression profiling and Chip-seq of mouse embryonic fibroblasts (MEFs) with either constitutive nuclear accumulation (Keap1−/−) or depletion (Nrf2−/−) of Nrf2, including genes downregulated in Nrf2−/− vs. wild-type and Nrf2 bound, and genes upregulated in Keap1−/− vs. wild-type and Nrf2 bound; “GSE28230,” from Gene Expression Omnibus data set of A549 adenocarcinoma lung cancer cells with siRNA knockdown of NRF2 ; “Osburn,” from GSE11287 data set of mouse liver with or without Keap1 knockout used ; across the urologic cancer profiles, we normalized the individual gene signature scores to s.d. from the median across samples, and a “summary score” for NRF2-ARE pathway was computed as the average of the four individual normalized signature scores. EMT signature scores were computed as previously described51 . Hypoxia signature was based on the set of canonical HIF1A targets from Harris19. Cyclin D1 and MAPK signatures were from Creighton21. Gene transcription signature scores of Yap1 pathway where generated as described here, on the basis of three different signatures: from GSE32567, genes differentially expressed  with knockdown of Yap1 in hepatocellular carcinoma cell line (SK-Hep1); from GSE49406, genes differentially expressed  with knockdown of Yap1 in HEK293 cells; from GSE7700, genes differentially expressed  with knockdown of Yap1 in normal breast luminal cell; across the urologic cancer profiles, we normalized the individual Yap1 gene signature scores to s.d. from the median across samples, and a “summary score” for Yap1 targets was computed as the average of the three individual normalized signature scores.Gene transcription signature scores associated with pathway  were computed as follows. For each gene in the TCGA pan-urologic cancer data set , expression values within each cancer type were centered on the median of tumor sample profiles and divided by the s.d. across tumors. For cell cycle, p53, EMT, NRF2-ARE, hypoxia, and KEGG signatures, the average expression of the set of genes within a given signature were computed. For cyclin D1, MAP Kinase, and Yap1 signatures, urologic cancer expression profiles  were scored for the above signatures using our previously described “t-score” metricWe examined an external gene expression profiling data set of multiple cancer types from the Expression Project for Oncology (expO) GSE2109), classifying each external tumor profile by genomic subtype as defined by TCGA data. Within each cancer type, genes in the expO data set were normalized to s.d. from the median. As a classifier, the top set of 900 mRNAs distinguishing between the nine COCA subtypes in TCGA  and the Broad Instititute’s Firehose data portal (https://gdac.broadinstitute.org). Somatic mutation calls made from TCGA whole exome sequencing data are available from synapse (https://www.synapse.org/#!Synapse:syn7214402). Additional molecular profiling data sets utilized as described above are available via the Gene Expression Omnibus.All data used in this study are publicly available. TCGA data are available through the Genome Data Commons (Supplementary InformationPeer Review FileSupplementary Data 1Supplementary Data 2Supplementary Data 3Supplementary Data 4Supplementary Data 5


Due to experimental batch effects, the application of a quantitative transcriptional signature for disease diagnoses commonly requires inter-sample data normalization, which would be hardly applicable under common clinical settings. Many cancers might have qualitative differences with the non-cancer states in the gene expression pattern. Therefore, it is reasonable to explore the power of qualitative diagnostic signatures which are robust against experimental batch effects and other random factors.Firstly, using data of technical replicate samples from the MicroArray Quality Control (MAQC) project, we demonstrated that the low-throughput PCR-based technologies also exist large measurement variations for gene expression even when the samples were measured in the same test site. Then, we demonstrated the critical limitation of low stability for classifiers based on quantitative transcriptional signatures in applications to individual samples through a case study using a support vector machine and a naïve Bayesian classifier to discriminate colorectal cancer tissues from normal tissues. To address this problem, we identified a signature consisting of three gene pairs for discriminating colorectal cancer tissues from non-cancer  tissues based on within-sample relative expression orderings (REOs) of these gene pairs. The signature was well verified using 22 independent datasets measured by different microarray and RNA_seq platforms, obviating the need of inter-sample data normalization.Subtle quantitative information of gene expression measurements tends to be unstable under current technical conditions, which will introduce uncertainty to clinical applications of the quantitative transcriptional diagnostic signatures. For diagnosis of disease states with qualitative transcriptional characteristics, the qualitative REO-based signatures could be robustly applied to individual samples measured by different platforms.The online version of this article (10.1186/s12864-018-4446-y) contains supplementary material, which is available to authorized users. In clinical, biopsy sampling with less-invasive techniques such as colonoscopy and endoscopic ultrasound-guided fine needle aspiration is often used for the initial clinical evaluation of cancer –6. HowevWith the wide application of high throughput gene expression profiling techniques, many classifiers based on quantitative transcriptional signatures for cancer subtyping –12 or eaNotably, among the vast number of reported quantitative disease signatures, several signatures have been approved by the Food and Drug Administration (FDA). One of the FDA approved signatures is MammaPrint® for predicting the recurrence risk of early stage (I and II) breast cancer patients with lymph node negative and tumor size < 5.0 cm treated with surgical resection –30. HoweIn contrast, it has been found that the within-sample relative expression orderings (REOs) of gene pairs, which is also called Relative Expression Analysis (RXA) , are robIn this article, in addition to the previous results for the high-throughput platforms reported by the MicroArray Quality Control (MAQC) project , we firsFirstly, we evaluated the CV of gene expression measurements in replicates for sample A and sample B measured in the same test site by two PCR-based technologies, StaRT-PCR™ Assays and TaqMan® Assays, respectively.For a total of 199 genes with non-zero measurements assayed by StaRT-PCR™ for 3 replicates of sample A, about 32.7% genes showed at least 10% CV and 15.1% genes showed at least 15% CV. Similarly, for a total of 195 genes with non-zero measurements assayed by StaRT-PCR™ for 3 replicates of sample B, about 34.4% genes showed at least 10% CV and 17.4% genes showed at least 15% CV, the results were also shown in Fig. For a total of 964 genes with non-zero measurements assayed by TaqMan® for sample A, about 13.1% genes showed at least 10% CV and 7.8% genes showed at least 15% CV. Similarly, for a total of 905 genes with non-zero measurements assayed by TaqMan® for sample B, about 10.7% genes showed at least 10% CV and 6.2% genes showed at least 15% CV, as shown in Fig. Due to large experimental batch effects, quantitative transcriptional measurement data from different experiments or profiled with different platforms could not be directly put together to train traditional SVM and naïve Bayesian classifiers. Because we could not find a single dataset with sufficient samples for colorectal cancer, normal and IBD tissues simultaneously, we were unable to train SVM and naïve Bayesian classifiers based on quantitative measurements for discriminating colorectal cancer and non-cancer  tissue samples. Thus, we constructed the SVM and naïve Bayesian classifiers for a simpler problem, discriminating colorectal cancer and normal tissue samples, to demonstrate the limitations of quantitative transcriptional signatures.t-test with 1% FDR control. Using these 7028 genes as feature genes, a SVM classifier with radial basis function (RBF) kernel was trained with tenfold cross-validation [Between the 32 cancer samples and 32 normal samples from dataset GSE8671, 7028 differentially expressed genes were detected using Student’s lidation , 55 usinMore comprehensive evaluation results were shown in Supplementary Result and Additional file k pairs, where k is an odd integer, to classify samples according to the majority vote rule. The results showed that for all possible k values ranging from 1 to 141, the largest geometric mean of sensitivity and specificity was 94.8% when k = 3  pairs from the 141 gene pairs to classify samples according to the majority vote rule. With different top-rank k, similar performances were achieved in the validation datasets, as shown in Additional file k increased to 77 or larger. The possible reason of the decreased performance for GSE68570 should be that the gene pairs with relatively low reversal rates in the training data might be unstable in data measured by other platforms [Moreover, to explore the generalization of the signature, we used all the possible top-ranked latforms . In geneWe demonstrated that, besides high-throughput gene expression profiling platforms, the low-throughput PCR-based quantitative measurements also exist large variation in replicate samples measured in the same or different test sites. Thus, the classifiers based on quantitative transcriptional signatures could not be robustly applied to individual samples measured by the same platform as the training samples, let alone those individual samples measured by different platforms. This could explain the problem mentioned in Introduction that some quantitative transcriptional signatures approved by FDA or incorporated into clinical guidelines must be sent to a central laboratory for measurement with strict quality control and data normalization. Besides the batch effects, the quantitative measurements of gene expression are commonly affected by partial RNA degradation  and diffFortunately, as demonstrated in our previous studies , 20, 36 GPAT3 and TRIP13, PYY and CKAP2, SDCBP2 and DAP3. These genes were found in the differentially expressed genes  detected between the 32 cancer samples and 32 normal samples in the GSE8671 dataset. For GPAT3-TRIP13 gene pair, both up-regulation of TRIP13 and down-regulation of GPAT3 contribute to the reversal REO in colorectal cancer samples. Similarly, for PYY-CKAP2 and SDCBP2-DAP3 gene pairs, up-regulation of CKAP2, DAP3 and down-regulation of PYY, SDCBP2 contribute to the reversal REO in colorectal cancer samples. Some of these genes, such as TRIP13 [PYY [TRIP13 is a novel mitotic checkpoint-silencing protein, whose overexpression is associated with poor prognosis in breast cancer patients [PYY may be relevant to the development and progression of colon adenocarcinoma [GPAT3 - TRIP13 gene pair, as shown in Fig. GPAT3 and TRIP13 took values ranging from 1.26 to 1.72 with the median of 1.38 in the normal samples, while in the tumor samples the fold change took values ranging from 0.63 to 1.06 with the median of 0.87. Similar results for the other two gene pairs, PYY - CKAP2 and SDCBP2 - DAP3, were shown in Fig. Even with sufficient high-quality data, it is difficult to interpret the signature used in complex classifiers to gain biological insights about the biomarkers . In conts TRIP13 , PYY [57patients , 58, 59.The REO-based method is based on a single binary “switch” that compares the ordering of expression between two genes. The simplicity does not necessarily limit its prediction performance and the method is not prone to the overfitting issue. Arguably, REO-based signatures may lose some subtle quantitative information on gene expression. However, considering that subtle quantitative information of gene expression measurements tends to be unreliable and even the ratios of expression values of gene pairs are affected by the batch effects , 60, theThe main purpose of this study is to systematically demonstrate the critical limitations of the traditional classifiers based on the quantitative transcriptional measurements, which are sensitive to batch effects and detection platforms and could not be applied directly to the data measured by different laboratories. As for the REO-based method, other approaches, such as TSP and k-TSP, could be applied to the data measured by different laboratories or platforms. Here, we additionally evaluated other rank based approaches using the same training and validation datasets. Using the tspair R package (version 3.3.3), we trained the TSP classifier in the training samples directly combined from data measured by the Affymetrix and Illumina platforms. In the training set, 97.0% of the 428 cancer samples and 94.3% of the 385 non-cancer sample were correctly identified. However, the classifier failed badly in many validation datasets as described in Additional file In conclusion, REO-based signatures circumvent the critical limitation of quantitative transcriptional signatures and the REO-based classifying method should be also applicable for classifying other tissue samples. Moreover, because the data normalization problem also exists in miRNA  and DNA Because the subtle quantitative information of gene expression measurements currently tends to be greatly affected by many random factors, the disease diagnostic signatures based on the quantitative measurements lack robustness for clinical applications. Thus, we should make more efforts to capture the qualitative differences of gene expression patterns between cancer and non-cancer and between cancer subtypes to exploit robust qualitative signatures for disease diagnosis.http://www.ncbi.nlm.nih.gov/geo/) and the mRNA-seq data measured by the Illumina platform were downloaded from ArrayExpress [http://www.ebi.ac.uk/arrayexpress/) and The Cancer Genome Atlas [http://cancergenome.nih.gov/).The gene expression profiles analyzed in this study are described in Table yExpress   and used the Robust Multi-array Average (RMA) algorithm for background adjustment without quantile normalization . For thehttps://biodbnet-abcc.ncifcrf.gov/db/db2db.php).For the array-based data, each probe ID was mapped to Entrez gene ID with the corresponding platform file. If a probe was mapped to multiple or zero genes, then the data of this probe were deleted. If multiple probes were mapped to the same gene, the expression value of the gene was defined as the arithmetic mean of the values of multiple probes. For the sequence-based data from ArrayExpress, the gene symbols were mapped to Entrez gene ID with the biological database network   project, two commercially available Reference RNA samples (sample A and sample B) with multiple replicates were measured by multiple microarray platforms and PCR-based technologies . The MAQThe MAQC PCR-based data, as described in Table For the gene expression levels of a certain gene in the replicates for sample A or sample B measured by each platform, the coefficient of variation (CV), calculated as the ratio of the standard deviation and arithmetic mean for the expression levels of this gene in the replicates, is used to measure the degree of variation of quantitative measurements. For the sample A and sample B measured by each platform, we calculated the percentage of genes that shows at least 10% and 15% CV, respectively, to reveal the degree of variation or uncertainty of quantitative measurements.The SVM classifier using radial basis function (RBF) kernel  and the We called cancer samples as positive samples, non-cancer samples, either normal or IBD, as negative samples, and evaluated the performance of the classification signature using sensitivity and specificity which are calculated as follows:i, j), the relationship of their relative ranks, Gi and Gj, within one sample, has only two possibilities, Gi > Gj or Gi < Gj. The relationship is called the relative expression ordering (REO). If the same REO pattern is maintained in a majority of samples, e.g. 90%, it is called a highly stable REO and the pair is a highly stable gene pair. Furthermore, if a gene pair  is highly stable in both a group of non-cancer samples and a group of cancer samples, respectively, but with reversal REO patterns (Gi < Gj in one group but Gi > Gj in the other group), the pair is called a reversal gene pair. Here, we selected the reversal gene pairs which are highly stable in non-cancer samples and cancer samples, respectively, but the REO patterns are reversed in the latter group. They form the candidate REO signature of the cancer.First, in the training dataset, each gene measurement is converted to its rank within each sample . Then, pairwise comparisons are performed for all genes to identify gene pairs with stable ordering in samples for a particular tissue type. For a pair of genes Additional file 2: Figure S1.k  gene pairs in the independent datasets measured by different platforms. The majority vote rule was used for classification. (TIFF 1848 kb)Performances of all possible top-ranked


High throughput mRNA expression profiling can be used to characterize the response of cell culture models to perturbations such as pharmacologic modulators and genetic perturbations. As profiling campaigns expand in scope, it is important to homogenize, summarize, and analyze the resulting data in a manner that captures significant biological signals in spite of various noise sources such as batch effects and stochastic variation. We used the L1000 platform for large-scale profiling of 978 representative genes across thousands of compound treatments. Here, a method is described that uses deep learning techniques to convert the expression changes of the landmark genes into a perturbation barcode that reveals important features of the underlying data, performing better than the raw data in revealing important biological insights. The barcode captures compound structure and target information, and predicts a compound’s high throughput screening promiscuity, to a higher degree than the original data measurements, indicating that the approach uncovers underlying factors of the expression data that are otherwise entangled or masked by noise. Furthermore, we demonstrate that visualizations derived from the perturbation barcode can be used to more sensitively assign functions to unknown compounds through a guilt-by-association approach, which we use to predict and experimentally validate the activity of compounds on the MAPK pathway. The demonstrated application of deep metric learning to large-scale chemical genetics projects highlights the utility of this and related approaches to the extraction of insights and testable hypotheses from big, sometimes noisy data. The effects of small molecules or biologics can be measured via their effect on cells’ gene expression profiles. Such experiments have been performed with small, focused sample sets for decades. Technological advances now permit this approach to be used on the scale of tens of thousands of samples per year. As datasets increase in size, their analysis becomes qualitatively more difficult due to experimental and biological noise and the fact that phenotypes are not distinct. We demonstrate that using tools developed for deep learning it is possible to generate ‘barcodes’ for expression experiments that can be used to simply, efficiently, and reproducibly represent the phenotypic effects of cell treatments as a string of 100 ones and zeroes. We find that this barcode does a better job of capturing the underlying biology than the original gene expression levels, and go on to show that it can be used to identify the targets of uncharacterized molecules. PLOS Computational Biology Methods Paper.This is a Pharmacology is generally explored in a linear and iterative manner, starting from the observation of an activity, that is then optimized for selectivity and potency via various methods, and subsequently tested in preclinical and randomized controlled clinical studies for safety and efficacy. In general, the activity of compounds on targets other than the intended one(s) is limited, though such ‘off-target’ activities may lead to adverse events. The detection of polypharmacology is desired and could be facilitated by comprehensive profiling of drug candidates on a general phenotyping platform, i.e. a consistent method to broadly assess the phenotypic effects of compound treatment. Such a platform could be used in order to uncover unexpected phenotypic signals beyond the originally identified mechanism-based phenotype. The inference of mode of action or mode of toxicity from gene expression changes has been extensively studied in the past –3. Diffeversus a target-based approach lies in the identification of the target(s) of molecules that show an activity in cell-based  assays [Phenotypic screening is considered by some to be a return to pharmacology’s roots , and by ) assays . A gener) assays .Lastly, a general phenotyping platform will allow one to monitor compounds through their maturation and optimization in order to prioritize series based on selectivity and to quickly identify potential polypharmacology and safety warning signals .We suggest that mRNA is a promising analyte for a general phenotyping platform, although the domain of applicability remains to be fully understood. Whereas gene expression changes are often distal to signaling and metabolic pathways that drug discovery aims to modulate, most perturbations of cellular pathways eventually lead to the nucleus , and to The L1000 platform  has the However, if such a platform is applied for large sets of perturbations, spanning years of different project stages and various programs, then data analysis, and particularly homogenization, become important. Large scale expression profiling projects such as the Connectivity Map  and applHerein we present a novel method of representing the expression profiles of the L1000 platform as a short binary barcode. The approach starts by training a deep model that learns to distinguish replicate from nonreplicate profiles. The internal state of the model thus learned demonstrated properties previously ascribed to deep neural network models, namely a hierarchical representation that captures the regularities and underlying structure of the input data , 18. TheWe first introduce the data generation process and highlight the need for improved analyses. The disparity between replicate profiles motivates the development of a metric learning approach that improves upon the current state of the art for this type of data analysis.The experimental approach used in this work is depicted in versus control. Looking at the similarity of each sample profile to every other in the dataset, the median rank of a sample’s biological replicate is approximately the 3rd percentile. While this ranking can be considered an enrichment (versus a null hypothesis of ~50th percentile), the 3rd percentile in our dataset nevertheless implies that >200 treatment profiles are more similar to a given treatment than a sample that was treated identically on a different day.Large scale gene expression profiling is known to have significant ‘batch’ effects that confound interpretation . These eAs mentioned above, the z-score data is confounded by batch effects that reduce the apparent similarity of expression responses deriving from identical treatments. In order to improve on the performance of the z-scores, another strategy that was explored is that of gene set enrichment analysis . GSEA iSince there is room for improvement over both of the previously described methods, we sought to evaluate alternate strategies to increase the sensitivity and specificity of interpretations and predictions derived from relatively large data sets that we are currently generating on the L1000 platform. In particular, as a first step, we sought to increase the concordance of samples that should exhibit the same phenotypic response, while separating them from those that should be distinct. Using the similarity of biological replicates as a gold standard, the goal was to recast the data in a way that maximizes the similarity of replicates in contrast to non-replicate samples. This can be thought of as a metric learning problem, whereby one learns a new measure of distance or similarity between points, one that distinguished replicates from non-replicates. Initial experimentation with linear and kernel methods  failed tDeep learning is an extension of decades-old artificial neural networks, and has recently shown impressive performance on a number of image and speech recognition tasks , 24. As Most machine learning methods consider a single input sample at a time, though each sample is generally represented by multiple variables, or features. The goal of this effort was to learn not about individual samples, but about the comparison of pairs of inputs in order to learn a new metric that better captures what is known about the data. Thus, a method was chosen that permits comparative learning in a neural network framework. We implemented a version of a siamese neural network , 26 to pA number of model architecture properties  need to be set prior to the optimization of model weights of a neural network by gradient descent. These include number of layers, layer sizes, activations of layers, regularization types and weights, and cost type and any parameters. We undertook a greedy manual search of hyperparemters, optimizing the validation set accuracy for predicting replicate vs. nonreplicate pairs. Significantly, other characteristics of the model and its representation, such as those described below and in versus the known target status of the pair (i.e. replicates or not replicates). The cost is used to train the network via backpropagation of the cost gradients [The final model that emerged from optimization of the neural network architecture performs the following: a pair of 978-dimensional z-score vectors representing two different expression profiles is given to the network as input, the data is transformed by two layers of noisy sigmoid layers, and then the representations of the members of the pair are compared to each other by calculating a Euclidean distance. A margin loss is calculated based on the distance radients . While sradients , 26), in1 = 0.87, [1 = 0.33). A nontrivial reference for comparison would be a model that thresholds the gene expression changes for each sample to generate a binary 978-dimensional representation. Depending on whether the threshold is shared among all genes or is gene specific, and on whether the threshold is applied to raw expression changes or absolute changes, a variety of models can be created. The best performance in terms of accuracy on the validation data was observed for a model that uses gene-specific thresholding on the original scale, with thresholds learned from the training data set, this model gave a test set accuracy of 67% (F1 = 0.66). Another nontrivial model one could apply would use random projections of the data as is done in locality sensitive hashing [1 = 0.50). Optimizing the projections to maximize training set discrimination yields a linear model. A 100-dimensional linear model yielded an accuracy of 71% (F1 = 0.47). These results show that the learned deep model is superior in classification accuracy to trivial and nontrivial comparator models.The network was trained on 80% of the compounds, using a validation held-out set of 10% to select model architecture and detailed configuration (hyperparameters), and a final test set of 10% was used to estimate generalization performance of the selected model. The trained model was able to correctly categorize 97% of test pairs as replicates/non-replicates . As a b hashing . Such a Having demonstrated that it is possible to recast the data in a way that captures the similarity of identically treated samples, we set out to determine if the learned model can be used to improve interpretability of large scale expression profiling campaigns. Since the model can separate replicates from nonreplicates, and since the model output is a function of a simple Euclidean distance calculated on the internal representation (hidden layer activations), we reasoned that this internal representation captures the discriminatory power learned by the model. We explored whether this internal representation has additional useful properties, as for example do the internal word representations of skip-gram language models  to repreThus the model yields a new encoding of the original data, a representation that was designed only to increase the similarity of biological replicates relative to non-replicates. Information is necessarily lost in reducing 978-dimensional continuous data to 100-dimensional binary representations. The question that remains is whether biologically interesting aspects of the data are retained in the simplified encoding.Thus, we evaluated whether the model described herein learns useful, generalizable aspects related to compound effects on cells as a byproduct of learning to identify replicates. The perturbation barcodes were assessed for their ability to identify similarities between compounds in a variety of contexts . FurtherThe median rank (out of 7573) of each sample’s biological replicate is shown in Next, one might ask whether a model trained to recognize samples treated with the same compound would also retain additional relationships related to biological function and chemical similarity. In order to avoid re-discovering the similarity of replicates, for this and subsequent comparisons, replicates were resolved to a single representation by averaging the hidden layer activations of the replicate samples before thresholding to barcodes, and analogously, z-scores and GSEA results represent the average of replicates.versus unshared targets, the comparison on the z-score data is not significant, with a t statistic of -1. The GSEA-summarized data is more significant with t~-38, while the barcode data shows the highest preferential similarity of profiles of compounds sharing a target annotation: t~-43. This indicates that the latter two methods are able to encode the expression profiles of compounds in a way that emphasizes biological similarity.We tested the ability of the different representations to recognize different compounds targeting the same cellular factor as having similar profiles. Due to compound-specific polypharmacologies, one shouldn’t expect any pair of distinct compounds to have identical profiles, but one should expect compounds sharing a target to have profiles that are more similar, on average, than pairs of compounds not sharing a molecular target. We looked at those compounds for which target annotation was available (1297 compounds having annotated targets modulated with potency < 1μM), and asked if compounds that share a target are more similar to each other in induced gene expression profiles than compounds that do not. Using a t statistic for the difference of mean distances of shared A well-studied approach to categorize compounds is by similarity of chemical structure, with one underlying assumption being that similarity of structure is related to similarity in function. Thus molecules are clustered using descriptors designed to allow molecules considered similar by a chemist to be grouped together, to the exclusion of dissimilar molecules . AnalogoNext we set out to determine whether the barcode reveals insight into biological function. In order to compare two different measures of biological activity, the profile of 3471 compounds across multiple High Throughput Screens (HTS) was compared to the L1000 expression profiles. The activity profiles across HTS assays could be thought of a biological fingerprint, as it represents the activity of the compound in a multitude of disparate assays, both cell-based and biochemical. Such profiles do not evenly span all potential biological responses since they opportunistically derived, depending on the history of each compound and the organization performing the HTS campaigns. However, in some ways the HTS profiles are more relevant to drug discovery since many of the constituent assays directly measure the molecular activities of interest rather than induced transcriptional programs. The correlation matrix of HTS fingerprints  of the 2) was used as measure of the ability of the models to predict properties of compounds on the basis of the profiling data . Promiscuity of compounds is of interest in drug discovery, wherein modulators that are both potent and specific (i.e. non-promiscuous) are generally sought. One way to ascertain if promiscuity information is present in the different expression profile representations is to assess the ability of each representation to generate a model that can quantitatively predict this property for samples not in the training set. Predictive support vector regression models were trained on each data representation, and the average cross-validation coefficient of determination (Ring data . Given tversus top 27% for z scores). Furthermore, average profiles of distinct compounds sharing a target are significantly closer to each other in the perturbation barcode space than in the original space: for each annotated target of compounds in the dataset, the distances of compounds sharing versus not sharing the target were compared via a t-statistic, and the mean of these statistics was found to be non-significantly higher (+1) for the z-score data but significantly smaller (-37) for the barcodes , but low efficacy (Emax~41%), thus not meeting the activity threshold. The additional actives discovered only in the barcode map demonstrate that the visualization derived from the barcodes has increased sensitivity to capture biological activities versus the raw data-derived visualization. Significantly, none of the newly identified AP-1 reporter inhibitors were structurally similar to the known actives (Dice similarity less than 0.6).The predictive fidelity of the t-SNE map derived from expression z-scores was compared to that of a t-SNE map derived from the perturbation barcodes in order to ascertain how much information one can hope to glean by studying the visualization of a large dataset . This usFinally, we looked at the neighborhood of the same known MAPK actives in the original data space for both the z-scores  and the barcodes  rather than the reduced 2-dimensional t-SNE space. Nearest neighbors were chosen by Euclidean distance and tested in the reporter assay as above. We present here a straightforward way to extract features from gene expression data that are in many ways more expressive and robust than the original gene expression changes. While training of a deep network is a somewhat specialized process, once an adequate model is trained, additional samples can efficiently be converted to perturbation barcodes with no (or little), further optimization needed, as evidenced by the simple transfer of the model to external (LINCS) data.Others have proposed novel methods for analyzing L1000 data. In particular, Liu et al  presenteWe demonstrate that similarity in the barcode space is indicative of more similarity in compound target and compound structure, activity across biological assays, and predictivity of biological action. It should be pointed out that even with the improved data representation, the level of concordance between the gene expression data and reference data , remains imperfect. There are a number of potential explanations for this. Firstly, the gene expression experiments were conducted in one or two cell lines per compound, and for some compounds the (known or unkown) efficacy targets are not expressed, thus limiting the potential correlation to target annotation. Additionally, many compounds were unoptimized screening hits or lightly characterized tools, and thus likely have significant polypharmacology, only some of which is known, and only some of which is consistent within chemotypes due to ‘activity cliffs.’ Next, target annotation is incomplete, so an apparent false positive association between a compound and an activity may in fact be a genuine connection that has not yet been discovered. Lastly, there is little doubt that obtaining higher data density , would complement the sparse dataset explored herein, and in doing so, possibly tie more compounds to benchmark annotations than was possible with the current data.We made the somewhat unexpected observation that models of general compound properties like promiscuity can be better built using the learned features than the data that the features were learned from, indicating that there are positive side effects of the compressive data encoding. We attribute this phenomenon to the denoising property of the learned features, or equivalently, extraction of robust underlying biological factors from the corrupted versions observed in experimentation .We anticipate that elaborations of approaches such as this one will be fruitful for other applications in biological and chemical domains, as they have been in artificial intelligence. A more ambitious goal of algorithmic design of molecules may be based on the combination of phenotypic information, quantitative structure-activity relationships, and pharmacokinetic/pharmacodynamic models with generative models for chemical structures. In the meantime, the direct application of the described metric learning-based representation technique to other high volume, high dimensional data has the potential to significantly reduce the effects of noise and to improve interpretability and the quality of generated hypotheses.80 was used. For compounds without relevant cell-based assay activity, the highest non-toxic dose was used, up to 20μM. Two cell lines were used: PC3 and ME180. A number of compounds were profiled in both cell lines, the rest were profiled in just one of the two. Most compounds were profiled as biological replicates by performing identical treatments and lysis on separate days, some compounds were only represented by a single instance. Compound treatment time was 6 hours, after which cells were lysed, frozen, and shipped to Genometry for processing. A total of 3699 compounds were screened. Compounds were selected based on being tool bioactive compounds, active compounds for ongoing phenotypic screening programs, and compounds of interest for particular compound optimization programs. Compound profiling was performed in four independent campaigns over the course of 14 months.Doses for treatment were chosen using the following hierarchy of approaches: for compounds with known biological activity and toxicity, the dose with the highest therapeutic window was selected. For compounds with only biological activity, the EC2 Genechip-equivalent intensities and normalized based on control gene intensities. Finally, the data on each plate was standardized based on the median and median absolute deviation of the vehicle control samples to calculate z-scores for each gene. These z-scores were the input for the deep metric learning, and were also analyzed directly as a baseline for data interpretation.Initial data processing was conducted by Genometry using a standard pipeline. Briefly, Luminex intensity measurements were assessed for consistency of relative expression of control genes. Samples passing well and plate level thresholds were summarized by conversion of intensity data to calculated logA metric learning cost module that takes consecutive pairs of training examples and calculates a distance between them, and thereby a cost, was implemented using the Pylearn2 framework . InternaThe model uses an input layer of 978 z-scores, followed by two hidden layers of 400 and 100 units, using a noisy sigmoid nonlinearity:Wx + b) before applying sigmoid nonlinearity σ. Noise was added to favor saturated (0/1) activations. Dropout [p = 0.5) and L1 weight decay (λ = 10−5–10−6) are also used in hidden layers for regularization. Finally, the cost layer uses a rectified loss function with a squared distance margin of 5:Independent Gaussian noise with mean 0 and variance 0.25  Dropout  , d2 is the squared Euclidean distance between hidden vectors. Theano [Here . Theano  automati. Theano  framewor. Theano  algorithAfter training the model, the weights were extracted and used to generate the activation state of the last hidden layer for each input sample. While noise was added during training to regularize the model and encourage saturation, no noise was added during barcode generation in order to yield a deterministic transformation. Perturbation barcodes are generated by thresholding activations of the last hidden layer to create a binary representation. See ‘Availability of supporting material’ for software.For comparison, several baseline models were created to compare replicate recognition accuracy to that of the neural network model. A model that randomly emits 1/-1 with probabilities 0.33/0.67 is a naïve model that predicts replicates independent of input with a frequency that matches that of replicate pairs in the training data. A series of models that threshold the gene expression data to generate binary representations  can also be made. The thesholding was applied to the original scale , or on the absolute value scale . The thresholds were either set to be equal for all genes or were selected separately for each gene. Some hashing approaches use random projections of the data, in order to determine if the learned neural model is superior to this approach, a 100 random projection model was tested for classification of pairs as replicates vs. nonreplicates. Finally, the random projections can be replaced with learned linear projections optimized to minimize classification error, also known as a linear model. The above models with tunable parameters , were learned using gradient descent, minimizing margin error on the training set, and tested on the test set.GSEA was conducted using 6034 gene sets derived from experimental gene signatures of up- and down-regulated genes curated by Nextbio . Enrichment was measured using a Wilcoxon test, and the resulting rank sum test z statistic was used as a score for a given gene set’s enrichment in a sample.Data downloaded from the NCBI GEO repository, accession GSE70138. The file GSE70138_Broad_LINCS_Level4_ZSVCINF_mlr12k_n78980x22268_2015-06-30.gct.gz was downloaded in December 2015. This dataset represents 273 compounds tested in 6-point dose response at two time points in 15 cell lines. The level 4 (z-score vs. vehicle control) data was utilized. The first 978 landmark (i.e. measured) gene measurements of the data matrix were analyzed. In order to train the perturbation barcode model on the LINCS data, a subset of the data was selected: 10 and 1 μM treatments at 24 hours. 80% of the data was used for training, and 20% for validation, and the same hyperparameters were used for the model (see section Deep Metric Learning). After the model was trained, the entire dataset was encoded with the perturbation barcodes learned from the training set. Targets for 149 of the compounds were found to be annotated in Chembl  or MetabCrossvalidated support vector regression was performed with the caret package , clusterFor visualization, data, either z-scores or barcodes, were reduced to two dimensions using the t-distributed stochastic neighbor embedding (t-SNE ) algoritCompound structures were clustered using in-house fragment-based descriptors , and hieMerck’s chemogenomic database, the Chemical Genetic Interaction Enterprise (CHEMGENIE) was used to annotate compounds with their known targets. CHEMGENIE contains harmonized data from external  sources as well as internal sources  where compounds are represented as desalted InChIKeys and targets are represented with their Entrez Gene ID and/or UniProt Accession. All target-based dose response data was added to each compound, and the highest affinity target was selected as a representative target for each compound.HTS fingerprints were constructed as described in the past . Brieflyt-SNE visualizations of z-scores representing the normalized L1000 data, and also of the 100-dimensional (nonthresholded) perturbation barcodes derived from the metric learning network were generated. The locations of known EGFR/MEK/MAPK inhibitors on these maps were plotted, and points (i.e. representing compound treatments), that were surrounded by these known actives were selected for testing from each of the two maps. Separately, in order to select actives from the full 978-dimensional z-score or 100-dimensional barcode space, the ten nearest neighbors (by Euclidean distance) of each of the seed compounds were identified, and available compounds were tested for activity. Compounds were tested in dose titration assay in 1536 well plate format using a previously optimized reporter gene assay. Cells  were plated in the manufacturer’s recommended assay medium at 3000 cells/well in 9μl in black/clear tissue culture treated 1536 well plates (Greiner), and allowed to adhere overnight. Compounds were added from DMSO serial dilution plates , using a 50nl pintool (GNF Systems), and the cells incubated 30 min in a tissue culture incubator. The cells were then stimulated with Epidermal Growth Factor  at 10ng/ml final concentration by adding 1μl 10x stock, and allowed to respond for 5 hours in a tissue culture incubator. Beta-lactamase detection reagents , were added per manufacturer’s instructions (2μl 6x mix), plates were incubated 2 hours at room temperature, and read using a bottom-reading multimode reader . Data was normalized to no-stimulation (100% inhibition), and stimulation + DMSO (0% inhibition), controls, and percent inhibition was plotted along with logistic regression curve fits using Spotfire (Tibco). Compounds were classified as active if they exceeded 50% inhibition of reporter activity without toxicity at concentrations below 10μM.https://github.com/matudor/siamese.Code for performing deep metric learning and a demonstration of the analysis on the LINCS data are available at S1 TextFigures showing sample correlation, sample space depiction, and details of prospective prediction results.(PDF)Click here for additional data file.


Molecular signatures identified from high-throughput transcriptomic studies often have poor reliability and fail to reproduce across studies. One solution is to combine independent studies into a single integrative analysis, additionally increasing sample size. However, the different protocols and technological platforms across transcriptomic studies produce unwanted systematic variation that strongly confounds the integrative analysis results. When studies aim to discriminate an outcome of interest, the common approach is a sequential two-step procedure; unwanted systematic variation removal techniques are applied prior to classification methods.MINT, that simultaneously accounts for unwanted systematic variation and identifies predictive gene signatures with greater reproducibility and accuracy. In two biological examples on the classification of three human cell types and four subtypes of breast cancer, we combined high-dimensional microarray and RNA-seq data sets and MINT identified highly reproducible and relevant gene signatures predictive of a given phenotype. MINT led to superior classification and prediction accuracy compared to the existing sequential two-step procedures.To limit the risk of overfitting and over-optimistic results of a two-step procedure, we developed a novel multivariate integration method, MINT is a powerful approach and the first of its kind to solve the integrative classification framework in a single step by combining multiple independent studies. MINT is computationally fast as part of the mixOmics R CRAN package, available at http://www.mixOmics.org/mixMINT/and http://cran.r-project.org/web/packages/mixOmics/.The online version of this article (doi:10.1186/s12859-017-1553-8) contains supplementary material, which is available to authorized users. High-throughput technologies, based on microarray and RNA-sequencing, are now being used to identify biomarkers or gene signatures that distinguish disease subgroups, predict cell phenotypes or classify responses to therapeutic drugs. However, few of these findings are reproduced when assessed in subsequent studies and even fewer lead to clinical applications , 2. The One way to increase sample size is to combine raw data from independent experiments in an integrative analysis. This would improve both the statistical power of the analysis and the reproducibility of the gene signatures that are identified . HoweverThe common approach to integrative classification is sequential. A first step consists of removing batch-effect by applying for instance ComBat , FAbatchThe major pitfall of the sequential approach is a risk of over-optimistic results from overfitting of the training set. This leads to signatures that cannot be reproduced on test sets. Moreover, most proposed classification models have not been objectively validated on an external and independent test set. Thus, spurious conclusions can be generated when using these methods, leading to limited potential for translating results into reliable clinical tools . For insMINT. MINT is the first approach of its kind that integrates independent data sets while simultaneously, accounting for unwanted (study) variation, classifying samples and identifying key discriminant variables. MINT predicts the class of new samples from external studies, which enables a direct assessment of its performance. It also provides insightful graphical outputs to improve interpretation and inspect each study during the integration process.To address existing limitations of current data integration approaches and the poor reproducibility of results, we propose a novel Multivariate INTegrative method, Basal, HER2, Luminal A and Luminal B). We use these datasets to demonstrate the reproducibility of gene signatures identified by MINT.We validated MINT in a subset of the MAQC project, which was carefully designed to enable assessment of unwanted systematic variation. We then combined microarray and RNA-seq experiments to classify samples from three human cell types (human Fibroblasts (Fib), human Embryonic Stem Cells (hESC) and human induced Pluripotent Stem Cells (hiPSC)) and from four classes of breast cancer (subtype X denote a data matrix of size N observations (rows) ×P variables  and Y a dummy matrix indicating each sample class membership of size N observations (rows) ×K categories outcome (columns). We assume that the data are partitioned into M groups corresponding to each independent study m: {(X(1),Y(1)),…,(XM)} so that nm is the number of samples in group m, see Additional file Xm)( and Ym)( and Ym)),…Y from a data matrix X by maximising a formula based on their covariance. Specifically, latent components are built based on the original X variables to summarise the information and reduce the dimension of the data while discriminating the Y outcome. Samples are then projected into a smaller space spanned by the latent component. We first detail the classical PLS-DA approach and then describe mgPLS, a PLS-based model we previously developed to model a group (study) structure in X.PLS approaches have been extended to classify samples PLS-DA Partial Least Squares Discriminant Analysis [Y is a dummy matrix indicating sample class membership. In our study, we applied PLS-DA as an integrative approach by naively concatenating all studies. Briefly, PLS-DA is an iterative method that constructs H successive artificial (latent) components th=Xhah and uh=Yhbh for h=1,..,H, where the hth component th (respectively uh) is a linear combination of the X (Y) variables. H denotes the dimension of the PLS-DA model. The weight coefficient vector ah (bh) is the loading vector that indicates the importance of each variable to define the component. For each dimension h=1,…,H PLS-DA seeks to maximize Analysis  is an exXh,Yh are residual matrices . The PLmgPLS Multi-group PLS is an extension of the PLS framework we recently proposed to model grouped data [X and Y. These global loading vectors thus allow the samples from each group or study to be projected in the same common space spanned by the PLS-components. We extended the original unsupervised approach to a supervised approach by using a dummy matrix Y as in PLS-DA to classify samples while modelling the group structure. For each dimension h=1,…,H mgPLS-DA seeks to maximize ped data , 23, whiah and bh are the global loadings vectors common to all groups, ah,bh) and global components  enable to assess overall classification accuracy, while the group-specific loadings and components provide powerful graphical outputs for each study that is integrated in the analysis. Global and group-specific components and loadings are represented in Additional file where mentclass2pt{minimMINTsimultaneously integrates independent studies and selects the most discriminant variables to classify samples and predict the class of new samples. MINT seeks for a common projection space for all studies that is defined on a small subset of discriminative variables and that display an analogous discrimination of the samples across studies. The identified variables share common information across all studies and therefore represent a reproducible signature that helps characterising biological systems. MINT further extends mgPLS-DA by including a ℓ1-penalisation on the global loading vector ah to perform variable selection. For each dimension h=1,…,H the MINT algorithm seeks to maximize Our novel multivariate integrative method λh is a non negative parameter that controls the amount of shrinkage on the global loading vectors ah and thus the number of non zero weights. Similarly to Lasso [ℓ1 penalisation in MINT improves interpretability of the PLS-components that are now defined only on a set of selected biomarkers from X (with non zero weight) that are identified in the linear combination ℓ1 penalisation in effectively solved in the MINT algorithm using soft-thresholding (see pseudo Algorithm 1).where in addition to the notations from Eq. , λh is to Lasso  or sparsto Lasso , the addIn addition to the integrative classification framework, MINT was extended to an integrative regression framework  to each test sample by using the maximal distance, as described in [Ytest.After scaling the test samples, the prediction framework of PLS is used to estimate the dummy matrix t Xtest , where eribed in . It consλh for each PLS-component h, which is usually performed using Cross-Validation (CV). In practice, the parameter λh can be equally replaced by the number of variables to select on each component, which is our preferred user-friendly option. The assessment criterion in the CV can be based on the proportion of misclassified samples, proportion of false or true positives, or, as in our case, the balanced error rate (BER). BER is calculated as the averaged proportion of wrongly classified samples in each class and weights up small sample size classes. We consider BER to be a more objective performance measure than the overall misclassification error rate when dealing with unbalanced classes. MINT tuning is computationally efficient as it takes advantage of the group data structure in the integrative study. We used a “Leave-One-Group-Out Cross-Validation (LOGOCV)”, which consists in performing CV where group or study m is left out only once m=1,…,M. LOGOCV realistically reflects the true case scenario where prediction is performed on independent external studies based on a reproducible signature identified on the training set. Finally, the total number of components H in MINT is set to K−1, K= number of classes, similar to PLS-DA and ℓ1 penalised PLS-DA models [The main parameter to tune in MINT is the penalty A models .MINT to identify the true positive genes on the MAQC project, then highlight the strong properties of our method to combine independent data sets in order to identify reproducible and predictive gene signatures on two other biological studies.We demonstrate the ability of The MicroArray quality control (MAQC) project. The extensive MAQC project focused on assessing microarray technologies reproducibility in a controlled environment [ironment . Two refStem cells. We integrated 15 transcriptomics microarray datasets to classify three types of human cells: human Fibroblasts (Fib), human Embryonic Stem Cells (hESC) and human induced Pluripotent Stem Cells (hiPSC). As there exists a biological hierarchy among these three cell types, two sub-classification problems are of interest in our analysis, which we will address simultaneously with MINT. On the one hand, differences between pluripotent (hiPSC and hESC) and non-pluripotent cells (Fib) are well-characterised and are expected to contribute to the main biological variation. Our first level of analysis will therefore benchmark MINT against the gold standard in the field. On the other hand, hiPSC are genetically reprogrammed to behave like hESC and both cell types are commonly assumed to be alike. However, differences have been reported in the literature [terature –28, justMINT against existing statistical methods to solve a rather ambitious classification problem.The stem cell dataset provides an excellent showcase study to benchmark Each of the 15 studies was assigned to either a training or test set. Platforms uniquely represented were assigned to the training set and studies with only one sample in one class were assigned to the test set. Remaining studies were randomly assigned to training or test set. Eventually, the training set included eight datasets (210 samples) derived on five commercial platforms and the independent test set included the remaining seven datasets (132 samples) derived on three platforms ), to efBreast cancer. We combined whole-genome gene-expression data from two cohorts from the Molecular Taxonomy of Breast Cancer International Consortium project  to clasignature . The METignature . TCGA coignature .The training set consisted in three cohorts (TCGA RNA-seq and both METABRIC microarray studies), including the expression levels of 15 803 genes on 2 814 samples; the test set included the TCGA microarray cohort with 254 samples , ComBatng . For LMng  in every single platform are true positive. We primarily focused on identifying biomarkers that discriminate C vs. D, and report the results of A vs. B in the Additional file −3 ), which MINT selected a single gene, BCAS1, to discriminate the two biological classes C and D. BCAS1 was a true positive gene, as part of the common DEG, and was ranked 1 for Illumina, 158 for AffyPrime and 1182 for AffyHuGene. Since the biological samples C and D are very different, the selection of one single gene by MINT was not surprising. To further investigate the performance of MINT, we expanded the number of genes selected by MINT, by decreasing its sparsity parameter see , and com−5, ANOVA, Additional file A meta-analysis of eight stem cell studies, each including three cell types  showed that methods that do not accommodate unwanted variation were able to rightly classify most of the samples from the training set, but failed at classifying any of the four subtypes on the external test set. As a consequence, all samples were predicted as MINT outperformed their non sparse counterparts LM+PLSDA and mgPLS . The popular method ComBat took 7.1s to run, and sPLS-DA 0.9s. The training step on the breast cancer data that includes 2817 samples and 15,755 genes was run in 37s for MINT and 71.5s for ComBat(30.8s)+sPLS-DA(40.6s).Finally, One of the main challenges when combining independent studies is to assess the concordance between studies. During the integration procedure, MINT proposes not only individual performance accuracy assessment, but also insightful graphical outputs that are study-specific and can serve as Quality Control step to detect outlier studies. One particular example is the Takahashi study from the stem cell data, whose poor performance Fig. d was fuMINT successfully built a space based on a handful of genes in which samples from each study are discriminated in a homogenous manner.The sample plots from each individual breast cancer data set showed the strong ability of MINT to discriminate the breast cancer subtypes while integrating data sets generated from disparate transcriptomics platforms, microarrays and RNA-sequencing Fig. a–c. ThoMINT selected 2 and 15 genes on the first two components respectively  indicated in an outcome variable. On the stem cell data, Basal, HER2, Luminal A and Luminal B [MINT successfully recovered 37 of the 48 PAM50 genes present in the data (77%) on the first three components . The overall signature included 30, 572 and 636 genes on each component  was superior than when performing a PLS-DA on the PAM50 genes only (BER of 20.8 on the training set and a very high 75 on the test set). This result shows that the genes selected by MINT offer a complementary characterisation to the PAM50 genes.On the breast cancer study, we performed two analyses which either included or discarded the PAM50 genes that were used to define the four cancer subtypes uminal B . In the MINT identified 11, 272 and 253 genes on the first three components respectively . Although several efficient methods have been proposed to address the unwanted systematic variation when integrating data , 7, 9–11MINT seeks to identify a common projection space for all studies that is defined on a small subset of discriminative variables and that display an analogous discrimination of the samples across studies. Therefore, MINT provides sample plot and classification performance specific to each study Fig. . Among tIntegrative approaches such as MINT are essential when combining multiple studies of complex data to limit spurious conclusions from any downstream analysis. Current methods showed a high proportion of false positives (44% on MAQC data) and exhibited very poor prediction accuracy  against non-pluripotent cells (hiPSC and hESC). Pluripotency is well-documented in the literature and OCT4 is currently the main known marker for undifferentiated cells –56. HoweSeveral important issues require consideration when dealing with the general task of integrating data. First and foremost, sample classification is crucial and needs to be well defined. This required addressing in analyses with the stem cell and breast cancer studies generated from multiple research groups and different microarray and RNA-seq platforms. For instance, the breast cancer subtype classification relied on the PAM50 intrinsic classifier proposed by , which wThe expertise and exhaustive screening required to homogeneously annotate samples hinders data integration, and because it is a process upstream to the statistical analysis, data integration approaches, including MINT, can not address it.A second issue in the general process of integrating datasets from different sources is data access and normalisation. As raw data are often not available, this results in integration of data sets that have each been normalised differently, as was the case with the breast cancer data in our study. Despite this limitation, MINT produced satisfactory results in that study. We were also able to overcome this issue in the stem cells data by using the stemformatics resource  where weMINT and other data integration methods dismiss the platform information and focus on the study effect only. Indeed, each study is considered as included in a single platform. MINT successfully integrated microarray and RNA-seq data, which supports that such an approach will likely be sufficient in most scenarios.A final important issue in data integration involves accounting for both between-study differences and platform effects. When samples clustered by study and the studies clustered by platform, then the experimental platform and not the study, is the biggest source of variation  and four subtypes of breast cancer . The gene signatures identified by MINT contained existing and novel biomarkers that were strong candidates for improved characterisation the phenotype of interest. In conclusion, MINT enables reliable integration and analysis of independent genomic data sets, outperforms existing available sequential methods, and identifies reproducible genetic predictors across data sets. MINT is available through the mixMINT module in the mixOmics R-package.We introduced MINT, a novel Multivariate INTegrative method, that is the first approach to integrate independent transcriptomics studies from different microarray and RNA-seq platforms by Additional file 1MINT in a regression framework, the application to the MAQC data (A vs B), the meta-analysis of the breast cancer data, the classification accuracy of the tested methods on the stem cells and breast cancer data, and details on the signature genes identified by MINT on the stem cells and breast cancer data. (PDF 4403 kb)Supplementary material. This pdf document contains supplementary methods and all supplementary Figures and Tables. Specifically, it provides the PLS-algorithm, the extension of


The Cancer Genome Atlas (TCGA) has generated comprehensive molecular profiles. We aim to identify a set of genes whose expression patterns can distinguish diverse tumor types. Those features may serve as biomarkers for tumor diagnosis and drug development.Using RNA-seq expression data, we undertook a pan-cancer classification of 9,096 TCGA tumor samples representing 31 tumor types. We randomly assigned 75% of samples into training and 25% into testing, proportionally allocating samples from each tumor type.BNC1, FAT2, FOXA1, and HOXA11. FOXA1 has been shown to play a role for sexual dimorphism in liver cancer. The differentially discriminative genes we identified might be important for the gender differences in tumor incidence and survival.We could correctly classify more than 90% of the test set samples. Accuracies were high for all but three of the 31 tumor types, in particular, for READ (rectum adenocarcinoma) which was largely indistinguishable from COAD (colon adenocarcinoma). We also carried out pan-cancer classification, separately for males and females, on 23 sex non-specific tumor types (those unrelated to reproductive organs). Results from these gender-specific analyses largely recapitulated results when gender was ignored. Remarkably, more than 80% of the 100 most discriminative genes selected from each gender separately overlapped. Genes that were differentially expressed between genders included We were able to identify many sets of 20 genes that could correctly classify more than 90% of the samples from 31 different tumor types using TCGA RNA-seq data. This accuracy is remarkable given the number of the tumor types and the total number of samples involved. We achieved similar results when we analyzed 23 non-sex-specific tumor types separately for males and females. We regard the frequency with which a gene appeared in those sets as measuring its importance for tumor classification. One third of the 50 most frequently appearing genes were pseudogenes; the degree of enrichment may be indicative of their importance in tumor classification. Lastly, we identified a few genes that might play a role in sexual dimorphism in certain cancers.The online version of this article (doi:10.1186/s12864-017-3906-0) contains supplementary material, which is available to authorized users. The Cancer Genome Atlas (TCGA) has generated comprehensive molecular profiles including somatic mutation, copy number variation, gene expression, DNA methylation, microRNA expression, and protein expression for more than 30 different human tumor types . Those lTumor classifications based on gene expression data have revealed distinct tumor subtypes and uncovered expression patterns that were associated with clinical outcomes –14. LandSex differences in cancer susceptibility are one of the most consistent, but least understood, findings in cancer epidemiology , 16. MalWe undertook a comprehensive pan-cancer classification of 9096 tumor samples from 31 tumor types from TCGA using RNA-seq gene expression data. We aimed to identify a set of genes whose expression levels can classify all 31 TCGA pan-cancer tumor types. We also carried out the same pan-cancer classification on the gene expression data from 602 “normal” tissue samples taken adjacent to tumors for 17 tumor types. We compared the top-ranked discriminative genes from both tumor and “normal” samples and concluded that most discriminative genes that we identified reflected tumor-type differences rather than simply tissue-of-origin differences. Moreover, we sought to identify, separately in men and in women, analogous sets of genes that can distinguish the 23 sex non-specific tumor types. We hope to gain insight into sexual dimorphism in some tumors from those analyses.https://tcga-data.nci.nih.gov) for 9096 patients representing 31 tumor types  UNC RNASeqV2 level 3 expression data from the TCGA data portal  due to small gender-specific sample sizes. At the time of analyses, data for two new tumor types (ESCA and STAD) became available and were included in the analysis. This brought the total number of sex non-specific tumor types to 23 with 2638 females and 4081 males RNA-seq samples. The numbers of samples for each tumor type from each gender are listed in Table k-nearest neighbors (KNN) algorithm as the classification tool. GA/KNN can identify gene signatures that not only can separate different classes of samples but also may uncover subtypes within a class. One valuable characteristic of GA/KNN is that, for each training/testing partition, it identifies many near-optimal feature sets and uses each feature set to predict the testing-set samples. Because the algorithm classifies each sample multiple times, one can calculate the proportion of times that each sample was predicted to be each of the 31 classes plus a category of unclassifiable due to ties (proportions sum to 1). Furthermore, one can also assess the relative importance of each gene for sample classification by counting how often that gene appears in those near-optimal feature sets.We used the GA/KNN method , 28 for k was set to 5 with a majority “voting” rule. We selected these parameters based on an earlier comprehensive analysis of the effect of the choice of parameters on both gene selection and classification accuracy ). Each bar represents one of the 24 proportions that samples from the actual tumor type were predicted to be. The 24 plotted proportions represent averages from the corresponding proportions for all samples of the actual tumor type. (DOCX 1745 kb)Additional file 12: Table S4.Genes ranked among the top 100 from either females and males. (DOCX 389 kb)Additional file 13: Figure S8.Heatmap representations of the expression patterns of the top genes across all male and female samples. Each row (gene) was centered by the median expression value across all samples. A hierarchical clustering analysis was carried out for both samples and genes using the Euclidean distance as the similarity metric. (DOCX 15 kb)Additional file 14: Figure S9.BNC1 expression data in the 23 sex non-specific tumors from males (blue) and females (pink). (DOCX 126 kb)Boxplot Additional file 15: Table S5.cc values for each tumor type from full female dataset, full male dataset, and the corresponding mean (sd) from the eight “matched” male datasets. (DOCX 283 kb)Mean and median of for π