Finally, we observed that prognostic biomarkers performed highest when combining radiomic, genetic, and clinical information  indicating complementary value of these data. In conclusion, we demonstrate that radiomic approaches permit noninvasive assessment of both molecular and clinical characteristics of tumors, and therefore have the potential to advance clinical decision-making by systematically analyzing standard-of-care medical images.Medical imaging can visualize characteristics of human cancer noninvasively. Radiomics is an emerging field that translates these medical images into quantitative data to enable phenotypic profiling of tumors. While radiomics has been associated with several clinical endpoints, the complex relationships of radiomics, clinical factors, and tumor biology are largely unknown. To this end, we analyzed two independent cohorts of respectively 262 North American and 89 European patients with lung cancer, and consistently identified previously undescribed associations between radiomic imaging features, molecular pathways, and clinical factors. In particular, we found a relationship between imaging features, immune response, inflammation, and survival, which was further validated by immunohistochemical staining. Moreover, a number of imaging features showed predictive value for specific pathways; for example, intra-tumor heterogeneity features predicted activity of RNA polymerase transcription  and intensity dispersion was predictive of the autodegration pathway of a ubiquitin ligase . Furthermore, Grossmann et al. found that combining data from radiomics, genomics and clinical parameters predicts how the cancer will progress better than any of these parameters can predict on their own. These results demonstrate the complementary value of radiomic data to genomic and clinical data.There are many different algorithms that can be used to process images for radiomics. Before radiomics can be used clinically to assess the biological processes underlying the tumors of patients, a specific algorithm needs to be decided upon and then tested in prospective clinical trials.DOI:http://dx.doi.org/10.7554/eLife.23421.002 A number of examples demonstrate successful translation of genomic information obtained from biopsies into clinical applications , but theRadiomics is an emerging field that translates these medical images into mineable data by extracting a large number of quantitative imaging features that objectively define tumor intensity, shape, size, and texture  in a robHere, we present a broad radiomic-genomic analysis in independent and large cohorts of patients with lung cancer. We rigorously investigated the mechanistic connections between imaging phenotypes and underlying molecular pathways. Furthermore, we validated key associations via immunohistochemical staining and related these associations to clinical factors. In addition, we developed and validated radiomic predictors of pathway activation status, and investigated the prognostic value of combining radiomic biomarkers with genetic and clinical data. In this study, we aimed at uncovering whether radiomic approaches have the potential to predict both molecular and clinical characteristics of tumors noninvasively and therefore have the potential to augment clinical decision-making using data extracted from standard of care medical images.n = 262), and validated our results on an independent validation cohort . We defined and extracted 636 radiomic features from CT scans . In general, we found that distinct radiomic features were associated with distinct biological processes. For example, texture entropy and cluster features, as well as voxel intensity variance features were associated with the immune system, the p53 pathway, and other pathways involved in cell cycle regulation in modules M2, M9, and M12 . In anotFurther examples for radiomic-pathway links included two modules (M13 and M7) that were highly enriched for pathways involved in mitochondrial pathways, transcription, translation, and RNA regulatory mechanisms; with only one exception, all features in the larger module (M13) were voxel intensity entropy features. In addition to this feature type, the smaller module (M7) contained mainly textural variance and information correlation features.For every module, we assessed prognostic association to overall survival (OS) and associations to stage and histology based on the radiomic features of a module . Three mWe examined and summarized the relationships of clinical status, module size, and overlap of modules in a network . We foun<10−4) in module M2, which was also associated with p53. Importantly, COP1 mediates p53 and may interact with autophagy (−4) in module 5, which was associated with lipoprotein metabolism and stage. Further, two shape features (sphericity and compactness) predicted TRAF6 mediated NFkB activation  in module 10, which was also associated with axon guidance and histology.To test whether radiomic features can predict if a pathway is activated or deleted in individual patients, we fitted univariate models of radiomic features on Dataset1 and selected for every module the strongest predictor in Dataset1 according to the area under the curve  (AUC) foutophagy , which aFurthermore, we assessed these representative features in terms of their predictive value for driver mutations in the discovery cohort; based on a subset of 60 patients whose tumors were profiled with Sanger sequencing, we estimate that the prevalence of mutated EGFR, KRAS, and TP53 are 15%, 35%, and 20%, respectively. In particular, we found strong performance for mutations in EGFR and KRAS by several features, but only one considerable performance for TP53 . InteresTo further investigate putative connections between radiomics, immune response pathways, and OS we performed immunohistochemical staining of 22 tumors for CD3, a T-cell co-receptor. These tumors were predicted to show relatively high or low immune response by a radiomics feature selected from the three modules  that were associated with OS. As represented in −9) when adding the radiomic and an existing gene signature  on Dataset2 . Furthermore, we tested combinations of clinical, genetic, and radiomic data and observed that the combinations of data types tended to result in higher performances than given by the individual data alone . In partignature ; this inignature , and fouignature .10.7554/Medical imaging plays a crucial role in cancer diagnosis, treatment, and response monitoring. Radiomics allows quantification of the radiographic phenotype of a tumor , but thePreliminary studies have previously investigated associations between imaging features, clinical factors, and molecular data for a number of cancer types as outlined in recent reviews . Our anaWe identified and independently validated thirteen radiomic-pathway modules with coherent expression patterns, eleven of which were significantly associated with OS, stage, or histology. By basing these clinical associations exclusively on radiomic features, we could demonstrate that the associated molecular pathways robustly matched radiomic- based hypotheses. For example, based on radiomic features modules M2, M9, and M12 were prognostic and also associated with stage. These modules were highly enriched for immune system, p53, and cell-cycle regulation pathways, biological processes that are widely recognized to play key roles in lung cancer. For example, it has been established that cell cycle regulation is of utmost importance in lung cancer . FurtherWe aimed at confirming our statistical results indicating connections between radiomics, immune response, and survival by immunohistochemical staining of lymphocytes in cases for which a relatively high or low immune response was predicted according to a radiomics score. We generally found high agreement between pathology and radiomics, especially in cases where immune response was predicted to be high. In cases of predicted low responders that showed high pathological immune response, the cause of disagreement may be a heavy distribution of CD3 clusters in the extreme periphery of the tumor with very little staining in the bulk of the tumor. In cases of predicted high responders that showed little to no immune response, this could be due to the lack of normal tissue margin around the edge of the tumor section or a sampling effect. Similarly, we stained tissue for RelA, the p65 subunit of NFkB, to validate radiomic predictions of inflammation. Overall, we found high agreement between pathology and radiomics, although at lower statistical significance. Future studies with whole mount sections stained with multi-plex phenotyping can help determine the relationship between a radiomic and a genetic immune or inflammation signature, and the gold standard.A variety of textural features were also associated with stage and histology (module M5). Similar associations have been reported by Our results further suggest that radiomic approaches could have the potential to predict molecular states of pathways. We found that the highest predictors of every module was also a suitable representative of the overall biological and radiomic themes of that module. Amongst these examples of pathways that showed high predictability in terms of radiomics, we found various pathways essential for tumorigenesis such as cell cycle pathways , signaling pathways , and tumor suppressor pathways . Furthermore, we tested those radiomic pathway predictors for predictive value of driver mutations. Thereby, the highest performances were found for mutations in EGFR and KRAS, which is in line with current radiomic-genetic literature . InteresFinally, we verified a previously described prognostic radiomic signature and observed that the best performance is achieved when combining radiomic, genetic, and clinical data. These results strongly suggest that radiomic data contain complementary prognostic information and are robust, as the published radiomic signature  has not Three research tracks have recently been proposed for clinical translation of such imaging biomarkers , includiOur study is limited by its retrospective nature. Imaging protocols are not standardized and hence variability in CT acquisition and reconstruction parameters is inherent in clinical practice. However, despite this, no corrections by cohort or scanner type were made in this study illustrating the translational aspect of our results that generalized across institutions. Hence, we expect that the performance of radiomics will further improve, as imaging data are becoming more standardized. In fact, multiple studies have already documented the robustness of radiomic feature extractions in terms of reproducibility and repeatability in test/re-test settings . AnotherBiological material investigated in this study has been acquired by single-needle biopsies, thus the interpretation of our genomic data is limited due to heterogeneity of lung cancer tumors. However, as our results validated in independent data and because known drivers of tumorigenesis were among the main pathways found to be associated with radiomic features, this suggests that these associations have been established in an early evolutionary step in tumorigenesis and are therefore reasonable representatives of the overall tumor. Prospective studies with defined spatial matchings of biopsies and/or single cell analyses could provide deeper insight into whether the strengths of these associations can be further increased. Prospective studies will also be required to assess clinical utility of combining radiomic, genomic, and clinical data into prognostic models.In conclusion, this study presented novel and consistent associations between radiomics, molecular pathways, and clinical factors. We applied an independent discovery and validation design on large patient cohorts from different continents with enough variability that allowed confidence in the generalization of our results. Furthermore, we performed biological validation and demonstrated that radiomics predicts molecular pathway status and thus improves the prognostic performances of clinical and gene signatures. The clinical impact of our results is illustrated by the fact that it advances the molecular knowledge of automated radiomic characterization of tumors, information currently not used clinically. This may provide opportunities to improve decision-support at low additional cost as imaging is routinely used in clinical practice as standard of care.Data underlying this study is made publically available with this article; Dataset1 can be downloaded as For analyses involving CT scans and gene expression data, 262 and 89 patients were available for Dataset1 and Dataset2, respectively. In addition, clinical data were available for 224 and 87 patients, respectively. Clinical outcomes investigated were overall survival (OS), pathologic TNM stage , and paFor tumors in both cohorts, expression of 60,607 probes was measured on a custom Rosetta/Merck Affymetrix 2.0 microarray chipset  by the Moffitt Cancer Center. Gene expression of Dataset2 is available also at Gene Expression Omnibus (GEO) through accession number GSE58661. Gene expression values were normalized with the robust multi-array average (RMA) algorithm  implemenGroup I features are first-order statistics  of all voxel intensity values in the tumor volume mask. Group II features describe the shape and size of a tumor (e.g. compactness). Group III features quantify texture in tumor images describing clustering of voxels with similar appearance by means of a gray-level co-occurrence matrix (GLCM), a run-length gray-level matrix (RLGL), or a gray-level size-zone matrix (GLSZM). These features quantify how frequent voxels of same gray-level are adjacent to each other (GLCM), how many voxels of the same gray-level appear in a consecutive run (RLGL), or the sizes of flat zones, areas of same gray-level in all directions (GLSZM). Group IV features are Group I-III features (except GLSZM) assessed after a wavelet decomposition of the image, which highlights sharp transitions in the intensity frequency spectrum. Group V consists of Group I features that have been calculated after applying a Laplace of Gaussian transformation to the image, which highlights edge structures. Detailed description and analytical definitions of the features added to the 3.We extracted 636 features grouped into I) tumor intensity (voxel statistics), II) shape, III) texture, IV) wavelet, and V) Laplace of Gaussian features. Group I-IV features have been defined as specified by rho was calculated for the expression of every gene across all patients and weighted by -log10(p), where p is the p-value of rho. The resulting gene rank was input to a preranked gene set enrichment analysis (GSEA) algorithm +mean(YC), where X and CYC are the Spearman rank correlations of all pairs of features and pathways in a module, respectively. The true r was calculated for every module in Dataset1 and validated on Dataset2 with random permutation tests (N = 1000). After correcting for multiple-hypothesis testing with the false-discovery-rate (FDR)  using the ‘isa2’ and ‘eisa’ packages in R and Bioconductor . As a rete (FDR) , the valn/N + m/M, where n and m are the number of features and pathways in a module, respectively, and N = 636 and M = 511 are the total numbers of features and pathways across all modules, respectively. Overlap of two modules was defined by the Jaccard index  in Bioconductor  to calcuAssociations to OS were assessed by calculating the mean concordance-index  of all fWe tested additive prognostic effects of integrating radiomic, gene expression, and clinical data by combining in a Cox proportional-hazards model the predictions of (I) a clinical Cox model with stage and histology, (II) an NSCLC OS gene signature, and (III) an NSCLC OS radiomic signature. We tested five published gene signatures  without All statistical analyses were carried out using the R software  version We selected 25 cases each that were predicted to have high and low immune response by using the value of the radiomic feature in the prognostic modules M2, M9, and M12 that showed the highest absolute correlation to the mean expression of genes in the CTLA4 inhibitory pathway that is supported to be associated with immune activity . In totaWe selected 25 cases each that were predicted to have high and low NFkB activity. The same procedure as for the CD3 staining was applied, with the exception that a standard cell conditioning was used with CC2 antigen retrieval buffer (Ventana). Furthermore, a rabbit polyclonal primary antibody for RelA (NFkB p65), (Spring Biosciences E2750) was used at 1:600 dilution* and incubated for 32 min. In total, 24 cases were available with enough tumor tissue and sufficient staining quality.The lymphocytes are highlighted by brown nuclear staining of CD3. The staining pattern was analyzed by a board-certified pathologist (MB) and scored into low and high enrichment. The percentage and intensity  of staining were recorded as well as the number and size of clustering of CD3 positive cells. The pathologist also chose the appropriate area from each sample for image analysis. We observed that the tissue section that has a complete cross section of the tumor with a complete rim of adjacent benign lung parenchyma is most ideal for image analysis. This is because the lymphocytic infiltration is commonly present at the periphery of the tumor. In addition to this assessment by a pathologist, a computational system was implemented for automatic evaluation . In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.eLife. Your article has been favorably evaluated by Aviv Regev (Senior Editor) and three reviewers, one of whom, Trever Bivona (Reviewer #1) who served as Guest Editor. The following individual involved in review of your submission has agreed to reveal their identity: James O'Connor (Reviewer #3).Thank you for submitting your article "Identification of Molecular Phenotypes in Lung Cancer by Integrating Radiomics and Genomics" for consideration by The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.Summary:This study presents of series of medical imaging based findings that link radiographic tumor features to clinical and molecular features in lung cancer (via radiomics). This radiomic approach was used in combination with clinical and genetic characteristics of the tumors to generate prognostic assessments, with improved performance. Overall, this study presents interesting insights and suggests the utility of radiomics for informing the understanding of the pathogenesis of lung cancer.Several issues should be addressed to confirm the validity and clarity of the data presented and better link the findings to the current molecular state of the lung cancer field.Essential revisions:1) The cohort analyzed by the authors is mostly early stage lung cancer. Were any stage IIIB or IV tumors included? This stage bias should be noted in the text and the findings contextualized accordingly.2) It is important to at least annotate the tumors studied for major driver mutations in lung cancer . Associating radiomic features with these genotypes that direct clinical management would greatly enhance the relevance and impact of the current study.3) There should be better molecular validation by biomarker staining in the tumor tissue of the gene sets that are used to score different molecular phenotypes. As it stands, it is unclear how and why the gene sets were selected for classification of the tumors and whether such in silico based classification translates into the human tissue in a meaningful way at the protein level (of key pathway biomarkers). An example is to perform RelA IHC to validate the NFkB finding in the relevant tumors.4) Were the 5 or so radiomic values "aggregated" into a single index per tumor to perform the clustering analysis with the gene set signature? If not, more elaboration as to how the clustering analysis was done would be very helpful.5) Please clarify the purpose and message of the network presented in 6) It is appreciated that the modules were verified with an additional patient data set independent of the discovery data set, but this study begs for validation of pathway activation in some of the tissues directly using immunohistochemistry. Modules 4, 6, and 8 seem like low hanging fruit given the enrichment of well characterized signaling pathways with an abundance of markers of pathway activation for IHC.7) Related to point #6, is it not troubling that the IHC for CD3 did not appear to agree with the prediction of high or low immune response for modules 2, 9, and 12?8) Can the authors elaborate on the clinical significance of developing a prognostic marker for a cancer that is overall highly aggressive? How will the prognostic power of the combination of R + G + C impact clinical decision making?9) What is the rationale for extending the 440 features in the previous Nat Commun study to 636 features? Please comment on this specific point for this study, but also for the subsequent translation of the radiomics idea could the authors discuss how to move from deriving and testing one or another 'radiomic signature' to deciding on a final signature and then locking this down, for future application. Is this approach planned, would it be tumor-therapy specific etc.? Some discussion of this is important in biomarker translation. The imaging biomarker roadmap in Nat Rev Clin Oncol 2017 should be cited in this regard.10) Since this study has several positives  it would be important to a) state how this study conforms with several of the roadmap recommendations and b) explicitly highlight how the biomarker = radiomics signature family (the one here is different from that in the previous paper in Nat Commun) has been advanced in this study down the biological/clinical validation pathway. This is alluded to, to some extent in the eighth paragraph of the Discussion, but this point could be strengthened.11) The section 'Immunohistochemical investigation of radiomic immune response assessment' seems less convincing that the data from other sections as there is N=15. Is this justified as an analysis based on the very small N? Please justify the inclusion of these data for this section.12) Many of the statistical associations are strong. But it is difficult to understand how the radiomic metrics relate to biological pathways that clearly measure something quite different . Is this really going to convince oncologists that a radiomic metric  is an adequate measure of say cell cycling? Does this seem plausible and if not then is that a translational barrier? Please discuss this a bit more in the text, as I think this will be seized upon by readers over the next few years. […] Several issues should be addressed to confirm the validity and clarity of the data presented and better link the findings to the current molecular state of the lung cancer field.In addition to the responses below, to better reflect the specific focus of study we would like to change the current title of the manuscript into: “Defining the biologic basis of radiomic phenotypes in lung cancer”.Essential revisions:1) The cohort analyzed by the authors is mostly early stage lung cancer. Were any stage IIIB or IV tumors included? This stage bias should be noted in the text and the findings contextualized accordingly.We thank the reviewer for the time in critically assessing our study. Indeed, our study cohorts contain mainly early stage tumors, although several stage IIIB and IV cases are included; specifically 26 in the discovery cohort. In the radiomic analysis of histology, these cases were grouped into “Other” to avoid imbalanced outcome groups. In our revised manuscript, we emphasized this important point in the Materials and methods and Discussion. Furthermore, a relevant issue with these late stage tumors is that they are difficult to segment, and are thus a challenge for radiomics of lung lesions. The approach that is currently being taken is to extract radiomic features from metastatic sites, e.g. liver, lymph nodes, adrenals; but this is quite distinct from the focus of the current work. Our text has now also been updated to reflect this, as well.2) It is important to at least annotate the tumors studied for major driver mutations in lung cancer . Associating radiomic features with these genotypes that direct clinical management would greatly enhance the relevance and impact of the current study.We thank the referee for this insightful comment. We agree that associating radiomic features with widely established lung driver mutations would enhance our study. Therefore, we retrospectively queried our institutional database to include all available sequencing records. For our revised manuscript, we can now present additional data on 60 patients whose primary tumors were profiled with Sanger sequencing. For these patients we analyzed how the representative features of every module of our analysis predict mutant EGFR, KRAS, and TP53. We found strong predictive value for EGFR and KRAS, but only moderate predictive value for TP53. In particular, features that predicted EGFR and KRAS were first-order statistics of the voxel intensities and textural features, complementing our previous associations to cell cycling and proliferation pathways. We included these new results in the revised manuscript. The mutation status of the current cohort will be made available in the revised text and shared publicly with all other data. Furthermore, we referenced prior work from our research groups and others wherein radiomic features were associated with EGFR, KRAS, and other lung cancer driving mutations according to the popular model of clonal evolution.3) There should be better molecular validation by biomarker staining in the tumor tissue of the gene sets that are used to score different molecular phenotypes. As it stands, it is unclear how and why the gene sets were selected for classification of the tumors and whether such in silico based classification translates into the human tissue in a meaningful way at the protein level (of key pathway biomarkers). An example is to perform RelA IHC to validate the NFkB finding in the relevant tumors.p = 0.06. We added these new results to our revised manuscript and included a corresponding discussion of these updates.We would like to thank the referee for the apparent interest in our study. Indeed, additional molecular validation of our computationally derived and, on a large and independent cohort, statistically validated results is a positive suggestion. Hence, we selected tumors that were predicted to have the highest and lowest activation of NFkB by shape sphericity, the representative feature of module 10 that was also associated with NFkB activation. Following the suggestion from the reviewer we stained available tissue from these tumors for RelA and assessed the enrichment by an experienced pathologist . Comparison of radiomic and pathological assessment revealed that radiomic feature values were higher for those cases that were also pathologically scored to be relatively higher enriched for RelA; according to a Wilcoxon rank sum test this was at a significance level of 4) Were the 5 or so radiomic values "aggregated" into a single index per tumor to perform the clustering analysis with the gene set signature? If not, more elaboration as to how the clustering analysis was done would be very helpful.We thank the referee for this request and apologize if clarity in our Materials and methods was limited. We leveraged a pre-ranked gene-set enrichment analysis using a correlation rank between every radiomic feature and the expression of every gene. This allowed us to calculate a normalized enrichment score for every feature and gene set , which was the basis for our clustering analysis. In this sense, indeed, we associate every feature with every gene set independently. We updated the text in our revised manuscript to increase clarity of this point.5) Please clarify the purpose and message of the network presented in We appreciate and thank for the detailed review of our manuscript. 6) It is appreciated that the modules were verified with an additional patient data set independent of the discovery data set, but this study begs for validation of pathway activation in some of the tissues directly using immunohistochemistry. Modules 4, 6, and 8 seem like low hanging fruit given the enrichment of well characterized signaling pathways with an abundance of markers of pathway activation for IHC.We thank this reviewer for noting our stringent statistical study design and for suggesting additional staining experiments, which is in line with similar comments. To address these comments, we present additional data on immunohistochemical staining for RelA (a p65 NKkB subunit) to validate radiomic associations with inflammation, and also present additional data on staining for CD3 to validate radiomic associations to immune response. We kindly refer to the corresponding sections of our revised manuscript and hope that the significance of these updates is noted.7) Related to point #6, is it not troubling that the IHC for CD3 did not appear to agree with the prediction of high or low immune response for modules 2, 9, and 12?We would like to thank the referee for taking time to assess the results of our manuscript, and apologize for any confusion. We did find pathological agreement for the association with immune response in modules 2, 9, and 12. We used modules 2, 9, and 12 to select the radiomic feature that representing the immune response score. We used this score to select cases that this radiomic feature predicted to show relatively high or low immune response. For these cases, we performed immunohistochemical staining for CD3 (as surrogate for immune response) and found that the pathological assessment of immune response agreed with the radiomic score in the majority of stained cases. In our revised manuscript, we present additional data on this to increase statistical power of our analysis. We updated the revised manuscript to highlight this important point.8) Can the authors elaborate on the clinical significance of developing a prognostic marker for a cancer that is overall highly aggressive? How will the prognostic power of the combination of R + G + C impact clinical decision making?According to the National Comprehensive Cancer Network (NCCN), an alliance of cancer centers in the United States, there are a number of treatment options for early stage cancers including, inter alia: adjuvant CTx, chemorads, immune Tx with and without RTx/CTx combinations. We propose that radiomic biomarkers can be used within a decision support system when planning therapy. Realizing that therapy choice is made between clinical oncologist and patient preference, more aggressive radiomic signatures may lead to more aggressive therapies. The superior prognostic power of combining radiomics with genomics and clinical data has long been hypothesized, but up to now no study rigorously investigated whether this holds true. The advantages of combining these inherently different data types seem obvious: while genomics quantifies molecular events happening in tumors, radiomics is able to provide a phenotypic quantification. The hope for clinical decision making is that these data types complement each other in prognostication. We included additional elaboration in the discussion on these fundamental points of our and other's work. At this time, we are not proposing that this be used in a CADx system. This important point was added to the Discussion.9) What is the rationale for extending the 440 features in the previous Nat Commun study to 636 features? Please comment on this specific point for this study, but also for the subsequent translation of the radiomics idea could the authors discuss how to move from deriving and testing one or another 'radiomic signature' to deciding on a final signature and then locking this down, for future application. Is this approach planned, would it be tumor-therapy specific etc.? Some discussion of this is important in biomarker translation. The imaging biomarker roadmap in Nat Rev Clin Oncol 2017 should be cited in this regard.We thank the referee for commenting on the various feature sets presented in current radiomic research and their translational impact. With regard to the referenced feature set from our previous study published 2014 in Nature Communications, we simply updated the 2014 feature set with a novel textural matrix  to derive new radiomic texture features from. Including this additional matrix in our feature extraction resulted in a total of 636 features; we describe this in more detail in our Materials and methods section and provide the mathematical derivation of those new features in the supplemental material. Generally, radiomics is a dynamic field in which feature sets continue to be enhanced which is a major focus of investigation by a number of groups internationally. What we show is that a parsimonious set derived from a large set of features can be used to improve prognostic value, but indeed in near-future studies we aim at fixing a radiomic signature for definitive clinical evaluation. We believe that the most reasonable way to conduct this would be to define such a radiomic signature from a research group, fix its weights at later stages, and have another, independent group evaluate its prognostic performance whilst having an institutional ethics committee oversee and verify double-blindness of this procedure. Unlike biological biomarkers, technical validation can occur at later stages; however, cost-effectiveness is a crucial track in translation that needs to be considered early on. We include an elaborate discussion including the suggested reference to reflect on all of these important points.10) Since this study has several positives  it would be important to a) state how this study conforms with several of the roadmap recommendations and b) explicitly highlight how the biomarker = radiomics signature family (the one here is different from that in the previous paper in Nat Commun) has been advanced in this study down the biological/clinical validation pathway. This is alluded to, to some extent in the eighth paragraph of the Discussion, but this point could be strengthened.We thank the referee for this very positive feedback in correctly noting the multiple efforts we undertook to provide a stringent analysis. We agree that in this regard it would be valuable to emphasize how our study can advance the urgently needed development for accurate and robust imaging biomarkers in clinical settings. We built on the signature that we proposed in the referenced study  by investigating the prognostic results of combining features of this signature with genetic and clinical data. In addition to this, we developed a novel signature to investigate whether the results of the previous signature hold for further signatures, as well. To convey this message more clearly, we extended the corresponding paragraphs with more elaborate discussions, including future perspectives.11) The section 'Immunohistochemical investigation of radiomic immune response assessment' seems less convincing that the data from other sections as there is N=15. Is this justified as an analysis based on the very small N? Please justify the inclusion of these data for this section.p-value with a standard Fisher’s exact test. As the p-value is a function of the sample size, insufficient sample size would be reflected in this p-value.We thank the reviewer for the critical assessment of our study. IHC of patient derived tissue samples is limited in clinical practice by availability of frozen tissue. Therefore, we chose a manageable, yet statistically reasonable sample size; initially, we selected N=20 for 10 samples of high radiomic expression and 10 for low radiomic expression. From these N=20 samples, N=15 samples that contained tumor tissue were located by the institutional tissue core, which still provided us with a sufficiently large sample size to calculate a valid p = 0.13 to p = 0.008. We include these updated results in the revised manuscript with a corresponding discussion of the results and methods , and thank the reviewers for suggesting additional stains of relevant tumors.We agree, however, that even more efforts in analyzing stained tissue would greatly enhance our study. We therefore performed additional staining for CD3 as described in the Materials and methods and now aimed at doubling our initial sample size in the resubmitted manuscript. We also updated our statistical method to a more quantitative one. Indeed, with these updates we observe a considerable increase in statistical power from 12) Many of the statistical associations are strong. But it is difficult to understand how the radiomic metrics relate to biological pathways that clearly measure something quite different . Is this really going to convince oncologists that a radiomic metric  is an adequate measure of say cell cycling? Does this seem plausible and if not then is that a translational barrier? Please discuss this a bit more in the text, as I think this will be seized upon by readers over the next few years.We would like to thank the referee for this positive remark and agree that this important point will have relevance for readers of our study over the next years, as our study for the first time directly addresses the numerous requests from the oncological community to gain better biological rationale of radiomics in general. Indeed, we do not propose to directly replace biospecimen-derived diagnostics with radiomic approaches, but rather augment and complement them in cases where biological measurements are not feasible, such as in situations where frequent treatment monitoring is essential. Furthermore, we aim at contributing to better understanding what the biological basis for the various prognostic associations that have been described for radiomic features over past couple of years. To address this challenging task, we implemented a broad genome-wide analysis with a stringent and thorough statistical design, including external validation of putative associations. We revised our Discussion to reflect the translational impact of our study more clearly.


Gathering additional valuable information contained in the datasets represents a crucial opportunity in the biomedical field. To facilitate and stimulate utilization of these datasets, we have recently developed an interactive data browsing and visualization web application, the Gene Expression Browser (GXB). In this note, we describe a curated compendium of 13 public datasets on human breast cancer, representing a total of 2142 transcriptome profiles. We classified the samples according to different immune based classification systems and integrated this information into the datasets. Annotated and harmonized datasets were uploaded to GXB. Study samples were categorized in different groups based on their immunologic tumor response profiles, intrinsic molecular subtypes and multiple clinical parameters. Ranked gene lists were generated based on relevant group comparisons. In this data note, we demonstrate the utility of GXB to evaluate the expression of a gene of interest, find differential gene expression between groups and investigate potential associations between variables with a specific focus on immunologic classification in breast cancer. This interactive resource is publicly available online at: NCBI Gene Expression Omnibus (GEO), a public repository of transcriptome profiles, holds over 2 million individual transcriptome profiles from more than 76,000 studies , which makes datasets more accessible and interactive . Three main methods have been described to achieve this, which have the same subtypes, but actually use different gene sets to stratify the patients (poor immunogenic disposition (PID),weak immunogenic disposition (WID) andfavorable immunogenic disposition (FID). The ability of IDS to distinguish patients with differential prognosis is dependent on the tumor’s immune benefit status (IBS), which is defined by IMS and the expression of cell proliferation markers. The IBS classification segregates immune benefit-enabled (IBE) and immune benefit-disabled (IBD) tumors. In IBE tumors, but not IBD tumors, FID status confers a protective survival benefit compared to WID and PID status  dataset and the Uppsala and Singapore (GSE4922) dataset.The datasets that comprise our collection are listed inNagalla 2013 reconstituted public dataset”.Data of the 1839 GEO-cases annotated with survival data that were previously combined and used in the Nagalla study, have been uploaded to GXB in the dataset “Guy's hospital dataset . Expression data in the SOFT file of this dataset was expressed as fold change. Therefore, we had to revert to reprocessing of the CEL files found attached to the GSE on GEO. In this case, the cell files were read into R (v3.2.2) using the ‘affy’ package (v1.50.0). Data was normalized using the RMA (Robust multichip averaging) and gene annotation data was added using the hgu133plus2.db package (v3.2.3).All datasets were downloaded from NCBI GEO in SOFT file format and were uploaded into GXB with the exception of thein vitro origin (GSE2603) have been split manually using the GXB Graphical interface.GSE records containing data generated with different or multiple platforms have been split by platform using the import process of GXB. GSEs containing data from both clinical asMetadata of the different studies was added to GXB both from the descriptive information found on GEO or from the method sections of the publications linked to these datasets. Short links to PMID (Pubmed) and GEO records were added.The constitution of the complete cohort has previously been described by (in vitro origin (GSE2603), survival annotations and immune classifications are not applicable. A final 281 cases were not annotated and non-classified, since for these cases either samples were not taken at the time of surgical resection, were neoadjuvant-treated or cases were not annotated with distant metastasis free survival (DMFS) time and event.Gene expression data is accompanied with clinical data in CSV file format. Gene expression data and clinical data are coupled to the sample via variable “Sample ID”. We annotated a total of 1839 cases with 10-year survival (time and event), IBS , IDS  ,Table 2To enable comparisons between datasets and to facilitate efficient data analysis, the clinical data was harmonized to reflect a nomenclature similar to that of The Cancer Genome Atlas (TCGA). Clinical variable names and availability in datasets are listed inStandardized clinical datasets can be found in the ‘downloads’ tab in GXB under the heading “additional files”. All datasets start with the following 21 clinical variables in fixed order: "sample.ID", "array sample id", "sample title", "series", "IMS", "IBS", "IDS", "ICR", "DMFS_10Y_EVENT", "DMFS_10Y_TIME", "disease free survival event", "disease free survival years", "distant met free survival event", "distant met free survival", "age at initial pathologic diagnosis", "lymph node status", "ER status", "PR status", "histology differentiation grade", "tumor size cm", and "pathology T stage". In case one of these variables is not available in a specific dataset, values in this column are all NA.vs IBE-FID (IBS/IDS); ICR1vs ICR4 (ICR1/ICR4); LN+vs LN- (LN status); G1vs G3 ; ER+vs ER- (ER status). For IMS, no rank list was generated, as this variable is not ordered. For tumor stage, no rank list was generated because the spread of samples between categories was small (R scripts for this harmonistaion have been made available onGithub.).Group sets for IBS/IDS, ICR cluster, Lymph Node (LN) Status, IMS, Histological grade, stage and Estrogen Receptor (ER) status were defined with matching differential gene expression rank lists. Rank lists are based on differential gene expression between two relevant groups for each group set: IBD-FIDhttp://breastcancer.gxbsidra.org/dm3/landing.gsp. A web tutorial is also available online:http://breastcancer.gxbsidra.org/dm3/tutorials.gsp#gxbtut.The GXB software has been described in detail in a recent publication  the gene of interest, HLA-G, can be identified using the search box in the upper left corner of the user interface. Upon selection of “HLA-G” in the left panel, the central panel displays the expression values of this gene for all samples as a bar chart. For some of the transcripts, as is the case for the HLA-G transcript, multiple probes are available. Probe ID can be displayed by clicking “Tools” in the central panel and clicking “Show Probe ID” in the dropdown list. In this example, which is for illustrative purposes only, we selected probe ID 211528_X_AT at random. 2) Sample grouping is default as “All sample”, it is changed by selecting “Immunologic Constant of Rejection” and 3) plot type is set to “Box Plot” in drop down menus in the central panel. The central panel now presents a graphical display of the observed abundance of HLA-G transcripts in breast cancer samples across the different ICR groups, each sample is represented by a single point in a boxplot . A tendeEach of the boxplots corresponding to the 13 datasets show a similar pattern, indicating an increased HLA-G expression in breast tumors with a high expression of ICR genes  markers of immune cell infiltration, including CD8, CD3, CD19 and CD2, show similar expression in IBD-FID and IBE-FID subgroups  is used to explore differential gene expression between IBE-FID and IBD-FID tumors in GXB. Group set “Immune Benefit Status” is selected and corresponding gene rank list “IBD-FIDbgroups ; while  with CXCL9 expression, as this is one of the chemokines included in the ICR gene signature. Graphical representation in GXB is set to bar plot and group set ICR is selected. As anticipated, the CXCL9 expression gradually increases from ICR1-ICR4. The drop down menu “Overlays” is used to add multiple layers of additional variables, “IBS”, “IDS” and “IMS”. Boxes underneath the individual bars (each bar represents a single case) display the assigned classifications . When coIDS relates to the ICR classification in a similar manner. FID tumors are mostly assigned to ICR4, while WID tumors are frequently classified to intermediate clusters (ICR2 and ICR3), and PID tumors prevail in the ICR1 cluster . This obFor a more comprehensive overview of the relationship between different immune classifications in breast cancer, the overlay of immune classifications was evaluated in the Nagalla 2013 reconstituted public dataset (n=1839). The observations made in the EMC2 dataset n=58; are alsoThe overlay of the different immune classifications demonstrates a coherency between the IBS/IDS classification and the ICR clusters. Bearing in mind that the ICR signature is associated with a broader phenomenon of immune-mediated, tissue-specific destruction, this coherency strengthens the hypothesis of a common final pathway of tissue destruction.In this data note, we highlighted the opportunities provided by the availability of public datasets. We uploaded 13 public datasets on human breast cancer, including a combined dataset, with harmonized clinical data annotation and immune classification to GXB to facilitate the reuse of gene expression data. The use of GXB to explore gene expression and the different possible approaches were illustrated by the following: (1) an example case of a specific gene of interest, HLA-G; (2) comparison of gene expression between specific subgroups, IBD-FID vs IBE-FID; and (3) the evaluation of the relationship between different categorical variables, IBS/IDS and ICR immune classifications. To conclude, GXB provides a convenient environment to explore gene expression profiles in the context of breast cancer.http://www.ncbi.nlm.nih.gov/geo/, and are referenced throughout the manuscript by their GEO accession numbers (e.g.GSE7390). Signal files and sample description files can also be downloaded from the GXB tool under the “downloads” tab.All datasets included in our curated collection are available publicly via the NCBI GEO website: I thank the authors for their response. Many of my suggestions have not been implemented. If these limitations are clearly stated in the paper, I think this database could benefit the community.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. My concerns/queries have been addressed in the revised version.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. et al present a web application that provides real-time analysis and visualization of a large compendium of microarray datasets generated from primary breast cancer biopsies. The uniqueness and distinct utility of this dataset is the focus on immune-related signatures. Before discussing potential shortcomings according to this reviewer's opinion, the authors should be commended for investing the energy and resources to develop and implement a focused public repository that would facilitate further analyses.RoelandsRegarding the protocols followed, I would echo the comments by the first reviewer and question the use of MAS5 and COMBAT, given that more robust algorithms are now available. This may be a key issue given the quantitative nature of many of the indices used. Also, it is not clear that this approach removes bias between the older Affymetrix platforms and the newer ones (PLUS2.0). On page 5 it is mentioned that "the 22,268 probe sets present in each of these platforms were included", but the PLUS 2.0  chips include twice as many. They probably meant to say "present in all platforms".In terms of whether sufficient details were provided to allow replication, it would be great if the authors provided the complete set of scripts used to pre-process and normalize the data included in the analysis. This will allow transparent assessment of the methods and replication of the dataset if needed.  et al, 20111) and also of additional breast cancer subtypes . This will allow a more thorough assessment of the overlap between the various signature-defined subtypes. Finally, although there are not as many profiles from breast cancer metastasis available in GEO, it may be worth considering curating those and including them in the platform. It would be extremely interesting to know how the immune signatures differ between primary and metastatic samples, and whether the trends suggest immune evasion.Related to the type of data included, I believe that the resource would be more valuable if additional clinically-relevant information was provided with each sample. For example, disease progression clearly depends on treatment and treatment information is missing entirely - we do not know whether patients even received any treatment at all. It would be useful to curate treatment information available in the original datasets. Also, it may be worth providing annotations for additional immune signatures  and heat maps. The one gene at a time visualization is very restrictive. Although the overall objective behind this website and tools is laudable, I believe that the approach followed, as presented in the manuscript, has several potential shortcomings that would be worth addressing.Worth considering having another table after Table 1 that summarizes the immune subtype signatures and annotations with references. Intrinsic subtype signatures used can also be summarized in that table. That will also eliminate the need to explain the acronyms .When running the HLA-G example as outlined in the manuscript, I was presented with 4 different HLA-G transcripts with no explanation as to what was the difference between them. Are these different probe sets? If so, that should probably be explained. Also, in that case, is there a recommendation as to which of these probe sets is the most representative . Minor points:I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Regarding the protocols followed, I would echo the comments by the first reviewer and question the use of MAS5 and COMBAT, given that more robust algorithms are now available. This may be a key issue given the quantitative nature of many of the indices used. Also, it is not clear that this approach removes bias between the older Affymetrix platforms and the newer ones (PLUS2.0). On page 5 it is mentioned that "the 22,268 probe sets present in each of these platforms were included", but the PLUS 2.0 chips include twice as many. They probably meant to say "present in all platforms".The primary goal of this data note was to share the transcriptomic and annotation data associated with this previous publication  and the breast cancer transcriptomic datasets from GEO that it constitutes of in a more interactive, comprehensible format to facilitate usage of the data and did not involve data processing. The selection of 22,268 probe sets, combination and normalization of the 13 datasets was also previously performed . For your second point, we indeed meant the probe sets that are present in all platforms, for this reasons probe sets that are exclusively present in Affymetrix U133 PLUS2.0 are not included in these 22,268 probe sets. We have revised this sentence in the manuscript accordingly.In terms of whether sufficient details were provided to allow replication, it would be great if the authors provided the complete set of scripts used to pre-process and normalize the data included in the analysis. This will allow transparent assessment of the methods and replication of the dataset if needed. The uploads from individual GEO datasets into GXB did not involve any data processing. For the Nagalla et al this has been performed as part of the work described in Nagalla et al. This data note did involve clinical data harmonization performed in R, scripts have been made available on Github. et al, 20111) and also of additional breast cancer subtypes . This will allow a more thorough assessment of the overlap between the various signature-defined subtypes. Finally, although there are not as many profiles from breast cancer metastasis available in GEO, it may be worth considering curating those and including them in the platform. It would be extremely interesting to know how the immune signatures differ between primary and metastatic samples, and whether the trends suggest immune evasion.Related to the type of data included, I believe that the resource would be more valuable if additional clinically-relevant information was provided with each sample. For example, disease progression clearly depends on treatment and treatment information is missing entirely - we do not know whether patients even received any treatment at all. It would be useful to curate treatment information available in the original datasets. Also, it may be worth providing annotations for additional immune signatures , a treatment parameter is available which can be selected as a parameter using the overlay function in the bar plot generated in GXB. We completely agree that information on adjuvant treatment is essential for interpretation of the supplied survival data . For proper survival analysis, we recommend users to download the csv file supplied which can be found under the “Downloads”-tab of each dataset and use available treatment information for stratification.Addition of more gene signatures can indeed be valuable to compare immune based classifications as demonstrated in this data note for Immunological Constant of Rejection (ICR) classification and Immune Benefit Status (IBS)/ Immune Disposition Status (IDS). Similarly, uploading additional GEO datasets that contain matched primary and metastatic samples would be very interesting and allow for comparison of both immune microenvironments. Although not in the scope of the current data note, we will definitely take these suggestions into consideration for further development of the GXB breast cancer instance.Finally, from a usability standpoint, I have a few suggestions outlined below: -Although the functionality may be available, I found it very difficult to filter samples based on a particular feature. For example, I was trying to do the HLA-G by ICR group analysis only for the ER-negative or basal cases, but could not find an easy way to do that. It is indeed possible to visualize the expression of a gene of interest, for example HLA-G, in a specific subgroup of samples like all ER-negative or basal cases. The most convenient way to do this is to (1) select barplot, (2) set group by Molecular Subtype and (3) overlay the barplots with ICR group annotation. To make the figure easier to analyze, you can subsequently sort by ICR. - It would be useful to provide tools for correlational analysis of two or more genes (e.g. co-expression patterns) and heat maps. The one gene at a time visualization is very restrictive.We realize that gene per gene visualization has its limitations and analysis of groups coordinately expressed genes provides more biological significance. For this reason, the GXB development team has already created a module analysis tool (MAT) that analyses expression data to find pre-defined groupings of co-expressed genes (modules) to obtain a molecular fingerprint of gene expression for each individual sample in your dataset. MAT has already been applied to many datasets with samples of various immune related diseases and we are considering to also use this platform for cancer transcriptomic datasets. As the desired number of group comparisons in breast cancer  is higher compared to the existing group comparisons, some work is required before we can implement these dataset in this tool.Worth considering having another table after Table 1 that summarizes the immune subtype signatures and annotations with references. Intrinsic subtype signatures used can also be summarized in that table. That will also eliminate the need to explain the acronyms .We have added the suggested table to the manuscript. This is both useful when reading the manuscript as well as for the use of this breast cancer GXB application, thank you for this suggestion.When running the HLA-G example as outlined in the manuscript, I was presented with 4 different HLA-G transcripts with no explanation as to what was the difference between them. Are these different probe sets? If so, that should probably be explained. Also, in that case, is there a recommendation as to which of these probe sets is the most representative .These different HLA-G transcripts are indeed different probe sets. If you are interested to find and select a specific probe ID, you can select “Show Probe ID” under “Tools”. This GXB data portal does not give an explanation on the difference between these probes. For this information we would like to refer to the Affymetrix website.  For the HLA-G example, we just took a single probe at random: 211528_X_AT. We added this information in version 2 of the manuscript.Minor points: et al. describe the implementation of the gene expression browser (GXB), a database and web-application integrating 13 breast cancer datasets containing gene expression and clinical data for a total of almost 2000 patients. The authors focused their case studies on the investigation of immune-related genes and subtypes and made their curated data publicly available. I think GXB is a welcome contribution to the breast cancer research community and its public availability will help other scientists to leverage this valuable collection of datasets. However, I have several concerns that need to be addressed before indexing, as listed below.In this manuscript, Roelands The manuscript is well written and easy to understand. Major comments In Table 1 and on the front page of the web-application, Princess Margaret Cancer Centre is listed in the sample set name even though these data have not been generated in this institution, which is misleading. When selecting "Breast Cancer", which I assumed contained the whole database, many datasets got filtered out, which is confusing. I tried AURKA and selected GSE9195 as dataset. Then I played a bit with the barplot to overlay different kinds of information. I added DMFS 10Y  and sorted the patients based on this but there were some patients with very low DMFS at the beginning and the end of the plot, which is confusing. Since the authors are dealing with survival data, I was expecting survival statistics and/or survival curves but I did not see any in GXB or the manuscript. Figure 1 is somehow misleading as these plots do not seem to be part of the web-application. The authors decided to limit their database to Affymetrix data. In that case, I suggest the authors reprocess all the CEL files consistently using (f)RMA and up-to-date chip description files (CDF), such as BrainArray CDFs. This would increase the consistency across all the datasets. Given the authors' large collection of datasets, mining each dataset separately is cumbersome. Implementing meta-analysis pipelines would help draw an overall conclusion before digging into dataset-specific results . This is what the authors seem to have done with the "Nagalla" dataset that is a compendium of all the datasets reprocessed using MAS5 and further corrected with ComBat. It is not clear why the authors used MAS% to normalize the data in this case vs (f)RMA for some other datasets. Please clarify. Why not recommending users to first explore Nagallia before going after each dataset separately to detect trends that may be (in)consistent with the majority of the datasets? There have been many molecular subtyping schemes published, including the Integrative Subtypes , the Subtype Classification Models  and the Absolute Assignment of Breast Cancer Intrinsic Molecular Subtype . The authors should discuss the rationale for their choice of relying solely on PAM50.Minor comments Table 2 would be better represented as a barplot or a similar figure. GXB would benefit from the inclusion of normal samples . METABRIC, TCGA and GTEx are relevant data sources for such gene expression profiles.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Answers are in Italic and boldMajor comments In Table 1 and on the front page of the web-application, Princess Margaret Cancer Centre is listed in the sample set name even though these data have not been generated in this institution, which is misleading.Names of datasets have been changed from “Princess Margaret Cancer Centre dataset - GSE6532.GPL96” and “Princess Margaret Cancer Centre dataset - GSE6532.GPL97”, to “John Radcliff Hospital  dataset- GSE6532.GPL96” and “John Radcliff Hospital  dataset- GSE6532.GPL97” respectively. “Princess Margaret Cancer dataset- GSE6532.GPL570” was changed to “Guys hospital (GUYT) dataset- GSE6532.GPL570”. Tables and figures throughout the manuscript have been revised accordingly. When selecting "Breast Cancer", which I assumed contained the whole database, many datasets got filtered out, which is confusing.Thank you for noticing this. Only a single disease type can be assigned per dataset. We can change ER+ and LN- datasets to disease type: Breast Cancer , effectively disabling the user to filter datasets based on ER+ and LN status. We preferred changing the name of Breast cancer  to "mixed breast cancer types" for clarity. I tried AURKA and selected GSE9195 as dataset. Then I played a bit with the barplot to overlay different kinds of information. I added DMFS 10Y  and sorted the patients based on this but there were some patients with very low DMFS at the beginning and the end of the plot, which is confusing.We were able to reproduce the barplot you described. By sorting by the categorical variable DMFS 10Y EVENT, you only sort based on this variable (DistantMetastasisFree or DistantMetastasis). The continuous counterpart of this variable (DMFS 10Y TIME) is not taken into account when you perform the sorting, explaining your observation. An alternative is to sort based on the continuous variable DMFS 10Y TIME, though this sorting results in a plot that mixes DistantMetastasisFree and DistantMetastasis categories. Unfortunately, subsequent sorting based on 2 variables is not possible using GXB. We have suggested this improvement to the GXB developing team.  Since the authors are dealing with survival data, I was expecting survival statistics and/or survival curves but I did not see any in GXB or the manuscript. Figure 1 is somehow misleading as these plots do not seem to be part of the web-application.Survival statistics are not yet part of the GXB web-application. To prevent any misconception, we have added an extra sentence in the legend of figure 1: “This figure is for explanatory purposes only and does not serve as a demonstration of the GXB web application.”  The authors decided to limit their database to Affymetrix data. In that case, I suggest the authors reprocess all the CEL files consistently using (f)RMA and up-to-date chip description files (CDF), such as BrainArray CDFs. This would increase the consistency across all the datasets.The purpose of this data note is to share existing data from GEO in a more comprehensible format and does not involve any data processing. We had to make an exception for dataset GSE9195.GPL570, since only raw format was available on GEO. Given the authors' large collection of datasets, mining each dataset separately is cumbersome. Implementing meta-analysis pipelines would help draw an overall conclusion before digging into dataset-specific results . This is what the authors seem to have done with the "Nagalla" dataset that is a compendium of all the datasets reprocessed using MAS5 and further corrected with ComBat. It is not clear why the authors used MAS5 to normalize the data in this case vs (f)RMA for some other datasets. Please clarify.Normalization of the complete Nagalla cohort was previously performed as described in Nagalla et al. 2013 and Miller et al. 2016 To make this data available as published in this article, we used the same data file to upload to GXB.Normalization is different between datasets because the data was transferred straight from GEO and different contributors use different methods depending on the times it was performed and the need of their specific work. Why not recommending users to first explore Nagallia before going after each dataset separately to detect trends that may be (in)consistent with the majority of the datasets?In this dataset collection, it is indeed possible to first explore the Nagalla dataset, which is a dataset that includes all samples from the collection. This advantage is specific for this breast cancer compendium of datasets, so your suggestion is a very good strategy to explore these datasets. We have added this recommendation in the text of the Dataset Demonstration section of the data note.   There have been many molecular subtyping schemes published, including the Integrative Subtypes , the Subtype Classification Models  and the Absolute Assignment of Breast Cancer Intrinsic Molecular Subtype . The authors should discuss the rationale for their choice of relying solely on PAM50.Categorization using the PAM50 molecular subtyping was performed as part of the research performed by Nagalla et al. 2013 and Miller et al. 2016. These annotated datasets were uploaded in GXB as described in this data note. The choice to rely on this 50-gene classifier was dependent on its high prognostic and predictive value of this subtyping method in combination with the high clinical applicability of PAM50 testing.  Minor comments Table 2 would be better represented as a barplot or a similar figure.For the purpose of listing which clinical variables are available in how many datasets from the collection, we found a table the more appropriate option. GXB would benefit from the inclusion of normal samples . METABRIC, TCGA and GTEx are relevant data sources for such gene expression profiles.Comparing healthy control tissue with cancerous tissue on gene expression level indeed represents an interesting aspect that can be explored using the GXB browser. At this moment, different types of dataset collections have already been uploaded to GXB, including case versus control comparisons. For example, a dataset with gene expression of whole blood samples from both  lung cancer patients and healthy controls and a dataset with head and neck cancer and cervical cancer tissue samples, matched with site-matched normal epithelial samples are currently available. The TCGA, METABRIC dataset and datasets in the GTEx portal are valuable resources that can be used to upload new datasets into GXB. We will definitely take this suggestion into consideration for our new dataset collection to upload into GXB.Discussed changes will be applied in version 2 of the manuscript.


We have assessed the performance of seven normalization methods for single cell RNA-seq using data generated from dilution of RNA samples. Our analyses showed that methods considering spike-in External RNA Control Consortium (ERCC) RNA molecules significantly outperformed those not considering ERCCs. This work provides a guidance of selecting normalization methods to remove technical noise in single cell RNA-seq data. M-values (TMM) (Single cell RNA-seq (scRNA-seq) is becoming a powerful tool to study many biological problems, such as tissue heterogeneity (es (TMM) , DESeq  collected two well-characterized human reference RNA samples: Universal Human Reference RNA (UHR) and Human Brain Reference (HBR) . These dM-values (TMM), DESeq, removed unwanted variation (RUV), and gamma regression model (GRM). This study thus provides a guidance of choosing normalization methods to remove technical noise in single cell RNA-seq data.Using this data set, we have assessed the performance of several commonly used normalization methods: FPKM, upper quartile (UQ), trimmed mean of Before applying different normalization methods on the data, we first selected variable genes using the following criteria: (1) across the 114 nonbulk samples, at least two samples with log(fpkm) >2; (2) across 114 nonbulk samples, variant of log(fpkm) >1. In this way, we found 13,375 genes for the following analyses.FPKM  methods After normalization, we clustered the normalized gene reads using hierarchical clustering with the Ward method, and the metric was Pearson correlation between normalized gene reads. We assessed the clustering results using the following statistical indices:n elements S = S into M clusters S into N clusters a + b) is the number of agreements between X and Y, while (c + d) is the number of disagreements between X and Y. The higher the Rand index, the more similar the two partitions. As we compared the clustering partitions to the prior known labels, the higher Rand index indicates the better the clustering is.The Rand index  evaluaten elements S = The Dunn index  evaluateA and B, the Jaccard index is computed as:B times for drawing B random subsets (here we chose B = 100), we then computed the mean of all the maximum Jaccard index . A total of 13,375 genes was selected with log(fpkm) >2 in at least two of the 114 nonbulk samples (100 or 10 pg). Using Pearson correlation as the similarity metric, we clustered all 120 samples (114 nonbulk and six bulk samples) using hierarchical clustering (File S1). For all methods, the UHR and HBR samples were well separated as expected, and we thus focused on evaluating how well the HBR and UHR samples were further clustered. In either the UHR or HBR group, samples normalized by FPKM, UQ, TMM, and DESeq tended to cluster by sequencing protocol rather than RNA amount, which indicates that the differences introduced by the protocols were not completely removed by normalization. In contrast, the HBR samples normalized by RUVr were largely clustered by RNA amount rather than sequencing protocol, although a significant portion of 10 pg samples sequenced using aRNA were clustered separately (File S1). Qualitatively, RUVr normalization alleviates the difference between scRNA-seq protocols and the clustering results are closer to the ground truth than the other methods, i.e., the samples are clustered based on the source (HBR) and the RNA amount . However, the UHR samples normalized with RUVr were still clustered according to protocols rather than RNA amount. The other four methods showed worse clustering results than RUVr because 10 and 100 pg samples were mixed in each protocol group (File S1).We first evaluated the performance of five methods not considering ERCCs: FPKM , DESeq (File S1). RUVr clearly outperformed the other methods.To quantify the similarity between the clusters generated from different normalization methods and the ground truth, we cut the clusters at different hierarchy levels, and calculated the Rand index between the clusters and the ground truth, which reflects the correctness of clustering. The Rand index is the ratio between the sample pairs that are correctly clustered and the total sample pairs . BecauseNext, we evaluated the compactness and separation of clusters using the Dunn index, which is the ratio of the smallest distance between clusters to the largest intracluster distance, and the distance between the two samples is defined as 1 − the Pearson correlation between the samples . We compMaterials and Methods) between the new cluster and the ground truth. The Jaccard index reflects the correctly clustered samples using the subsets of samples, and thus the robustness of the clustering. All methods except RUVr showed similar Jaccard index values. Next, to evaluate the robustness of clustering to genes used to calculate Pearson correlation coefficient between samples, we ranked the 13,375 genes using their coefficients of variance (CV)  gave the highest Dunn index, which represents the best separation of the clusters  . Among the 120 RNA-seq runs, 45 samples containing spike-in ERCCs were normalized using the two methods, and then clustered with six bulk samples together  (together . The cluFile S1).The GRM showed a slightly higher Rand index than RUVg when the number of clusters was the same as the ground truth (6) or more . GRM alsNotably, both methods considering ERCC outperformed the methods not considering ERCC on the 51 samples (45 10 or 100 pg and six bulk samples) containing spike-in ERCCs . RUVr, aHere, we evaluated the performance of seven normalization methods on 120 RNA-seq runs in terms of correctness (Rand index), compactness (Dunn index), and robustness  of clustering. The results showed that, for methods not considering spike-in ERCCs, RUVr showed higher Rand index and lower Jaccard index than FPKM, UQ, DESeq, and TMM; all methods showed similar Dunn index values. Considering ERCC, such as in the models of RUVg and GRM, significantly improved performance. Between RUVg and GRM, GRM is more robust in terms of selecting different sets of genes that generate similar clusters. Spike-in ERCCs would reduce the sequencing depth of mRNAs of interest, and there is also a concern about whether the synthesized ERCC molecules behave the same as mRNAs from the cell. Our analyses suggest that calibration of scRNA-seq to the spike-in ERCC is a powerful means of removing technical noise when the ERCCs are correctly modeled.www.g3journal.org/lookup/suppl/doi:10.1534/g3.117.040683/-/DC1.Supplemental material is available online at Click here for additional data file.


Exponentially increasing numbers of NGS-based epigenomic datasets in public repositories like GEO constitute an enormous source of information that is invaluable for integrative and comparative studies of gene regulatory mechanisms. One of today’s challenges for such studies is to identify functionally informative local and global patterns of chromatin states in order to describe the regulatory impact of the epigenome in normal cell physiology and in case of pathological aberrations. Critically, the most preferred Chromatin ImmunoPrecipitation-Sequencing (ChIP-Seq) is inherently prone to significant variability between assays, which poses significant challenge on comparative studies. One challenge concerns data normalization to adjust sequencing depth variation.Currently existing tools either apply linear scaling corrections and/or are restricted to specific genomic regions, which can be prone to biases. To overcome these restrictions without any external biases, we developed Epimetheus, a genome-wide quantile-based multi-profile normalization tool for histone modification data and related datasets.Epimetheus has been successfully used to normalize epigenomics data in previous studies on X inactivation in breast cancer and in integrative studies of neuronal cell fate acquisition and tumorigenic transformation; Epimetheus is freely available to the scientific community.The online version of this article (doi:10.1186/s12859-017-1655-3) contains supplementary material, which is available to authorized users. Epigenetics is a complex and multi-layered process with potentially profound implications in cell fate decisions, including phenomena such as differentiation or tumorigenesis. With the advancements and cost-reduction in high-throughput sequencing, next generation sequencing (NGS) has become inevitable for epigenome research. Studying the epigenome and its dynamics involves sequencing to associate histone and DNA modifications to the genome of a particular cell with the aim of characterising the state of chromatin – the cumulated histone and DNA modifications in different genomic regions, over time during (patho)physiological processes (cell fate changes) and between different samples.www.ngs-qc.org) [However, its assessment via chromatin immunoprecipitation is inherently prone to significant variability, posing different bioinformatic challenges for comparative studies - a general caveat in Big Data integrative analysis. Multiple factors like antibody efficacy, sequencing library accuracy and depth have a direct impact on data quality and thus on any downstream analysis. Therefore, it is imperative to evaluate the quality of data prior to comparative studies  . However, proposed a two-step non-linear approach, based on a locally weighted regression (LOESS) method to correct such differences among ChIP-seq data [While significant computational efforts have been made in the past for single ChIP-seq data analysis, sophisticated computational and experimental methods to correct technical variability among multi-sample ChIP-seq analyses is acquiring importance recently. Initially, a simple linear normalization approach was widely used, where the counts were represented relative to the total number of reads to correct the throughput differences. However, such linear normalization, as used by RNA-seq-based tools like DESeq  and Edgeseq data . LOESS’sseq data , a multiseq data . Since tseq data  or Epigeseq data , both ofAll the above-mentioned tools suffer from a number of important limitations, namely (i) their annotation dependency, (ii) their restriction to specific regions, (iii) less user-friendliness, especially for non-bioinformaticians, and (iv) their inability to produce output files that are compatible with downstream analyses. Moreover, the existing approaches are mostly intended for a particular analysis, thus their normalization outputs are not readily exportable to other tools for multi-dimensional sample analysis and require programming skills. To overcome all these restrictions, we developed Epimetheus, a quantile-based multi-profile normalization tool. The genome-wide normalization procedure applied by Epimetheus enables optimal processing of datasets from different enrichment patterns, including broad/sharp histone modification or PolII-seq profiles, chromatin accessibility profiles generated by FAIRE-seq  and ATACThe basic assumption underlying quantile normalization is the presence of a common read-count distribution in the compared datasets. In cases where the compared enrichment events comprise factors that are implicated in house-keeping events, it is reasonable to assume that the distribution of the read counts for a given target will be similar across cell types .As for gene expression analysis (RNA-seq and microarrays) or RNA polymerase II enrichment (Polyphemus ), where As quantile normalization is an absolute read count-based approach, any region-specific or technical bias will over/under-represent the read counts and lead to inaccurate downstream analyses. Clonal reads  constitute one such technical bias. Unfortunately, some level of clonal read contamination is unavoidable in sequencing datasets involving PCR. Epimetheus will remove such clonal reads from the raw alignment data, unless otherwise specified by the user. There are a few alignment and platform-specific biases that should be addressed prior to analysis as these are specific to each data and pipeline. Particularly recommended is to remove reads with more than one perfect alignment and those aligned to repeat and centromere regions. Also, a user can opt to exclude tricky regions from analysis using the respective option available in Epimetheus. Reads are elongated to a specified length to represent the average fragment length (150-300 bp), as typically only the first 50–100 base pairs are sequenced in ChIP-seq.For quantile normalization, an approach similar to that of Xu et al.  and Menda & Xb are two samples of same target, then genomic bins for Xa will be xa1, xa2, xa3…xan and for Xb they will be xb1, xb2, xb3…xbn, where ‘n’ depends on the sizes of ‘S’ and ‘G’ (G/S). Reads in each bin are counted to calculate the reads per bin (RpB) distribution for each sample, thus generating the two libraries Xa = {xai | 1 ≥ i ≤ n} and Xb = {xbi | 1 ≥ i ≤ n}. Similarly, Ya and Yb will specify two different libraries for another target profile.If X is a target histone mark and XUsing RCI calculation results, a B × N matrix is built, where B is the total number of bins (for a given ‘G’ and ‘S’) and N is the number of samples. In case of multiple histone marks, B × N1, B × N2, etc., will be similarly generated. The differences in coverage among samples are adjusted to same level by (i) sorting each sample’s RpB in ascending order individually, (ii) ranking the values for each sample individually, (iii) calculating the average of the corresponding rank values and (iv) assigning normalized values back to the original positions. This results in a normalized matrix norm(B × N), where each sample has normalized-RpB (nRpB). Subsequently, Z-score scaling is applied to the normalized matrix to generate znorm(B × N), which is calculated from the distance of each nRpB to a mean value of total nRpB in the sample, divided by the standard deviation.www.ngs-qc.org) can be consulted for guidance [Note that quantile-based normalization cannot be applied to profiles of different histone marks, as the signal distribution and amplitude can be highly dissimilar, which is incompatible with the initial assumption. Similarly, users should not apply quantile normalization in cases where major differences are expected between the datasets. For example, inhibitors of epigenetic enzymes are likely to exert global effects histone mark deposition ; quantilguidance .In contrast to previously described methods, Epimetheus produces normalized BED files by adding/removing reads with respect to normalized per-bin RCIs using raw alignment BED files as reference. Increasing counts is done by adding new reads aligned randomly to a new position within the concerned bin; existing reads are randomly removed to decrease read counts. As the BED format is the preferred input format in most of the ChIP-seq tools, Epimetheus enables the direct use of normalized data for downstream analysis.Along with normalized BED output, Epimetheus produces three additional types of outputs: (a) visualization files, (b) plots and (c) normalized BED files. Visualization files are text files (in bedgraph format) generated for raw and normalized RCI, which can be used for other downstream analyses as well. To assess the difference among samples and the effect of normalization, MA transformation plots  are genewww.ngs-qc.org) to assess any influence of datasets quality in normalization. To avoid bias, clonal reads were excluded from the analysis in all datasets. The detailed materials and methods about the tools and their parameters used in different analysis followed are explained in the Additional file Before initiating normalization, the datasets were subjected to quality control using NGS-QC  . The comTo evaluate the consequence of quantile normalization on a regular ChIP-seq peak calling approach, MACS peak calling was performed for the H3K4me3 HepG2 dataset with raw and normalized BED files . While normalization led to very small differences in the number of identified peaks Fig. , which wbona fide common peaks.We conclude from these studies with biological replicates, which should in principle generate identical profiles that normalization results in a significant adjustment of peak intensities and increased the accuracy of identifying To illustrate Epimetheus’ performance in multi-profile integrative analysis, chromatin state analysis was performed using ChromHMM  with pubMYO7A gene in H1 cells changed from “active” to “poised”; this is due to a more prominent enrichment of H3K27me3 after normalization  cell model, which differentiates upon retinoic acid treatment . Cells wthe time , 25. HowHoxa cluster using quantitative PCR (qPCR). As illustrated in Fig. To further support the normalization results, we validated the H3K27me3 enrichment levels on various regions of the Here we have demonstrated the effect of quantile normalization on a variety of datasets and show that normalization is imperative when performing a comparative analysis of the relative signal amplitude levels of NGS-profiles. Even though normalization appears to have a minor effect on the global data analysis by peak-finding algorithms, we show here that (i) normalization minimizes amplitude differences in replicates and increases confidence in the temporal evolution of signal amplitude of enrichment patterns and (ii) improves chromatin state annotations when using binarized enrichment data, as with ChromHMM.In contrast to existing normalization tools, Epimetheus provides analytical outputs compatible with a variety of downstream analyses and illustrative plots. Indeed, previously described tools depend on peak caller predictions and/or external control datasets, the latter of which is generally the profile of the “input”, i.e. the whole cell extract (WCE) used for the ChIP. Given the diversity in available peak callers – each of which can be used with different parameters - and the potential bias introduced by a poor quality WCE, this approach could possibly lead to artefactual normalization. Specifically, even though control datasets exhibit in general an “ideal”  pattern, a few WCE controls show an artefactual enrichment-like pattern (examples are GSM788366 and GSM768313). Such patterns will significantly influence the normalization outcomes produced by tools like ChIP-norm.In this context we have compared genome-wide and target-specific normalization using Epimetheus for genome-wide and a ChIP-norm (7)-like approach for normalization using a pre-selected bins displaying significant enrichment over the input control. We compared both the approaches by using datasets with different enrichment patterns . In fact, datasets with systemic modifications of enrichment patterns or major differences in quality should obviously not be normalized. While most of the existing tools focus on normalization only for differential analysis, the present study with biological replicates and chromatin state analyses supports the necessity of normalization for any comparative, integrative or differential analysis. Relative to existing tools, the more robust and sophisticated options in Epimetheus are that (i) it can be customised to variety of requirements, (ii) it can be applied genome-wide or to specific regions (when justified), and (iii) it can exclude specific regions, such as repetitive elements, which would bias global normalization. We and others have shown previously that linear scaling-based tools cannot correct for the technical variations in ChIP-seq data. Other less user-friendly non-linear normalization tools are restricted to specific regions and their outputs cannot be usable easily for downstream analyses, such as binarized chromatin state annotations. For all these reasons, we have used Epimetheus in previous studies , 26, 27


Deconvolution is a mathematical process of resolving an observed function into its constituent elements. In the field of biomedical research, deconvolution analysis is applied to obtain single cell-type or tissue specific signatures from a mixed signal and most of them follow the linearity assumption. Although recent development of next generation sequencing technology suggests RNA-seq as a fast and accurate method for obtaining transcriptomic profiles, few studies have been conducted to investigate best RNA-seq quantification methods that yield the optimum linear space for deconvolution analysis.Using a benchmark RNA-seq dataset, we investigated the linearity of abundance estimated from seven most popular RNA-seq quantification methods both at the gene and isoform levels. Linearity is evaluated through parameter estimation, concordance analysis and residual analysis based on a multiple linear regression model. Results show that count data gives poor parameter estimations, large intercepts and high inter-sample variability; while TPM value from Kallisto and Salmon shows high linearity in all analyses.Salmon and Kallisto TPM data gives the best fit to the linear model studied. This suggests that TPM values estimated from Salmon and Kallisto are the ideal RNA-seq measurements for deconvolution studies.The online version of this article (doi:10.1186/s12859-017-1526-y) contains supplementary material, which is available to authorized users. Next-generation sequencing based technology for RNA profiling (RNA-seq) has become the predominant method to quantify the transcript abundance in cells. Compared to microarray technology, RNA-seq offers broader quantification range and enables the detection of novel transcripts . HoweverWe employed the benchmark dataset used to assess RNA-seq measurement performance in different application sites and platforms from the Sequencing Quality Control (SEQC) project . In ordeWe performed a literature survey and selected seven prevalent quantification methods for comparison. To increase the comparability of the estimated transcript abundance, all the alignment-based quantification methods were applied on mappings processed with Tophat2 . HTSeq-cTo our surprise, the distribution of the estimated abundance from each quantification method results in distinct distribution at both the gene level Fig. a and isC∼m×A+n×B+ε. Since sample C is derived by mixing sample A and B at ratio of 3:1, the expected values for the parameters m, n, and ε should be 0.75, 0.25, and zero respectively. We reason that if the RNA-seq data are linear, the fitted multiple linear regression model will provide precise estimation for the parameters m, n, and ε. Since there are four biological replicates for each sample types , there is a total of 64 possible models to be fitted. We fitted all 64 models and studied the performance based on the average of the parameters and predicted values. As shown in Fig. C is linearly correlated to the fitted values from models with known mixture proportion 0.75 and 0.25 as parameters in all quantification methods. The linear relationships are especially pronounced at the gene level  a concordance analysis between measured C and fitted value We observed that the estimation of m resulted in two clusters from count data represents consistent result to the concordance analysis between C against In the concordance analysis, we evaluated if the rank of genes from the fitted value entclass1pt{minimat}Ĉ Fig. a. In thtruth 0.7×A+0.25×BIn the residual analysis, in order to make the residual comparable across quantification methods in different ranges, we rescaled quantifications and use the model D, using the model D∼m×A+n×B+ε. Since D is made up of A and B with proportion 1:3, the expected value for m and n is 0.25 and 0.75 respectively in this D-based model. All of the results from the D-based model are consistent with the findings observed from the C-based model discussed  project. Raw data in fastq format were obtained from GEO website with accession number GSE47774. List of downloaded samples and their details is specified in Additional file Raw sequence reads from multiple lanes were first merged into one for each replicate of each sample. To prepare sequence map files for alignment-based quantification methods, the merged sequenced reads were mapped to human reference genome (hg19) using Tophat2 (version 2.1.0) with default parameters. The mapped files were then used by quantification methods HTSeq (version 0.6.1) to obtain count data, Cufflinks (versin 2.1.1) to obtain FPKM data. RSEM (version 1.2.31) will first mapped the raw reads using its default aligner Botwtie2 (version 2.1.0) onto human genome (hg19) and then estimate count, FPKM, and TPM. EdgeR (version 3.12.1) is used for count data transformation. Alignment-free quantification methods Kallisto (version 0.42.5) and Salmon (version 0.6.0) estimate the count and TPM data based on index built from human transcriptome (GRCh37). All the quantification methods were run at both the gene and isoform levels. All the statistical analysis and plots were carried out in R environment (version 3.2.3).Additional file 1Figure S1. Concordant analysis between rank of quantifications of Additional file 2Figure S2. Jitter boxplot of estimated coefficients and intercepts from linear model D∼m×A+n×B+ε at gene level (a) and isoform level (b). Red line indicates expected estimates if D, A and B satisfy linear assumption. (PDF 1520 kb)Additional file 3Figure S3. Concordant analysis between rank of estimated quantifications and rank of measured abundance value at gene level (a) and isoform level (b). The fitted value in the y-axis is estimated from model D∼m×A+n×B+ε. Ranks were normalized by the number of quantifications in each plot. (PDF 5950 kb)Additional file 4Figure S4. ROC-like curve evaluating linearity of quantified abundance at gene level (a) and isoform level (b) based on residuals from model D∼m×A+n×B+ε. Proportion of variables with residuals smaller than a threshold is computed. (PDF 1160 kb)Additional file 5Figure S5. Residual plot for rescaled model Additional file 6Table S1. Column 1 and 2 show the sample type and replicates index; Column 3 shows total read pairs, column 4-7 show the mapping rate from different quantification methods . (XLS 61 kb)