Targeted therapy for non-small cell lung cancer is histology dependent. However, histological classification by routine pathological assessment with hematoxylin-eosin staining and immunostaining for poorly differentiated tumors, particularly those from small biopsies, is still challenging. Additionally, the effectiveness of immunomarkers is limited by technical inconsistencies of immunostaining and lack of standardization for staining interpretation.KRT5 and AGR2, which has the stable REO pattern of KRT5 > AGR2 in pSCC and KRT5 < AGR2 in pADC. In the two test datasets with relative unambiguous NSCLC types, the apparent accuracy of the signature were 94.44 and 98.41%, respectively. In the other integrated dataset for frozen tissues, the signature reclassified 4.22% of the 805 pADC patients as SCC and 12% of the 125 pSCC patients as ADC. Similar results were observed in the clinical challenging cases, including FFPE specimens, mixed tumors, small biopsy specimens and poorly differentiated specimens. The survival analyses showed that the pADC patients reclassified as SCC had significantly shorter overall survival than the signature-confirmed pADC patients , consisting with the knowledge that SCC patients suffer poor prognoses than ADC patients. The proliferative activity, subtype-specific marker genes and consensus clustering analyses also supported the correctness of our signature.Using gene expression profiles of pathologically-determined lung adenocarcinomas and squamous cell carcinomas, denoted as pADC and pSCC respectively, we developed a qualitative transcriptional signature, based on the within-sample relative gene expression orderings (REOs) of gene pairs, to distinguish ADC from SCC. The signature consists of two genes, The non-subjective qualitative REOs signature could effectively distinguish ADC from SCC, which would be an auxiliary test for the pathological assessment of the ambiguous cases. Lung cancer is the most frequent cause of cancer-related deaths worldwide. Non-small cell lung cancer (NSCLC) represents around 80% of lung cancers , with twTTF-1 and p63 [Microscopic morphological features observed from hematoxylin-eosin (HE) staining are currently the “golden” standard for the lung cancer histological classification. In general, if there is an adequate tumor specimen and the tumor is well or moderately differentiated, imaging technique is sufficient to determine ADC or SCC . However and p63 , there i and p63 .Therefore, in recent years, considerable efforts have been devoted to extracting signatures based on gene expression profiles to stratify ADC and SCC , 16. HowFortunately, the within-sample relative expression orderings (REOs) of genes, which are the qualitative transcriptional characteristics of samples, are robust against to experimental batch effects and disease signatures based on REOs can be directly applied to samples at the individualized level –26. BesiIn this study, we developed a REOs-based qualitative signature for individualized NSCLC histological reclassification. We tested the robustness of the signature in two datasets with relative unambiguous NSCLC types, concordantly determined by two independent routine pathologists. For the other test datasets, we performed the survival analyses, proliferative activity analyses, subtype-marker genes expressions and consensus clustering analyses to provide evidences that the signature could rectify some misclassifications of histological subtypes by routine pathological assessments. Especially, the sample reclassifications by the signature were validated in various specimen types, including the frozen tissue specimens, formalin fixed paraffin-embedded (FFPE) tissue specimens, small biopsy specimens, mixed tumor specimens with high varied proportions of tumor cells and poorly differentiated tumor (LCC) specimens. Therefore, this signature would be an effective auxiliary tool for precise diagnoses of lung SCC and ADC.a > Eb in pSCC or equally Eb > Ea in pADC) occurring significantly more frequently in pSCC samples than in pADC samples . Next, for each subtype-opposite gene pair, we calculated the apparent accuracy of the gene pair for distinguishing ADC from SCC in the training data, as the pathological assessments are not 100% reliable [KRT5 and AGR2, as the addition of any other gene pair did not increase the apparent accuracy. The other set, consisting of two gene pairs, also contained the gene pair (KRT5 and AGR2), indicating that this gene pair had the optimal performance. Therefore, the gene pair, KRT5 and AGR2, was selected as the signature for distinguishing ADC from SCC. The classification rule of the signature is that a sample was classified as SCC if the mRNA expression level of KRT5 was higher than that of AGR2; otherwise ADC. According to the classification rule, two of the 61 pSCC samples in the GSE30219 dataset and one of the 31 pSCC samples in the GSE18842 dataset were reclassified as ADC and all the 99 pADC samples in the two datasets were confirmed by the signature.Figure reliable . FinallyKrt5 and Agr2 proteins was performed for 96 pADC samples and 80 pSCC samples, derived from Anenabio, Xi’an, China. The IHC results for Krt5 and Agr2 proteins are shown in Fig. Agr2 protein was highly expressed in 63 (65.63%) samples, while Krt5 protein was only highly expressed in 7 (7.29%) samples  samples, while Agr2 protein was only highly expressed in 8 (10.00%) samples  pADC and 13 (16.25%) pSCC samples were low expressed of both Krt5 and Agr2 proteins, suggesting the limitation of IHC of immunomarkers in distinguishing ADC from SCC.Immunohistochemical analysis of the KRT5 and AGR2) had the optimal performance  with relative unambiguous NSCLC types, which were concordantly determined by two independent routine pathologists. In the GSE19188 dataset with 45 pADC and 27 pSCC samples, the apparent accuracy of the signature for pADC (sensitivity) was 93.33%, the apparent accuracy for pSCC (specificity) was 96.30%, and the overall apparent accuracy was 94.44%  using consensus clustering . Here, the signature reclassified 12 (2.41%) pADC samples as SCC, whose proliferation scores were significantly higher than that of the signature-confirmed pADC samples . The 82 pSCC samples were reclassified as ADC by the signature and their proliferation scores were significantly lower than the signature-confirmed pSCC samples  and specificity (pSCC prediction) were 95.24 and 88.89%, respectively (Table NAPSA/TTF1) and SCC markers (Krt5/P40), respectively. The apparent sensitivity (pADC prediction) and specificity (pSCC prediction) of the signature for poorly differentially samples were 100 and 50%, respectively , and all the 20 genes were down-regulated , which can accurately distinguish ADC from SCC and is independent of the subjective experiences of pathologists. It is known that Krt5 protein is expressed primarily in basal keratinocytes in the epidermis, whose overexpression is the unique characteristic of SCC [Agr2, is known as an adenocarcinoma antigen, which promotes cell migration and metastasis [krt5 protein was preferentially expressed in pSCC, while Agr2 protein was preferentially expressed in pADC, providing biological evidences of the signature in distinguishing ADC from SCC. However, it is note that, there were 8 (4.55%) samples with high expressions of the two proteins and 25(14.20%) samples with low expressions of the two proteins, indicating that a certain percentage of patients could not be accurately classified by the immunohistochemical assessment. The limitation was also observed in the commonly used immunomarkers for ADC and SCC [In this study, we developed a robust qualitative transcriptional signature consisting of two genes  was 96.48%, the apparent specificity (SCC prediction) was 84.90%, and the overall apparent accuracy was 92.55%. Notably, there were large differences in the apparent accuracy (or reclassification) rates among cohorts. It has been reported that different pathologists had discordant histological classifications for about one third of the cases because of their subjective diagnoses using varying criteria or interpretations , which mp = 0.0012, HR = 1.60, 95% CI = 1.20–2.12, Fig. p = 0.0130, HR = 1.45, 95% CI = 1.08–1.95, Additional file Previously, several gene expression signatures to stratify ADC and SCC have been reported , 16, 36.However, a limitation of this study is that most of the samples we used were diagnosed according to WHO 2004 criteria because few samples diagnosed according to WHO 2015 criteria could be found in public data. It is known that the current criteria auxiliary with immunostaining could reclassify a subset of cases diagnosed with the WHO 2004 criteria . HoweverThe non-subjective qualitative REOs signature would be an auxiliary test for the pathological assessment of the ambiguous cases, including the cases lacking the evidences of glandular or squamous differentiation. Notably, it is worthwhile to measure all possible qualitative transcriptional signatures together  for disease diagnosis, histological classification, prognosis and drug evaluation of lung cancer through a single whole (or a large panel of genes) RNA-sequencing for a sample to aid or even replace multiple conventional clinical detection, which could preserve precious tissue for the other molecular testing such as EGFR mutation detection.http://www.ncbi.nlm.nih.gov/geo/), ArrayExpress (https://www.ebi.ac.uk/arrayexpress/) and The Cancer Genome Atlas .The 15 public gene expression datasets of NSCLC tissues (Table The training data for extracting a REOs-based signature was integrated from two microarray datasets (GSE30219 and GSE18842), including 99 pathologically-determined ADC (denoted as pADC) samples, 92 pathologically-determined SCC (denoted as pSCC) samples and 59 normal controls.In this study, 2 datasets (GSE19188 and E-MTAB-2435), were selected as “golden”standard datasets to test the performance of the signature, as their histological classifications were concordantly determined by two independent pathologists, whose subtypes were relative unambiguous. Seven datasets recording survival information were integrated for survival analyses, including 805 pADC and 125 pSCC samples of stage I-III patients treated with curative surgery resection only. The clinical information of the 7 datasets is displayed in Additional file For data generated by the Affymetrix microarray platforms, the Robust Multi-array Average algorithm  was usedp values were adjusted using the Benjamini-Hochberg procedure for multiple testing to control the false discovery rate (FDR) [First, Significance Analysis of Microarrays (SAM) algorithm  was usedte (FDR) . We selea and b) both derived from the subtype-opposite genes, Fisher’s exact test was used to evaluate whether the frequency of pSCC samples with a specific REO pattern (Ea > Eb), where Ea and Eb represent the expression levels of gene a and b, respectively, was significantly higher than the frequency in pADC samples. The significant gene pairs were defined as subtype-opposite gene pairs.Then, for a pair of genes (a and Gb) between the two groups was calculated as Formula a and Gb) in all samples of pADC and pSCC groups, respectively.Finally, for each subtype-opposite gene pair, we calculated the apparent accuracy (Formula 1) of the gene pair for distinguishing ADC from SCC, as the pathological assessments are not 100% reliable . We chosKrt5  or anti-Agr2  primary antibody and then with secondary antibody conjugated with HRP. The slides were developed with DAB substrates, counterstained with haematoxylin, dehydrated and mounted with neutral balsam. Each tissue sample of the microarray was then imaged under miscroscope Carl Zeiss GmbH with same settings. The protein expression was quantified and scored basing on a multiplicative index of the average staining intensity (1–4) and the extent of staining  to 16 . The staining intensity was scored as follows: 1, negative; 2, weak; 3, moderate; and 4, strong. The extent of staining was scored as follows: 1, less than 10% positive cells; 2, 10 to 30%; 3, 30 to 70%; and 4, more than 70%. Score of staining index less than 5 was considered low expression, score of 5–10 considered medium expression and score of 11–16 considered high expression. Fisher’s exact test was used to compare the frequencies of the Krt5 and Agr2 proteins expressions between 96 pADC samples and 80 pSCC samples.The lung cancer array for immunohistochemical analysis was bought from Anenabio, Xi’an, China (Cat#LC2161), including 96 pADC samples and 80 pSCC samples. The slides of human lung cancer tissue microarray used in this study were dewaxed with xylene and rehydrated through gradient ethanol into water. Antigens were unmasked by 10 mM citric acid buffer. Endogenous peroxidase activity was blocked by 3% hydrogen peroxidase. After further blocking for non-specific binding, the slides were incubated with specific anti-To avoid the bias of patient follow-up duration among different datasets, the overall survival (OS) of patients were truncated at 60 months. Survival curves were estimated using the Kaplan-Meier method and were compared using the log-rank test . A multiThe proliferation score for each sample was calculated by averaging the expressions of the 44 genes in a proliferation signature  in a cohKRT5, TP63) [NAPSA, TTF1) [CD56, SYP, CHGA) [The RankProd (RP) algorithm , which i5, TP63) , ADC marA, TTF1)  and neurP, CHGA) .k = 10 groups (k represents the number of clusters) by the agglomerative hierarchical clustering algorithm using Pearson correlation distance. To identify the optimum number of clusters, k corresponding to the first downwards inflection in cumulative distribution function (CDF) was used. Additionally, we also performed the consensus clustering using top 2000 and 3000 most variable genes, respectively.Consensus clustering was performed using the ConsensusClusterPlus package according to the Ward method for hierarchical clustering . The samhttp://www.r-project.org/).All statistical analyses were performed using the R 2.15.3 (Additional file 1: Table S1. Clinical characteristics of patients treated with curative surgery resection only. Table S2. The top 50 subtype-opposite gene pairs. Table S3. The histological classification of samples classified by pathological assessment, signature and clustering in the GSE50081. Table S4. The histological classification of samples classified by pathological assessment, signature and cluster in the GSE58661. Figure S1. Kaplan-Meier curves of OS respectively for the SCC and ADC groups reclassified by the signature and original pathological assessment in the data integrated by 6 datasets. Figure S2. The consensus clustering of the samples based on top 1000 most variable genes in the GSE50081 dataset. Figure S3. The consensus clustering of the samples based on top 2000 most variable genes in the GSE50081 dataset. Figure S4. The consensus clustering of the samples based on top 3000 most variable genes in the GSE50081 dataset. Figure S5. The consensus clustering of the samples based on top 1000 most variable genes in the GSE58661 dataset for small biopsy specimens. Figure S6. The consensus clustering of the samples based on top 2000 most variable genes in the GSE58661 dataset for small biopsy specimens. Figure S7. The consensus clustering of the samples based on top 3000 most variable genes in the GSE58661 dataset for small biopsy specimens. Figure S8. The Kaplan-Meier curves of overall survival respectively for the ADC and SCC groups of patients treated with curative surgery resection only.


Significant advances in biotechnology have allowed for simultaneous measurement of molecular data across multiple genomic, epigenomic and transcriptomic levels from a single tumor/patient sample. This has motivated systematic data-driven approaches to integrate multi-dimensional structured datasets, since cancer development and progression is driven by numerous co-ordinated molecular alterations and the interactions between them. We propose a novel multi-scale Bayesian approach that combines integrative graphical structure learning from multiple sources of data with a variable selection framework—to determine the key genomic drivers of cancer progression. The integrative structure learning is first accomplished through novel joint graphical models for heterogeneous  data, allowing for flexible and interpretable incorporation of prior existing knowledge. This subsequently informs a variable selection step to identify groups of co-ordinated molecular features within and across platforms associated with clinical outcomes of cancer progression, while according appropriate adjustments for multicollinearity and multiplicities. We evaluate our methods through rigorous simulations to establish superiority over existing methods that do not take the network and/or prior information into account. Our methods are motivated by and applied to a glioblastoma multiforme (GBM) dataset from The Cancer Genome Atlas to predict patient survival times integrating gene expression, copy number and methylation data. We find a high concordance between our selected prognostic gene network modules with known associations with GBM. In addition, our model discovers several novel cross-platform network interactions (both cis and trans acting) between gene expression, copy number variation associated gene dosing and epigenetic regulation through promoter methylation, some with known implications in the etiology of GBM. Our framework provides a useful tool for biomedical researchers, since clinical prediction using multi-platform genomic information is an important step towards personalized treatment of many cancers. The last decade has seen a proliferation of multi-platform genomic data, aided partly by the rapid evolution and declining costs of modern technologies, producing high-throughput multi-dimensional data. It is now technologically and economically feasible to collect diverse data on matched patient/tumor samples at a detailed molecular resolution across multiple modalities such as genomics (DNA copy number), epigenomics (methylation), transcriptomics (mRNA/gene expression) and proteomics. Such large scale coordinated efforts include worldwide consortiums such as the International Cancer Genome Consortium , The Cancer Genome Atlas  and more recently the Genomic Data Commons , which have collated data over multiple types of cancer on diverse molecular platforms, to accelerate discovery of molecular markers associated with cancer development and progression. The resulting analytical challenges are to integrate these vast amounts of data into models that accurately predict the complex pathophysiology of cancer and to translate this complexity into clinically actionable outputs, towards the holy grail of precision medicine.Initial studies in cancer genomics relying on single platform analyses (mostly gene expression- and protein-based) have discovered multiple candidate “druggable” targets such as KRAS mutation in colon and lung cancer , BRAF inOur methods are motivated by a TCGA based study in glioblastoma multiforme (GBM), where-in diverse platform-specific features are obtained at genomic, epigenomic and transcriptomic resolutions across matched tumor samples. Our goals are two-pronged: first assess dependence within and between platform-specific features, and second, incorporate the dependence in finding important molecular markers associated with relevant clinical outcomes. Integrating data across platforms has sound biological justifications due to interplay of features between and within the platforms. For example, between platforms, attributes at the genomic/DNA level such as methylation and copy number variation can directly affect mRNA expression, which in turn is known to influence clinical outcomes such as cancer progression times and pathobiology of the tumors. Within platform interactions arise from pathway-based dependencies  as well as dependencies based on chromosomal/genomic location (e.g. copy number data). Furthermore, the molecular features are inherently on different scales: discrete (copy number variation) and continuous (DNA methylation and mRNA expression). In addition, there exist substantial prior knowledge on pathway/graphical interactions between these genes (e.g. from public databases and literature), which can be incorporated to achieve improved estimation, increase signal to noise ratio and more refined biological interpretations. Our proposed approach combines all the above aspects to develop an integrative model for predicting clinical outcomes.There has been a growing but limited literature on statistical and computational approaches exploiting the information garnered from data integration in relating the platforms to the clinical outcome—which is usually the goal of translational research in finding markers of cancer progression. Choi et al.  propose Given that the associations between genes and gene products can be captured efficiently via networks, there is a growing variable selection literature for graph structured genomic covariates coming from a single platform –13] whi whi13] wUnlike previous approaches incorporating prior information to estimate the graph based on single platform data , ), 17]), y ∈ ℜ, to be regressed on a p-dimensional vector of mixed covariates x =  obtained from D*(≥ 2) distinct platforms. However, we note that our approach can be generalized to binary or ordinal outcomes via latent variable based thresholding approaches , where the covariates may be continuous, binary or ordered categorical. The mixed covariates have an underlying graphical structure which is to be estimated (e.g. panel (c) in G0 (e.g. panel (b) in We focus on a univariate continuous response, proaches , 20]. I. Iy ∈ ℜ,V = {1, …, p} and edge set E, and is used to: (1) model dependence between features within and across platforms—in our application, measurements for different platforms are available for the same set of genes, so that the joint modeling across platforms allows for both cis-acting  and trans-acting (across gene locations); and (2) detect potentially overlapping subgroups of features within and across platforms, which define functional modules that work together to drive the clinical outcome. Such modules correspond to cliques in the graph, which are defined as a subgroup of V such that each node in this subgroup is connected to every other node in the subgroup.The graphical modeling approach for mixed data models ordered categorical variables by rounding continuous latent variables , 20], a, a20], aC, O denoting continuous and ordinal (and/or binary) covariates respectively. Let zO denote the generic notation for the latent continuous variable corresponding to ordinal predictor xO, and consider the following graphical model for mixed covariatesND], an, an49], S1 Interactive PlotWe have generated an additional interactive pdf figure containing subpanels (a) and (c) in (PDF)Click here for additional data file.S1 AppendixThis Appendix contains guidance on the choice of the belief parameter based on the degree of confidence that one would like to put on the prior belief.(PDF)Click here for additional data file.S2 Appendixκ = 0); and (b) only 75% confidence was placed on the prior graph knowledge for within platform interactions.This Appendix contains further results on the sensitivity of the BVS-SL approach when the mis-specification of the prior knowledge is varied in simulations. It further contains separate analysis of the TCGA data when (a) no graph information was used ((PDF)Click here for additional data file.S1 DataThe data file is a. Rdata file which contains the survival time and censoring status for subjects, as well as the copy number variation, gene expression and methylation measurements of the probes matched to the 48 genes considered in the real data analysis.(7Z)Click here for additional data file.


DECtp achieved the highest area under curves (AUCs) for all the comparisons, suggesting that cancer purity information is critical for DEG calling between tumor and normal samples. In addition, we applied DECtp into cancer and normal samples of 14 tumor types collected from The Cancer Genome Atlas (TCGA) and compared the DEGs with those called by limma. As a result, DECtp achieved more sensitive, consistent, and biologically meaningful results and identified a few novel DEGs for further experimental validation.Identifying differentially expressed genes (DEGs) between tumor and normal samples is critical for studying tumorigenesis, and has been routinely applied to identify diagnostic, prognostic, and therapeutic biomarkers for many cancers. It is well-known that solid tumor tissue samples obtained from clinical settings are always mixtures of cancer and normal cells. However, the tumor purity information is more or less ignored in traditional differential expression analyses, which might decrease the power of differential gene identification or even bias the results. In this paper, we have developed a novel differential gene calling method called DECtp by integrating tumor purity information into a generalized least square procedure, followed by the Wald test. We compared DECtp with popular methods like  Nowadays, RNA sequencing (RNA-Seq) has become a routine for measuring RNA expression levels   and those from “contaminated” tumor samples as ki and j and μj+wiδj are the adjusted means for normal and tumor samples, and ϕj is the dispersion. The DE is obtained by testing if δj is 0. UNDO is designed for deconvoluting array-based gene expression data of tumor samples  to identify DEGs between tumor and normal samples. DECtp models expression profiles of tumor samples as a mixed Gaussian distribution, where the mixing proportion is tumor purity. With known or estimated tumor purity, differential expressions are then called based on a generalized least square procedure followed by the Wald test. We performed analyses on extensive simulated data with different sample sizes and noise levels and TCGA data of various cancers. DECtp achieves more accurate, consistent, and biologically meaningful results than those from other state of the art methods, such as limma  such that they will follow a Gaussian distribution. This transformation allows for the introduction of a linear model with Gaussian noise in subsequent steps.Supposing that the input data consists of expression profiles of i, let Xi be its transformed expressions on all normal samples. We assume that mi and Xi. Similarly, let Yi be the transformed expressions on “pure” cancer samples for gene i, which also admits a normal distribution. Without loss of generality, we assume Yi = Xi+δi, where δi represents the difference between cancer and normal samples. Clearly, δi is a random variable following a normal distribution with mean μi and variance H0:μi = 0. However in practice, the expression profile of “pure” cancer sample Yi is not observed. Instead, the observed expressions of solid tumor samples are always a mixture of expressions on cancer and normal cells.Specifically, for any gene i on observed tumor samples. For a tumor sample with known purity λs estimated by existing methods, we use i on sample s. Then Xis and Let Xis and i, the linear regression model is trained as follows: Zi = Wβi+ϵi, whereTo solve this problem, we propose a simple linear model and a generalized least square procedure by taking n0+n1) × 1 vector Zi represents expressions from normal and tumor samples with the first n0 entries from normal samples, and the last n1 entries from tumor samples. In addition, W is a matrix of dimensionality (n0+n1) × 2 with the first column consisting of all 1 s and the second column consisting of n0 0 s and n1 tumor purities  for respective tumor samples. βi is the linear model parameter to be determined, and ϵi is the random error. The objective is to test H0:μi = 0.Here, the (H = (WTW)−1WT, and Zi is 2 and σ′2, the residual variances from normal and cancer groups respectively. Given estimated H0:μi = 0 is calculated as t distribution with n0+n1−2 degrees of freedom, and the p-values can be obtained accordingly. False discovery rate (FDR) can be estimated using established procedures such as the Benjamini-Hochberg method . Before stepping into detailed analyses, it is insightful to first examine the relationship between gene expression and tumor purity.We applied and compared DECtp with canonical DEG calling algorithms like limma on a few simulated datasets and cancer datasets downloaded from The Cancer Genome Atlas  in TCGA as an example to illustrate the correlation between gene expression and tumor purity. Specifically, after quantile-normalizing the expression profiles . Similarly, we also classified the genes into 10 subgroups according to their correlations with tumor purity and observed a positive correlation between the t-test statistics and group labels , the gene expressions are more correlated with purities. However, when there is no difference between tumor and normal samples , the gene expressions will have a low correlation with purities. These results revealed that tumor purity will bias differential expression analysis if not correctly accounted for, and our method was motivated from this observation.We conducted the above analyses across 14 cancer types with sufficient normal tissues  including Bladder Carcinoma (BLCA), Breast Invasive Carcinoma (BRCA), Esophageal Carcinoma (ESCA), Head-Neck Squamous Cell Carcinoma (HNSC), Kidney Chromophobe (KICH), Kidney Renal Clear Cell Carcinoma (KIRC), Cervical Kidney renal papillary cell carcinoma (KIRP), Liver Hepatocellular Carcinoma (LIHC), Lung Adenocarcinoma (LUAD), Lung Squamous Cell Carcinoma (LUSC), PRAD, Stomach Adenocarcinoma (STAD), Thyroid Cancer (THCA), and Uterine Corpus Endometrial Carcinoma (UCEC). The top 1000 genes with the largest correlations for each cancer type were shown in Supplementary Table To evaluate DECtp and compare it with other methods, we simulated a few datasets resembling true biological scenarios with different sample sizes and noise levels.We first downloaded from TCGA the LUAD gene expression data  consisting of 517 tumor and 59 matched normal samples. Each RSEM value was transformed to log2 (RSEM + min), where min is the minimum non-zero RSEM value. The log2-transofmred data was quantile normalized, which was then used to generate simulation data.It is worth mentioning that our purpose is to call DEGs between pure normal and pure tumor samples. However, both kinds of samples are infeasible to retrieve in reality, thus we adopted a compromised strategy as follows:i, we simulated expression profile of “pure” normal sample j as mi is the mean expression of gene i across all 59 LUAD normal samples, and For each gene j as i is a true DEG if Similarly, we simulated expression profile of “pure” tumor sample j uniformly from . Plugging in Xij, Yij and λj into the formula j at gene i, which is a mixture of expression profile from “pure” tumor and “pure” normal samples.We generated tumor purity values λXij) and mixed  curve analysis  for BLCA and PRAD. Similar to previous findings, we found that top differentially expressed genes are more correlated with purity than other genes for both DECtp and limma. The trend is clearer for DECtp, indicating that it is better in identifying tumor purity-associated differential genes. The observation holds for all 14 cancer types  are overlapped with those identified by DECtp. limma identified 5,393 DEGs, among which 4180 (78%) are overlapped with those identified by DECtp. Similarly for PRAD, the t-test identified 9,223 DEGs, among which 8696 (94%) are overlapped with those identified by DECtp. Limma identified 8682 DEGs, among which 8271 (95%) are overlapped with DECtp. The overlaps of DEGs for other cancer types were shown in Supplementary Figure t-test , 107621 DEGs overlapped between DECtp and limma , 112772 DEGs overlapped between t-test and limma, suggesting that the three methods are generally consistent. We also have downloaded RNA-seq count data of six cancer types from TCGA, including BLCA, BRCA, HNSC, LUAD, LUSC, and PRAD to investigate the overlaps of DEGs called from DECtp, limma and edgeR. To have a fair comparison, we selected the same tumor and normal samples from the two different data type  when using DECtp and edgeR . The overlaps of DEGs for the three methods were shown in Supplementary Figure We also examined the overlaps of DEGs (at FDR 0.001) called from the Next, we examined the Pearson correlation among test statistics for different cancer types. Even though different cancer types have distinct etiologies, they might still share many genomic and transcriptomic features. We plotted in Figure IRF8, CECR1 and IL10RA for tumor and normal samples. It is clear that the p-values are not statistically significant for limma, i.e., the p-value is 0.872 for IRF8, 0.959 for CECR1, and 0.867 for IL10RA. The middle panel shows the scatter plot of expression profiles versus InfiniumPurify purities, in which the correlations are all very high, especially, −0.68 for IL10RA. The high correlation indicates that the large within group variance of cancer samples is mostly caused by variation in purities for different samples, which dilutes the signals of DEGs. And thus, after removing the effect of tumor purity, we could observe significant difference on expressions of these genes between normal and tumor groups. Indeed, there are many studies linking these 3 genes to breast cancer  models, thus DECtp cannot be directly applied. In the future, it will be interesting to develop similar models using the NB distributions incorporating tumor purity information.Finally, we would like to point out that DECtp may have a few further applications. Similar to differential gene analysis, differential protein and differential methylation analyses have also been widely performed between cancer and normal samples. In principle, DECtp could be applied to any differential analysis between cancer and normal samples given the data is Gaussian. In addition, Aran et al. found that identifying co-expression networks from genomics data without accounting for tumor purity is problematic (Aran et al., WZ and JY conceived the concept of the work. WZ, BH, and HL performed the experiments. WZ, JY, and BH wrote the paper.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Tumors are mixtures of different compartments. While global gene expression analysis profiles the average expression of all compartments in a sample, identifying the specific contribution of each compartment remains a challenge. With the increasing recognition of the importance of non-neoplastic components, the ability to breakdown the gene expression contribution of each is critical. Here, we develop DECODER, an integrated framework which performs de novo deconvolution and single-sample compartment weight estimation. We use DECODER to deconvolve 33 TCGA tumor RNA-seq data sets and show that it may be applied to other data types including ATAC-seq. We demonstrate that it can be utilized to reproducibly estimate cellular compartment weights in pancreatic cancer that are clinically meaningful. Application of DECODER across cancer types advances the capability of identifying cellular compartments in an unknown sample and may have implications for identifying the tumor of origin for cancers of unknown primary. Separating different cell compartments from bulk gene expression data can be challenging. Here the authors present DECODER, which can perform de novo deconvolutions on non-negative matrices including microarray, RNA-seq and ATAC-seq data sets. Therefore, with bulk tumor samples, the analysis of tumor gene expression can be significantly confounded by the presence of nonneoplastic cell types, while the contribution of the tumor microenvironment is difficult to separate. Although laser-capture microdissection (LCM) and single-cell sequencing techniques strive to tackle these problems, both of them present certain limitations. LCM is labor intensive and may influence the quality of the microdissected tissue for further analysis4. Single-cell sequencing is still expensive, computing resource heavy, and currently limited by the lack of comprehensive cell-sorting biomarkers6.Tumor samples are mixtures of distinct cell populations that contribute to intra-tumor heterogeneity, including immune, stroma, and normal cells7, and DNA methylation profiles, e.g., MethylPurify8 and InfiniumPurify9, focus on inferring tumor purity, while expression-based deconvolution methods mainly handle estimation of compartment fractions, as well as extraction of compartment-specific expression profiles2. However, the current expression-based deconvolution methods still pose a number of limitations. Some methods are limited to the presupposition of a certain combination of compartments, such as DeMix 10, UNDO (tumor and stroma)11, and ESTIMATE 12. Other methods, such as DeconRNAseq13 and CIBERSORT14, provide the flexibility to measure any number of specific compartments. However, they require knowledge of the pure expression of compartments as the reference, which is difficult to obtain in practice. Similarly, DSA15 requires lists of marker genes. However, frequently the exact compartments in a sample are unknown, and samples are inherently heterogenous. Therefore, the incomplete knowledge of the number of compartments may hamper the accuracy of calculations of the compartment proportions. In addition, like many other quadratic programming-based algorithms, DSA15 has a minimum sample size requirement to perform the estimation, imposing a need for larger data sets.To eliminate the need of relying on LCM or single-cell-based techniques, a plethora of computational strategies have been developed to deconvolve the mixed signal present in a bulk tumor sample using RNA gene expression, DNA copy number data, or DNA methylation data. Algorithms based on DNA copy-number alterations, e.g., ABSOLUTE16. As a result, we identified two tumor-specific  and two stroma-specific  subtypes, together with exocrine, endocrine, and immune factors. Like other standard NMF applications, the number of factors (K) that the input matrix may be decomposed into must be determined a priority. Although the performance of NMF at different K may be evaluated by silhouette and cophenetic correlation coefficient, this evaluation assumes the exclusive classification of each sample into one of the K clusters18, which may not be as biologically clear cut. In our previous study, we empirically determined K by dedicated manual association of biological relevance to each factor at multiple trial runs of K, which can be time consuming and resource intensive16. Thus, developing a streamlined framework that is able to automatically determine K is very appealing and will have potential applicability to the bulk tumor sample deconvolution of any cancer type.Pancreatic ductal adenocarcinoma (PDAC) is characterized by relatively low tumor purity and large amounts of desmoplastic stroma. Therefore, identifying tumor-specific alterations in PDAC is a continuing challenge. To perform virtual microdissection and study compartment-specific signatures, we previously deconvolved bulk PDAC samples by adapting the nonnegative matrix factorization (NMF) algorithmde novo compartment deconvolution and weight estimation of tumor samples (DECODER), an NMF-based integrated and sophisticated framework for the de novo deconvolution of tumor mixture samples for compartments, and estimation of compartment weights for samples 19, COMPASS20, and ICGC  pancreatic cancer18 RNA-seq data sets, we demonstrate that DECODER provides insight into pancreatic cancer biology with clinical implications. This framework is not only a useful algorithm for de novo and single-sample deconvolution of heterogenous samples, but also to the best of our knowledge, for the first time, provides cancer type-specific derived compartment information.Here, we present les Fig. . We showA, which is a N × M matrix of N rows of genes and M columns of samples, the aim of the de novo deconvolution is to factorize A into two matrices W and H, where W is a N × K matrix, H is a K × M matrix, and K is the number of compartments  projection . Then the ranked list of genes for each compartment was subjected to gene set analysis using the Molecular Signatures Database v3.1 (MSigDB)23 for the biological interpretation and annotation of compartments. We manually annotated five compartments common across cancer types by using MSigDB: immune, basal tumor, activated stroma, histone, and olfactory. Analysis of overlap in the top 250 genes of 269 compartments showed four clusters of closely correlated compartments across cancer types: immune, activated stroma, histone, and basal tumor   data set, nine major compartments were identified using DECODER de novo deconvolution. We analyzed each compartment by examining enriched MSigDB gene sets, and found that seven of the nine were dominant compartments similarly defined in the microarray data set for PDAC: basal tumor, classical tumor, activated stroma, normal stroma, immune, endocrine, and exocrine Fig. . For eacH) was also derived from the de novo deconvolution (Supplementary Data n = 37) showed significantly higher immune weights than those with no tertiary lymphoid structures (n = 84)  Fig. . We also84) Fig.  and ESTI84) Fig. . In addi Fig. . W19, we show that Moffitt Basal-like samples (n = 37) are associated with higher DECODER basal compartment weights and Moffitt Classical samples (n = 113) with higher DECODER classical compartment weights  but not difference (B.-C.)  between DECODER basal and classical tumor weight is associated with survival as continuous variables.Based on previous tumor subtype calls (the Moffitt schema) derived by consensus clustering using exemplar genes for these sampleshts Fig. . Hereafthts Fig. . To deten = 28) and Classical samples (n = 122) were even more pronounced for the calls derived by DECODER B./C. than by the consensus clustering based calls of Moffitt schema  using Moffitt tumor subtype calls as the gold standard. We found that basal compartment weight alone, B./C., and B.-C., have similarly high area under the receiver-operating curve  and applying NNLS to indicate the compartment weights for even a single sample (see the Methods section). Ten-fold cross-validation demonstrated good reproducibility of compartment weights derived from gene weights compared with those derived from de novo deconvolution , where samples were microdissected, the endocrine, exocrine, and immune weights were significantly lower than those in the TCGA PAAD and ICGC data sets , similar to our TCGA PAAD results, we found that ABSOLUTE-based tumor purity, ESTIMATE-based immune and stromal scores correlate with respective compartment weights derived by DECODER  are associated with higher exocrine weights, and the immunogenic samples (n = 20) are associated with higher immune weights  showed the lowest B./C., while the adenosquamous carcinoma samples (n = 4) showed the highest, suggesting that the B./C. is able to identify the extremes of tumor histology26  and Classical samples (n = 52) subtyped by B./C. were differentially associated with patient outcome , skin cutaneous melanoma (D29.22:SKCM), adrenocortical carcinoma (D28.15:ACC), pheochromocytoma and paraganglioma (D29.9:PCPG), prostate adenocarcinoma (D29.10:PRAD), thyroid carcinoma (D21.12:THCA), uterine corpus endometrial carcinoma (D28.6:UCEC), testicular germ cell tumors (D28.23:TGCT), lung adenocarcinoma (D29.27:LUAD), and bladder urothelial carcinoma (D29.24:BLCA) Fig. . Similar28.DECODER also identified organ system-associated compartments. For instance, brain (D29.13:GBM + LGG[Brain]), pan-kidney (D14.8:KIRC + KIRP[Pan-Kidney]), pan-gastrointestinal (D21.5:COAD + STAD + ESCA[Pan-GI]), digestive (D29.26:STAD + ESCA[Digestive]), and pan-squamous (D16.12:Pan-Squamous). For both of the brain tumors (glioblastoma multiforme [GBM] and brain lower grade glioma [LGG]), D29.13:GBM + LGG(Brain) showed exclusively high weight. Intriguingly, the compartment of D29.9:PCPG also showed relatively high weight in GBM and LGG, reflecting the fact that they may be anatomically similar. Comparing with ATAC-seq clusters and iClusters, D29.13:GBM + LGG(Brain) was associated with A5:Brain, and C11:LGG(IDH1 mut) or C23:GBM/LGG(IDH1 wt), respectively. D14.8:KIRC + KIRP(Pan-Kidney) was distinctly highly weighted for kidney clear cell carcinoma (KIRC) and kidney renal papillary cell carcinoma (KIRP), represented as a pan-kidney compartment, which is associated with A1:Kidney/Bile duct and C28:Pan-Kidney. Similarly, D21.5:COAD + STAD + ESCA(Pan-GI) and D29.26:STAD + ESCA(Digestive) were found to be related to the pan-GI system and clusters, with D21.5:COAD + STAD + ESCA(Pan-GI) exhibiting the highest weights in colon adenocarcinoma (COAD) and stomach adenocarcinoma (STAD), and the second highest weights in esophageal carcinoma (ESCA). For ESCA, which may have squamous morphology components, the compartment with highest weight was annotated as the pan-squamous compartment (D16.12:Pan-Squamous), showing the strongest associations with the squamous clusters in ATAC-seq and iCluster as well. D16.12:Pan-Squamous is also the most highly represented in head and neck squamous cell carcinoma (HNSC), lung squamous cell carcinoma (LUSC), and cervical and endocervical cancers (CESC), known to have squamous histologies. Interestingly, while D16.12:Pan-Squamous showed the second highest weights in BLCA, we found that the cancer-specific D29.24:BLCA compartment was overrepresented as well. This is in agreement with the finding that BLCA has very diverse iCluster membershipsN = 141), enabled by sample sufficiency in the data set and as expected by its known heterogeneity29. Less prominent compartments were found in cholangiocarcinoma , TGCT (n = 18), and mesothelioma , which may be due to the small samples sizes available.Compartments D27.24:BRCA-Basal, D20.18:BRCA-Her2 + , D28.11:BRCA-Chr8qAmp, and D27.13:BRCA-Luminal were all found to be associated with breast invasive carcinoma (BRCA) (K) and assume the presence of K compartments in a data set. In contrast, the de novo compartment deconvolution of DECODER is fluid and allows each of the compartments in a data set to be identified at runs of different DECODER is an integrated framework, which we developed to perform de novo compartment deconvolution for any data set with nonnegative values, and conduct efficient compartment weight estimation for even a single sample of cancer types in TCGA. Standard NMF methods pre-define the number of factors , bladder urothelial carcinoma (BLCA), as well as several other cancer types , which may lead to an unbalanced number of compartments identified for different cancers. Therefore, it is possible that more unbiased compartment information may be derived in the setting of a greater number of samples, especially for the cancers with smaller sample sizes.DECODER may be applied to any solid or hematologic malignancy, as well as liquid biopsies, and may also be used for platforms other than gene expression. From a computational perspective, DECODER could be applied to any data type with nonnegative values derived from multiple platforms, e.g. ChIP-seq data, copy-number variations, DNA methylation data and single-cell sequencing platforms. As a proof of concept, we applied DECODER on the PanCan ATAC-seq data set containing 23 cancer types in a combined fashion, and identified compartments associated with cell-of-origin and organ systems, which highly reproduced previous clustering-based interpretations of the PanCan analysis22. Our successful deconvolution on PanCan ATAC-seq data supports the fact that profiled open-chromatin regions are extremely tissue specific35, making ATAC-seq a promising method for distinguishing a complex mixture of cancer types. More ATAC-seq cohorts in metastatic samples will be needed to further validate our results. The application of DECODER using ATAC-seq data may provide more biological information that can better guide treatment for cancers of unknown primary and will need to be studied in the context of future clinical trials.From a clinical perspective, the ability of DECODER to identify sample compartments without the a priori assumption of an actual number of compartments, makes it a powerful tool for comparing tumor heterogeneity across samples and for the very challenging clinical conundrum of cancers of unknown primary. Cancers of unknown primary are metastatic cancers where the primary site of origin cannot be identified. Optimal treatment of these cancers remains a challenge, as knowledge of the primary site guides treatment decisions. We and others have previously shown that metastatic samples are molecularly conserved compared with their primary tumorsA (a N × M matrix of N genes and M samples), highly expressed genes in the top third quartile were subjected to selection of the 5000 most variable ones resulting in M matrix). R  repetitions of fivefold resampling . The pair of gene weights and compartment weights with the lowest residual solution from these 20 instances were then used to seed NMF of C . Subsequently, NNLS was used to findith column/sample in ith column/sample in A′ (a 5000 × M matrix). Of note, while discarding H′, The final NMF outputs two matrices, i.e., gene weight matrix jth row/gene in aj is the jth row/gene in A (a N × M matrix). As a result, With The range of increasing ith column in ith column and the largest loading value in the rest of the columns. Based on the ranking, the top 250 genes are identified. The overlapping percentages converted to decimals of the top 250 genes between each of the factors at ith factor at iith factor at ith factor at ith at iith at ith factor at ith factor at ith factor at The factor ID is generated by concatenating the number of factors at current run  of all the scores, (c) has more than two continuous (adjacently linked) (major) or one (unstable) factor(s) greater than the median quartile (Q2) of all the scores, (d) is with the maximum score among the continuous factors with score higher than Q2 (candidates in a high-score factor block), and (e) has the largest number of factors in the high-score block or has the greatest score if multiple high-score blocks and candidates found. Finally, if a major compartment is found to have over 100 overlapping genes in the top 250 with another major compartment, and both of them have the same linkage, the compartment identified at larger K, and combined to form a final gene weights matrix W. W is saved as a reference for later single-sample compartment weight estimation. Similarly, rows of H for the samples in the bulk measurement A.The linking of all the related factors at multiple runs of plot Fig.. Along e21, assessed by the Kolmogorov–Smirnov statistic with Benjamini–Hochberg correction for significance16. For RNA-seq data, final compartments of TCGA PAAD were annotated based on MSigDB terms and prior knowledge. For the remaining cancer types, the immune, activated stroma, basal tumor, olfactory, and histone compartments were annotated according to MSigDB, with the remaining compartments unannotated  were normalized so that the sum of them equals to 1 for each sample. For PanCan ATAC-seq data set, weights for the 22 major compartments were normalized so that the sum of them equals to 1 for each sample.In TCGA PAAD, samples were randomly partitioned into ten subsets. De novo deconvolution was then performed on each combination of ninefold of samples to derive final gene weights. Then for each onefold of samples, the compartment weights were then estimated by the gene weights derived by the other ninefold of samples by the nonnegative least square (NNLS) algorithm. Finally, the estimated compartment weights were compared with those derived from performing de novo deconvolution on the whole data set.Two-sided paired Wilcoxon rank-sum test was used for comparison of basal and classical tumor weights in subtyped samples. Two-sided Wilcoxon rank-sum test was used to compare ratios and differences of basal and classical tumor weights in subtyped samples. The Kruskal–Wallis test by ranks was used for testing whether the compartment weights originated from the same distribution when more than two categories of samples were involved. In correlation analysis, Spearman correlation was used to compare DECODER-derived weights with ESTIMATE-derived scores, where the scales were largely different. Otherwise, Pearson correlation was involved. For survival analysis, continuous variables were analyzed by Cox proportional-hazards regression model, and categorical variables were analyzed by log-rank test.Further information on research design is available in the Supplementary InformationPeer Review FileDescription of Additional Supplementary FilesSupplementary Data 1Supplementary Data 2Supplementary Data 3Reporting Summary


The potential for astrocyte participation in central nervous system recovery is highlighted by in vitro experiments demonstrating their capacity to transdifferentiate into neurons. Understanding astrocyte plasticity could be advanced by comparing astrocytes with stem cells. RNA sequencing (RNA-seq) is ideal for comparing differences across cell types. However, this novel multi-stage process has the potential to introduce unwanted technical variation at several points in the experimental workflow. Quantitative understanding of the contribution of experimental parameters to technical variation would facilitate the design of robust RNA-Seq experiments.log2fold change| > 2 were applied to the data, no significant differences were observed. The technical component quantified variation arising from particular steps in the research pathway, and compared the ability of different normalization methods to reduce unwanted variance. To facilitate this objective, a liberal false discovery rate of 10% and a |log2fold change| > 0.5 were implemented for the differential expression threshold. Data were normalized with RPKM, TMM, and UQS methods using JMP Genomics. The contributions of key replicable experimental parameters  to variance in the data were evaluated using principal variance component analysis. Our analysis showed that, although the variance for every parameter is strongly influenced by the normalization method, the largest contributor to technical variance was library preparation. The ability to detect differentially expressed genes was also affected by normalization; differences were only detected in non-normalized and TMM-normalized data.RNA-Seq was used to achieve biological and technical objectives. The biological aspect compared gene expression between normal human fetal-derived astrocytes and human neural stem cells cultured in identical conditions. When differential expression threshold criteria of |The similarity in gene expression between astrocytes and neural stem cells supports the potential for astrocytic transdifferentiation into neurons, and emphasizes the need to evaluate the therapeutic potential of astrocytes for central nervous system damage. The choice of normalization method influences the contributions to experimental variance as well as the outcomes of differential expression analysis. However irrespective of normalization method, our findings illustrate that library preparation contributed the largest component of technical variance.The online version of this article (10.1186/s12859-018-2382-0) contains supplementary material, which is available to authorized users. RNA “sequencing by synthesis” (RNA-Seq) emerged over a decade ago as part of a suite of next generation analytical methods that enable high throughput interrogation of genomes and transcriptomes. RNA-Seq is becoming the method of choice for gene expression analyses due to technological advances that have increased genome coverage and reduced sequencing costs. RNA-Seq data acquisition mandates a substantial investment from the investigator; therefore, it is important to understand choices that may introduce bias or decrease the quality of the data. RNA-Seq poses particular challenges for researchers because standardized best practices have yet to be universally adopted . AlthougRNA-Seq has proven especially informative for the detection of genes that are differentially expressed during biological processes such as organism development and disease progression. The identification of differentially expressed genes is achieved with a multi-step workflow, and bias can be introduced in both the data-generation and data-analysis phases. The selection of appropriate normalization and data analysis methods have received considerable attention, and statistical algorithms that are specifically designed to address RNA-Seq data postprocessing and evaluation continue to evolve –9. HowevIn the current study, we quantify the variation introduced during the experimental workflow, with the long term goal of increasing the fidelity of ongoing experiments with human cell lines derived from brain tissue , 16. To RNA-seq data can be confounded by the introduction of unwanted statistical variance at any procedural step of the data generation and data analysis procedures in RNA-Seq experiments Table . VariatiGibco® H9 hESC-Derived Human Neural Stem Cells  were cultured in accordance with previously described protocols . Briefly2. When cultures were ~ 80% confluent, they were rinsed in DPBS  and partially digested with 2 mL of 37 °C StemPRO Accutase for subculturing. When detachment was observed under the microscope, cells were transferred with 9 mL of media to tubes for centrifugation at 210 G for 5 min. Supernatant was removed, cells were triturated in prewarmed media and transferred to T-25 flasks coated with CellStart.Cells were thawed, resuspended in complete hNSC serum free media, and centrifuged. The supernatant containing cryoprotectant was removed before resuspending in complete hNSC serum free media and transferring cells (passage 0) to T-25 flasks (one flask per ampule) coated with CellStart . Media were replenished following every 48 h of incubation at 37 °C, 5% CO2. At ~ 80% confluence (day 5) cells were subcultured by partial digestion and plated in vessels recommended for hNSCs . 48 h after the first passage, media were changed to complete hNSC serum free media. NHA and hNSC were cultured in parallel conditions after this point.Normal human fetal-derived astrocytes  from two donor lots  were cultured according to previously established protocols , 22. Via2 in complete hNSC serum free media. After 24 h, media were replenished with 97% Knockout DMEM/F-12, 1% Glutamax, and 2% StemPro neural supplement. Every 48 h, 75% of the media were replenished with 97% Knockout DMEM/F-12, 1% Glutamax, and 2% StemPro neural supplement while ensuring that cells were not exposed to air. After the 10-day differentiation protocol suggested by the manufacturer, images were captured and RNA was isolated as described in the following sections.The second passages of lot A and lot B from NHA and hNSC were subcultured by partial digestion as described above, and cultured according to the manufacturer’s specifications for spontaneous differentiation as described previously . BrieflyPhase contrast images of living cells were acquired with Metavue image capture software (Molecular Devices) prior to RNA isolation. Images were captured with a Coolsnap HQ CCD camera (Photometrics) attached to the projection port of an inverted Nikon TE-2000 microscope.n = 4) with RIN Values greater than 8.9 were divided in half and submitted to the BioMicro Center at the Massachusetts Institute of Technology where two different libraries for each condition were prepared as follows . 10 ng of total RNA was used as input for cDNA preparation with the SMART-Seq v3 Ultra Low Input RNA Kit for Sequencing (Clonetech) according to the manufacturer’s specifications. Fragmented samples were transferred to the SPRI-works for BioMicro Center adapter ligation, multiplex barcoding, size selection, and enrichment using BioMicro Center PCR primers. An AATI Fragment Analyzer™  was used to assess the libraries for fragment size and distribution. Multiplexed samples were sequenced twice according to the protocol for 150 base pair (bp) paired end (PE) reads on an Illumina NextSeq sequencer. Reads mapping to the forward and reverse strands were pooled because the libraries were not prepared with strand-specific protocols.RNA was isolated as previously described , 22 usinPhred scores were assessed with FastQC, a quality control software program, to evaluate base call accuracy in accordance with previous methods , 23. Rea2 transform the sequence data and to normalize it using three different methods: (1) reads per kilobase of exon per million reads mapped (RPKM), (2) trimmed means of M component (TMM), (3) upper-quartile scaling (UQS). RPKM-normalized data is scaled by a factor that considers the both the library size and gene length. TMM-normalized data represents the average after removing the highest and lowest data points and does not consider library size. UQS-normalized data is adjusted based on the size of the library.Read counts were normalized to account for varying lane sequencing depth and other potential technical effects as described previously . JMP Genp-values for multiple comparisons in the statistical analysis undertaken with JMP Genomics [log2fold change| value of 0.5. For a detailed description of the statistical analyses used in this study, refer to [In accordance with previous approaches, the data were fit to a Poisson model as part of the JMP Genomics ANOVA analysis . PoissonGenomics . The mulGenomics . The varrefer to , 29 and Our experimental design was influenced by the guidelines for preclinical research set forth by the NIH, as previously described . The WA0At the MIT BioMicro center, a single blind protocol was used to collect sequence data without prior knowledge of the nature of the biological samples. Groups were assigned to the sequence data by researchers at NMSU who assessed the outcomes. The standards set forth by the HUGO Gene Nomenclature Committee guided the use of official gene symbols in this manuscript.Morphological differences between human neural stem cells and normal human fetal-derived astrocytes cultured in an identical spontaneous differentiation environment are visible in phase contrast images Fig. . Human nr > 0.98) indicate a strong positive correlation between all technical replicates for both cell lines.FastQC evaluation of RNA-Seq read quality revealed that in forward reads, the average Phred scores for all read positions met the criteria for 99% base call accuracy (Phred score > 20). In reverse reads, base pairs 140–150 dropped in quality from ~ 28 to ~ 14. Sample, library preparation and flow cell showed differences in the raw read count output Table . The fou2 transformed data, RPKM normalized data, and UQS normalized data cluster the samples based on cell line. In contrast TMM normalized data did not cluster samples based on any of the experimental parameters. The UQS normalized data produced the most coherent heat maps . In contrast, the proportion of variance contributed by library preparation was increased to over 50% in TMM normalized data, as compared to under 20% with other normalization methods. In TMM normalized data, the percentage of variance based on cell line decreased to 16% from over 50% with other normalization methods. The greatest contribution from cell line was seen in UQS normalization (75%) as compared to other methods . UQS normalization also resulted in the greatest contribution from the cell lot to the experimental variance (5%), which was almost double the contribution from lot in non-normalized data (3%) and in data normalized with other methods . The increased contributions from cell line and lot that were observed with UQS normalization were accompanied by decreased contributions from library preparation (11%) flow cell (6%), and residual (5%) variance as compared to other normalization techniques.The proportions of variance resulting from different experimental parameters were uncovered using principal variance component analysis with JMP Genomics Table . The norlog2fold change values between replicate samples  to 0.62 .We expected to observe similar gene expression patterns in the two cell lines selected for this study because both are of central nervous system lineage, and were cultured under identical conditions. Previous RNA-seq analyses by our laboratory have implemented differential expression criteria of |dj < 0.1 . When thes Table . Even wilog2fold change| values between cell lines were identified by all normalization methods . Three genes were common to three methods, while six genes were identified by two methods, and one gene was exclusive to one method  and ranged from 413 bp to ~ 28 kb.Four of the top ten genes with the largest |od Table . Nine oflog2fold change| > 0.5), and analyzed collectively. The online resource STRING identified 125 protein nodes, which formed 9 distinct networks with 31 edges under the application of a high stringency filter  revealed that the most enriched metabolic pathways were “Metabolic pathways” , “Neuroactive ligand-receptor interaction”  samples, and marginally reduced the variance introduced by technical parameters of the experimental workflow by 1%  was one of the second most enriched networks for this gene list with five genes present in the network. (TIF 9788 kb)Additional file 2:Figure S2 KEGG analysis of metabolic networks. KEGG analysis identified the most enriched metabolic networks for 143 hNSC-upregulated genes. “PI3K-Akt signaling pathway”  was one of the second most enriched networks for this gene list with five genes present in the network. (TIF 9123 kb)


There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task.P values 0.580 and 0.273, respectively) and inferior performance on data from each other relative to an internal test set . The highest internal performance was achieved by combining training and test data from MSH and NIH , but this model demonstrated significantly lower external performance at IU . To test the effect of pooling data from sites with disparate pneumonia prevalence, we used stratified subsampling to generate MSH–NIH cohorts that only differed in disease prevalence between training data sites. When both training data sites had the same pneumonia prevalence, the model performed consistently on external IU data (P = 0.88). When a 10-fold difference in pneumonia rate was introduced between sites, internal test performance improved compared to the balanced model , but this outperformance failed to generalize to IU . CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH  and 99.98% MSH  radiographs. The primary limitation of our approach and the available public data is that we cannot fully assess what other factors might be contributing to hospital system–specific biases.A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center , Mount Sinai Hospital , and Indiana University Network for Patient Care . These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset. Models trained on data from either NIH or MSH had equivalent performance on IU  on X-rays to diagnose disease have been promising, but it has not yet been shown that models trained on X-rays from one hospital or one group of hospitals will work equally well at different hospitals.Before these tools are used for computer-aided diagnosis in real-world clinical settings, we must verify their ability to generalize across a variety of hospital systems.n = 112,120 from 30,805 patients), Mount Sinai Hospital , and Indiana University Network for Patient Care .A cross-sectional design was used to train and evaluate pneumonia screening CNNs on 158,323 chest X-rays from the National Institutes of Health Clinical Center  with extremely high accuracy and calibrate predictions accordingly.The performance of CNNs in diagnosing diseases on X-rays may reflect not only their ability to identify disease-specific imaging findings on X-rays but also their ability to exploit confounding information.Estimates of CNN performance based on test data from hospital systems used for model training may overstate their likely real-world performance. There is significant interest in using convolutional neural networks (CNNs) to analyze radiology, pathology, or clinical imaging for the purposes of computer-aided diagnosis (CAD) –5. TheseAn underlying premise of the test set implying future generalizability to new data is that the test set is reflective of the data that will be encountered elsewhere. Recent work in computer vision has demonstrated that the true generalization performance of even classic CIFAR-10 photograph classification CNNs to new data may be lower than previously believed . In the The primary aim of this study was to obtain data from three separate hospital systems and to assess how well deep learning models trained at one hospital system generalized to other external hospital systems. For the purposes of this assessment, we chose the diagnosis of pneumonia on chest X-ray for both its clinical significance as well as common occurrence and significant interest . We reprThis study was approved by the Mount Sinai Health System Institutional Review Board; the requirement for patient consent was waived for this retrospective study that was deemed to carry minimal risk. Three datasets were obtained from different hospital groups: NIH , Indiana University Network for Patient Care , and Mount Sinai Hospital  [Deep learning encompasses any algorithm that uses multiple layers of feed-forward neural networks to model phenomena . ClassifNIH data contained only frontal chest radiographs, whereas IU and MSH data contained both frontal and lateral chest radiographs and were found to contain inconsistent frontal and lateral labels on manual review. A total of 402 IU and 490 MSH radiographs were manually labeled as frontal/lateral and randomly divided into groups  and used to train ResNet-50 CNNs to identify frontal radiographs . A totalIU radiographs were manually labeled by curators after review of the accompanying text radiology reports . NIH radMSH radiographs did not initially include labels, so a subset of radiographic reports was manually labeled to train an NLP algorithm that could infer labels for the full dataset. A total of 405 radiographic reports were manually labeled for cardiomegaly, emphysema, effusion, hernia, nodule, atelectasis, pneumonia, edema, and consolidation. To evaluate the NLP algorithm’s performance, these were split into train and test groups . A previously described NLP concept extraction model based on 1- and 2-gram bag-of-words features with Lasso logistic regression was trained to identify reports mentioning pneumonia . AUC, seAs NIH and MSH data contained patient identifiers, all NIH and MSH patients were separated into fixed train (70%), tune (10%), and test (20%) groups . IU dataOf 42,396 MSH radiographs, 39,574 contained a label indicating whether they were portable radiographs; 31,838 were labeled as portable. We identified a subset of 31,076 MSH portable radiographs that documented the department of acquisition, with 28,841 from inpatient wards and 2,235 from the emergency department.n = 15) attached to the original bottleneck layer and sigmoid activation (for binary classification) or a linear layer with output dimension equal to that of the classification label followed by softmax activation (for n > 2 multiclass prediction) [PyTorch 0.2.0 and torchvision were used for model training . All imaTo assess how individual models trained using single datasets would generalize compared to a model trained simultaneously on multiple datasets, we trained CNNs to predict nine overlapping diagnoses  using three different train set combinations: NIH, MSH, and a joint NIH–MSH train set. We were interested only in the prediction of pneumonia and included other diagnoses to improve overall model training and performance. For each model, we calculated AUC, accuracy, sensitivity, and specificity for four different test sets: joint NIH–MSH, NIH only, MSH only, and IU. We report differences in test AUC for all possible internal–external comparisons. We consider the joint MSH–NIH test set the internal comparison set for the jointly trained model. We additionally report differences in test AUC between a jointly trained MSH–NIH model and individual MSH–NIH test sets. The classification threshold was set to ensure 95% sensitivity on each test set to simulate model use for a theoretical screening task. After external review of this analysis, a trivial model that ranked cases based only on the average pneumonia prevalence in each hospital system’s training data and completely ignored radiographic findings was evaluated on the MSH–NIH test set to evaluate how hospital systems alone can predict pneumonia in the joint dataset. We include calibration plots across all reported comparisons as supporting information.After training models for pneumonia and evaluating their performance across sites, additional analysis was planned to better understand a CNN’s ability to detect site and department and how that could affect pneumonia prediction. We trained a CNN to predict hospital systems from radiographs to assess whether location information was directly detectable from the radiograph alone. Radiographs from all three hospital systems were utilized to learn a model that could identify the hospital system from which a given radiograph was drawn. To develop this concept more granularly, for MSH radiographs, we further identified from which hospital unit individual radiographs were obtained . In all cases, we report the classification accuracy on a held-out test set.n = 100), we averaged the softmax probability for each subregion calculated ask,i,j represents the activation at the ith row and jth column of the kth final convolutional layer. To characterize how many different subregions were typically involved in NIH hospital system classification, we report the mean, minimum, and maximum number of subregions that predicted NIH decisively (probability ≥ 95%). To illustrate the contribution of particularly influential features  to classification, we present several examples of heatmaps generated by calculating Yi,j NIH − Yi,j MSH − Yi,j IU for all i,j subregions in an image and subtracting the mean. This additional calculation was necessary to distinguish their positive contribution in the context of many subregions contributing positively to classification probability.We created 7 × 7 sample activation maps, following Zhou and colleagues, to attempt to understand which locations in chest radiographs provided strong evidence for hospital system . For thiWe wished to assess the hypothesis that a CNN could learn to exploit large differences in pathology prevalence between two hospital systems in training data by calibrating its predictions to the baseline prevalence at each hospital system rather than exclusively discriminating based on direct pathology findings. This would lead to strong performance on a test dataset consisting of imbalanced data from both hospital systems but would fail to generalize to data from an external hospital system. To test this hypothesis, we simulated experimental cohorts that differed only in relative disease prevalence and performed internal and external evaluations as described above. Five cohorts of 20,000 patients consisting of 10,000 MSH and 10,000 NIH patients were specifically sampled to artificially set different levels of pneumonia prevalence in each population while maintaining a constant overall prevalence: NIH Severe , NIH Mild , Balanced , MSH Mild , and MSH Severe . The sampling routine also ensured that males and females had an equal prevalence of pneumonia. We refer to these as “engineered prevalence cohorts.” Train, tune, and test splits consistent with prior modeling were maintained for these experiments. CNNs were trained on each cohort in the fashion previously described, and test AUCs on internal joint MSH–NIH and external IU data were compared.2 tests, and proportion CIs were calculated using the Clopper-Pearson interval. All P values were assessed at an alpha of 0.05. Statistical analysis was performed using R version 3.4 with the pROC package and scikit-learn 0.18.1 [To assess AUC differences between classification models, we used either the paired or unpaired version of DeLong’s test for ROC curves as appropriate . Comparin 0.18.1 ,22.The average age of patients in the MSH cohort was 63.2 years (SD 16.5 years), compared to 49.6 years (SD 17 years) in the IU cohort and 46.9 years (SD 16.6 years) in the NIH cohort .2P < 0.001) or IU  data.Positive cases of pneumonia were remarkably more prevalent in MSH data (34.2%) than in either NIH , and comparable external test AUC 0.725 at IU . CNNs trained to detect pneumonia at MSH had internal test AUC 0.802 (95% CI 0.793–0.812), significantly worse external test AUC 0.717 at NIH , and comparable external test AUC 0.756 at IU . A jointly trained MSH–NIH model had internal test AUC 0.931 (95% CI 0.927–0.936), significantly greater than external test AUC 0.815 at Indiana . The jointly trained model had stronger internal performance compared to either constituent site individually   –S14 FigsN = 100 NIH radiographs). Laterality labels were particularly influential  of NIH, 8,386 / 8,388  of MSH, and 737 / 771  of IU test radiographs, and hospital system was a larger source of variation in image data principal components than pneumonia . To idenluential .P < 0.001).A CNN trained to identify individual departments within MSH accurately identified 5,805 / 5,805  of inpatient radiographs and 449 / 449  of emergency department radiographs. Patients who received portable radiographs on an inpatient floor had a higher prevalence of pneumonia than those in the emergency department , MSH Mild AUC 0.860 , NIH Mild AUC 0.807 , and NIH Severe AUC 0.849  than when it was balanced with AUC 0.739 (95% CI 0.707–0.772)  .P < 0.001; MSH Mild AUC 0.650, 95% CI 0.548–0.752, P < 0.001; NIH Mild AUC 0.703, 95% CI 0.616–0.790, P = 0.027; NIH Severe AUC 0.683, 95% CI 0.591–0.775, P < 0.001). Internal MSH–NIH performance did not significantly exceed external IU performance for Balanced .Internal MSH–NIH performance of all models trained on imbalanced cohorts was significantly better than their corresponding external performance on IU  was higher than performance on either individual dataset , likely because the model was able to calibrate to different prevalences across hospital systems in the joint test set but not individual test sets. A simple calibration-based non-CNN model that used hospital system pneumonia prevalence only to make predictions and ignored image features achieved AUC 0.861 because of the large difference in pneumonia prevalence between the MSH and NIH test sets. Calibration plots confirmed that a model trained on NIH data was poorly calibrated to MSH and vice versa.By engineering cohorts of varying prevalence, we demonstrated that the more predictive a hospital system was of pneumonia, the more it was exploited to make predictions, which led to poor generalization on external datasets. We noted that metallic tokens indicating laterality often appeared in radiographs in a site-specific way, which made site identification trivial. However, CNNs did not require this indicator: most image subregions contained features indicative of a radiograph’s origin. These results suggest that CNNs could rely on subtle differences in acquisition protocol, image processing, or distribution pipeline  and overlook pathology. Radiological imaging of the same modality from different hospital systems can have markedly different CNN image feature distributions, as we highlighted in A difficulty of using deep learning models in medicine is that they use a massive number of parameters, making it difficult to identify the specific variables driving predictions and complicating the traditional understanding of model overfitting. Best-practice guidelines within the biomedical literature put a heavy emphasis on identifying all features used and transparently reporting model coefficients to promote the development of reproducible, clinically relevant models ,24. ThisEven the development of customized deep learning models that are trained, tuned, and tested with the intent of deploying at a single site are not necessarily a solution that can control for potential confounding variables. At a finer level, we found that CNNs could separate portable radiographs from the inpatient wards and emergency department in MSH data with 100% accuracy and that these patient groups had significantly different prevalences of pneumonia. It was determined after the fact that devices from different manufacturers had been used in the inpatient units (Konica Minolta) and emergency department (Fujifilm), and the latter were stored in the picture archiving and communication system (PACS) in an inverted color scheme  along with distinctive text indicating laterality and use of a portable scanner. While these identifying features were prominent to the model, they only became apparent to us after manual image review. If certain scanners within a hospital are used to evaluate patients with different baseline disease prevalences , these may confound deep learning models trained on radiological data. Fully external testing—ideally on a collection of data gathered from a varied collection of hospitals—can reveal and account for such sampling biases that may limit the generalizability of a model.The development of CNN architectures specifically adapted to accommodate radiological imaging is an important step towards building stronger models. Entire high-resolution radiological images are often aggressively downsampled  to facilitate “transfer learning,” i.e., fine-tuning preexisting CNN architectures that have been pretrained to ImageNet . While pWhile our analysis found degradation of model performance on external test sets, we note that it is possible for external test set performance to be either better or worse than internal. Many different aspects of dataset construction  and the underlying clinical data  can affect performance. For example, a model trained on noisily labeled data that included all available imaging might reasonably be expected to have lower internal test performance than if tested externally on a similar dataset manually selected and labeled by a physician as clear examples of pathological and normal cases.In addition to site-specific confounding variables that threaten generalizability, there are other factors related to medical management that may exist everywhere but undermine the clinical applicability of a model. As has been noted, chest tubes that treat pneumothorax frequently appear in studies positive for pneumothorax in NIH data; a CNN for pneumothorax may learn to detect obvious chest tubes rather than a subtler pneumothorax itself . If suchThere are several limitations to this study. Most notably, without more granular details on the underlying patient populations, we are unable to fully assess what factors might be contributing to the hospital system–specific biasing of the models. The extremely high incidence of pneumonia in the MSH dataset is also a point of concern; however, we attribute this to differences in the underlying patient populations and variability in classification thresholds for pathology. First, a majority of MSH radiographs were portable inpatient scans, ordered for patients too unstable to travel to the radiology department for a standard radiograph. In contrast, all IU radiographs were outpatient. While the inpatient/outpatient mix from NIH is not reported, we believe it likely contains a substantial outpatient percentage given that the incidence of pneumonia is similar to IU. Second, our NLP approach for MSH assigned positive ground truth labels more liberally than NIH or IU, marking a study as positive for pathology when a radiologist explicitly commented on it as a possibility in a report, indicating that the radiographic appearance was consistent with the finding. Different radiologists may have different thresholds at which they explicitly include a possible diagnosis in their reports. Researchers working in this area will continually have to make decisions about their classification threshold for labeling a study positive or negative. We believe that either of these two factors can drive large differences in prevalences of pathology across datasets, and this variation can confound diagnostic CNNs.An additional limitation was that radiologic diagnoses are made in the context of a patient’s history and clinical presentation, something not incorporated into our approach. Positive findings on chest radiograph are necessary but not sufficient for the diagnosis of pneumonia, which is only made when the patient also exhibits a “constellation of suggestive clinical features” . FinallyPneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons. When models were trained on pooled data from sites with different pneumonia prevalence, they performed better on new pooled data from these sites but not on external data. CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.S1 TableNLP, natural language processing.(XLSX)Click here for additional data file.S2 TableMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(XLSX)Click here for additional data file.S1 Fig(TIF)Click here for additional data file.S2 Fig(TIF)Click here for additional data file.S3 FigIU, Indiana University Network for Patient Care; MSH, Mount Sinai Hospital.(TIF)Click here for additional data file.S4 FigMSH, Mount Sinai Hospital.(TIF)Click here for additional data file.S5 FigMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S6 FigMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S7 FigIU, Indiana University Network for Patient Care; MSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S8 FigMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S9 FigMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S10 FigMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S11 FigIU, Indiana University Network for Patient Care; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S12 FigMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S13 FigMSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S14 FigNIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.S15 FigPrincipal component analysis was performed on each image’s bottleneck features and colored to reveal the distribution of hospital systems and pneumonia. MSH images are largely separable from radiographs acquired at IU and NIH. IU, Indiana University Network for Patient Care; MSH, Mount Sinai Hospital; NIH, National Institutes of Health Clinical Center.(TIF)Click here for additional data file.