This bias had great effect on the downstream Gene Ontology over-representation analysis. However, such a bias has not been systematically analyzed for different replicate types of RNA-seq data.In differential expression analysis of RNA-sequencing (RNA-seq) read count data for two sample groups, it is known that highly expressed genes (or longer genes) are more likely to be differentially expressed which is called We show that the dispersion coefficient of a gene in the negative binomial modeling of read counts is the critical determinant of the read count bias (and gene length bias) by mathematical inference and tests for a number of simulated and real RNA-seq datasets. We demonstrate that the read count bias is mostly confined to data with small gene dispersions , and many biological replicate data from unrelated samples do not suffer from such a bias except for genes with some small counts. It is also shown that the sample-permuting GSEA method yields a considerable number of false positives caused by the read count bias, while the preranked method does not.We showed the small gene variance  is the main cause of read count bias (and gene length bias) for the first time and analyzed the read count bias for different replicate types of RNA-seq data and its effect on gene-set enrichment analysis.The online version of this article (doi:10.1186/s12864-017-3809-0) contains supplementary material, which is available to authorized users. High-throughput cDNA sequencing (RNA-seq) provides portraits of the transcriptome landscape at an unprecedented resolution , 2. RNA-normalization: In contrasting RNA-seq counts between different cell conditions, each sample can have different sequencing depths and RNA compositions. Therefore, appropriate normalization should be applied to make the gene expression levels comparable or to estimate the model parameters [probability modelling: Since they are counting data, discrete probability models  have been used to test the differential expression (DE) of genes. Parameter estimation is a critical issue especially for data with small replicates [biases in DE analysis: striking biases with DE analysis of RNA-seq count data were found in that highly expressed genes or long genes had a greater likelihood of being detected to be differentially expressed, which are called the read count bias and gene length bias, respectively [GO analysis) such that GO terms annotated to many long genes had a greater chance of being selected. A resampling based method was eventually developed to account for the selection bias in GO analysis [Poisson model and mostly for the technical replicate data [Poisson model  and biological replicate data.Molecular biological research has focused on questions such as ‘what happens in the cell’ and ‘what changes between differing cell conditions’. While the sequencing technology has shown advantages for answering the former question, the latter gave rise to some complicated issues as follows: (1) rameters –12. (2) plicates , 13, 14.ectively . These banalysis  and follanalysis , 18. Becanalysis , 20. Notate data . Thus, sgenetically identical (GI) replicate data (generated from cell lines or inbred model organisms). In contrast, the replicate data from unrelated individuals, denoted by unrelated replicates, had overall tens to hundreds times greater gene dispersion values than those of technical replicate data, and DE analysis with such unrelated replicate data did not exhibit the read count bias except for genes with some small read counts (< tens). Such a pattern was observed for different levels of DE fold changes and sequencing depths. Although DE analysis of technical replicates is not meaningful, it is included to contrast the patterns and pinpoint the cause of read count bias. Lastly, it is shown that the sample-permuting gene-set enrichment analysis (GSEA) [In this study, it is shown that the gene dispersion value as estimated in the negative binomial modelling of read counts , 14 is ts (GSEA)  is highls (GSEA) . We alsos (GSEA) .gi is defined as follows:In DE analysis of RNA-seq count data between different sample groups, it is known that genes with a larger read count (or longer genes) are more likely to be differentially expressed , 16. We μik and σik are the mean and standard deviation of ith gene gi and sample group k (k = 1 or 2) for the read count data normalized with the DESeq median method [negative binomial variance across the samples. In other words, SNR score can largely represent the distribution of gene differential expression score (effect size/standard error). Thus, these normalized counts have been used for GSEA of RNA-seq data [where n method . Althougseq data –26.The SNR scores for the four datasets were plotted in the ascending order of the mean read count of each gene in Fig. sample replicate type  is a key factor that determines the read count bias, whereas the replicate number and the depth exercise only a limited effect. To corroborate the evidence, we analyzed probability models and conducted a simulation test in the following sections.A possible reason for the two distinctly different SNR patterns was the sample replicate type: The former two (Marioni and MAQC-2 dataset) were composed of technical replicate samples while the latter two (TCGA KIRC and TCGA BRCA) of biological replicates obtained from different patient samples. Besides, the replicate size and sequencing depth may affect the power of DE analysis. Because the replicate numbers are equally set to be seven for all the four datasets, we examined the effect of the sequencing depth by down-sampling the counts. The read counts in the two TCGA datasets were down-sampled to the Marioni dataset level which had the lowest depth among the four: We computationally down-sampled the data using binomial distribution  because naïve LRT) statistic for the significance cutoffs  was also plotted in Fig. The SNR scores are also depicted for the voom (TMM)-transformed data  which exith gene in jth sample, denoted by Xij, can be simply assumed to have a Poisson distribution Xij ~ Poisson(μij) where the mean and variance are the same as μij [Xij is modelled by a negative binomial (NB) distribution to account for the increased variability, and denoted as Xij ~ NB where μij and σij2 are the mean and variance, respectively. Its variance is given as σij2 = μij + αiμij2, where αi is the dispersion coefficient for gi that determines the amount of additional variability [αi approaches 0.The main difference between technical and unrelated replicates is the gene-wise variance across the samples. The technical replicate data are generated from the same samples, so most of its variation comes from the experimental noise such as random sampling. In such a case, the read count of  as μij . However as μij , 30. In iability . In partαi for each gene can be estimated using the edgeR package [αi’s for ten publicly available RNA-seq count datasets are shown in Fig. The dispersion coefficient  package  and the The SNR score for biological replicate data is represented asμik and σik are the mean and standard deviation of the normalized counts for ith gene in the sample group k = 1 or 2. For the technical replicate case where the dispersion coefficient αi is close to 0, the SNR value is approximated to,where i is not negligible in (1) and the SNR is estimated aswhich directly depends on the read counts. This accounts for the increasing SNR variation with the technical replicate data in Fig. μi1f) ≥ 1/(μi1f2) where f = μi1/μi2 is the fold change value . Similarly, the lower bound is obtained using inequality αi/f2 ≤ αi/f asusing the inequality 1/(f. The upper bound (2) indicates the SNR values for biological replicate data are bounded by a constant μi1) is demonstrated in Fig. f) and dispersion values. For a dispersion value of 0.1 or higher, the SNR exhibited nearly a ‘flat’ distribution except for some small read counts (< tens), while the SNR rapidly increased for smaller dispersion values. This pattern was observed across different levels of the fold change values. This result accounts for both the ‘divergent’ SNR distribution with the technical replicates and the ‘even’ SNR distribution with the unrelated replicates shown in Fig. where mentclass2pt{minimSNRi| value in (2) is also bounded by T-statistic results in similar relationships between dispersion, read count, fold change as well as replicate size . Read count data for 10,000 genes were simulated using Poisson or negative binomial distributions for four different dispersion values 0, 0.01, 0.1 and 0.3. The means of the 10,000 genes were randomly sampled from the TCGA KIRC RNA-seq data. Therefore, this simulation compares the SNR distributions of the technical (α ≤ 0.01) and unrelated replicate (α ≥ 0.1) data at the same ‘high depth’ of a TCGA dataset. Among the genes, 30% of the genes were chosen and the mean of their test group counts were increased or decreased by 1.3 ~ 4-folds to generate the DE genes (see Methods). Then, the SNR values for each dispersion value were depicted in Fig. μi1 is represented as cNiLi where c is a proportionality constant, Ni is the total number of transcripts and Li is the length of gene i, it can be easily shown that the SNRi in (1) is also bounded by the same constant Li is, while the SNRi becomes proportional to The gene length bias  can simiThe down-sampling analysis in a previous Section is useful for prioritizing the key factor for the read count bias. However, the Marioni data were generated at quite a low depth with a specific purpose of comparing RNA-seq with microarray, and hence the influence of genes with low counts can be amplified. The key point of this paper is that the well-known read count bias (and gene length bias) nearly dissipates in many (or most) unrelated replicate data with a commonly used depth (more than hundreds of median read count) and the small dispersion is the primary cause of the read count bias.To demonstrate this, the SNR distributions of ten publicly available RNA-seq read count datasets were depicted (as boxplots) in Fig. https://www.bioconductor.org) and CRAN (MASS) (https://cran.r-project.org). The proportions of DE genes in each bin of 1000 genes for each method were depicted in Fig. Then, the DE gene distributions along the read count were analyzed using seven different DE analysis methods and corresponding R packages which are available from the Bioconductor  and its effect on GO analysis , 16. HowOverall, this study recommends using unrelated replicates for RNA-seq differential expression analysis and warns of read count bias for some of the genetically identical replicates for which an appropriate adaptation algorithm or the preranked GSEA may be applied for an unbiased functional analysis , 20.Xij of gene i and sample j was generated using Poisson or negative binomial distribution depending on the gene dispersion of each simulation datasetThe read count μij is the mean and σij2 is the variance. Each simulated dataset contained 10000 genes and 20 samples (ten samples for each group). The mean read counts for simulated genes were determined by randomly selecting 10000 median gene counts from TCGA KIRC Inference for two-sample Additional file 2: Figure S1-S4.(DOCX 649 kb)


Diagnostic metagenomics is a rapidly evolving laboratory tool for culture-independent tracing of foodborne pathogens. The method has the potential to become a generic platform for detection of most pathogens and many sample types. Today, however, it is still at an early and experimental stage. Studies show that metagenomic methods, from sample storage and DNA extraction to library preparation and shotgun sequencing, have a great influence on data output. To construct protocols that extract the complete metagenome but with minimal bias is an ongoing challenge. Many different software strategies for data analysis are being developed, and several studies applying diagnostic metagenomics to human clinical samples have been published, detecting, and sometimes, typing bacterial infections. It is possible to obtain a draft genome of the pathogen and to develop methods that can theoretically be applied in real-time. Finally, diagnostic metagenomics can theoretically be better geared than conventional methods to detect co-infections. The present review focuses on the current state of test development, as well as practical implementation of diagnostic metagenomics to trace foodborne bacterial infections in fecal samples from animals and humans. In its widest definition, metagenomics is the study of all genetic material in an environmental sample . The samIn the laboratory, next-generation sequencing (NGS) is the most common way to study metagenomes, either as amplicon or shotgun sequencing. The three main advantages of 16S ribosomal RNA (rRNA) amplicon sequencing, as compared to shotgun sequencing, are that (i) it is cost-effective; (ii) the data analysis can be performed by established bioinformatic pipelines; and (iii) the available reference databases are already quite comprehensive ,4. The tAn obvious area to apply metagenomics is in diagnostics. Diagnostic metagenomics has been defined by Pallen  as detecAt present, foodborne bacterial infections are diagnosed by many different laboratory methods, e.g., culturing, serotyping, resistance profiling, phage typing, hybridizations, and PCR. These methods are often laborious and take some days before a complete characterization is available . It shouThe present review focuses on the current state of test development, as well as practical implementation of diagnostic metagenomics to trace foodborne bacterial infections in fecal samples from animals and humans. First, the methods available for data analysis will be described. Subsequently, the laboratory steps necessary to include in a study are reviewed. Finally, some relevant studies are described. It should be noted that metagenomics of food samples are not included in this review due to the normally very low concentration of foodborne pathogens in such samples that often require a (pre)-enrichment step. Data analysis is an important, but also difficult and time consuming, part of metagenomic studies. In practice, it can be summarized as classifying metagenomic sequences, either reads or assemblies. Classification of metagenomic sequences can be taxonomy-dependent or taxonomy-independent. The first approach, taxonomy-dependent classification, is dependent on a reference database and a comparison method, and it can be based on all sequencing data or only the marker genes ,11. ThisTaxonomy-dependent classification methods are more often used in diagnostic metagenomics, due to the importance for the treatment of a patient with an unknown causative pathogen. This makes databases an integrated and important part of diagnostic metagenomics. However, most available databases at present are insufficient and/or biased to include more model organisms and human pathogens , which iThe second approach, taxonomy-independent classification, also called binning, is only dependent on the data itself and can be based on sequence composition, differential abundance, or both ,22. ThisTo make the classification of metagenomic sequences as correct as possible, a preprocessing of the shotgun data is often necessary. Preprocessing may include trimming, masking, and assembly. Assembly of reads into contigs improves the analysis, but is often a difficult task from a bioinformatics point of view ,4,22. ItCampylobacter jejuni in 10-fold dilutions were analyzed by the taxonomic classifiers Kraken and CLARK. Both classifiers identified false positive reads from Campylobacter in non-spiked, quantitative polymerase chain reaction (qPCR) negative samples. A sorting of Kraken hits to remove false positives was developed in that study. Briefly, the sorting was done by assigning each hit a score, keeping the high-scoring hits, and from these, removing hits for phage and plasmid DNA using the Kraken and basic local alignment search tool (BLAST) [Andersen et al.  showed t (BLAST) . The stuOne of the challenges of metagenomics is to predict the presence and abundance of common, as well as rare, bacterial species: Ranjan et al. , who invStatistical analyses are often used in metagenomic studies to compare relative abundances of single organisms within a sample  or diversity among samples (beta-diversity). The most widely used methods to make results comparable are normalization and rarefying. In normalization, the ratio of an organism in proportion to the library size is calculated. In rarefying, a minimum library size is defined, then libraries with fewer reads are discarded and the remaining libraries are subsampled for all libraries to have the minimum library size. However, McMurdie and Holmes  argue thCampylobacter in chicken; the high bacterial richness in this type of sample complicates detection of the spiking organism.At present, an interesting discussion is the differences between diarrheal patient samples and asymptomatic control samples. Bacterial richness is significantly higher in asymptomatic controls than in patients with diarrhea , and patAnother issue also addressed in a high-prevalence setting is co-infections, found to be quite common . Becker Campylobacter jejuni, and found great variations between the test kits, but no correlation with the total DNA yield. A well-recognized problem in choosing an appropriate method for DNA extraction is the following dilemma: some DNA extraction methods can be gentle and thereby avoid shearing of DNA from those bacteria that are easily lysed, while other DNA extraction kits can be more harsh in order ensure that DNA is extracted from all, and not just easily lysed, bacteria [Diagnostic metagenomics is still at an early stage of development, especially with regards to the choice of laboratory techniques. However, there is no doubt that the choice of laboratory methods strongly influences the result. For example, fecal samples are complex matrices containing many different assay inhibitors and a high load of microbial commensals. Several studies have shown how sample storage, e.g., freezing, influences the bacterial composition of the extracted DNA ,41, and bacteria . In a cobacteria ,47.Andersen et al.  investigDuring the library preparation of sequencing libraries, fragmentation, size selection and PCR steps are known to introduce bias, and the choice of library preparation kit can influence the extracted community. Fragmentation can be done enzymatically or mechanically. Enzymatic fragmentation is often used because of the ease of use, although it results in a relatively high span in the fragment size. Mechanical fragmentation is more precise because of the smaller span in the fragment size, but in return requires specialized and expensive laboratory equipment. For the size selection, the use of solid-phase reversible immobilization beads, also known as AMPure beads , is recommended over gel extraction to avoid GC-bias, though the GC-bias provides a more precise size selection. The step in library preparation that introduces the most bias is PCR amplification, which is more efficient for sequences with a balanced GC-content than for sequences that are either GC- or AT-rich. PCR-free library preparation kits are recommended and are available from Illumina (TruSeq), but in return they require large quantities of input DNA (>1 µg). To minimize the GC-bias, Kapa HiFi polymerase is recommended rather than Phusion polymerase, which is the standard in Illumina library preparation kits. A deeper sequencing gives a higher precision in beta-diversity and, in addition, the detection of more OTUs ,51,52.The most widely used sequencers for metagenomic studies are Illumina HiSeq and MiSeq. Illumina sequencing errors do not randomly occur over the forward and reverse reads, and not all bases are equally often substituted, inserted or deleted. Most of the errors are caught by the quality scores, but some remain and can affect the result of metagenomic studies of very rare bacteria . A knownMetagenomics improve foodborne diagnostics, but we need references. Although the number of studies available is limited, they all indicate the relevance and the diagnostic potential of the method. Taxonomy-dependent methods, as opposed to taxonomy-independent methods are often used in these studies because of the necessity to identify the pathogen. It is often reported that only 1% of the sequencing data are from the target pathogen . The tasC. jejuni because only the illness sample contained reads aligning to C. jejuni. Although the pathogen could be detected in metagenomic data, it was a very limited study, and besides, the dependency of the recovery sample makes the method inapplicable to real-time surveillance situations. One of the first studies to apply diagnostic metagenomics on clinical fecal samples was published in 2008 by Nakamura et al. . The stuEscherichia coli O104:H4 in Germany in 2011. Of the samples in the study, 40 were known to contain the pathogen. On an HiSeq (Illumina), 48 samples were paired-end sequenced with 2 × 151 bp yielding a total of 180 giga base pairs. It was possible to recover a draft genome of the outbreak strain with a coverage of >10× from 10 samples and >1× from 26 samples. Additionally, Shiga-toxin genes were identified in 27 samples. The study illustrates the potential of diagnostic metagenomics in an outbreak, but the method is not as sensitive as the conventional method used for comparison.Loman et al.  performeIn 2016, Schneeberger et al.  publisheE. coli, which is also a commensal, it was necessary to search for the pathogen-specific virulence factors. This method is actually the first one that theoretically could be applied in real-time and, interestingly, without any need for known positive or negative reference samples. However, this would require that the abundance-based threshold is calculated based on a much larger set of samples.In a study published in 2017 by Joensen et al. , 55 clinn = 5) and whole genome sequencing (WGS) (n = 6) as infections with two different strains of Salmonella Heidelberg. Metagenomic datasets comprising 0.06–5.5 Gigabases (GB) of microbial sequencing data were de novo assembled, and the genomes from WGS of the outbreak isolates were used as reference for extracting contigs from the pathogens by BLAST. The metagenomic results were consistent with those from culturing and WGS and it was possible to type the infection strains from the metagenomic dataset. This study proves that a metagenomic sample contains enough sequence data from a pathogen to type it, and additionally, distinguish between two outbreaks with different strains of the same species. However, as this was done using the outbreak strains as references, the difficult part will be to extract the same amount of data from the pathogen in real-time and without this very specific reference. Another study from 2017  conducteC. jejuni with estimated infection loads from 9.2 × 104–1.0 × 109 colony forming units (CFU)/mL were analyzed. The study detected Campylobacter in all clinical samples by Kraken followed by a filtering of hits to remove false positives. The results were in line with the detection limits defined in the study from spiked human fecal samples (7.75 × 104 CFU/mL). The study presents a method that could theoretically be applied in real-time situations, because it does not depend on known positive or negative reference samples. However, this would require that the method be validated on a much larger set of known samples to ensure that the filtering method is robust enough and that the hits are never false positives. The detection limits obtained in the study showed that the sensitivity of the method is comparable to that of conventional methods, although illustrating that the method, as well as the conventional methods, cannot detect low infection loads. In a third study published in 2017 , seven cDue to the limited number of metagenomic studies in foodborne pathogen detection, it is not possible to draw clear conclusions. However, there appear six potential strengths from the studies of diagnostic metagenomics used on human clinical samples: (1) As a proof-of-concept, pathogen reads can be identified from a patient sample when using a healthy sample from the same patient as the control; (2) Relative abundance of sequences matching the pathogen can be used to define true positive hits; (3) It can be useful and, even, necessary to search for pathogen-specific marker genes; (4) A draft genome of the pathogen can be extracted from outbreak samples; (5) A metagenome can contain enough sequence from the pathogen to type it to the strain level, and finally (6) Diagnostic metagenomics can be superior to the conventional methods when diagnosing co-infections.Considering sequence data analysis, there has been much progress in recent years. Many new software tools and pipelines are being continuously developed, as well as new databases and platforms for sharing of data. In all the studies of human clinical samples, a reference strain or a database is used. However, future studies should consider applying a taxonomy-independent classification method. A large fraction of shotgun metagenomic data, often around 50%, is not classified due to missing reference matches. This may be overcome by the use of taxonomy-independent classification methods, depending on the fraction of data possible to assign to an OTU. In spite of this, it would still be necessary to classify these new organisms to enable a comparison of the results from various laboratories. A final consideration on shifting to the taxonomy-independent methods is, whether isolates of outbreak strains are still needed as references, or if it is enough to have a draft genome extracted from shotgun metagenomic data.Diagnostic metagenomics has the potential for earlier and faster identification of samples with antimicrobial resistance genes, which has not been the focus of the present review but deserves a separate effort due to the clinical importance of the issue. There is an urgent need for developing standards throughout the method pipeline. To this end, an important issue in the data analysis is how to identify false positives and false negatives and how to handle them. In several of the clinical studies mentioned, both false positives and false negatives were observed. This knowledge was based on a known reference sample, either from the same patient, from healthy controls or from non-spiked samples. This is a major challenge to overcome before the methods can be used in real-time diagnostics. For diagnostic metagenomics to become truly useful, the method must be able to analyze each sample in an outbreak rapidly and independently of other samples, and to provide a robust and reproducible result. On the laboratory side, it is necessary to develop internationally agreed, standard operating procedures for DNA extraction that ensure a correct representation of the actual microbial community. It can be discussed if the method should be focused on extracting DNA from certain organisms, e.g., bacteria or virus. By sequencing “target DNA” rather than host DNA, the sequencing capacity could be utilized more efficiently. In the current state, all steps in the laboratory, from sample storage and DNA extraction to library preparation and shotgun sequencing, are known to influence the sequence data output. However, not all of the details about the bias that can be introduced in each step are known, thus optimization is still necessary and ongoing. Ideally, diagnostic metagenomics should feature a universal method to be used on all kinds of samples with a consistent, high-quality result. Currently, this may be difficult to achieve when apparently similar (faeces) matrices show different results ,49.Contributing to the harmonization of DNA purification steps, a recent and important study reported a large inter-laboratory comparison of 21 DNA extraction protocols on the same fecal samples, in which differences in microbial community composition were studied . It concWhile sharing of data, preferably raw sequencing data and associated metadata, should be the ultimate diagnostic aim in tracking of foodborne infections , there a


Multivariate analysis incorporating the four classifiers together retained as significant variables the Moffitt “stroma” and Bailey classifications, highlighting the complementarity of classifiers based on tumor epithelium (Bailey) and tumor stroma (Moffitt stroma). Our results reinforce the clinical validity of subtyping in PDAC, which should be regarded as a collection of separate diseases. Beside their clinical utility that remains to be demonstrated, the clinical interest of the subtypes, notably those from Bailey’s and Moffitt’s “stroma” classifiers that show independent prognostic value, will be reinforced by the identification of new biomarkers and/or therapeutic targets in each subtype for designing and testing novel specific targeted therapies.Four molecular classifications of pancreatic ductal adenocarcinoma (PDAC), biologically and clinically relevant and based on gene expression profiles, were established in the recent years, including the Collisson’s, Moffitt’s (“tumor” and “stroma” classifications), and Bailey’s classifications. The aim of this study was to validate the prognostic value of the Moffitt’s classifications and to compare the Collisson’s, Moffitt’s, and Bailey’s classifications in a large series of samples. We collected clinical and gene expression data of PDAC samples from 15 public data sets, resulting in a total of 846 primary cancer samples, including 601 with survival annotation. All samples were classified according to each of the four multigene classifiers. We confirmed the independent prognostic value of the Moffitt “tumor”, Moffitt “stroma”, and Bailey’s classifications, but not that of the Collisson’s classification. Despite a relatively low gene overlap, all classifications were associated with pathological grade, an important prognostic feature and reflect of intrinsic molecular characteristics of tumors. The concordance rate in term of “good-prognosis” The online version of this article (10.1186/s12943-017-0739-z) contains supplementary material, which is available to authorized users. Pancreatic ductal adenocarcinoma (PDAC) is one of the most aggressive human cancers . Its incTwo important questions remain regarding these classifications established in relatively small series. The first one concerns the robustness of their prognostic value, notably the Moffitt’s classification, the Collisson’s and the Bailey’s classifications having been recently challenged in respective series of 118 patients  and 364 http://smd.stanford.edu/cgi-bin/source/sourceSearch) and EntrezGene  to retrieve and update the non-Affymetrix gene chips annotations, and NetAffx Annotation files  for the Affymetrix annotations. The probes were then mapped according to their EntrezGeneID. For the TCGA data, we used the available normalized RNASeq data that we log2-transformed. Finally, we defined the molecular subtype of each sample in each data set separately as defined in the original publications: the three Collisson’s subtypes [We collected clinical and gene expression data of PDAC samples from 15 public data sets , and 49% in the “normal” “stroma subtype” vs. 34% in the “activated” (p = 7.56E-05). In univariate analysis  were associated with OS, as were the AJJCC stage, the pathological type, grade, tumor size, lymph node status. In multivariate analysis .We first searched for correlations between the Moffitt’s classifications (“tumor” and “stroma”) and clinical data  showed . Some classifications were associated with age (Moffitt “tumor”), pathological type (Moffitt “tumor”), pathological tumor size , whereas all were associated with pathological grade, an important prognostic feature of PDACs. Based on the glandular cell differentiation, mitotic index, and nuclear atypia [We compared the correlations between all classifications and clinicopathological data r atypia , the graNext, we assessed the concordance of the four classifications in term of assignment to the poor-prognosis and good-prognosis groups in the 601-sample series. We combined the Bailey’s pancreatic progenitor, immunogenic and ADEX subtypes into a single good-prognosis group because their survival curves were not different. Similarly, we combined the Collisson’s classical and exocrine-like subtypes into a single good-prognosis group. In the Moffitt’s “tumor classification”, the “basal-like” subtype represented the poor-prognosis group, as did respectively the “activated” subtype in the “stroma classification”. Next, we compared the results of the classifications by using two-way contingency-table analyses. All comparisons showed significant correlations. The concordance rate  between the three classifications based on gene lists derived from tumor tissues, and low when considering the concordance with the Moffitt’s stroma classification. Despite higher prognostic value for the Moffitt’s “tumor classification” taken alone, the multivariate analysis incorporating the four classifiers together retained as independent variables the Moffitt’s “stroma” and Bailey’s classifications, highlighting the complementarity of classifiers based on tumor epithelium and stroma.Our study displays some limitations related to the retrospective nature of data sets and associated biases, including the absence of information with respect to survival for all samples, However, our results reinforce the clinical validity of subtypes in PDAC. Of course, their clinical utility , or abilAdditional file 1: Table S1.List of pancreatic cancer data sets included (XLSX 10 kb)Additional file 2: Table S2.Clinicopathological characteristics of samples and correlations with molecular subtypes of the four classifications (XLSX 14 kb)Additional file 3: Table S3.Gene overlapping between the four classifiers (XLSX 8 kb)Additional file 4: Table S4.Common genes between the four multigene classifiers (XLSX 14 kb)Additional file 5: Table S5.Concordance rate between all classifications compared two-by-two in term of outcome prediction (good- or poor-prognosis). All comparisons were significant by using two-way contingency-table analyses (Chi2-test) (XLSX 8 kb)


Gene regulation is one of the most important cellular processes, indispensable for the adaptability of organisms and closely interlinked with several classes of pathogenesis and their progression. Elucidation of regulatory mechanisms can be approached by a multitude of experimental methods, yet integration of the resulting heterogeneous, large, and noisy data sets into comprehensive and tissue or disease-specific cellular models requires rigorous computational methods. Recently, several algorithms have been proposed which model genome-wide gene regulation as sets of (linear) equations over the activity and relationships of transcription factors, genes and other factors. Subsequent optimization finds those parameters that minimize the divergence of predicted and measured expression intensities. In various settings, these methods produced promising results in terms of estimating transcription factor activity and identifying key biomarkers for specific phenotypes. However, despite their common root in mathematical optimization, they vastly differ in the types of experimental data being integrated, the background knowledge necessary for their application, the granularity of their regulatory model, the concrete paradigm used for solving the optimization problem and the data sets used for evaluation.Here, we review five recent methods of this class in detail and compare them with respect to several key properties. Furthermore, we quantitatively compare the results of four of the presented methods based on publicly available data sets.The results show that all methods seem to find biologically relevant information. However, we also observe that the mutual result overlaps are very low, which contradicts biological intuition. Our aim is to raise further awareness of the power of these methods, yet also to identify common shortcomings and necessary extensions enabling focused research on the critical points.The online version of this article (doi:10.1186/s12918-017-0419-z) contains supplementary material, which is available to authorized users. Gene regulation is one of the most important biological processes in living cells. It is indispensable for adapting to changing environments, stimuli, and developmental stage and plays an essential role in the pathogenesis and course of diseases. Mechanistically, the transcription of DNA into RNA is predominantly controlled by a complex network of transcription factors (TFs) see Fig. . These pDue to this importance, many efforts have been devoted to the elucidation of human regulatory relationships and networks. Wide-spread experimental techniques are transcriptome measurements to quantify gene and transcription factor co-expression , chromatSuch measurements and predictions are used by network reconstruction algorithms to predict regulatory relationships and regulatory networks . A plethHere, we review five rather recent methods and conduct a quantitative comparison of their results with the goal to identify their mutual strengths and weaknesses. They all have in common that they assume both the set of regulators (transcription factors or micro RNAs) to be known and the topology of the regulatory network to be given. By combining this background knowledge with specific omics data sets, especially transcriptome data, they try to infer the activity of regulators in a certain experimental condition or disease using mathematical optimization. All presented methods are global methods in the sense that they compute activities genome-wide (as much as represented by the underlying network), thus removing the shortcomings of local methods which ignore cross-talk between sub-models and global effects within samples. The methods predominantly produce a ranked list of regulators, sorted by their activity in a given group of samples; given that a multitude of biological influences is ignored during inference, especially kinetic and temporal effects, their goal cannot be to produce absolute snapshots of regulatory activity. We describe each method in detail and compare them with respect to the most important properties, such as the data being used, the method applied for deriving optimized activity values, or the evaluation performed to show effectiveness. We further implemented a quantitative comparison including four of the presented methods to objectively analyze their results. As contrast, we also include ARACNE  as sixthWe describe in detail five methods which infer transcription factor activity from omics data sets using a background network of transcription factors and the genes they regulate. All use some form of mathematical optimization. To emphasize the common ground of these at-first-sight rather different methods, we explain their underlying models using a simple framework for defining the relationships of transcription factors and genes. This framework is presented first; it should be understood as a least common denominator, not as a proper method for network inference by itself. We then describe five recently published methods for genome-wide TF activity estimation as extensions or constraints to this general framework, namely the approach by Schacht et al.   and directed edges indicate a regulatory relationship between the two connected nodes, for example the influence of a TF on the expression of a gene  is minimized which is achieved via mixed-integer linear programming using the Gurobi 5.5 optimizer.t-test was computed between melanoma and other cell lines of the NCI-60 panel to find differentially expressed genes of melanogenesis. For the resulting genes, regulation models were built and used to predict gene expression in the melanoma cell line data set yielding good prediction performances.Evaluation of the results was performed using expression data from 59 cell lines of the NCI-60 panel , 38 and RACER  aims to integrate generic cell-line data with sample-specific measurements using a two-stage regression see Fig. . FirstlyCompared to our general framework, RACER includes additionally miRNA binding information. It assumes a linear combination, which is not further justified, of the regulatory effects of TFs and miRNAs on mRNA level. RACER can incorporate a variety of sample specific data including mRNA and miRNA expression values, CNV and DNA methylation. Optimization is applied twice to reduce model complexity, where the method first infers sample-specific TF and miRNA activities and uses these, in a second step, to compute general TF/miRNA – gene interactions.i in sample s, c is an intercept, βt,s describes the estimated activity of TF t in sample s and bt,i is the TF – gene binding score for TF t and gene i. The parameter βmi,s stands for the estimated activity of miRNA mi in sample s and is multiplied by ci,mi, the number of conserved target sites on 3’UTR of the target gene i for miRNA mi, and by the expression level of miRNA mi in sample s. θCNV,s  are the regression parameters for CNV signals CNVi,s . Using βt,s and βmi,s from the first regression step, TF – gene and miRNA – gene interactions across all samples are calculated in a second model:bt,i > 0 and conserved target sites ci,mi > 0. The resulting parameters γi,t and γi,mi indicate the strength of a TF/miRNA – gene relationship across all samples. To obtain robust estimates, γi,mi is additionally weighted by the averaged activities of the miRNA.In the first regression step, mRNA, miRNA, CNV and DNA methylation data are used to calculate the sample specific activities:1 penalty on the linear coefficients to induce a sparse solution and to set irrelevant parameters to zero after the fitting. This sparse LASSO solution is obtained through elastic-net regularized generalized linear models. A supplementary feature selection procedure comparing the full model to a restricted model leaving one TF or miRNA out provides the most predominant TF/miRNA regulators. TF binding scores are collected from the generic cell line of erythroleukemia cells K562 from ENCODE for 97 TFs and 16653 genes. Further, the number of conserved target sites on 3’UTR is taken from sequence-based information from TargetScan for 470 miRNAs and 16653 genes. The RACER method is implemented in R and publicly available under http://www.cs.utoronto.ca/~yueli/racer.html.In each of the two regression steps, the optimization criterion is to minimize the sum of squared errors with LThe method was evaluated using expression data from an acute myeloid leukemia (AML) data set from TCGA with 173 samples  via a saRegression Analysis with Background Integration (RABIT) is a method for finding expression regulators in cancer by a large scale analysis across diverse cancer types. It integrates TF binding information with tumor profiling data to search for TFs driving tumor-specific gene expression patterns see Fig. . It can i, B includes values of the f different background factors for gene i, b contains RBP or TF binding information and θ and β are the respective regression parameter vectors. The regression coefficients β are estimated by minimizing the squared difference between measured and predicted gene expression. The regulatory activity score for each TF/RBP is defined by a t-value (regression coefficient divided by standard error) and its significance by the corresponding t-test. If multiple profiles exist for the same TF from different conditions or cell lines, the profile with the highest absolute value of TF regulatory activity score is selected. In a second step, a stepwise forward selection is applied to find a subset of TFs among those screened in step one optimizing the model error. Lastly, TFs with insignificant cross-tumor correlation are removed from the results.In contrast to our general framework, RABIT can, like RACER, make use of CNV and DNA methylation data additionally integrating promoter CpG content and promoter degree information  and takes RBP or TF binding information as regulatory input. The computational model consists of three steps see Fig. . First, http://rabit.dfci.harvard.edu/download.Computationally, the regression coefficients are calculated via the efficient Frisch-Waugh-Lovell method. TF binding information is taken from 686 TF ChIP-seq profiles from ENCODE representing 150 TFs and 90 cell types. Additionally, recognition motifs for 133 RBPs and their putative targets are collected by searching recognition motifs over the 3’UTR regions . An implRABIT was applied to 7484 tumor profiles of 18 cancer types from TCGA using gene expression, somatic mutation, CNV and DNA methylation data. To systematically assess the results, the cancer relevance level of a TF was calculated as percentage of tumors with the TF target genes differentially regulated . A comparison to cancer gene databases, i.e., the NCI cancer gene index project , the BusIn contrast to the previous three methods and to our general framework which directly scores TFs or other regulators, ISMARA  infers the activity of regulatory motifs (short nucleotide sequences) and thereby indirectly deduces the effects of TFs and miRNAs see Fig. . ISMARA p in sample s, cpand cs are intercepts for each promoter and sample, Np,m summarizes the TF/miRNA binding site predictions  and βm,s stands for the estimated motif activities. Like in the other presented methods, the optimization criterion is to minimize the sum of squared error terms between predicted and measured gene expression. Primarily, ISMARA provides the inferred motif activity profiles  sorted by significance and a set of TFs and miRNAs that bind to these motifs representing the key regulators. Further, a list containing their predicted target promoters, associated transcripts and genes, a network of known interaction between these targets and a list of enriched gene ontology categories is displayed. The web service ISMARA is available under http://ismara.unibas.ch.ISMARA takes sample specific measurements and information about regulatory motifs for TFs and miRNAs into account. Based on the input of gene expression data or chromatin state measurements, the input signal is calculated for each promoter in each sample. The input signal levels are modelled linearly in terms of the binding site predictions and unknown motif activities:βm,sto avoid overfitting. Information about regulatory motifs is provided via the annotation of promoters based on deep sequencing data of transcription start sites. To obtain a set of promoters and their associated transcripts, the 5’ ends of mRNA mappings from UCSC genome database are clustered with the promoters. TF binding site predictions in the proximal promoter region are collected using 190 position weight matrices representing 350 TFs from JASPAR, TRANSFAC, motifs from the literature and their own analyses of ChIP-seq and ChIP-chip data. Additionally, miRNA target sites for about 100 seed families are annotated in the 3’UTRs of transcripts associated with each promoter.ISMARA employs a Bayesian procedure with a Gaussian likelihood model and a Gaussian prior distribution for For evaluation, ISMARA was applied to data from well-studied systems and results were compared to the literature. Inferred motif activities were highly reproducible and even more robust than the expression profiles from which motif activities were derived. When comparing samples from 16 human cell types (GEO accession number GSE30611) from younger and older donors, ISMARA was able to identify a key regulator of aging-related changes in expression of lysosomal genes. A joint analysis of the human GNF atlas of 79 tissues and cell lines  and the BiRte  takes a mathematically different approach compared to the abovementioned methods integrating TF/miRNA target gene predictions with sample specific expression data into a joint probabilistic framework see Fig. . CompareR which can be seen as hidden variables) based on the entire gene expression data D and certain model parameters θ:BiRte takes as input differential gene expression data (mRNA), an underlying regulatory network including TF/miRNA – target gene binding information and optionally CNV data, miRNA and TF expression measurements. As opposed to our general framework, biRte defines a likelihood model for the set of active TFs/miRNAs  method where a regulator can switch from an active to an inactive state (switch) or an inactive and an active regulator exchange their activity states (swap). Thereby, the posterior probability for each regulator and condition to influence the expression of its target genes is estimated. Simultaneously, a variable selection procedure is applied to achieve sparsity of the model. The optimization goal is not, as one would expect, to return the configuration with highest posterior probability among all sampled ones but to take marginal selection frequencies during sampling into account and filter those above a defined cutoff. After the determination of active regulators, the associated transcriptional network containing TFs and miRNAs is inferred from the observable differential expression of target genes and target gene predictions for individual regulators.Here https://bioconductor.org/packages/release/bioc/html/birte.html.In practice, the stochastic sampling scheme based on MCMC allows swap operations only when regulators show a significant overlap of regulated targets. The variable selection procedure is implemented via a spike and slab prior  which caSeveral simulations were conducted to study model behavior. On the basis of a human regulatory sub-network and accordingly simulated expression data of 900 target genes biRte was compared to BIRTA , GEMULA http://califano.c2b2.columbia.edu/aracne.We compare ARACNE (Algorithm for the Reconstruction of Accurate Cellular Networks)  as an esI defined asIi,j measures the relatedness of genes gi and gj and equals zero if both are independent. In a second step, the mutual information values are filtered using an appropriate threshold depending on the distribution of all mutual information values between random permutations of the original data set and indirect interactions are removed.ARACNE uses as input only microarray expression profiles and estimates candidate interactions by calculating the pairwise gene expression profile mutual information gi and gk interact only through a third gene gj, thenComputationally, a Gaussian kernel operator is used to calculate mutual information scores. In a subsequent step, the data processing inequality (DPI)  is appliThus, the least of the three mutual information scores can come from indirect interactions only .ARACNE’s performance was evaluated on the reconstruction of realistic synthetic datasets  and on aWe described five recent methods for the genome-wide inference of regulatory activity, namely the approach by Schacht et al., RACER, RABIT, ISMARA, and biRte. They all assume the topology of the regulatory network to be known, cast activity estimation as an optimization problem regarding the difference between predicted and measured values, take different types of sample specific omics data into account, and eventually produce a list of regulators like transcription factors or miRNAs, ranked by their estimated activities in the samples under study. We also included ARACNE which is background knowledge-free and uses only local dependency measures to reconstruct a regulatory network and indirectly infer activities. All of the presented methods essentially follow the same goal, i.e., accurate ranking of regulatory activity, but differ in the types of measurements being integrated, the background knowledge necessary for their application, the complexity and refinement of the underlying model of gene regulation, and the concrete paradigm used for solving the optimization problem. Most of the methods, except for the approach by Schacht et al., are available online via a downloadable implementation, a web service, or an R package providing an operable solution for the interested user. Whereas an overview of the main features of each method ca be found in Table The data sets used for evaluation vary between all methods. Therefore, we further implemented an evaluation framework to compare the method by Schacht et al., RACER, RABIT and biRte in an objective and quantitative way. We used experimental data of three publicly available data sets from TCGA  and a reThe methods differ in the types of measurements being integrated, which corresponds to the level of detail of their model of gene regulations. All six methods use mRNA as input. RACER, RABIT and biRte can also integrate CNV, DNA methylation, TF/miRNA expression data, or somatic mutations. ISMARA calculates an input signal from microarray, RNA-seq, or ChIP-seq data.Additionally, all presented methods use prior knowledge about the underlying regulatory network. These networks are extracted from different data sources and pre-processed in different manners. All methods require at least knowledge about TF – gene relationships, yet RACER, biRte and ISMARA also incorporate information about miRNAs. When using RABIT, the user can choose whether to provide TF or RNA binding protein information. The approach of Schacht et al. and biRte extract regulatory information partly from the commercial MetaCore™ database, whereas the other methods use only publicly available databases, like ENCODE, JASPAR or TRANSFAC. The networks which are used for the evaluations published in the respective papers are publicly available for the case of RACER , RABIT (predicted binding scores of 63 RBP motifs and 17463 genes) and biRte (network for E.coli including 160 TFs). Neither Schacht et al. nor ISMARA make this data available.The methods use different mathematical models to infer regulatory activity. The approach by Schacht et al., RACER, RABIT and ISMARA use linear regression whereas biRte applies a probabilistic framework. ARACNE, as a local method, is based on mutual information. RACER and RABIT can be seen as extensions of the approach by Schacht et al. since they essentially use the same model structure but incorporate more input data types and more classes of regulatory information. Further, RACER applies a two-stage regression to infer regulatory activity.For assessing regulator activities, Schacht et al., RACER, RABIT and ISMARA minimize the sum of error terms between measured and predicted gene expression. However, the methods use rather different algorithms for solving the resulting optimization problem, and also apply different constraints to achieve model sparsity, robustness of inference, and feature selection. In the approach by Schacht et al., the regression model is computed for each gene separately and allows only a maximum number of six regulating TFs. RACER uses a LASSO approach, while ISMARA follows a Bayesian model that infers regulator activities as posterior distributions. LASSO can be interpreted as a Bayesian model using Laplacian priors instead of Gaussian priors in the regression framework obtaining point estimates of the regulatory activities and enforcing sparseness of the solution . In contSchacht et al. and biRte determine activity of regulators over all samples at once, whereas RACER and biRte first infer sample-specific activities which are combined to cross-tumor activities only in a second optimization step. In contrast, ISMARA in first place infers motifs activity; these activities are used to deduce the effects of TFs and miRNAs by their motif binding profiles. ISMARA primarily provides sample specific TF and miRNA activity but also offers an option to group samples and compare average regulatory activity between different conditions. Like biRte and ARACNE, it also infers the network of the regulators themselves.The type and extent of evaluation performed for the different methods vary greatly. They range from direct application to biological problems over the comparison of results to the biological literature to simulation studies. All methods published evaluations results on publicly available datasets, e.g., from the National Cancer Institute, TCGA or GEO, but unfortunately address different tissues and cancer types. Sample-based cross-validation is applied in the work by Schacht et al., RACER, RABIT and ISMARA. The first two of these methods use correlation coefficients between measured and predicted gene expression for assessing prediction quality. RACER, RABIT and biRte compare their results to the outcome of other algorithms and to those of restricted models, for example excluding one type of the input variables. All methods search the literature to compare their predictions to previously published studies on the respective biological question. Overall, ISMARA provide the most extensive biological evaluation using a battery of relevant use cases, whereas biRte excels in systematic simulation studies. Sadly, there are very few works which compare any of the methods presented on the same problem; the only result we are aware of compared ARACNE and biRte regarding their performance in network reconstruction on simulated data, in which biRte attained higher robustness against false positive and false negative target gene predictions .Although certain evaluation steps were carried out for all methods, results in the original papers are not comparable as they used different input datasets, different background regulatory networks, and different evaluation metrics. Therefore, in addition to the comparison of general properties of the methods, we implemented an evaluation framework using three independent and publicly available test data sets to compare the method by Schacht et al., RACER, RABIT and biRte in an objective and quantitative way. All evaluated methods were given the same regulatory network as input.For the evaluation we used experimental data from TCGA  for threFor mRNA gene expression we used processed RNA-seq data in the form of FPKM (fragment per kilobase of exon per million mapped reads) values. The files included Ensembl Gene IDs which were converted to HGNC symbols using the Ensembl  BioMart We restricted our analyses to the samples for which all four input data types were available. When multiple measurements for one sample and data type were available, we used only the first one in alphabetical order of the file name. After this selection procedure, 165 samples remained for COAD, 404 for LIHC and 180 for PAAD. A list including sample and file information is available in Additional file Together with the experimental data, all evaluated methods were given the same regulatory network as input. We used a publicly available human TF – gene network  based onFor the approach by Schacht et al. we re-implemented their method as closely as possible to the original design using Python and the Cuneiform workflow language , 69. DueFor RACER we used the available R scriptsRABIT published a C++ implementation which they provide on their websiteBiRte is available as a bioconductor R package. We used R version 3.3.2 with biRte version 1.10.0 and applied the method “birteLimma” to estimate regulatory activities with the options niter and nburnin set to 10000. As biRte has a randomized component, the resulting TF activities are not exactly the same for different runs. We averaged the final activity scores over 1000 iterations of birteLimma.We conducted the quantitative comparison for the method proposed by Schacht et al., RACER, RABIT and biRte. ISMARA was not included since it is (a) only available as a web service, (b) can only be used with its own, proprietary underlying regulatory network model, and (c) requires the upload of raw data which is prohibited by TCGA’s terms of use. Also ARACNE  was not For our re-implemented method by Schacht et al. and RACER we computed separate models for case and control group and ranked the TFs by their activity difference between the two groups.To ensure the result’s comparability, we first used only mRNA expression data as input to the four methods. In a second evaluation, we included also other omics data sets where possible. BiRte was evaluated on mRNA and CNV data, RABIT on mRNA, CNV and DNA methylation data, and RACER additionally used miRNA expression as input. We obtained lists with the regulators ranked according to the absolute value of their computed activity for each cancer type and method, with and without the use of additional inputs. For each cancer type we calculated the size of the overlaps in the four different results using the top 10 and top 100 regulators. The results for the top 10 regulators using either only mRNA or multiple omics data sets as input are shown in Table When only mRNA is used as input, one TF is commonly found by the three methods RACER, RABIT and biRte in each data set, respectively: PHOX2B for COAD, EPAS1 for LIHC and ELF1 for PAAD. A literature search of these TFs and their targets revealed clear associations to the respective cancer type. The TF obtained commonly for COAD, PHOX2B, is related to TLX2, a gene which has been shown to play a role in the tumorigenesis of gastrointestinal stromal tumors . EPAS1, p-value) is below 0.006. Additionally, the methods do identify different TFs for different data sets, indicating the importance of the actual cancer specific mRNA expression values and that results are not dictated by the background network.Several TFs in the top 10 are found by two of the four methods For instance, RACER and RABIT have four common top 10 TFs  in the COAD data set. However, the top 10 TFs found by the method by Schacht et al. do not overlap with any top 10 TFs of the other methods in any data set. The agreement of RACER, RABIT and biRte in the top 10 TFs hints to the biological importance of the found TFs since this overlap is statistical significant as the probability of finding common TFs in three sets of ten randomly chosen ones out of 429 TFs , their association to liver hepatocellular carcinoma shows that the methods reach their goal of identifying relevant TFs.When looking at the overlap of three of the four methods, the number of overlapping TFs is still the highest for the triplet RACER, RABIT and biRte. For the LIHC dataset two TFs are found in the top 100 of all four methods (E2F4 and SOX10). E2F4 is a downstream target of ZBTB7, which was associated to the expression of cell cycle-associated genes in liver cancer cells . Two tarHowever, when comparing different data sets, the methods tend to rank the same TFs under the top 100 to a greater or lesser extent. For example, the overlap of all top 100 TFs of the three cancer types is only one TF for RABIT and nine TFs for biRte, but 16 TFs for the method by Schacht et al. and even 32 TFs for RACER. Therefore, the results from RABIT and biRte seem to be more cancer type specific and less dependent on the regulatory network than the results from RACER. However, we did not specifically investigate the influence of the underlying network and its topology on the results which would be an interesting point for further research.When not only taking mRNA into account but also miRNA, CNV and DNA methylation, the results are more difficult to compare between the methods, since they all use a different way of combining different types of data due to their models and implementations.We are aware of the lower level of comparability of this approach regarding the multi-omics results in contrast to a scenario, where all methods are evaluated on the same set of input data. However, we intended to use maximum set of input data for each method to cover the effect of the use of multiple omics data sets compared to only mRNA as input.BiRte was evaluated on mRNA and CNV data, RABIT on mRNA, CNV and DNA methylation data, and RACER additionally used miRNA expression as input. Whereas RACER and RABIT considered CNV or DNA methylation data as one background factor and compute only one activity value, biRte evaluated the influence of each CNV separately.The results , and one TF in PAAD . Three of them  were already found when using only mRNA data as input.The overlap of the top 10 of RABIT and biRte in the multi omics case is considerable with three TFs in LIHC , and two apply cross validation techniques to estimate prediction performance . Further, the effect of temporal buffering between TF binding and the actual effect on gene expression is not included in any of the methods.All methods produce a ranked list of regulators. Comparing these results across different methods, even when applied on the same data set and using the same background network, is difficult since no generally accepted benchmarks are available. Therefore, there currently is no objective measure to designate a best method. The closest comparable evaluation effort we are aware of is implemented in the “DREAM5 – Network Inference” challenge , which tWe therefore compared the results of four methods in a quantitative way. The experimental data and the regulatory network we used as input are publicly available to ensure transparency of our results. The results suggest that the methods are able to find biologically relevant information about regulation processes in cancer. However, the result overlaps are rather low . This seems surprising as all methods essentially follow the same goal, i.e., identification of the most differentially active TFs or genes. We think further research is necessary to exactly characterize the specific strengths of each method. Furthermore, we did not investigate the influence of the underlying network on the results, which is another topic for further research.Despite their often rather involved procedures and models, none of the presented methods adequately reflects the biological reality of regulatory activity in cells. A specific disease phenotype is rarely caused by a single gene but rather a product of the interplay of genetic variability, epigenetic modifications and post-transcriptional regulation mechanisms . The preAdditional file 1:Lists information about the samples and files from TCGA included in our quantitative evaluation for all three cancer types . (XLS 697 kb)Additional file 2:Includes an adjacency list of the connected nodes of the TF – gene network. The list includes three columns  where each row indicates an association with the value of “edge” between a TF and a gene. Complexes of TFs are indicated with a separating “.” between their components. (TXT 39 kb)


CD274 (PD-L1), GATA3, IFNG, IL12R, IL12RB2, IL4, PDCD1 (PD-1), PDCD1LG2 (PD-L2), and TBX21 (T-bet) using data of 158 glioblastoma multiforme (GBM) patients with clinical information available at The Cancer Genome Atlas. Principal component analysis of the expression profiling data was used to derive an equation for evaluating the status of Th1 and Th2 cells. GBM specimens were divided based on the median of the Th scores. The results revealed that Th1HighTh2Low and Th1LowTh2Low statuses indicated better prognosis than Th1HighTh2High, and were evaluated based on the downregulation of PD-L1, PD-L2, and PD-1. Furthermore, Th2Low divided based on the threshold, as well as CD274Low and PDCD1Low, were associated with good prognosis. In the Th2Low subgroup, 14 genes were identified as potential prognostic markers. Of these, SLC11A1Low, TNFRSF1BLow, and LTBRLow also indicated good prognosis. These results suggest that low Th2 balance and low activity of the PD-L1/PD-1 axis predict good prognosis in GBM. The set of genes identified in the present study could reliably predict survival in GBM patients and serve as useful molecular markers. Furthermore, this set of genes could prove to be novel targets for cancer immunotherapy.Common cancer treatments include radiation therapy, chemotherapy including molecular targeted drugs and anticancer drugs, and surgical treatment. Recent studies have focused on investigating the mechanisms by which immune cells attack cancer cells and produce immune tolerance-suppressing cytokines, as well as on their potential application in cancer immunotherapy. We conducted expression profiling of  Gliomas are the most common type of primary central nervous system (CNS) tumors and represent 40% of brain tumors . The WorRecent studies have targeted CNS tumors for cancer immunotherapy, and this approach has yielded progress in neurobiology, oncology, and immunology in malignant gliomas . Cancer + T-cells, CD8+ T-cells, and IgG B-cells are the primary effectors of Th1 immunity. Th1 cells are identified based on TBX21 (T-bet) and STAT4 expression [+ T-cells, and B-cells. The key Th2 transcription factors include GATA3 and STAT6 [A recent study performed expression profiling and clinical characterization of the PD-L1 gene using 301 microarray data from the Chinese Glioma Genome Atlas (CGGA) and RNA-Seq data from 675 samples of grade II–IV gliomas, including primary and secondary tumors, from The Cancer Genome Atlas (TCGA) . Howeverpression , 16. On nd STAT6 . TherefoIn this study, we performed the gene expression profiling of CD274 (PD-L1), GATA3, IFNG, IL12R, IL12RB2, IL4, PDCD1 (PD-1), PDCD1LG2 (PD-L2), and TBX21 (T-bet) using 158 primary GBM samples from the TCGA data set, followed by principal component analysis (PCA) to obtain Th1 and Th2 scores. GBM specimens were grouped based on the calculated Th1 and Th2 scores, and patient prognosis was estimated based on the expression levels of the PD-L1/PD-1 axis genes. Furthermore, we evaluated the utility of the obtained signature genes as prognostic markers in GBM to identify patient subgroups with good prognosis and identified novel candidate prognostic markers in GBM. The genes identified in the present study may serve as novel targets for cancer immunotherapy.*P = 0.0121 in the training data set; HR = 2.29, 95% CI 1.77–2.99, and *P < 0.0001 in the validation data set), and the HR for the “living” status was lower than that for the “deceased” status . The other results of multivariate analyses for OS were not consistent with each other because of the small number of samples and the biases.The present study recruited 158 non-treated primary GBM patients (WHO grade IV) whose data were deposited in the “Glioblastoma Multiforme ” data set between 2008 and 2012 Table . The medr2) were not statistically significant . Thus, the expression levels of these genes were independent and were thus used as a valid input for the calculation of Th1 and Th2 scores to estimate prognosis in GBM.Expression data were obtained from 158 primary GBM patients from TCGA, which were derived post-RNA-Seq performed on the tissues obtained by tumor resections and biopsies. However, the tissues from which the samples were derived, e.g. from a central or peripheral tumor or from a hypoxic or normoxic region, are unknown. Nine genes involved in Th1 and Th2 cell functions and/or cancer immunotherapy were selected as predictors Table . The scaP > 0.05, log-rank test), except for GATA3 (*P = 0.0324)  . The Th1Th1 score = 0.7671 × TBX21 + 0.506 × IL12RB1 + 0.2899 × IFNG + 0.2673 × IL12RB2,Th2 score = 1.0 × GATA3 + 0.0057 × IL4.2 values were statistically significant to a certain degree  for Th1-PDCD1 (r2 = 0.649), CD274-PDCD1LG2 (r2 = 0.617), Th2-PDCD1 (r2 = 0.313), PDCD1-PDCD1LG2 (r2 = 0.303), Th2-CD274 (r2 = 0.290), Th1-PDCD1LG2 r2 = 0.242), Th2-PDCD1LG2 (r2 = 0.214), and Th1-Th2 (r2 = 0.202), except for Th1-CD274 and CD274-PDCD1   Figure . On the ) Figure . In addi) Figure , suggestLow group, but not the Th1Low group, showed better prognoses than the Th2High group    , PDCD1LG2 (PD-L2), and PDCD1 (PD-1), which reliably divided the Kaplan-Meier curves and showed the best prognosis among all subgroups , and results of Kaplan-Meier analysis returned a median survival time of 678 days Figure . GBM pat) Figure  and 2D. s Figure , especiaLow subgroup predicted good prognosis based on Kaplan-Meier analysis , PDCD1LG2 (PD-L2), and PDCD1 (PD-1), was associated with the best prognosis among all subgroups  in each analysis , PDCD1LG2 (PD-L2), and PDCD1 (PD-1) were assessed for prediction of GBM prognosis. The Th1 and Th2 subgroups were divided based on thresholds of 6.018 and 3.20, respectively Figure . The Th2s Figure . This tes in GBM . These rLow subgroup with those in the Th2High subgroup (98 genes), CD274Low subgroup with those in the CD274High subgroup (65 genes), and PDCD1Low subgroup with those in the PDCD1High subgroup (90 genes); the genes are presented in a Venn diagram , tumor necrosis factor receptor superfamily member 1B (TNFRSF1B)Low , and lymphotoxin beta receptor (LTBR)Low  subgroups based on the median expression levels. These were found to predict good prognosis compared to each corresponding high subgroup , myeloid cells , and natural killer (NK) cells  in subgroups divided by the Th1 score, the Th2 score, and a combination of the two, compared with the median expression of each gene . The expLow subgroup showed the same magnitudes of downregulation as those in the Th2High subgroup, but not in both the CD274Low and PDCD1Low subgroups, when compared to their corresponding high subgroups , which constitute a loop to IFN-α signaling, causes a reduction in PD-L1 and MHC class I/II expression in glioma cells, as well as an enhanced susceptibility to immune cell lysis of NK cells. Thus, the above findings suggested that autocrine IFN-α signaling, but not IFN-β or IFN-γ signaling, contributes to the immune evasion of gliomas . Similar+ T-cells in cancer and chronic viral infection. PD-1 expression is partly regulated by an exhaustion-specific enhancer that contains the retinoic acid receptor (RAR), TBX21 (T-bet), and SRY-related HMG-box 3 (SOX3) [In addition, sustained expression of PD-1 has been observed in exhausted CD83 (SOX3) . Similar3 (SOX3) . Althoug3 (SOX3) , the cor3 (SOX3) . In addi+ cells, which is differentiated into Th17 cells [Low, PD-L1Low, and PD-1Low subgroups compared with the corresponding high subgroups  deposited between 2008 and 2012. Out of 604 GBM clinical samples, 166 samples with available clinical information and RNA-Seq data were downloaded. Duplicate samples, such as those from recurrent tumors, were removed from the analysis. The final data set comprised 158 samples available for survival distribution analysis and gene expression profiling, as a training data set. And also, 1120 low grade glioma (LGG) and GBM data were downloaded. 558 LGG samples and duplicated samples, such as identical 145 GBM samples including into the training data set as described above, were removed. Finally, out of 417 GBM samples, 413 samples with available clinical information and RNA-Seq data were remained, as a validation data set.Clinical information and RNA-Seq gene expression data were obtained from The Cancer Genome Atlas (TCGA) . Hazard ratios (HR) and 95% confidence intervals (CI) were calculated based on a logistic regression model with respect to clinical variables that were assessed via multivariate analysis with stepwise selection to compare groups. Overall survival (OS) was defined as the date of diagnosis of GBM to the date of death or last follow-up.Clustering was performed using a hierarchical clustering method using JMP built-in modules . Functional GO annotation was performed using GOstat  and The t-test, chi-square (χ2) test, one-way analysis of variance (ANOVA), and Wilcoxon/Kruskal-Wallis test as appropriate. P* < 0.05 was considered statistically significant.Statistical analyses were performed using R software , Biocond


Identify lncRNA competitively regulated subpathway not only can gain insight into the initiation and progression of disease, but also help for understanding the functional roles of lncRNAs in the disease context. Here, we present an effective method, Subpathway-LNCE, which was specifically designed to identify lncRNAs competitively regulated functions and the functional roles of these competitive regulation lncRNAs have not be well characterized in diseases. Moreover, the method integrated lncRNA-mRNA expression profile and pathway topologies. Using prostate cancer datasets and LUAD data sets, we confirmed the effectiveness of our method in identifying disease associated dysfunctional subpathway that regulated by lncRNAs. By analyzing kidney renal clear cell carcinoma related lncRNA competitively regulated subpathway network, we show that Subpathway-LNCE can help uncover disease key lncRNAs. Furthermore, we demonstrated that our method is reproducible and robust. Subpathway-LNCE provide a flexible tool to identify lncRNA competitively regulated signal subpathways underlying certain condition, and help to expound the functional roles of lncRNAs in various status. Subpathway-LNCE has been developed as an R package freely available at  In recent years, with the development of next generation sequencing technologies, large scale long non-coding RNA (lncRNA) have been identified , 2. It het al. demonstrate that key local subregions, rather than completely pathways, is more subtly explainable to the etiology of diseases [Recently, several useful tools that investigate lncRNA function has been developed , 17. Fordiseases , 19. It In this article, we proposed a novel method called Subpathway-LNCE, to identify lncRNAs competitively regulated signal subpathways underlying certain conditon. Firstly, KEGG signal pathways were converted into undirected graphs with genes as nodes and regulated relations as edges. Then, we reconstructed condition-specific lncRNA competitively regulated signal pathways (LRSP) based on matched lncRNA-mRNA expression profiles and their shared miRNAs. We mapped interesting lncRNAs and genes into LRSP, then located subpahtways within pathways according to the “lenient distance” similarity method . FinallyWe evaluated Subpathway-LNCE method using prostate cancer data sets and KIRC data set. Firstly, we demonstrated the effectiveness of our method by identifying lncRNAs competitively regulated subpathways for prostate cancer. We then detected KIRC key competitively regulated lncRNAs that were biologically meaningful. Meanwhile, our results displayed reproducible by analyzing there independent prostate cancer data sets from different data sources. We also tested robust of our method by randomly disturbing matched expression profiles and LRSP.We first applied Subpathway-LNCE with SRA data set of prostate cancer to assess the effectiveness . SubpathIn further analysis, we focused on three subpathways that competitively regulated by lncRNAs Figure . The firin vivo of prostate cancer [The third subpathway, path: 04070_1, was a part of Phosphatidylinositol signaling system Figure . In thise cancer . Though To examine the utility of Subpathway-LNCE for diseases, we applied Subpathway-LNCE with TCGA data set of LUAD. Subpathway-LNCE identified 28 significant lncRNAs competitively regulated subpathways in LUAD data sets with FDR<0.01 , of whicIn this section, we focused on applying our method to dissect key lncRNAs that implicated with disease and further explore its functional roles under disease condition. To do this, we performed Subpathway-LNCE on KIRC dataset. Firstly, we constructed the KIRC-relevant lncRNA competitively regulated signal subpathways network, in which the top rank 20 subpathways in Supathway-LNCE result list and lncRNA that competitively regulated these subpathways were considered. These 20 subpathways belonging to 19 complete pathways of which up to 14 had been well reported associated with cancers . Hub nodin vitro and in vivo and frequently deleted in malignancy [The modulatory relationship HCG18- EGFR appeared up to 10 pathways, of which 7 pathways were cancer-related such as focal adhesion, such as MAPK signaling pathway and etc. EGFR was a receptor tyrosine kinase, its signaling ultimately increased angiogenesis and decreases apoptosis . Overexplignancy . The abop value = 0.0378  in the same data sets for all three different data resources. And we focused on pathways that rank on top 15 of the result list from different P value of differential expression gene in the same data sets for all three different data sets. The number of pathways shared under different P value was up to 10 in the TCGA data set, and in the other data set the number also up to 8 and 7, respectively , TCGA Data Portal (TCGA) and Gene Expression Omnibus (GEO), details see methods. We focused on pathways that rank on top 15 of the result list from these three different data sets. Hypergeometric test was used to evaluate the significance of shared pathways between any two data sets from different resource. We found that pathways shared between SRA and TCGA with 5 Figure . Futherm5 Figure , the resectively . In any We used two methods to test the stability of Subpathway-LNCE, namely randomly introducing noise in the matched mRNA-lncRNA expression profiles and disturbing LRSP. We firstly randomly deleted mRNAs and lncRNAs from mached mRNA-lncRNA expression profiles, and deleted percentage from 5% to 30%, at 5% intervals. Then we repeated the deleted process 100 times and applied the Subpathway-LNCE method on noisily matched mRNA-lncRNA expression profiles. We calculated the mean ratio of recall significant pathways comparing to original significant pathways (FDR < 0.01). In general, the deleted percentage increased, the overlap of significant pathways fell slowly. The Subpathway-LNCE shown the best stability when the deleted ratio was 5%, recalling more than 90% of the pathways. Even after removal of up to 15% of the expression data, the recalling still more than 70%. It indicated that Subpathway-LNCE was robust when the expression data was noise.We next randomly deleted the edges within LRSPs from 5% to 30%, at 5% intervals, then we repeated the deleted process 100 times and applied the Subpathway-LNCE method on SRA prostate data sets for each disturbed LRSP. The mean ratio of recall significant pathways was calculated comparing to original significant pathways (FDR < 0.01). We found that as the deleted percentage increased, the overlap of significant pathways fell slowly Figure . Above aRecently, lncRNAs have been found can function as competitors of mRNAs for miRNA binding, thereby competitively regulating mRNA expression levels and maintain normal biological functions , 10, 14.Subpathway-LNCE which integrated lncRNA-mRNA expression profile and pathway topologies was specifically designed to identify lncRNAs competitively regulated functions. It considers several important aspects as follows. Firstly, lncRNA, which play important roles in various biological processes , 6, reprp value = 0.0378. It suggested LncRNA may be responsible for explaining disease processes thereby presenting opportunities for new therapies.In the result section, we applied Subpathway-LNCE with SRA data set of prostate cancer and choose three different sub region from different aspects to elaborate the effective of our methods. The first is the significant subpathway was path:04020_1, which was a subregion of calcium signaling pathway. Down regulation of lncRNA MEG3 had been reported closely associated with several cancers , 26. MorIn this study, there are differences among three data sets, including experimental method, operational approach etc. So we considered false discovery rate (FDR) to identify differentially expressed genes in different data sets. Actually, many researchers took different threshold values among different data sets for identifying differential genes. For example, Liang et al. applied different FDRs in their study . We thenin vitro and in vivo and frequently deleted in malignancy [In order to ensure the reliability of data, the mRNA-miRNA interactions were experimentally validated, collected from TarBase , mirTarBlignancy . Althoughttps://cran.rstudio.com/web/packages/SubpathwayLNCE/, and our tool provided a flexible usage for changing the significant positive threshold of r value.In summary, Subpathway-LNCE, a novel method that designed to identify lncRNAs competitively regulated signal subpathways, can not only help to understanding the molecular mechanism of diseases, but also will lead to important insight into the functional roles of lncRNAs in pathological states. Moreover, we have implemented Subpathway-LNCE method as an R-based package, which is publicly available on https://tcga-data.nci.nih.gov/tcga/) and Gene Expression Omnibus  [We analyzed three independent prostate data sets, the three prostate data sets was obtained from different data sources: Sequence Read Archive  , The CanSE23316) , and eac(i) SRA data set of prostate cancerTo obtain transcription level of lncRNAs and mRNAs, firstly we used TopHat mapping reads to the reference genome (hg19) , next we(ii) TCGA data set of prostate cancerThe PRAD data set was download from TCGA, and corresponding clinical information was also obtained. The matched mRNA/lncRNA expression profile for prostate cancer were extracted according to our previous study . In brie(iii) GEO data set of prostate cancert test.We mapped 604 258 probe sequences (25 bp in length) to annotated files from GENCODE (GRCH37) with BLAST under probe re-annotation pipeline (see Supplementary Files), resulted with 17254 mRNAs and 3495 lncRNAs. We obtained matched lncRNA and mRNA expression profiles, which included 6 normal samples and 12 cancer samples. MRNAs were considered differentially expressed under a threshold of FDR = 0.1 of Student's The KIRC data set was download from TCGA, and corresponding clinical information was also obtained. Details of these data sets were processed similar with TCGA data set of prostate cancer, which has described above. And the matched lncRNA and mRNA expression profiles included 71 normal samples and 255 cancer samples.The LUAD data set was download from TCGA. Details of the data-set were processed similar with TCGA data set of prostate cancer, which has described above. And the matched lncRNA and mRNA expression profiles included 52 normal samples and 494 cancer samples.Subpathway-LNCE was developed to identify lncRNA competitively regulated signal subpathways underlying certain conditon. The schematic overview is shown in Figure p = 0.05 (ii) Jaccard Coefficient of shared miRNAs rank at top 20%. In order to ensure the reliability of data, those relationships which satisfied with both criteria were retained. Finally, we obtained candidate LncRNA-mRNA competitively regulated network included 6722 lncRNA-mRNA interactions among 1527 genes and 798 LncRNAs.Firstly, we collected lncRNA-associated competing triplets (lncRNA-miRNA-mRNA relationships). The lncRNA-miRNA interactions were obtainted from our previously work and StarBase, whereas experimentally validated mRNA-miRNA interactions were collected from TarBase , mirTarBr value had reached a significant positive threshold were retained (p < 0.05) based on Fisher's Z transform [We converted 191 KEGG signal pathways into undirected graphs kept original pathway structural information using our previously developed R packages [ransform . Then, tn and s parameters control the intensity of regulated signals and the size of candidate subpathways, respectively. We used n = 1 and s = 8 as default parameters.LncRNAs involved in the competing regulation and genes of interests were regarded as signature nodes. These nodes combined with topology of LRSP can help us efficiently positioning lncRNA-regulated subregions. We first mapped signatures nodes into LRSP, then locating subpathways competing regulated by lncRNAs used “lenient distance” similarity combined with network topology structure. In brief, we calculated the shortest path between any two signature nodes, if the number of molecules between each signature pairs was no longer than n, then they were merged into one nodes. Finally, the number of nodes in the molecule sets within pathway no less than s were regarded as candidate subpahtways competing regulated by LncRNAs. The To estimate whether the candidate subpathways were competing regulated by lncRNAs comparing random, we used the Wallenius approximation methods to estimate the significance of candidate subpathways. The following parameters were needed: (i) the number of interesting mRNAs (x); (ii) the number of background mRNAs (n); (iii) the number of background mRNAs involved in this subpathways (m1); (iv) the number of interesting mRNAs annotated into this subpathway (m2); (v) the weight of this subpathway (w), which suggested the intensity of competing regulation by lncRNAs involved in this subpathways. The weight of the subpathway was computed as follows:G is the number of mRNAs of this subpathways, and parameter GL is the number of mRNAs competitively regulated by lncRNAs within this subpathway. β is parameter of control (In this study β = 1). The Wallenius approximation methods was executed using R package BiasedUrn [In above formula, parameter PiasedUrn .K-mean clustering was used to classify KIRC (PRAD) patients into two groups by expression of hub lncRNAs in the lncRNA competitively regulated subpathway network. Then, the Kaplan-Meier method was used to evaluate the difference of survival time between these two groups, and statistical significance was estimated by log-rank test.


To better understand genome regulation, it is important to uncover the role of transcription factors in the process of chromatin structure establishment and maintenance. Here we present a data-driven approach to systematically characterise transcription factors that are relevant for this process. Our method uses a linear mixed modelling approach to combine datasets of transcription factor binding motif enrichments in open chromatin and gene expression across the same set of cell lines. Applying this approach to the ENCODE dataset, we confirm already known and imply numerous novel transcription factors that play a role in the establishment or maintenance of open chromatin. In particular, our approach rediscovers many factors that have been annotated as pioneer factors. Transcription factor binding occurs mainly in regions of open chromatin. For many transcription factors, it is unclear whether binding is the cause or the consequence of open chromatin. Here, we used datasets on open chromatin and gene expression provided by the ENCODE project to predict which transcription factors drive transitions between open and closed states. A signature of such a factor is that its expression values are correlated to chromatin accessibility at its motif across the same set of cell lines. Our method assesses this correlation while accounting for the fact that some tested cell lines are more related than others. We find many transcription factors showing evidence of driving transitions and a high proportion of these transcription factors are known pioneer factors, i.e., they play a role in opening up closed chromatin. For each setting, we filtered motifs that did not overlap at least 150 DHS regions per cell line on average. Only subfamilies passing this filter in all settings were included . Power mildly increased at low CAR ranks for more stringent cutoffs at the cost of fewer motifs passing filtering. However, at false discovery rate of 10% power was nearly identical.Cumulative distribution of CAR ranks at the subfamily level using the three different motif cutoffs: 10(PNG)Click here for additional data file.S6 FigShown are pairwise comparisons of different motif cutoffs. For each cutoff we derived CAR ranks for all tested subfamilies yielding one CAR rank list per cutoff. Pairwise comparisons of these lists were performed in the following manner: For each pair of rank lists, the first list was used to split the tested subfamilies into a ‘CAR set’ and its complement based on whether a subfamily had CAR rank below 10. For the second results list, two separate CAR enrichment curves were drawn, one curve for the ‘CAR set’ defined via the first list (black) and its complement (grey). Rows denote the cutoff used to derive the ‘CAR set’ and columns denote the cutoff used to draw the enrichment curves. For each setting, we filtered motifs that did not overlap at least 150 DHS regions per cell line on average. Only subfamilies passing this filter in all settings were included . We see that CARs predicted are stable with respect to varying motif cutoffs.(PNG)Click here for additional data file.S7 Fig−5 (used throughout the paper) is compared to variable motif cutoffs guided by ChIP-seq data, where motif cutoffs are adjusted such that called binding sites (i.e. DHS sites containing a motif instance) have a fixed validation rate compared to a gold standard defined by ChiP-seq. Chosen validation rates are 0.3, 0.5 and 0.7. For each setting, we filtered motifs that did not overlap at least 150 DHS regions per cell line on average. Only subfamilies passing this filter in all settings were included . While we see some variation in power, the variation is not systematic.Shown are cumulative distribution of CAR ranks at the subfamily level comparing fixed motif cutoff of 10(PNG)Click here for additional data file.S8 Fig−5  is compared to variable motif cutoffs guided by ChIP-seq data, where motif cutoffs are adjusted such that called binding sites (i.e. DHS sites containing a motif instance) have a fixed validation rate when compared to ChiP-seq. Chosen validation rates are 0.3, 0.5 and 0.7. For each setting, we filtered motifs that did not overlap at least 150 DHS regions per cell line on average. Only subfamilies passing this filter in all settings were included . We see that CARs predicted are stable with respect to varying motif cutoffs.Shown are pairwise comparisons of different motif cutoff methods. For each cutoff method we derived CAR ranks for all tested subfamilies yielding one CAR rank list per method. Pairwise comparisons of these lists were performed in the following manner: For each pair of rank lists, the first list was used to split the tested subfamilies into a ‘CAR set’ and its complement based on whether a subfamily had CAR rank below 10. For the second results list, two separate CAR enrichment curves were drawn, one curve for the ‘CAR set’ defined via the first list (black) and its complement (grey). Rows denote the cutoff method used to derive the ‘CAR set’ and columns denote the cutoff method used to draw the enrichment curves. A fixed motif cutoff of 10(PNG)Click here for additional data file.S9 FigRMA normalization and averaging expression values for experiments derived from the cell types. Displayed is a density distribution of the maximal expression variance observed in each subfamily. We partitioned TF subfamilies into two groups depending on whether they had family level CAR ranks of 1 or not. We observe that top ranked subfamilies do have substantially higher variance on average than other subfamilies .We derived the variance of expression for all transcription factors across micro-arrays after (PNG)Click here for additional data file.S10 FigH3K4me3 peak data for 51 cell lines were downloaded from ENCODE and histone-wise motif activity was computed and normalized analogously to for DHS data, regressing out the first principal component. We performed the mixed model regression where H3K4me3-based motif accessibility data are regressed on gene expression adding a random effect with the same covariance structure as the expression matrix (denoted ‘histone’). To assess the DHS-independent contribution of H3K4me3 histone activities, we added DHS-based motif accessibility as a covariate (denoted ‘DHS-adjusted histone’). We see that subfamily ranks for both of these strategies do not substantially enrich in low ranks. While ‘histone’ performs mildly better, this is likely due to correlation between the histone activity and DHS activity. In contrast, when DHS-based motif accessibility data was adjusted for H3K4me3-based motif accessibility, we see a still substantial enrichment (see “histone-adjusted DHS” curve). This experiment was performed by regressing DHS motif accessibility on gene expression while adding H3K4me3-based motif accessibilities as a covariate plus a random effect with the same covariance structure as the expression matrix. This shows that of the two activity measures, only DHS activity substantially associates with expression.(PNG)Click here for additional data file.S11 FigPOU5F1 motif to gene expression values across all genes. We see the strongest association to SOX2 expression.The QQ-plot shows the p-value distribution obtained from the LMM associating the accessibility of the (PNG)Click here for additional data file.S12 FigDisplayed are the precision-recall curves using annotation from Iwafuchi-Doi et al. (2014) as true set. Motif wise PIQ pioneer scores were extracted from Sherwood et al. (2014). For each subfamily, we defined its PIQ pioneer score as the maximal pioneer score for its subfamily members. For 77 subfamilies, data were available from both approaches of which 7 were in the true set. For both CAR ranks and PIQ pioneer scores, precision-recall curves were drawn . Additionally, both scores were combined: For each scoring method, results were ranked (rank ties was replaced by the minimum). For each subfamily, its combined rank is the maximal rank across both methods. A low rank can therefore only be achieved when both methods yielded low ranks. We see that the combined strategy outperforms both base strategies.(PNG)Click here for additional data file.S13 FigDisplayed are pair-wise correlation matrices with squared entries across cell lines for motif accessibilities (a); motif accessibilities with the first principal component removed (b) and (c) for expression values. Further, the first 25 eigenvalues of these matrices are shown in (d). The motif accessibility matrix has a very dominant first principal component. After removal of the first principal component, the correlation structure of motif accessibility and expression show a similar structure.(PNG)Click here for additional data file.