The initial GDC dataset include genomic, epigenomic, proteomic, clinical and other data from the NCI TCGA and TARGET programs. Data production for the GDC started in June, 2015 using an OpenStack-based private cloud. By June of 2016, the GDC had analyzed more than 50,000 raw sequencing data inputs, as well as multiple other data types. Using the latest human genome reference build GRCh38, the GDC generated a variety of data types from aligned reads to somatic mutations, gene expression, miRNA expression, DNA methylation status, and copy number variation. In this paper, we describe the pipelines and workflows used to process and harmonize the data in the GDC. The generated data, as well as the original input files from TCGA and TARGET, are available for download and exploratory analysis at the GDC Data Portal and Legacy Archive ( The Genomic Data Commons repository contains genomic, epigenomic, proteomic and clinical data from the TCGA and TARGET datasets. Here, the authors describe the analysis methods for how these divergent datasets were integrated together. Each of these projects contains a variety of processed and unprocessed molecular data types, including genomics, epigenomics, proteomics, imaging, clinical, and others.The National Cancer Institute’s (NCI’s) Genomic Data Commons (GDC)These data, as well as data from future projects, are often generated using different methods, so that joint analysis of multiple datasets are often confounded by batch effects. Based on the lessons learned from TCGA, one of the major goals of the GDC is to create a uniform set of molecular datasets that minimize batch effects due to differences in reference genomes, gene models, analytical algorithms, and processing pipelines. In the GDC, this process is called harmonization and is broken up into two stages: alignment and higher level data generation. In the alignment stage, reads from Next Generation Sequencing (NGS) data are extracted and aligned, or re-aligned, to a single human reference genome sequence using a single pipeline for that particular data type. In the second stage, different higher level data generation pipelines utilize the GDC aligned data to derive summary results, such as somatic mutations or gene expression.https://portal.gdc.cancer.gov/) and API. Users are able to get the up-to-date file summary directly from data repository facet view.Data production started with processing of Whole Exome Sequencing (WXS) alignment and somatic mutation calling, Whole Genome Sequencing (WGS) alignment, mRNA-Seq alignment and gene/exon level quantification, miRNA-Seq alignment and quantification, TCGA Affymetrix Genome-Wide Human SNP Array 6.0 array copy number segmentation, and Illumina Infinium HumanMethylation450/27 methylation array re-annotation. By the end of May 2016, we had successfully processed more than 56,168 BAM files and FASTQ bundles, including 27,594 WXS, 4691 WGS, 11,969 RNA-Seq and 11,914 miRNA-Seq, from 12,487 patients with a total input file size of 1001.7 TB. From GDC harmonized BAMs, more than 100,000 higher level derived data files have been generated. A total of more than 1400 TB of generated data, including intermediate files, have been produced by GDC data processing pipelines, and stored in GDC storage. Many of them have been imported into the GDC database (described in a companion submission), and have become queryable and downloadable from the GDC Data Portal  on Dec 2013 with GenBank assembly accession GCA_000001405.15. The complete assembly downloaded from NCBI contains 456 sequences, including 25 continuous chromosomal and mitochondrial sequences, 42 unlocalized scaffolds, 127 unplaced scaffolds, 261 alternative scaffolds, and 1 EBV decoy sequence.The GDC chose GRCh38 as the reference human genome build for all data analyses, because of its improved coverage and accuracy over the previous major build GRCh37At the time of the development of the GDC pipelines, there was a lack of tool support for alternative scaffolds analysis. As a result, the GDC excluded alternative contigs from the GDC reference sequence, and used GCA_000001405.15_GRCh38_no_alt_analysis_set from the NCBI ftp site as a modified GCA_000001405.15 reference contig set without alternative loci and patches.4.In addition to the continuous chromosomal and mitochondrial sequences, unlocalized and unplaced scaffolds described above, the GDC reference sequence also includes 2385 human decoy sequences, together named hs38d1 with GenBank assembly accession GCA_000786075.2. These human decoy sequences are additional unlocalized sequences not officially recognized by GRC at the time of reference release, and having them in the GDC reference helps to reduce false alignments of such reads in other regionshttp://pave.niaid.nih.gov/) instead of NCBI, because this group updated the sequences as new information is confirmed5. The GDC reference including all human and viral decoys can be downloaded from the GDC at the following link (https://api.gdc.cancer.gov/data/62f23fad-0f24-43fb-8844-990d531947cf).Similar to the idea of improving alignment quality with human decoy sequence, we collected genomic sequences from ten types (200 subtypes) of cancer related viruses in the reference genome, together called “virus decoy” as a collection. The additional benefit of having such virus sequences in the reference genome is to give users the opportunity to directly identify virus genome derived reads. These viruses include Human Cytomegalovirus , Epstein-Barr virus , Hepatitis B virus (HBV), Hepatitis C virus (HCV), Human immunodeficiency virus (HIV), Human herpesvirus 8 , Human T-lymphotropic virus 1 (HTLV-1), Merkel cell polyomavirus (MCV), Simian vacuolating virus 40 (SV40), and Human papillomavirus (HPV) . In each workflow, a main CWL file describes how tools and sub-workflows, also written in CWL, can be used in clearly defined steps. All major tools have been containerized using Docker containers to support reproducibility and portability of the workflows. The GDC will be redistributing the main GDC workflows to the research community to support reproducible research.All major GDC data production pipelines are written in the Common Workflow Language  to process all BAMs from the same patient together for a post-alignment optimization process called “co-cleaning” in which includes GATK IndelRealigner and BaseQualityScoreRecalibration (BQSR). IndelRealigner performs local realignment to further improve mapping quality acrossing all reads at loci close to indels, and BQSR detects and fixes systematic errors made by the sequencer when it estimates the quality score of each base call12. For post-alignment optimization of WGS data, we only employ BQSR, but not IndelRealigner.After initial alignment of WXS data, we follow GATK Best Practices . The GDC provides these point mutations and small INDELs together as Simple Nucleotide Variants (SNV). INDEL calls from VarScan2 were not included in the initial GDC data release, but are included since release 10.GDC uses four somatic variant callers, MuSE16, such as the one used to generate the WGA libraries by REPLI-g. This phenomenon is most obvious in MuTect2 calls as we observed about 10 folds increase of INS/SNP ratio when tumor is a WGA sample (data not shown), compared to other cases in the same project, and most of such INDELs are primarily supported by soft-clipped reads. In order to increase specificity, we re-analyzed all TCGA tumor WGA samples with MuTect2 option –dontUseSoftClippedBases, and successfully removed most of these false-positive INDELs. However, as these artifacts were introduced during library preparation, they could also exist in MuTect2 SNPs, as well as variants called by other tools, in a smaller scale. We recommend users to consume these WGA calls with care.Artificial chimera reads can form during the Multiple Displacement Amplification reaction17 to generate Annotated Somatic VCFs.Raw Somatic Variant Call Format files (VCFs) generated from each pipeline are further processed by caller-specific filters to tag low quality variants in the FILTER column in VCF. For SomaticSniper, variants with Somatic Score (SSC) < 25 are removed. These VCFs are then annotated using Variant Effect Predictor18. This is done with a custom tool based upon VCF2MAF (https://github.com/mskcc/vcf2maf) from Memorial Sloan Kettering Cancer Center. VCF and MAF files may contain germline variants and therefore all VCFs and MAFs described above are only available as controlled-access files. To access controlled-access files in the GDC, users must obtain approval from the Data Access Committee through dbGaP (database of genotypes and phenotypes)19.To allow easier investigation of variants and annotations, the VCFs are transformed into project-level tab-delimited Mutation Annotation Format (MAF) filesWe have also created open-access MAF files by applying stringent criteria to remove potential germline variants. These open-access MAF files are used to support variant visualization and simpler data sharing. We did not produce open-access MAFs for the TARGET program because of privacy concerns of child sample donors. Mutation loads of point mutations and INDELs from both open-access (public) and controlled-access (protected) MAFs of all TCGA projects are displayed in Fig. 23. As shown in Fig. Somatic variant detection is still in a stage of rapid algorithm development, and no single caller is superior to others in every respect23, or using a simulated dataset. In a previous effort to evaluate quality of somatic variant callers, TCGA re-sequenced regions of many called variants using orthogonal sequence technologies, such as Sanger Sequencing, and therefore have produced a set of validated variants that GDC can use to evaluate False-Negatives in current GDC callers. However, no validation experiments were designed for GDC-called variants, thus False-Positives or specificity will not be evaluated in this paper. An independent analysis has been performed by comparing GDC-generated somatic variants to variants previously identified by the TCGA Consortium the same samples, and concluded that both datasets are highly concordant24.Evaluation of somatic variant callers often requires comparison with a so-called gold standard dataset that has been extensively sequenced using multiple independent methodsWe extracted and lifted over all TCGA validation information from 196 MAFs available on May 2016 to GRCh38 coordinates, and selected mutation calls from the same tumor samples that GDC has also successfully analyzed with all four pipelines. This results in 1,911 unique tumor samples and 115,476 validated variants, across 13 TCGA projects . Comparison to GDC called somatic variants of the sample tumor sample shows that only 3.2% of TCGA validated variants are not re-discovered by any of the GDC pipelines Fig. . This nuTo further investigate the impact of cancer type or project to each caller, we analyzed the validated variant recall rate for each pipeline Fig. . In mosthttps://github.com/ucscCancer/icgc_rnaseq_align) created by the International Cancer Genome Consortium. Input RNA-Seq FASTQ files were aligned to GRCh38 using the STAR 2-pass method25, and quantified using HTSeq26 and DEXSeq27. The GDC uses GENCODE v22 as the default gene model for both mRNA-Seq alignment and quantification. DEXSeq exon-level quantification results have not yet been imported into GDC Data Portal at the time of publication.RNA-Seq analysis by the GDC makes use of a modified workflow  and unstranded libraries; however, the read assignment is different between these two different modes resulting in an inability to compare across these library types. Because the GDC emphasizes data comparability across projects, we have run HTSeq on all projects as if they were unstranded libraries.28 and FPKM-UQ 29. The definition of these units is described in the “Materials and Methods” section. Note that the denominators of such normalizations are read counts of all the protein coding genes, instead of all genes. If users are interested in a different set of genes, they are encouraged to perform a normalization based on the genes they are working on, or to use a more sophisticated method, such as DESeq30 or EdgeR31. In addition, GDC does not perform RNA expression batch corrections.For convenience to the scientific community, the GDC also produces gene level quantification in the units of FPKM (Fragments Per Kilobase of transcript per Million mapped reads)We also compared GDC FPKM-UQ expression data to the original TCGA upper-quartile normalized RSEM expression values using Spearman correlation. This comparison was performed over 10,243 shared aliquots and 18,038 shared genes in both dataset. The average correlation between the same sample from two datasets  based on their average expression values in the GDC, and then examined gene level correlations within each of these four groups with TCGA results  normalized miRNA gene expression from GDC GRCh38 pipeline to the original TCGA Hg19 pipeline of the same aliquot using Spearman correlation. Similar to what we have described before for mRNA expression, only 9516 shared aliquots, and 641 shared mature miRNAs are used in this comparison. As shown in the boxplots at the bottom of Fig. Expression level groups are also created in the miRNA datasets for detailed correlation comparisons Fig. . Bottom.33 and uses the DNAcopy version 1.44.034 R-package to perform a Circular Binary Segmentation (CBS) analysis35 with GRCh38 probeset metadata. This pipeline normalizes noisy intensity measurements into chromosomal regions of equal copy number in the form of segment mean values, which are equal to log2(copy-number/2), so that diploid regions will have a segment mean of zero, amplified regions will have positive values, and deletions will have negative values.TCGA used Affymetrix Genome-Wide Human SNP 6.0 (SNP6) array data to identify genomic regions that have Copy Number Variations (CNV) by aggregation of typed loci into larger contiguous regions. Direct liftover of region boundaries from hg19 to GRCh38 results in fragmented segments with poor data quality due to change of probe loci between different genome builds. For this reason, the GDC SNP6 pipeline is built onto the existing TCGA level 2 tangent-normalized copy number data generated by BirdsuiteTo be consistent with the original TCGA data, two different output files were produced for each input: copy number segment files that generated from all probes, and masked copy number segment files, equivalent to the original TCGA “nocnv” file, generated by excluding certain probes that have been previously identified to carry copy number variations in a pool of germline samples.TCGA used Illumina Infinium HumanMethylation27 (HM27) and HumanMethylation450 (HM450) BeadChip to measure the level of methylation at known CpG sites as beta values, calculated from array intensities (Level 2 data) as Beta = M/(M + U), where M is the methylated probe intensity and U is the unmethylated probe intensity. The GDC inherited these beta values from existing hg19-based TCGA Level 3 DNA methylation data, and re-annotated each probeset with new metadata information based on GRCh38 and GENCODE v22.37 to reduce high dimensional information in the molecular data to two dimensions ons Fig. . We util38 and that esophageal adenocarcinomas resembles a subtype of stomach cancer39. Our method supports those observations. In our 2D-clustering the majority of the colon cancer (COAD) samples are co-clustered with rectum cancer (READ) samples, and some esophageal cancer samples cluster together with stomach cancer (STAD) samples. We also identify a few samples designated as primary tumor in one cancer type, but which cluster together with samples from another cancer type. For example, a paraganglioma and phenochromocytoma sample is located within a cluster of adrenocortical carcinoma samples.In our result, different cancer types are well separated and some interesting patterns arise. For example, it has been previously suggested that colon and rectum cancer should be grouped as one colorectal cancerThe rapid decrease in sequencing costs has led to a rapid increase in the resources needed for storage and computation. The GDC provides a solution for this problem by centralizing storage and processing of genomics data. This model currently enables researchers to perform analysis in three different ways: (1) Quick data analysis and exploration using existing GDC visualization tools without the need to download files; (2) Data analysis using GDC generated high level processed data. These files are much smaller than the raw data; (3) Resource-extensive data analysis by downloading raw sequencing data to other data centers or commercial clouds.The genomic data available through the GDC are analyzed uniformly using common algorithms and pipelines, and will be reanalyzed in the future as improved algorithms and methods are developed. As there is no consensus among the scientific community on the best algorithms on somatic variant detection, the GDC implements four callers, and each generates its own set of variant calling output. As shown in Fig. The GDC data harmonization process also makes cross-project analysis much simpler, and reduces artificial findings due to differences in algorithms. Users will be then able to perform joint analysis of data originating from different projects because all data are processed by the same tools and represented in the same format.https://github.com/NCI-GDC/gdc-workflow-overview. As workflow updates are inevitable for an on-going data processing project like the GDC, users are able to find workflow versions using the GDC data portal and API. In addition, major version changes are also described in detail in the GDC workflow documentation. When new workflows are introduced, GDC will reprocess old data if necessary.Major GDC production workflows are available in GitHub The NCI’s Genomic Data Commons has provided a feasible and scalable model for easy sharing of large set of genomics data. We hope our efforts in data sharing and uniform data processing will be valuable not only to researchers, but also to clinicians, patients, and other interested parties, and help accelerate the long-term goal of precision oncology.www.docker.com) to ensure pipeline scalability, portability, and reproducibility. The alignment was managed by an in-house job management system that creates new virtual machines of the designated configuration on demand in an OpenStack environment. The corresponding Docker container that holds the entire workflow then runs inside of the virtual machine for an additional layer of security.The GDC takes advantage of containerization technology and built pipelines using Docker (www.commonwl.org) to describe analysis workflows of multiple Dockerized tools. CWL provides an additional transparent layer between workflow description and workflow execution, that allows even better scalability through parallel execution and portability. The GDC managed CWL pipeline production uses the Slurm Workload Manager (slurm.schedmd.com).During development of high-level data generation pipelines, we transitioned from single Docker workflows to using Common Workflow Language . The main steps include regenerating FASTQ files from BAM input on a per-read group basis using biobambam240 and alignment by read group using BWA (version 0.7.12) in both paired-end and single-end mode. This was followed by BAM sort, merge, and MarkDuplicates using Picard42. The GDC maps reads using either BWA MEM mode if length is equal or larger than 70 bp or BWA Aln mode if below. BWA Aln is also used when mapping older FASTQ reads formatted with Illumina-1.3 and Illumina-1.5 quality scores. Multiple QC metrics were collected both before and after realignment using FastQC43, samtools41 and Picard42. Re-aligned BAM files from the same patient are then collected together for co-cleaning using GATK 3.6 IndelRealigner and BaseQualityScoreRecalibration.The GDC DNA-Seq alignment pipeline follows GATK Best Practices . Some of the workflows had introduced incorrect information into the data, such as sample swapping and mislabeled experimental strategies, which were corrected in the GDC harmonization process.The initial GDC release includes four somatic variant callers: MuTect2, VarScan2, MuSE, and SomaticSniper.11. The GDC uses MuTect2 tools from the GATK nightly-2016-02-25-gf39d340 version. Before tumor normal pairs can be used for somatic variant calling, it is important to generate a Panel of Normals (PoNs) filter that contains calling artifacts and potential germline variants. As mentioned previously, whole genome amplified (WGA) samples are analyzed with dontUseSoftClippedBases turned on.MuTect2 is built upon the capability of local de novo assembly by HaplotypeCaller and somatic genotyping engine of Mutect. Mutect applies a Bayesian classifier to detect somatic mutations13. GDC uses VarScan2 version 2.3.9. The first step of VarScan2 calling is to generate a mpileup file of both tumor and normal BAMs using samtools for a single mpileup file. We set the quality cutoff for samtools to be 1 and also disabled Base Alignment Quality score computation. The mpileup is then used as input to VarScan Somatic to generate a VCF file that contains both SNP and INDEL calls. The resulting VCF is filtered for significant calls using VarScan ProcessSomatic.VarScan2 is another somatic variant caller that identifies both SNV and INDELS. It uses heuristics and statistics to identify variants and considers the confounding impacts of read depth, base quality, variant allele frequency and statistical significance12. The first step, “MuSE call”, estimates the equilibrium frequencies of all four alleles and presents the maximum a posteriori on every genomics locus. The second step, “MuSE sump”, performs a tier based cutoff based on a sample-specific error model which also takes dbSNP information into account. GDC uses MuSE version 1.0rc_submission_c039ffa. Parallelization can be implemented for the first step of MuSE, based on genomic chunks, which can accelerate the production close to linear. The GDC currently only passes calls with quality filter “PASS” to the GDC public MAF files; however, variants with other quality Tier values could also be considered a user’s discretion.MuSE calls somatic variants using Markov Substitution model for Evolution14. GDC uses the default parameter settings of SomaticSniper version 1.0.5.0.SomaticSniper is a somatic variant caller that only identifies SNPs. It uses a bayesian inference to compare genotype likelihoods between tumor and normalsA.https://github.com/ucscCancer/fpfilter-tool) was applied to both VarScan2 and SomaticSniper VCFs.False Positive Filter  as TCGA normal samples that were previously identified to be free of hematopoiesis events (unpublished) at the time of GDC data processing. This PoN is not only used as a MuTect2 built-in filterD.http://archive.broadinstitute.org/cancer/cga/dtoxog) is used to remove oxoG artifacts from point mutation calls. These artifacts were generated due to oxidative DNA damage during sample preparation45.d-ToxoG (E.https://github.com/eilslabs/DKFZBiasFilter) is used to tag variants that are supported with significant bias from one strand direction compared to the other.DKFZ Strandbias Filter  is a tab-delimited text file with aggregated mutation information from VCF Files and are generated on a project-level. The GDC currently produces two types of MAF files: controlled-access MAFs that contain all variants in VCFs, and open-access somatic MAFs that contain filtered variants and reduced germline contaminations and thus considered “high quality”. Any user can explore the open-access somatic MAF for high quality calls; while a more sophisticated user may want to apply for dbGaP access to obtain the superset of mutations in the controlled-access MAF. With the larger set of mutations they may perform custom filtering based on FILTER and GDC_FILTER columns, or collect information that was removed from the open-access version, such as supporting read depth in the normal samples.https://docs.gdc.cancer.gov/Data/File_Formats/MAF_Format/.The specification of the GDC MAF can be found at 25. The first pass alignment recognizes splice junctions in the sample, and the second pass uses those splice junctions to perform the final alignment. STAR version 2.4.0f1 was initially used and we then switched to version 2.4.2a that fixed a bug and allowed us to complete processing all the input files. If both BAM and FASTQ input files existed, only FASTQ files were used.The RNA-Seq alignment pipeline performs alignments of raw reads against the reference genome using a two-pass approach using STAR26 pipeline is used for calculating the number of reads that align to different genes in the genome. As mentioned previously, only reads that can be uniquely assigned to a gene are counted. The GDC ran HTSeq on all samples as unstranded libraries in order to maintain consistency for cross-sample comparisons.The HTSeq (version 0.6.p1)RCg: Number of reads mapped to the geneRCpc: Number of reads mapped to all protein-coding genesRCg75: The 75th percentile read count value for genes in the sampleL: Length of the gene in base pairsThe raw counts are normalized into Fragment per kilobase million mapped reads (FPKM) and Upper Quartile Normalized FPKM (FPKM-UQ) using all protein-coding genes as the denominator.46 pipeline. The first step in this pipeline is to create the flattened General Feature Format (GFF) file, which essentially collapses the information for multiple transcripts spanning the same exon into exon counting bins for that exon. Once the flattened GFF file is obtained, the number of reads that overlap with each exon counting bin are calculated. The result is a flat file which has raw counts for each exon. This data type is not currently available in the GDC data portal and will appear in a later data release.The GDC has also generated exon-level quantification using the DEXSeqThe GDC miRNA harmonization pipeline begins with a realignment of TCGA and TARGET miRNA-Seq reads using a similar strategy of the GDC DNA-Seq alignment pipeline. Because reads of miRNA-Seq are typically short, only BWA Aln was used.32 from BCGSC (British Columbia Genome Sequencing Center). In this pipeline, miRNA species and miRNA isoforms are counted differently, and normalized Reads Per Million values are also derived. The final results from each miRNA-Seq sample is a miRNA species quantification file and a miRNA isoform quantification file, in a human-readable format compatible to the original TCGA data.miRNA quantification is done with a modified version of the miRNA Profiling Pipeline v0.2.7The hg19-based probeset metadata were obtained from the Affymetrix website, and then lifted over to GRCh38. Probes with reference bases not matching between hg19 and GRCh38 were removed.To generate Copy Number Segment file, all SNP and CNV probes are used for CBS calculation, with the only exception that probes in the Pseudo-Autosomal (PAR) regions were removed in males prior to calculation. To generate the Masked Copy Number Segment file from this result, all probesets in chromosome Y and in the frequent copy number variant regions in germlines obtained from GenePattern were also removed prior to calculation.47. Type II probes with a mapping quality of <10, or Type I probes for which the methylated and unmethylated probes map to different locations in the genome, and/or had a mapping quality of <10, had an entry of ‘*’ for the ‘chr’ field, and ‘−1’ for coordinates47. These coordinates were then used to identify the associated transcripts from GENCODE v22, the associated CpG island, and the CpG sites’ distance from each of these features. Multiple transcripts overlapping the target CpG were separated with semicolons. Beta values were inherited from existing TCGA Level 3 DNA methylation data (hg19-based) based on Probe IDs.Using probe sequence information provided in the manufacturer’s manifest, HM27 and HM450 probes were remapped to the GRCh38 reference genome48, and many of these discrepancies can not be easily solved by existing normalization tools. In order to reduce false-positive annotations, GDC requires a strict matching of Chromosome, Position, and Alternative Alleles during implementation of MAF annotations. However, in various variant comparisons in this paper, we applied a loose matching strategy to regard two variants the same if they have overlapping regions between starting and ending positions. This is particularly useful when a non-INDEL caller, such as SomaticSniper or MuSE, represents INDEL sites as point mutations.The same genetic variant can be represented in VCF format in multiple different waysmRNA expression count, miRNA expression count, Copy Number Segmentation, and Methylation Beta Values were collected from the GDC Data Portal. For mRNA expression and miRNA expression, we removed low-expressed genes and miRNAs if 99% or more samples have less than or equal to 1 count. For Copy Number Segments, we computed average segmentation means on each gene weighted on overlapped length between segments and genes. For methylation data, we removed probes that are empty in more than 5% of the samples, and imputed the remaining empty values with the probeset mean. The methylation data is also randomly down-sampled by 25% of the probes to reduce computational burden.49.To integrate these data together for t-SNE clustering, we first computed the top 200 Principal Components (PCs) from each of the four data types using the R function prcomp. The top 200, ranked by “variation explained”, from these 800 combined PCs were selected and further scaled to the arbitrary weights of 3:3:1:1 for mRNA:Methylation:miRNA:CNV, and finally used as input for the t-SNE analysis with the R package Rtsne. We gave less weights to miRNA and copy number because they don’t separate cancer types as well compared to the other two data types. We found that the 3:3:1:1 ratio gives the best visual performance in the scatter plot compared to the other combinations tested. We ran t-SNE 1000 times with random seeds and displayed the result that minimizes the cost functionFurther information on research design is available in the Supplementary InformationReporting Summary


Predicting lithium response prior to treatment could both expedite therapy and avoid exposure to side effects. Since lithium responsiveness may be heritable, its predictability based on genomic data is of interest. We thus evaluate the degree to which lithium response can be predicted with a machine learning (ML) approach using genomic data. Using the largest existing genomic dataset in the lithium response literature , we evaluated the degree to which lithium response could be predicted based on 47,465 genotyped single nucleotide polymorphisms using a supervised ML approach. Under appropriate cross-validation procedures, lithium response could be predicted to above-chance levels in two constituent sites . Variants with shared importance in these models showed over-representation of postsynaptic membrane related genes. Lithium response was not predictable in the pooled dataset , although non-trivial performance was achieved within a restricted dataset including only those patients followed prospectively . Genomic classification of lithium response remains a promising but difficult task. Classification performance could potentially be improved by further harmonization of data collection procedures. One contributing factor to this lag is the number of medication trials that are typically undertaken to find the optimal treatment for a given patient. Since medication trials are based on an empirical approach , it may be possible to reduce this delay by predicting the best treatment for a given individual a priori. However, we currently lack predictive markers for this purpose.Bipolar disorder (BD) is a neuropsychiatric illness characterized by recurrent episodes of mania and depression. It is associated with a significant risk of suicide3. At present, the best evidence of genomic correlates of lithium response is from the largest genome-wide association study (GWAS) of lithium response to date4. This study yielded a single associated locus on chromosome 21. Although the authors demonstrated that the response-related alleles were associated with lower rates of relapse in an independent sample of 73 patients treated for 2 years of lithium monotherapy, the out of sample predictive power of the genomic data overall remains unknown. This is due to the fact that GWAS is (A) not designed to evaluate predictive capacity, and (B) cannot account for epistatic effects.Genomic data may be useful for the prediction of lithium responsiveness, given the familial aggregation of BD (particularly the lithium responsive type) among lithium respondersThus, the primary objective of the present study was to evaluate the degree to which lithium response can be predicted with a machine learning (ML) approach—which explicitly tackles the question of out-of-sample predictive power—using only genetic data. To do this, we employ the largest-ever set of genomic data from BD patients treated with lithium for maintenance therapy over a duration of at least one year, from 14 international site members of the Consortium on Lithium Genetics (ConLiGen). We also had two secondary objectives. First, we sought to evaluate–through pathway analysis–whether any above-chance classification performance was informed by genetic variants in specific biological pathways. Second, we sought to explore factors that might limit classification performance in this large multi-site dataset, such as assessment method  and strength of lithium response.4. Briefly, a sample of 3013 patients were evaluated for long term response to lithium in one of 22 collaborating centres. For these analyses, we limited the feature set to the genotyped (non-imputed) SNPs that overlapped between all platforms. The same quality control measures described in the supplementary materials of Hou et al.4 were employed. We also removed data from sites that had either fewer than ten responders or fewer than 50 subjects in total. Table For this work we use the largest-ever set of genomic data from BD patients treated with lithium for maintenance therapy over a duration of at least one year, from 14 international site members of the Consortium on Lithium Genetics (ConLiGen). A detailed description of the data collection procedure was given by Hou et al.Analyses were performed in four stages: (A) analysis of pooled data , (B) analysis of individual site-level data , (C) an analysis in which we trained classifiers using data from all but one site, which was used as a validation set , and (D) an analysis in which we repeated the aggregate analysis after leaving each site out, one at a time .4, we define a lithium responder as a subject with Alda score As in the study by Hou et al.It is possible that the mode of follow-up (prospective vs. not) could influence the labeling of patients as lithium responders or non-responders. Thus, we split the data into two sets: the first included data from sites where patients were followed prospectively , and the second included only patients assessed retrospectively or cross-sectionally . Within each of these datasets, we repeated the aggregate, leave-one-site-out, and predict-one-site-out analyses under the same cross-validation scheme as our other analyses.To provide an indication of the degree to which classification performance was sensitive to classifier architecture, we implemented two classification models: L2-penalized logistic regression (LR) and extreme gradient boosted trees.5.Logistic regression is a simple method that learns a linear decision boundary between classes. Given the high dimensionality of the feature space and the fact that we expect there to be relatively few features that carry predictive capacity, we add an L2 regularization penalty to the log-likelihood loss function according to the default parameterization in the SciKit-Learn v.0.20.1 package6, similar to a random forest classifier7, pools a number of weak classifiers (decision trees) in order to create a single strong classifier with a reduced variance in comparison to the weak classifiers. On top of this, the XGB algorithm trains subsequent trees including information regarding mistakes that previous trees made  which serves to reduce bias in the strong classifier. We use the python implementation of XGB from the distributed machine learning community  and use the default parameters in all of our experiments.The XGBoost  algorithmAll analyses employed five-fold cross-validation stratified by lithium response. We did this to ensure that model performance is measured out of sample, thus minimizing the possibility of over fitting. By definition, the predict-one-site-out analysis validation phase is conducted out of sample.Performance measures included accuracy, area under the receiver operator characteristic curve (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F-1 score (F1), and Cohen’s kappa. Our primary outcome measure of interest was Cohen’s kappa, which is generally more conservative under class imbalance.p values that represented the probability p that a “trivial” or “null” classifier applied to a data set with the same proportion of positive examples would achieve greater performance. For the experiments in which we perform a stratified k-fold cross-validation, we combine the results from each testing set before applying this technique. Mathematical details for this procedure are provided in the Appendix.For the Cohen’s kappa metric, we simulated p value, with the statistical significance threshold set to For the best classified sites within the site-level analysis, we first compared the feature importance values (or LR coefficients) between classifiers trained on a given site. This was done to evaluate whether above-chance classification performance obtained with different models was attributable to the same features. For models in which multiple sites were classified with above-chance performance, we compared the feature importance/coefficient values between those sites. In both cases, we computed the mean feature importances/coefficients (model specific) over the respective site’s cross-validation folds. The expected values were then compared using Kendall’s Tau (using the Scipy v. 1.3.0 implementation). Where we compared feature importances from the XGBoost algorithm  against LR coefficients, we did so using the absolute value of the LR coefficients. However, when comparing coefficients between LR models , no such transformation was made. We report the tau statistic and 8 on the Dalhousie sample, we first computed the median LR coefficient  for all SNPs whose LR coefficients had the same sign across folds. We then ranked SNPs by the absolute value of the median of their regression coefficients and selected the top quartile (hereafter referred to as the effect set) for gene ontology analyses. Next, we used the biopython package9 to determine the gene(s) with which each SNP in the data set is associated (if any) and separated the effect set of genes from the total (reference) set. We then compare the reference and effect sets using the statistical overrepresentation test in PANTHER8. To limit the penalty incurred by multiple comparisons, in both the effect and reference gene lists we omitted all duplicates and genes that were either uncharacterized or not catalogued by PANTHER.To determine a within site feature importance ranking for the LR classifier for gene ontology analysisWe investigated the overlap in feature importance for LR classifiers trained on the Dalhousie and Würzburg samples separately. In this scenario, we found the intersection of all SNPs whose LR coefficients matched in sign between both sites and took this to be the effect set. We then subjected the genes associated with these SNPs to the same gene set analysis outlined above.p = 0.001).Demographic statistics are reported in Table p = 0.2) nor XGBoost  classifiers obtained classification performance in exceedance of chance.Figure p = 0.0006). The performance of the XGB classifier remained relatively similar for the Würzburg sample .Figure p = 0.0019), accuracy of 0.61 (95% CI [0.57–0.64]), AUC-ROC 0.62 (0.58–0.65), sensitivity 0.23 (0.16–0.31), specificity 0.91 (0.87–0.95), PPV of 0.68 (0.58–0.78), and NPV of 0.59 (0.57–0.62). However, the Cohen’s kappa obtained by the XGB classifier on the Dalhousie University sample was 0.1 (0.02–0.17), with a simulated p value of 0.03, suggesting this was more likely due to chance than the performance obtained with the LR classifier.The LR classifier trained on the Dalhousie University sample (45%/55% responders/non-responders) achieved a Cohen’s kappa of 0.15 . Removal of data from San Diego and Barcelona result in Cohen’s kappa values of 0.06  and 0.05 . Conversely, under the predict-one-site-out analysis, no combination of Table p = 0.28; Dalhousie vs. Würzburg XGB tau = 0.007, p = 0.109).Rank correlations between the feature importance/coefficients are shown in Table p = 2.4e−05, and FDR 8.14e−03). A summary of the postsynaptic membrane genes is shown in Table Tables Table Table In the LR model trained on the prospectively collected data, the most informative variants were associated with genes that showed statistically significant overrepresentation in synaptic membranes  and Würzburg, Germany . In both cases, the classifiers showed better specificity  than sensitivity , suggesting that improvements are needed for reducing the false negative rate. Classification performance remained trivial when data from the Halifax site was excluded , likely due to poor sensitivity . Interestingly, when the Würzburg dataset was left out of the aggregate analysis, the LR classifier’s performance increased to a small but above chance level , which was largely driven by an improvement in PPV , without loss of NPV. The low performance in the aggregate analysis combined with variability in the class imbalance and site-level classification performance indicate that there may be a non-trivial level of between-site heterogeneity in our data, and this may be negatively impacting classification performance. It is also possible that there exists significant between-site heterogeneity in clinical phenotype—as recently demonstrated by Nunes et al.10—which could also negatively impact classification performance. Future research must investigate the relationship between heterogeneity in clinical phenotype and classification performance in multi-site studies.When trained on data pooled across all 14 sites, classification performance with the LR and XGBoost classifiers was effectively trivial, with Cohen’s kappa values of 0.02  and 0.0 , respectively. However, the inability to classify lithium response was relatively consistent across analyses at the site-level, with the exception of data from Dalhousie University in the Canadian Maritimes  which have a significant role in regulation of cellular excitability17. The DISC1 gene, which encodes an integral postsynaptic regulator of plasticity18, was notably associated with schizophrenia and affective disorders in a Scottish family study20. However, lithium response more specifically may have narrower associations with postsynaptic elements involved in G-protein coupled receptor signaling pathways23. This pathway was not statistically overrepresented in gene set overlapping between the Würzburg and Halifax datasets. Since genetic variants’ LR coefficient ranks were uncorrelated between these sites— suggesting a degree of between-site heterogeneity in feature importance—it may consequently be easier to find enrichment of genes in more general biological processes/components. Indeed, when we repeated the aggregate analysis by restricting data to only those collected in a prospective fashion , we found further overrepresentation of genes involved in some biological processes . A plausible hypothesis is that (A) informative variants from models trained on substantially heterogeneous data will associate with relatively broad and non-specific biological pathways, and that (B) the associated biological pathways may increase in specificity for models trained on more homogeneous datasets. Testing this hypothesis will require further work toward identifying potentially more homogeneous subgroups within each of the lithium responder and non-responder classes.Although there was no statistically significant rank correlation between the most important individual features from the LR classifiers trained on the Halifax and Würzburg samples, respectively, there were interesting patterns of gene enrichment among the specific variants whose direction of effect was consistent between these sites. Specifically, we noted the greatest enrichment among genes associated with the postsynaptic membrane . The overrepresented gene set included genes such as ANK3, DISC1, various glutamate receptor genes, scaffolding-related genes such as HOMER1, and adhesion-molecule related genes such as the cadherin (CDH) family. Postsynaptic membrane function, and several of the enriched genes above, have previously been associated with BD. For example, ankyrin 3 (ANK3), which is involved in the aggregation of voltage-gated sodium channels, has been associated with BD in multiple studies24 opined that between-site heterogeneity could reduce classification performance in multi-site ML studies, supporting their argument with results aggregated from ML studies in schizophrenia neuroimaging. They suggested that small studies are more likely to consist of homogeneous and biased samples, but that increasing sample size entails a commensurate increase in heterogeneity. If sampling biases are related to the site from which data are sourced, then increasing sample size by way of pooling data in multi-site studies should consequently result in a more heterogeneous dataset for which classification performance degrades. However, we have previously found contradictory evidence in both BD neuroimaging and in prediction of lithium response, where pooling data across sites improved classification performance25. In both of those studies, however, there was a relatively strict set of harmonization procedures; the former study employed the harmonized data-handling procedures of the ENIGMA consortium , while the latter included only subjects that were followed prospectively for at least one year before assessment of lithium responsiveness using the Alda scale. It is possible that this standardization may have reduced the degree of between-site heterogeneity in those data, thereby allowing phenotypic heterogeneity to be the primary source of variation in their data.Schnack and Khanp = 0.001). Clinical features such as history of psychosis, clinical course, and history of rapid cycling may indeed represent feature dimensions along which we may identify clinical exemplars of lithium responsiveness and non-responsiveness, respectively28. Future studies should therefore (A) maximize inclusion of subjects who are prospectively followed, and (B) investigate alternative methods for identifying exemplars of lithium responsiveness.Data collection practices between sites in our study may have influenced classification performance. A LR classifier applied to the aggregated dataset from only those sites employing prospective follow up achieved superior performance  in comparison to classification on aggregated data from the non-prospective sites (kappa 0.0 ; Table 29 who found a differential pattern of mi-RNA One limitation of the present study is the use of only SNPs that overlapped across all genotyping platforms in the dataset. This may have biased the available feature set, and had lower overall genome coverage. However, there were no large gaps in genome coverage when using only this restricted set of SNPs. Furthermore, use of the fully imputed data would have been infeasible without some degree of feature pre-selection, for which optimal performance in high-dimensional feature spaces can be intractable. Moreover, imputation will introduce additional assumptions and nevertheless also depends in some way on the observable data. For this reason, we opted to not replace directly genotyped features with imputed ones. Given that nearly 50,000 features were directly genotyped in all subjects, of which there were only 2210, we opted to not increase the feature space dimensionality by adding imputed variants to our model. Our analysis was based on SNPs and thus it does not account for changes in gene expression. This could be another possible explanation of differences between responders and non-responders as shown by Hunsberger et al.Another limitation of our study is that hyperparameter tuning was not performed on the models implemented. This was avoided since appropriately guarding against overfitting would necessitate a nesting of cross-validation procedures. Nested cross-validation requires larger sample sizes, and some of the sites in our data had relatively few subjects. As a result, significant inequality between sites would exist in terms of the benefits of hyperparameter optimization. Comparability of performance in the site-level analyses, and the inference of more general conclusions, would therefore have been more challenging if hyperparameter optimization was used. Furthermore, for the logistic regression analysis, the In conclusion, the present study suggests promise for the out-of-sample genomic classification of lithium response, but highlights the ongoing challenges of this task. Future modeling decisions should include not only the goal of achieving above-chance omnibus classification performance, but also a robust sensitivity. Several concrete steps in this direction include increasing sample sizes and further harmonizing data collection procedures.Supplementary Information.


High throughput RNA sequencing is a powerful approach to study gene expression. Due to the complex multiple-steps protocols in data acquisition, extreme deviation of a sample from samples of the same treatment group may occur due to technical variation or true biological differences. The high-dimensionality of the data with few biological replicates make it challenging to accurately detect those samples, and this issue is not well studied in the literature currently. Robust statistics is a family of theories and techniques aim to detect the outliers by first fitting the majority of the data and then flagging data points that deviate from it. Robust statistics have been widely used in multivariate data analysis for outlier detection in chemometrics and engineering. Here we apply robust statistics on RNA-seq data analysis.PcaHubert and PcaGrid, to detect outlier samples in multiple simulated and real biological RNA-seq data sets with positive control outlier samples. PcaGrid achieved 100% sensitivity and 100% specificity in all the tests using positive control outliers with varying degrees of divergence. We applied rPCA methods and classical principal component analysis (cPCA) on an RNA-Seq data set profiling gene expression of the external granule layer in the cerebellum of control and conditional SnoN knockout mice. Both rPCA methods detected the same two outlier samples but cPCA failed to detect any. We performed differentially expressed gene detection before and after outlier removal as well as with and without batch effect modeling. We validated gene expression changes using quantitative reverse transcription PCR and used the result as reference to compare the performance of eight different data analysis strategies. Removing outliers without batch effect modeling performed the best in term of detecting biologically relevant differentially expressed genes.We report the use of two robust principal component analysis (rPCA) methods, PcaGrid function is an accurate and objective method to detect outlier samples. It is well suited for high-dimensional data with small sample sizes like RNA-seq data. Outlier removal can significantly improve the performance of differential gene detection and downstream functional analysis.rPCA implemented in the  In statistics, an outlier is an observation that lies outside the overall pattern of a distribution . HoweverHigh throughput mRNA sequencing, known as RNA-seq , has eme2 > 0.9), but no clear standard exists for biological replicates”. The authors argued that if gene expression differences exist among experimental conditions, it should be expected that biological replicates of the same condition will cluster together in a principal component analysis (PCA) [Research of outlier sample detection in RNA-seq data analysis has been scarce in the literature. Because of the importance of quality control at sample level, outlier sample detection has been studied extensively in microarray data analysis –14. Howeis (PCA) . HoweverPcaCov, PcaGrid, PcaHubert and PcaLocantore (see methods for details). All of them are implemented in the rrcov R package as different functions with a common interface for computation and visualization [Robust Principal Component Analysis (rPCA) is designed to use robust statistics to detect outliers objectively, rather than subjectively as currently carried out using classical PCA (cPCA) . cPCA islization . rPCA stlization , 23. To PcaHubert) outperforms several robust PCA tools [PcaGrid method has the lowest estimated false positive rate [In this study, we explore the feasibility of using rPCA statistics on RNA-seq data sets with small sample sizes for outlier detection. Previous studies have shown that ROBPCA  was used as input. RNA transcripts with varying degree of coverage and fold change between two conditions were simulated to mimic real biological situations. For baseline samples we specified fold change matrix to have 500 differentially expressed genes between two conditions. Three biological replicates each , six or twelve biological replicates each  were simulated using error model of “Illumina 4”, positional bias model of “rnaf” with default error rate of 0.005. Error rate is a simulation parameter which represents the probability that the sequencer records the wrong nucleotide at any given base in the uniform error model. To simulate samples with higher error rates we used the same parameters as those used for the baseline sample simulation except that the error rates were varied from 0.01, 0.05, 0.1 to 0.2. These experiments replicate noises introduced by sequencing errors.Polyester  was usedTo simulate outlier samples, we designed two type of outliers. The first type of outliers had a distinct set of genes that were differentially expressed between two conditions than the baseline sample set and were referred as outlierH (Outliers with high “outlierness”). This scenario represents experiments where samples were coming from a completely different population. For example, individuals with a wrong diagnosis for the disease of interest. The second type of outliers had 50% DEGs overlapped with that of the baseline sample set but with different fold changes and were referred as outlierL (Outliers with low “outlierness”). This scenario represents experiments where outlier samples were coming from individuals with a correct diagnosis but in the severe end of the disease spectrum or respond differently to a drug treatment. Three biological replicates each for each group were simulated using error model of “Illumina 4”, positional bias model of “rnaf” and error rate of 0.01, 0.05, 0.1 and 0.2.SnoN knockout mice (n = 6 each group) [The RNA-Seq data set (Gene Expression Omnibus database accession number GSE120279) were generated by laser capture microdissection followed by RNA-Seq. It was used to profile specifically gene expression of the external granule layer in the cerebellum of control and conditional h group)  and humah group)  were dowvst) or regularized log (rlog) normalization. P-values were adjusted for multiple testing with the Benjamini and Hochberg method for controlling of the false discovery rate (FDR). Genes were called significant when the adjusted P-value was < 0.1. All homologous genes with identical gene names between the human and mouse were used in the human cerebellum data analysis as described [Sequences were adapter-trimmed using Cutadapt , qualityescribed .rrcov R package as different functions with a common interface for computation and visualization [PcaCov; 2) the PCA projection pursuit algorithm is represented by the function PcaProj and PcaGrid [PcaHubert function tries to combine the advantages of the above two approaches; 4) the spherical principal components algorithm [PcaLocantore function.There are four major algorithms and all of them are implemented in the lization . In addi PcaGrid ; 3) the  PcaGrid , 37 impllgorithm  is implePcaGrid and PcaHubert in our study because of their superior performance comparing to other methods [PcaGrid function implemented the PCA projection pursuit algorithm [PcaHubert function combines ideas of projection pursuit and robust covariance estimation. The projection pursuit part is used for the initial dimension reduction and the minimum covariance determinant estimator are then applied to this lower-dimensional data space.We chose to use  methods , 24. Thelgorithm . It is blgorithm . The ROBlgorithm , 37 implThe purpose of a robust PCA is twofold: 1) to find those linear combinations of the original variables that contain most of the information, even if there are outliers, and (2) to flag outliers and to determine their type . Hubert  to find SnoN knockout experiment data matrix. The combined count matrix was used as input to DESeq2 for DEGs detection and rlog or vst transformation. Transformation normalized matrix with n rows (samples) and p columns (genes) was provided as input to the PcaHubert and PcaGrid function implemented in the rrcov R package for outlier detection. Default parameters were used for the analysis unless specifically mentioned otherwise.The count matrix of each sample being tested as an outlier was individually combined with the baseline data matrix or the n = 4 or 5) using RNAeasy Micro Kit (Qiagen). cDNA was synthesized from 250 ng of RNA using Superscript III (Invitrogen) with random hexamer and olig-dT combined primers. PCR reactions were performed using lightcycle480 Roche. The primers used in validation are listed in Supplement Table t-test with a predefined nominal α level of 0.05 was used to determine significant differences between SnoN KO mice and wild-type littermate controls.External granule layer granule neuron precursors were purified using traditional Percoll fractionation method as described . Total RTo test whether rPCA can be used to identify outliers, we first tested rPCA on simulated data. We simulated an RNA-Seq data set for two treatment groups with 3 biological replicates each using Polyester  and usedPcaGrid did not report any outlier for any of the data set tested whereas PcaHubert reported one outlier in 3 out 4 of the cases when rlog was used for normalization  or two (1 case) baseline samples as outliers  or two baseline samples (one case) as outliers .We then simulated RNA-Seq data with low “outlierness” comparing with that of the baseline samples and used each sample as a positive control outlier  or 0.669 (outlierH) lead to multiple false positives, any value between 0.643–0.982 (outlierL) or 0.669–0.978 (outlierH) resulted in one false positive, and any value between 0.983–0.999 (outlierL) or 0.979–0.999 (outlierH) lead to the correct identification of the outlier without any false positives. A value of 1 or above was not able to identify any outlier. The value of crit.pca.distances required to correctly identify the outlier without any false positives was a very narrow range and the default value was too low in both cases.PcaGrid behaved very differently on outlierH and outlierL. A sample with low “outlierness” (outlierL) required higher cutoff values (0.913–0.999) to be identified correctly without calling false positives than when a sample had high “outlierness” . The default value of crit.pca.distances required to correctly identify the outlier without any false positives was appropriate in both situations. PcaGrid had more accurate and stable performance than PcaHubert for a wider range of parameter values for this data set.PcaHubert has an additional parameter alpha, which measures the fraction of outliers the algorithm should resist. Alpha can take any value between 0.5 and 1 and the default is 0.75. For samples with low “outlierness” (outlierL), all allowed values lead to the correct identification of the outlier as well as calling false positive outliers. For samples with high “outlierness” (outlierH), certain values will allow the correct identification of the outlier without calling false positives (0.5–0.714) but the default value (0.75) is not an appropriate cutoff.PcaGrid had wider range of parameter values with stable performance and the default cutoff value allowed the correct identification of outliers without yielding false positives in most tested cases. The default cutoff values for PcaHubert was too low which allowed the correct identification of outliers but also led to false positive calls.In conclusion SnoN knockout mice [SnoN knockout mice and six littermate control mice were subjected to RNA-Seq. This data set was sample of interest and we henceforth refer to this as the mouse cerebellum data set. Samples were collected from two litters and processed in 4 batches with litter and batch matched controls (Table n = 1) with that of the mouse cerebellum data . The combined data matrices were referred as human cerebellum data set and DRG neuron data set, respectively.To mimic true biological situations where outlier samples exist, we used three published data sets and performed analysis where samples of different sources (different species or different tissues) were mixed with the samples of interest. The first data set was generated by laser capture microdissection followed by RNA-Seq to profile specifically gene expression of the external granule layer (EGL) in the cerebellum of control and conditional out mice . The cerls Table . We repoPcaHubert and PcaGrid correctly identified the human cerebellum sample as an outlier using default parameters  as an outlier. PcaGrid classified an additional sample from control animals (WT-1) as an outlier. To test whether rPCA can distinguish samples from the same species but from different tissues, we applied the three methods on the DRG neuron dataset. Similarly, all three methods correctly classified the mouse DRG neuron sample as an outlier using default parameters  or modeled only batch effect (~ batch + condition) or only RIN (~ RIN + condition) in the design formula of DESeq2. Together we compared a total of eight strategies of data analysis:To determine how the presence of outliers affected DEG detection in real RNA-Seq data analysis, we compared DEG detection before and after outlier removal in the Before outlier removal:Strategy 1 (S1): no batch effect modeling.Strategy 2 (S2): modeling of only batch effect in DESeq2.Strategy 3 (S3): modeling of only RIN in DESeq2.Strategy 4 (S4): modeling of both batch effect and RIN in DESeq2.After outlier removal:Strategy 5 (S5): no batch effect modeling.Strategy 6 (S6): modeling of only batch effect in DESeq2.Strategy 7 (S7): modeling of only RIN in DESeq2.Strategy 8 (S8): modeling of both batch effect and RIN in DESeq2.Before outlier removal, four different analysis strategies S1-S4) detected only 2 to 14 DEGs  with the same trends as the qRT-PCR results. Therefore, outlier removal without using batch effect modeling led to detecting significantly more biological relevant DEGs as validated by qRT-PCR on independent biological replicates prepared using a different cell isolation method than all the other strategies.First, we calculated sensitivity, specificity, positive predictive value, negative predictive value, overall agreement, concordance with qRT-PCR results, false positive rate and false negative rate of the DEG prediction by different strategies. The most dramatic change after outlier removal was the significant increase of sensitivity from 3.7–22.2% S1-S4) to 51.9%  analyses of DEGs identified by S5 in conditional SnoN KO samples demonstrated that genes upregulated in SnoN KO EGL were related to neuronal development, dendrite morphogenesis, and neuron projection whereas genes downregulated in conditional SnoN KO mice were associated most notably with the control of cell adhesion, mRNA splicing, and protein phosphorylation [SnoN KO samples demonstrated that DEGs identified by S5 and S6 are enriched for similar GO terms related to dendrite morphogenesis, mRNA splicing etc.  led to completely loss of detecting biologically meaningful DEGs. SnoN, the gene that was knocked out in the experiment, was not identified as differentially expressed by S7 and S8. Adjusting batch effects could introduce spurious group differences [Unexpectedly using batch effect modeling of batch and RIN in DESeq2 had varying impact on DEG detection. Before outlier removal, batch effect modeling of batch and RIN individually or in combination slightly improved the number of DEGs and sensitivity but decreased specificity, PPV and overall PCR concordance. On the contrary, after outlier removal, batch effect modeling strategies all significantly reduced the number of DEGs and sensitivity. Modeling only batch (S6) did improve specificity, PPV and overall agreement comparing with S5 but with significantly decreased sensitivity and missing known genes and functions regulated by ferences  or removferences . Even woferences  dependinPcaGrid function of the rrcov R package is an accurate method to detect outlier samples from RNA-Seq data sets with small sample sizes. Removing outliers can significantly improve the performance of differential gene detection and downstream functional analysis. Robust statistics have been widely used in multivariate data analysis for outlier detection in chemometrics and engineering. To our best knowledge, this study is the first one to apply robust statistics on RNA-Seq data analysis. rPCA should be applicable to other similar high-dimensional genomic data including but not limited to ChIP-seq, DNase-seq, and bisulfite sequencing data.Our results demonstrate that robust principal component analysis implemented in the Additional file 1. Supplemental Table 1: Performance of rPCA for outlier detection on simulated data with 3 biological replicates in each treatment group using vst transformation.Additional file 2. Supplemental Table 2: Performance of rPCA for outlier detection on simulated data with 6 biological replicates in each treatment group using rlog transformation.Additional file 3. Supplemental Table 3: Performance of rPCA for outlier detection on simulated data with 6 biological replicates in each treatment group using vst transformation.Additional file 4. Supplemental Table 4: Performance of rPCA for outlier detection on simulated data with 12 biological replicates in each treatment group using rlog transformation.Additional file 5. Supplemental Table 5: Performance of rPCA for outlier detection on simulated data with 12 biological replicates in each treatment group using vst transformation.Additional file 6. Supplemental Table 6: List of genes validated by qRT-PCR with primer sequences, results and DEG status in each analysis strategy.


Analysis of bulk RNA sequencing (RNA-Seq) data is a valuable tool to understand transcription at the genome scale. Targeted sequencing of RNA has emerged as a practical means of assessing the majority of the transcriptomic space with less reliance on large resources for consumables and bioinformatics. TempO-Seq is a templated, multiplexed RNA-Seq platform that interrogates a panel of sentinel genes representative of genome-wide transcription. Nuances of the technology require proper preprocessing of the data. Various methods have been proposed and compared for normalizing bulk RNA-Seq data, but there has been little to no investigation of how the methods perform on TempO-Seq data. We simulated count data into two groups (treated vs. untreated) at seven-fold change (FC) levels (including no change) using control samples from human HepaRG cells run on TempO-Seq and normalized the data using seven normalization methods. Upper Quartile (UQ) performed the best with regard to maintaining FC levels as detected by a limma contrast between treated vs. untreated groups. For all FC levels, specificity of the UQ normalization was greater than 0.84 and sensitivity greater than 0.90 except for the no change and +1.5 levels. Furthermore, K-means clustering of the simulated genes normalized by UQ agreed the most with the FC assignments [adjusted Rand index (ARI) = 0.67]. Despite having an assumption of the majority of genes being unchanged, the DESeq2 scaling factors normalization method performed reasonably well as did simple normalization procedures counts per million (CPM) and total counts (TCs). These results suggest that for two class comparisons of TempO-Seq data, UQ, CPM, TC, or DESeq2 normalization should provide reasonably reliable results at absolute FC levels ≥2.0. These findings will help guide researchers to normalize TempO-Seq gene expression data for more reliable results. We show that based on sensitivity and specificity performance measures as well as the adjusted Rand index (ARI) as a measure of agreement, Upper Quartile (UQ) performed the best with respect to maintaining absolute FC levels ≥2.0 as detected in a two-group comparison. Counts Per Million (CPM), Total Counts (TCs), and DESeq2 normalization methods also performed reasonably well. The importance of this study is centered on providing the research community an assessment of which method to use for normalization of TempO-Seq data to ensure the reliable results from downstream analyses.HepaRG cells  in cryopreserved form were thawed and seeded at approximately 20,000 cells/well onto collagen(I)-coated 384-well plates . Differentiated HepaRG cultures (2D-DIFF) were re-differentiated from cryopreserved suspension form over 10 days prior to vehicle exposures. Proliferated HepaRG cultures (PROLIF) were seeded at approximately 2,000 cells/well, grown for 3 days, and proliferated during vehicle exposures. For this, HepaRG were plated using William’s E medium  which was supplemented with MHPIT maintenance additive . Vehicle exposures durations were 96 h, and at the final time point, incubation media were removed and cultures were washed once with 50 μl phosphate-buffered saline . Cells were subsequently lysed for high-throughput transcriptomics using 20 μl of 1 × Tempo-Seq lysis buffer (Biospyder) with a 15-min room temperature incubation with subsequent freezing at −80°C. Edge effects were minimized by excluding lysates in rows A, B, O, P, and Columns 1, 2, 23, and 24.via PCR amplification introduces adaptors required for sequencing and well-specific “barcodes” that link sequencing data to a specific well of origin. Barcode sequences flank the target sequence and are inserted into standard illumina adaptors to permit dual-index sequencing and deconvolution of sample-specific reads using standard illumina software. All PCR-amplified and barcoded samples were pooled into a single library for sequencing on a HighSeq 2500 sequencer  using a 50 cycle single-end read flow cell. Processing of the sequencing data was conducted using Illumina’s bcl2fastq software employing default parameter settings allowing for 1 mismatch per read. Sequencing reads were de-multiplexed using standard instrument software for each sample using barcodes to give FASTQ files linked to each well. Downsampled data was generated to obtain 500 mapped reads per gene on average. The 50 bp reads in the fastq files were aligned using bowtie version 1.2.2 (using parameters: -v 3 -k 1 -m 1 –best –strata –trim3 1) to a manifest of the TempO-Seq target genes sequences (a subset of the human transcriptome [Refseq release 70 downloaded July 23rd 2015)] reflecting the 50 bp sequences targeted by the DOs). The utility “idxstats” in the samtools package1 was used to generate read count data matrices.Tempo-Seq analysis was performed as previously described by Biospyder, Inc. . BrieflyYgs_m denote the read count of a gene g ∈{1,…,G} belonging to a group m∈{1,2} of a sample S∈{S1, S2,…,SN} such that:Let gs_m is the parameter measuring the dispersion in the data, μgs_mis the true mean of the data, and NB is the negative binomial distribution. We set the dispersion parameter for each gene to be the same for all samples. Thus, θgs_m = θg_. Here,where θMs_m is the sequencing depth for the mth group in sample S. Ms_m = 1 × 106Us_m for Us_m∼Unif where Unif is the uniform distribution. The bounds (minimum and maximum limits) of Unif and the estimation of the sample mean g were obtained from a DESeq analysis  and the ratiowhere analysis  of TempOanalysis  where thanalysis . Then, twgs_m denoted in m, 35 samples were generated  normalized using the ge limma . The metcalcNormFactors function in the Bioconductor package EdgeR  approach  in the BestimateSizeFactors function in the Bioconductor package DESeq2  was carried out using the ARI  cluster assignment by any of the normalizations . R′ ranges between 0 and 1. R′ = 1 when the two partitions agree 100% and R‘ = 0 when the two partitions are selected by chance.where m = 2 vs. m = 1 denoting simulated treated samples vs. untreated samples. Sets A - F representing -4, -2, -1.5, +1.5, +2, and +4 FC, respectively, contain 50 simulated genes each, while set G with 50 simulated genes also represents no FC. The proportion of genes satisfies the assumption that the majority of genes are differentially expressed in a perturbed biological system with samples assayed on the TempO-Seq targeted platform. Thus, the comparisons of the normalization approaches would be more representative of the FC distribution of the genes in a typical targeted TempO-Seq analysis.Using the RNA extracted from 240 wells containing HepaRG control samples and interrogated on the TempO-Seq platform, we simulated count data for treated (perturbed) and untreated groups (35 samples in each) using a negative binomial distribution with mean and dispersion estimates randomly sampled from the sentinel genes in the control samples. 2 FC approximately equal to 0  values are greater than +0.98  at absolute FC levels greater than or equal to 2.0 . Other n2 ratio values of the treated to the average of the untreated with K = 7. The ARI measures the amount of agreement between the genes in the clusters and their FC set assignment. The ARI ranges from 0 to 1, where 0 defines the agreement is essentially random, and 1 indicates that the agreement is perfect. As shown in To test the agreement of the normalized data with the FC group assignment, we K-means clustered the genes using the logde facto normalization method that are preferred by analysts. With single-channel microarray, RMA became the standard normalization approach. For bulk RNA-seq, many users gravitated to DESeq2 for normalization. For single-cell RNA-Seq, the path forward continues to emerge. In the case of newer platforms such as TempO-Seq targeted RNA sequencing, there has not been an evaluation of the performance of several normalization methods. Recent publications using the TempO-Seq platform used DESeq2 or CPM for normalization (The various gene expression platforms that researchers rely on for whole genome transcriptomics have their own lization . In thislization . We founlization  as well lization , UQ at tlization . This ad2 FC approximately equal to 0 (Our analysis reveals some interesting findings regarding some of the statistics surrounding TempO-Seq normalized data. For instance, the dispersion of the genes used to generate random variates for normalization is typically less than 0.2 . This suual to 0 . If the ual to 0 –4. Previual to 0 .Our investigation does have some limitations that are important to keep in mind. UQ, CPM, TC, and DESeq2 performed reasonably well in normalizing the simulated TempO-Seq data at an absolute FC level ≥2.0, with UQ being the best overall. Below that threshold, the performances suffered. Furthermore, our analysis is for a two-class comparison of groups. More investigation is needed to determine which normalization is superior when comparing more groups or data with a dose or time dependency. Our analysis focused on simulated data from control HepaRG wells and not treated cells. We simulated various FC levels to mimic perturbations, but we assumed that the dispersion for each gene was the same across the two groups. It may be the case that genes in a perturbed system might have dispersion and other statistical properties different from the control cells which may presumably impact the normalization. Note that the TempO-Seq platform that we elevated the normalization methods is a targeted, human version with about 3,000 probes/transcripts as content. Normalizations for data from whole genome TempO-Seq platforms and/or targeted content from other species may perform differently. Finally, the simulated data used DESeq, the previous version of DESeq2, for estimating the dispersion and mean of the genes. This may have provided an unfair advantage to the DESeq2 performance in the comparison. Despite the aforementioned limitations, our results shed some light on the utilization of various standard methods for normalization of TempO-Seq data and that if used in a proper way, several choices will hopefully provide reliable analysis results. Future work will investigate more sophisticated normalization methods for TempO-Seq data and concentrate on data generated from whole genome platforms, other species’ gene content, and factorial or series experimental designs.The original contributions presented in the study are included in the article/SF and SR generated HepaRG cultures and TempO-Seq data. PB designed the simulation study, implemented the computations, performed the analyses, and wrote parts of the manuscript. SF and SA provided interpretation of the results and wrote part of the manuscript. RP provided interpretation of the results. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.