Expression fold changes are similarly affected. Many of the impacted genes are widely studied disease-associated genes. We show that impacted genes exhibit diverse patterns of discordance among pipelines, suggesting that many inter-pipeline differences contribute to overall uncertainty in mRNA abundance estimates. A concerted, community-wide effort will be needed to develop gold-standards for estimating the mRNA abundance of the discordant genes reported here. In the meantime, our list of discordantly evaluated genes provides an important resource for robust marker discovery and target selection.RNA-sequencing data is widely used to identify disease biomarkers and therapeutic targets using numerical methods such as clustering, classification, regression, and differential expression analysis. Such approaches rely on the assumption that mRNA abundance estimates from RNA-seq are reliable estimates of true expression levels. Here, using data from five RNA-seq processing pipelines applied to 6,690 human tumor and normal tissues, we show that nearly 88% of protein-coding genes have similar gene expression profiles across all pipelines. However, for >12% of protein-coding genes, current best-in-class RNA-seq processing pipelines differ in their abundance estimates by more than four-fold when applied to exactly  RNA-sequencing data provide a quantitative view of the transcriptome which can be used to study human diseases. Analyzing RNA-seq data from large scale studies requires efficient and uniform processing of raw sequencing reads from all the samples, while controlling for platform-related biases and noise.1 , and the Genotype-Tissue Expression Project (GTEx)2 . GTEx RNA-seq data processed using the TOPMed pipeline (https://github.com/broadinstitute/gtex-pipeline) was one of the first such efforts. Later, uniformly processed cancer genomic data  from various large-scale cancer studies including TCGA were made available  by the NCI Genomic Data Commons (GDC)3.In recent years, a number of ground-breaking projects have processed RNA-seq data from large scale studies such as The Cancer Genome Atlas (TCGA)4et al., re-processed raw sequencing reads from all TCGA samples to reduce technical inter-sample variability. They provide a readily accessible resource, which provides both normalized and integer gene counts for downstream analysis.Beyond these efforts, Rahman5 project at the European Bioinformatics Institute likewise uniformly processed publicly available and controlled RNA-Seq data from Array Express, NCBI’s Gene Expression Omnibus (GEO), European Nucleotide Archive (ENA) and various large scale studies such as GTEx. Uniformly processed data are available via the project website and through the Bioconductor package Expression Atlas.The Expression Atlas6et al., developed recount2, where >4.4 trillion RNA-seq raw reads from >70,000 human samples deposited in TCGA, GTEx and Short Read Archive (SRA) were processed uniformly using the Rail-RNA pipeline7. Raw sequencing reads from all studies were processed identically to maximize cross-sample comparability. The resulting “ready-to-analyze summaries” contain expression data for genes, exons, and exon-exon splice junctions, and can be downloaded via an easy to search web interface (https://jhubiostatistics.shinyapps.io/recount/) or R/Bioconductor package8.Collado-Torres9et al., developed Toil, a robust workflow which processed >20,000 RNA-seq samples from four large scale studies (including TCGA and GTEx) in under four days using 32,000 commercial cloud-computing preemptable cores. They demonstrated a 30-fold reduction in cost and time compared to existing pipelines which process similar large-scale datasets. Additionally, their workflow can be shared easily using docker containers, increasing reproducibility and reducing the burden on the researcher to install and configure individual tools.Vivianet al.10, used RNA quality information to select and process TCGA and GTEx samples not affected by RNA-degradation issues using two different approaches. One approach minimizes differences between normal tissues from TCGA and matching tissues from GTEx. The second approach (used in this paper) uniformly processes both data sets without the inter-project normalization step.Wang Each of these pipelines, which re-process raw sequencing RNA-seq reads from large scale studies are valuable in their own right and provide widely-used gene expression data for scientific research. However, each of these pipelines use different bioinformatic tools and were also aligned to different assemblies of the human genome using diverse genomic annotations. We compared gene expression values from common samples  processed by the pipelines to understand how gene expression quantification is impacted by differences in data processing. We show that although the majority of protein coding genes show arguably acceptable concordance between pipelines, for >12% of protein coding genes in the genome, different pipelines produce gene expression estimates that are different by more than four-fold.To explore the effects of RNA-seq data processing differences on gene expression estimates, we downloaded pan-tissue uniformly-processed RNA-seq abundance values from five different best-in-class processing pipelines for 4,800 tumor samples from TCGA, and calls from four pipelines for 1,890 normal-tissue samples from GTEx. For completeness, we also included an additional dataset for the same samples with batch effect correction between TCGA and GTEx. To ensure fair comparisons, we limited all our analyses to protein coding genes that appear across all data sets . Figure 13. Differences among the RNA-seq pipelines discussed here include methods/software, software versions, different annotations, different reference assemblies, run-time parameter values, and counting/normalization methods .Several of the above data sources are only available in units of fragments per kilobase of transcript per million mapped reads (FPKM)within either TCGA or GTEx data sets, irrespective of data source . We searched for genes whose expression in a given sample is >32 TPM according to one pipeline (suggesting high expression), but <8 TPM (i.e. >4-fold lower) in the same sample according to at least one other pipeline. To ensure that such large differences between abundance estimates are not due to a small number of outlier samples, we further required that the expression of a gene should be discordant (i.e. meet the above criteria) in at least 1% of all samples . We refer to genes that meet all the preceding criteria as “discordantly quantified” (DQ) genes.). In addition to poor inter-pipeline correlations in mRNA abundance  in TCGA data and 1,214 DQ genes (~7%) in GTEx data Importantly, discordant abundance estimates are not due to outlier estimates from a single pipeline. As shown in Fig. S2c,dnce Fig. , for appnce Fig. .Figure 3relative expression of a given gene between two samples may be expected to be more comparable across pipelines17. However, of the above 2,068 DQ genes, 1,958 genes (~95%) have >2-fold inter-pipeline differences in fold-change estimates for the same sample pairs . Of note, many DQ genes have divergent expression values in large numbers of samples , which found that absolute RNA-seq abundance estimates were generally not trustworthy, and that reliable differential expression analysis was only feasible in less than two-thirds of the genomeCEBPA, and NPM1, and the splicing regulator U2 small nuclear RNA auxiliary factor 1 (U2AF1), which is frequently mutated in Myelodyplastic Syndrome19) in five versions of TCGA data . We note that for NPM1, estimates often differ by a simple scaling factor. For CEBPA on the other hand, estimate differences appear to be sample-specific. In some pipelines, U2AF1 is essentially not expressed in a given sample, but highly expressed in the same sample according to another pipeline, demonstrating that abundance estimates for this gene are simply poorly correlated across two pipelines. Depending on which pipeline is used by a researcher to inform subsequent laboratory experiments, caution should be used when designing experiments for DQ genes such as U2AF1.For most DQ genes, differences between pipelines arise in a variety of ways. As an example, Fig. 20, Supplementary Fig. 21, Supplementary Fig. 22, Supplementary Fig. Gene expression data is the starting point for cohort stratification, pathway enrichment analyses, and many other investigations. The five best-in-class RNA-seq pipelines compared here produce consistent results for nearly 88% of protein-coding genes, including many disease-associated genes, such as genes associated with breast cancer  are known to be frequently affected by multi-mapping reads  protein coding genes and 1073 out of 2068 DQ (51.88%) genes did not have the same exon count per gene across four of our datasets that use different gencode24 annotation versions  are widely-studied and play important roles in diverse human tissues and diseases. As such, the finding that their abundance estimates may vary by greater than four-fold depending on the data processing pipeline will be important for biomedical research and translational medicine.Our list of 2,068 DQ genes includes 784 disease-associated genes  using Bioconductor (https://www.bioconductor.org/) packages. To maximize transparency and reproducibility, we have deposited all scripts, associated data, and a large number of additional plots in a Github repository: https://github.com/sonali-bioc/UncertaintyRNA. R/Bioconductor package TCGAbiolinks27 was used to download TCGA data from GDC, SummarizedExperiment28 was used to store TCGA and GTEx data from each data source, rtracklayer29 was used to import GTF files. All plots were made using ggplot230, RcolorBrewer31, eulerr32, UpSetR33 and pheatmap34.All analyses were performed in R .RNA Seq gene expression data was downloaded from five and four different sources for TCGA and GTEx respectively  more than 5) and the log2 fold change should be more than 2 (i.e. >4-fold difference in expression).For all sample pairs within each data source, expression fold changes were calculated for discordant genes (as defined above) and compared with fold change differences across other data sources.2. In addition to these within pipeline comparisons, we also compared the original versions of the TCGA GDC data and the GTEx (v6) data using the same approach.Four pipelines provide both TCGA and GTEx data . Pathway enrichment analysis using DEGs was done using clusterProfiler37 (v 3.4.4). Enrichment results were visualized using dot plots made with ggplot230.For each cancer type, raw counts from each pipeline for the same samples were used for differential expression analysis using DESeq238 analysis using Java GSEA software , for all three cancer types.log2(TPM+0.001) counts from each pipeline were used to conduct a GSEASupplementray informationSupplementary Table S1.Supplementary Table S2.Supplementary Table S3.Supplementary Table S4.Supplementary Table S5.Supplementary Table S6.Supplementary Table S7.Supplementary Table S8.Supplementary Table S9.Supplementary Table S10.


Deep learning analysis of images and text unfolds new horizons in medicine. However, analysis of transcriptomic data, the cause of biological and pathological changes, is hampered by structural complexity distinctive from images and text. Here we conduct unsupervised training on more than 20,000 human normal and tumor transcriptomic data and show that the resulting Deep-Autoencoder, DeepT2Vec, has successfully extracted informative features and embedded transcriptomes into 30-dimensional Transcriptomic Feature Vectors (TFVs). We demonstrate that the TFVs could recapitulate expression patterns and be used to track tissue origins. Trained on these extracted features only, a supervised classifier, DeepC, can effectively distinguish tumors from normal samples with an accuracy of 90% for Pan-Cancer and reach an average 94% for specific cancers. Training on a connected network, the accuracy is further increased to 96% for Pan-Cancer. Together, our study shows that deep learning with autoencoder is suitable for transcriptomic analysis, and DeepT2Vec could be successfully applied to distinguish cancers, normal tissues, and other potential traits with limited samples. Indeed, the approach has also been successfully employed for a number of biomedical analyses, including mutation detection6 and morphological classifications for fMRI data7, tumor histopathological or photographic images8, and eye diseases9.Deep learning with artificial neural networks has recently surpassed traditional approaches, and the performance of trained specialists in handling massive and complex data in multiple areas. For a range of tasks, particularly in image recognition13. Systematic and quantitative measures of gene expression by microarray and next-generation sequencing, i.e., transcriptomics, could significantly facilitate research, diagnosis, and therapeutics. However, the collective expression status of the genes, the transcriptome, in the genome of a specific type of cell or tissue is large and complex, making it difficult to analyze and conceptualize for research and application. Since the beginning of the millennium, a number of studies have attempted to employ different machine learning methods to characterize gene expression for several cancer types18. These studies illuminated a new path for analyzing transcriptomic data. However, they were limited by small sample size, biomarker selection, or narrow model generalizability. Furthermore, the approaches of linear transformation in these studies are used to extract complex features, which restricts the model from handling linearly inseparable data.Cancers or other pathological and biological processes are largely depended on gene activity or gene expression. Biomarkers using individual gene expression or mutation information have shown predictive power in diagnosis or therapy for several diseases19, or sequential structure20, convolutional neural networks for analyzing imaging and language are not particularly suitable for analyzing transcriptomic data. Extracting meaningful or key features from complex transcriptomic data remains a daunting challenge. It remains uncertain whether deep learning could be used to reduce the dimensionality and complexity of transcriptomic data and extract characteristic features for predicting biological and pathological behavior such as cancer diagnosis.The idea that deep learning could be applied to transcriptomic analysis is tantalizing. However, because transcriptomes do not share similar intrinsic characteristics or structure with images or text, such as local correlationHere we report the construction and training of an unsupervised deep autoencoder to extract features of high dimensional transcriptomic data from a large collection of normal human tissues and of tumor samples of different types. We show that the extracted transcriptomic features have retained adequate information such that they could be decoded back to original gene expression profiling. Furthermore, when coupled those extracted features with the question to separate different cancer or normal tissue samples. Combined with supervised classification, the extracted features can be effectively used to distinguish normal samples of different tissue origins and successfully diagnose tumors in Pan-cancer or tissue-specific cancer analyses.22. The total data set consisted of 20,654 individual microarrays, 17,258 tumors and 3396 normal tissue samples. The original dataset were normalized and preprocessed to remove outliers and duplicates, in the end, there were n = 16,532 tumors and n = 3202 normal tissue samples were then used in subsequent analyses.In order to develop a deep neural network to learn features from human transcriptomic data, we collected gene expression data from various types of human tumors and corresponding normal tissues from the “Gene Expression Across Normal and Tumor Tissue” database , which collects the majority of the published tumor microarray data from NCBI Expression Omnibus23 from each sample. As gene expression is highly correlated, this set of Landmark Genes has been reported to represent approximately 80% of the information within the transcriptome24 and multiple studies have demonstrated their applicability27.To guard against overfitting the data with the available sample size, the training was conducted using a reduced gene set consisting of 978 Landmark Genes (L1000)1. A deep autoencoder processes the data through successive layers of progressively fewer nodes (neurons), by extracting features to compress highly complex information into low dimensional space1. We designed our autoencoder, DeepT2Vec, with five layers, each yielding a 30–50% reduction in neurons  was constructed with two parts: 1) DeepT2Vec (Deep Transcriptome to Vector): that extracts transcriptomic features and (2) DeepC to classify normal and tumor samples  algorithm was used to minimize the reconstruction error during training. The hyperparameters  were adjusted to optimize the training “”. 10% ofTo assess the effectiveness of DeepT2Vec in extracting informative features from human transcriptomic data, we applied DeepT2Vec to the test dataset to examine its ability to encode the gene expression data into TFVs and then to decode them back. We found that DeepT2Vec can successfully extract the transcriptomic features and decode to recapitulate the original data We next examined the TFVs from tumor samples with unsupervised t-SNE. Interestingly, the majority of tumor samples irrespective of their tissues of origins not only segregated from the normal clusters but also coalesced into a common area Fig. C. WithinThese results that tumors of different tissue origins have a shared profile of transcriptomic features are particularly interesting. Although it is known that cancers share common biological properties such as uncontrolled growth, there has been no molecular or simple quantitative measure for “cancerousness”. Our results not only suggest a common underlying molecular mechanism of tumorigenesis but also provide a transcriptome-based quantitative measurement that could be developed for research and therapeutic applications.29. After the generation of DeepC for the Pan-Cancer classification, the test dataset was used for evaluation. Our data showed that, using the 30-d feature vectors only without prior knowledge and manual gene set curation, the classifier correctly identified more than 90% of the tumor and normal samples . Tumor samples used in training were from brain, colon, breast, kidney, lung, skin, liver, prostate, ovary, and blood “”. Again,We also compared other state-of-the-art methods for dimensionality reduction and its predictive power with DeepDCancer Fig. . PrincipTogether these results indicate that DeepDCancer which combines unsupervised training DeepT2Vec with supervised training DeepC can successfully analyze complex transcriptomic data for cancer diagnosis. Furthermore, DeepT2Vec trained with a large dataset has extracted powerful features and successfully achieved dimensionality reduction. It generates highly informative and low-dimensional TFVs that enable the generation of classifiers, DeepCs, with relatively small datasets for distinguishing biological and pathological samples.30. In recent years, specific mutations and biomarkers have been associated with specific grades and stages of cancers31. Therefore, it is possible that cancers of different grades and stages would display distinctive transcriptomic features, and such features could be extracted using deep learning and used for diagnostic prediction. We, therefore, applied DeepT2Vec to grading and staging data and used the resulting TFVs to train classifiers for cancer grading and staging predictions.Cancer grading and staging classifications are crucial diagnostic parameters that direct therapeutic practice19, which involves training a single classifier for each class or each grade or stage of cancers . For classification of cancers at different grades and stages, we employed the one-versus-rest strategyers Fig. ”. 10 folers Fig. .Figure 532. Interestingly, the classifiers, which trained with the tumor samples with grading annotation, generated accurate classification for tumor grade ade Fig. A. While 33. The TNM system is the most widely used cancer staging system30, in which the T refers to the size of the primary tumor and its extend growing into nearby tissues . The N refers to whether there is no or different degrees of lymph node metastasis , while the M refers to whether there are distant metastases or not (M1 and M0). While the prediction performance for T is similar to tumor grading, the identification for N and M were less robust  based on the contributions of the genes identified in the maximum activation analysis. 165/1217 GO annotations were identified as significant (p < 0.05)  enrichment analysis using DAVID , mimicking the mammalian visual system for local feature extraction and combinatorial information processing, have enabled the significant recent advances in image recognition. Similarly, Recurrent Neural Networks (RNNs) and its derivations are appropriately designed to extract features from sequential data and, therefore, have resulted in the successful development of natural language processing (NLP). However, much of the biomedical data do not share structural similarity with imaging or language. Gene expression data are high dimensional, sparse, nonlinear, and complex. Unlike images and language, transcriptomic data do not exhibit local or sequential correlation but are intrinsically organized and highly correlated. Here our results indicate that deep autoencoders, which are specialized to extract intrinsic features from complex data in a nonlinear way, can successfully extract important features from transcriptomic data. The deep autoencoder approach could be applicable to other molecular and physiological data in research, diagnosis, or therapeutics.The approach to extract transcriptomic features by training a deep autoencoder in an unsupervised fashion on a large collection of transcriptomic data is highly successful. The resulting DeepT2Vec could convert complex transcriptomic data from each sample into a low dimensional vector, a 30-dimensional TFV. The success of the approach is evident by the finding that TFVs could be used to decode gene expression faithful to their original patterns. Furthermore, TFVs for samples from the same type of normal tissues are automatically clustered, while those from different tissue types are separated from each other. Interestingly, The TFVs for most of the tumor samples are gathered into a different space, away from the normal tissues, regardless of their tissue origins. The results indicate that tumors share cancer transcriptomic features, suggesting a common underlying molecular mechanism of carcinogenesis despite distinct tissue origins and oncogenic alterations. While it is a shared view that cancers exhibit common behaviors such as uncontrolled growth, this shared cancer transcriptomic profile, as defined by TFVs provides a molecular-based quantifiable measure that could be used in research and in the clinic.CCancer further improved prediction accuracy for Pan-Cancer to 96%.Feature extraction without prior knowledge of the nature of genes provides the opportunity to a myriad of downstream analyses, including but not limited to clustering, data visualization, classification, and gene pathway analyses, especially across multiple datasets and systems. For cancer diagnosis, supervised training of a neural network classifier, DeepC, with TFVs from labeled samples yielded a high accuracy for Pan-Cancer (90%). Sensitivity analysis revealed that DeepT2Vec resulted from unsupervised training with a large dataset is powerful enough for analyzing data with relatively small sample size. This is further confirmed by analyses of tumors from the same tissue origin , which have much fewer samples and yet are more biologically similar. Indeed, DeepDCancer has yielded even better predictions for specific types of tumors. Moreover, DeepDCCancer framework as not only predictive but also comparable with human knowledge of cell biology . We examine this idea by conducting Gene Ontology analysis  database [We filtered the dataset by genes using the LINCS L1000 list of m = 978 landmark genes. All the data  were then preprocessed to remove outliers by the expression level of GAPDH, which was considered as a negative control and assumed to be uniform between samples . Samples with missing information about normal/cancer or tissue of origins and duplicate samples were also removed. After all the quality control filters, we ended up with a normalized dataset with in total 16,532 cancer samples and 3202 normal tissue samples.https://tensorflow.org) for all the computation procedures and the package is publicly available at https://github.com/DesmondYuan/deepD. The expression profiling was centralized to a distribution with mean = 0 and stand deviation = 1 on each gene and then on each cell. For stable training of neural networks, the entire data matrix was clipped by min = − 1 and max = 1 before being fed into autoencoders. 10% of the data were randomly selected as test dataset and not used for further training, and the remaining 90% was defined as experimental data. All the experimental data for training were subject to tenfold cross-validation whereas those data were split into 10 pieces and trained for 10 times where each piece served as validation dataset, respectivelyData preprocessing scripts were implemented using scikit-learn v0.23 and included in the DeepD.data package as a module.We designed DeepD package in Python 3.7 using the Google Tensorflow framework  was trained with carefully finetuned hyperparameters, including activation function: tanh; iterations: 100,000; batch size: 1024; optimization algorithm: Adam optimizer; learning rate: 0.001; L1 regularization: 0.0001; L2 regularization: 0.01). Subject to tenfold cross-validation, 90% of experimental data were used for training, and the rest 10% experimental data were withheld as test set. Among the 90% experimental data, 70%  were used for gradient descent and the other 30%  used to monitor the training progress and early stopping when overfitting is detected the performance stops improving for a continuous period of time (patience: 1000 iterations). For Tumor Grading and Staging analyses, the same hyperparameters were applied. Model performance was evaluated by accuracies of each class ; mean and standard deviation were calculated among all of the cross validation. For multiclass classification, all the cross validation models were combined in an unbiased fashion to make assembly predictions.The other dimensionality reduction methods are implemented using sklearn.decomposition package in Python (scikit-learn 0.23). Those methods are trained on the same 70/30% data partition and then connected to a Perceptron  classifier. For all of those methods, m = 30 components are calculated to match the vector width with Tp2vec.28 and Matlab® R2013a. During training, real-time accuracy for supervised classification was calculated with the evaluation functions in DeepD.utils and was then used to monitor overfitting. The final accuracy was determined by selecting the point with the lowest MSE on the ROC curve. The average and standard deviation were calculated for tenfold cross validation. For multiclass classification , separate one-versus-rest classifiers for each class were trained independently.Features identified through DeepT2Vec were visualized with unsupervised t-Distributed Stochastic Neighbor Embedding (t-SNE) using the Laurens van der Maaten’s t-SNE packageSupplementary Information


Metagenomics and marker gene approaches, coupled with high-throughput sequencing technologies, have revolutionized the field of microbial ecology. Metagenomics is a culture-independent method that allows the identification and characterization of organisms from all kinds of samples. Whole-genome shotgun sequencing analyses the total DNA of a chosen sample to determine the presence of micro-organisms from all domains of life and their genomic content. Importantly, the whole-genome shotgun sequencing approach reveals the genomic diversity present, but can also give insights into the functional potential of the micro-organisms identified. The marker gene approach is based on the sequencing of a specific gene region. It allows one to describe the microbial composition based on the taxonomic groups present in the sample. It is frequently used to analyse the biodiversity of microbial ecosystems. Despite its importance, the analysis of metagenomic sequencing and marker gene data is quite a challenge. Here we review the primary workflows and software used for both approaches and discuss the current challenges in the field. All supporting data, code and protocols have been provided within the article or through supplementary data files.The development of metagenomics and marker gene-based approaches combined with high-throughput sequencing has revolutionized the field of microbial ecology. Two approaches have been extensively used to study microbial diversity: whole-genome shotgun (WGS) and marker gene sequencing. WGS sequencing allows the characterization of whole genomes, genes and genetic features, while marker gene analysis provides an in-depth description of the diversity of specific taxonomic groups. These approaches produce millions of reads even in a single study. Thus, a significant number of methods and software have been developed in parallel to deduce meaningful information from this vast amount of data that is generated. Each microbial community varies considerably in structure and composition, making it complicated to select the optimal methodology for the analysis and interpretation of such data. A challenge in this field is choosing methods, software and databases fitting the data and the questions of the study. In this review, we provide an updated guideline for the analyses of WGS and marker gene sequencing data. We also discuss recent comparisons of the available methods, software and databases to perform those metagenomic analyses.Metagenomics refers to the application of sequencing techniques to analyse the totality of the genomic material present in a sample . CurrentThe main advantage of WGS metagenomics compared to marker gene sequencing is that it offers the possibility to characterize the genetic and the genomic diversity of the analysed community as well as potential and novel functions that are present in the studied community. Further, when using an appropriate sequencing depth, it is possible to assemble full genomes from metagenome data to gain insights into the ‘genomic diversity’ of microbial ecosystems and to obtain draft genomes of uncultured organisms . AlthougFurthermore, when sequencing metagenomes, some specific chromosomal parts may be undercovered, depending on the properties of the genomic regions , the sequencing depth and the chosen sequencing technology . WGS metIn this review, we focus on the analysis of Illumina platform-derived data, since this sequencing technology is most commonly used in metagenomic studies. Illumina standard sequencing produces large numbers of reads (up to 1.5 Tb per run) with high accuracy (error rate ranging from 0.1–1 %), generally with a length of 150–300 bp per read. For example, the Illumina NovaSeq 6000 System can be coupled with WGS metagenomics to produce up to 6Tb per run. Other sequencing technologies, such as Oxford Nanopore MinION/GridION and Pacific Biosciences Sequel, are also used, since they can yield up to 10 Gb per run, as well as very long reads (hundreds of kb). However, the quality of MinION and PacBio sequencing is still lower than that of the Illumina system (PacBio has an error rate of 2.5 %) . MinION Moreover, BGI NGS Platforms offer DNBSEQ technology that can be used for a range of applications. For instance, DNBSEQ-T7 is suitable for WGS metagenomics, generating 1–6 Tb of high-quality data, with read lengths of 100–150 bp. Choosing the appropriate technology among the variety of sequencing systems available will depend on the goal of the project. For example, Illumina sequencing is excellent for marker gene studies, since the analyses are based on short fragments (amplicons), and high quality is necessary to discriminate between reads. However, it is not always necessary to choose, but it may be better to apply more than one sequencing technology in a single study and combine, for example, the high-quality reads of Illumina with the long reads of MinION or Pacific Bioscience.http://hannonlab.cshl.edu/fastx_toolkit/), PRINSEQ . T. T189]. BOX 1.Alpha diversity: the diversity measured within a particular ecosystem or sample. It is commonly characterized by OTU/ASV richness, evenness and phylogenetic diversity. Beta diversity: diversity comparison between particular ecosystems or samples. It is commonly analysed through ecological and phylogenetic distances estimated from the sample composition. Rarefaction curves: plots representing the number of samples on the x-axis and the number of ‘species’ or diversity variants on the y-axis. The curves show how the richness increases with the increase of sequencing depth, and they are commonly used to select a threshold value (plateau) to perform diversity analyses. Metadata: all the variables and data relevant for the study providing information about the samples included. For example, in a theoretical study of a water-associated microbial community, typical metadata could be measures of temperature, pH, salinity, or oxygen levels from the samples analysed.Since significant variations are associated with distinct methods, it is better not to compare results obtained from different pipelines. There is not ‘a universal method’, but choosing the best method depends on the type of data and the question to answer. For example, to compare samples enriched in phylogenetically related species requiring a high resolution (very similar sequences), ASV-based methods are more appropriate than OTU-based methods, since they detect single-nucleotide differences. However, ASV estimation methods have some limitations. For example, a single genome can contain multiple ASVs that can differ in more than one nucleotide, which may lead to erroneous taxonomic annotations. Thus, when genomic heterogeneity is essential in the study, it is better to choose a more conservative approach, such as the OTU-based methods . FurtherAssigning taxonomy to OTUs/ASVs is a critical step in microbial community analyses since it answers the question, ‘who is there?’. Different classification methods and databases have been developed for the taxonomic assignment of the most common marker genes, the 16S rRNA and 18S RNA genes, and the ITS region.blast [blast are based on alignment methods [blast. As part of the QIIME 2 project, the q2-feature-classifier, a taxonomic classification method based on novel machine learning and alignment-based methods, was developed [blast+ [blast. The new naive Bayes, VSEARCH and blast+ classifiers (included in the q2-feature-classifier plug-in) perform equally well or better than the previous generation of methods. Importantly, the optimization of parameter settings is as critical as choosing the correct method, because the accuracy of the classification and the ability to detect novel taxa may vary considerably depending on these choices [Typical classification software is based on different algorithms, such as RDP classifier , UCLUST blast , coupledblast . On the  methods . UCLUST,eveloped . The q2- [blast+  and VSEA [blast+ , and a m choices . Furthersilva [silva also includes small and large subunits of the rRNA gene (16S/18S and 23S/28S). These curated databases are often integrated into the most common pipelines for marker gene analyses, such as QIIME 2 [silva, for the 18S rRNA gene, eukaryotic databases have been designed, such as EukRef [2) [2) [2, an 18S rRNA curated database focused on protists, was constructed. It includes other eukaryotes, such as metazoa, land plants, macrosporic fungi and eukaryotic organelles  [2 is a curated database of 18S rRNA from planktonic protists [The primary databases used for 16S rRNA gene analyses are Greengenes , the Ribsilva . silva a QIIME 2 , mothur  QIIME 2 , or RDP  QIIME 2 . Micro-es EukRef , or a mokRef [2)  and the  [2) [2) . The Euk [2) [2) . Protistplastid) . In contprotists .The most up-to-date databases to analyse fungal diversity based on the ITS marker region are UNITE  and the A primary difference between the the whole genome sequencing and marker gene approaches is that the latter does not give information about the functional capabilities associated with the taxa, although some approaches have been developed to link marker genes with their corresponding functional profiles . One of After the prediction of OTUs/ASVs, the starting point for ecological and statistical analyses is the establishment of an abundance table that reports the number of reads for each OTU/ASV per sample in columns and the OTU/ASV definitions in rows. This abundance table can also be based on the taxonomic information at specific levels  depending on the purpose of the study. The data inform us about the differential sequencing effort between samples and the diversity of microbial communities. Abundance tables in the field of microbial communities are characterized by (i) different numbers of reads between samples and (ii) a high number of single variants  . Since aThe second group of methods is based on scaling the data. The estimation of the relative abundance table is already a normalization process known as total sum scaling (TSS). However, to deal with the compositional nature of the data, a promising group of scaling methods based on log-ratio transformations has been developed. A simple scaling method, the log upper quartile (logUQ), basically scales each sample by the 75th percentile of the distribution of counts before log transformation . PaulsonP-values and control the false discovery rate (FDR) [Recent studies have analysed how normalization methods influence the output of standard analyses in microbial ecology studies (mainly from 16S rRNA data). A study comparing rarefaction to various scaling methods  showed that either one or the other type could perform well in combination with specific beta diversity estimations . For exate (FDR) . It showIn conclusion, choosing normalization methods is not trivial. So far, there is no consensus about which approach shows the best performance. However, the best result is obtained from a trade-off between the data features  and the normalization method, combined with the coupled ecological and statistical analyses.In marker gene studies, one of the well-known biases is the variation in gene copy numbers between species . For exaIn addition to the possibility of analysing marker genes in a step-by-step way, there are pipelines that facilitate complete analysis, from quality filtering to statistical comparisons. The most commonly used pipelines are QIIME (now updated to QIIME 2) , mothur In microbial ecology, diversity is typically described within  or between (beta diversity) samples . Most ofOrdination techniques such as principal coordinate analysis (PCoA), canonical correspondence analysis (CCA), principal component analysis (PCA), or non-metric multidimensional scaling (NMDS) are practical exploratory approaches . These tt-test, analysis of variance (ANOVA), or multivariate analysis of variance (MANOVA). A comparative study examined the effects of heterogeneity of multivariate dispersions of the data on PERMANOVA, ANOSIM and Mantel test [Statistical analyses of alpha and beta diversity data should be based on the biological question asked and the results of the exploratory analyses. Some of the most used statistical analyses for microbial ecology studies are shown in tel test . The auttel test . To identel test . The lintel test .The analysis of the composition of microbiomes (ANCOM) is a more recent test adjusted to the structure of microbial community structures, and it accounts for the compositional data . Other mMany studies on microbial communities incorporate network analyses to infer microbial ecological interactions and/or external variables . These iTo test the significance of associations between environmental variables from the metadata  and microbial composition, multivariate ANOVA based on dissimilarities (Adonis) is widely used . RecentlNew tools are continuously emerging and evolving to adapt to sequencing technologies and metagenomic approaches. The most difficult issue for the user is definitively not to get lost among all possible choices. Indeed, the best approach is to compare the results obtained using several of the available tools. However, this is very time consuming and also requires one to acquire knowledge on how to use each tool. Finally, correct interpretation of the results obtained from the application of different analyses tools is necessary to form conclusions. Hence, when one is not making such comparisons using one’s own data, the comparative studies and reviews published regularly to guide the user are constructive and should be read and analysed carefully to guide one’s final decision on which tool to use.


Vast amounts of transcriptomic data reside in public repositories, but effective reuse remains challenging. Issues include unstructured dataset metadata, inconsistent data processing and quality control, and inconsistent probe–gene mappings across microarray technologies. Thus, extensive curation and data reprocessing are necessary prior to any reuse. The Gemma bioinformatics system was created to help address these issues. Gemma consists of a database of curated transcriptomic datasets, analytical software, a web interface and web services. Here we present an update on Gemma’s holdings, data processing and analysis pipelines, our curation guidelines, and software features. As of June 2020, Gemma contains 10 811 manually curated datasets , over 395 000 samples and hundreds of curated transcriptomic platforms (both microarray and RNA sequencing). Dataset topics were represented with 10 215 distinct terms from 12 ontologies, for a total of 54 316 topic annotations (mean topics/dataset = 5.2). While Gemma has broad coverage of conditions and tissues, it captures a large majority of available brain-related datasets, accounting for 34% of its holdings. Users can access the curated data and differential expression analyses through the Gemma website, RESTful service and an R package.Database URL: https://gemma.msl.ubc.ca/home.html As of June 2020, NCBI Gene Expression Omnibus (GEO) contains 97 379 transcriptomic studies, 71 233 (73%) of which are generated in human, mouse and rat ; generatWhile the data in GEO are plentiful, in our experience, substantial effort in curating and reprocessing is necessary before high-quality secondary analyses can be performed. This effort is necessary because GEO was originally intended mainly to be an archive of published data . AccordiIn some datasets, the available metadata consist of a very brief description of the study and cryptically named samples; in others, there is insufficient sample information to easily reconstruct the underlying study design, despite the MIAME standard (“Minimum Information About a Microarray Experiment”) that GEO follows . This prTo help address these issues, we created Gemma , a bioinThe Gemma data model was influenced by (but does not fully adopt) an early microarray data model, MAGE-OM , and somProcessing of large amounts of gene expression data into Gemma involves numerous steps. Major grouping of these steps include dataset selection, platform processing, expression data processing, metadata curation and downstream analyses. Many of these steps are automated, with some human intervention and manual curation at key stages, described in the subsequent sections. Before delving further, we first lay out the decisions that influenced the framework’s design.The main goal of Gemma’s data processing and curation pipelines is to provide a rich yet usable representation of a dataset; the representation can be described as having three parts. First, the expression data are needed, such that the data can be usefully represented as a single matrix where rows represent ‘platform elements’  and columns represent RNA samples, with entries being expression levels. Ideally, these data are processed using a uniform set of procedures. Second, the samples need to be described: what tissues, conditions etc. they were derived from and what role they play in the study (e.g. control or treated samples). Third, for microarrays, the probes need to be mapped to genes , and this also should be done in a consistent manner.The information provided by GEO is only a starting point to meeting these needs. Expression levels can be obtained from GEO for many datasets, but they are provided by submitters and thus are not processed in a consistent manner. Samples are described in GEO using free text with limited structure provided by key–value pairings, but the descriptions are written by data submitters and are generally not adequate for our purposes, especially in the lack of harmonization that the use of formal ontologies affords. Mappings of microarray probes to genes are generally provided by data submitters (i.e. microarray manufacturers) and are often based on provided GenBank IDs for mRNAs. GenBank IDs rarely correspond to the actual sequences on the microarray, which are generally short oligonucleotides or (for some older platforms) PCR products.https://github.com/PavlidisLab/Gemma/).To address these issues, Gemma implements a comprehensive data processing pipeline . Gemma nHomo sapiens, Mus musculus, Rattus norvegicus, Danio rerio, Drosophila melanogaster, Caenorhabditis elegans and Saccharomyces cerevisiae). More recently, we have specifically increased our coverage of studies relating to the nervous system and on datasets studying human, mouse or rat. We de-prioritize datasets that have small sample sizes  as these are less suitable for the downstream analyses implemented in Gemma. We also prefer studies that have biological replication and a clear experimental design affording specific comparisons between contrasting conditions. That said, we do accommodate user requests for loading datasets that do not meet these criteria. A notable current omission is single-cell RNA-sequencing (scRNA-seq) datasets, because most of the available datasets to date have minimal (if any) biological replication and are often surveys intended for cell type identification. Additionally, there is little consistency in how single-cell datasets are represented in GEO, with some datasets having GSM identifiers for each individual cell while others for each processed pool of cells. Anticipating that future scRNA-seq studies will eventually shift to designs with replicates, incorporating single-cell data into Gemma is a key area of future work.For Gemma, our goal is not to capture all available datasets, in part because of resource limitations. Initially, we accumulated data on any domain of biology for seven taxa of interest , but since these are less commonly used and less suitable for quantification of expression levels, implementing support for these technologies have not been a priority. Gemma can also handle microRNA (miRNA) microarray platforms. Finally, we no longer routinely add data from two-color microarrays, partly because they are no longer in much active use. They also tend to have poor gene coverage and/or do not provide sufficient metadata to permit probe sequence analysis (see ‘Processing platform information’), as well as have other complications such as dye-swap or ‘loop’ designs that hinder automated analysis.Gemma has a robust security model that permits control of ownership and visibility of underlying entities. All datasets when first loaded into Gemma are marked ‘private’ and can only be viewed by curators and administrators. This allows us to curate and preprocess the data before making it available publicly. Gemma also has an audit trail system, in which updates and key events are recorded during processing. Curators can attach notes to datasets, mark datasets as needing curation or flag datasets as unusable (we support most of these features for platforms as well). Platforms or datasets that are confirmed as unusable are blacklisted, to avoid inadvertently reloading them in the future. Collectively, these tools enable curators and administrators to easily assess the curation state of a dataset, track down the source of errors and coordinate curation efforts.A key step in Gemma data processing is linking expression data to genes. This is especially important for microarray platforms, where the probe sequences on the arrays were often designed prior to the availability of high-quality reference genome sequences and annotations. We decouple the expression data from the underlying microarray platform, enabling us to update the relationships between probes and genes with minimal disruption.We apply different probe-to-gene mapping strategies depending on the type of platform ; this is done internally by adopting a previously described protocol  with somFor platforms with probe sequences, the processing proceeds as follows. For Affymetrix platforms specifically, we perform an additional step where we collapse the Affymetrix probe sets (consisting of multiple 25-bp oligonucleotides each) to a single representative sequence, resolving overlaps, as described previously . Next, fhttps://tinyurl.com/Gemma-HumanNCBI). The output of our RNA-seq data processing pipeline (see ‘Processing of expression data’ below) can then be linked to these ‘generic’ platforms based on NCBI gene IDs.For RNA-seq data, a different strategy is employed while still accommodating them in the same general framework. For each taxon, we define a ‘pseudo-platform’, where the entire platform’s elements are the set of known genes recorded in the reference genome annotations (e.g. ad hoc using a simple tab-delimited file). This is generally based on a GEO Series (GSE ID), and the system automatically retrieves any associated ‘GEO DataSets’ (GDS IDs) if available. Unlike GEO Series, DataSets are curated by GEO and provide additional metadata including basic information on the experimental design; they are, however, only available for <5% of GEO Series. Where possible, free-text sample annotations from GEO are mapped to an appropriate ontology term based on a pre-defined dictionary, which currently contains 674 text-to-term mappings. Parsing of these files and mapping them to the Gemma data model results in an initial representation of the dataset in Gemma.First, basic dataset metadata  are parsed and loaded from GEO SOFT files , we attempt to extract the data for both channels separately (including background levels) for assessing spot quality; spots which have low signal in both channels are flagged as missing data.https://tinyurl.com/Affy-APT; specifically ‘apt-probeset-summarize’ with the RMA algorithm  using the Affymetrix Analysis Power Tools provided by the manufacturer [lgorithm ]. These https://github.com/PavlidisLab/rnaseq-pipeline) is used to download FASTQ files from the SRA or EMBL-EBI European Nucleotide Archive  and then quantile normalized.After the data are loaded into Gemma (through either of the above streams), the resulting expression data are processed through a common post-processing pipeline, including steps described below. The expression data are logWhere possible, the data are batch corrected, which is performed only after the curation of the experimental design. Importantly, batch information is almost never explicitly provided as sample annotations by the submitters in GEO. In Gemma, batches are defined automatically using information in the raw data files. For microarrays, date stamps can usually be found in CEL files for Affymetrix datasets and in GenePix output files for Agilent (and other spotted platforms) datasets. Occasionally, batch information is instead obtained manually from supplementary information provided by the data submitter or the associated publication. Through the date stamps, batches are defined using simple heuristics : samples that are processed closely in time are considered a separate batch if a larger time gap occurred before processing of the next sample. In practice this almost always yields a batch factor that is correlated with one of the first three principal components of the expression level data. For RNA-seq datasets, no relevant dates are associated with the records in GEO or in SRA. Instead, we attempt to use information from the FASTQ headers, which for Illumina platforms often contain information on the ‘device’, ‘run’, ‘flow cell’ and/or ‘lane’. Any batch information obtained is stored as a factor in the experimental design. Despite these efforts, for many datasets we are unable to obtain batch information.Batch correction is conducted using an in-house implementation of the ComBat algorithm , enhanceA number of diagnostics are computed and stored for each dataset, such as principal component analysis (PCA) of the expression levels, the relationship between the principal components and curated experimental factors, the mean–variance relationship of expression levels [which is especially important for RNA-seq data ] and samA common data quality problem in expression datasets is the presence of one or more samples that deviate markedly  from the properties of other samples. Gemma flags potential outlier samples automatically, which are then manually reviewed. Samples confirmed to be outliers are not permanently removed from the system: we represent them as missing data instead. This ensures transparency of the process and also affords us the possibility to reverse the decision easily. Our outlier detection algorithm uses the sample–sample expression level correlation matrix, the idea being that samples which have low correlations with all other samples are potentially outliers. Omitting some details, the algorithm is as follows: the correlation matrix is first adjusted by regressing out the effect of major experimental factors such as tissue type, so that identifiable biological groups of samples are not treated as outliers. Samples are considered potential outliers if their adjusted median correlation is outside the interquartile range of correlations of the sample with the closest median correlation to them. All outliers called by the algorithm are reviewed and approved by a curator. With outlier removal and batch correction, the data are now finalized for downstream analyses.Datasets in GEO are very diverse in how they are represented. Gemma is designed to automatically handle many complexities that arise; in some cases, a curator needs to review the dataset and take an appropriate action.When a dataset’s SOFT file is downloaded from GEO and parsed in preparation for loading, Gemma checks to see if any of the samples are already included in another dataset in Gemma, based on the ‘GEO Sample’ ID (GSM IDs). Any duplicated samples are removed and noted in the dataset description . This is necessary to avoid double counting when performing meta-analysis in Gemma. Occasionally, this results in a suboptimal representation of either dataset or both datasets, e.g. it can result in the removal of all control samples from a dataset if the controls are shared between two datasets. To resolve this issue, we determine which of the overlapping datasets results in the best use of the samples and then reload the ‘better’ dataset to have a usable design.https://tinyurl.com/Gemma-GSE23579-1) for human and GSE23579.2 (https://tinyurl.com/Gemma-GSE23579-2) for mouse]. Samples from taxa that are not supported by Gemma are rejected.Datasets that include samples from multiple species are detected and split apart so that a separate Gemma ExpressionExperiment is created for the samples from each taxon . This design decision favors simplicity of implementation and speed of curation at some cost of power for inference. Next, we will discuss the standards used for the ‘Category’ and ‘Value’ pairs separately.IGF1, assigning the category ‘genotype’ indicates that there is a genotypic modification of the gene IGF1 in the sample’s cells, whereas assigning the category ‘treatment’ indicates that external IGF1 proteins were introduced into the sample’s culture media. In terms of grouping entities, when ‘categories’ are used as factors, it enables us to group samples by the respective factors’ levels and perform differential expression analysis. Additionally, categories can be used as filters when searching for annotations in Gemma. Thus, it is essential that the use of categories is standardized. In As shown in the previous example, we include ‘Categories’ when annotating Gemma entities. This is done for two key reasons: by assigning a category to the terms’ ‘Value’, we provide additional context to those terms and we enable easier grouping of entities. To illustrate the point: if a sample is annotated with the gene symbol Gemma currently supports 12 different ontologies . There whttps://github.com/PavlidisLab/TGEMO) to capture these terms, with the intent of replacing them with the appropriate ontology terms in the future. Currently, TGEMO contains 101 ontology terms and implements concepts for genetic modifications (e.g. knockdowns), some mouse/rat strains (e.g. DB/2) and some terms for our internal usage.During the curation process, we found that a number of concepts are not represented in the ontologies we use. We created The Gemma Ontology , sampling time intervals (e.g. 3 h) and dosages . We established style standards for free-text annotations to ensure consistent and unambiguous annotation https://tinyurl.com/Gemma-GSE1463, , the input box automatically searches and provides a list of recommended matching terms, along with an indication as to whether (and how often) the term has been used previously in the system. Frequently used terms are favored. When no suitable term is identified, the curator can choose to use free text to annotate the concept by following formatting guidelines of common cases. This also promotes the reuse of previously used free-text strings, which are also presented in the concept recommendation results .https://tinyurl.com/Gemma-GSE2198;  and also in dataset searches. Experimental designs are modeled as follows. A dataset can have one or more ExperimentalFactors that describe a feature (or factor) that varies across samples, while the different levels of the feature are called FactorValues. For example, the dataset GSE2198  has thrMost datasets, when first imported into Gemma, do not have any information on the experimental design. Exceptions to this are datasets  that have an associated GEO DataSet ‘GDS’, which have been manually curated by NCBI GEO staff. For such datasets, we automatically initialize an experimental design layout based on that in the GDS. In either case, we manually curate the experimental design to add ontology terms and to enforce uniformity in how factors and levels are described.Considering the diversity in experimental design of the datasets submitted to GEO, we elect to prioritize key experimental factors during curation. The rationale is that while we could invest more time to exhaustively curate every single potential factor within a dataset, it would be a large effort with diminishing returns. Instead, we prioritize factors that are highly recurrent across many studies, which would give us the greatest value. These factors include ‘genotype’ for genetic manipulations; ‘treatment’ for both chemical and physical treatments; ‘disease’ for diseases and disorders; and ‘disease staging’ for the varying stages or severity of disease. For time-course datasets, factors such as ‘timepoint’ for sampling time points; ‘developmental stage’ and ‘age’ for patient ages are also prioritized. Other popular factors that are taken into account include ‘organism part’ for tissues and organs; ‘cell types’; ‘cell lines’; ‘strain’ for mouse or rat strains and ‘biological sex’.https://tinyurl.com/Gemma-GSE40463;  rely on the presence of a transgene vector, we consider the samples with the empty transgene vector to be the baseline condition.Another aspect of design curation is the identification of the ‘baseline condition’ of the study, which is important for differential expression analysis. Assignment of the baseline condition is not performed for all factors, as there might not be any to begin with . For factors that do have a baseline condition , the condition is annotated with specific ontology terms such as ‘reference substance role’ etc. . For more complex datasets that involve positive and negative controls, we use the ‘control’ term to annotate what we perceive as the ‘true’ baseline condition of the dataset. For example, in dataset GSE40463 , for onhttps://tinyurl.com/Gemma-GSE36051), a study of breast tumor cells (https://tinyurl.com/Gemma-GSE14499), a dataset concerning a mouse model for Alzheimer’s disease , and ontology Uniform Resource Identifiers can be used directly as search terms. In either case, retrieved ontology terms are expanded to include all child terms via ‘part-of’ and ‘is-a’ relationships. This means that searching for the term ‘cerebral cortex’ will not only retrieve datasets that contain that term either in free-text or in ontology term but also datasets associated with the term ‘hippocampus’ (in both free-text or ontology term), since ‘hippocampus’ (UBERON_0001954) is a child term of ‘cerebral cortex’ (UBERON_0000956).An important use case for Gemma is searching for datasets based on keywords. The system we have developed performs search based on both full-text indexing as well as ontology inference. In text searches, we search all dataset-associated text fields ; while ontology searches are performed in two different ways—free-text searches are used to retrieve ontology terms based on an index of the ontology-associated text , the experimental design and other relevant metadata populated and a final manual review has been performed.Next, differential expression analysis is performed on the dataset based on the annotated experimental design. In cases where certain terms are used , Gemma automatically assigns these conditions as the baseline control group; in absence of a clear control condition, a baseline is arbitrarily selected. To perform the analysis, a generalized linear model is fit to the data for each platform element (probe/gene). For RNA-seq data, we use weighted regression, using an in-house implementation of the voom algorithm  to compuP-value and associated false discovery rate , and at the contrast level, the test statistics (t-statistic), the log2 fold change and its associated P-value. The results are available for download as tab-delimited files, and the top-most differentially expressed platform elements can be visualized on Gemma.The output of the differential expression analysis is then stored in the database. For each platform elements, this includes at the factor-level, the main effect’s The other major type of analysis supported by Gemma is co-expression. This analysis has been performed on selected datasets that have sufficient number of samples . Additionally, aggregation of data across datasets, i.e. meta-analysis, for both co-expression and differential expression can also be performed in Gemma. As these components of Gemma are currently undergoing redesign, we omit a detailed description of the associated methods.https://github.com/PavlidisLab/GemmaPaper-2020).Figure generation and numerical reporting was performed with code using Jython, Python and R. Scripts and relevant data files are provided on Github . For most of this paper, we will focus our analyses and descriptions to the three primary taxa: human, mouse and rat; each having 4593, 4933 and 894 datasets, respectively . These scores are publicly viewable (see ‘User interface’), thus researchers could use these scores in determining which datasets of interest they could focus on (or omit) in their downstream analyses. We stress that these scores are only a rough guide and a dataset’s quality score can sometimes be improved by curation.median ≥ 0.9). As high sample agreement is indicative of data quality, such datasets are given a higher GEEQ score. For the next property, 8% (825/10 420) of datasets had at least one outlier sample flagged and manually confirmed during QC. The number of outliers removed ranges from 1 to 21 samples in those datasets; when converted to a ratio of outlier samples over the total number of samples in each dataset, the outlier removal ratio ranges from 0.002 to 0.36, with a mean ratio of 0.04. In relation to the GEEQ score, only the presence of predicted outliers negatively affects it. Once they have been reviewed and accepted by a curator, they are marked as missing data (see ‘Methods’) and their contribution to the GEEQ score is also removed. Finally, 62% (6460/10 420) of all datasets contain inferred batch information, of which 47% (3047/6460) were judged as not requiring batch correction, 19% (1231/6460) were batch corrected and the remainder  could not be corrected due to confounds between the batches and another experimental factor. It is concerning that almost 38% of all datasets lack batch information, which raises questions on whether the data can be fully trusted (and accordingly we downgrade these datasets with lower GEEQ scores); we found that this is prevalent in datasets using the Illumina BeadArray technology , as the submitted data files (both raw or processed) and metadata in GEO almost never contain any batch information or date stamps. We have observed that a fair number of batch-confounded datasets could be ‘rescued’ by splitting the datasets on one of the factors that the original submitters did not intentionally control for (oftentimes either by tissue or cell line). Rescuing of these datasets is a recent addition to our curation process that is still being put into effect.Multiple dataset properties are used in determining the GEEQ score. Here, we present some additional data for three of the properties: median sample–sample correlation, presence of outliers in the dataset and ‘batching’ information. It is well established that a high correlation of gene expression profiles between replicate samples is a good indicator of data reproducibility , 51. UsiSeparate from the GEEQ score framework, we explored two other dataset properties that may influence the dataset and curation quality: availability of raw expression data and association of a PubMed publication to the dataset. For the first, 64% (6686/10 420) had raw expression data that were successfully reprocessed through our pipeline [70% of which were microarray platforms (4704/6686)]. Since all of our RNA-seq data are reprocessed, this indicates that 56% (4704/8429) of microarray data had raw data that could be successfully reprocessed. For the second, 86% (8981/10 420) had an association with a PubMed publication ID, indicating that for a vast majority of datasets, our curators will be able to better curate those datasets using additional information from the publications.N = 150, 125 and 54 for human, mouse and rat, respectively). All but seven are microarray platforms (the remainder being RNA-seq ‘pseudo-platforms’). Gemma also contains 21 ‘merged platforms’ to accommodate datasets in which the same sample is analyzed on different (but closely related) platforms.In Gemma, a total of 811 platforms have been loaded, though only 357 platforms are linked to datasets; the remaining 454 platforms are not used actively in Gemma for various reasons which we will discuss. Out of the 357 fully curated platforms, 92% (329/357) of them are used in human, mouse or rat  are collapsed into a single taxon-specific ‘pseudo-platform’ that contains all known genes of the taxon (see ‘Methods’). Similarly, there are 88/454 alternative Affymetrix ‘platforms’ that are listed separately in GEO despite using the same physical array, because they were processed using ‘custom CDFs’; datasets using these are remapped in Gemma to use the ‘official’ manufacturer-designated probe set layout, except in cases where raw CEL files are not available. A further 160/454 unused platforms have no datasets currently linked with them and are retained for record keeping . 34% (153/454) of the remaining platforms have been considered unusable for various reasons, such as microarray platforms lacking specific probe sequence information, and have been blacklisted. Much of the processing and harmonization of platforms in Gemma reduces the complexity of managing platform redundancies and improves platform comparability across technologies within each taxon.N = 1335, 1325 and 263 respectively; The platforms with the most associated datasets for human, mouse and rat are GPL570 (Affymetrix GeneChip Human Genome U133 Plus 2.0 Array), mouse RNA-Seq and GPL1355  were used to annotate human, mouse and rat datasets—which we will further describe. Due to use of ontologies during curation, 98% (51 479/52 762) of dataset topics are represented by 9379 distinct ontology terms and the remainder are free-text terms (836 distinct free-text terms). The mean number of topics per dataset is 5.2.In Gemma, the displayed dataset topics originate from three different levels of curation: dataset, experimental design and sample-level curation. Dataset-level annotations are used to describe the general intent of the dataset and background information not captured at the other levels ; they are appended to the datasets directly and represent 38% (20 332/52 762) of all dataset topics. Design-level annotations are used to describe the underlying experimental design of the study (i.e. different ‘factors’ or sample groupings), while sample-level annotations are used to describe various sample characteristics. As design and sample-level annotations are not appended to the dataset directly, relevant annotations are algorithmically selected to reflect key information of the dataset and are presented alongside dataset annotations as dataset topics. Design and sample-level annotations represent 56% (29 393/52 762) and 6% (3037/52 762) of all dataset topics.Separately, we also track the source of dataset annotations by adopting a framework derived from GO evidence codes; this reflects the two main methods in which annotations are appended to various Gemma entities: manual curation and automation (see ‘Methods’). The bulk of dataset topics are manually curated (code ‘IC’ or ‘inferred by curator’), representing 95% of all dataset topics (49 890/52 762), while the remainder are automated . We next describe summaries of dataset topics using three different groupings (reflective of the core ontologies used): tissue/cell-type, disease and chemical substances.N = 8292) of them contain a dataset topic annotation related to tissues or cell types (i.e. usage of ontology terms from the UBERON or CL ontologies). Collectively, these topic annotations represent 29% (15 202/52 762) of dataset topics. After ontology inference, the number of ‘dataset-to-topic’ associations increases to 265 087 in total. After manual inspection, we show the top 10 terms of varying resolutions that describe the variety of tissues and cell types represented in Gemma’s datasets  of human, mouse and rat datasets contain a dataset topic annotation related to diseases and disorders (i.e. usage of ontology terms from the DO ontology). These topic annotations represent 9% (4849/52 762) of dataset topics, with the number of topic-dataset associations increasing to 36 797 after inference. In Next, 23% (2362/10 420) of datasets contain a topic annotation related to small molecules (i.e. terms from the ChEBI ontology). These annotations represent 7% (3837/52 762) of dataset topics, with the number of topic–dataset associations increasing to 15 229 after inference. N = 582); the inferred associations deriving from 180 children terms used, including ‘dexamethasone’  and ‘tamoxifen’ , both of which are listed in N = 173), deriving its associations from 25 children terms, such as ‘dexamethasone’. Separately, we found that ‘doxorubicin’ , a widely used chemotherapeutic was not included under the term ‘antineoplastic agent’  and ‘application’ (CHEBI_33232); a selection of 10 terms is shown in c agent’ , indicatc agent’ .https://tinyurl.com/Gemma-GSE8030;  These tags have three different background colors: green, yellow and blue. Green tags indicate the tag was added at the dataset level, yellow tags at the experimental design level (i.e. levels of a factor) and blue tags at the sample level. Both the design and sample level (i.e. yellow and blue) tags are propagated automatically to the dataset page.A mix of color-coded emoticons and visuals are used to display the state and quality of the dataset. Here, we see a green smiley which represents a high GEEQ score. We also display batching information  and data reanalysis state (GSE8030 has Affymetrix CEL files and was successfully reprocessed).P-values for each factor or a heatmap showing the top differentially expressed platform elements. A complete table containing the differential expression values  for all the platform elements can also be downloaded for further inspection.Different visualizations are used to present the level of differential expression observed in each dataset . The useThrough the ‘Experimental Design’ tab, the layout of the experimental design for the dataset is shown, with the number of samples in each combination of factor levels . AdditioIn the ‘Diagnostics’ tab, we present different visualizations, including a heatmap of the sample–sample gene expression correlations, PCA scree plots, bar plots showing the association of factors to the principal components and a mean–variance scatterplot . Part ofAnother important page is the search page, where the end user could search for various entities, including genes, datasets (‘Experiments’) and platforms that are included in Gemma. Searches can be initiated through the search box found on the homepage or through the ‘Explore’ panel on the upper banner present across Gemma . For dathttps://tinyurl.com/Gemma-REST. For R users, we also provide an R package that utilizes the Gemma REST-API, returning the output as R objects that is more convenient to use  were imported from NCBI GEO; 32% (N = 802) of those GEO studies are also contained in Gemma. While GXA’s taxon coverage is fairly wide and would benefit researchers interested in taxa not included by Gemma, Gemma contains far more datasets; more so for human, mouse and rat. Gemma’s more limited taxon coverage also enables us to focus our curation efforts on those three taxa. Another core difference is that GXA includes proteomics data, while Gemma solely processes transcriptomic data. Additionally, there are notable differences in the tooling and procedures for processing data. Batch correction of data is performed in Gemma using ComBat, while it is unclear if correction is implemented in GXA. For ontology support, while GXA relies exclusively on EFO (which does import terms from other ontologies), we rely on 12 different ontologies (including EFO), providing us greater coverage of concepts. The Gene Perturbation Atlas (GPA), on the other hand, was created to reuse expression data generated from human and mouse studies where genes were manipulated . For experiments involving the manipulation of protein-coding genes, this atlas contains data from 1749 GEO microarray datasets; 40% (N = 692) of which are also contained in Gemma. Due to GPA’s focus, users are constrained to genetic manipulation datasets, whereas Gemma provides the flexibility to also work with non-genetic manipulation datasets at the same time. Gemma also provides data from both microarray and RNA-seq studies, while GPA is currently solely microarray-based. Additionally, it is unclear whether the data in GPA were reprocessed uniformly or if reprocessing from raw data were performed. Ontologies are not used in GPA, potentially limiting the potential for cross-database connectivity in the future. The All RNA-seq and ChIP-seq Sample and Signature Search  was creThroughout this paper, we have emphasized Gemma’s utility in allowing access to data and analysis results for datasets in GEO. We have not detailed other analytical capabilities  in Gemma, largely because these features are currently undergoing redesign and revision. As a data resource, Gemma’s main shortcoming is that we only have a fraction of the data contained in GEO. This is a gap that is unlikely to be closed, because the generation of data outstrips available resources to perform curation. As such, we have elected to focus our efforts on curating rodent and human datasets we deem relevant to neurodevelopment, neurological and neuropsychiatric conditions—a more manageable goal considering the annual submission rate of such datasets to GEO is approximately 500. However, we do continue to add data on other topics and are able to respond to user requests for specific datasets.There are several features that we plan to implement in Gemma. First, single-cell RNA-seq is rapidly gaining wider adoption, and we anticipate a growing number of cell type–specific studies with biological replication, which we will be working to accommodate. Second, we will be enhancing the display and interpretation of differential expression on Gemma’s ‘Gene Information’ pages. Recently, we showed that for certain genes, there is some level of predictability in their differential expression . This im


Flow and mass cytometry are important modern immunology tools for measuring expression levels of multiple proteins on single cells. The goal is to better understand the mechanisms of responses on a single cell basis by studying differential expression of proteins. Most current data analysis tools compare expressions across many computationally discovered cell types. Our goal is to focus on just one cell type. Our narrower field of application allows us to define a more specific statistical model with easier to control statistical guarantees.R package and workflow for both strategies on a pregnancy dataset.Differential analysis of marker expressions can be difficult due to marker correlations and inter-subject heterogeneity, particularly for studies of human immunology. We address these challenges with two multiple regression strategies: a bootstrapped generalized linear model and a generalized linear mixed model. On simulated datasets, we compare the robustness towards marker correlations and heterogeneity of both strategies. For paired experiments, we find that both strategies maintain the target false discovery rate under medium correlations and that mixed models are statistically more powerful under the correct model specification. For unpaired experiments, our results indicate that much larger patient sample sizes are required to detect differences. We illustrate the CytoGLMM Our approach to finding differential proteins in flow and mass cytometry data reduces biases arising from marker correlations and safeguards against false discoveries induced by patient heterogeneity. Flow  and massStatistical workflows that analyze data generated by flow and mass cytometry usually begin by clustering cells into both known and novel cell types.  provide We can classify differential analysis methods into marginal regression—analyses that focus on individual markers—and multiple regression—analyses that work on multiple markers simultaneously. The Bioconductor workflow by , cydar, Citrus and CellCnn are multiple regression methods. Their advantage is that they can provide a conditional interpretation of the effect of a protein onto the outcome, and thus reduce the bias due to marker correlations. A disadvantage is that Citrus summarizes protein expressions by taking the median for each cell type which can lead to a decrease in statistical power. The power decrease comes from the reduction in cell sample size from thousands of cells to one cell per sample. On the other hand, CellCnn uses a neural network for which it is currently unclear how to build confidence intervals, derive STAT1 and IRF1. Assume that applying a stimulus to STAT1 activates transcription of IRF1. Further assume that the stimulus does not directly activate IRF1. If we performed separate differential analyses on protein STAT1 and IRF1, we would observe differential expressions for both STAT1 and IRF1, even though only STAT1 had been directly activated. In contrast, a multiple regression method would report STAT1 as differentially expressed given IRF1, and IRF1 as not differentially expressed given STAT1.It is helpful to consider an example to further illustrate the differences between the marginal and the multiple regression method. Consider two intracellular proteins involved in interferon-CytoGLMM implements multiple regression that accounts for marker correlations without the aforementioned limitations. The main difference between our method and current methods is that we focus on cell-specific differential analysis and one fixed cell type, whereas current methods  learn cell types and perform differential analysis jointly. The narrower field of application allows us to define a more specific statistical model with easier to control statistical guarantees. Only the Bioconductor workflow by  focuses We present two versions of multiple regression: (1) A Generalized Linear Model (GLM) for unpaired samples. A GLM is a regression model that allows for a response and error terms that follow different distributions than the normal. (2) A restricted Generalized Linear Mixed Model (GLMM), which is a GLM that allows for random and fixed effects, for paired samples—when the same donor provides two samples, one for each condition. GLMs and GLMMs are generalizations of least squares to non-normal data. In our case, we will use logistic regression to model the experimental condition as unfair coin flips—when the coin flip comes up heads then the cell is declared to be stimulated, otherwise it is unstimulated. We model the coin fairness with a linear model of marker expressions after applying a transformation that ensures each coin flip has a probability of heads between zero and one.Our models depart from the classic model where the marker expressions are the response variables. In our GLMs, the experimental condition is independent of the marker expression of interest given the other markers if the regression coefficient is zero (Proposition 2.2 in ). In conWe present a conditional differential analysis to avoid biases arising from marker correlations by using multiple regression instead of marginal regression.We present two multiple regression strategies that work with the unsummarized expression data to maximize statistical power and account for patient heterogeneity to safeguard against false discoveries: (1) GLMs with a patient-level bootstrap, and (2) GLMMs with a patient-level random effect.The “R package CytoGLMM on different simulated datasets, and illustrates the full workflow for real pregnancy data. In the “In summary, our two main contributions are: We first evaluate the GLM and GLMM procedures for both paired and unpaired samples on simulated datasets. We then test them on a real pregnancy dataset.We generate simulated data with both cell and donor level variability. We allow for negative and positive correlations between markers and a wide range of correlation strengths. We simulate different scenarios ranging from weak to strong patient/cell variability. To make sure that we generate positive counts we use a Poisson noise model after transforming the generated expressions to positive real numbers using the exponential function. This is similar to using the log link function for Poisson GLMs. Overall, there are four main parameters: correlation We perform simulations with a variety of different parameters. All simulations have 16 samples. For paired samples, those 16 samples come from 8 donors. For unpaired samples, those 16 samples come from 16 donors. Each sample has 1000 cells. We compared the observed False Discovery Rate (FDR) and the power. The FDR measures the statistical type 1 errors, the expected proportion of falsely declared discoveries over the total number of reported discoveries. The statistical power represents the proportion of correctly reported discoveries over the total number of true discoveries.Figures First, let’s consider the paired samples experiment. The plots on the left show results when we vary cell and donor-level correlations at a fixed amount of cell mentclass2pt{minimIn the unpaired samples experiment, we only show GLM results as the GLMM results have zero power, there is no data to estimate the donor-level random effect term. We observe up to 20% FDR with a target FDR of 5%. To have non-zero power we need to increase the effect size to 15 . Furthermore, FDR is only controlled under medium cell-level marker correlations using the more conservative BY procedure, with BH exceeding We reanalyze a published dataset on the maternal immune system during pregnancy . The stuThe data were collected at early, mid, late pregnancy, and six weeks postpartum. Samples were left unstimulated or stimulated. Stimulation conditions included: formance .R package openCyto [In our analysis, we focus on comparing early  and the method of moments estimates (for GLMMs) with 95% confidence intervals for the fixed effects t}β Fig. . We trandings by  reportinThe GLMM method takes 1–2 s for the pregnancy dataset with 178,872 NK cells. The GLM requires resampling the data multiple times. For 1000 bootstrap replicates it takes 5 min for the pregnancy dataset. We obtained these running times on a laptop with an 2.3 GHz quad-core processor.R package CytoGLMM provides functions which are applicable to a wide range of cytometry studies. Besides comparisons on paired samples, where samples are available for the same subject under different experimental conditions, our CytoGLMM is applicable to unpaired samples, where samples are collected on two separate groups of individuals.Our new R package. In simulated paired samples experiments, both GLMM with Benjamini-Hochberg (GLMM-BH) and Benjamini–Yekutieli (GLMM-BY) procedures control the FDR below the target FDR under cell-level marker correlations with an autoregressive structure with correlations up to Our simulation experiments compare multiple regression GLM and GLMM, as implemented in cytoglm and cytoglmm in our Interestingly, our power analysis suggests that GLMM-BH achieves adequate power with 1000 cells per sample. Not much can be gained by going to 10,000 or more cells. Such cell counts are not uncommon in cytometry studies. Our findings suggest that CytoGLMM will not detect any differential expression for rare cell types with around 100 cells per sample. Citrus showed low power in our simulation analysis. This makes sense as Citrus was not intended to be used for predefined cell types—its main focus is cell type discovery.Overall, larger cell-level and donor-level correlations increase power and reduce the observed FDR. Hypothesis testing under arbitrary dependency structure is still an active research topic , 18, 19.In addition to corroborating a differential expression of pSTAT1 in the original study , we alsop-value tables—using the summary function—to control the overall FDR.We analyze 10 functional markers in the pregnancy data. CytoGLMM scales computationally to larger number of markers as GLMM can be implemented with the method of moments, and GLM with fast numerical optimization procedures. For example, a GLMM analysis on 40 markers, 16 samples, and 10,000 cells per sample takes 10 s on a laptop with an 2.3 GHz quad-core processor. There is however a statistical tradeoff as the effective sample size will be anywhere between the number of samples and the cells. To extend our methodology to more than two groups, we recommend to run a separate two-group CytoGLMM analysis on each pair, and combine the R package openCyto [Our simulations are limited to a Poisson mixed effect model for protein marker expression. Our conclusions are only valid with respect to this model. The real data generating process might be different. Two main caveats are to be noted. First, we can only encode an experimental design comparing two groups. Second, we require gated cell types. To reduce the person-to-person bias of manual gating, we employed the openCyto . The curA possible alternative to GLMMs are Generalized Estimating Equations (GEEs). GEEs are statistically more efficient when the covariance structure of the residuals are known. In our case, the covariance structure is unknown and needs to be estimated from the data. In most immunology studies, we only have a few donors without a given covariance structure (e.g. no time dependency), resulting in a hard and possibly unstable covariance estimation problem, which could result in an overall loss of efficiency .We presented a conditional differential analysis to avoid biases arising from marker correlations. We built statistical models of the unsummarized expression data to maximize statistical power, and modeled patient heterogeneity to safeguard against false discoveries. The main difference between our and related procedures is that we assume that the cell type is known or can be estimated with high accuracy. This assumption is reasonable in many studies with cytometry data. In our own work, we applied CytoGLMM in wide range of immunology studies: In , we idenBoth the GLM and GLMM method build on generalized linear models that can model other data types than binary responses. Therefore it would be possible to extend CytoGLMM to continuous response variables. A more challenging next step is extending CytoGLMM to include more complicated experimental designs; e.g. twin studies .We recommend that marker expressions be corrected for batch effects , 24–27 aThe goal of the GLM is to find protein expression patterns that are associated with the condition of interest, such as a response to a stimulus. We set up the GLM to predict the experimental condition from protein marker expressions, thus our experimental conditions are response variables and marker expressions are explanatory variables. The response variable R. Our logistic regression model, which is part of a general class of GLMs, can be summarized in the following form:The response probabilities al tails . We use al tails  and BY [al tails  procedurWe make additional modeling assumptions by adding a random effect term in the standard logistic regression model to account for the subject effect. The covariates The mixed effect model is a compromise between two extremes. On the one hand, we could estimate separate regression coefficients for each donor. This corresponds to random effects modeled with a multivariate normal distribution with infinite standard deviations and no constraint on how coefficients are related to each other. On the other hand, we could pool all donors into one group and ignore the donor information. This corresponds to a GLM with no random effects, with no additional variability besides the fixed effect term. A compromise between these two extremes is to estimate the standard deviations of the random effects from data, allowing the regression model to learn from the other donors. Mixed effects procedures are related to empirical Bayes procedures . The firR package mbest to estimate the model parameters We use the method of moments as implemented in the rived by . The autC, have a non-zero effect size, whereas, all markers that are not, have a zero effect size:rth row and sth column of the correlation matrix We construct our simulated datasets by sampling from Poisson GLMs. In prior work, we confirmed—with predictive posterior checks—that Poisson GLMs with mixed effects provide a good fit to mass cytometry data on the same pregnancy dataset . We consR package openCyto [https://christofseiler.github.io/CytoGLMM/articles/pregnancy_dataset.html.We reproduce the original gating strategy according to the supplementary material  from  using thopenCyto . In our