They play an important role in post-transcriptional regulation of gene expression. Evidences of microRNA acting as promoter/suppressor of several diseases including cancer are being unveiled. Recent studies have shown that microRNAs are differentially expressed in disease states when compared with that of normal states. Profiling of microRNA is a good measure to estimate the differences in expression levels, which can be further utilized to understand the progression of any associated disease.Machine learning techniques, when applied to microRNA expression values obtained from NGS data, could be utilized for the development of effective disease prediction system. This paper discusses an approach for microRNA expression profiling, its normalization and a Support Vector based machine learning technique to develop a Cancer Prediction System. Presently, the system has been trained with data samples of hepatocellular carcinoma, carcinomas of the bladder and lung cancer. microRNAs related to specific types of cancer were used to build the classifier.When the system is trained and tested with 10 fold cross validation, the prediction accuracy obtained is 97.56% for lung cancer, 97.82% for hepatocellular carcinoma and 95.0% for carcinomas of the bladder. The system is further validated with separate test sets, which show accuracies higher than 90%. A ranking based on differential expression marks the relative significance of each microRNA in the prediction process.Results from experiments proved that microRNA expression profiling is an effective mechanism for disease identification, provided sufficiently large database is available.The online version of this article (doi:10.1186/s12885-016-3042-2) contains supplementary material, which is available to authorized users. Recent feration . Downregferation . Landi eferation . Aberranferation . Presentferation . Phenomiferation .microRNA profiling application ranges from identification of microRNAs involved in cell differentiation, novel microRNA discovery, microRNA : mRNA and microRNA : protein interactions and as biomarkers. It is a difficult task due to very low presence (0.01%) of microRNA in total RNA mass, lack of common start or stop sequence and very short sequence length. Despite these challenges, three different strategies were established - a hybridization based method , Quantitative reverse transcription PCR (qRT-PCR) and Next Generation Sequencing (RNA-Seq) . qRT-PCRO(mn). Btrim [O(mn/w), where w is word length of the computer. FastX, TagCleaner [NGS reads may contain adapter or fragments of adapter sequence, which were added during library preparation step of sequence generation. Adapters are not part of biological sequence and if not trimmed would be a reason for wrong downstream analysis. Given a read and an adapter sequence the problem of adapter removal can be modelled as an optimal semi global sequence alignment problem. Several of preprocessing tools have been developed recently for efficient adapter removal and quality trimming. They differ in accuracy, speed, memory requirement, capability to trim at 5’ end or 3’ end or both the ends and capable of handling single or paired end reads. Quite a few algorithms are based on Watermann Smith sequence alignment algorithm, having a time complexity of ). Btrim  is a vergCleaner  and SeqTgCleaner  are usefgCleaner , AllenTrgCleaner , CutadapgCleaner , AdaptergCleaner  and SkewgCleaner .read has an approximate/exact match. Around 60 sequence mapping tools were developed after Next Generation Sequencing came into existence. Accuracy, speed, read length support and memory requirement are critical parameters in measuring performance of sequence mapping tools. Speed of operation can be enhanced by applying limits to the number of mismatches permitted and the gap lengths allowed. Another possible way to increase speed is by ignoring read quality score and single nucleotide polymorphism(SNP) information. The number of mapped reads decreases with increase in read length for a given threshold of mismatches permitted. Paired ends map to a reference sequence if the reads are within a threshold value of insert size. Throughput of tools that consider paired ends are lesser than those of single end reads [In genome scale mapping, the reference sequence consists of millions of nucleotides, there could be many locations where a nd reads .Generally, sequence mapping starts with the creation of an index of the reference sequence. Most popular indexing techniques used in the tools are hash table and Burrows Wheeler Transform (BWT). Hash table based indexing keeps a pair - keyword and value, where keyword is a k-mer generated from the reference sequence and value returned is the coordinate of matched location in the reference sequence. A new index with small memory requirement, based on BWT, namely FM Index is the backbone of another set of tools. When compared with Hash tables, index built time is higher for BWT, but it works efficiently in cases when a single read matches with multiple locations. Examples of tools based on BWT indexing are BWA , Bowtie The objective of the present study is to develop a cancer prediction system using microRNA profiling. This is accomplished by the application of machine learning technique on differential expression of specific set of microRNAs in normal and in tumour samples. Profiling of microRNAs is performed by removing the adapter sequences, sequence mapping to quantify mature microRNA sequences, followed by a normalization procedure.microRNA transcriptome data for lung cancer, hepatocellular carcinoma and bladder cancer were used to build a cancer prediction system. Data in Sequence Read Archives (SRA)format was downloaded from National Center for Biotechnology Information (NCBI). The lung cancer data set consists of 41 samples (SRP009408- microRNA expression profiles in lung cancer tissues versus adjacent lung tissues using next-gen sequencing). 20 samples of bladder cancer (SRP007946) and 46 samples of both normal and tumour for hepatocellular carcinoma (SRP049590) were downloaded. Illumina Genome Analyzer II was the sequencing machine in all these three cases. Another 9 samples of lung cancer were downloaded to conduct an independent test (SRP040720). The microRNAs that are linked with up/downregulation of lung cancer, hepatocellular carcinoma and carcinomas of the bladder were obtained from microRNA - disease association databases such as miRCancer , Phenomiread with a maximum of one mismatch was considered as a hit [read and a mature microRNA were made mandatory to prevent the reads mapped to paralogs of a given microRNA and to avoid multiple ambiguous hits [Quantification of mature microRNAs is preferred to pre-microRNAs since the former show active role in gene regulation. Several approaches have been employed by researchers/scientists to quantify microRNAs in NGS reads. In one of the profiling experiments, mature microRNA sequence aligned to a as a hit . Two or as a hit . There aous hits .The proposed architecture for cancer prediction system from NGS data is depicted in Fig. TrimGalore, which in effect uses two popular tools cutadapt and FastQC. Adapter removal was done by cutadapt and quality trimming was done by FastQC. In this experiment, we insist the length of resultant reads to be atleast 20 and the quality threshold to be at least 30. To align reads to MirBase, a memory efficient and ultra fast sequence mapping tool, Bowtie is used [bowtie−build, to create the index. Bowtie alignment policy can be set either by seed length (-l) and number of mismatches in seed region (-n) or total number of mismatches (-v) in the entire alignment. When a sequence mapping tool like Bowtie is used to map reads to a mature microRNA sequence, the total number of aligned reads can be taken as the measure of its expression level. We used idxstats of samtools [To perform preprocessing we used  is used . Memory samtools  to get tIn microRNA profiling, normalization is a critical step as it tries to correct bias in the data. Several normalization methods are available, specifically applicable to microarray analysis, Real time PCR and Next Generation Sequencing. In Next Generation Sequencing, relative count of microRNA is found by normalizing reads against total number of reads in the sample or total number of maps to microRNAs in the sample. The resultant value is expressed as reads per million to respective library. Z-score normalization determines the variation of expression value from the mean in units of standard deviation. In this experiment, the normalized expression of a microRNA is Z-score value of microRNA expression with respect to the total mapped microRNAs in the sample.n dimensions, where n is the number of samples. Differential expression in normal and tumour sample was obtained by finding Euclidean distance as a measure of degree of difference in expression values between the samples. If P =  and Q =  are two vectors, the distance between P and Q is obtained by Normalized expression values of a microRNA from all samples can be viewed as a vector of f(x) = ωTx + b, where ω is weight vector and b is bias. ωTx is dot product between two vectors and it is defined as ωTx=Σiωixi. A hyperplane is the set of all points with ωTx= 0, which separates input data into two classes. The bias, b translate the hyperplane away from the origin. The closest points to the hyperplane among positive and negative samples define a margin. Instances in the training set can be viewed as pair  ∀i=1, m, where m is number of instances in the training set. SVM minimizes the risk of misclassification by maximizing the margin between the data points. Therefore, SVM is basically an optimization problem to find out a Lagrangian multiplier, αi>0, such that L is maximum with a constraint. A prediction model based on Support Vector Machine(SVM) is used to classify the data samples. SVM is a supervised machine learning algorithm. It works by projecting data in input space to a feature space of higher dimensions. SVM has been selected for this experiment due to its ability to handle high dimensional data and due to its higher prediction accuracy. A linear classifier is based on a discriminant function of the form f(x)=ωTϕ(x) + b, where ϕ is a non-linear function. Direct computation of the function ϕ is not scalable with the number of input features. An efficient way of computation known as kernel trick k is employed to limit the size of resultant feature space and thus memory and computational requirements. A Pearson VII kernel(PUK) is defined as ω and σ control half width and trailing factor of peak, respectively.A linear SVM is of no use, if input data instances are not separable by a linear boundary. Solution to this problem is mapping of data in to a higher dimensional space which may exhibit a linear pattern. A non-linear SVM classifier is based on discriminant function of form We trained and tested classifiers for lung cancer, carcinomas of the bladder and hepatocellular carcinoma separately. We evaluated the performance of the models with three different kernel functions, namely Normalized polynomial kernel, RBF Kernel and Pearson VII kernel(PUK). Normalized polynomial kernel and Pearson VII kernel(PUK) functions were giving almost same performance. The performance of the classifier model was evaluated with the following measures: microRNA data with respect to sequencing experiments in lung cancer, hepatocellular carcinoma and carcinomas of the bladder were retrieved from NCBI. The lung cancer data set contains 21 positive and 20 negative samples, whereas the data sets of hepatocellular carcinoma contains 23 positive and 23 negative samples, and the data sets of carcinomas of the bladder contains 10 positive and 10 negative samples. Accession codes and total number of reads in each sample are given in Additional file β-catenin and TCF-4 [st in our experiment. The second ranked microRNA is hsa-miR-21. Clinical evidence shows that miR-21 acts as tumour suppressor by targeting MAP2K3 gene [Differential expression of a microRNA was computed as the Euclidean distance between expression values in normal and tumour samples. microRNAs were ranked by arranging them in descending order of differential expression. Table nd TCF-4 , which i2K3 gene . microRN2K3 gene  and list2K3 gene . Our stuprecision and Fig. recall obtained from the experiment. Predicted results did not contain any false positive in the case of positive samples of lung cancer, negative samples of hepatocellular carcinoma and carcinomas of the bladder.The cancer prediction system were developed using Support Vector based classifier model. When the system is trained and tested with 10 fold cross validation, the prediction accuracy is 97.56% for lung cancer, 97.82% for hepatocellular carcinoma and 95.0% for carcinomas of the bladder. Figure Even though cross validation is an effective method for validation when limited number of samples are available, we verified the developed model using separate test set and independent test set. Out of 41 samples of lung cancer, 32 were used for training the model and the remaining 9 samples were used to test it. There was only one wrong prediction, which was a false negative prediction. Similarly, for hepatocellular carcinoma we trained with 33 instances and tested the model using 13 instances. Again there was only one wrong prediction and the accuracy is 92.8%. Table We extended the validation of classifier model by increasing the number of negative samples in the data set. microRNA profiling has been repeated for each set of microRNAs specific to each type of cancer. For example, lung cancer specific microRNAs are profiled using data samples of hepatocellular carcinoma and carcinomas of the bladder. Expression values were normalized, appended to the negative set and the cancer prediction system was trained again. Table Figures Early detection is important in the successful treatment of cancer or any other chronic disease. There are many molecular biological techniques available, but they may often expensive and may undergo long diagnostic procedures. We developed a computational method to predict incidences of cancer by using differential expression of specific set of microRNAs. NGS sequencing techniques have evolved to an extent where the cost of experiment is becoming cheaper. This makes NGS based microRNA profiling a feasible option to find aberrant expression of microRNAs. Our prediction system is designed in such a way to predict any type of cancer, provided the system has been trained with data for that particular type of cancer. Presently, the system is capable of predicting three different types of cancer.One of the challenges in accurate detection and quantification of microRNAs when compared with mRNA profiling is handling of shorter length of mature microRNA sequence. This makes the annealing of primers in reverse transcription and PCR to a difficult process. Another barrier in annealing is the inability to selective enrichment due to the absence of a common sequence and wide variation in melting temperature due to the variance in GC content of microRNAs. An implicit assumption in mRNA profiling studies is that there exists a correlation of protein level and differential expression of mRNAs. Normally, a correlation coefficient of mRNA expression versus protein expression is calculated in genome wide studies . When miBiomarkers help to assess disease status and act as an aid for early diagnosis for many types of cancer , 50. BetThe exact molecular mechanism behind gene expression regulation of microRNAs is not unveiled completely. But, increasing evidences with experimental proofs are available for the association between microRNAs and different diseases. The progress in Next Generation Sequencing added great momentum in microRNA research. Many studies related to differential expression of microRNAs in specific diseases/cancer are in literature, but development of cancer prediction system using microRNA profiling is a novel approach.In this paper, we present a method to predict the incidence of cancer by analyzing the NGS data based on disease specific microRNAs. When the experiments were conducted with lung cancer, hepatocellular carcinoma and carcinomas of the bladder samples, the obtained accuracies of prediction were around 97% in cross validation. Independent and separate tests too gave promising results. Thus, profiling of microRNA in any accepted manner is a useful method in forecasting human cancers as well as other diseases in which the system is trained. We hope this could be further extended for the development of more comprehensive prediction systems.


In this paper, we target a small group of biomedical graduate students to show rapid crowd-curation of precise sample annotations across all phenotypes, and we demonstrate the biological validity of these crowd-curated annotations for breast cancer. STARGEO.org makes GEO data findable, accessible, interoperable and reusable  to ultimately facilitate knowledge discovery. Our work demonstrates the utility of crowd-curation and interpretation of open ‘big data’ under FAIR principles as a first step towards realizing an ideal paradigm of precision medicine.The Gene Expression Omnibus (GEO) contains more than two million digital samples from functional genomics experiments amassed over almost two decades. However, individual sample meta-data remains poorly described by unstructured free text attributes preventing its largescale reanalysis. We introduce the Search Tag Analyze Resource for GEO as a web application ( Thus far, precision medicine has gained the most traction in cancer7 where for both non-small cell lung cancer and breast cancer, for instance, the standard-of-care now includes sequencing of genes such as EGFR or quantitating panels of RNA such as those included in Oncotype DX, respectively, to drive therapeutic decisions for new subtypes of patients7. Moreover, clinical trials are ongoing to develop a precision medicine approach to other diseases such as those that affect the cardiovascular8–11 and neuropsychiatric13 systems among others. In fact, the National Research Council recently affirmed that to realize the practice of precision medicine requires building a molecular taxonomy or nosology from functional gene targets defined across many different diseases14. However, the dearth of machine readable public genomics data appropriately curated over a great number of diseases has largely precluded such efforts.The paradigm of precision medicine15–17, and GEO data is rich and complete over a great many diseases and phenotypes. There are currently gene expression measurements openly available on over 2 million samples drawn from experiments amassed since the year 200018–20. Funding agencies, such as the National Institutes of Health (NIH), mandates the public sharing this data from functional genomics experiments, and GEO doubles in size every two years on average. To date, over 21,000 PubMed publications have been derived from over 1,000,000 digital samples (see http://STARGEO.org/stats) measured by microarrays, the single largest type of genomic data within GEO19. While this data can openly be used to define a precision medicine ideal, GEO itself and almost all other attempts at crowd-curation of sample level annotations have largely failed to embrace the guiding principles to make data generated more findable, accessible, interoperable and reusable 21 to ultimately enhance the ability of machines and individuals to leverage GEO data for downstream scientific inquiry.Meanwhile, the National Center for Biotechnology Information (NCBI) Gene Expression Omnibus (GEO) is perhaps the largest of a number of open functional genomics data repositories17 employs a combination of small scale biomedical manual curation by experts with sophisticated text-mining tools to annotate samples with a structured bioontology across studies22, their approach is neither open nor embraces FAIR principles and thus cannot scale GEO’s millions of individual samples. As of October 2015, ArrayExpress had annotated 2,330 datasets studying samples in 6,345 differential comparisons across 25 different organisms (http://www.ebi.ac.uk/gxa/release-notes.html). Moreover, the few other crowd curation attempts24, including some with an interactive meta-analysis portal (http://metasignature.stanford.edu/), have either failed in their scale to annotate GEO and / or failed to embrace FAIR principles that encourage sustained and ever more useful crowd curation.For instance, while GEO itself elicits a basic level of sample curation for contrasts of phenotypes in GEO DataSets, this curation is largely used to visualize gene expression in a given study (or Series), and most importantly DataSet annotations disregard FAIR principles and are not standardized across studies. Similarly, while the Gene Expression Atlas by ArrayExpressWith this immediate and increasing need to better mine large open data stores to foster new knowledge discovery, the NIH had a Big Data to Knowledge (BD2K) initiative to maximize the use of biomedical big data for individual investigators and the overall research community. Towards this end, we introduce the Search Tag Analyze Resource for GEO (STARGEO.org) as a NIH / BD2K-funded online platform to share open crowd-curation of digital samples. Currently, no large-scale central repository of annotations exists that the biomedical community can leverage to characterize the molecular genomic pathology of disease. STARGEO.org fills that gap by providing a convenient web-based annotation interface to facilitate precise curation of digital samples, as well as an analysis portal to easily generate robust genomics signatures from meta-analysis of the genomics data and crowd curated annotations.25. Therefore, we hypothesize that leveraging a small crowd of biomedical graduate students for the curation of biological features with STARGEO.org will result in a precisely annotated dataset of samples that may be used for large-scale translational discovery.Towards that end, we recruited a small group of up to 10 biomedical students to develop STARGEO.org into a structured functional genomics database of digital samples curated for relevant biological features. Specifically, we targeted graduate biomedical students as a scientific crowd enriched with disease knowledge whose members are most incentivized to learn about disease genomics in preparation for their careers. Indeed, studies have shown that levels of intrinsic motivation far outweigh extrinsic motivation in inducing crowd participation and to maintain precision of task performanceIn this work, we demonstrate a rapid crowd-curation of sample annotations across all phenotypes, we report a high precision of annotation among curators to characterize common annotation mistakes, and we demonstrate high and significant biological validity of crowd-curated annotations on open data for characterizing the genomic pathology of breast cancer.The general STARGEO.org workflow for using GEO data to define genomic signatures is shown . Curator26 to immediately make their crowd-curation data Interoperable and machine readable. Also, we provide a snapshot interface for users to quickly assemble and ultimately freeze snapshots of annotations and digitally publish their snapshots to Zenodo (https://zenodo.org) to promote Reusability.Furthermore, we keep the STARGEO.org data in sync with GEO data, which is continually being updated. Once appropriate studies to curate are found, STARGEO.org curators make annotations on Tags that represents knowledge about digital samples. Specifically, we define Tags and annotations as key:value bindings where Tags are the keys that hold annotation values. We allow users to map Tags to formal ontologies sourced from the National Center for Bioontology’s BioPortal27 to allow users to perform robust meta-analyses across Series to define differentially expressed genes. These results we describe here are based on raw GEO data downloaded for 465,770 digital Samples from 11,903 Series (experiments) across 1,682 different Platforms (chipsets) as of December 19, 2013, and we report on 490,110 total sample annotations made on 5,798 series across 278 independent Tags made on that data through December 31, 2015.In addition, we automatically map all probe sets or Platforms deposited in GEO to the National Center for Biotechnology Information’s Entrez gene IDs28, and they allow curators to select subsets of Samples in a given Series for mass annotation.We implemented the STARGEO.org annotation process  to allowWith STARGEO.org, curators design RegExs at the Series level to match and thus discriminate linked Samples in order to assign appropriate Tags. Therefore, a Series with thousands of Samples is Tagged with the same effort as a Series with only ten samples once an appropriate RegEx is used to discriminate Sample level annotations. The web application features real-time highlighting of annotations being applied to samples to make clear the result of any RegEx being applied to Tag samples . Most ofTo instantiate the database with high quality annotations, we recruited ten biomedical graduate students from across the country to curate samples for disease and other biological phenotypes, and we designed a reimbursement scheme to reward their precision in making annotations. We used word of mouth and social media to reach out to potential curators. Our sole criterion was that curators had at least some graduate level training in the biomedical sciences. We used Twitter to strategically recruit curators that would be interested in learning about disease and defining genomic disease signatures. Specifically, we sent direct messages to Twitter users with any keywords like ‘biomedicine’, ‘translational medicine’ or ‘research’ in their profile descriptions as well as key words like ‘student’ and ‘MD’ and/or ‘PhD’ to capture their educational exposure. In all, we recruited three different biomedical graduate students from the local Bay area (Stanford and UCSF), and we recruited an additional six biomedical graduate students across the United States with our Twitter outreach.With this small crowd of curators from 12/1/2014 through 12/31/2015, we made 490,110 total sample annotations using 278 Tags across 149,380 distinct Samples drawn from 11,903 distinct Series. This represents 32% of the 465,770 digital Samples we downloaded from GEO that we have annotated with at least one Tag or 14% of the 1,639 series we downloaded from GEO . We foun30post-hoc to further standardize the semantic consistency of Tags across studies and to facilitate cohort selection of contrasts for meta-analyses.Strategically, curators were allowed the freedom to define new Tags in order to represent any phenotype of interest. We employed a text based system to define new Tags to facilitate complete flexibility to describe any biological or experimental feature. In the initial six-week period of reimbursement, virtually all Tags represented disease states except for demographics such as Age, Gender, etc. We manually controlled the vocabulary of Tags by collapsing obvious duplicates (such as ‘Breast Cancer’ and ‘BRCA’) where appropriate. For disease related phenotypes, such as case or control status, we mapped curator-supplied tags to the Disease OntologyTo test the precision in making the 490,110 sample annotations we acquired, we implemented a validation interface for blinded cross-annotation among the curators—i.e., different curators made independent annotations to check the annotation concordance as a measure of precision of already Tagged Samples. We reimbursed pairs of curators 5 cents for every concordant sample annotation to drive precision, and curators were only reimbursed for 100% concordant Sample annotations per Series. To minimize the potential for abuse of our reimbursement scheme and to ensure the highest reliability of our measured cross-annotation precision, we sought to facilitate true independence of the cross-annotations among different curators. Specifically, we hid all GEO identifier fields to completely blind the cross-annotation interface such that curators cannot easily duplicate concordant Sample annotations for a given Series. Similarly, we strategically hide RegExs submitted from users to again discourage automated cross-annotation without manual review.post-hoc to bioontology.org covering 84%  of distinct cross-annotations made . We found this random pattern of agreement between pairs of expert annotators involved mistakes in defining RegExs such as with capturing Age, the most frequent phenotype annotated initially and subsequently validated. However, other examples of random agreement involved poorly designed Tags that asked for ambiguous annotations. For instance, one example was the MB_Histology Tag for the GSE21140 Series, which is supposed to represent a histological annotation for medulloblastoma. The original RegEx captured categorical annotations of medulloblastoma histology (RegEx=‘(Classic|Desmoplastic|Large cell anaplastic|MBEN)’). However, the validation RegEx matched on whether the patient had primary medulloblastoma (RegEx=‘Primary medulloblastoma’). When grouped by Tags across multiple Series and Samples, the most discordant tags  all deriAdditionally, there was a distinct subset of 10 pairs of sets of Sample annotations with perfect disagreement where Kappa=−1. Almost inevitably, these were mistakes made in matching the RegEx for case or control status. For instance, the largest Series with a Kappa=−1 on cross annotation of 144 Samples was for the RCC_control Tag for the GSE53757 Series, which represents control samples for renal cell carcinoma. The original annotation matched samples with normal kidney  while the validation annotation matched renal cell carcinoma patients .34 as a gold standard for a well annotated set of samples of functional genomics data, and we compared the rank correlation of the summary statistics for the tumor-normal differential expression of STARGEO.org versus TCGA samples. In particular, breast cancer is the best represented disease among TCGA samples, and we performed differential gene analysis on RNA-Seq data from 1,119 cases of breast cancer tumors relative to 113 normal breast tissue samples as controls. We generated a comparable STARGEO.org measure of differential gene expression for breast cancer with meta-analysis  using our crowd-curated annotations. In all, we used 1,234 tumors (cases) versus 535 normal (control) samples of breast tissue over 27 different GEO studies from STARGEO.org. Overall, we found a significant (P<=0.01) Spearman rank correlation of 0.77  of 0.77  across a of 0.77  shared b35 across transplant rejection36, lung cancer37, pancreatic cancer38, chronic renal disease39, preeclampsia41, and sepsis42 among others. However, defining robust gene signatures from public data involves a laborious process requiring substantial technical expertise to download, curate, and analyze digital samples across different datasets. While physicians and scientists are the disease experts most incentivized to annotate and subsequently interpret GEO data, the significant bioinformatics burden to do so precludes their efforts. STARGEO.org immediately solves this problem for individual researchers by providing robust meta-analyzed genomics signatures to users based on their curated annotations of digital samples through the convenient web application. Moreover, STARGEO.org provides a natural mechanism to check those curations for precision and consistency by embracing FAIR principles for crowd curation. This stands in stark contrast of other attempts to annotate GEO, including GEO itself, that disregard FAIR principles thereby handicapping the sustainability of such efforts and development of any robust digital curation community.Robust gene signatures discovered through public disease-related datasets have had tremendous translational impact for biomarker and drug discoveryIn this work, we introduce STARGEO.org as a novel web-based application to gain better descriptions of GEO sample phenotypes uniformly across different studies and to define robust differentially expressed gene signatures of disease by meta-analysis of gene expression. Most importantly, STARGEO.org specifically makes every free text attribute we source from GEO as well as all curation and analysis data we generate immediately FAIR. Moreover, by targeting and reimbursing a specialized crowd of biomedical graduate students, we are able to leverage STARGEO.org to curate biological features with high precision. We found that without any bioinformatics training or experience, the students we recruited were able to dynamically conduct sophisticated meta-analyses to define robust signatures of disease and ultimately discover the molecular pathology of different diseases. As a proof of principle, we demonstrate the biological accuracy of these crowd-curated annotations by significantly recapitulating differentially expressed genes that define breast cancer relative to a TCGA gold standard for a well annotated functional genomics dataset.43–45, and this relationship is robust and independent of any explicit bias among curators46. Therefore, while for lack of a gold standard it remains unclear how sensitive or specific the crowd-curation annotations are, we assume accurate annotations with high inter-rater reliability metrics we demonstrate here despite of any individual curator’s unknown bias.We acknowledge that we cannot estimate the performance of Tags to accurately capture crowd-curation of sample annotations for lack of an appropriate gold standard of annotations from the original data depositor. In fact, the gold standard of open data curation is manual review by human curators as we perform here twice with high precision. The high inter-rater reliability we observed among curators suggests that Tags can reproducibly capture the features of biological samples that the original data depositor intended to share. In the absence of an appropriate gold standard, however, it is reasonable to asses curation performance by consensus theory or majority opinion because aggregation of independent responses across curators is more accurate than any individual curator’s responsehttps://zenodo.org) to promote Reusability. Therefore, by adopting FAIR principles, we may transform STARGEO.org into a translational community resource that can be used to capture open digital curation to characterize the functional genomics of disease on a large scale towards discovery of novel drugs and biomarkers in this age of precision medicine.Finally, STARGEO.org is designed to be for crowd curation of open data what GitHub has been for open source code development: i.e., a community of curators that can openly build large sets of annotations together. Specifically, STARGEO.org is designed to support existing best practices to make research data more findable, accessible, interoperable and reusable  to ultimately facilitate knowledge discovery. We embrace FAIR principals for both the crowd-curated sample annotations we generate and the raw sample attribute free-text data that we download from GEO. By making raw GEO data Findable and Accessible, we immediately provide a valuable tool beyond the standard search interface that GEO provides. By building in ontology-mapping functionality from bioontology.org to map our Tags, we immediately make or crowd-curation data Interoperable. We provide a snapshot interface for users to quickly assemble and ultimately freeze snapshots of annotations and digitally publish their snapshots to Zenodo.org (https://www.postgresql.org) relational database management system backend. With this schema, we implemented a web application in Python (https://www.python.org) programming language using the Django (https://www.djangoproject.com) web development framework that allowed us to crowd-curate a semantic network of Tags and appropriate sample annotations representing biological diseases and other phenotypes. We also implemented the functionality for users to quickly assemble and ultimately freeze snapshots of annotations on STARGEO.org and digitally publish their annotation datasets to Zenodo.org (https://zenodo.org) for formal citation.Using the Amazon Web Services cloud infrastructure, we downloaded over 1.7 TB of public data for all processed expression data and associated attributes for series, samples, and platforms catalogued in GEO (ftp://ftp.ncbi.nih.gov/pub/geo/DATA/), and we developed a scalable database schema to represent their attributes as Tags defined as curator-assigned key:value bindings for digital samples where key sample tags are bound to sample annotation values  on an open-source PostgreSQL  across 1,682 different Platforms (chipsets) as of December 19, 2013, and we report on curations made on this raw GEO data through 12/31/2015. We full text indexed all 14,874,580 sample and 283,883 series attributes to facilitate rapid searches at the sample attribute level, a task currently impossible on GEO. We leveraged regular expressions (RegExs) in Python to design an annotation interface for curators to use to quickly annotate sample with Tags to represent biological interpretation. We integrated a blinded validation scheme that allowed for cross-annotation of Tags on which we derived measurements of precision. We used simple concordance estimates as well as Cohen’s Kappa statistic47 gene annotation service, to map all probe identifiers to Entrez gene identifiers. For STARGEO.org, we used samples with crowd-curated annotations that were made across 1,234 cases versus 535 control samples from 27 different GEO experiments. For every gene measured in each study, we estimated the mean difference of contrasts for expression as well as the standard deviation of that mean difference. We used a standard meta-analysis with 1) fixed and 2) random effects model to combine these estimates across studies to generate meta P-values and meta effects across studies. Specifically, we used inverse variance weighting for pooling of the data across studies, and calculated weights for estimates of random effects with continuous outcome data via the DerSimonian-Laird estimate49. We use Python to implement these analyzes in STARGEO.org. All raw GEO data, curations, and analyses that we generate are available at the http://STARGEO.org web application portal with documentation for programmatic download via a representational state transfer (ReST) application programmer interface (API) through http://STARGEO.org/docs.We used STARGEO.org to define a genomic signature for breast cancer on crowd-curated and compared it with a genomic signature for breast cancer using TCGA data. We used STARGEO.org mappings, based on the mygene.infoT-test based on fragments per kilobase per million sequenced reads (FPKM) estimates50, and (2) differential gene expression analysis based on the negative binomial distribution (DESeq2) method51. We used Spearman rank correlation across all four comparisons of differentially expressed genes between STARGEO.org (random versus fixed effects) meta-analyses and TCGA (FPKM versus DESeq2) analyses. Although all the comparisons were both highly and significantly correlated by Spearman rank correlation, we found that the highest correlation of the STARGEO.org breast cancer genomic signature under random effects for STARGEO.org and the FPKM for TCGA, and these results are reported as our results. To correct significance for multiple tests, we applied the Benjamini–Hochberg procedure52 and selected genes with false discovery rate (FDR)<0.1 (10%). For both STARGEO.org and TCGA analyses, we scaled the fold change of each gene’s effect by the significance  × fold change), and used this score to rank genes by their differential expression and estimate the overlap among the top 200 (1%) of genes53 shared between the two datasets. All calculations are provided as For TCGA, we downloaded RNA-Seq data already preprocessed to transcript counts across genes and deposited in GEO with clinical annotations from thousands of samples from TCGA and matched controls (GSE62944). We selected 1,119 breast cancer cases versus 113 controls and performed two standard types of analyses to define differentially expressed genes: (1) A statistical How to cite this article: Hadley, D. et al. Precision annotation of digital samples in NCBI’s gene expression omnibus. Sci. Data 4:170125 doi: 10.1038/sdata.2017.125 (2017).Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


Stromal-derived intratumoural heterogeneity (ITH) has been shown to undermine molecular stratification of patients into appropriate prognostic/predictive subgroups. Here, using several clinically relevant colorectal cancer (CRC) gene expression signatures, we assessed the susceptibility of these signatures to the confounding effects of ITH using gene expression microarray data obtained from multiple tumour regions of a cohort of 24 patients, including central tumour, the tumour invasive front and lymph node metastasis. Sample clustering alongside correlative assessment revealed variation in the ability of each signature to cluster samples according to patient-of-origin rather than region-of-origin within the multi-region dataset. Signatures focused on cancer-cell intrinsic gene expression were found to produce more clinically useful, patient-centred classifiers, as exemplified by the CRC intrinsic signature (CRIS), which robustly clustered samples by patient-of-origin rather than region-of-origin. These findings highlight the potential of cancer-cell intrinsic signatures to reliably stratify CRC patients by minimising the confounding effects of stromal-derived ITH. Tumour expression profiling is currently used for prognostic and predictive purposes without taking into account the intra patient heterogeneity. Here the authors show that cancer cell specific signatures overcome the tumour heterogeneity effect and result in better classification of colorectal cancer patients. The application of transcriptional gene signatures to stratify tumours into prognostic and predictive subtypes has evolved rapidly since the original landmark studies demonstrated the clinical utility of this approach in breast cancer1Recently, an international colorectal cancer subtyping consortium, tasked with defining the transcriptional landscape of CRC, published their assignment of four consensus molecular subtypes in CRC (CMS1-4)8By evaluating differentially expressed genes between the central tumour (CT) and invasive front (IF) regions of primary colorectal tumours, we previously demonstrated that stromal-derived ITH can be a major confounder of transcriptomics-based CMS patient subtypingWe have previously demonstrated how stromal-derived gene expression could undermine patient classification when using a gene signature associated with the recently published CRC CMSet al.et al.et al.et al.n=24) and invasive front   . To incl4) et al.. Using t4) et al.. In addi4) et al.. In addi4) et al.. The sten=24, total dataset n=72). Utilizing this cohort, we then employed a semi-supervised analysis of each signature, using hierarchical clustering of region-specific samples from both the primary tumour CT and IF, alongside the LN metastasis samples. This analysis confirmed the ability of our previously published stromal-associated 30 gene signature to identify regions-of-origin rather than patient-of-origin, with similar results observed for the stem-like CMS4 classifier . To allow a quantitative comparison of both the intra- and inter-patient similarities for each signature, we implemented an additional normalization step in this analysis (detailed in the Pearson similarity section of the Methods section), by assessing the correlation between samples specifically from the same patient, compared to samples from different patients . Using tBRAF mutational status using gene expression data from the PETACCGiven the results of the clustering and similarity analyses, we hypothesized that the level of performance observed for each signature would be relative to the proportion of stromal and epithelial genes in each classifier. We also proposed that the Popovici signature genes would be predominantly epithelial tumour cell derived, given the superior performance of this signature in our study. The Popovici signature was developed by examining the transcriptional profile associated with epithelial P<0.0001, Tukey's multiple comparison test P<0.05, These results further reinforced the findings of our previous work, in which we reported that cancer-cell extrinsic genes can confound transcriptomics-based patient classification signaturesTo directly compare the patient classification results using the published methodologies for both the CRIS and CMS classifiers, we performed sample classification with the random-forest CMS classifier algorithm, alongside the CRIS classifier, which uses a nearest template prediction (NTP) classifier, on our complete cohort. We observed that while CMS classification results in concordant assignment of 9/24 (38%) of patient-matched CT and IF samples, the CRIS classifier concordantly assigns 17/24 (71%) of patient-matched CT and IF samples . More deFurther comparison of the CRIS signature using the patient similarity normalized index as before , indicatThe potential clinical utility of both the Popovici and CRIS signatures for patient classification is further reinforced by their ability to maintain patient-specific clustering throughout this step-wise analysis, with only a minimal decline in their efficiency as the stringency increases . By plotWe and others have previously demonstrated how transcriptional-based patient classifiers, such as the CMS89Unsupervised clustering of large transcriptomic array data will generally cluster samples according to patient-of-origin. However, classification of patient samples into prognostic/predictive subtypes is generally dependent on small pre-defined gene signatures that can themselves be confounded by region-of-origin of the tissue. When patient stratification is distilled to this level of granularity, subtle changes in the global gene expression profiles at different regions can have profound implication for subtype assignment. Similar to results with our 30 gene signature, the stem-like CMS4 classifier is confounded by region-of-origin when attempting to classify individual patient samples. Signatures such as the Jorissen and Eschrich signatures11In contrast, the Popovici signatureLarge individual and collaborative efforts aimed at defining CRC molecular subtypes have confirmed the important role that the stroma plays in patient stratification and prognosis7192021222425et al.4The platforms used in the generation of the gene signatures in this study include Affymetrix and custom cDNA arrays, alongside next generation sequencing (NGS) technology. Inevitably, when comparing the utility of these signatures, there will be some cases when individual genes/probes are not universally represented across all platforms, resulting in gene dropout. To ensure that this dropout was minimized, we utilized consensus ‘core genes' for the signatures  and principal components analysis (PCA) to assess the quality of the data before analysis. The distribution of sample data  should approximate a normal distribution and should be free of outliers, demonstrated through hierarchical clustering and decomposition techniques (PCA and/or MDS). Hierarchical clustering (condition tree) shows the relationships among the expression levels of samples, allowing identification of any spurious samples. From the initial QC analyses of the microarray data, we identified 2 samples as outliers from the PCA and clustering tree methods  which weet al.https://github.com/Sage-Bionetworks/crcsc) and scripts executed in R studio version 3.3.1. The colorectal intrinsic subtypes (CRIS) were determined using the nearest template prediction (NTP) classifier, available from GenePattern (https://genepattern.broadinstitute.org/gp/pages/login.jsf). The thresholds used for statistical significance were generated using Benjamini-Hochberg–corrected false discovery rate (BH.FDR) <0.2 (ref. The consensus molecular subtype (CMS) for each region profiled was determined using the random forest classifier described by Guinney 0.2 ref. .ftp://ftp.ebi.ac.uk/pub/databases/genenames/new/tsv/hgnc_complete_set.txt)To avoid variations inherent during probeset-to-gene annotation conversion from signatures developed using different platforms, we utilized previously published consensus gene lists for each signature to be evaluated, as implemented by Sanz-Pamplona and colleaguesIn addition, we have used a combination of Entrez ID, Unigene, HUGO and Refseq to comprehensively update the annotation file for the DSA array. Probes without Entrez Gene IDs that had a RefSeq or Unigene ID in the original annotation were assigned an Entrez Gene ID using the NCBI Batch Entrez tool. Probes that hadn't any identifiers apart from a gene symbol were updated by matching that gene symbol to a previous gene symbol or alias found in the hgnc_complete_set file. Despite these efforts, inevitably no match was found for a small number of genes in some signatures and they were lost for subsequent analysis. The overall levels of gene dropout were minimal; the signature with the highest dropout  was due to a loss of only three genes missing from the core gene signature. Given this limitation, the signatures used in our study can only serve as a representation of the original signatures. Importantly however, our study is focussed on dissecting the cellular source of the core genes related to their ability to robustly cluster patient samples to outline parameters that could potentially improve future signature/classifier design. Each individual signature, in its original form, has demonstrated prognostic value and was selected as they represent the most clinically relevant and widely employed signatures to support the aims and findings in this study.The 30 gene signature was previously generatedet al.n=30), poor <36 months survival (n=48) across Dukes stage B, C and D. Three adenoma samples were included within the good prognostic group. Transcription profiles were generated on cDNA arrays. Using a leave-one-out cross validation approach, a 43 gene signature was developed  or no recurrence (n=143). Transcriptional data was generated on the Almac Colorectal Cancer DSA platform. Using cross validation, a 634 probeset signature was developed which identified a prognostic subtype in stage II, which was further validated in an independent cohort of 144 samples. This signature has subsequently been validated in an additional stage II clinical trial cohortThe Kennedy et al.The Popovici The colorectal intrinsic signature (CRIS)For all analyses, we have used the gene sets as detailed in As outlined above, patients were assigned an alphabetical label A–Y; patient sample M was removed as an outlier before analysis. Divisive analysis clustering (DIANA) was performed on the expression values of the patient matched IF and CT samples, ascertaining to each of the gene signatures on default settings. This was completed using the ‘cluster' package in R statistical environment (v3.2.3). The DIANA method is particularly suited to test inter-patient heterogeneity in sample pairs, as it continuously splits samples into two clusters until it reaches single samples, which allows an assessment of both the final patient clustering and the extent to which an individual signature could be undermined by stromal derived transcriptional ITH. Resulting dendrograms were assessed for patient clustering and values were tabulated.Partek Genomics Suite software, version 6.6; 2016 Partek, Inc., St Louis, MO, USA., was used for independent dataset analysis. For the purpose of clustering, data matrices were standardized to the median value of probe sets expression. Standardization of the data allows for comparison of expression levels for different probe sets. Following standardization, two-dimensional hierarchical clustering was performed (samples × probe sets/genes). Euclidean distance was used to calculate the distance matrix, a multidimensional matrix representing the distance from each data point (probe set-sample pair) to all the other data points. Ward's linkage method was subsequently applied to join samples and genes together, with the minimum variance, to find compact clusters based on the calculated distance matrix.The Pearson correlation coefficient was used to define the ratio between the covariance and the standard deviation of multi-region samples for each individual patient. By generating a score for each sample compared to each other sample, this method allowed us to build a matrix based on an enumeration of the similarities of all three samples  for each individual patient. Increased scores indicate that samples display a higher similarity with other matched region-specific samples from the same patient. As the standard Pearson method allows direct correlation of one sample to another, we wished to test if each individual patient score was higher than that observed across all of the samples. To this end, we used a normalized method, which calculates the relative similarity between the three samples from the same patient, compared to their similarity to samples from all other patients within our dataset, from a score of 0 (no increased correlation of patient matched samples compared to samples from different patients) to 1 .We used sequential analyses to give an assessment of the ability of each signature to both cluster patient samples into the same higher-order prognostic/predictive subtype, followed by the ability of the signature to robustly differentiate and cluster primary and metastatic samples according to patient of origin. The various signatures indicated above were used to cluster the data, using hclust, with Ward's linkage and Euclidean distance metric. The resultant dendrogram was then analysed using the cutree function to extract the group membership, as the number of groups is sequentially increased, from 1 to 24 (the number of patients). At each level, the Patient Group Overall Ratio (PGOR) was calculated as: PGOR=Number of Patients Grouped in the Same Cluster/Total Number of Patients,that is, the PGOR=1 if all samples for all patients are found in the same cluster at a particular level, and PGOR=0 if none of the samples group together consistently. The evolution of the PGOR was plotted against the number of clusters, showing the consistency of patient clustering.http://www.ncbi.nlm.nih.gov/geo/) and is available under accession numbers GSE95109 and GPL23083. For testing of the cell lineage-specific source of the transcripts, gene expression profiles from an independent CRC dataset were downloaded from NCBI Gene Expression Omnibus (GEO) (http://www.ncbi.nlm.nih.gov/geo/) under accession number GSE39396. The GSE39396 dataset contains microarray profiles from fresh colorectal specimens where FACS has been used to divide cells into specific endothelial (CD45+EPCAM−CD31−FAP−), epithelial (CD45− EPCAM+CD31−FAP−), leukocyte (CD45−EPCAM−CD31+FAP−) and fibroblast (CD45−EPCAM−CD31−FAP+) populations before microarray profiling. Plots based on transcriptional data were plotted using GraphPad Prism version 5.03 for Windows, GraphPad Software, La Jolla CA, USA, www.graphpad.com. In addition, for the comparison of CRC sample classification by the Sadanandam signatureOur transcriptional data and updated annotation files, alongside patient and region identifiers has been uploaded to the NCBI Gene Expression Omnibus (GEO) repository .Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Supplementary Figures and Supplementary Table30 gene signatureEschrich et al. signatureJorissen et al. signatureSadanandam et al. signatureKennedy et al. signaturePopovici et al. signatureColorectal intrinsic signature (CRIS) signature


We apply in silico phenotyping to a set of 70 000 RNA-seq samples we recently processed on a common pipeline as part of the recount2 project. We use gene expression data to build and evaluate predictors for both biological phenotypes  and experimental conditions (sequencing strategy). We demonstrate how these predictions can be used to study cross-sample properties of public genomic data, select genomic projects with specific characteristics, and perform downstream analyses using predicted phenotypes. The methods to perform phenotype prediction are available in the phenopredict R package and the predictions for recount2 are available from the recount R package. With data and phenotype information available for 70,000 human samples, expression data is available for use on a scale that was not previously feasible.Publicly available genomic data are a valuable resource for studying normal human variation and disease, but these data are often not well labeled or annotated. The lack of phenotype information for public genomic data severely limits their utility for addressing targeted biological questions. We develop an  To daterecount2  resourceRail-RNA . The userecount2  resourcerecount2  (N = 11 284), and the sequence read archive (SRA) (N = 49 657)  ,17 levelrecount2 . Expressg bwtool . The funrecount2.The GTEx data are well-characterized and are comprised of data generated from hundreds of individuals, with each individual having been sampled at multiple different tissues. Given the study design and well-characterized nature of these data, when possible , the GTEx expression data were used to build the phenotype predictors. When invariability of a phenotype within GTEx limited this approach , the SRA data were used to train the predictor. The first step of our general approach was to randomly sample half of the training data set. Expression data from this subset of individuals comprised our training set. For each predictor, accuracy was then assessed in samples from three tests sets: (i) The remaining samples from the training data set that were not used to build the predictor (ii) TCGA and (iii) the data set not used to build the predictor either SRA (in the case of sex and tissue) or GTEx (in the case of sample origin and sequencing strategy). Accuracy was assessed by comparing the predicted phenotype to the reported phenotype in any sample for which this information was available. Upon achieving sufficient prediction accuracy (≥85% in the training data), phenotypes were then predicted across all samples within R package phenopredict which can use expression data and phenotype information from any study to build a phenotype predictor. Functionality within this package allows for (i) a predictor to be built for either continuous or categorical variables (build_predictor), (ii) the predictor to be tested on the training data to assess resubstitution error (test_predictor), (iii) data to be extracted from a new data set for the same regions upon which the predictor was built (extract_data) and (iv) phenotype prediction in this new data set (predict_pheno). Additionally, for analyses in which data have to be filtered across multiple files (i.e. across chromosomes), there are two functions (filter_regions and merge_input) that can be used to limit the number of regions included for prediction from each file and to merge the filtered regions prior to building the predictor, respectively. Finally, to optimize prediction accuracy, the function optimize_numRegions can be used to determine the optimal number of regions to be used in build_predictor.We have built the limma’s lmFit framework (l) in the phenotype (P),r. The l of the phenotype of interest (P), but it can optionally include other covariates. To ensure predictors herein were as generalizable across data sets as possible, no covariates were included during region selection. r to phenotype Pl. We use limma (l), the topTable function from limma is used to chose the n most significant regions. For predictions herein, the number of regions (n) was optimized using optimize_numRegions function within phenopredict. This choses the fewest regions needed to maximize prediction accuracy in the training data; however, users can alternatively specify their desired number of regions for prediction.The predictor is built by first employing a linear model within ramework  to selecse limma  to modervalidationCellType from the R package minfi (r in the set of regions selected r) in the selected set or regions (l × N matrix where l is the number of levels in the phenotype and N is the number of samples in the training data. We fit the model subject to the conditions that: (i) et al. (minfi (r) across all levels (l) of phenotype  .The same approach as above is carried out with the exception that, in Equation , we replSelected regions are then modeled as in Equation . We fit build_predictor function. Here, using the framework established in projectCellType from minfi (l) in phenotype \Predictions are made from model 4 and we measure error as the root mean squared error from the observed phenotype to the predicted phenotype in the training set.extract_data can be used to obtain expression estimates in a new data set at the same exact regions selected by filter_regions and used to build the predictor. By using expression from a fixed set of genomic regions in the new data set, rather than identifying a new set of predictive regions, we limit the possibility of overfitting our model.Before prediction can be carried out in a new data set, expression data at the precise regions used to build the predictor must be extracted. test_predictor with the exception that expression data in this case comes from a new expression data set (r) upon which the predictor was built, but from an independent set of samples (N*), such that:N*). l) in phenotype P) from Equation  and RMSE between the reported and predicted values, for samples where information was available.Accuracy was again assessed using the correlation coefficient , where TP are the number of true positives and FN are the number of false negatives. Specificity was defined such that Specificity = TN/(TN + FP), where TN are true negatives and FP are false positives.For all phenotypes included in R package recount. Predicted phenotypes can be included using the add_predictions function. For each predicted phenotype, this function adds reported phenotype, predicted phenotype, and prediction accuracy. The most recent release will be used by default; however, older versions will remain accessible through recount. An example of this function is provided at https://gist.github.com/ShanEllis/?add_pred.R and demonstrates how to add predicted phenotypes to the GTEx data within recount.Predicted phenotypes are available through the data. This imports the prediction coefficients and region information for each of the four predictors: predictor_sex, predictor_tissue, predictor_samplesource, and predictor_sequencing strategy. The predictor object for the phenotype the user is interested in predicting (i.e. ‘predictor_sex’ if interested in predicting sex) would then be used as the predictordata argument in the extract_data function within phenopredict.To predict sex, age, sequencing strategy, or sample source in a new data set using the predictors presented herein, users would first obtain the predictor data using the following: recount2. Predicted sex was first summarized overall, where each sample received a predicted sex of male, female, or unassigned (when sex could not be disambiguated). Secondarily, sex was summarized by project. When all the samples within a project were of the same predicted sex, they were classified accordingly as female only, male only, or unassigned only; however, projects with samples of more than one predicted sex were determined to be ‘mixed’. Finally, sample size broken down by sex within each project type was summarized.Sex within the SRA was studied across the 49,657 samples in recount2 requiring that possible projects must (i) employ paired RNA-seq, (ii) not have reported sex included in the study metadata, (iii) have ‘tissue’ as its predicted sample source (rather than be derived from a cell line) and (iv) have at least 20 samples in the study. Studies fitting these criteria were considered for use in Use case #3.Predicted phenotypes can be used to identify samples or studies of interest for future study. To exemplify this, constraints were placed on the predicted phenotype data to search for a study within the cancer studies in recount2  and the abstract of this study included the breakdown of females and male samples , the corresponding abstracts for each study were searched within  samples . Thus, drecount2  and liver metastases (MC) were compared to primary colorectal tissue (PC) samples. To recapitulate the analysis from the initial publication, no outlier genes or samples were removed, no covariates were included, and no adjustment was made for the fact that the PC samples were used in both comparisons. As was done in the initial publication, to protect against hypervariable genes driving differences detected, gene dispersion was estimated for the scaled gene counts for 58 037 genes across 54 samples . For each gene , and each comparison  a negative binomial generalized log-linear model (glm) was fit. To be in line with the initial publication  and colon (N = 376) samples from the GTEx data were utilized for further comparison. GTEx gene expression data (‘SRP012682’) were downloaded from recount2  between GTEx liver and colon samples and the results from both the NC:PC and MC:PC comparisons were generated.Liver ; Genotype Tissue Expression Project (GTEx); The Cancer Genome Atlas (TCGA); Sequence Read Archive (SRA); expressed region (ER); generalized log-linear model (glm); log fold change (logFC); colorectal cancer (CRC); normal colon epithelium (NC); liver metastases (MC); primary colorectal tissue (PC); Differential gene expression analysis (DGEA); Root mean square error (RMSE); concordance at the top (CAT).recount2 (N = 9538), TCGA (N = 11 284) and the SRA (N = 49 657). Region-level expression estimates  and technical (i.e. sequencing strategy) phenotypes from the expression data itself. All phenotype predictors were built using the data available in recount2 . Data inN = 49 65. Region-stimates , which anumRegions argument were tested for each predictor. For categorical variables, numRegions determines how many regions should be selected for prediction from each level in the phenotype; thus, the total number of regions can be up to numRegions × levels. The minimum numRegions value that maximized accuracy in the training set was then utilized for phenotype prediction. This corresponded to the numRegions argument being equal to 20 for both sex and sequencing strategy , 100 for sample source , and 80 for tissue   for the regions) .To characterize these regions further, the genomic location for each region was summarized. Aside from sex, where regions were intentionally only selected from the sex chromosomes, no single chromosome contributed >17.5% of the regions used in any predictor . AdditioN = 4769) and a test set (N = 4769). The 40 regions (20 each from the X and Y chromosomes) that best discriminated males and females in the GTEx training data were selected and the predictor built. In the GTEx training data, when applying the predictor to the data upon which it was trained, the resubstitution error  was 0.1%. Sex prediction was then carried out in the GTEx test set, an expression data set in which the data were generated by the same consortia but whose samples were not included to build the predictor. Prediction accuracy was 99.8% in these data. Sex prediction was carried out in data from TCGA (N = 11 284), a completely independent and well-characterized data set, where prediction accuracy was 99.0%, and in the SRA samples for which sex information was available (N = 3640), where accuracy was 86.3%  demonstrated notably low accuracy (1.14%)  was zero in both samples reported to be male and those samples predicted to be female, while samples predicted to be male had a median log2(expression) of 6.3  was determined. Median study accuracy was 100% (range: 0–100%), with 72% of studies having a prediction accuracy >90% . One rel (1.14%) . Upon cl) of 6.3 . These r) of 6.3 . Taken trecount2 (N = 4769) and 76.8% in the TCGA data (N = 7317)  (http://www.cs.cmu.edu/∼ckingsf/sharq/) was employed. sharq was previously used to scrape the abstracts for a subset of studies included in the SRA, extracting its best guess as to the tissue used for study from each. Using this tool, tissue type was assigned to 8951 SRA samples. These predictions (‘sharq_beta’) were previously made available in recount2 (Tissue prediction is more complex than sex prediction for two reasons: (i) region selection must be carried out across all chromosomes (not just the sex chromosomes) and (ii) there are more than two levels for which a predictor must be built. In the case of the data in recount2 , the GTE) Figure . We note = 6704) , suggest2 N = 476 and 76.8recount2 . By usinN = 10 830). Among these samples, there were 336 unique reported tissues within the SRA with 1442 (13.3%) samples directly matching the predicted tissue from the expression data and sharq_beta predictions, respectively. This relatively low concordance often reflects nonuniform or distinct coding within the SRA. Specifically, there are 1692 cases in which sharq_beta and expression predictions are concordant with one another but discordant with what is reported in the SRA. Of these, 1131 samples (65.2%) include the predicted tissue in the reported tissue . Of the remaining 561 samples  concordant between sharq_beta and expression predictions but discordant with reported tissue, there are a number of explanations. To assess what could be going on, we randomly sampled one sample from each of the 32 unique studies from which these 561 samples were generated. Of these 32 samples, (i) 62.5% of the time (20/32) the predicted tissue was a more general term for the reported tissue , (ii) 15.6% (5/32) of the time the reported tissue was concordant with the predicted tissue, but the reported tissue also included diagnosis , (iii) 9.3% of the time (3/32) the tissue reported did not indicate an actual tissue (predicted tissue is ‘breast’ but the reported tissue is ‘tumor’), or (iv) 12.5% of the time (4/32) the predictions are discordant . Taken together these data suggest that true accuracy is likely between the lower bound of 51.9% determined by comparison to sharq_beta predictions and the 87.5% accuracy found in this less rigid comparison to the subset of SRA samples with defined tissue annotations.To assess this claim directly, we compared predictions from the expression data and sharq_beta predictions to reported phenotypes for the subset of SRA samples where tissue was reported  and a test set (N = 24 828). Forty regions that distinguished single end sequencing libraries from paired end sequencing libraries were identified and the predictor built. In the training set, the resubstitution error was 8.9%. Accuracy was then assessed in the SRA test samples (N = 24 828), GTEx (9538) and TCGA (N = 11 284) samples, where accuracy for sequencing strategy prediction was 90.9%, 99.9%, and 94.1%, respectively  or a primary tissue  were used to select 200 regions upon which the predictor was built. The resubstitution error for this predictor was 10.8%. To assess accuracy in the SRA test set samples, all samples for which sample source information were utilized. Prediction accuracy was 89.2% in these data. Applying this predictor across the GTEx (N = 9538) and TCGA (N = 11 284) data demonstrated its accuracy to be 97.0% and 99.8%, respectively , the lack of correlation in the test data (R2 = 9.9 × 10−3) suggested limited predictability of this phenotype from expression data (https://github.com/ShanEllis/phenopredict_phenotypes). Given these results, some phenotypes will likely be better identified through other means—either by manually curating SRA metadata . By including information about the accuracy of our predictions we make it possible for downstream models to incorporate this uncertainty. While the overall accuracy for any of the predicted phenotypes assessed was at minimum 85%, assessing phenotype accuracy at the sample level is non-trivial. When available, we have included reported phenotype and utilized it to calculate sensitivity and specificity, but this information is not available for many samples and even when it is available, it may be incomplete or incorrect. Phenotype files often use manual data entry and can themselves be prone to error . For exaTo assess discordance levels globally, sensitivity and specificity were calculated for every level of each predicted phenotype. Across predictions, mean sensitivity was 78.6% (range: 18.6–99.1%) and mean specificity was 98.9% (range: 97.2–99.7%) . Upon clWe have reported measures of accuracy for each predictor; however, both reporting error and prediction error affect this measure. Prediction error is error attributable to true limitations of our predictors. This is what we have aimed to minimize in the construction of each of our predictors. Reporting error is error attributable to mislabeling or misreporting within the metadata provided with each data set.To quantify the level of reporting error directly, the sequencing strategy phenotype was used. Unlike biological phenotypes, where the truth is not always known, for sequencing strategy, a technical phenotype, whether the type of reads in the archive (the truth) matched the strategy reported within the SRA metadata could be used to estimate the level of reporting error within recount. The reads counted by Rail-RNA  were comIn the context of the predictions for sequencing strategy made from the expression data, in the SRA training data, of the 224 samples determined to likely be misreported, 207 (92.4%) were also identified as discordant in the expression predictions, suggesting that the true accuracy of sequencing strategy prediction is 92.0% (22834/24828), rather than the reported 91.1%. Similarly, in the SRA test data, 239 of the 251 (95.2%) samples determined to have been misreported were also discordant between reported and predicted sequencing strategy, suggesting that the true accuracy of sequencing strategy prediction is 92.8% (22798/24829), rather than the 90.9% reported accuracy. This reporting error negatively affects our accuracy measure, further supporting that the accuracy measures we have are reported are conservative, lower-bounds for accuracy.N = 1720)  , it becoWhen broken down by project type, it becomes clear that almost half of the projects within the SRA (43.1%) include exclusively female samples. Of the remaining projects, 18.2% include exclusively male samples, 1.2% include only samples that could not be disambiguated, and 37.4% of studies include some combination of predicted sexes Figure . For prorecount2  predicted to be generated from paired end sequencing, (ii) where sex was not reported in the SRA metadata, (iii) where the data were predicted to have come from a tissue (rather than a cell line), and finally (iv) that had at least 20 samples. Upon applying these filters, we identified three studies within the SRA meeting these criteria.P < 0.001 and logFC ≥ 2. No covariates were included for analysis. The authors then, in an attempt to remove the effects of the fact that the MC samples were sampled from the liver while the PC and NC samples were sampled from the colon, filtered out ‘liver specific genes (309 genes)’ using the TiGER database . This study includes expression data from 54 samples from 18 unique individuals. For each individual, RNA-seq was carried out on a normal colonic epithelial sample (NC), the primary colon cancer epithelium (PC), and a metastatic cancer sampled from the liver (MC). The initial analysis sought to identify gene expression changes that correspond to aggressiveness of colorectal cancer (CRC). To do so, they authors carried out differential gene expression analyses between NC and MC samples  and between MC and PC samples (identifying 1846 significant genes). Significance was defined such that database  from therecount2 . Given the experimental design, we checked to ensure that, for each individual, the same sex was predicted across each of the three samples. Predicted sex was concordant across samples within individual for all samples included for study. Analysis was first carried out as in the initial publication , comparirecount2 . Thus, r  plots  comparinet al. MC:PC comparison, relative to overlap of 31 and 26 of the top 1000 genes in either the NC:PC and MC:PC comparison where tissue has been included in the model, respectively (To test this directly, the level of overlap between the aforementioned differential expression analyses and a differential gene expression analysis comparing healthy liver and colon in a completely independent set of samples from the GTEx Project  was asseectively . These rectively  from theFinally, while inclusion of predictions improves the accuracy of this analysis, we note that the the inclusion of predicted phenotypes in downstream analyses is currently limited, as sex and tissue predictions were made from the expression data, the same data upon which the predictions are modeled as covariates to look for expression differences. Novel statistical approaches will be required to most appropriately utilize these predictions in downstream analyses.recount2 (R package phenopredict and can be accessed in the R package recount using the add_predictions function. We demonstrate how predicted phenotypes can be used to characterize the samples currently within recount2, to identify samples and studies of interest for future study, and how predicted phenotypes can be incorporated into analyses to improve accuracy. Taken together, the availability of this critical phenotype information across the 70,000 samples currently included within recount2 (We demonstrate that accurate phenotype prediction from expression data is possible for both biological and technical phenotypes and demonstrate, through three use cases, how predicted phenotypes can be utilized going forward. We have built predictors for sequencing strategy, sex, sample origin, and tissue. These predictors have been applied to the samples within recount2  and can recount2  make dowSupplementary DataClick here for additional data file.