Previously, we classified colorectal cancers (CRCs) into five CRCAssigner (CRCA) subtypes with different prognoses and potential treatment responses, later consolidated into four consensus molecular subtypes (CMS). Here we demonstrate the analytical development and validation of a custom NanoString nCounter platform-based biomarker assay (NanoCRCA) to stratify CRCs into subtypes. To reduce costs, we switched from the standard nCounter protocol to a custom modified protocol. The assay included a reduced 38-gene panel that was selected using an in-house machine-learning pipeline. We applied NanoCRCA to 413 samples from 355 CRC patients. From the fresh frozen samples (n = 237), a subset had matched microarray/RNAseq profiles (n = 47) or formalin-fixed paraffin-embedded (FFPE) samples (n = 58). We also analyzed a further 118 FFPE samples. We compared the assay results with the CMS classifier, different platforms (microarrays/RNAseq) and gene-set classifiers . The standard and modified protocols showed high correlation (> 0.88) for gene expression. Technical replicates were highly correlated (> 0.96). NanoCRCA classified fresh frozen and FFPE samples into all five CRCA subtypes with consistent classification of selected matched fresh frozen/FFPE samples. We demonstrate high and significant subtype concordance across protocols (100%), gene sets (95%), platforms (87%) and with CMS subtypes (75%) when evaluated across multiple datasets. Overall, our NanoCRCA assay with further validation may facilitate prospective validation of CRC subtypes in clinical trials and beyond. The median overall survival of metastatic (m)CRC patients with unresectable disease remains around 24 months with standard chemotherapies. Targeted therapies including anti-EGFR antibodies, anti-angiogenic and immunotherapy agents may extend survival up to 30 months in selected patients2. However, selecting patients who will benefit from different standard-of-care treatment regimens remains challenging. Additional predictive biomarkers are required to spare patients from unnecessary toxicities, improve outcomes and increase cost-effectiveness of treatments.Colorectal cancer (CRC) is the fourth leading cause of cancer-related deaths worldwide3. We demonstrated significantly poorer disease-free survival (DFS) in untreated patients for the stem-like subtype, intermediate DFS for inflammatory and enterocyte, and better DFS for goblet-like and TA3. Then, from two different datasets that included drug response information, we observed increased responses within the stem-like subtype to irinotecan, fluorouracil and leucovorin treatment combination (FOLFIRI), and within the TA subtype to the anti-EGFR monoclonal antibody cetuximab5 using small sample size cohorts. These treatment responses were further validated by other studies7. However, additional validation is warranted.In order to classify CRCs into groups, we previously identified five distinctive gene expression subtypes  using a 786-gene signature: goblet-like, enterocyte, stem-like, inflammatory and transit-amplifying (TA)12. To address the ambiguity caused by these independent studies reaching differing conclusions , these and our findings were aggregated by our CRC Subtyping Consortium (CRCSC) into 4 consensus molecular subtypes (CMS): CMS1 (associated with inflammatory subtype); CMS2 (enterocyte and TA); CMS3 (goblet-like); and CMS4 (stem-like), plus a “mixed” subtype representing either the existence of additional subtypes or the presence of multiple subtypes in a single sample13. CRCA and CMS subtypes are generally highly similar (discussed below), with the primary exception that the enterocyte and TA subtypes of CRCA were merged into CMS2 of the consensus classification13. The current work for assay development and evaluation of the CRCA subtypes started before we published our CMS subtypes, however the clear connection between these two subtyping schemes suggests that our subtyping assay results are representative of real and consistent biological heterogeneity in CRCs.Five other groups independently identified between 3 and 6 molecularly distinct CRC subtypes based on expression profiles14. The prospective validation of these findings into routine clinical practice remains challenging, mainly due to the lack of a fit-for-purpose assay that can classify patient samples into subtypes within a clinically relevant turnaround time and reasonable costs using formalin-fixed paraffin-embedded (FFPE) samples. Microarray or RNAseq gene expression profiles for subtype classification are expensive, time consuming, require dedicated bioinformatics expertise, and have turnaround times incompatible with prospective clinical applications. Our previously published proof-of-concept assays - immunohistochemistry (IHC) and quantitative reverse transcriptase polymerase chain reaction (qRT-PCR) methods3 - may not be easily applicable when higher numbers of genes are required as biomarkers. Hence, we applied nCounter platform (NanoString Technologies) to develop a clinically-relevant biomarker assay for CRCA subtype classification. A summary of the relative costs and turnaround times of these platforms can be found in Supplementary Table SRecently, it was reported that addition of oxaliplatin to fluorouracil-leucovorin in early-stage disease revealed an advantage in disease-free survival in the enterocyte subtype compared to the other subtypes15, as well as assays to predict medulloblastoma16 and lymphoma17 subtypes. This platform measures gene expression in the form of discrete counts of barcoded mRNAs, and requires no amplification step, eliminating a potential source of bias. In the present study, we evaluated the suitability of this platform for a gene expression-based assay for our CRCA subtypes using a modified protocol to classify CRCs in multiple independent cohorts (fresh frozen and FFPE samples). The results were compared to the CMS subtype classification and other platforms. A summary of the classifiers utilised in this study is given in Supplementary Table SThe nCounter platform has previously been exploited to develop the Food and Drug Administration (FDA)-approved Prosigna® Breast Cancer Prognostic Gene Signature Assay to predict risk of recurrence in patients treated with adjuvant hormonal therapy4. A second cohort of RNA samples  was purchased from OriGene . A third cohort included 145 fresh frozen samples (Singapore FF) and a fourth cohort included 106 FFPE samples (Singapore FFPE) from patients participating in an on-going observational study. The fifth cohort consisted of 12 FFPE CRC samples from a retrospective tissue collection from The Royal Marsden Hospital, UK (RETRO-C cohort). The sixth and final cohort consisted of 58 stage II-III CRC patients  with matched prospectively collected fresh frozen and FFPE tissue.Six CRC cohorts of primary tumour samples collected prior to treatment were studied, as summarised in Supplementary Table The protocol for the Singapore cohorts (both FF and FFPE) of this study was approved by the SingHealth Institutional Review Board: 2013/110/B. The protocol for the INCLIVA-Valencia cohort was approved by the Comité Etico de Investigacion Clinica del Clínico Universitario de Valencia: F-CE-GEva-15. The Montpellier cohort had ethical approval as described in their original publication and OriGene cohort is available from the commercial vendor. The protocol for the RETRO-C cohort was approved by NRES Committee East of England-Cambridge Central: 10/H0308/28. All patients provided informed consent, and all experiments were performed in accordance with relevant guidelines and regulations.3: (a) seven genes that were proposed in said report as subtype biomarkers for qRT-PCR and IHC3; (b) among the top 2 to 9 highest scoring genes for each subtype from predictive analysis of microarrays (PAM) centroids3; (c) three additional genes that distinguish TA (cetuximab-sensitive and –resistant) sub-subtypes3; and (d) those representing the characteristics of certain subtypes such as WNT signalling, epithelial-mesenchymal transition (EMT)/stroma, MET tyrosine kinase signalling, and NF-κB signalling. An additional 10 housekeeping genes were used for normalization. All genes and relevant annotations are listed in Supplementary Table The following 50 genes were initially selected for inclusion in the CRCA subtype custom nCounter assay based on our previous reportnCounter® Max Analysis System  was used to perform the assay using either standard or modified (Elements chemistry) protocol as per the manufacturer’s instructions Fig. . For theThe TagSets consist of fluorescently-labelled specific Reporter Tags to resolve and count individual nucleic acid target sequences during data collection (Probe A), and biotinylated universal Capture Tags to capture the hybridised target nucleic acid sequence to the streptavidin-coated imaging surface (Probe B). Final hybridising concentration was 20 pM for Probe A and 100 pM for Probe B.Supplementary Information.For both standard and modified protocols, 100 ng of total mRNA (20 ng/uL) from fresh-frozen or FFPE tissues was used. Hybridisation reactions were prepared according to manufacturers’ instructions for either 18 h at 65 °C using Standard CodeSets reagents for standard protocol or for 20 h at 67 °C using Elements™ TagSets reagents for the modified protocol. Hybridised samples were pipetted using the nCounter Prep Station and immobilised on to the sample cartridge for data quantification and collection using nCounter Digital Analyzer (NanoString Technologies). The nCounter Prep Station and Digital Analyzer together constitute the nCounter® Max Analysis System. For the PanCancer Progression Panel (NanoString Technologies), the standard protocol from the manufacturer was used. The processing and quality control steps are available in idSample employed support vector regression (SVR)18 based on CIBERSORT19 to train a model using reference data, such as centroids of values representing different subtypes - in this case, our published 786-gene signature-based PAM centroids3. The model was then used to predict the gene expression profiles of each sample from a test data set such that it provides weights for each subtype. Finally, the subtypes showing negative weights for each sample (meaning they were not represented in that sample) were set to zero, and the rest of those subtypes showing positive weights were normalised to compute the proportion of each subtype in each sample. Those samples with maximum proportion greater than 70% were selected as representing a single subtype.idSample and another in-house built machine-learning approach-based R package, intPredict. This tool employed a pipeline of different gene selection and class prediction methods to develop a robust gene classifier to predict subtypes by randomly splitting the original data set of samples into training and test data sets and executing the pipeline repeatedly 50 times . Gene selection methods included prediction strength (PS)20, prediction analysis of microarrays (PAM)21 and between-within group sum of squares ratio (BW)22. Furthermore, the best performing gene set from the gene selection methods was identified using multiple class prediction methods such as random forest (RF)23, diagonal linear discriminant analysis (DLDA)22 and two support vector machine (SVM) approaches – linear and radial methods18. The MCR for each method was determined as follows,k is the number of test samples, and ei is the misclassification of each test sample compared to known subtype. The gene set with the lowest median MCR of all the methods was chosen. For each gene set, a 95% credible interval on the median MCR was calculated.We selected a robust set of genes that best predict the CRC subtypes with lowest misclassification error rate (MCR) using the samples selected by r) ≤0.15, or if the correlation was high for multiple subtype centroids (Pearson’s r difference between first and second highest subtypes ≤0.06), in line with the published CMS classifier13.CRCA subtypes were assigned by performing Pearson correlation of gene-wise median-centred expression profiles for each sample with corresponding centroids for the subtypes. The subtype with the highest correlation was then assigned to that sample. Samples were marked as having “undetermined” subtype if the sample’s correlation with the subtype centroid had a value (Pearson’s CMSclassifier R package (v1.0.0) and the classifyCMS function, using the single sample prediction (SSP) classifier13.CMS subtypes were determined from microarray or RNAseq data using the Supplementary Information contains further detailed information of all methods employed.Materials and Methods). Initially, we applied a standard protocol from the manufacturer , we again achieved successful clustering by sample rather than by protocol between the standard and modified protocols for 12 samples . Since mixed subtype samples comprising more than one subtype are present in CRC13, we selected only the samples from our published training dataset (n = 387)3 that showed at least 70% probability of belonging to a single tumour subtype . Furthermore, the intPredict tool, which contains a pipeline of supervised gene selection  and class prediction methods , was used to identify 38 robust genes with the lowest misclassification error rate  using two in-laboratory developed bioinformatics tools – 13  to fresh frozen CRC samples . Subtypes were determined by the correlation of gene expression profiles with the CRCA-38 centroids. All five subtypes were identified by the assay (with 89% (159/179) of the samples being classifiable) and demonstrated distinct patterns of gene expression Fig. . A small3 in 179 samples from three independent cohorts.We further confirmed the molecular characteristics of the subtypes from NanoCRCA, especially stem-like samples, which showed increased expression of genes associated with EMT, stem cells, metastatic response, extracellular matrix (ECM) structure and receptor interaction, cellular differentiation, the collagen family and others Fig. . This waMaterials and Methods; n = 47). Subtypes from CRCA-38 (using the same centroids as utilised to classify samples using NanoCRCA), the original CRCA-786 and CMS were determined on these platforms. Figure We sought to assess if subtyping using the NanoCRCA assay (with the modified protocol and 38-gene classifier) mirrored the results of subtyping using CMS subtypes and platforms such as microarrays and RNAseq. Although these platforms are highly multiplexed, they can be expensive and have long sample and data processing time  were CMS2 ; 11.8% (2/17) each of CMS3 (goblet-like) and CMS4 (stem-like); and 5.9% (1/17) of CMS1 (inflammatory). However, we found 23.5% (4/17) samples with undetermined subtype . Again, the NanoCRCA assay showed good and significant concordance (84.6% (11/13); Fisher’s exact test p = 0.01) with the CMS classifier excluding the undetermined samples . Two separate assays  may be more appropriate if translation of CRC subtypes into the clinic is to be achieved.We additionally compared our NanoCRCA assay with the microarray-based CMS subtypes in this cohort. We classified the Montpellier cohort of 17 microarray gene expression profiles into CMS subtypes using the published CMS classifierles Fig. . FurtherWe also compared the performance of the NanoCRCA assay against classification using Affymetrix Human Genome U133 Plus 2.0 (HG-U133 Plus2) microarray profiles and the CRCA-38 classifier in this cohort Fig. . We clasSimilarly, we compared the CRCA-786 classification using microarray data to the results of NanoCRCA. The 786-gene signature-based classification yielded 29.4% (5/17) stem-like, 23.5% (4/17) each enterocyte and goblet-like, 11.8% (2/17) inflammatory and 5.9% (1/17) TA samples . In addition to our prior finding of high reproducibility of the assay in FFPE samples  and nCounter data, allowing for equivalent interpretation of results from both of these platform types. However, care must be taken to ensure sufficient cellularity of the samples and careful selection of the type of samples (fresh frozen vs. FFPE) prior to subtyping to avoid misclassification. Most importantly, our NanoCRCA is highly applicable to clinically available FFPE samples, potentially more so than even fresh frozen samples. Since multiple CRC clinical trials require low-cost, reproducible and rapid clinically implementable assays to prospectively validate CRC subtypes for subtype-specific studies, our NanoCRCA assay may potentially facilitate this process in the clinic using FFPE samples.Supplementary Information & FiguresSupplementary Table S1Supplementary Table S2Supplementary Table S3Supplementary Table S4Supplementary Table S5Supplementary Table S6Supplementary Table S7Supplementary Table S8Supplementary Table S9


High-throughput gene expression profiles have allowed discovery of potential biomarkers enabling early diagnosis, prognosis and developing individualized treatment. However, it remains a challenge to identify a set of reliable and reproducible biomarkers across various gene expression platforms and laboratories for single sample diagnosis and prognosis. We address this need with our Data-Driven Reference (DDR) approach, which employs stably expressed housekeeping genes as references to eliminate platform-specific biases and non-biological variabilities.MET or HER2-positive, and mutant KRAS, EGFR or PIK3CA) with smaller sets of biomarkers. We demonstrate on the three microarray datasets that our method is capable of identifying robust biomarkers for subgrouping medulloblastoma samples with data perturbation due to different microarray platforms. In addition to identifying the majority of subgroup-specific biomarkers in CodeSet of nanoString, some potential new biomarkers for subgrouping medulloblastoma were detected by our method.Our method identifies biomarkers with “built-in” features, and these features can be interpreted consistently regardless of profiling technology, which enable classification of single-sample independent of platforms. Validation with RNA-seq data of blood platelets shows that DDR achieves the superior performance in classification of six different tumor types as well as molecular target statuses (such as In this study, we present a simple, yet powerful data-driven method which contributes significantly to identification of robust cross-platform gene signature for disease classification of single-patient to facilitate precision medicine. In addition, our method provides a new strategy for transcriptome analysis. Identification of reliable and reproducible biomarkers can contribute to reveal patterns of disease heterogeneity. Recent advances in High-throughput sequencing (HTS) technology, such as microarray , 2 and RSince gene expression data are high-dimensional data, an important research aim in analysis of transcription profiles is the discovery of small subset of biomarkers containing the most discriminant information, also known as feature selection , for accTo address these challenges, we present a Data-Driven Reference (DDR) approach to identify robust cross-platform gene signature for classification of single-sample from various platforms. Our DDR algorithm consists of three main steps: 1) the stably expressed housekeeping genes are employed as references to create a contingency table for each gene using given gene expression dataset; 2) Fisher’s exact tests are applied in contingency tables to identify differentially expressed genes (DEGs) as potential biomarkers between two conditions; 3) the categories which the expression levels of biomarkers fall into based on selected reference genes serve as input to the classifier. The reference genes are the housekeeping genes whose expression values remain relatively constant across all samples from different conditions. The categories generated by stably expressed reference genes represent the relative positions of biomarkers, which have a consistent interpretation across gene expression platforms and eliminate sample-specific biases. We illustrate DDR’s utility through various evaluations and comparisons with gene signatures identified by existing methods. We demonstrate that DDR method contributes significantly to identification of robust cross-platform gene signature for disease classification of single-patient to facilitate precision medicine.STARD7-AS1, ZCCHC9, RBM14, HNRNPH1, and EIF4G2) from TCGA-BRCA RNA-Seq dataset were selected, so that log2-fold-changes between expressions of two consecutive reference genes were around 2  experimentally validated for clinical or functional relevance in growth and metastasis of TNBC [B3GNT5, PPP1R14C, RGMA, and HAPLN3) were also computationally selected as signature genes in TNBC[p-value  [p-value) was less than 0.01 in EdgeR and DESeq2 methods, or FDR  was less than 0.1 in DESeq and DDR methods. We measured the precision and recall of the identified DEGs using the DEGs from the datasets of 230 TCGA-BRCA samples (115 TNBC samples and randomly selected 115 other subtypes) and 118 TCGA-LUAD samples  as the gold standard. The precision and recall values in both datasets for different methods and different numbers of samples per group are illustrated in Fig. EdgeR methods reported high values for precision in both datasets across different sample sizes   uses linEdgeR methods were higher compared with the other methods in both datasets  of the tools in analysis of RNA-Seq data, we generated mock comparisons from two datasets: the first consisted of 115 triple-negative breast cancer samples by randomly dividing the samples into two non-overlapping groups (57 samples for one group and 58 samples for the other group) and second consisted of 59 normal samples from TCGA-LUAD dataset by randomly dividing the samples into two non-overlapping groups (29 samples for one group and 30 samples for the other group). The median FDRs of both EdgeR (FDR <0.01) or DESeq2 . The use of DDR and DESeq2 (or EdgeR) algorithms achieved higher overlap rate in DEG results from BRCA dataset. DESeq generated DEG list overlapped poorly with that from DDR (<52%) in both datasets. It is no surprise that the highest overlap percentages were observed between DESeq and DESeq2 DEG lists or between EdgeR_EXACT and EdgeR_GLM DEG lists.Finally, we compared the overlaps of DEGs identified by the different methods through computing overlap coefficient (Szymkiewicz-Simpson coefficient) . The ovelimma by using prostate cancer microarray data. Similarly as in analysis of RNA-Seq data, we used 240 samples (randomly selected 120 samples from each group) as the gold standard and measured the precision   was selected as biomarkers for classification based on adjusted p-value (< 1×10−60) and Expression Distance (ED >2.5). Combination of FDR and ED for selection of signature genes enables not only identifying genes containing most discriminant information but also leading to more reproducible biomarkers. Fisher’s exact tests were employed to identify top DEGs as potential biomarkers, which come with the most differentially relative positions in comparison to reference genes, named as “built-in” features. Since the positions are relative, they can be robust features and be used for single sample classification. These “built-in” features . DDR was applied to TCGA-BRCA RNA-Seq dataset to identify a list of ranked DEGs   and emplKRAS-mut vs KRAS-wt in CRC, HBC, NSCLC, and PAAD patients, EGFR-mut vs EGFR-wt in NSCLC patients, MET+ vs MET- in NSCLC patients, PIK3CA-mut vs PIK3CA-wt in BRCA, HER2+ vs HER2- in BRCA, as well as triple-negative breast cancer. The number of biomarkers , and then medulloblastoma samples were subgrouped by measuring the expression level of 22 subgroup-specific genes [C1orf127, ZNF347, WDR70 and HNRNPK)  for each subgroup were identified by performing DDR for each subgroup against the other subgroups , SHH , G3 , G4 , and their expression heatmap is presented in Fig. Scikit-learn [Medulloblastoma (MB) is the most common malignant brain tumor in children and represents approximately 20% of childhood brain tumors . Transcric genes , 43. Heric genes ), and thit-learn  using SVNWD2 from training dataset (GSE37418) was not available in GSE21140 datasets, the other 19 signature genes  samples from GSE21140 were assigned to appropriate subgroups (Table NWD2 was not included) were used for subgrouping. Using OneVsRest SVM classifier trained on GSE37418 dataset, 51/51 (100%) SHH samples, 33/46 (72%) G3 samples and 185/188 (98%) G4 samples were correctly classified to appropriate subgroups, respectively, which resulted in accurate classification of 94% in GSE37382 dataset  and G4   for classification of diseases using gene expression profiling. However, these approaches posed serious reproducibility challenge when classifying cross-platform samples individually due to technological and platform biases. To overcome this limitation, we present a simple, yet powerful data-driven method that does not require distribution-based modeling for gene expression analysis and it identifies potential biomarker genes with “built-in” features (categorized tiers based on reference genes) for the classification of single-sample from distinct platforms.edgeR [DESeq [DESeq2 [limma [DESeq2 (and edgeR) were higher (>80%). These results demonstrated that DDR retains information of expression values well on analysis of DEGs. More importantly, DDR controlled the number of false positives better, which guaranteed identifying reliable biomarkers. Unlike edgeR, DESeq, DESeq2 and limma, DDR is a data-driven non-parametric method which requires fewer assumption about data and is robust to outliers. As a result, it deals with cross-platform profiling gene expression and technical bias well. The utilization of reference genes at different expression levels effectively combines p-value and fold-change to identify reliable DEGs (biomarkers). In addition, employing logarithmic expression levels when selecting reference genes provides wiggle room to deal with overdispersion in RNA-Seq data. Our method provides a better methodological advantage to identify reliable and reproducible potential biomarkers from various expression profiling platforms. DDR can also be employed to detect DEGs among multiple conditions by designing appropriate contingency tables.The huge amount of gene expression profiling data has been accumulated over the past decades and deposited in public databases such as GEO  and TCGAedgeR , DESeq [R [DESeq  and DESe [DESeq2  on two R2 [limma  on a micTCGA is a comprehensive molecular profiling project that compiles clinical and genomic data from samples of different human tumor types , and proedgeR using RNA-Seq profiling of TEP to yield DEGs for pinpointing the location of primary tumor with 71% accuracy across six types of tumors and distinguish cancer patients with different molecular subtypes as well. In this study, we applied the DDR method to identify potential biomarkers using RNA-Seq data of TEP and identify multiclass cancer and molecular subclass. We were successful in achieving comparable classification performance with fewer biomarkers compared with results reported by Best et al. [MET and HER2-positive, or EGFR, PIK3CA and KRAS mutations) were identified by DDR as compared to biomarker sets from Best et al., which makes it more practical in blood-based cancer diagnostics and therapeutic target identification. Selection of smaller subset of biomarkers can reduce over-fitting and computational complexity by removing redundant features.Tumor-educated platelets (TEP) is able to serve as potential noninvasive source of tumor-related RNA biomarkers , 39, 49.t et al. . Much smDDR employs Fisher’s exact test, a non-parametric method, to analyze gene expression profiling, so it could be equally applied in the analysis of microarray data. To evaluate the performance of DDR in identification of signature genes and classification of microarray data, we applied DDR to a published microarray dataset of medulloblastoma GSE37418) and derived 20 signature genes for medulloblastoma subgroups. Among 20 signature genes, 12 genes overlay NanoString codeset  from a c8 and derThe main technical novelty of this work is the combination of data-driven reference genes with non-parametric Fisher’s exact test for discovering potential biomarkers. This not only allowed us to identify differentially expressed genes but also help to extract corresponding “built-in” features based on reference genes. One of the exciting outcomes of these “built-in” features is their reliability and reproducibility in classifying disease samples involved with technical bias and cross-platform, which allow us to analyze single sample of disease. This study has shown that DDR can be a promising tool for the identification of biomarkers for precision medicine. And some expression assay (e.g. quantitative PCR) based on these biomarkers and reference genes can be easily developed for diagnosis, prognosis and developing individualized treatment in the future. Finally, although this study has focused on cancer classification, it could be equally useful in classification of other diseases such as Parkinson or Alzheimer’s. In conclusion, we have developed a novel, reliable and reproducible data-driven method for identification of potential biomarkers for single-sample classification.RNA-Seq from The Cancer Genome Atlas (TCGA). All RNA-Seq read count data from TCGA lung adenocarcinoma (LUAD) project (n=594) were retrieved using the GDC Data Transfer Tool [RNA-Seq from Gene Expression Omnibus (GEO). RPKM normalized RNA-Seq data of LUAD were downloaded from the Gene Expression Omnibus  [Microarray data from GEO. We retrieved mRNA expression microarray data set from GEO under the accession number GSE62872 [RNA-Seq of Tumor-Educated Platelets. The gene expression profiles of 285 blood platelet samples were downloaded from GEO under the accession number GSE68086 [Microarray data of Medulloblastoma. All microarray data are downloaded from the GEO database under assession number GSE37418 [fer Tool . Both LUSE40419) . The datGSE62872  which waGSE62872 .RNA-Seq GSE68086 . The samGSE37418 , GSE2114GSE37418 , respectaffy were used as input to DDR. Before running DDR, the input data were re-arranged to group the samples under the different conditions.When analyzing RNA-Seq data, DDR expects count data as input. The count data can be obtained through TopHat-HTSeq pipeline , 55. Foret al [,i by dividing the standard deviation (σi) of its CPM values by the mean (μi): Reference genes from RNA-Seq data were identified using a data-driven approach similar to that developed by Hoang et al . The noret al ,57, and k, so that j ≤n) was assigned to Tier j in sample k, here The genes with the lowest product scores and those that were also included in the list of human housekeeping genes  were selmentclass2pt{minimn reference genes were selected from previous step, n+1 tiers were generated based on the expression levels of these reference genes. The relative expression position (tier) of each gene in comparison to the reference genes from the same sample was obtained, and then frequency counts of the tiers were obtained across the samples from the same condition. These frequency counts were filled in the cells of a contingency table (each column represents the relative position (tier) of each gene in comparison to the reference genes and each row represents the different conditions. So, a C×T contingency table was created for each gene, here C is the number of conditions and T is the number of tiers, to display the numbers of samples from a particular condition in which that gene was assigned to a particular tier . In addition to adjusted p-values, we defined expression distance (ED) for each gene to describe a quantity change of the gene expression between conditions. The ED can be used to select up- and down-regulated genes.When DESeq [DESeq2 [EdgeR [limma [To assess how well DDR performs for identification of DEGs in comparison to the current methods  and high EDs. The feature tables consisting of tiers to which selected biomarkers belong for samples served as input to the classifiers. An example of feature table for classifying TNBC and non-TNBC is shown in Additional file Small subset of biomarkers for classification was obtained from DEGs by filtering on the basis of low adjusted Scikit-learn [Scikit-learn to assess the performance of our classification model.Several machine learning classifiers from it-learn  Python lAdditional file 1 The file contains eight figures  and four tables (Tables S1-S4) as supplementary results.Additional file 2 The table contains reference genes and DEGs identified by DDR in TCGA-BRCA RNA-Seq data.Additional file 3 The table contains reference genes and DEGs identified by DDR in TCGA-LUAD RNA-Seq data.Additional file 4 The table contains reference genes and DEGs identified by DDR in prostate cancer microarray data from GEO (Accession: GSE62872).Additional file 5 The table contains “built-in” features of four selected biomarkers from TCGA-BRCA RNA-Seq data and microarray data from GEO (Accession: GSE27447) as input to classifiers.Additional file 6 The table contains DEGs identified by DDR in RNA-Seq data of Tumor-Educated Platelets from GEO (Accession: GSE68086).Additional file 7 The table contains “built-in” features from Tumor-Educated Platelets (Accession: GSE68086) as input to classifiers.Additional file 8 The table contains DEGs identified by DDR in microarray datasets of medulloblastoma from GEO .Additional file 9 The table contains “built-in” features from medulloblastoma  as input to classifiers.


Many biological data acquisition platforms suffer from inadvertent inclusion of biologically irrelevant variance in analyzed data, collectively termed batch effects. Batch effects can lead to difficulties in downstream analysis by lowering the power to detect biologically interesting differences and can in certain instances lead to false discoveries. They are especially troublesome in predictive modelling where samples in training sets and test sets are often completely correlated with batches. In this article, we present BARA, a normalization method for adjusting batch effects in predictive modelling. BARA utilizes a few reference samples to adjust for batch effects in a compressed data space spanned by the training set. We evaluate BARA using a collection of publicly available datasets and three different prediction models, and compare its performance to already existing methods developed for similar purposes. The results show that data normalized with BARA generates high and consistent prediction performances. Further, they suggest that BARA produces reliable performances independent of the examined classifiers. We therefore conclude that BARA has great potential to facilitate the development of predictive assays where test sets and training sets are correlated with batch. Data acquisition techniques designed to quantify biological signals from gene- or protein expression are often associated with batch effects. The problem with batch effects is that it leads to the inclusion of biologically irrelevant variance in the obtained data, which can lower the power of subsequent analyses or lead to false discoveries –4. The vSome actions have been shown to reduce the impact of batch effects. One such action is to carefully design experiments to minimize the correlation between possible sources of technical variance and known biological factors. However, this action is not possible for all types of experiments. For predictive modelling, for example, the correlation between biological factors and batches cannot be eliminated. This is due to the inherent nature of these experiments, where fixed training sets are often used to infer parameter values used to predict subsequently acquired test sets. This leads to total confoundment between batches and samples in the test sets, which can result in poor predictive performances on test sets . AnotherBatch Adjustment by Reference Alignment (BARA) to adjust for batch effects in predictive modelling. The method has the advantage that only a few reference samples are necessary to perform batch corrections. Also, rather than attempting to clean the data by removing batch effects from both training set and test sets, BARA aims to transform the test set to make it more similar to the training set. BARA performs the adjustment in a compressed data space spanned by the training set, thereby alleviating the number of necessary batch estimates that needs to be performed. We test the BARA method on a collection of 25 publicly available datasets and show that BARA consistently aids the classifier to achieve high prediction performances. We further show that the performance of BARA is better or comparable to the performance of existing methods on the examined datasets. By reducing the negative impact of batch effects, the prediction performances observed with BARA can facilitate the development of predictive assays.In this paper, we introduce https://github.com/gradinetal2018/BARA.The R software environment was used to perform the analyses presented in this paper . Figuresgender/sex.25 datasets compiled by Hornung et. al.  were dowCross-study prediction performances were used to evaluate the performance of BARA and to compare it to existing normalization methods. The normalization methods included in this analysis were; BARA, ComBat , FABatchTo assess the performance of the BARA algorithm as the number of reference samples was varied, the cross-study prediction approach described above was repeated. The performance estimation was repeated 6 times, where the number of utilized reference samples was varied from 1 sample to 6 samples. The predictive performances were summarized as described above.The BARA algorithm was created specifically for predictive modelling, where a fixed training set is used to classify test samples possibly affected by batch effects. The training set is used to identify a set of directions that captures the largest part of the variance in the data, using singular value decomposition (SVD). This step allows the data to be compressed into a lower dimensional space, which reduces the number of necessary batch estimates, and simultaneously decreases the complexity of the data.k, is an adjustable parameter that can be set by using a predetermined value, estimated with for example cross-validation, or determined by setting an acceptable loss of variance in the training set, for example 10%. The centered training data is then multiplied by the first k columns of the right singular vector to obtain a transformed training set.The training dataset, X, contains m samples in rows and n variables in columns. Each variable in X is centered by its mean value, and the matrix is decomposed using SVD to identify the directions where the batch adjustment is performed.The test dataset, Z, contains p samples in rows and n variables. The variables are first adjusted by subtracting the mean values of the training data, and is then projected onto the identified directions.j, is estimated for all retained dimensions, using the reference samples present in both the training set and the test set. For example, the adjustment factor of dimension j is estimated by comparing the mean value of the reference samples in the training set for dimension j, to the mean value of the reference samples in the test set for dimension j.A batch adjustment factor, aWhere To achieve a level of expression to what was originally observed, the mean values estimated from the training set are finally added to the reconstructed data.k, and the predictions are made on the adjusted test set, Zk.The predictions are then performed as normal, where the model is built from the compressed training dataset, XThe following methods were used through their implementations in the R-package bapred; ComBat, FAbatch, fSVA exact, mean centering, ratio A and standardization. For FAbatch, the default values of the parameters were used, i.e. the number of factors to estimate for each batch was left unspecified, the preliminary probabilities were estimated using leave-one-out cross-validation, maximum number of iterations were 100, and the maximum number of factors were 12. For fSVA exact, the algorithm parameter was changed to correspond to the exact algorithm instead of the fast, while the default values were used for the remaining parameters. For the other methods implemented in the bapred package, no additional parameter values could be specified.The two reference-based methods, reference mean centering and reference ratio A, where implemented in R. Reference mean centering subtracted each batch’s genes by the mean expression of its reference samples, and reference ratio A scaled each batch’s genes expression by the mean expression of the reference samples.BARA was implemented by specifying the loss parameter as a criterion for selecting the number of dimensions to retain. The loss parameter was set to 10%. Thus, at most 10% of the variance in the training data was lost in the normalization.Three types of prediction models were used to assess the performance of the normalization methods in the cross-study analysis. The prediction models were; random forest, kNN and SVM with linear kernel. The prediction models were implemented using the R-packages randomForest, class and e1071 , 22, 26.○Number of nearest neighbors: 1, 2, 3, 4, 5, 6, 7, 8, 9kNN○Mtry: 5, 7, 9, 10, 11, 13, 15, 17○Ntree: 500, 1000, 1500, 2000, 2500, 3000Random Forest○Cost: 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50SVMTo assess the performance of BARA and to compare it to existing normalization methods, cross-study predictions were examined. Because the acquired datasets originated from separate studies, the biological annotation used for classification was sex/gender. To assess the different normalization methods, each dataset was iteratively used as training set to define a prediction model. Batch effects between the training sets and the test sets were adjusted with the examined normalization methods. Normalized test sets were classified with the trained prediction models and MCCs were calculated to estimate the prediction performances. The resulting MCCs for the normalized data and for the unnormalized data on the 3 different prediction models are shown in Figs Because all information used to adjust for batch effects in the BARA method is estimated from the reference samples, the performance of the method as the number of reference samples was varied was examined. The number of reference samples was varied from 1 sample to 6 samples, and the cross-study prediction approach described above was repeated. Again, the performances were summarized by calculating the mean MCC for each run. The performance for each repeat can be seen in Batch effects are a widespread problem that exist in most biological data acquisition platforms and hinder the development and implementation of promising biomarker signatures –5. BatchOnly a few reference samples are required by BARA; three were used in the comparison described above and BARA was also found to achieve robust performance metrics when only a single reference sample was used. However, the optimal number of reference samples could generally be assumed to be based on a trade-off between costs and accuracy. Because the reference samples are used to estimate mean adjustments for every batch in a compressed data space, additional reference samples should provide more certain estimates of the mean values. This was also observed when the number of reference samples used by BARA was varied, see BARA estimates the batch adjustments in a compressed data space spanned by the training set. The compression is calculated with SVD and is thus a linear combination of the original variables along the directions of maximum variance. SVD was chosen because we hypothesized that it would be suitable for many datasets associated with predictive modelling where biomarker signatures have been identified to optimize prediction performances, which suggests that low-dimensional representations of the data exist that captures large fractions of the important variance. Further, SVD is a well-known operation that offers a convenient way of compressing data, performing batch adjustment and reconstructing the original variables. Because SVD is used in BARA, multiple levels of compression can be selected for a dataset depending on the number of directions that are retained. In the cross-study analyses described above, a maximum of 10% of the variance in the training set was considered an acceptable loss, and the smallest number of dimensions satisfying this condition was selected in the normalization steps. However, other approaches for selecting an optimal number of dimensions to retain in a specific dataset could be pursued. For example, cross-validation performances could be compared as the number of dimensions retained are varied, or an external validation set affected by batch effects could aid the selection by better mimicking an actual prediction scenario.The ability of BARA to restore the predictive performances in datasets suffering from batch effects was assessed by cross-study predictions, and the obtained performances were compared to those obtained using existing normalization methods or no normalization. Due to the difficulty in procuring public datasets containing training sets suitable for predictive modelling where external test sets affected by batch effects exist, a collection of datasets previously compiled  were useIn conclusion, we have introduced a novel method to adjust for batch effects in predictive modelling and compared it to already existing methods. We show that BARA improves the prediction performances in the examined datasets compared to applying no normalization. Further, the BARA-normalized datasets achieved higher or comparable prediction performances compared to datasets normalized with the other examined methods. These results suggest that BARA can be considered a useful tool to reduce the negative impact of batch effect in predictive modelling.


Large-scale immune monitoring experiments  are a promising direction for biomarker discovery and responder stratification in immunotherapy. Mass cytometry is one of the tools in the immune monitoring arsenal. We propose a standardized workflow for the acquisition and analysis of large-scale mass cytometry experiments. The workflow includes two-tiered barcoding, a broad lyophilized panel, and the incorporation of a fully automated, cloud-based analysis platform. We applied the workflow to a large antibody staining screen using the LEGENDScreen kit, resulting in single-cell data for 350 antibodies over 71 profiling subsets. The screen recapitulates many known trends in the immune system and reveals potential markers for delineating MAIT cells. Additionally, we examine the effect of fixation on staining intensity and identify several markers where fixation leads to either gain or loss of signal. The standardized workflow can be seamlessly integrated into existing trials. Finally, the antibody staining data set is available as an online resource for researchers who are designing mass cytometry experiments in suspension and tissue. Immune monitoring (IM) is a systems biology approach for the quantitative evaluation of the state of the immune system , 2. Chande novo with no clear guidance on best practices.Any large-scale study will introduce challenges such as sample quality control, batch effects, and inter-operator variability. There are a plethora of methods to address potential data quality issues in mass cytometry. These include the incorporation of normalization beads into the sample , reductiThe situation is even more problematic in the computational biology arena. Numerous mass cytometry analysis methods have been published. These can be broadly classified into one of two categories. Clustering algorithms, such as SPADE , PhenoGrFinally, the insights gained from mass cytometry ultimately depend on the antibodies used in a given staining panel, and as with any other antibody-guided assay, antibody selection is a central component of mass cytometry experiment design. While there is some consensus on appropriate markers to identify major circulating immune subsets , much ofhttps://www.antibodystainingdataset.com. This represents a powerful resource that allows researchers to quickly identify potential markers for inclusion in novel mass cytometry studies. Finally, the overall workflow represents a systematic framework that can readily by applied for performing IM in large experiments such as clinical trials.In order to address the above, we developed a streamlined mass cytometry pipeline that combines a lyophilized antibody panel, two-tier barcoding, efficient batched sample acquisition and a novel cloud-based analytics service. We applied this efficient sample and data processing pipeline to screen the expression of 326 antibodies across all major peripheral blood mononuclear cell (PBMC) subsets from multiple donors on both fresh and fixed cells. This represents one of the largest mass cytometry data sets to date, with approximately 63 million events acquired over a month of operation. The workflow incorporates multiple mechanisms that address and monitor intra- and inter-sample variability, quality control, standardization and automation. The result is a comprehensive antibody staining data set, which screens marker expression in every major immune subset on a single-cell level. These antibody expression data have been made available as an interactive companion website at Peripheral blood mononuclear cells (PBMCs) for the primary LEGENDScreen experiment were isolated by Ficoll gradient centrifugation from leukapheresis products derived from 3 independent de-identified donors (New York Blood Center). Additional validation experiments used blood collected from consented healthy donors under an existing IRB protocol at the HIMC. For the primary screen experiment, approximately 120 million cells from each donor were incubated for 20 min at 37°C in RPMI media containing 10% FBS, 1 μM Rh103 to label dead cells and 50 μM IdU to label actively cycling cells. The samples were then washed, Fc-blocked  and stained for 30 min on ice with a lyophilized core antibody cocktail comprised of markers to allow identification of all major immune subsets . All theAfter staining, the samples were then divided into two aliquots, one of which was fixed with freshly diluted 1.6% formaldehyde in PBS for 20 min, while the other was left untreated. Each of the 6 samples was then barcoded using a combinatorial CD45-based barcoding scheme , allowinSamples were thawed immediately prior to acquisition, washed once in PBS, once in CAS buffer (Fluidigm) and then resuspended in CAS buffer containing a 1/20 dilution of EQ normalization beads (Fluidigm). Following routine instrument tuning and optimization, the samples were run at an acquisition rate of <300 events per second on a Helios mass cytometer (Fluidigm) modified with a wide-bore injector (Fluidigm). Upon completion of the acquisition, FCS files associated with each barcoded batch of wells were concatenated and normalized using the bead-based normalization algorithm in the Fluidigm software resulting in 38 FCS files.FCS files were uploaded to the Astrolabe Cytometry Platform  where transformation, debarcoding, cleaning, labeling, and unsupervised clustering was done. Data was transformed using arcsinh with a cofactor of 5 and the marker intensities presented in the paper are all after transformation. Batches were debarcoded using the Ek'Balam algorithm (see below), resulting in 2,232 individual samples corresponding to one  combination. Data from 12 antibody wells were excluded due insufficient cell recovery or ambiguous barcoding resulting from a known pipetting errors during sample preparation, resulting in 2,160 samples. For batches 23, 25, and 34, between 50 and 75% of events were removed due to loss of stability, as described in the main text.The individual samples were then labeled using the Ek'balam algorithm . Each ceEk'Balam is a hierarchy-based algorithm for labeling cell subsets which combines the strength of a knowledge-based gating strategy with unbiased clustering. It receives a user-defined subset hierarchy which details gating rules such as “Cells which are CD3+ are T Cells.” Subsets can branch through additional rules, for example, “T Cells which are CD4+ are CD4+ T Cells.” The hierarchy is organized into levels which correspond to parallel steps when gating. For example, the first level could include “CD3+ are T Cells,” “CD19+ are B Cells,” and “CD33+ are Myeloids.” Ek'Balam then iterates over the levels. At each iteration, the data is clustered with FlowSOM , using oProfiling refers to a variation of unsupervised clustering using the FlowSOM algorithm. The variant differs from classic FlowSOM in two significant aspects. One, each cell subset is clustered separately. This guarantees that the output will not include biologically irrelevant clusters that combine multiple cell subsets. Two, the clusters are labeled according to the markers that differentiate between them the most, according to the MCC. The labeling makes the output more accessible to the researcher by providing an initial intuition about the differences between the clusters. A formal definition of the profiling algorithm is provided in the supplement.The following metrics were employed when comparing the computational debarcoding and labeling results to manual methods. Metrics were calculated for each class separately, where class is either a barcode (for debarcoding) or a cell subset (for labeling). The class was set as the target and all other classes as not-target. In all cases, the manual method is assumed to be the correct solution.TP, FP, TN, and FN are true positive, false positive, true negative, and false negative, respectively.Precision is the frequency of correctly classified target events out of all events classified as target, or TP / (TP + FP).Recall is the frequency of correctly classified target events out of all target events, or TP/(TP + FN).The F1 score is the harmonic mean of precision and recall, orThe Matthews Correlation Coefficient (MCC) is the correlation coefficient between the computational and manual classification, orThe average overlap frequency is a metric of staining and clustering quality of a given marker . It assuX− is the values of all events in the negative modality, X+ is the values of all events in the positive modality, where Scaled2AOF for each  pair:Given a set of samples, we can extend the AOF into a sample quality score by calculating the Quality2AOF for each sample:where m indexes over markers and i indexes over samples, and then calculating the For each , the percent of positive events is the percent of events whose intensity is greater than the 99th percentile of all events in the Blank LEGENDScreen well  is compared to control (fresh samples). The six patient x treatment combinations were barcoded and pooled using a live cell-compatible doublet-free barcoding strategy leveraging CD45 antibodies conjugated to 4 distinct isotopes. This barcode approach streamlines sample processing and minimizes potential variability due to acquiring different patients or treatments at different times. The isotopes used for barcoding were specifically chosen to ensure that potential spillover due to isotopic impurities or oxide formation from these barcoding channels would not influence any of the other antibody channels being measured in this experiment. Next, the samples were evenly distributed across each of the 372 wells of a LEGENDScreen kit, each of which includes a PE-conjugated antibody against a distinct epitope. Following this with a metal-conjugated anti-PE antibody enabled the measurement of a comprehensive set of surface markers across all the cell subsets identified by the broad lyophilized panel. Finally, to streamline data acquisition, sets of 10 wells were further barcoded and combined using a combinatorial strategy leveraging five palladium channels.Figure 3C). The Astrolabe analysis took 24 h, and the platform's “Analysis” export was employed in all follow-up analyses.The resulting 38 batched samples were then acquired using a Helios mass cytometer . AcquisiThe antibody staining data set involves a high number of samples, complex experiment design, a long acquisition period, and advanced computational analysis, any of which could potentially introduce variability or other artifacts. Several tests inspect the various stages of the experiment . First aThe starting point for the data set was blood from three healthy donors. After the fixed vs. fresh treatment and the introduction of the kit's antibodies, each of these individuals leads to several hundred different samples. However, the individual donor immune profile across each set of samples are expected to be identical and therefore the acquired data should be highly comparable. This is reflected in the principal component analysis (PCA) map over the sample cell subset frequencies . The samWe further applied Average Overlap Frequency (AOF) as a metric to evaluate individual marker staining quality across all sample batches . This QCExcept for the batch effects identified by the AOF QC, the data set was consistent across cell subsets and marker intensities . For fouThe combination of the above quality control measures highlights the overall robustness of the antibody staining data set. The overall staining data were cohesive for each donor, and for each cell subset across donors, and specific acquisition issues were identified and addressed using automated QC metrics.The Astrolabe platform automatically labeled canonical immune cell subsets . As withThe marker intensity profiles for each of the subsets labeled by the platform largely follow the consensus HIPC definitions [The discovery of novel cell subsets defined by previously unappreciated marker expression patterns is one of the most exciting promises of high-complexity cytometry such as mass cytometry. While cell subset labeling follows established trends, unsupervised clustering has the potential to unearth previously unknown signals. Astrolabe includes a profiling step, where each defined cell subset is clustered separately . The numWith 350 measured antibodies over 71 profiling subsets, the antibody staining data set is a rich source of information about expected expression patterns in a healthy immune system. In order to provide an initial view into the full expression dataset, we calculated two metrics for each profiling subset and antibody combination . The firFocusing on any specific section of the heat map reveals a plethora of relevant patterns. The top of the map is populated with well-established markers  such as Lower sections of the heat map allow investigation of many surface markers that appear less frequently in the scientific literature . Notablyp-value is lower than 2.2–16). Together, these tests show that the trends seen in this data set are generalizable. With that said, unlike the main data set, CD5 is uniformly expressed across all T Cell subsets in the validation data  cells . MAIT ceTwo of these trends overlap between the two cell subsets: an increase in CD26 and a decrease in CD49d. CD26 has been previously associated with MAIT cells . When exCD192 (CCR2) was differentially expressed between CD161hi and CD161low CM cells, with a x3.6 average fold increase in intensity in the CD161hi subset , left. IOne of the limitations of this screening approach is that each of the antibodies is profiled independently, which precludes co-expression analyses of markers in the screen. To validate and further explore the co-expression patterns of the markers identified in the screen, we independently stained a healthy donor PBMC sample with a panel incorporating several of the differentially expressed markers identified in the screen together with Va7.2 TCR to definitively identify MAIT cells . tSNE anFormaldehyde fixation is a useful approach to preserve cell samples but has been associated with changes in cell surface epitopes and marker expression profiles , 48, Sup. HoweverThe antibody staining data set includes two conditions for each donor and antibody samples: one stained fresh and stained following fixation with 1.6% formaldehyde. Two hundred fifty-five of the LEGENDScreen markers have cells whose intensity is higher than the blank threshold. For each of these markers, we calculated the ratio between median expression in each of the conditions over all cell subsets . We arbiSixty-five of the markers have a 2-fold or more increase in fixed samples relative to fresh . In otheExamining the ratio between the medians enables a broad survey of all antibodies over all subsets. However, it ignores the single-cell nature of the data. Closer examination of several marker intensity distributions reveals that when the ratio is around zero, the underlying distribution is usually maintained from fresh to fixed as well . When maThe LEGENDScreen kit includes antibodies conjugated to PE which are then measured by mass cytometry using an anti-PE secondary. It is possible that the effects of fixation observed here are not due to effects on the underlying antibody, but rather due to a more complex interaction that potentially includes the marker antibody, PE, and anti-PE. We therefore performed a validation experiment where seven of the gain or loss markers were incorporated into the mass cytometry panel . For theCytometry experiment design can be a daunting task due to the high number of variables that needs to be considered. There are many factors that could influence results in unknown ways, especially when employing a method such as fixation that has the potential to perturb the chemistry and kinetics underlying the assay. This antibody staining data set represents an accessible resource to identify and anticipate such potential effects.We present a standardized workflow for the acquisition and analysis of large-scale immune monitoring studies using mass cytometry. The workflow incorporates several established experimental techniques in order to reduce signal variation within samples, across samples, and across operators. One, it utilizes a lyophilized core antibody panel that allows clear identification of major compartments of the immune system and provides higher resolution into T Cell, B Cell, and other subsets. Lyophilization streamlines sample processing and eliminates the variability inherent in pipetting small volumes from a large numbers of individual antibody vials. Two, a two-tiered barcoding scheme assures that all donors and treatments are acquired together and that samples are organized into batches. This reduces the technical variation associated with the instrument and its operation. Three, a fully automated cloud-based analytics platform (Astrolabe) runs the same quality control, data cleaning, cell subset labeling, and unsupervised clustering over the entire data set. Taken together, the workflow provides a flexible framework that can be easily adapted to clinical trial immune monitoring or other large-scale experiments and greatly improve the quality, reproducibility, robustness and utility of mass cytometry data.We leveraged this standardized workflow as part of a comprehensive screen to establish the expression of 350 surface markers across all major circulating immune subsets at single cell resolution. Acquisition of the entire expression dataset across three donors required more than a month of Helios operation and culminated in over 60 million events; one of the largest single mass cytometry datasets recorded to date.Several quality control approaches were included in order to ensure the accuracy and quality of the antibody staining dataset. First, we employed a two-tier barcoding approach to minimize technical variability in performing the screen. The barcoded samples were deconvolved using an automated debarcoding approach that was directly compared and shown to perform comparably to manual debarcoding. Second, we used average overlap frequency (AOF) as a metric to evaluate the consistency of individual marker staining quality across all samples, which allows us to identify and address acquisition batch effects. Third, we used an automated approach to identify and label cell subsets, the accuracy of which was validated against manual gating of each of the analogous subsets, demonstrating high overlap and consistency between these approaches. Fifth, we performed the screen using three independent donor blood samples to allow for an evaluation of the biological reproducibility of individual marker expression profiles, and each donor presented a consistent and distinct cell subset profile across the entire experiment with both the frequencies of the major immune compartments and the intensities of their canonical markers showing low variability across the entire acquisition period. Finally, the reproducibility of the antibody expression profiles in our primary screen were further validated using a second independent screen performed using an additional donor. Taken together, these steps highlight the fidelity of the antibody staining resource. However, it is still important to note the limitations of this data set as a high-throughput screen; any findings require independent follow-up to confirm whether the reported expression patterns truly reflect hitherto unknown phenotypic diversity or may reflect specific biological or technical aspects of this screen. As an illustration of this approach, we used the screen to identify potential markers to characterize CD161+ MAIT cells, and then performed an independent experiment where we incorporated these markers as part of a single CyTOF panel. This allowed us to both independently validated the markers identified the screen and to further explore their co-expression patterns, confirming that CD161hi MAIT cells can be further characterized as being CD26hi, CD192hi, CD183low, and CD57low.In addition to screening marker expression patterns on fresh cells, we also introduced formaldehyde fixation as a treatment, thoroughly examining the influence that this standard perturbation could have on surface marker staining. When examining the effect of fixation on marker expression patterns, 173 out of 255 expressed markers had no change in their intensity. Sixty-five gained some signal from fresh to fixed. We hypothesize that this gain is an artifact of the fixation protocol rather than a novel biological signal since it was subset agnostic and only affected some of the cells in each profiling subset. Seventeen markers lost their existing signal after fixation. In almost all cases, the loss of signal affected all expressing subsets. The one exception was CD22, where one expressing subset (basophils) lost the signal, while another (B Cells) did not. It has previously been suggested that the CD22 epitope on basophils is conformationally distinct from that on B cells . Our dathttps://www.antibodystainingdataset.com. We included two aggregate statistics for each  combination: median anti-PE intensity and percent positive cells . In addition to interacting with the dataset through heat maps, survey aggregate statistics for their marker(s) and cell subset(s) of choice, the website allows investigators to delve deeper into the single-cell resolution and the relevant distributions. Overall, this dataset represents an accessible and unbiased resource for assessing potential expression of various markers over a large range of immune subsets in healthy individuals and surveying the statistics in the entire data set reveals intriguing signals for potential expression of less-studied markers. This study offers a valuable new resource to aid in the design of high dimensional antibody panels for immune monitoring studies, and further offers a template for a robust experimental workflow incorporating several components to ensure the accuracy and robustness of data generated using mass cytometry technology.The overall antibody staining data set is a powerful asset for immunologists seeking to investigate the immune system through the lens of less-explored markers and develop antibody panels to focus on specific cell subsets. To maximize the utility of this versatile qualitative resource, these results are fully accessible through an interactive website at https://flowrepository.org/id/FR-FCM-Z23S) and on ImmPort (https://www.immport.org) upon manuscript publication.All datasets acquired in the course of this study will be available on FlowRepository (EA contributed to experiment design, analysis, and writing the manuscript. BL, PB, and XG contributed to sample acquisition and analysis. MG contributed to analysis. MM contributed to experiment design and writing the manuscript. AR contributed to experiment design, sample acquisition and analysis, and writing the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Colorectal cancer is a heterogenous and mostly sporadic disease, the development of which is associated with microbial dysbiosis. Recent advances in subtype classification have successfully stratified the disease using molecular profiling. To understand potential relationships between molecular mechanisms differentiating the subtypes of colorectal cancer and composition of gut microbial community, we classified a set of 34 tumour samples into molecular subtypes using RNA-sequencing gene expression profiles and determined relative abundances of bacterial taxonomic groups. To identify bacterial community composition, 16S rRNA amplicon metabarcoding was used as well as whole genome metagenomics of the non-human part of RNA-sequencing data. The generated data expands the collection of the data sources related to the disease and connects molecular aspects of the cancer with environmental impact of microbial community.  Most cases of CRC are sporadic with no known genetic link. Environmental factors are therefore likely to play a critical role in the development of the disease, and a key characteristic of the colon is that it houses the largest proportion of the human microbiome, suggesting that this might play a role in causing CRC. Recent data points to the importance of the microbial communities in the gut, the microbiome, and possible links to the development of CRC5. If this is the case, understanding the role of the microbiome in CRC will have profound effects on cancer rates, since it is potentially relatively easily to manipulate, using diet, pre- and probiotics and faecal transplants9. However, despite the intense interest in the field and increasing evidence pointing to a role for the microbiome in CRC, convincing connections with clinical parameters and outcome are rarely seen.Colorectal cancer (CRC) is one of the most common types of cancer worldwide, in terms of both incidence and mortality13.CRC is a highly heterogeneous disease, with varying clinical outcomes, response to therapy, and morphological features, and molecular subtyping systems based on CpG-island methylation, microsatellite instability and gene mutations have shown strong associations with outcome and response to therapy in CRC14. We stratified a set of CRC tumour samples into CMS according to their gene expression profiles15 and assessed differences in bacterial communities among CMS. The gene expression profiles were generated using RNA sequencing, and 16S rRNA metabarcoding as well as metagenomic analysis of non-human portion of the RNA sequencing data were employed for bacterial taxa quantification. We analysed the enrichment/depletion of bacterial species in one subtype compared to the other subtypes and showed enrichment of certain oral bacteria associated with CMS, which was validated using targeted quantitative PCR.Contrary to other microbiome studies, where CRC is treated as a single disease entity, we focused on the association between Consensus Molecular Subtypes (CMS) of colorectal cancer and gut microbiome patterns in the accompanying primary publicationThe data generated in this study combine various views of each sample as multiple different methods were used to obtain information about the samples. This allows us to study associations between the results of the particular methods. Making the raw sequencing data available together with the scripts used for data processing and analysis, we enable reuse of the data and extend the collection of the data sources related to CRC, for which the aetiology is not yet well understood.14. The workflow is shown in Fig. 16, and scripts together with more downstream analysis results are accessible as the Zenodo dataset17.Here, we present a more condensed version of the methods that led to data and analyses in the primary publicationTM (Qiagen), and equilibrated for at least 48 hours at −20 °C, prior to nucleic acid extraction. RNA and DNA were extracted from 15–20 mg each of tissue using RNEasy Plus Mini Kit (Qiagen) and DNeasy Blood and Tissue Mini Kit (Qiagen), respectively. Tissue disruption was carried out using a Retsch Mixer Mill. RNA extraction included a DNAse treatment step, and DNA extraction included overnight incubation with proteinase K, and treatment with RNAse A. Purified nucleic acids were quantified using the NanoDrop 2000c spectrophotometer , and stored at −80 °C. Nucleic acids were extracted from all tumour samples in a single batch by one operator, to avoid inter-batch variation.Tumour tissue was collected from 34 patients undergoing surgery for resection of colorectal tumours. None of the patients had received chemotherapy prior to surgery, and all patients provided written, informed consent. This study was carried out with approval from the University of Otago Human Ethics Committee . Table Library preparation and ribosomal RNA depletion was carried out using Illumina TruSeq stranded total RNA library prep V1 and Ribo-Zero Gold. The ribosomal RNA depletion step has potentially removed a portion of bacterial ribosomal RNA alongside of the human one, hence losing some information on bacteria. However, the same method of depletion was used on all the samples thus the potential loss would effect all of them in a similar manner. RNA sequencing was carried out using the Illumina HiSeq. 2500 V4 platform, to produce 125 bp paired end reads. Each sample library was split equally to two HiSeq lanes and the sequences from the two lanes were merged for each sample during the data processing phase.18 and SolexaQA++19. The cleaned reads were mapped to the GRCh38 reference human genome with STAR20 and the read count for each HAVANA annotated gene in every sample was calculated with htseq-count21. The read counts were transformed to gene expression profiles measured in transcripts-per-million (TPM) with DESeq222. The published CMS classifier15 was used to assign a molecular subtype of the disease to each sample based on the gene expression profiles (for more details see14). We identified six samples as CMS1, 13 samples as CMS2 and nine samples as CMS3. No samples were classified as CMS4, and six samples were unclassified.Adapters and low quality segments were removed from the sequenced reads using fastq-mcf from EA Utils23 database was built containing all NCBI Refseq complete genomes or chromosome-level genomes (January 2017) and additional genomes of bacteria proposed to play a role in CRC, disregarding their genome status. The used bacterial genomes are listed in the files Supplementary_table_K1.xlsx  and Supplementary_table_K2.xlsx  in the folder data/kraken of the accompanying repository. All RNA-seq reads that were not uniquely mapped to the human genome reference sequence were used as input to Kraken using this custom database for taxonomic classification per sample. Altogether, 2231 different bacterial species were detected in at least one sample and only 1.4% of the analysed reads were not assigned to any bacterial species. We visualised bacterial abundances per CRC subtype using Krona24 and the interactive plots are available at http://crc.sschmeier.com.A Kraken25, we identified bacterial taxa with considerable abundance differences among the subtypes. For each CRC sample we used the assigned CMS subtype, the list of identified bacterial species, and the read counts corresponding to the identified species as input data. We treated all samples of a certain CMS subtype as replicates belonging to the subtype and ran differential analysis of each CMS subtype against all the other classified samples. This analysis identified bacterial species that are enriched (or depleted) in a subtype as compared to all other subtypes. For further details regarding the analysis, please refer to the primary publication14.We analysed the enrichment/depletion of bacterial species in one subtype compared to the other subtypes employing a strategy similar to differential expression analysis. Using edgeRLibraries containing 16S rRNA were prepared with 20 ng of DNA for each sample using primer pairs flanking the V3 and V4 hypervariable regions of the 16S rRNA gene and Illumina sequencing adaptors and barcodes were added using limited cycle PCR. Amplicon sequencing was carried out using the Illumina MiSeq platform, and paired end reads of length 250 bp were generated.26 to form sequences of the V3-V4 hypervariable 16S rRNA region. Afterwards, low quality regions were removed from the resulting fragments with SolexaQA++19. Microbiome analysis was carried out with the QIIME bioinformatics pipeline27 using the Greengenes database28 for taxonomy assignment. No further normalisation of the data was performed.Short overlapping forward and reverse reads coming from the same fragment were joined together with FLASh16. Data resulting from the analyses presented here are located in the folder data of the Zenodo repository17. The data are separated into several subfolders:The folder expr contains raw read counts in subfolder raw_counts, tpm-based expression profiles of all samples stored in file tpm.readyForClassifier.tsv and also file CMSclassifiedCRC.tpm.havana.tsv containing the CMS subtype classification. These files are the main outcomes of gene expression profile classifications.Results of the metagenomics analysis of the non-human genomic content of RNA-seq are located in folder kraken together with two tables (Supplementary_table_K*.xlsx) containing lists of bacterial species used in this metagenomics analysis.The folder 16S contains the biom file otu_table.biom resulting from the 16S rRNA metabarcoding analysis with QIIME and two partial abundance tables otu_table_sorted_*.txt.gz. The abundance tables are derived from the biom file and were used further for data visualisation in the primary publication as well as for the metagenomics method comparison.Sequenced genomic data from both RNA-seq and 16S rRNA metabarcoding experiments are stored in the Sequence Read Archive as the study SRP117763The quality of raw sequenced reads from RNA-seq experiments was assessed with FASTQC and was very good. A pair of representative per base quality plots of corresponding forward and reverse read pairs for one sample is shown in Fig. In Fig. Download metadata file


Single-cell RNA-seq has the potential to facilitate isoform quantification as the confounding factor of a mixed population of cells is eliminated. However, best practice for using existing quantification methods has not been established. We carry out a benchmark for five popular isoform quantification tools. Performance is generally good for simulated data based on SMARTer and SMART-seq2 data. The reduction in performance compared with bulk RNA-seq is small. An important biological insight comes from our analysis of real data which shows that genes that express two isoforms in bulk RNA-seq predominantly express one or neither isoform in individual cells.The online version of this article (10.1186/s13059-018-1571-5) contains supplementary material, which is available to authorized users. RNA-seq has transformed genomics by making it easy and relatively inexpensive to obtain genome-wide, quantitative measurements of the transcriptome. To be able to take full advantage of this data, however, it is critical to have robust and accurate computational methods for quantifying transcript abundances. Isoform quantification is considered a hard problem –3, and tAlthough bulk RNA-seq data of cells indicate the expression of multiple isoforms, scRNA-seq of the same cells show evidence for a single or a small number of isoforms per gene –18. ThisDespite the intrinsic benefits of single-cell resolution, to date most scRNA-seq research has focused on gene-level quantification and analysis . This paIn addition to software concerns, there are experimental and technical issues which could impact on isoform quantification in single cells. A wide range of library preparation protocols have been developed for scRNA-seq –29, someTo assess isoform quantification for scRNA-seq, we present a simulation-based benchmarking study using data generated from three different scRNA-seq projects. While benchmarking studies have been performed previously for bulk RNA-seq , 14, to The first dataset considered in this benchmark consisted of 96 mouse quiescent B lymphocytes collected as part of the BLUEPRINT epigenome project   . The performance of all three modes was evaluated in this benchmark. For most isoform quantification tools, the mean F1 score was remarkably similar and in the range of 0.777–0.888od) Fig. . The meaIn addition to determining whether an isoform is expressed, it is often of interest to estimate isoform abundance. To evaluate how well isoform quantification tools perform this task, two measures were considered—Spearman’s rho and the normalized root mean square error (NRMSE) Fig. , c. SpeaConsistent with the results for isoform detection, mean Spearman’s rho was similar between isoform quantification tools and simulation methods and in the range 0.782–0.891. The exception was eXpress, which had much lower mean Spearman’s rho than the other tools with values from 0.550 to 0.574. eXpress also performed poorly relative to the other isoform quantification tools when considering the NRMSE. Interestingly, although the overall pattern of NRMSE results was similar for both simulation methods, the NRMSE was consistently far higher for the Splatter and Polyester simulations compared to the RSEM simulations. One possible explanation is that the difference in the NRMSE is due to a small number of outliers. However, this did not appear to be the case  dataset published by Kolodziejczyk et al. . On averFrom the Kolodziejczyk et al. dataset, 271 mESCs grown in standard 2i media + LIF which passed quality control were used for the benchmark see . This daTo perform the benchmark, simulated data was generated as described previously from the selected cells from Kolodziejczyk et al. see . Again, The highest mean value of Spearman’s rho was obtained by Salmon run in SMEM mode (0.896), with Salmon run in quasi mode, Kallisto, RSEM, and Sailfish obtaining similar values. The lowest mean value of the NRMSE was also obtained by Salmon run in SMEM mode (19.5), with Salmon run in quasi mode, Kallisto, RSEM, and Sailfish obtaining similar values. Again, of the tools considered, eXpress performed most poorly by a substantial margin.We find that the performance of existing isoform quantification tools is generally good when run on simulated data based on SMART-seq2 and SMARTer scRNA-seq data. We next consider the performance of isoform quantification tools when scRNA-seq data is used compared with bulk RNA-seq data. Although previous benchmarks of isoform quantification have been performed using bulk RNA-seq data , 14, a dTo gain further insights regarding the performance of the tools, we made use of the bulk RNA-seq data generated for the BLUEPRINT B lymphocytes and Kolodziejczyk et al. standard 2i media + LIF mESCs. We used RSEM to simulate the bulk RNA-seq data and collected the same performance statistics for our bulk RNA-seq benchmark as in our scRNA-seq benchmark. As the data used in our bulk and scRNA-seq benchmark came from the same source, the same method was used to generate the simulated bulk and scRNA-seq data, and the same performance statistics were collected in both benchmarks, we were able to carry out a meaningful comparison of isoform quantification tool performance on bulk and scRNA-seq data.We find that all isoform quantification tools performed well on the simulated bulk data, but since most methods also performed well on single-cell data, the improvement was generally small  into isoform expression estimates could improve accuracy of quantification. While UMI information could reduce the effects of PCR amplification noise , UMI-basA number of previous studies have attempted to determine whether genes from which multiple isoforms are expressed in a population of cells are expressing all of these isoforms in individual cells –18. HoweFor high-quality simulated scRNA-seq datasets with a high number of reads/cell, RSEM, Kallisto, Salmon, and Sailfish can accurately detect and quantify isoform expression. Isoforms with a high number of dropouts appear to be relatively challenging to quantify, possibly because such isoforms are often lowly expressed. We find that genes which express two isoforms in bulk RNA-seq predominantly express only one (or neither) isoform in equivalent scRNA-seq. In our benchmark of bulk RNA-seq, we discover the performance of most isoform quantification tools is slightly worse for scRNA-seq compared with bulk, but that the difference is small.Taken together, our findings show that isoform quantification is possible with scRNA-seq for SMARTer and SMART-seq2 data. As single cells do not generally express all of the isoforms seen at the population level, scRNA-seq may eventually provide advantages over bulk RNA-seq for isoform quantification by essentially deconvoluting the problem of isoform quantification. Future isoform quantification tools designed explicitly for scRNA-seq could improve on the performance of existing tools by being more conservative in calling isoforms as expressed, and by explicitly modeling the technical noise inherent to scRNA-seq.The Ensembl release 89 genome and transcriptome with 92 spike-in sequences developed by the External RNA Control Consortium (ERCC) appended were used wherever genome files in FASTQ format and/or transcriptomes in GTF format were required , 47. To Sequencing adaptors were trimmed from the BLUEPRINT data using Cutadapt . Reads fTwo simulation methods were used in this study. The first method used to simulate single-cell RNA-seq data was RSEM. RSEM is an isoform quantification tool which makes use of a generative model and an expectation maximization algorithm to perform isoform quantification . When peFor each cell in the Kolodziejczyk et al. and the BLUEPRINT datasets that passed quality control, one RSEM simulation was performed. Isoform quantification was performed on each cell and the isoform expression estimates and inferred estimates for RSEM’s latent variables were used to perform the simulation. Consequently, each RSEM-simulated cell used in this study was simulated using variables inferred from a real cell.The second simulation method was based on two tools, Splatter and Polyester. A flowchart showing the steps followed to generate the Splatter- and Polyester-simulated data is displayed in Fig. The lun2sim method was used to simulate a matrix of counts based on an expression matrix of counts from the BLUEPRINT B lymphocytes generated by Kallisto . The simReads from each cell in the datasets simulated by RSEM based on the Kolodziejczyk et al. and BLUEPRINT datasets were aligned to the Ensembl genome release 89 using STAR. RSeQC was used to collect alignment quality statistics for each cell. The alignment quality statistics and the number of reads for each simulated cell were used to remove low-quality cells from each dataset  is:N is the number of isoforms that could have been simulated, S is the isoform expression estimates for the isoform quantification tool of interest, O is the ground truth expression estimates, and sd(O) is the sample standard deviation of the ground truth expression estimates.where Prior to calculating the NRMSE, the ground truth and the isoform expression estimates were transformed using the formula:Soriginal was the original value of the ground truth or the expression estimate. This transformation reduces the impact of a small number of highly expressed isoforms on the value of the NRMSE.where Additional file 1:Supplementary Figures. This file contains all of the supplementary figures for this paper. (PDF 2136 kb)