SEQC cohort patients to develop supervised and unsupervised models that can predict disease prognosis.Neuroblastoma is a heterogeneous disease with diverse clinical outcomes. Current risk group models require improvement as patients within the same risk group can still show variable prognosis. Recently collected genome-wide datasets provide opportunities to infer neuroblastoma subtypes in a more unified way. Within this context, data integration is critical as different molecular characteristics can contain complementary signals. To this end, we utilized the genomic datasets available for the SEQC cohort can accurately predict overall survival and event-free survival profiles of patients in two independent cohorts. We also performed extensive experiments to assess the prediction accuracy of high risk patients and patients without MYCN amplification. Our results from this part suggest that clinical endpoints can be predicted accurately across multiple cohorts. To explore the data in an unsupervised manner, we used an integrative clustering strategy named multi-view kernel k-means (MVKKM) that can effectively integrate multiple high-dimensional datasets with varying weights. We observed that integrating different gene expression datasets results in a better patient stratification compared to using these datasets individually. Also, our identified subgroups provide a better Cox regression model fit compared to the existing risk group definitions.Our supervised model trained on the Altogether, our results indicate that integration of multiple genomic characterizations enables the discovery of subtypes that improve over existing definitions of risk groups. Effective prediction of survival times will have a direct impact on choosing the right therapies for patients.This article was reviewed by Susmita Datta, Wenzhong Xiao and Ziv Shkedy.The online version of this article (10.1186/s13062-018-0223-8) contains supplementary material, which is available to authorized users. Neuroblastoma is the second most common solid tumor in childhood. The disease can have a large variety of clinical outcomes ranging from spontaneous regression to relentless progression despite extensive therapies. As such, accurate prediction of disease prognosis is critical to improve the choice of therapies. MYCN gene is a well-established prognostic marker in neuroblastoma. Chromosomal amplification of the MYCN locus occurs in 25% of all neuroblastomas and is associated with poor prognosis . HoweverSEQC) cohort  2017) to develop statistical models that can predict clinical outcomes in neuroblastoma. Using a linear Support Vector Machine  and event-free survival (EFS) variables in two independent cohorts. Our results indicate that predicting OS and EFS is more challenging for high risk (HR) patients. We observed that focusing on only high risk patients during training improves the prediction accuracy of HR patients. In the last part of the study, we employed an unsupervised learning strategy based on kernel k-means. We used MVKKM to integrate multiple data types with the aim to identify subgroups that have significantly diverse survival profiles. We observed that integrating all gene expression datasets  improves over using these datasets individually. Also, integrating these data types with learned weights is a better option than integrating them uniformly. We also confirmed that the Cox regression model that uses our identified clusters as covariates yields a better model compared to the regression model with existing high risk / low risk (LR) labels. We repeated these analyses for the subset of patients that have Array Comparative Genomic Hybridization (aCGH) data, and reached similar conclusions. Altogether, our results suggest that utilizing genomic characterizations of tumors improve over current definitions of risk groups.Recently, several methods have been developed to integrate multiple genomic data for cancer subtype discovery –9. In nene , we werne . Based on this comparison, the sign of the intensity values for 32 patients were reversed. The list of these patient ids can be found in Additional file SEQC_NB_MAV_G_log2.txt downloaded from CAMDA website and GSE62564_SEQC_NB_RNASeq_log2RPM.txt downloaded from GEO website for entry GSE62564. The data for the Versteeg cohort (GSE16476 [http://r2.amc.nl), and the data for the TARGET cohort were downloaded from the following link: https://ocg.cancer.gov/programs/target/data-matrix. Note that gene-level expression measurements were used for all our experiments. We used the limma package in R to perform differential expression analysis with RNA-seq MAV data [limma outputs adjusted p-values that are corrected for multiple testing using the Benjamini-Hochberg method [seaborn.clustermap function, (https://seaborn.pydata.org/generated/seaborn.clustermap.html) in Python  was used to generate the heatmap.RNA-seq, microarray and aCGH datasets for the GSE16476 ) were doMAV data . limma og method . seabornhttp://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) function available in Python’s scikit-learn library for training SVM models. Feature selection is performed with the sklearn.feature_selection.SelectKBest method, (http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html) that uses ANOVA F-values to rank the features. The only parameter that needs to be tuned is the C parameter that determines the cost of misclassification. Models with lower values of C allow for more errors and learn a large margin whereas models with higher values of C aim to classify all examples correctly and learn a small margin. We considered the values {106,105,104,103,0.1,0.5,1,2,5,10,20,50,100,1000} for the C parameter and chose the one with the best accuracy within a nested cross-validation framework. As clinical endpoints  have an unbalanced distribution in neuroblastoma cohorts, we further checked whether using class-specific weights improves the performance. Namely, we set the class-weight parameter to balanced so that the C parameter of each class is multiplied with a weight value that is inversely proportional to the corresponding class frequency in the input data. Nested cross-validation was used to decide between setting this parameter to balanced or to none.We used the Support Vector Classification =exp(−γ||x−y||2). To avoid selecting a specific γ parameter, we used six different γ parameters  for each view. Among the set of values {1.5,2,2.5}, 1.5 was chosen as optimal for the sparsity parameter p based on mean silhouette score [i is calculated as (bi−ai)/max where ai is the average distance of sample i to all other samples in sample i’s cluster and bi is the average distance of sample i to all the samples in the closest cluster that i is not a part of. Lastly, silhouette scores of all data points are averaged to determine the mean silhouette coefficient. The number of clusters was determined by aggregating three measures: mean silhouette score, DUNN index [We used the code provided by Tzortzis et al.  with theNN index  was usedsurvival package in R to perform Kaplan-Meier analysis. Because more than 80% of the SEQC data is right-censored, we applied the Cox proportional hazards regression analysis with Firth’s correction (hereafter named Coxphf) using the R package [loglik+mln(n) where loglik is the log likelihood of the regression model, n is the number of samples and m is the number of parameters [We used the  package . To comprameters . As an arameters  was applrameters ) and Harrameters  were userameters . The repSEQC dataset. The mean cross-validation accuracy of the models that predict OS  and EFS  labels is close to the best accuracy reported for the same dataset [SEQC cohort can be sufficiently captured by linear models.Table  dataset   and balanced accuracy as performance metrics. Several interesting observations can be derived from these results. First, when we compare the results of predicting OS and EFS labels, we observe that we predict OS more accurately than EFS in All→All and HR→HR contexts. Surprisingly, EFS prediction is more accurate when we focus only on patients with no MYCN amplification. Here we should note that some of the models listed in Table All→All and HR→HR contexts use gene expression data from the same platform whereas this is not the case for MYCN_NA→MYCN_NA context. For MYCN_NA→MYCN_NA context, if we compare models that are derived from the same platform only, we still observe that EFS prediction has better performance than OS prediction.We used our model trained on the All→Alland HR→HR contexts are indeed due to the different training datasets used, we repeated the comparison with models that use gene expression data from the same platform. When RNA-seq based gene expression data is used, models that are learned from only HR patients perform better than models that are learned from all patients in predicting OS of HR patients. However, we do not observe the same improvement for microarray based models. We also checked whether accounting for unbalanced class labels improves the performance. Turning on the balanced option for class weights gives a better model in terms of training set performance except for EFS prediction in All→All model. This could be due to the fact that the EFS label is less unbalanced than the OS label in the SEQC cohort.We investigated whether using only HR patients for training improves the prediction performance of OS and EFS of HR patients. Indeed, this is the case for OS prediction as balanced accuracy increases from 0.71 to 0.79. However, the same effect is not seen in predicting EFS. To confirm that the differences in accuracy between Versteeg cohort data itself to confirm that the expression profile of these genes correlate with MYCN mRNA levels. Using these 157 genes, they were able to predict OS in Versteeg cohort with a balanced accuracy of 0.84. Figure SEQC) performs remarkably well on predicting OS in Versteeg cohort.Figure Versteeg cohort for the TARGET cohort. . This result is likely due to the high fraction of HR patients in TARGET cohort. When we restrict the comparison to models that use the same type of gene expression data, models derived from RNA-seq data predict both OS and EFS of HR patients more accurately when the training was performed with HR patients only  for the TARGET cohort. The OS models trained only with HR patients showed a lower overlap size: 70 between SEQC and Versteeg cohorts and 81 between SEQC and TARGET cohorts. The smaller size of the HR training datasets is likely to be associated with these small overlaps.We investigated the 1000 selected features across the different models that we trained. First, we compared the OS and the EFS models that we used to predict Versteeg or TARGET cohorts and tested on the SEQC cohort. The linear SVM model trained on the Versteeg cohort predicts OS in SEQC cohort with a balanced accuracy of 0.80 and an AUROC of 0.86. This analysis reveals that even a small number of patients  is satisfactory to learn a gene signature that can predict OS in a much larger cohort . Predicting EFS resulted in a similar accuracy . We repeated the same experiments where TARGET cohort is used as the training set. As expected, the predictive accuracy was lower compared to training on the Versteeg cohort . Though, we see the opposite pattern when we try to predict the outcome of HR patients using only HR patients as the training data. Namely, TARGET-trained models  achieve a better predictive accuracy than Versteeg-trained models . This is likely related to the much larger set of HR-only training data in TARGET cohort (TARGET: 217 vs Versteeg: 36).The experiments were performed in the opposite direction where we trained models using the SEQC and other cohorts, we also investigated integrative clustering approaches to identify neuroblastoma subtypes in an unsupervised manner. To this end, MVKKM method was utilized which can cluster samples by integrating multiple data types. In MVKKM method, each data type is considered as a view, and multiple kernels can be used to represent each view. Because the relevance of the different views (or even the relevance of the different kernels in the same view) to the clustering task can vary, MVKKM learns a weight distribution across the kernels. This weight distribution enables the contribution of different kernels in varying degrees. Also, clustering the samples in the kernel space provides flexibility in applying non-linear feature transformations.In addition to supervised learning approaches to predict OS and EFS in First, we applied MVKKM to all the patients to integrate different types of gene expression data. We used six RBF kernels with different gamma parameters for each view (See Methods). Table Figure limma method. The top 500 genes with the smallest adjusted p-values are listed in Additional file We investigated the genes that are differentially expressed between the two clusters with p-value . Interestingly, including aCGH data to this model resulted in no improvement. On the other hand, if we focus only on patients with no MYCN amplification (121 patients), including aCGH on top of gene expression datasets results in a slight improvement compared to using gene expression datasets only (BIC: 219 vs 220). For 145 patients, the MVKKM-learned weights for RNA-seq (MAV), RNA-seq (RPM) and microarray datasets were 0.30, 0.34 and 0.36 respectively. Figure We repeated the same analysis with a subset of patients i.e., 145 patients) that have aCGH data. Similar to what we observed in our experiments with all patients, combining the RNA-seq and the microarray datasets gives the best model  all patients; (ii) subset of patients with aCGH data and compared the different models that use distinct sets of data types with Cox proportional hazards regression results. Application of Firth’s correction to Cox regression was critical as we obtained completely different results without this correction (data not shown). Based on most of the evaluation criteria calculated from Cox regression and AFT models, combining all the available gene expression datasets resulted in the best model both for all the patients and for the subset of patients. For all patients, we discovered two clusters that have significantly distinct survival profiles. In parallel with the observed variability of prognosis within LR and HR patients, our identified clusters contain a mixture of LR and HR patients. For subset of patients with aCGH data, we discovered three clusters that correspond to low risk, intermediate risk and high risk groups. We observed that the proportion of literature-defined HR patients increases as the risk level of the group increases. Interestingly, addition of aCGH data on top of the gene expression datasets resulted in no improvement for this group of patients. This could be due to the high noise level of the aCGH data. The clusters that we identified in both patient groups indicates that the current stratification of patients to high risk and low risk groups can be improved via the integrated use of genomic datasets.The performance of both supervised and unsupervised learning models can be further improved as genomic data becomes available for larger cohorts. Also, additional types of genomic data such as DNA methylation, protein expression or microRNA expression can provide opportunities to understand this complex disease from different angles and to pave the way toward improved choice of therapies. Indeed, a recent study discovered that the disruption of the let-7 microRNA family is an important mechanism in understanding cancer pathogenesis for neuroblastoma .Our study demonstrates that supervised learning models built from genomic datasets are suitable for clinical endpoint prediction in independent cohorts. Also, unsupervised integration of multiple genomic datasets with MVKKM reveals neuroblastoma subtypes with distinct survival profiles. Both supervised and unsupervised approaches can contribute to improved treatment stratification of neuroblastoma patients. Altogether, these results indicate that the use of multi-dimensional genomic datasets has the potential to improve current cancer risk models.In this manuscript authors demonstrate that supervised learning models built from multiple genomic datasets are suitable for clinical endpoint prediction. Also, they use a form of Multi-view kernel k-means (MVKKM) algorithm to identify subtypes of Neuroblastoma which has distinct survival profiles. Both supervised and unsupervised approaches can contribute to improved treatment stratification of neuroblastoma patients. It is an interesting idea in include both supervised and unsupervised methods for the survival prediction. I have some major concerns:Reviewer comment: Authors have used mean silhouette co-efficient to choose the number of clusters. Please note that there are many other indices to determine the quality of clusters such as Dunn-index and connectivity. One may be able to use them as well to determine the number of clusters. So a holistic way will be to use many such evaluation measures and rank them using methods such as Pihur et al., Bioinformatics, 2007 paper.We thank the reviewer for this suggestion. In addition to mean silhouette index, we now calculate the DUNN-index and connectivity for each value of k. Other measures mentioned in Pihur et al. were either meaningful only in the context of gene expression data; or they required substantial additional code for implementation. We used RankAggreg method proposed by Pihur et al. to combine the resulting rankings. The number of clusters chosen by the aggregation of these three methods remained the same in our experiments. As such, we did not update the results. However, we are glad that the aggregation method could improve the choice of number of clusters for future studies that focus on cancer subtyping.Author’s response: Reviewer comment: The wonder whether the assumptions of Cox proportional hazards model will work here or not. I do think Accelerated failure time model with penalty for high dimensional data would work better. So please comment why that wasn’t used.We think there is misunderstanding about how we run the Coxph model. We do not input high dimensional data  directly to the Coxph model. We first learn clusters from the kernel-transformed data and input these clustering assignments to the model together with age and tumor stage. However, what is critical is that we do account for the large proportion of censored samples with Firth correction. One advantage of using the Coxph model is that we can easily compare the hazard ratios that we obtained with those from literature. Nevertheless, we have also tried Accelerated failure time model where right censoring is handled by imputation as in Dr. Datta’s recent paper . We now add adjusted root mean squared error (rmse) and Harrell’s c-index measures to Tables4andWe do not report low-predicted survival (LPS) classification of patients as with the cutoffst=2andt=5, we routinely got an error from survdiff function regarding the existence of only a single group. We think rmse and Harrell’s c-index are representative of model performance.Author’s response: In summary, even though the ranking of the models have changed in some cases, our main conclusions remained the same based on the results of the AFT model.Also, during these experiments, we discovered a bug that leads to reading HR/LR labels incorrectly. As such, Figs. Reviewer comment: How do you choose p related to sparsity in composite kernel?We thank the reviewer for this comment as we realized that we forget to include the choice of p in the “section. Our choice of p parameter is based on the following assumptions and prior results:p>=4) correspond to the case where views contribute uniformly to the composite kernel. We think that p values that lead to non-uniform weights would work best for our context.In this manuscript, we integrate datasets that are highly correlated . However, these datasets could still contain complementary information and have to potential to improve clustering when combined. As described in the manuscript, in MVKKM model, p=1 corresponds to the selection of only one view  and large values of p , the authors have used MVKKM for several datasets and their results indicate that the choice of p within the range of {1.5-2.5} give the optimal results. Even though the datasets they used are of different context, we still think that the results are informative.We tried the values 1.5, 2 and 2.5 based on the discussion above. p = 1.5 was chosen according to mean silhouette coefficient. We now mention this in “Author’s response: Reviewer comment: What is the range of p?p has to be greater than or equal to 1. We state this in the second paragraph of “section.Author’s response: In this manuscript, Baali etc. described their work comparing gene prediction models across multiple independent datasets of neuroblastoma and using genomic data integration modeling to study cell signaling mechanisms of high-risk neuroblastoma and to predict disease outcomes.The paper touched upon a number of observations and issues that this reviewer thinks are important to the integration of multiple genomic data sets. However, more clarity is needed in the paper: while a reader can follow the methods and results described in the paper, some of the statements appear to be weak and confusing, and it is hard to draw conclusions.Reviewer comment: A technical issue often came up in integration of multiple datasets is between data from arrays and data from RNA-seq. Tables We thank the reviewer for raising this important point. We checked whether our observations still hold when we compared the models that use data from the same platform. The balanced accuracy and AUROCs of all the models are now listed in Additional fileAuthor’s response: We changed the result section accordingly. In particular, in section “Here we should note that some of the models listed in Table2use gene expression data derived from different platforms (e.g. RNA-seq or microarray). As such, we checked whether these observations still hold if we restrict the comparison to models that use gene expression data from the same platform . As such it does not require us to compare models of the same gene expression platform.Turning on the balanced option for class weights gives a better model in terms of training set performance except for EFS prediction inAll→Allmodel. This could be due to the fact that the EFS label is less unbalanced than the OS label in the SEQC cohort.Reviewer comment: On Page 6 line 7–11, the author stated that “we see that the microarray-based models are preferred over RNA-seq. This could be due to the fact that the microarray platform is used to measure gene expression in the TARGET cohort”; is this true for the Versteeg cohort as well, since it used microarrays as well? Would the models from arrays show the best performance in general? It would help if the authors can either discuss the test results of the models derived from each platform, or show these results as supplemental information.We thank the reviewer for this comment. Indeed, we have realized that the same observation does not hold for the Versteeg cohort where gene expression data is also array-based. As such, we have now removed those sentences from the “section and instead included the following sentences to Discussion.Author’s response: Our experiments indicate that the differences in platforms of gene expression data between training and test cohorts may not be critical as RNA-seq-derived models are found to perform better than microarray-derived models for many of the prediction tasks where the test cohort contains array-based gene expression data.Reviewer comment: As shown in Table In addition to the “section, we also mention the difficulty of predicting the outcomes of HR patients in the first paragraph of the “section.Author’s response: We could not understand the second part of this comment as we have not applied MVKKM on TARGET cohort. Since MVKKM is an unsupervised approach that outputs clustering information of patients, it is unclear to us how it could be used to predict patient outcomes directly.Reviewer comment: Besides, on Page 2 line 55–58, “Chen Suo and her colleagues have identified a potential mislabeling of 32 neuroblastoma patients in aCGH data . As such, we updated the aCGH data accordingly.” The authors then stated that the aCGH data did not improve the results of prediction. Can the authors reference the information or include details of the corrections they made so readers can potential reproduce the results?We now included a Additional fileAdditional filethat lists the ids of 32 patients and explained the correction in more detail in text as follows:Author’s response: Chen Suo and her colleagues have identified a potential mislabeling problem between normal and tumor samples when they compared the aCGH data against the MYCN status derived from FISH experiments . Based on this comparison, the sign of the intensity values for 32 patients were reversed. Ids of these patients are listed in Additional fileReviewer comment: Figure Fig.3is now referenced in the beginning of the section.Reviewer comment: The paper describe supervise and unsupervised methods for the development of multi- source signature for High risk /Low risk survival patients. It is an applied paper which presents an analysis for different datasets. The paper is interesting but poorly written and a strong language editing is needed in order to transform the current text to a level of a scientific publication . I listed below few examples (there are much more).We thank the reviewer for detailed suggestions on language usage. We made the necessary changes for all the items listed below. Detailed replies are available below. Additionally, we revised several other parts of the manuscript to improve clarity. We hope that our revised manuscript reads better.Author’s response: Reviewer comment: Page 2, line 28: “Using a linear SVM” should be “Using a linear Support vector machine ”. See for example, line 37.We have fixed this now and inserted the reference for SVMs.Author’s response: Reviewer comment: Page 2, line 55 “.....CGH datasets for the SEQC cohort”. You should use the same font for SEQC, see for example page 4 lines 38, 48 etc. Author’s response: We have fixed the font and used the full name for the SEQC acronym.Reviewer comment: Page 3, line 25: “the SVC function available”, SVC mentioned for the first time, use the full name and give reference.We have spelled out the full name for SVC and inserted its reference.Author’s response: Reviewer comment: Page 3, line 35: "the C parameter" it is not clear what the C parameter is.We have now inserted a description for the C parameter.Author’s response: Reviewer comment: Page 3, line 47 “Multi-view kernel k-means (MVKKM)”, you do not need to use the ?full name since it was already mentioned in the abstract and in page 2, line 47.This has been corrected.Author’s response: Reviewer comment: Page 3, line 60-page 4 line 5: All parameters of equation (1) should be in ONE sentence with “,” between the parameters.The parameters are now explained in a single sentence.Author’s response: Reviewer comment: Page 4, line 23: add space before “where”.This has been corrected.Author’s response: Reviewer comment: Page 4, lie 52: the title of the subsection should not be numbered 0.0.1.We have removed the numbering before the subsection title.Author’s response: Reviewer comment: Page 4 line 57-page 5 line 8: this text should be a part of the caption of Table We have moved the model definitions to the caption of TableAuthor’s response: Reviewer comment: Page 5, line 43: “We also plot the survival profiles of these different approaches in Fig. This is now fixed. Thank you.Author’s response: Reviewer comment: Page 6, line 43: “We also performed the experiments.....” should be “The experiments was performed in the opposite...”. In general try to reduce the number of times that you write “We also plot.....”, “we also compared...”, “ we also investigated...”, etc.We have edited the entire manuscript based on this comment and reduced the number of sentences that start with we.Author’s response: Reviewer comment: Page 7, line 29: “silhouette score” is mentioned for the first time, it is not clear what is it, add a ref.Silhouette score is already defined in page 4, we have now included a reference for it.Author’s response: Reviewer comment:Page 7, line 48: “The log-rank test p-value of this analysis confirms that the patients in the two clusters show distinct prognosis .” should be “The log-rank test p-value is equals to XXXX confirms that the patients in the two clusters show distinct prognosis.”This has been corrected.Author’s response: Reviewer comment:Page 7, line 60 “adjusted p-values” is mentioned for the first time. Based on which methods the p values are calculated?We have defined adjusted p-values in Methods and included the multiple-testing correction method.Author’s response: Reviewer comment: Page 7, line 61, give a reference to the limma method.The reference for the limma method is already provided in the “section.Author’s response: Reviewer comment: Page 9, line 36: Abbreviation should be excluded. Give the Abbreviation for each method in the text in the first time that it is mentioned. For example, see page 2, line 27 for MVKKM.Abbreviation section is inserted due to the journal instructions.Author’s response: Reviewer comment: Page 13, Tables We have now rephrased the table captions to include titles.Author’s response: We have converted numeral representations to words to express the numbers below 10.The captions for Tables We have converted a number of sentences in the “The full names for SVM and MVKKM are removed in the “All occurrences of the word Versteeg and SEQC are now italicized.The abbreviation HR is introduced after the first use of High Risk.Some of the other changes we made are listed below: Additional file 1List of patients for which aCGH data is corrected. This text file contains the list of 32 patients for which aCGH data intensities are reversed. (TXT 191 b)Additional file 2limma analysis on RNA-seq-MAV data. The data for the top 500 genes with smallest adjusted p-values are included. The columns indicate gene ID, log fold change, average expression of the gene, t-statistic, p-value, adjusted p-value, B-statistic and a column that indicates whether this gene has been found to be associated with neuroblastoma in literature. (XLS 110 kb)Differential Expression Results. This spreadsheet contains the results of Additional file 3Figure S1. Heatmap of differentially expressed genes. The heatmap plots the expression (RNA-seq-MAV) of the top 50 genes with smallest adjusted p-values for the two clusters. (PDF 669 kb)Additional file 4Versteeg and Target cohorts. This spreadsheet lists the performance metrics of models trained from SEQC cohort. Models that use different types of gene expression data , RNA-seq (RPM)) are listed individually. (XLSX 40 kb)Performance metrics of models for predicting OS and EFS in


DNA methylation is a widely investigated epigenetic mark that plays a vital role in tumorigenesis. Advancements in high-throughput assays, such as the Infinium 450K platform, provide genome-scale DNA methylation landscapes in single-CpG locus resolution, and the identification of differentially methylated loci has become an insightful approach to deepen our understanding of cancers. However, the situation with extremely unbalanced numbers of samples and loci  makes it rather difficult to explore differential methylation between the sick and the normal. In this article, a hybrid approach based on ensemble feature selection for identifying differentially methylated loci (HyDML) was proposed by incorporating instance perturbation and multiple function models. Experiments on data from The Cancer Genome Atlas showed that HyDML not only achieved effective DML identification, but also outperformed the single-feature selection approach in terms of classification performance and the robustness of feature selection. The intensive analysis of the DML indicated that different types of cancers have mutual patterns, and the stable DML sharing in pan-cancers is of the great potential to be biomarkers, which may strengthen the confidence of domain experts to implement biological validations. DNA methylation is one of the essential epigenetic mechanisms, which plays a vital role in normal development and is closely correlated with the cell growth, differentiation, and transformation in eukaryotes .Failure t statistics analysis based on linear model and empirical Bayes  has become an insightful attempt for detection of tumor markers . In the al Bayes . We consHowever, considering that the HM450 data have a small number of samples but high dimensional features , the results from general feature selection methods for identifying DML will have poor robustness . The robIn this article, we aggregate data diversity and function diversity to propose a hybrid ensemble approach for identification of DML (HyDML). Under the framework of ensemble feature selection, this newly proposed method not only can realize the effective identification of DML, but also can accommodate for the robustness of the results. Additionally, taking advantage of the large-scale Infinium 450K methylation data produced by The Cancer Genome Atlas (TCGA) project, we performed intensive analysis to look further into interrelationships between differential methylation and cancers and found that different cancers have common patterns, and robust DML sharing in pan-cancers is of the great potential to be biomarkers.For feeding the algorithm and analysis, in total 13 cancers are selected with both normal and tumor samples. Specifically, these cancers are bladder urothelial carcinoma (BLCA), breast invasive carcinoma (BRCA), colon adenocarcinoma (COAD), esophageal carcinoma (ESCA), head and neck squamous cell carcinoma (HNSC), kidney renal clear cell carcinoma (KIRC), kidney renal papillary cell carcinoma (KIRP), liver hepatocellular carcinoma (LIHC), lung adenocarcinoma (LUAD), lung squamous cell carcinoma (LUSC), prostate adenocarcinoma (PRAD), thyroid carcinoma (THCA), and uterine corpus endometrial carcinoma (UCEC). In all, there are 6,189 samples including 699 normal samples and 5491 tumor samples (https://tcga-data.nci.nih.gov/tcga/) for our selected samples. The methylation data are generated by Illumina Infinium HumanMethylation450k BeadChip technique. The Illumina Infinium assay utilizes a pair of probes for each CpG site, one probe for the methylated allele and the other for the unmethylated version. The methylation level is then estimated, based on the measured intensities of this pair of probes, as the ratio of methylated signal to the sum of methylated and unmethylated signal, which ranges from 0 (absent methylation) to 1 (completely methylated). To assess the ability of the selected DML to distinguish between the two types of samples , we retrieved three independent test sets from the NCBI database. The three data sets are also obtained by HM 450 technique, including samples of breast (GSE52635), liver (GSE54503), and lung (GSE66836) cancer, as well as corresponding normal tissue data records  Eli records . iii) FiFirst, in order to obtain a diverse set of feature selectors, we perform multiple samplings on training samples to generate data subsets. To this end, we make use of resampling and cross-validation, integrating classifier training into the ensemble feature selection framework for selecting loci that are informative for classifying tumor and normal samples. In each sampling, the whole data set is divided into 10 pieces with the same number of samples, and each of them can be regarded as a test subset to validate subsequent classification performance, while the rest automatically becomes a training set for feature selection and classifier training (constructed with support vector machine) . The insm represents the number of samples, and p represents the total number of features in the data set. λ is used to balance the empirical risk and model complexity, whereas α is used to balance the regularization of L1 and L2. In MDFS, we apply feature selection with max information gain criterion, which measures the worth of a feature by computing the information gain values with respect to the class. For rmcfs, it relies on a Monte Carlo approach to select informative features and is capable of incorporating interdependencies between features. The three basic feature selection algorithms can be well adapted to the high-dimensional and small-sample characteristics of 450k methylation data, and the whole calculation amount is moderate, while classification performance can be guaranteed. For each data subset, aggregating the results of multiple feature selection methods could further enhance the stability. More formally, consider an ensemble feature selector E = {F1, F2,… ,Fs} and each Fi provides a feature ranking fi denotes the feature weight of each Fi and N represents the nth feature. Hence, a general aggregation formulation for the ensemble ranking f, obtained by weighted summing the ranks over all fi, is as follows:where w represents the feature weight coefficient, acci donates the accuracy of the corresponding test set on the classifier trained by feature selector Fi, and f also can be regarded as the aggregation ranking for the ensemble feature selector. Here, s = 3, which represents the three basic feature selection methods, and we can get the preliminary DML at this level of aggregation. Then, taking the union set of obtained loci subsets is the second level of aggregation, and the corresponding formula representation is as follows:where s donates the number of data subsets, and fi is the feature ranking of corresponding data subset. In this way, one aggregated ranking of all the features for each sampling can be yielded. We perform 10 iterations for generating more data subsets to further improve the stability of selected loci, and with the idea of bagging, the final DML set consisted of loci that appear more than five times in 10 iterations. The overall algorithm framework for one sampling is shown in where To measure the effect of our hybrid ensemble technique on the feature selection results, following fi represents the signature obtained by the selection method on subsampling i(1 ≤ i ≤ k); k is the number of data subsets; S is a similarity measure for feature subsets, which denotes the stability of fi and fj. Here, we use Jaccard index :where rd index  as S.where the indicator function I(.) returns 1 if its argument is true, and zero otherwise. In the sequel, the overall stability To evaluate the classification performance and perform comparisons, we use several characteristics of classification performance all derived from the confusion matrix. These characteristics are TP, TN, FP, and FP, which denote true-negatives, true-positives, false-negatives, and false-positives, respectively. All the performance metrics are calculated by these characteristics, including TPR (true-positive rate), FPR , ACC (classification accuracy), Precision, Recall, and F1 score. We also include the area under the receive operating characteristic curve, which is defined by a function of sensitivity and specificity, further abbreviated as AUC.For each of the 13 cancers, we finally obtained a set of DML, which varies from 5,700 in COAD to 14,516 in THCA , we founWe first explored the distribution of DML in 22 pairs of autosomes for each cancer, which could help us to find out which chromosome gets potential extensive genetic variation when cancer occurs. To this end, we calculated the distribution density of the DML on each autosome, using ratio of the number of DML to the number of CpG sites determined by the 450K chip . Most DMWe also observed that biologically similar cancers shared more mutual DML through hierarchical clustering using similarity metric based on Jaccard index . KIRC anStot , and the results in all 13 cancers showed the hybrid ensemble approach (HyDML) performed better than single-feature selection methods, which could be a benefit in performing subsequent analysis with the selected differential methylation sites  were used to compare HyDML with classical DML identification methods, including FastDMA and RnBeads, for analyzing the differences between the ensemble feature selection approach and the statistical test method. Using the original DML previously selected from the three cancers as training sets, we constructed a classification model based on SVM and performed the verification with the test sets. The results showed that DML selected by HyDML performed better than FastDMA and RnBeads Table 1.In order to further analyze the association between DNA methylation and cancer, we investigated the differential methylation sites that occurred in multiple cancers, which could help us reveal the pan-cancer–associated methylation patterns. First, we defined a selected site as a pan-cancer differentially methylated locus (pDML) if it occurred no less than 10 times in 13 cancers. We in total obtained 338 pDML, in which some of them presented as hypermethylated, whereas the others presented obvious hypomethylation, expressed by median value in normal and tumor samples Figure 4TBX15. It has been demonstrated that TBX15 plays a vital role in multiple cancers, such as non–small cell lung cancer  in all 13 cancers Figure 5g cancer , thyroidg cancer , and ovag cancer , and espg cancer . MoreoveP = 5.55e-21)  . Baylin .55e-21) .P < 0.05). Then, we summarized the pathways that occurred in at least seven cancers and denoted as pan-cancer methylation-related pathways (PMPs) and obtained in total 11 PMPs, where 10 of them have been reported to be associated with cancers , feature selection–based approaches could select more informative loci that are closely related to the difference between the normal and the sick, as well as eliminating noisy and irrelevant loci, especially when dealing with microarray data of sparse samples and high-dimensional features. By t-SNE clustering, the results showed that the selected loci could distinguish between the normal and the sick well in each cancer, and the results from the independent test sets demonstrated that the classification model constructed by loci from HyDML had better generalization ability.Additionally, comprehensive investigation of the pDML showed that different cancers shared some common patterns in methylation variability at CpG locus resolution and revealed the potential similarities in different cancers. We found that same tissues share more abnormal methylation patterns with different subtypes of tumorigenesis, such as KIRC and KIRP, and LUAD and LUSC. This may indicate that the tissue specificity of methylation is preserved even when the tissue is cancerous. We also found a locus (cg02829688), which was hypermethylated in 13 cancers, located in a functional region on the genome, and could be of great potential to be an oncogenesis biomarker. Enriched motifs analysis from the background sequences of pDML revealed the potential influence on transcription function by CpG methylation, and the most significantly enriched motif, IRF3, has been reported playing a vital role in tumorigenesis. Through pathway analysis, some pan-cancer–related pathways were also determined, which have been reported playing a vital role in start, development, and metastasis of tumors.As an import epigenetic mark, DNA methylation has been widely investigated to deepen our understanding of its mechanism and correlation with human illness, and it is possible to analyze methylation at all levels with the massive data generated by high-throughput detection technology. However, how to effectively identify DML from high-throughput methylation data is still a tough challenge even if feature selection methods have been extensively explored in the context of gene expression data. Innovatively, combining the instance perturbation and function diversity, the newly proposed method HyDML achieved effective identification of DML, and this demonstrated that ensemble feature selection could be used in dimension reduction for large-scale biological data. This will not only facilitate future early diagnosis of cancers based on the DNA methylation signatures but also enable additional investigations into the utilization of feature selection on other biomarker analysis domains. In the future, we will continue to study in depth the application of machine learning in biomarker identification and achieve better selection and prediction effect by combining more related information.In this article, a hybrid ensemble approach is proposed by incorporating instance perturbation and multiple functions to identify differential methylation sites across 13 cancers from TCGA. The specially designed framework makes it possible to select robust differential methylation sites, which not only improves the accuracy of the classifier built by the selected sites, but also enhances the confidence of domain experts to implement biological validations. Further intensive analysis reveals that different cancer types have common methylation patterns, and part of the differential methylation sites shared in pan-cancers is of great potential to be crucial in the early diagnosis of cancers. All findings demonstrate that abnormal DNA methylation could be regarded as a marker that expresses the difference between the normal and the sick.The data sets and materials for this study can be found in the following links:https://tcga-data.nci.nih.gov/tcga/HM 450 methylation data: Independent test sets:https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE52635For BRCA: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc= GSE54503For LIHC: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE66836For LUAD: https://github.com/TQBio/HyDML.git.Source codes of HyDML, DML files, and single-nucleotide polymorphism files have been provided as an open source available at QT, JZ, and SF conceived and designed the experiments. QT, ZY, YF, JT, YS, and SF performed the analysis and edited the manuscript. JZ and SF led the research and reviewed the manuscript. All authors read and approved the manuscript.This work was supported by the National Natural Science Foundation of China (no. 61503061 and no. 61872063) and the Fundamental Research Funds for the Central Universities (no. ZYGX2016J102).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Intratumor heterogeneity (ITH) represents the diversity of cell populations that make up cancer tissue. The level of ITH in a tumor is usually measured by a genomic variation profile, such as copy number variation and somatic mutation. However, a recent study has identified ITH at the transcriptome level and suggested that ITH at gene expression levels is useful for predicting prognosis. Measuring ITH levels at the spliceome level is a natural extension. There are serious technical challenges in measuring spliceomic ITH (sITH) from bulk tumor RNA sequencing (RNA-seq) due to the complex splicing patterns.We propose an information-theoretic method to measure the sITH of bulk tumors to overcome the above challenges. This method has been extensively tested in experiments using synthetic data, xenograft tumor data, and TCGA pan-cancer data. As a result, we showed that sITH is closely related to cancer progression and clonal heterogeneity, along with clinically significant features such as cancer stage, survival outcome and PAM50 subtype. As far as we know, it is the first study to define ITH at the spliceome level. This method can greatly improve the understanding of cancer spliceome and has great potential as a diagnostic and prognostic tool. Intratumor heterogeneity (ITH) represents the diversity of cell populations that make up cancer tissue . This reIn a recent study by Morris et al. , The ITHITH can be deduced using molecular profiles extracted from genomic, epigenomic and transcriptomic data such as whole-genome sequencing (WGS), CNVs, bisulfite-seq, and RNA-seq data. Approaches using each domain have been used to assess the level of ITH in cancer tissues and to identify molecular features associated with tumor evolution . For exaGenome-level ITH has been extensively studied using bulk tumor sequencing data. ABSOLUTE  is a genRecently, an ITH model using a methylation profile was developed. This model, proposed by Mazor et al. , used a A transcriptome-level ITH model was recently developed . They usSplicing variations have been regarded as important cancer drivers in various types of cancer , 20, 21.The presence of intercellular spliceomic differences has also been proposed by numerous studies , 25. A rSince Park et al.  proposedThe flow of subsequent manuscripts is as follows: 1) definition of the proposed model, 2) basic verification of the validity of the model, 3) performance verification of the model using synthetic data, 4) performance verification using xenograft tumor data, and 5) performance verification using TCGA pan-cancer data.In this study, we used bulk-tumor RNA-seq data to measure sITH. It is a technique that combines bulk sampling with short-read sequencing. A possible alternative for each part is single-cell analysis and single-molecule real-time sequencing (SMRT-seq). Single-cell analysis has been improved in terms of stability and efficiency and has been used in many biological studies. This technique is very useful for studying ITH because it provides a molecular profile of every single cell in bulk tumors . HoweverAs discussed in the previous section, transcript assembly in cancer is very difficult due to complex splicing patterns, noncanonical splice sites, and short-length sequence reads. So we decided to use a local analysis approach to avoid transcriptome assembly. Here, each RNA-seq read that supports each splicing event is collected and grouped by each intron region. . Splicedintronic splicing unit or splicing unit, which is defined as a collection of splicing events for each intron , PS = , CNTk(i) represents the number of RNA-seq reads that support i-th alternative splice site in k-th splicing unit. Pk(i) is the fraction of RNA-seq reads that support the i-th splice site of the k-th splicing unit. Nk is the total number of alternative sites in k-th splicing unit.Then we defined a local unit called an h intron . In thisNormal tissues are also known to have heterogeneity in the use of isoforms between cells . To dealJSD represents the Jensen-Shannon divergence between the two distributions Pk and Qk, where Pk and Qk denote the splice-site usage distribution of the k-th splicing unit in samples P and Q, respectively. Note that Pk and Qk are defined in Mk represents the intermediate distribution introduced between two distributions Pk and Qk designed to calculate bi-directional KLD (KLD(Pk‖Mk) represents the Kullback-Leibler divergence of the distribution Pk from Mk. Ther might be a case that two samples have different sets of splice sites. In that case, the pseudo-count is added to the splice site, which is not found in one sample, where the pseudo-count is calculated to be 1/100 of the total number of reads in the corresponding splicing unit.sITH represents the increased sITH of a sample P from the origin sample Q to be compared. In the actual case, the target sample P corresponds to a bulk tumor sample, and the origin sample Q corresponds to a normal sample. In this case, sITH may be called sITH of sample P for convenience . Here, the two samples to be compared are pre-processed using the pseudo-counting described above to have the same number of splicing units for the compatibility issue. Therefore, the i-th splicing unit of samples P and Q represents the same intronic region.JSD can be calculated for each intronic region . Becauseonal KLD . KLD of these RNA-seq samples was defined using Shannon’s entropy as ITHintron) as ITHtranscript and ITHintron calculated for the 1,000 samples was analyzed. The results are discussed in the “Results” section.Pgi indicates the ratio of the i-th isoform of the g-th gene and implies the sequencing coverage ratio in the context of this analysis. Ng indicates the number of isoforms of the g-th gene.Pki is the selection ratio for the i-th splice site of the k-th intron unit, which means the read count ratio between splicing junctions. Nk is the number of splice sites of the k-th intron unit. L indicates the total number of intron units found in the sample.The whole-transcript ITH  represents the count of the j-th junction of the resulted mixture sample with i ITH level. NT(j) represents the count of j-th junction in the selected normal tissue sample. SC represents the number of j-th junction in the l-th selected single-cell cancer sample. To avoid sampling bias we randomly extracted 10 times at each of 39 ITH levels. Thus, a total of 390 mixture samples were synthesized (10 iterations per 39 ITH levels). In conclusion, samples mixed with more single cells are expected to have a larger ITH by design.We randomly selected normal tissue data from 112 normal tissue pools. Single cells were then selected for mixing. Here, the number of cells represents the ITH level . Selected single cells were mixed in normal tissue at a rate of 1% per cell. The detailed definition of each mixture sample is in We conducted a correlation analysis between the number of single cells mixed and the sITH value measured in the sample, and the results are discussed in detail in the “Results” section.HRAS transduction before transplantation to enhance malignancy. After the single-cell origin derived from MCF10A-HRAS was transplanted into immunocompromised mice, the xenograft tissues were cultured until the tumor completely progressed and metastasized. DNA and RNA samples were collected at various points during the process. Therefore, we can compare the ITH from two different molecular domains (genome and spliceome) as the tumor grows. A total of 10 samples were collected while culturing xenograft tissue. They collected two samples for metastatic tissue and one sample for each of the eight-time points.The main limitation of the previous synthetic data experiment is the lack of an appropriate evolutionary model in the mixture generation. In this experiment, we used xenograft tumor data (SRA accession: SRP050242) generated by actual clonal evolution [First, we calculated the sITH for each xenograft tumor sample, using normal breast tissue samples from TCGA as the origin . At thishttps://gdc.cancer.gov/about-data/publications/pancanatlas).Unlike xenograft samples from one common ancestral cell, samples from cancer patients were from a diverse population. This patient-to-patient variation in the genetic background can be a confounding factor that can mask actual ITH. It is also unclear whether the origins of the tissue can affect the results of the analysis because only breast cancer tissues were used in previous experiments. To test whether sITH could overcome potential problems and demonstrate its clinical significance, we performed a comprehensive pan-cancer level experiment using the TCGA pan-cancer dataset . WeinsteWe first collected all the RNA-seq data provided by TCGA pan-cancer. The TCGA pan-cancer data validation consists of four steps: 1) comparison of genomic ITH with sITH, 2) comparison of cancer stage with sITH, 3) comparison of survival outcome with sITH, and 4) comparison of PAM50 subtype and sITH. The comparison results are covered in the “Results” section.ITHtranscript and ITHintron for 1,000 samples generated with arbitrary heterogeneity levels shows that the two measures have a significant positive correlation   . This mesITH values were measured for samples made by mixing actual single-cell data with actual breast cancer tissues at a defined ratio and compared to the single-cell blending ratio of each sample. Spearman correlation test showed that the ratio of single cells mixed with sITH was positively correlated  . This meOne of the problems with this experiment using synthetic data is that an appropriate evolutionary model is not taken into account, which is problematic in the following two points. 1) In this scheme, the ITH level is defined in such a way that different single cells are keep being added to the normal tissue. There is no way to know whether this reflects the actual cancer evolution model. 2) In this scheme, combine single cells is done by simply dividing the junction count, which may be different from the actual experimental results, because there is a limit to the number of reads that can be captured by the RNA-seq technology. These factors may be the cause of the linear and gradual increase of the pattern without the saturation shown in Spearman correlation showed a positive correlation between sITH and time-point of each sample . This is plausible because tumor ITH levels are known to increase with cancer progression . SpearmaA study by Weinstein et al.  used ABSSignificant but not very large correlations between sITH and gITH suggest that ITH definition at the genome level and ITH definition at the spliceome level may be different. This is a phenomenon also mentioned in other transcriptomic ITH (ie tITH) studies , suggestThe cancer stage is a well-known indicator of cancer progression, which is determined based on pathological observations of cancer tissues such as size, location, the extent of invasion, and extent of spread. The level of ITH is generally related to the progression of cancer. Since 3,363 samples out of 7,599 samples did not have cancer stage information, 4,236 samples were used in this analysis . SpearmaOverall survival represents survival time after treatment, which in this case implies surgical resection of the tumor. The ITH level is associated with the degree of malignancy of cancer and affects the survival rate of cancer patients . So we tTo test the clinical significance of sITH, we performed a survival outcome association test on 5,265 patient RNA-seq samples. First, the patients were divided into two clusters (scikit-learn python 2.7) with K-means clustering using the sITH value of each sample and log-rank test was performed to test the correlation (p = 3.79e-32) ). The saWe tried to create synergy by modeling the two indicators at once. For each sample, a vector with two variables was used as an input and the same analysis was performed. The log-rank test result was p-value 6.28e-08, which showed better performance than the model using the only gITH, but rather a decrease in performance compared to the model using the only sITH. This means that simply combining the two indicators can not create synergy.Additional analysis has been prepared to help visual understanding. Initially, 7,599 samples were classified into six groups with different survival outcomes. The first five groups were classified by the time of death. For example, the first group contains samples that died in the first year after treatment, and the second group contains samples that died in the second year. The sixth group includes samples reported to be alive for more than 5 years, where the 5-year threshold is based on criteria commonly used to determine cancer remission. As a result, 2,268 samples were classified into six survival groups and the remainder were excluded because they could not be classified into six groups because of the short follow-up period .The molecular subtype is derived from molecular information in cancer. One of the most studied cancer type in terms of the molecular level is breast cancer, where the well-known molecular subtype system, PAM50, has been broadly used . PAM50 cAlso, we arranged the samples in the order of Luminal A, Luminal B, Her2-enriched, and Basal-like using the relative malignancy of each subtype. The correlation of the sITH values to the subtype order was then calculated through Spearman correlation. Both ITHs had significant correlation with PAM50 subtype , while sThe previous tests answered two important questions. The first is whether sITH can measure the ITH of a bulk tumor, and the second is what is the biological and clinical significance of it. The last important question left would be why sITH works, and how does it work. This can be answered using two concepts. One is genetic diversity, and the other is regression toward the mean.There is no debate that cancer progression and malignization is the product of somatic evolution, whose most important driver is genetic diversity. Genetic diversity is a critical factor in helping tumors survive in various environments, and the ITH is the very indicator of such diversity. As genomic DNA can be used to estimate genetic diversity, so does spliceome. sITH was designed to capture the increase of spliceomic diversity in each intron. The other important factor is the regression toward the mean. Considering the huge differences between cancer patients, We thought that it might not be easy to pinpoint which intronic loci are important in each patient. So we just averaged the entire intronic loci with expecting to capture the average level of genetic diversity for each bulk-tumor. And the results supported our expectation.Our study suggests that any type of genetic diversities can be used to predict malignancies of bulk-tumors and it also suggests that integrating various sources of genetic diversities might lead to a more comprehensive model of ITH and a better understanding of malignization.Despite studies that show intercellular differences at the spliceome level , 26, thehttp://biohealth.snu.ac.kr/software/SpliceHetero. It was implemented in Python 2.7 and tested on CentOS Linux release 7 and Ubuntu 16.04, and 18.04. Lastly, note that all the abbreviations used in this article are listed in The proposed model can help to develop diagnostic and prognostic tools by providing a tool to understand the inherent heterogeneity of cancerous spliceome. The whole process is implemented as a software package and is available free at S1 Table(XLSX)Click here for additional data file.


This global bias can lead to drastic over- or under-representation of RNA in non-linear length-dependent fashion due to enzymatic reactions during cDNA production. It is currently not corrected by any RNA-seq software, which mostly focus on local bias in coverage along RNAs. This paper describes LiBiNorm, a simple command line program that mimics the popular htseq-count software and allows diagnostics, quantification, and global bias removal. LiBiNorm outputs gene expression data that has been normalized to correct for global bias introduced by the Smart-seq2 protocol. In addition, it produces data and several plots that allow insights into the experimental history underlying library preparation. The LiBiNorm package includes an R script that allows visualization of the main results. LiBiNorm is the first software application to correct for the global bias that is introduced by the Smart-seq2 protocol. It is freely downloadable at  RNA sequencing (RNA-seq) is now widely used for determining gene expression levels. While originally developed for assessing average gene expression over a large number of cells from a cell culture or tissue sample, it is increasingly being used for measuring gene expression within individual cells .Most sample preparation protocols for RNA-seq involve RNA isolation and reverse-transcription, followed by fragmentation, amplification, and sequencing, not necessarily in this order; this is followed by aligning the sequencing reads to a reference sequence and allowing the reads to be associated with a gene or transcript so that expression levels can be calculated.A major issue with most RNA-seq strategies is the question of how to translate sequencing read numbers into abundances of the original RNAs.There are currently two main strategies for using RNA-seq data to quantify transcript expression levels: (i) tagging RNAs with unique molecular identifiers (UMIs), which allows labelling and in turn counting absolute numbers of original RNA molecules ; (ii) seThe first strategy is implemented by protocols such as CEL-seq  and is aThe default assumption is that there is a linear dependence between the expected read numbers and the RNA length, as embodied by length-normalizing measures such as Fragments per Kilobase Million (FPKM)  and TranHowever, virtually all bias correction efforts target deviations from a uniform read distribution along RNAs, also referred to as “coverage.” It was only recently recognized that cDNA conversion of protocols such as Smart-seq2 not only affects coverage, but introduces a global, length-dependent bias that is potentially orders of magnitude stronger than “local” coverage bias. We established this as a problem in our previous work , which iWhile local biases are mostly related to DNA sequence and affect the likelihood that the DNA will fragment or the likelihood that primers will associate at a particular location, we showed, based on a set of abstracted models, that global biases arise from aspects of polymerase processivity during cDNA conversion; the latter are less dependent on the DNA sequence and often cause a variation in read coverage along the transcript, but not always. This resulting length-dependent overall bias for a transcript can introduce significant errors when calculating relative expression levels of genes with different transcript lengths. In fact, the read numbers obtained for longer RNAs in Smart-seq protocols deviate so much from FPKM/TPM measures that the assumption of linear scaling appears wrong altogether and will lead to an underestimation of long genes’ expression levels .Several methods and software releases have been published which attempt to cater for different sources of bias that can occur when assessing expression levels using RNA-seq data. To our knowledge, none of these consider the global bias that we address here. Instead, they mostly assume an overall linear scaling between read numbers and RNA lengths and focus on local deviations from uniform coverage. These are documented in more detail in the Here, we present the standalone tool “LiBiNorm” that builds on our previous work and permits easy and user-friendly implementation of analysis strategies aimed at global bias. Its main target is Smart-seq2 data, but it can be applied broadly to diagnose library preparation characteristics underlying different types of datasets.We have developed LiBiNorm , a software package that can quantify the degree of global bias present in a Smart-seq2 RNA-seq dataset using the framework described in LiBiNorm takes aligned bam-format Smart-seq2 RNA-seq data as input together with a reference genome annotation in gtf or gff3 format. The RNA-seq data can be single or paired end and must be aligned to a reference genome by an aligner such as Tophat2 , its sucLiBiNorm associates individual reads with the transcripts associated with each gene and from their distribution determines parameters that are appropriate for normalizing the effect of bias on gene expression values. These parameters are based on mathematical models, as introduced in , which cUsing the gene transcript lengths that can be derived from the genome annotation, it determines parameters corresponding to the model using non-overlapping, single isoform genes only and then produces a modified TPM measure of the gene expression which has been normalized to compensate for the global bias.It is possible to specify that an alternative to the default BD model be used for bias normalization, or that all models should be evaluated and their goodness-of-fits based on log likelihoods used to select the best model for a specific data set.LiBiNorm is in principle suited to all Smart-seq data. However, we have found that local bias in Smart-seq1 data is so strong that it impedes global bias correction or masks its effects. This is probably due to RNA secondary structures that cause erroneous reverse transcription initiation and termination, which disturbs the read distribution, thus frustrating accurate determination of the model parameters. This problem is overcome in Smart-seq2 through the addition of betaine .The primary input data for LiBiNorm is a bam file containing aligned RNA-seq data. Core command line options allow identifying the type of RNA-seq data , choosing the genomic feature to which the reads should be aligned, and setting the rules governing whether a read is associated with a feature or not. As well as the bam file, LiBiNorm also requires a gene annotation file that allows the reads to be assigned to specific genes which can either be in gff3 or gtf format.The LiBiNorm takes as input single or paired end RNA-seq data in bam format after having been aligned to a complete reference genome. LiBiNorm must be informed if the data is paired end and not ordered by read name. RNA-seq protocols can generate reads that are aligned with, or are in the opposite sense to, the sense of the original RNA, or alternatively may not indicate the original sense at all. The read sense status must be provided to LiBiNorm through the associated command line option.The bias correction performed by LiBiNorm is a function of the lengths of the transcripts and the distribution of the RNA fragments within these transcripts.By default, LiBiNorm associates reads with the combined exons of all of the transcript variants of a gene. The length of the mRNA transcript is assumed to be the total length of the combined exons, and the position of a read is taken as the position within this total length. The presence of some transcript variants with retained introns is recorded in some reference annotations, but such variants are unusual and give misleading gene lengths and read positions, so are discarded by LiBiNorm.LiBiNorm adopts the same three alternative algorithms for determining whether a read is associated with a feature as is available in htseq-count. However, the LiBiNorm default is the “strict” option, which requires the read to lie completely within annotated exons and not extend outside the exons in order to be associated with a gene. This avoids incorrect association of reads with small RNA genes which can provide misleading information when estimating model parameters.Our models are specified by up to five parameters that need to be estimated for each given dataset . Paramet9) and reads from each gene (200) that are selected and the number and lengths of MCMC runs  that are performed. These choices of settings combine high precision and quick run time, but can be changed by the user. Finally, the initial Nelder-Mead step is performed by default as it reduces the fraction of MCMC runs that get trapped in local modes; it can be switched off by command line option, though, if random starting points are preferred.LiBiNorm ignores mRNAs longer than 20 kb in this process in order to limit the potential influence of very long transcripts with spurious reads. We have found the default cutoff of 20 kb to be adequate for all our analyses, however a command line option permits changing this setting. We have also included default settings for the maximum total reads  are inferred from the data and make it possible to “reverse-engineer” aspects of the sample preparation process. These parameters correspond to different aspects of the sample preparation process. This includes reduced fragmentation efficiencies (by factor d + 1) at fragment ends (over distance h from ends), reverse transcriptase and second-strand polymerase processivities , and PCR efficiency of Smart-seq protocols  as a representative dataset took 1,613 s with htseq-count, whereas LiBiNorm in htseq-count compatible mode took 125 s. The output files are identical.LiBiNorm can also produce simple read count data from the RNA-seq data that can be used by packages such as DESeq2  for calcLiBiNorm count -z -i gene_id -s no -c <countfilename> <fileroot>.bam "../Drosophila_melanogaster.BDGP6.91.gtf"andhtseq-count -i gene_id -f bam -s no <fileroot>.bam "../Drosophila_melanogaster.BDGP6.91.gtf" > <countfilename>The commands used were:Library bias normalization can, if specified through the command line, generate additional files that contain more information about the bias analysis it has performed. The full set of output files is shown in The output files can be processed by an R script that forms part of the LiBiNorm software package in order to produce various graphical representations of the bias and parameter estimation process. The plots include a heatmap that visualizes the global bias and is by itself a highly useful tool for RNA-seq QC and diagnostics.LiBiNorm is implemented in C++. It uses a slightly modified version of the bamtools library for reading bam files and uses the standard C++11 template library. The associated makefile supports compilation using gcc or clang in order to generate executables for Linux or macOS. A Microsoft Visual Studio solution file supports compilation for Windows platforms, for which the source for the zlib compression library is also required.Drosophila study that compares different RNA-seq protocols. This study is particularly well suited for our analyses, as it includes  global bias free RNA-fragmentation datasets (TruSeq) and Smart-seq2 data SRR1743153–SRR1743166 from the short read archive . Comparing these datasets will be used to demonstrate the reduction in global bias provided by LiBiNorm compared to that of other software packages.We illustrate usage of LiBiNorm based on datasets from a SRR1743153–SRR1743166 were aligned using HISAT2 (http://ccb.jhu.edu/software/hisat2/index.shtml). The output was converted to bam file format using samtools version 1.3.1  was analyzed using the LiBiNorm “−f” option which produces results for the full set of six models that are described in LiBiNorm count -p 3 -f -i gene_id -u <fileroot> -s no -c <countfilename> <fileroot>.bam Drosophila_melanogaster.BDGP6.91.gtfSample LiBiNormPlot.R was then used to produce plots from the data files and which are shown in SRR1743160Rscript LiBiNormPlot.R SRR1743160 data in two different formats. The heatmap (Plots in  heatmap  displays heatmap  shows “s10 (except a) parameter estimates and predicted bias for SRR1743160 is shown in <fileroot>_norm.txt contains parameter estimates and predicted bias relative to a linear model for each of the six models. The Model BD log−0.103 + 1 = 1.788 (d), over a length of 101.89 = 78 bases (h) at each end of the fragments. The first- and second-strand processivities (inverses of t1 and t2) are 104.27 = 18.6 kb (t1) and 103.52 = 3.31 kb (t2), respectively. Finally, PCR selection has increased the proportion of complete second strands to 59% (a). While these values appear sound, this diagnostics information potentially allows troubleshooting; for example, processivities or PCR efficiency could be increased by optimizing reaction conditions etc.These parameters allow insights into characteristics of the library preparation process and can be interpreted in the following way: the library’s fragmentation/tagmentation efficiency was reduced by 10The bias vs. length table  further TruSeq data is virtually bias-free, since it is based on RNA fragmentation, which drastically reduces the effects of the cDNA conversion. We used the four TruSeq samples in the Combs et al. data as the gold standard for mRNA expression levels; other popular benchmarking methods, such as qPCR quantification are inadequate, as they are subject to the same global cDNA bias.2 expression levels, expressed as TPM  and its fitted model (BD), we demonstrate based on scatter plots how correlation improves after bias correction .For comparison, we performed the same process using four of the software packages which contain some degree of bias removal: Cufflinks , Salmon 2 . Both Sa2  (See Sup% poorer . LiBiNorLiBiNorm is a multi-platform software application designed to identify, visualize, and correct global biases in RNA-seq data that are introduced by the library preparation protocols.LiBiNorm functions by learning parameters from the datasets that allow reverse engineering parts of the library preparation history. The parameter estimates are characteristic for the library preparation protocol and allow LiBiNorm to infer the protocol that was used. This permits insights and diagnostics of the sample preparation history and enables LiBiNorm to calculate and apply the appropriate bias correction, which is most relevant for the Smart-seq2 protocol. A benchmarking effort confirms that LiBiNorm is superior to other available software for this purpose.A companion R script is able to produce graphical representations of the results and analyses LiBiNorm generates. LiBiNorm was designed to be consistent with the gene expression calculations provided by htseq-count  and incl10.7717/peerj.6222/supp-1Supplemental Information 1Click here for additional data file.


Environmental health and exposure researchers can improve the quality and interpretation of their chemical measurement data, avoid spurious results, and improve analytical protocols for new chemicals by closely examining lab and field quality control (QC) data. Reporting QC data along with chemical measurements in biological and environmental samples allows readers to evaluate data quality and appropriate uses of the data . However many studies do not adequately describe or interpret QC assessments in publications, leaving readers uncertain about the level of confidence in the reported data. One potential barrier to both QC implementation and reporting is that guidance on how to integrate and interpret QC assessments is often fragmented and difficult to find, with no centralized repository or summary. In addition, existing documents are typically written for regulatory scientists rather than environmental health researchers, who may have little or no experience in analytical chemistry.We discuss approaches for implementing quality assurance/quality control (QA/QC) in environmental exposure measurement projects and describe our process for interpreting QC results and drawing conclusions about data validity.Our methods build upon existing guidance and years of practical experience collecting exposure data and analyzing it in collaboration with contract and university laboratories, as well as the Centers for Disease Control and Prevention. With real examples from our data, we demonstrate problems that would not have come to light had we not engaged with our QC data and incorporated field QC samples in our study design. Our approach focuses on descriptive analyses and data visualizations that have been compatible with diverse exposure studies with sample sizes ranging from tens to hundreds of samples. Future work could incorporate additional statistically grounded methods for larger datasets with more QC samples.This guidance, along with example table shells, graphics, and some sample R code, provides a useful set of tools for getting the best information from valuable environmental exposure datasets and enabling valid comparison and synthesis of exposure data across studies. Chemical measurements play a critical role in the study of links between the environment and health, yet many researchers in this field receive little if any training in analytical chemistry. The growing interest in measuring and evaluating health effects of co-exposure to a multitude of chemicals , 2 makesSome important considerations when measuring and interpreting environmental chemical exposures have been discussed in other peer-reviewed articles or official guidance documents. For example, a recent document from the Environmental Protection Agency (EPA) provides citizen scientists with guidance on how to develop a field measurement program, including planning for the collection of quality control (QC) samples . The CenDespite these helpful documents, there is still a lack of readily accessible, practical guidance on how to interpret and use the results of both field and laboratory QC checks to qualify exposure datasets  and this gap is reflected in the environmental health literature. While the vast majority of environmental health studies report robust findings based on high quality measurements, questions about measure validity have led to confusion and lack of confidence in some topic areas. For example, a number of studies have measured rapidly metabolized chemicals such as phthalates and bisphenol A (BPA) in blood or other non-urine matrices, despite the fact that urine is the preferred matrix for these chemicals. Phthalates and BPA are present at higher levels in urine and, when the proper metabolites are measured, there is less concern about contamination from external sources, including contamination from plastics during specimen collection .More commonly, however, exposure studies simply do not adequately report on QA/QC or describe how QC results informed reporting and interpretation of the data. In the context of systematic review and weight of evidence approaches, not reporting on QA/QC may result in a study being given less weight. For example, the risk of bias tool employed in case studies of the Navigation Guide for Systematic Review includes reporting of certain QA/QC results in its criteria for a “low risk of bias” rating . When wOver many years of collecting and interpreting environmental exposure data, we have developed a standard approach for (1) using field and laboratory QA/QC to validate and qualify chemical measurement data for environmental samples and 2) presenting our QC findings in our research publications . These  presentiWe present a detailed approach based on our own studies, acknowledging that this is an example, not a one-size-fits-all approach. Every study is unique and some will require specialized quality assessment not covered here. Still, we anticipate that many environmental health scientists will find this example to be a useful framework for building their own processes.Our guide is organized by a series of questions that we ask when we start a new study and then again when we receive measurement data from the lab. Key QA/QC concepts are introduced in the Not every question is relevant to every study; for example, researchers working with a lab to develop a new analytical method will need to focus more on method validation and quality control than those using a well-established method and credentialed lab. Still, controlling for issues related to sample collection and transport remain important in the latter scenario, as does variation in method performance and/or sources of contamination when samples are analyzed at the laboratory in multiple batches. Our guidance is most relevant to targeted organic chemical analyses, which use liquid or gas chromatography, often in combination with mass spectrometry, to determine whether a pre-defined set of chemicals are present in samples. QA/QC approaches for non-targeted methods, where tentative identities are established by matching to a library of mass spectra such as the National Institute of Standards and Technology (NIST) database , are addThis guide is not a set of rules, but rather establishes a framework for evaluating and reporting QC data for chemical measurements in environmental or biological samples. While it may be most useful to environmental health scientists who have little or no experience in analytical chemistry, we hope that researchers with a range of experience will find it helpful to consult our approach for evaluating and presenting QC data in publications.Because the number of QC samples available is often limited by budgetary constraints, many of the methods we use rely on visualization and conservative action  rather than on statistical methods. Whether statistical methods are incorporated or not, tabulating, visualizing, and communicating about QA/QC for environmental exposure measurements is important in order to reveal systematic error in the laboratory  or in thOne of our first priorities when designing a new study is to consult with a chemist to establish an analyte list and method for analysis.Given the complexity of chemical synonyms, it is helpful to be as specific as possible when communicating about the chemicals to be analyzed. One approach is to send the lab a list of the chemical names , Chemical Abstracts Service (CAS) numbers, and configurations  of all desired analytes  or will be collected. As discussed previously, certain biological matrices are preferred over others for measurement, depending on the chemicals . MatrixThe process of determining a final list of analytes will differ depending on whether the lab has an established method or is developing a new method, and whether it is targeted to a few chemicals with similar structure versus many chemicals with different properties . Targeting a broad suite of chemicals may limit the degree of precision and accuracy that can be achieved for each individual chemical, and the lab may need to invest substantial effort to develop a multi-residue method – that is, a method that can analyze for many chemicals at once – and determine a final list of target chemicals with acceptable method performance. In any case, a new method should be validated to characterize performance measures – precision, accuracy, expected quantitation and method detection limits, and the range of concentrations that can be quantitated with demonstrated precision and accuracy – before analyzing study samples. If the lab already has an established method for the chemicals of interest, the research team should review method performance measures to ensure they are consistent with study objectives.The method of quantification affects the types of QC data that are expected from the lab. Three common approaches include external calibration, internal calibration and isotope dilution . External calibration, where the response  from the sample is compared to the response from calibration standards containing known amounts of the analytes of interest, is a simple method that can be used for a variety of different analyses. However results can be influenced by interference from other chemicals present in the sample matrix and resulting fluctuations in the analytical instrument response . With inWhat QA/QC is needed?Another important factor in selecting a method is to make sure it is sensitive enough to detect the anticipated concentrations in the field samples (samples submitted to the lab) down to levels that are relevant to the research question. For example, commercial labs measuring environmental chemicals may establish reporting limits to meet the needs of occupational or regulatory safety compliance testing; these limits may be much higher than levels that are meaningful for research questions about general population exposure and could result in most data being reported as non-detect or qualified as estimated and imprecise. On the other hand, lower reporting limits generally translate to more expensive testing, so researchers have the opportunity to balance sensitivity and cost.There are ample opportunities for sample contamination during collection, storage, shipment and analysis, especially when targeting ubiquitous chemicals commonly encountered in consumer products and in home and office furnishings or laboratory equipment. An important aspect of method validation is to check for contamination of samples during field activities, from collection containers, during transport and storage, and during laboratory extraction and analysis  along with chemical names  or method reporting limit (MRL), which is either the lowest value that the lab will report or the lowest value that the lab will report without flagging the data as estimated, is often  the same as the quantitation limit or LOQ.Common terms used by laboratories to discuss reporting limits include instrument detection limit (IDL), method detection limit (MDL) and limit of quantitation (LOQ). The IDL and MDL are both related to the level of an analyte that can be detected with confidence that it is truly present. The IDL captures the smallest true signal  that can be distinguished from background noise (variation in the instrument response to blank samples), while the MDL takes into account additional sources of error introduced during sample preparation  and thus is higher than the IDL. The MDL is also often referred to as the limit of detection (LOD) or detection limit (DL). The LOQ, on the other hand, describes the lowest mass or concentration that can be detected with confidence in the Before submitting samples for analysis, it is helpful to find out (1) the methods and terminology that the laboratory will use to describe reporting limits  and (2) whether reporting limits will be consistent within a chemical or whether limits could vary between samples or batches. Equally critical is to clarify how the lab will report non-detects. Several different values could appear in the amount or concentration fields for non-detects, including but not limited to zeroes, the detection limit, the reporting limit or “ND.”Another important point to discuss in advance with the laboratory is how they will report values for compounds with a confirmed identity but measured at levels below what can be accurately quantitated. For example, when measuring chemicals of emerging interest, we ask laboratories to report estimated values below the RL and we flag them during data analysis. This practice has some limitations  but is pQA/QC occurs both inside and outside the analytical laboratory  for target chemicals to evaluate the accuracy of the analytical method, surrogate compounds added to field samples to estimate recovery during extraction and analysis, and blanks to assess contamination with target chemicals from some source in the laboratory. While laboratories generally conduct rigorous review of their own QC data, considering lab and field QC together can help to identify specific sources of contamination, imprecision, and systematic error, so we typically request to review the lab’s raw QC data in conjunction with the field QC data.Spiked samples and CRMs establish the accuracy of the method by assessing the recoveries of known amounts of each target chemical from a clean or representative matrix. A CRM is a matrix comparable to that used for sampling  that has been certified to contain a specific amount of analyte with a well-characterized uncertainty. If CRMs aren’t available, the laboratory can prepare laboratory control samples (LCSs) by spiking known amounts of target chemicals into a clean sample of the matrix of interest, such as a dust wipe, air sampler, purified water or synthetic urine or blood that has been analyzed and shown to be free of the analytes of interest, or to contain a consistent amount of analytes of interest that can be subtracted from the amounts measured in the spiked sample to calculate a percent recovery. The LCS or CRM – at least 1 per analytical batch – is run through the same sample preparation, extraction, and analysis as the field samples to capture the accuracy of the complete method; calculating the percent of the known/spiked amount recovered for each analyte tells us whether the method is accurate in the matrix.Another type of spiked sample, called a matrix spike, can be used to check the extraction efficiency for a complex sampling matrix that may interfere with the analysis. These samples are typically included if there is concern about interference from the sampling matrix, for example, with house dust, soil or sediment samples, consumer products, or biological samples like blood. Instead of recovery from a clean matrix, these QC checks capture recovery from a representative field sample. Here the “matrix” refers to all elements of the sample other than the targeted analytes; this includes the sampling medium  itself as well as any other chemicals present in the sample that might interfere with measurement of target chemicals. A matrix spike can be created, for example, by splitting a representative sample collected in the field and spiking the target analytes into one half prior to extraction and analysis. The recovery of spiked analyte is determined as the amount measured in the spiked sample minus the amount measured in the non-spiked sample divided by the spike amount. A limitation of this approach is that the analytes are spiked in an already dissolved state, so it is possible that the analytes in the environmental matrix would not be extracted as readily from the matrix as the spiked chemicals. Thus, the true extraction efficiency may be lower than represented by the matrix spike.For newly developed methods where performance is not characterized, we request results for all recoveries of spiked samples and/or CRMs so that we can perform visual checks that have at times revealed systematic problems with the analytical method that were not noted by the lab  that are transported to the sampling location and exposed to all of the same conditions as the real samples  except the actual collection process. We aim for at least 10% of our samples to be field blanks, with an absolute minimum of 3 field blanks.Collecting and preparing several types of blank samples helps us to distinguish sources of contamination. Laboratory blanks alert us to possible contamination originating in the lab. These blanks can capture contamination during sample extraction (solvent blanks), from reagents and other materials used in the analytical method (solvent method blanks) or from “typical” background levels of target analytes present in the sampling matrix (matrix blanks). Field blanks, on the other hand, capture Unfortunately, in some cases there aren’t good options for representative field blanks. For example field blanks can be created for biomonitoring programs by taking empty collection containers into the field and using purified water or synthetic urine or blood to create a blank . HoweverCollecting side-by-side duplicate samples in the field helps assess the precision of both the sample collection and analytical methods. Duplicate samples can also be created by collecting a single sample and splitting it prior to analysis, which is the only option for biological samples; however, this method only captures the precision of the analysis process , 17 and Analytical performance can shift over time and even between multiple extractions or instrument runs within a short time window. Laboratories often analyze samples in multiple batches, that is, sets of field samples and associated laboratory QC samples that are analyzed together in one analytical run. The time between batches can vary from days to months or even years, though ideally this time span is minimized in order to maintain consistent equipment and procedures throughout the study.Two approaches help address batch-to-batch variability: (1) randomizing participant samples between batches by specifying the order and grouping of samples (and blind field QC samples) when submitting samples to the lab (this may require corresponding with the lab to determine the batch size in advance), and (2) running CRMs – such as standard reference material (SRM) from NIST  – in eacIn Additional files No amount of QA/QC can save a dataset from basic misunderstandings about what is being reported. After receiving data, it is helpful to ask the chemists to double check the analyte list  against the list of standards used in the analysis, particularly if this information was not included in the report from the lab. It is worthwhile to make this verification even when chemical identities were specified in advance of the analysis as it is possible that the standard used for analysis was slightly different than planned. Only through this process, for example, did we discover that a lab had accidentally purchased a standard for 2,2,4-trimethyl-1,3-pentanediol isobutyrate rather than 2,2,4-trimethyl-1,3-pentanediol diisobutyrate .Table Examining results by batch or even by sample run order can reveal trends in QC samples over time, identifying systematic laboratory errors that may be missed by summary statistics or visualizations . Shifts Table Figure A summary of the recovery information should be included in the peer-reviewed manuscript to demonstrate accuracy. See Additional file When isotope dilution quantitation with automatic recovery correction is not employed, we review the surrogate recovery standard data for each individual sample, generally considering 50–150% recovery to be acceptable. Interpretation of an out-of-range surrogate recovery depends both on its direction and on the levels of the associated analytes  measured in the sample. In samples with low surrogate recoveries, the concern is that if similar target analytes are present in the sample, the measurements will be underestimated/biased low. For samples with high surrogate recoveries, on the other hand, we can be confident that similar target compounds should be detected if present, but the amount may be overestimated or biased high. If surrogate recoveries are out-of-range in all samples, and particularly if they are also out-of-range in blank samples, this is likely indicative of a broader problem with the analytical method , 29. TabFigure Once we have determined that we can accurately measure the target analytes in our sampling matrix, the next step is to ensure that we are confident about whether those target analytes came from the study site or participant – or from somewhere else. Table Figure Typically, we use blanks to qualify values rather than remove measurements from our data. Specifically, we use detected values in field blanks and sometimes other blanks  summary table shell. Table S4. Findings and actions from review of quality assurance and quality control (QA/QC) data table shell. Section III: Figure S1. Distribution of surrogate recoveries by study visit. Additional file 3: Example of report formatting request to send to the lab. Additional file 4: Example QA/QC report.