Impact of Clinical Data Veracity on Cancer Genomic Research

Abstract Genomic analysis of tumors is transforming our understanding of cancer. However, although a great deal of attention is paid to the accuracy of the cancer genomic data itself, less attention has been paid to the accuracy of the associated clinical information that renders the genomic data useful for research. In this brief communication, we suggest that omissions and errors in clinical annotations have a major impact on the interpretation of cancer genomic data. We describe our discovery of annotation omissions and errors when reviewing an already carefully annotated colorectal cancer gene expression dataset from our laboratory. The potential importance of clinical annotation omissions and errors was then explored using simulation analyses with an independent genomic dataset. We suggest that the completeness and veracity of clinical annotations accompanying cancer genomic data require renewed focus by the oncology research community, when planning new collections and when interpreting existing cancer genomic data.

Translational research combining clinical data with genomic data [eg, (1)(2)(3)] has dramatically increased our understanding of cancer and lays the foundation for precision oncology (4,5). Clinically annotated cancer genomic datasets from individual laboratories (4), biobanks (6), or large international initiatives (1,7) can now be accessed without bioinformatics expertise using web interfaces such as cBioPortal (8), Xena (9), or StratomeX (10). However, as clinical and genomic data were integrated by large-scale initiatives, inaccuracies were recognized (11), and unexpected issues such as hidden duplicates emerged (12). In clinical databases and registers in general, clinical data errors are surprisingly pervasive (13)(14)(15)(16), including missing clinical annotations that reduce analysis power (17). Despite the considerations described above, relatively little has been done to evaluate the effects of clinical data omissions and errors on the interpretation of cancer genomic datasets.
When a senior clinician reviewed the clinical data accompanying a colorectal cancer gene expression dataset from our laboratory, we discovered a substantial number of omissions and errors, despite the high degree of care initially taken when manually extracting data from the contemporaneous clinical record. Data review using clinical notes, direct communication with medical staff, and publicly available records such as death certificates identified 1 or more missing data points in 142 of 205 (69%; Figure 1, A) tumor records and errors in 32 of 205 (16%; Figure 1, B) tumor records. Additional clinical data were identified that clarified metastatic status for 103 of the 205 (50%) tumors and, on review, the American Joint Committee on Cancer stage (7th edition) was revised for 12 tumors, and the designation of recurrence status was corrected for 21 tumors, which impacted on disease-free survival (DFS) for 16 of 205 (8%) tumors. The changes made are summarized diagrammatically in Figure 1, C, and the overall effect on DFS is shown in Figure 1, D. As an example, the impact of these corrections to clinical data on conclusions drawn from downstream genomic analysis was assessed using Affymetrix gene expression data from these 205 tumors. Cox proportional hazards analysis of associations between transcriptome-wide gene expression and DFS identified 79 RNAs with statistically significant associations with DFS in the original (uncorrected) data (at P .01) being lost after revision and a different set of 73 RNAs gaining a statistically significant association (at P .01) with DFS following revision.
To estimate the impact of omissions and errors in clinical data fields associated with genomic datasets in general, we  undertook a simulation analysis using a publicly available colorectal cancer gene expression dataset (GSE17536, n ¼ 177) (18). The effect of clinical data omissions and errors on significant associations between gene expression and DFS was determined using Cox proportional hazards analysis with a statistical significance cut-off of a P value less than .05, with and without multiple testing correction. Fixed proportions of clinical annotations were omitted or changed 1000 times in randomly chosen patients. The results confirm that omissions or changes to relatively small subsets of high-impact clinical data fields have a substantial impact on RNAs associated with DFS ( Figure 2). For example, omission of clinical annotations for 20% of patients caused almost all previously statistically significant (P .05 after multiple testing correction) RNAs associated with DFS to be lost and reduced sensitivity for detecting survival associations by one-third on average, when compared with the original data set (Figure 2, A and B). Changes to disease status (Figure 2, C and D) and time to death (Figure 2, E and F) also had a substantial impact on the sets of genes with statistically significant association with DFS and on sensitivity. For example, random switching of survival status for only 10% of patients caused more than 50% of the RNAs associated with DFS to be lost ( Figure 2, C and D).
Missing or erroneous clinical information accompanying genomic datasets can originate either in the original clinical record or during the extraction, transcription, or interpretation of the data (15). In our study, data was collected from 1996 to 2013, with a period (2000-2010) of substantially reduced clinician oversight. Secondly, several recurrent tumors were found on expert review to be second primary cancers. This is particularly important when investigating associations between gene expression and behavior of the incident tumor. Thirdly, substantial changes in clinical practice occurred during this study, including the introduction of routine pre-operative computed tomography staging, which may have led to stage migration during the study (19). Similar iterative improvements to imaging and pathology technologies (20) are likely to affect many genomic studies. Changes to the type and accuracy of clinical data recorded electronically may also have led to ascertainment biases (16,21), as may the more readily available survival data for patients who have died compared with patients who are still alive but temporarily lost to follow-up (22).
Lessons we have learned from this exercise include clear documentation of any systematic changes in clinical practice over time, providing embedded clinician oversight at the point of data entry, following standardized training for all data collection staff, creating logically arranged synoptic data collection forms, censoring of the survival status of all patients at the same point in time, and undertaking periodic expert data reviews.
We suggest that the effects of omissions and errors may similarly impact more nuanced associations between clinical and genomic information, such as associations between mutational profiles and the efficacy of targeted therapies. Our findings may not be surprising to statisticians, however, many clinicians and cancer biologists who regularly use genomic data are blissfully unaware of these effects. We suggest to our colleagues that 1) it is worth considerable investment to ensure that clinical annotations of genomic data are as complete and accurate as possible, especially when genomic resources will be used in population-scale analyses (23), developing clinical tests, or in linked clinical-genomic workflows (24) and 2) ideally, standardized sets of clinical variables should be included in all clinical metadata. Our conclusions also have implications for interpretation of large international cancer genomic projects such as The Cancer Genome Atlas or International Cancer Genome Consortium, as well as for smaller scale genomic analyses in individual laboratories. In addition, the revision of clinical annotations may change the conclusions drawn from already published cancer genomic studies.

Notes
Role of the funder: The funders had no role in the design of the study; the collection, analysis, and interpretation of the data; the writing of the article; or the decision to submit the article for publication.
Disclosures: The authors have no conflicts of interest to disclose.

Data Availability
The study follows the FAIR principles, however the data set used in this analysis will be made available through the corresponding author, after consideration by a data access committee of its potential use. The data are not made freely available through international genomic data sharing initiatives due to ethical requirements of the studies in which this data was generated, which respect cultural considerations about genomic data of New Zealand's indigenous Ma ori people.