Rationale: Machine learning may augment conventional epidemiologic methods of detecting COVID-19. Prior studies of machine learning to screen for COVID-19 have relied largely upon radiographic and laboratory data. Few have used clinical symptoms alone to screen for COVID-19, and most are limited by sample size. We adapted an existing, open source machine learning pipeline to identify symptoms associated with COVID-19 using a large national cohort of Veterans with polymerase chain reaction (PCR)-confirmed COVID-19. Methods: We included members of the Veterans Affairs (VA) Birth Cohort (n=4,221,135) who had PCR-confirmed COVID- 19 between 01/01/2020-06/15/2020. Veterans with an outpatient VA visit from 03/01/2018-02/29/2020 were analyzed. To identify symptoms associated with COVID-19, we adapted the open source v3NLP platform. This symptom pipeline consists of several features (Figure) and was restricted to identify fever, cough, anosmia/ageusia, dyspnea, congestion, sore throat, diarrhea, nausea, loss of appetite, headache, dizziness, chills, confusion/altered mental status, myalgia, and fatigue. Before running this pipeline, a multidisciplinary team of 9 clinicians first developed an annotation guideline. Each will use the guideline to review of 100 random notes that include information about symptoms to evaluate interrater agreement. This guideline will be revised until a kappa of 0.85 for inter-rater reliability is achieved. The multidisciplinary team will then review a random subset of 900 notes to assess agreement between pipeline- and human-identified symptoms. Each clinician will review and annotate 150 notes so that each note is reviewed by at least 2 clinicians. We will run the symptom pipeline on text from clinical notes of Veterans with PCR-confirmed COVID-19 and evaluate the performance of symptom identification with snippet annotation. Results: The sample included all clinical notes (n=1,641,935) identified two weeks before through two weeks after a PCR-confirmed diagnosis of COVID-19. Initial testing found the opensource symptom pipeline achieved an F-measure (harmonic mean of positive predictive value and sensitivity) of 0.71 to detect symptoms and an overall F-measure of 0.87 indicating excellent performance when combined with detecting negative symptoms. Conclusions: This work demonstrates the method by which an existing open source machine learning pipeline can be adapted to identify symptoms associated with COVID-19. This work has broad applications and is distinguished by its focus on symptoms commonly reported in clinical notes. Future studies may consider applying this methodology to screen for symptoms in populations of interest (e.g., persons with prior COVID-19 or immunization against COVID-19) or to develop an electronic medical recordbased syndromic surveillance system.
Machine learning to identify symptoms associated with coronavirus disease 2019 (covid-19) in a national cohort of veterans