which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses
EXTRACTIVE
contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses
EXTRACTIVE
We aim to address this issue. Our research goal is to develop natural language processing methods to collectively analyze the study results reported by many hospitals and medical institutes all over the world, reconcile these results, and make a holistic and unbiased conclusion regarding the correlation between radiological findings and COVID-19. Specifically, we take the CORD-19 dataset BIBREF2, which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. We develop sentence classification methods to identify all sentences narrating radiological findings from COVID-19. Then constituent parsing is utilized to identify all noun phrases from these sentences and these noun phrases contain abnormalities, lesions, diseases identified by radiology imaging such as X-ray and computed tomography (CT). We calculate the frequency of these noun phrases and select those with top frequencies for medical professionals to further investigate. Since these clinical entities are aggregated from a number of hospitals all over the world, the population bias is largely mitigated and the conclusions are more objective and universally informative. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19.
We aim to address this issue. Our research goal is to develop natural language processing methods to collectively analyze the study results reported by many hospitals and medical institutes all over the world, reconcile these results, and make a holistic and unbiased conclusion regarding the correlation between radiological findings and COVID-19. Specifically, we take the CORD-19 dataset BIBREF2, which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. We develop sentence classification methods to identify all sentences narrating radiological findings from COVID-19. Then constituent parsing is utilized to identify all noun phrases from these sentences and these noun phrases contain abnormalities, lesions, diseases identified by radiology imaging such as X-ray and computed tomography (CT). We calculate the frequency of these noun phrases and select those with top frequencies for medical professionals to further investigate. Since these clinical entities are aggregated from a number of hospitals all over the world, the population bias is largely mitigated and the conclusions are more objective and universally informative. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19.
2004.01862
Introduction </s> Cor on av irus Ġdisease Ġ2019 Ġ( CO VID - 19 ) Ġis Ġan Ġinfectious Ġdisease Ġthat Ġhas Ġaffected Ġmore Ġthan Ġone Ġmillion Ġindividuals Ġall Ġover Ġthe Ġworld Ġand Ġcaused Ġmore Ġthan Ġ55 , 000 Ġdeaths , Ġas Ġof ĠApril Ġ3 Ġin Ġ2020 . ĠThe Ġscience Ġcommunity Ġhas Ġbeen Ġworking Ġvery Ġactively Ġto Ġunderstand Ġthis Ġnew Ġdisease Ġand Ġmake Ġdiagnosis Ġand Ġtreatment Ġguidelines Ġbased Ġon Ġthe Ġfindings . ĠOne Ġmajor Ġstream Ġof Ġefforts Ġare Ġfocused Ġon Ġdiscovering Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġin Ġthe Ġlung Ġareas Ġand ĠCO VID - 19 . ĠThere Ġhave Ġbeen Ġseveral Ġworks ĠB IB REF 0 , ĠB IB REF 1 Ġpublishing Ġsuch Ġresults . ĠHowever , Ġexisting Ġstudies Ġare Ġmostly Ġconducted Ġseparately Ġby Ġdifferent Ġhospitals Ġand Ġmedical Ġinstit utes . ĠDue Ġto Ġgeographic Ġaffinity , Ġthe Ġpopulations Ġserved Ġby Ġdifferent Ġhospitals Ġhave Ġdifferent Ġgenetic , Ġsocial , Ġand Ġethnic Ġcharacteristics . ĠAs Ġa Ġresult , Ġthe Ġrad iological Ġfindings Ġfrom ĠCO VID - 19 Ġpatient Ġcases Ġin Ġdifferent Ġpopulations Ġare Ġdifferent . ĠThis Ġpopulation Ġbias Ġinc urs Ġinconsistent Ġor Ġeven Ġconflicting Ġconclusions Ġregarding Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . ĠAs Ġa Ġresult , Ġmedical Ġprofessionals Ġcannot Ġmake Ġinformed Ġdecisions Ġon Ġhow Ġto Ġuse Ġrad iological Ġfindings Ġto Ġguide Ġdiagnosis Ġand Ġtreatment Ġof ĠCO VID - 19 . </s> We Ġaim Ġto Ġaddress Ġthis Ġissue . ĠOur Ġresearch Ġgoal Ġis Ġto Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġcollectively Ġanalyze Ġthe Ġstudy Ġresults Ġreported Ġby Ġmany Ġhospitals Ġand Ġmedical Ġinstit utes Ġall Ġover Ġthe Ġworld , Ġreconcile Ġthese Ġresults , Ġand Ġmake Ġa Ġholistic Ġand Ġunbiased Ġconclusion Ġregarding Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . ĠSpecifically , Ġwe Ġtake Ġthe ĠC ORD - 19 Ġdataset ĠB IB REF 2 , Ġwhich Ġcontains Ġover Ġ45 , 000 Ġscholarly Ġarticles , Ġincluding Ġover Ġ33 , 000 Ġwith Ġfull Ġtext , Ġabout ĠCO VID - 19 , ĠS ARS - Co V - 2 , Ġand Ġrelated Ġcoron av ir uses . ĠWe Ġdevelop Ġsentence Ġclassification Ġmethods Ġto Ġidentify Ġall Ġsentences Ġnarr ating Ġrad iological Ġfindings Ġfrom ĠCO VID - 19 . ĠThen Ġconstituent Ġparsing Ġis Ġutilized Ġto Ġidentify Ġall Ġnoun Ġphrases Ġfrom Ġthese Ġsentences Ġand Ġthese Ġnoun Ġphrases Ġcontain Ġabnormalities , Ġlesions , Ġdiseases Ġidentified Ġby Ġrad iology Ġimaging Ġsuch Ġas ĠX - ray Ġand Ġcomputed Ġtom ography Ġ( CT ). ĠWe Ġcalculate Ġthe Ġfrequency Ġof Ġthese Ġnoun Ġphrases Ġand Ġselect Ġthose Ġwith Ġtop Ġfrequencies Ġfor Ġmedical Ġprofessionals Ġto Ġfurther Ġinvestigate . ĠSince Ġthese Ġclinical Ġentities Ġare Ġaggreg ated Ġfrom Ġa Ġnumber Ġof Ġhospitals Ġall Ġover Ġthe Ġworld , Ġthe Ġpopulation Ġbias Ġis Ġlargely Ġmit igated Ġand Ġthe Ġconclusions Ġare Ġmore Ġobjective Ġand Ġuniversally Ġinformative . ĠFrom Ġthe ĠC ORD - 19 Ġdataset , Ġour Ġmethod Ġsuccessfully Ġdiscovers Ġa Ġset Ġof Ġclinical Ġfindings Ġthat Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 . </s> The Ġmajor Ġcontributions Ġof Ġthis Ġpaper Ġinclude : </s> We Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġperform Ġunbiased Ġstudy Ġof Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . </s> We Ġdevelop Ġa Ġboot stra pping Ġapproach Ġto Ġeffectively Ġtrain Ġa Ġsentence Ġclass ifier Ġwith Ġlight - weight Ġmanual Ġannotation Ġeffort . ĠThe Ġsentence Ġclass ifier Ġis Ġused Ġto Ġextract Ġrad iological Ġfindings Ġfrom Ġa Ġvast Ġamount Ġof Ġliterature . </s> We Ġconduct Ġexperiments Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġour Ġmethod . ĠFrom Ġthe ĠC ORD - 19 Ġdataset , Ġour Ġmethod Ġsuccessfully Ġdiscovers Ġa Ġset Ġof Ġclinical Ġfindings Ġthat Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn ĠSection Ġ2 , Ġwe Ġintroduce Ġthe Ġdata . ĠSection Ġ3 Ġpresents Ġthe Ġmethod . ĠSection Ġ4 Ġgives Ġexperimental Ġresults . ĠSection Ġ5 Ġconcludes Ġthe Ġpaper . </s> Dat as et </s> We Ġused Ġthe ĠCO VID - 19 ĠOpen ĠResearch ĠDat as et Ġ( C ORD - 19 ) ĠB IB REF 2 Ġfor Ġour Ġstudy . ĠIn Ġresponse Ġto Ġthe ĠCO VID - 19 Ġpand emic , Ġthe ĠWhite ĠHouse Ġand Ġa Ġcoalition Ġof Ġresearch Ġgroups Ġprepared Ġthe ĠC ORD - 19 Ġdataset . ĠIt Ġcontains Ġover Ġ45 , 000 Ġscholarly Ġarticles , Ġincluding Ġover Ġ33 , 000 Ġwith Ġfull Ġtext , Ġabout ĠCO VID - 19 , ĠS ARS - Co V - 2 , Ġand Ġrelated Ġcoron av ir uses . ĠThese Ġarticles Ġare Ġcontributed Ġby Ġhospitals Ġand Ġmedical Ġinstit utes Ġall Ġover Ġthe Ġworld . ĠSince Ġthe Ġoutbreak Ġof ĠCO VID - 19 Ġis Ġafter ĠNovember Ġ2019 , Ġwe Ġselect Ġarticles Ġpublished Ġafter ĠNovember Ġ2019 Ġto Ġstudy , Ġwhich Ġinclude Ġa Ġtotal Ġof Ġ20 81 Ġarticles Ġand Ġabout Ġ36 0000 Ġsentences . ĠMany Ġarticles Ġreport Ġthe Ġrad iological Ġfindings Ġrelated Ġto ĠCO VID - 19 . ĠTable ĠT AB REF 4 Ġshows Ġsome Ġexamples . </s> Methods </s> Our Ġgoal Ġis Ġto Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġmethods Ġto Ġanalyze Ġa Ġlarge Ġcollection Ġof ĠCO VID - 19 Ġliterature Ġand Ġdiscover Ġunbiased Ġand Ġuniversally Ġinformative Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . ĠTo Ġachieve Ġthis Ġgoal , Ġwe Ġneed Ġto Ġaddress Ġtwo Ġtechnical Ġchallenges . ĠFirst , Ġin Ġthe Ġlarge Ġcollection Ġof ĠCO VID - 19 Ġliterature , Ġonly Ġa Ġsmall Ġpart Ġof Ġsentences Ġare Ġabout Ġrad iological Ġfindings . ĠIt Ġis Ġtime - consuming Ġto Ġmanually Ġidentify Ġthese Ġsentences . ĠSimple Ġmethods Ġsuch Ġas Ġkeyword - based Ġretrieval Ġwill Ġfalsely Ġretrieve Ġsentences Ġthat Ġare Ġnot Ġabout Ġrad iological Ġfindings Ġand Ġmiss Ġsentences Ġthat Ġare Ġabout Ġrad iological Ġfindings . ĠHow Ġcan Ġwe Ġdevelop ĠN LP Ġmethods Ġto Ġprecisely Ġand Ġcomprehens ively Ġextract Ġsentences Ġcontaining Ġrad iological Ġfindings Ġwith Ġminimum Ġhuman Ġannotation ? ĠSecond , Ġgiven Ġthe Ġextracted Ġsentences , Ġthey Ġare Ġstill Ġhighly Ġun struct ured , Ġwhich Ġare Ġdifficult Ġfor Ġmedical Ġprofessionals Ġto Ġdigest Ġand Ġindex . ĠHow Ġcan Ġwe Ġfurther Ġprocess Ġthese Ġsentences Ġinto Ġstructured Ġinformation Ġthat Ġis Ġmore Ġconcise Ġand Ġeasy Ġto Ġuse ? </s> To Ġaddress Ġthe Ġfirst Ġchallenge , Ġwe Ġdevelop Ġa Ġsentence Ġclass ifier Ġto Ġjudge Ġwhether Ġa Ġsentence Ġcontains Ġrad iological Ġfindings . ĠTo Ġminimize Ġmanual - label ing Ġoverhead , Ġwe Ġpropose Ġeasy Ġways Ġof Ġconstructing Ġpositive Ġand Ġnegative Ġtraining Ġexamples , Ġdevelop Ġa Ġboot stra pping Ġapproach Ġto Ġmine Ġhard Ġexamples , Ġand Ġuse Ġhard Ġexamples Ġto Ġre - train Ġthe Ġclass ifier Ġfor Ġreducing Ġfalse Ġpositives . ĠTo Ġaddress Ġthe Ġsecond Ġchallenge , Ġwe Ġuse Ġconstituent Ġparsing Ġto Ġrecognize Ġnoun Ġphrases Ġwhich Ġcontain Ġcritical Ġmedical Ġinformation Ġ( e . g ., Ġlesions , Ġabnormalities , Ġdiseases ) Ġand Ġare Ġeasy Ġto Ġindex Ġand Ġdigest . ĠWe Ġselect Ġnoun Ġphrases Ġwith Ġtop Ġfrequencies Ġfor Ġmedical Ġprofessionals Ġto Ġfurther Ġinvestigate . </s> Methods Ġ: :: ĠExtract ing ĠSent ences ĠCont aining ĠRad iological ĠFind ings </s> In Ġthis Ġsection , Ġwe Ġdevelop Ġa Ġsentence - level Ġclass ifier Ġto Ġdetermine Ġwhether Ġa Ġsentence Ġcontains Ġrad iological Ġfindings . ĠTo Ġbuild Ġsuch Ġa Ġclass ifier , Ġwe Ġneed Ġto Ġcreate Ġpositive Ġand Ġnegative Ġtraining Ġsentences , Ġwithout Ġlabor - intensive Ġannotations . ĠTo Ġobtain Ġpositive Ġexamples , Ġwe Ġresort Ġto Ġthe ĠMed P ix Ġdatabase , Ġwhich Ġcontains Ġrad iology Ġreports Ġnarr ating Ġrad iological Ġfindings . ĠMed P ix Ġis Ġan Ġopen - access Ġonline Ġdatabase Ġof Ġmedical Ġimages , Ġteaching Ġcases , Ġand Ġclinical Ġtopics . ĠIt Ġcontains Ġmore Ġthan Ġ9000 Ġtopics , Ġ59 000 Ġimages Ġfrom Ġ12 000 Ġpatient Ġcases . ĠWe Ġselected Ġdiagnostic Ġreports Ġfor ĠCT Ġimages Ġand Ġused Ġsentences Ġin Ġthe Ġreports Ġas Ġpositive Ġsamples . ĠTo Ġobtain Ġnegative Ġsentences , Ġwe Ġrandomly Ġsample Ġsome Ġsentences Ġfrom Ġthe Ġarticles Ġand Ġquickly Ġscreen Ġthem Ġto Ġensure Ġthat Ġthey Ġare Ġnot Ġabout Ġrad iological Ġfindings . ĠSince Ġmost Ġsentences Ġin Ġthe Ġliterature Ġare Ġnot Ġabout Ġrad iological Ġfindings , Ġa Ġrandom Ġsampling Ġcan Ġalmost Ġensure Ġthe Ġselect Ġsentences Ġare Ġnegative . ĠA Ġmanual Ġscreening Ġis Ġconducted Ġto Ġfurther Ġensure Ġthis Ġand Ġthe Ġscreening Ġeffort Ġis Ġnot Ġheavy . </s> Given Ġthese Ġpositive Ġand Ġnegative Ġtraining Ġsentences , Ġwe Ġuse Ġthem Ġto Ġtrain Ġa Ġsentence Ġclass ifier Ġwhich Ġpredicts Ġwhether Ġa Ġsentence Ġis Ġabout Ġthe Ġrad iological Ġfindings Ġof ĠCO VID - 19 . ĠWe Ġuse Ġthe ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( BER T ) ĠB IB REF 3 Ġmodel Ġfor Ġsentence Ġclassification . ĠB ERT Ġis Ġa Ġneural Ġlanguage Ġmodel Ġthat Ġlearns Ġcontextual Ġrepresentations Ġof Ġwords Ġand Ġsentences . ĠB ERT Ġpret r ains Ġdeep Ġbid irection al Ġrepresentations Ġfrom Ġunl abel ed Ġtext Ġby Ġjointly Ġconditioning Ġon Ġboth Ġleft Ġand Ġright Ġcontext Ġin Ġall Ġlayers . ĠTo Ġapply Ġthe Ġpret rained ĠB ERT Ġto Ġa Ġdownstream Ġtask Ġsuch Ġas Ġsentence Ġclassification , Ġone Ġcan Ġadd Ġan Ġadditional Ġlayer Ġon Ġtop Ġof Ġthe ĠB ERT Ġarchitecture Ġand Ġtrain Ġthis Ġnewly - added Ġlayer Ġusing Ġthe Ġlabeled Ġdata Ġin Ġthe Ġtarget Ġtask . ĠIn Ġour Ġcase , Ġsimilar Ġto ĠB IB REF 4 , Ġwe Ġpret rain Ġthe ĠB ERT Ġmodel Ġon Ġa Ġvast Ġamount Ġof Ġbiomedical Ġliterature Ġto Ġobtain Ġsemantic Ġrepresentations Ġof Ġwords . ĠA Ġlinear Ġlayer Ġis Ġadded Ġto Ġthe Ġoutput Ġof ĠB ERT Ġfor Ġpredicting Ġwhether Ġthis Ġsentence Ġis Ġpositive Ġ( containing Ġrad iological Ġfinding ) Ġor Ġnegative . ĠThe Ġarchitecture Ġand Ġhyper param eters Ġof Ġthe ĠB ERT Ġmodel Ġused Ġin Ġour Ġmethod Ġare Ġthe Ġsame Ġas Ġthose Ġin ĠB IB REF 4 . ĠFigure ĠFIG REF 7 Ġshows Ġthe Ġarchitecture Ġof Ġthe Ġclassification Ġmodel . </s> When Ġapplying Ġthis Ġtrained Ġsentence Ġclass ifier Ġto Ġunseen Ġsentences , Ġwe Ġfound Ġthat Ġit Ġyields Ġa Ġlot Ġof Ġfalse Ġpositives : Ġmany Ġsentences Ġirrelevant Ġto Ġrad iological Ġfindings Ġof ĠCO VID - 19 Ġare Ġpredicted Ġas Ġbeing Ġrelevant . ĠTo Ġsolve Ġthis Ġproblem , Ġwe Ġiter atively Ġperform Ġhard Ġexample Ġmining Ġin Ġa Ġboot stra pping Ġway Ġand Ġuse Ġthese Ġhard Ġexamples Ġto Ġret rain Ġthe Ġclass ifier Ġfor Ġreducing Ġfalse Ġpositives . ĠAt Ġiteration Ġ$ t $, Ġgiven Ġthe Ġclass ifier Ġ$ C _ t $, Ġwe Ġapply Ġit Ġto Ġmake Ġpredictions Ġon Ġunseen Ġsentences . ĠEach Ġsentence Ġis Ġassociated Ġwith Ġa Ġprediction Ġscore Ġwhere Ġa Ġlarger Ġscore Ġindicates Ġthat Ġthis Ġsentence Ġis Ġmore Ġlikely Ġto Ġbe Ġpositive . ĠWe Ġrank Ġthese Ġsentences Ġin Ġdescending Ġorder Ġof Ġtheir Ġprediction Ġscores . ĠThen Ġfor Ġthe Ġtop - K Ġsentences Ġwith Ġthe Ġlargest Ġprediction Ġscores , Ġwe Ġread Ġthem Ġand Ġlabel Ġeach Ġof Ġthem Ġas Ġeither Ġbeing Ġpositive Ġor Ġnegative . ĠThen Ġwe Ġadd Ġthe Ġlabeled Ġpairs Ġto Ġthe Ġtraining Ġset Ġand Ġre - train Ġthe Ġclass ifier Ġand Ġget Ġ$ C _{ t + 1 } $. ĠThis Ġprocedure Ġis Ġrepeated Ġagain Ġto Ġidentify Ġnew Ġfalse Ġpositives Ġand Ġupdate Ġthe Ġclass ifier Ġusing Ġthe Ġnew Ġfalse Ġpositives . </s> Methods Ġ: :: ĠExtract ing ĠN oun ĠPh r ases </s> The Ġextracted Ġsentences Ġcontaining Ġrad iological Ġfindings Ġof ĠCO VID - 19 Ġare Ġhighly Ġun struct ured , Ġwhich Ġare Ġstill Ġdifficult Ġto Ġdigest Ġfor Ġmedical Ġprofessionals . ĠTo Ġsolve Ġthis Ġproblem , Ġfrom Ġthese Ġun struct ured Ġsentences , Ġwe Ġextract Ġstructured Ġinformation Ġthat Ġis Ġboth Ġclinically Ġimportant Ġand Ġeasy Ġto Ġuse . ĠWe Ġnotice Ġthat Ġimportant Ġinformation , Ġsuch Ġas Ġlesions , Ġabnormalities , Ġdiseases , Ġis Ġmostly Ġcontained Ġin Ġnoun Ġphrases . ĠTherefore , Ġwe Ġuse ĠN LP Ġto Ġextract Ġnoun Ġphrases Ġand Ġperform Ġfurther Ġanalysis Ġthere from . ĠFirst , Ġwe Ġperform Ġpart - of - speech Ġ( POS ) Ġtagging Ġto Ġlabel Ġeach Ġword Ġin Ġa Ġsentence Ġas Ġbeing Ġa Ġnoun , Ġverb , Ġadjective , Ġetc . ĠThen Ġon Ġtop Ġof Ġthese Ġwords Ġand Ġtheir ĠPOS Ġtags , Ġwe Ġperform Ġconstituent Ġparsing Ġto Ġobtain Ġthe Ġsyntax Ġtree Ġof Ġthe Ġsentence . ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 9 . ĠFrom Ġbottom Ġto Ġtop Ġof Ġthe Ġtree , Ġfine - gr ained Ġlinguistic Ġunits Ġsuch Ġas Ġwords Ġare Ġcomposed Ġinto Ġcoarse - gr ained Ġunits Ġsuch Ġas Ġphrases , Ġincluding Ġnoun Ġphrases . ĠWe Ġobtain Ġthe Ġnoun Ġphrases Ġby Ġreading Ġthe Ġnode Ġlabels Ġin Ġthe Ġtree . </s> Given Ġthe Ġextracted Ġnoun Ġphrases , Ġwe Ġremove Ġstop Ġwords Ġin Ġthem Ġand Ġperform Ġle mm at ization Ġto Ġeliminate Ġnon - essential Ġlinguistic Ġvariations . ĠWe Ġcount Ġthe Ġfrequency Ġof Ġeach Ġnoun Ġphrase Ġand Ġrank Ġthem Ġin Ġdescending Ġfrequency . ĠThen Ġwe Ġselect Ġthe Ġnoun Ġphrases Ġwith Ġtop Ġfrequencies Ġand Ġpresent Ġthem Ġto Ġmedical Ġprofessionals Ġfor Ġfurther Ġinvestigation . </s> Exper iment Ġ: :: ĠExperimental ĠSettings </s> For Ġbuilding Ġthe Ġinitial Ġsentence Ġclass ifier Ġ( before Ġhard - example Ġmining ), Ġwe Ġcollected Ġ23 50 Ġpositive Ġsamples Ġfrom ĠMed P ix Ġand Ġ3000 Ġnegative Ġsamples Ġfrom ĠC ORD - 19 . ĠWe Ġused Ġ90 % Ġsentences Ġfor Ġtraining Ġand Ġthe Ġrest Ġ10 % Ġsentences Ġfor Ġvalidation . ĠThe Ġweights Ġin Ġthe Ġsentence Ġclass ifier Ġare Ġoptimized Ġusing Ġthe ĠAdam Ġalgorithm Ġwith Ġa Ġlearning Ġrate Ġof Ġ$ 2 \ times Ġ10 ^{ - 5 } $ Ġand Ġa Ġmini - batch Ġsize Ġof Ġ4 . ĠIn Ġboot stra pping Ġfor Ġhard Ġexample Ġmining , Ġwe Ġadded Ġ400 Ġfalse Ġpositives Ġin Ġeach Ġiteration Ġfor Ġclass ifier - ret raining Ġand Ġwe Ġperformed Ġ4 Ġiterations Ġof Ġboot stra pping . </s> Exper iment Ġ: :: ĠResults Ġof ĠSent ence ĠClassification </s> Under Ġthe Ġfinal Ġclass ifier , Ġ9 98 Ġsentences Ġare Ġpredicted Ġas Ġbeing Ġpositive . ĠAmong Ġthem , Ġ7 17 Ġare Ġtrue Ġpositives Ġ( according Ġto Ġmanual Ġcheck ). ĠThe Ġclass ifier Ġachieves Ġa Ġprecision Ġof Ġ71 . 8 %. ĠFor Ġthe Ġinitial Ġclass ifier Ġ( before Ġadding Ġmined Ġhard Ġexamples Ġusing Ġboot stra pping ), Ġamong Ġthe Ġtop Ġ100 Ġsentences Ġwith Ġthe Ġlargest Ġprediction Ġscores , Ġ53 Ġare Ġfalse Ġpositives . ĠThe Ġinitial Ġclass ifier Ġonly Ġachieves Ġa Ġprecision Ġof Ġ47 %. ĠThe Ġprecision Ġachieved Ġby Ġclass ifiers Ġtrained Ġafter Ġround Ġ1 - 3 Ġin Ġboot stra pping Ġis Ġ55 %, Ġ57 %, Ġand Ġ69 % Ġrespectively , Ġas Ġshown Ġin ĠTable ĠT AB REF 12 . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġhard Ġexample Ġmining . ĠTable ĠT AB REF 13 Ġshows Ġsome Ġexample Ġsentences Ġthat Ġare Ġtrue Ġpositives , Ġtrue Ġnegatives , Ġand Ġfalse Ġpositives , Ġunder Ġthe Ġpredictions Ġmade Ġby Ġthe Ġfinal Ġclass ifier . </s> Exper iment Ġ: :: ĠResults Ġof ĠN oun ĠPh rase ĠExt raction </s> Table ĠT AB REF 15 Ġshows Ġthe Ġextracted Ġnoun Ġphrases Ġwith Ġtop Ġfrequencies Ġthat Ġare Ġrelevant Ġto Ġrad iology . ĠMedical Ġprofessionals Ġcan Ġlook Ġat Ġthis Ġtable Ġand Ġselect Ġnoun Ġphrases Ġindicating Ġrad iological Ġfindings Ġfor Ġfurther Ġinvestigation , Ġsuch Ġas Ġconsolidation , Ġple ural Ġeff usion , Ġground Ġglass Ġopacity , Ġthick ening , Ġetc . ĠWe Ġmark Ġsuch Ġnoun Ġphrases Ġwith Ġbold Ġfont Ġin Ġthe Ġtable . ĠTo Ġfurther Ġinvestigate Ġhow Ġa Ġnoun Ġphrase Ġis Ġrelevant Ġto ĠCO VID - 19 , Ġmedical Ġprofessionals Ġcan Ġreview Ġthe Ġsentences Ġmentioning Ġthis Ġnoun Ġphrase . ĠTable ĠT AB REF 16 , TA B REF 17 , TA B REF 18 Ġshow Ġsome Ġexamples . </s> For Ġexample , Ġreading Ġthe Ġfive Ġexample Ġsentences Ġcontaining Ġconsolidation , Ġone Ġcan Ġjudge Ġthat Ġconsolidation Ġis Ġa Ġtypical Ġmanifestation Ġof ĠCO VID - 19 . ĠThis Ġis Ġin Ġaccordance Ġwith Ġthe Ġconclusion Ġin ĠB IB REF 5 : ĠâĢ ľ Cons olid ation Ġbecomes Ġthe Ġdominant ĠCT Ġfindings Ġas Ġthe Ġdisease Ġprogresses ." ĠSimilarly , Ġthe Ġexample Ġsentences Ġof Ġple ural Ġeff usion , Ġground Ġglass Ġopacity , Ġthick ening , Ġfib rosis , Ġbron chi ect asis , Ġlymph aden opathy Ġshow Ġthat Ġthese Ġabnormalities Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 . ĠThis Ġis Ġconsistent Ġwith Ġthe Ġresults Ġreported Ġin Ġthe Ġliterature : </s> P le ural Ġeff usion : ĠâĢ ľ In Ġterms Ġof Ġple ural Ġchanges , ĠCT Ġshowed Ġthat Ġsix Ġ( 9 . 7 %) Ġhad Ġple ural Ġeff usion ." ĠB IB REF 6 </s> Ground Ġglass Ġopacity : ĠâĢ ľ The Ġpredominant Ġpattern Ġof Ġabnormalities Ġafter Ġsymptom Ġonset Ġwas Ġground - glass Ġopacity Ġ( 35 / 78 Ġ[ 45 %] Ġto Ġ49 / 79 Ġ[ 62 %] Ġin Ġdifferent Ġperiods ." ĠB IB REF 7 </s> Th ick ening : ĠâĢ ľ Furthermore , Ġground - glass Ġopacity Ġwas Ġsub c ategor ized Ġinto : Ġ( 1 ) Ġpure Ġground - glass Ġopacity ; Ġ( 2 ) Ġground - glass Ġopacity Ġwith Ġsmooth Ġinter l ob ular Ġse pt al Ġthick ening ." ĠB IB REF 7 </s> F ib rosis : ĠâĢ ľ In Ġfive Ġpatients , Ġfollow - up ĠCT Ġshowed Ġimprovement Ġwith Ġthe Ġappearance Ġof Ġfib rosis Ġand Ġresolution Ġof ĠG GO s .", ĠB IB REF 8 </s> Bron chi ect asis Ġand Ġlymph aden opathy : ĠâĢ ľ The Ġmost Ġcommon Ġpatterns Ġseen Ġon Ġchest ĠCT Ġwere Ġground - glass Ġopacity , Ġin Ġaddition Ġto Ġill - defined Ġmargins , Ġsmooth Ġor Ġirregular Ġinter l ob ular Ġse pt al Ġthick ening , Ġair Ġbron ch ogram Ġ, Ġcrazy - p aving Ġpattern , Ġand Ġthick ening Ġof Ġthe Ġadjacent Ġple ura . ĠLess Ġcommon ĠCT Ġfindings Ġwere Ġnod ules , Ġcy stic Ġchanges , Ġbron chio lect asis , Ġple ural Ġeff usion Ġ, Ġand Ġlymph aden opathy ." ĠB IB REF 9 </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġautomatically Ġextract Ġunbiased Ġrad iological Ġfindings Ġof ĠCO VID - 19 . ĠWe Ġdevelop Ġa ĠB ERT - based Ġclass ifier Ġto Ġselect Ġsentences Ġthat Ġcontain ĠCO VID - related Ġrad iological Ġfindings Ġand Ġuse Ġboot stra pping Ġto Ġmine Ġhard Ġexamples Ġfor Ġreducing Ġfalse Ġpositives . ĠConst itu ent Ġparsing Ġis Ġused Ġto Ġextract Ġnoun Ġphrases Ġfrom Ġthe Ġpositive Ġsentences Ġand Ġthose Ġwith Ġtop Ġfrequencies Ġare Ġselected Ġfor Ġmedical Ġprofessionals Ġto Ġfurther Ġinvestigate . ĠFrom Ġthe ĠC ORD - 19 Ġdataset , Ġour Ġmethod Ġsuccessfully Ġdiscovers Ġrad iological Ġfindings Ġthat Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 .
Introduction Coronavirus disease 2019 (COVID-19) is an infectious disease that has affected more than one million individuals all over the world and caused more than 55,000 deaths, as of April 3 in 2020. The science community has been working very actively to understand this new disease and make diagnosis and treatment guidelines based on the findings. One major stream of efforts are focused on discovering the correlation between radiological findings in the lung areas and COVID-19. There have been several works BIBREF0, BIBREF1 publishing such results. However, existing studies are mostly conducted separately by different hospitals and medical institutes. Due to geographic affinity, the populations served by different hospitals have different genetic, social, and ethnic characteristics. As a result, the radiological findings from COVID-19 patient cases in different populations are different. This population bias incurs inconsistent or even conflicting conclusions regarding the correlation between radiological findings and COVID-19. As a result, medical professionals cannot make informed decisions on how to use radiological findings to guide diagnosis and treatment of COVID-19. We aim to address this issue. Our research goal is to develop natural language processing methods to collectively analyze the study results reported by many hospitals and medical institutes all over the world, reconcile these results, and make a holistic and unbiased conclusion regarding the correlation between radiological findings and COVID-19. Specifically, we take the CORD-19 dataset BIBREF2, which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. We develop sentence classification methods to identify all sentences narrating radiological findings from COVID-19. Then constituent parsing is utilized to identify all noun phrases from these sentences and these noun phrases contain abnormalities, lesions, diseases identified by radiology imaging such as X-ray and computed tomography (CT). We calculate the frequency of these noun phrases and select those with top frequencies for medical professionals to further investigate. Since these clinical entities are aggregated from a number of hospitals all over the world, the population bias is largely mitigated and the conclusions are more objective and universally informative. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19. The major contributions of this paper include: We develop natural language processing methods to perform unbiased study of the correlation between radiological findings and COVID-19. We develop a bootstrapping approach to effectively train a sentence classifier with light-weight manual annotation effort. The sentence classifier is used to extract radiological findings from a vast amount of literature. We conduct experiments to verify the effectiveness of our method. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19. The rest of the paper is organized as follows. In Section 2, we introduce the data. Section 3 presents the method. Section 4 gives experimental results. Section 5 concludes the paper. Dataset We used the COVID-19 Open Research Dataset (CORD-19) BIBREF2 for our study. In response to the COVID-19 pandemic, the White House and a coalition of research groups prepared the CORD-19 dataset. It contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. These articles are contributed by hospitals and medical institutes all over the world. Since the outbreak of COVID-19 is after November 2019, we select articles published after November 2019 to study, which include a total of 2081 articles and about 360000 sentences. Many articles report the radiological findings related to COVID-19. Table TABREF4 shows some examples. Methods Our goal is to develop natural language processing (NLP) methods to analyze a large collection of COVID-19 literature and discover unbiased and universally informative correlation between radiological findings and COVID-19. To achieve this goal, we need to address two technical challenges. First, in the large collection of COVID-19 literature, only a small part of sentences are about radiological findings. It is time-consuming to manually identify these sentences. Simple methods such as keyword-based retrieval will falsely retrieve sentences that are not about radiological findings and miss sentences that are about radiological findings. How can we develop NLP methods to precisely and comprehensively extract sentences containing radiological findings with minimum human annotation? Second, given the extracted sentences, they are still highly unstructured, which are difficult for medical professionals to digest and index. How can we further process these sentences into structured information that is more concise and easy to use? To address the first challenge, we develop a sentence classifier to judge whether a sentence contains radiological findings. To minimize manual-labeling overhead, we propose easy ways of constructing positive and negative training examples, develop a bootstrapping approach to mine hard examples, and use hard examples to re-train the classifier for reducing false positives. To address the second challenge, we use constituent parsing to recognize noun phrases which contain critical medical information (e.g., lesions, abnormalities, diseases) and are easy to index and digest. We select noun phrases with top frequencies for medical professionals to further investigate. Methods ::: Extracting Sentences Containing Radiological Findings In this section, we develop a sentence-level classifier to determine whether a sentence contains radiological findings. To build such a classifier, we need to create positive and negative training sentences, without labor-intensive annotations. To obtain positive examples, we resort to the MedPix database, which contains radiology reports narrating radiological findings. MedPix is an open-access online database of medical images, teaching cases, and clinical topics. It contains more than 9000 topics, 59000 images from 12000 patient cases. We selected diagnostic reports for CT images and used sentences in the reports as positive samples. To obtain negative sentences, we randomly sample some sentences from the articles and quickly screen them to ensure that they are not about radiological findings. Since most sentences in the literature are not about radiological findings, a random sampling can almost ensure the select sentences are negative. A manual screening is conducted to further ensure this and the screening effort is not heavy. Given these positive and negative training sentences, we use them to train a sentence classifier which predicts whether a sentence is about the radiological findings of COVID-19. We use the Bidirectional Encoder Representations from Transformers (BERT) BIBREF3 model for sentence classification. BERT is a neural language model that learns contextual representations of words and sentences. BERT pretrains deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. To apply the pretrained BERT to a downstream task such as sentence classification, one can add an additional layer on top of the BERT architecture and train this newly-added layer using the labeled data in the target task. In our case, similar to BIBREF4, we pretrain the BERT model on a vast amount of biomedical literature to obtain semantic representations of words. A linear layer is added to the output of BERT for predicting whether this sentence is positive (containing radiological finding) or negative. The architecture and hyperparameters of the BERT model used in our method are the same as those in BIBREF4. Figure FIGREF7 shows the architecture of the classification model. When applying this trained sentence classifier to unseen sentences, we found that it yields a lot of false positives: many sentences irrelevant to radiological findings of COVID-19 are predicted as being relevant. To solve this problem, we iteratively perform hard example mining in a bootstrapping way and use these hard examples to retrain the classifier for reducing false positives. At iteration $t$, given the classifier $C_t$, we apply it to make predictions on unseen sentences. Each sentence is associated with a prediction score where a larger score indicates that this sentence is more likely to be positive. We rank these sentences in descending order of their prediction scores. Then for the top-K sentences with the largest prediction scores, we read them and label each of them as either being positive or negative. Then we add the labeled pairs to the training set and re-train the classifier and get $C_{t+1}$. This procedure is repeated again to identify new false positives and update the classifier using the new false positives. Methods ::: Extracting Noun Phrases The extracted sentences containing radiological findings of COVID-19 are highly unstructured, which are still difficult to digest for medical professionals. To solve this problem, from these unstructured sentences, we extract structured information that is both clinically important and easy to use. We notice that important information, such as lesions, abnormalities, diseases, is mostly contained in noun phrases. Therefore, we use NLP to extract noun phrases and perform further analysis therefrom. First, we perform part-of-speech (POS) tagging to label each word in a sentence as being a noun, verb, adjective, etc. Then on top of these words and their POS tags, we perform constituent parsing to obtain the syntax tree of the sentence. An example is shown in Figure FIGREF9. From bottom to top of the tree, fine-grained linguistic units such as words are composed into coarse-grained units such as phrases, including noun phrases. We obtain the noun phrases by reading the node labels in the tree. Given the extracted noun phrases, we remove stop words in them and perform lemmatization to eliminate non-essential linguistic variations. We count the frequency of each noun phrase and rank them in descending frequency. Then we select the noun phrases with top frequencies and present them to medical professionals for further investigation. Experiment ::: Experimental Settings For building the initial sentence classifier (before hard-example mining), we collected 2350 positive samples from MedPix and 3000 negative samples from CORD-19. We used 90% sentences for training and the rest 10% sentences for validation. The weights in the sentence classifier are optimized using the Adam algorithm with a learning rate of $2\times 10^{-5}$ and a mini-batch size of 4. In bootstrapping for hard example mining, we added 400 false positives in each iteration for classifier-retraining and we performed 4 iterations of bootstrapping. Experiment ::: Results of Sentence Classification Under the final classifier, 998 sentences are predicted as being positive. Among them, 717 are true positives (according to manual check). The classifier achieves a precision of 71.8%. For the initial classifier (before adding mined hard examples using bootstrapping), among the top 100 sentences with the largest prediction scores, 53 are false positives. The initial classifier only achieves a precision of 47%. The precision achieved by classifiers trained after round 1-3 in bootstrapping is 55%, 57%, and 69% respectively, as shown in Table TABREF12. This demonstrates the effectiveness of hard example mining. Table TABREF13 shows some example sentences that are true positives, true negatives, and false positives, under the predictions made by the final classifier. Experiment ::: Results of Noun Phrase Extraction Table TABREF15 shows the extracted noun phrases with top frequencies that are relevant to radiology. Medical professionals can look at this table and select noun phrases indicating radiological findings for further investigation, such as consolidation, pleural effusion, ground glass opacity, thickening, etc. We mark such noun phrases with bold font in the table. To further investigate how a noun phrase is relevant to COVID-19, medical professionals can review the sentences mentioning this noun phrase. Table TABREF16,TABREF17,TABREF18 show some examples. For example, reading the five example sentences containing consolidation, one can judge that consolidation is a typical manifestation of COVID-19. This is in accordance with the conclusion in BIBREF5: “Consolidation becomes the dominant CT findings as the disease progresses." Similarly, the example sentences of pleural effusion, ground glass opacity, thickening, fibrosis, bronchiectasis, lymphadenopathy show that these abnormalities are closely related with COVID-19. This is consistent with the results reported in the literature: Pleural effusion: “In terms of pleural changes, CT showed that six (9.7%) had pleural effusion." BIBREF6 Ground glass opacity: “The predominant pattern of abnormalities after symptom onset was ground-glass opacity (35/78 [45%] to 49/79 [62%] in different periods." BIBREF7 Thickening: “Furthermore, ground-glass opacity was subcategorized into: (1) pure ground-glass opacity; (2) ground-glass opacity with smooth interlobular septal thickening." BIBREF7 Fibrosis: “In five patients, follow-up CT showed improvement with the appearance of fibrosis and resolution of GGOs.", BIBREF8 Bronchiectasis and lymphadenopathy: “The most common patterns seen on chest CT were ground-glass opacity, in addition to ill-defined margins, smooth or irregular interlobular septal thickening, air bronchogram , crazy-paving pattern, and thickening of the adjacent pleura. Less common CT findings were nodules, cystic changes, bronchiolectasis, pleural effusion , and lymphadenopathy." BIBREF9 Conclusions In this paper, we develop natural language processing methods to automatically extract unbiased radiological findings of COVID-19. We develop a BERT-based classifier to select sentences that contain COVID-related radiological findings and use bootstrapping to mine hard examples for reducing false positives. Constituent parsing is used to extract noun phrases from the positive sentences and those with top frequencies are selected for medical professionals to further investigate. From the CORD-19 dataset, our method successfully discovers radiological findings that are closely related with COVID-19.
What is the CORD-19 dataset?
45f7c03a686b68179cadb1413c5f3c1d373328bd
What Ġis Ġthe ĠC ORD - 19 Ġdataset ?
which Ġcontains Ġover Ġ45 , 000 Ġscholarly Ġarticles , Ġincluding Ġover Ġ33 , 000 Ġwith Ġfull Ġtext , Ġabout ĠCO VID - 19 , ĠS ARS - Co V - 2 , Ġand Ġrelated Ġcoron av ir uses
<s> What Ġis Ġthe ĠC ORD - 19 Ġdataset ? </s> Introduction </s> Cor on av irus Ġdisease Ġ2019 Ġ( CO VID - 19 ) Ġis Ġan Ġinfectious Ġdisease Ġthat Ġhas Ġaffected Ġmore Ġthan Ġone Ġmillion Ġindividuals Ġall Ġover Ġthe Ġworld Ġand Ġcaused Ġmore Ġthan Ġ55 , 000 Ġdeaths , Ġas Ġof ĠApril Ġ3 Ġin Ġ2020 . ĠThe Ġscience Ġcommunity Ġhas Ġbeen Ġworking Ġvery Ġactively Ġto Ġunderstand Ġthis Ġnew Ġdisease Ġand Ġmake Ġdiagnosis Ġand Ġtreatment Ġguidelines Ġbased Ġon Ġthe Ġfindings . ĠOne Ġmajor Ġstream Ġof Ġefforts Ġare Ġfocused Ġon Ġdiscovering Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġin Ġthe Ġlung Ġareas Ġand ĠCO VID - 19 . ĠThere Ġhave Ġbeen Ġseveral Ġworks ĠB IB REF 0 , ĠB IB REF 1 Ġpublishing Ġsuch Ġresults . ĠHowever , Ġexisting Ġstudies Ġare Ġmostly Ġconducted Ġseparately Ġby Ġdifferent Ġhospitals Ġand Ġmedical Ġinstit utes . ĠDue Ġto Ġgeographic Ġaffinity , Ġthe Ġpopulations Ġserved Ġby Ġdifferent Ġhospitals Ġhave Ġdifferent Ġgenetic , Ġsocial , Ġand Ġethnic Ġcharacteristics . ĠAs Ġa Ġresult , Ġthe Ġrad iological Ġfindings Ġfrom ĠCO VID - 19 Ġpatient Ġcases Ġin Ġdifferent Ġpopulations Ġare Ġdifferent . ĠThis Ġpopulation Ġbias Ġinc urs Ġinconsistent Ġor Ġeven Ġconflicting Ġconclusions Ġregarding Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . ĠAs Ġa Ġresult , Ġmedical Ġprofessionals Ġcannot Ġmake Ġinformed Ġdecisions Ġon Ġhow Ġto Ġuse Ġrad iological Ġfindings Ġto Ġguide Ġdiagnosis Ġand Ġtreatment Ġof ĠCO VID - 19 . </s> We Ġaim Ġto Ġaddress Ġthis Ġissue . ĠOur Ġresearch Ġgoal Ġis Ġto Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġcollectively Ġanalyze Ġthe Ġstudy Ġresults Ġreported Ġby Ġmany Ġhospitals Ġand Ġmedical Ġinstit utes Ġall Ġover Ġthe Ġworld , Ġreconcile Ġthese Ġresults , Ġand Ġmake Ġa Ġholistic Ġand Ġunbiased Ġconclusion Ġregarding Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . ĠSpecifically , Ġwe Ġtake Ġthe ĠC ORD - 19 Ġdataset ĠB IB REF 2 , Ġwhich Ġcontains Ġover Ġ45 , 000 Ġscholarly Ġarticles , Ġincluding Ġover Ġ33 , 000 Ġwith Ġfull Ġtext , Ġabout ĠCO VID - 19 , ĠS ARS - Co V - 2 , Ġand Ġrelated Ġcoron av ir uses . ĠWe Ġdevelop Ġsentence Ġclassification Ġmethods Ġto Ġidentify Ġall Ġsentences Ġnarr ating Ġrad iological Ġfindings Ġfrom ĠCO VID - 19 . ĠThen Ġconstituent Ġparsing Ġis Ġutilized Ġto Ġidentify Ġall Ġnoun Ġphrases Ġfrom Ġthese Ġsentences Ġand Ġthese Ġnoun Ġphrases Ġcontain Ġabnormalities , Ġlesions , Ġdiseases Ġidentified Ġby Ġrad iology Ġimaging Ġsuch Ġas ĠX - ray Ġand Ġcomputed Ġtom ography Ġ( CT ). ĠWe Ġcalculate Ġthe Ġfrequency Ġof Ġthese Ġnoun Ġphrases Ġand Ġselect Ġthose Ġwith Ġtop Ġfrequencies Ġfor Ġmedical Ġprofessionals Ġto Ġfurther Ġinvestigate . ĠSince Ġthese Ġclinical Ġentities Ġare Ġaggreg ated Ġfrom Ġa Ġnumber Ġof Ġhospitals Ġall Ġover Ġthe Ġworld , Ġthe Ġpopulation Ġbias Ġis Ġlargely Ġmit igated Ġand Ġthe Ġconclusions Ġare Ġmore Ġobjective Ġand Ġuniversally Ġinformative . ĠFrom Ġthe ĠC ORD - 19 Ġdataset , Ġour Ġmethod Ġsuccessfully Ġdiscovers Ġa Ġset Ġof Ġclinical Ġfindings Ġthat Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 . </s> The Ġmajor Ġcontributions Ġof Ġthis Ġpaper Ġinclude : </s> We Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġperform Ġunbiased Ġstudy Ġof Ġthe Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . </s> We Ġdevelop Ġa Ġboot stra pping Ġapproach Ġto Ġeffectively Ġtrain Ġa Ġsentence Ġclass ifier Ġwith Ġlight - weight Ġmanual Ġannotation Ġeffort . ĠThe Ġsentence Ġclass ifier Ġis Ġused Ġto Ġextract Ġrad iological Ġfindings Ġfrom Ġa Ġvast Ġamount Ġof Ġliterature . </s> We Ġconduct Ġexperiments Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġour Ġmethod . ĠFrom Ġthe ĠC ORD - 19 Ġdataset , Ġour Ġmethod Ġsuccessfully Ġdiscovers Ġa Ġset Ġof Ġclinical Ġfindings Ġthat Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn ĠSection Ġ2 , Ġwe Ġintroduce Ġthe Ġdata . ĠSection Ġ3 Ġpresents Ġthe Ġmethod . ĠSection Ġ4 Ġgives Ġexperimental Ġresults . ĠSection Ġ5 Ġconcludes Ġthe Ġpaper . </s> Dat as et </s> We Ġused Ġthe ĠCO VID - 19 ĠOpen ĠResearch ĠDat as et Ġ( C ORD - 19 ) ĠB IB REF 2 Ġfor Ġour Ġstudy . ĠIn Ġresponse Ġto Ġthe ĠCO VID - 19 Ġpand emic , Ġthe ĠWhite ĠHouse Ġand Ġa Ġcoalition Ġof Ġresearch Ġgroups Ġprepared Ġthe ĠC ORD - 19 Ġdataset . ĠIt Ġcontains Ġover Ġ45 , 000 Ġscholarly Ġarticles , Ġincluding Ġover Ġ33 , 000 Ġwith Ġfull Ġtext , Ġabout ĠCO VID - 19 , ĠS ARS - Co V - 2 , Ġand Ġrelated Ġcoron av ir uses . ĠThese Ġarticles Ġare Ġcontributed Ġby Ġhospitals Ġand Ġmedical Ġinstit utes Ġall Ġover Ġthe Ġworld . ĠSince Ġthe Ġoutbreak Ġof ĠCO VID - 19 Ġis Ġafter ĠNovember Ġ2019 , Ġwe Ġselect Ġarticles Ġpublished Ġafter ĠNovember Ġ2019 Ġto Ġstudy , Ġwhich Ġinclude Ġa Ġtotal Ġof Ġ20 81 Ġarticles Ġand Ġabout Ġ36 0000 Ġsentences . ĠMany Ġarticles Ġreport Ġthe Ġrad iological Ġfindings Ġrelated Ġto ĠCO VID - 19 . ĠTable ĠT AB REF 4 Ġshows Ġsome Ġexamples . </s> Methods </s> Our Ġgoal Ġis Ġto Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġmethods Ġto Ġanalyze Ġa Ġlarge Ġcollection Ġof ĠCO VID - 19 Ġliterature Ġand Ġdiscover Ġunbiased Ġand Ġuniversally Ġinformative Ġcorrelation Ġbetween Ġrad iological Ġfindings Ġand ĠCO VID - 19 . ĠTo Ġachieve Ġthis Ġgoal , Ġwe Ġneed Ġto Ġaddress Ġtwo Ġtechnical Ġchallenges . ĠFirst , Ġin Ġthe Ġlarge Ġcollection Ġof ĠCO VID - 19 Ġliterature , Ġonly Ġa Ġsmall Ġpart Ġof Ġsentences Ġare Ġabout Ġrad iological Ġfindings . ĠIt Ġis Ġtime - consuming Ġto Ġmanually Ġidentify Ġthese Ġsentences . ĠSimple Ġmethods Ġsuch Ġas Ġkeyword - based Ġretrieval Ġwill Ġfalsely Ġretrieve Ġsentences Ġthat Ġare Ġnot Ġabout Ġrad iological Ġfindings Ġand Ġmiss Ġsentences Ġthat Ġare Ġabout Ġrad iological Ġfindings . ĠHow Ġcan Ġwe Ġdevelop ĠN LP Ġmethods Ġto Ġprecisely Ġand Ġcomprehens ively Ġextract Ġsentences Ġcontaining Ġrad iological Ġfindings Ġwith Ġminimum Ġhuman Ġannotation ? ĠSecond , Ġgiven Ġthe Ġextracted Ġsentences , Ġthey Ġare Ġstill Ġhighly Ġun struct ured , Ġwhich Ġare Ġdifficult Ġfor Ġmedical Ġprofessionals Ġto Ġdigest Ġand Ġindex . ĠHow Ġcan Ġwe Ġfurther Ġprocess Ġthese Ġsentences Ġinto Ġstructured Ġinformation Ġthat Ġis Ġmore Ġconcise Ġand Ġeasy Ġto Ġuse ? </s> To Ġaddress Ġthe Ġfirst Ġchallenge , Ġwe Ġdevelop Ġa Ġsentence Ġclass ifier Ġto Ġjudge Ġwhether Ġa Ġsentence Ġcontains Ġrad iological Ġfindings . ĠTo Ġminimize Ġmanual - label ing Ġoverhead , Ġwe Ġpropose Ġeasy Ġways Ġof Ġconstructing Ġpositive Ġand Ġnegative Ġtraining Ġexamples , Ġdevelop Ġa Ġboot stra pping Ġapproach Ġto Ġmine Ġhard Ġexamples , Ġand Ġuse Ġhard Ġexamples Ġto Ġre - train Ġthe Ġclass ifier Ġfor Ġreducing Ġfalse Ġpositives . ĠTo Ġaddress Ġthe Ġsecond Ġchallenge , Ġwe Ġuse Ġconstituent Ġparsing Ġto Ġrecognize Ġnoun Ġphrases Ġwhich Ġcontain Ġcritical Ġmedical Ġinformation Ġ( e . g ., Ġlesions , Ġabnormalities , Ġdiseases ) Ġand Ġare Ġeasy Ġto Ġindex Ġand Ġdigest . ĠWe Ġselect Ġnoun Ġphrases Ġwith Ġtop Ġfrequencies Ġfor Ġmedical Ġprofessionals Ġto Ġfurther Ġinvestigate . </s> Methods Ġ: :: ĠExtract ing ĠSent ences ĠCont aining ĠRad iological ĠFind ings </s> In Ġthis Ġsection , Ġwe Ġdevelop Ġa Ġsentence - level Ġclass ifier Ġto Ġdetermine Ġwhether Ġa Ġsentence Ġcontains Ġrad iological Ġfindings . ĠTo Ġbuild Ġsuch Ġa Ġclass ifier , Ġwe Ġneed Ġto Ġcreate Ġpositive Ġand Ġnegative Ġtraining Ġsentences , Ġwithout Ġlabor - intensive Ġannotations . ĠTo Ġobtain Ġpositive Ġexamples , Ġwe Ġresort Ġto Ġthe ĠMed P ix Ġdatabase , Ġwhich Ġcontains Ġrad iology Ġreports Ġnarr ating Ġrad iological Ġfindings . ĠMed P ix Ġis Ġan Ġopen - access Ġonline Ġdatabase Ġof Ġmedical Ġimages , Ġteaching Ġcases , Ġand Ġclinical Ġtopics . ĠIt Ġcontains Ġmore Ġthan Ġ9000 Ġtopics , Ġ59 000 Ġimages Ġfrom Ġ12 000 Ġpatient Ġcases . ĠWe Ġselected Ġdiagnostic Ġreports Ġfor ĠCT Ġimages Ġand Ġused Ġsentences Ġin Ġthe Ġreports Ġas Ġpositive Ġsamples . ĠTo Ġobtain Ġnegative Ġsentences , Ġwe Ġrandomly Ġsample Ġsome Ġsentences Ġfrom Ġthe Ġarticles Ġand Ġquickly Ġscreen Ġthem Ġto Ġensure Ġthat Ġthey Ġare Ġnot Ġabout Ġrad iological Ġfindings . ĠSince Ġmost Ġsentences Ġin Ġthe Ġliterature Ġare Ġnot Ġabout Ġrad iological Ġfindings , Ġa Ġrandom Ġsampling Ġcan Ġalmost Ġensure Ġthe Ġselect Ġsentences Ġare Ġnegative . ĠA Ġmanual Ġscreening Ġis Ġconducted Ġto Ġfurther Ġensure Ġthis Ġand Ġthe Ġscreening Ġeffort Ġis Ġnot Ġheavy . </s> Given Ġthese Ġpositive Ġand Ġnegative Ġtraining Ġsentences , Ġwe Ġuse Ġthem Ġto Ġtrain Ġa Ġsentence Ġclass ifier Ġwhich Ġpredicts Ġwhether Ġa Ġsentence Ġis Ġabout Ġthe Ġrad iological Ġfindings Ġof ĠCO VID - 19 . ĠWe Ġuse Ġthe ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( BER T ) ĠB IB REF 3 Ġmodel Ġfor Ġsentence Ġclassification . ĠB ERT Ġis Ġa Ġneural Ġlanguage Ġmodel Ġthat Ġlearns Ġcontextual Ġrepresentations Ġof Ġwords Ġand Ġsentences . ĠB ERT Ġpret r ains Ġdeep Ġbid irection al Ġrepresentations Ġfrom Ġunl abel ed Ġtext Ġby Ġjointly Ġconditioning Ġon Ġboth Ġleft Ġand Ġright Ġcontext Ġin Ġall Ġlayers . ĠTo Ġapply Ġthe Ġpret rained ĠB ERT Ġto Ġa Ġdownstream Ġtask Ġsuch Ġas Ġsentence Ġclassification , Ġone Ġcan Ġadd Ġan Ġadditional Ġlayer Ġon Ġtop Ġof Ġthe ĠB ERT Ġarchitecture Ġand Ġtrain Ġthis Ġnewly - added Ġlayer Ġusing Ġthe Ġlabeled Ġdata Ġin Ġthe Ġtarget Ġtask . ĠIn Ġour Ġcase , Ġsimilar Ġto ĠB IB REF 4 , Ġwe Ġpret rain Ġthe ĠB ERT Ġmodel Ġon Ġa Ġvast Ġamount Ġof Ġbiomedical Ġliterature Ġto Ġobtain Ġsemantic Ġrepresentations Ġof Ġwords . ĠA Ġlinear Ġlayer Ġis Ġadded Ġto Ġthe Ġoutput Ġof ĠB ERT Ġfor Ġpredicting Ġwhether Ġthis Ġsentence Ġis Ġpositive Ġ( containing Ġrad iological Ġfinding ) Ġor Ġnegative . ĠThe Ġarchitecture Ġand Ġhyper param eters Ġof Ġthe ĠB ERT Ġmodel Ġused Ġin Ġour Ġmethod Ġare Ġthe Ġsame Ġas Ġthose Ġin ĠB IB REF 4 . ĠFigure ĠFIG REF 7 Ġshows Ġthe Ġarchitecture Ġof Ġthe Ġclassification Ġmodel . </s> When Ġapplying Ġthis Ġtrained Ġsentence Ġclass ifier Ġto Ġunseen Ġsentences , Ġwe Ġfound Ġthat Ġit Ġyields Ġa Ġlot Ġof Ġfalse Ġpositives : Ġmany Ġsentences Ġirrelevant Ġto Ġrad iological Ġfindings Ġof ĠCO VID - 19 Ġare Ġpredicted Ġas Ġbeing Ġrelevant . ĠTo Ġsolve Ġthis Ġproblem , Ġwe Ġiter atively Ġperform Ġhard Ġexample Ġmining Ġin Ġa Ġboot stra pping Ġway Ġand Ġuse Ġthese Ġhard Ġexamples Ġto Ġret rain Ġthe Ġclass ifier Ġfor Ġreducing Ġfalse Ġpositives . ĠAt Ġiteration Ġ$ t $, Ġgiven Ġthe Ġclass ifier Ġ$ C _ t $, Ġwe Ġapply Ġit Ġto Ġmake Ġpredictions Ġon Ġunseen Ġsentences . ĠEach Ġsentence Ġis Ġassociated Ġwith Ġa Ġprediction Ġscore Ġwhere Ġa Ġlarger Ġscore Ġindicates Ġthat Ġthis Ġsentence Ġis Ġmore Ġlikely Ġto Ġbe Ġpositive . ĠWe Ġrank Ġthese Ġsentences Ġin Ġdescending Ġorder Ġof Ġtheir Ġprediction Ġscores . ĠThen Ġfor Ġthe Ġtop - K Ġsentences Ġwith Ġthe Ġlargest Ġprediction Ġscores , Ġwe Ġread Ġthem Ġand Ġlabel Ġeach Ġof Ġthem Ġas Ġeither Ġbeing Ġpositive Ġor Ġnegative . ĠThen Ġwe Ġadd Ġthe Ġlabeled Ġpairs Ġto Ġthe Ġtraining Ġset Ġand Ġre - train Ġthe Ġclass ifier Ġand Ġget Ġ$ C _{ t + 1 } $. ĠThis Ġprocedure Ġis Ġrepeated Ġagain Ġto Ġidentify Ġnew Ġfalse Ġpositives Ġand Ġupdate Ġthe Ġclass ifier Ġusing Ġthe Ġnew Ġfalse Ġpositives . </s> Methods Ġ: :: ĠExtract ing ĠN oun ĠPh r ases </s> The Ġextracted Ġsentences Ġcontaining Ġrad iological Ġfindings Ġof ĠCO VID - 19 Ġare Ġhighly Ġun struct ured , Ġwhich Ġare Ġstill Ġdifficult Ġto Ġdigest Ġfor Ġmedical Ġprofessionals . ĠTo Ġsolve Ġthis Ġproblem , Ġfrom Ġthese Ġun struct ured Ġsentences , Ġwe Ġextract Ġstructured Ġinformation Ġthat Ġis Ġboth Ġclinically Ġimportant Ġand Ġeasy Ġto Ġuse . ĠWe Ġnotice Ġthat Ġimportant Ġinformation , Ġsuch Ġas Ġlesions , Ġabnormalities , Ġdiseases , Ġis Ġmostly Ġcontained Ġin Ġnoun Ġphrases . ĠTherefore , Ġwe Ġuse ĠN LP Ġto Ġextract Ġnoun Ġphrases Ġand Ġperform Ġfurther Ġanalysis Ġthere from . ĠFirst , Ġwe Ġperform Ġpart - of - speech Ġ( POS ) Ġtagging Ġto Ġlabel Ġeach Ġword Ġin Ġa Ġsentence Ġas Ġbeing Ġa Ġnoun , Ġverb , Ġadjective , Ġetc . ĠThen Ġon Ġtop Ġof Ġthese Ġwords Ġand Ġtheir ĠPOS Ġtags , Ġwe Ġperform Ġconstituent Ġparsing Ġto Ġobtain Ġthe Ġsyntax Ġtree Ġof Ġthe Ġsentence . ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 9 . ĠFrom Ġbottom Ġto Ġtop Ġof Ġthe Ġtree , Ġfine - gr ained Ġlinguistic Ġunits Ġsuch Ġas Ġwords Ġare Ġcomposed Ġinto Ġcoarse - gr ained Ġunits Ġsuch Ġas Ġphrases , Ġincluding Ġnoun Ġphrases . ĠWe Ġobtain Ġthe Ġnoun Ġphrases Ġby Ġreading Ġthe Ġnode Ġlabels Ġin Ġthe Ġtree . </s> Given Ġthe Ġextracted Ġnoun Ġphrases , Ġwe Ġremove Ġstop Ġwords Ġin Ġthem Ġand Ġperform Ġle mm at ization Ġto Ġeliminate Ġnon - essential Ġlinguistic Ġvariations . ĠWe Ġcount Ġthe Ġfrequency Ġof Ġeach Ġnoun Ġphrase Ġand Ġrank Ġthem Ġin Ġdescending Ġfrequency . ĠThen Ġwe Ġselect Ġthe Ġnoun Ġphrases Ġwith Ġtop Ġfrequencies Ġand Ġpresent Ġthem Ġto Ġmedical Ġprofessionals Ġfor Ġfurther Ġinvestigation . </s> Exper iment Ġ: :: ĠExperimental ĠSettings </s> For Ġbuilding Ġthe Ġinitial Ġsentence Ġclass ifier Ġ( before Ġhard - example Ġmining ), Ġwe Ġcollected Ġ23 50 Ġpositive Ġsamples Ġfrom ĠMed P ix Ġand Ġ3000 Ġnegative Ġsamples Ġfrom ĠC ORD - 19 . ĠWe Ġused Ġ90 % Ġsentences Ġfor Ġtraining Ġand Ġthe Ġrest Ġ10 % Ġsentences Ġfor Ġvalidation . ĠThe Ġweights Ġin Ġthe Ġsentence Ġclass ifier Ġare Ġoptimized Ġusing Ġthe ĠAdam Ġalgorithm Ġwith Ġa Ġlearning Ġrate Ġof Ġ$ 2 \ times Ġ10 ^{ - 5 } $ Ġand Ġa Ġmini - batch Ġsize Ġof Ġ4 . ĠIn Ġboot stra pping Ġfor Ġhard Ġexample Ġmining , Ġwe Ġadded Ġ400 Ġfalse Ġpositives Ġin Ġeach Ġiteration Ġfor Ġclass ifier - ret raining Ġand Ġwe Ġperformed Ġ4 Ġiterations Ġof Ġboot stra pping . </s> Exper iment Ġ: :: ĠResults Ġof ĠSent ence ĠClassification </s> Under Ġthe Ġfinal Ġclass ifier , Ġ9 98 Ġsentences Ġare Ġpredicted Ġas Ġbeing Ġpositive . ĠAmong Ġthem , Ġ7 17 Ġare Ġtrue Ġpositives Ġ( according Ġto Ġmanual Ġcheck ). ĠThe Ġclass ifier Ġachieves Ġa Ġprecision Ġof Ġ71 . 8 %. ĠFor Ġthe Ġinitial Ġclass ifier Ġ( before Ġadding Ġmined Ġhard Ġexamples Ġusing Ġboot stra pping ), Ġamong Ġthe Ġtop Ġ100 Ġsentences Ġwith Ġthe Ġlargest Ġprediction Ġscores , Ġ53 Ġare Ġfalse Ġpositives . ĠThe Ġinitial Ġclass ifier Ġonly Ġachieves Ġa Ġprecision Ġof Ġ47 %. ĠThe Ġprecision Ġachieved Ġby Ġclass ifiers Ġtrained Ġafter Ġround Ġ1 - 3 Ġin Ġboot stra pping Ġis Ġ55 %, Ġ57 %, Ġand Ġ69 % Ġrespectively , Ġas Ġshown Ġin ĠTable ĠT AB REF 12 . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġhard Ġexample Ġmining . ĠTable ĠT AB REF 13 Ġshows Ġsome Ġexample Ġsentences Ġthat Ġare Ġtrue Ġpositives , Ġtrue Ġnegatives , Ġand Ġfalse Ġpositives , Ġunder Ġthe Ġpredictions Ġmade Ġby Ġthe Ġfinal Ġclass ifier . </s> Exper iment Ġ: :: ĠResults Ġof ĠN oun ĠPh rase ĠExt raction </s> Table ĠT AB REF 15 Ġshows Ġthe Ġextracted Ġnoun Ġphrases Ġwith Ġtop Ġfrequencies Ġthat Ġare Ġrelevant Ġto Ġrad iology . ĠMedical Ġprofessionals Ġcan Ġlook Ġat Ġthis Ġtable Ġand Ġselect Ġnoun Ġphrases Ġindicating Ġrad iological Ġfindings Ġfor Ġfurther Ġinvestigation , Ġsuch Ġas Ġconsolidation , Ġple ural Ġeff usion , Ġground Ġglass Ġopacity , Ġthick ening , Ġetc . ĠWe Ġmark Ġsuch Ġnoun Ġphrases Ġwith Ġbold Ġfont Ġin Ġthe Ġtable . ĠTo Ġfurther Ġinvestigate Ġhow Ġa Ġnoun Ġphrase Ġis Ġrelevant Ġto ĠCO VID - 19 , Ġmedical Ġprofessionals Ġcan Ġreview Ġthe Ġsentences Ġmentioning Ġthis Ġnoun Ġphrase . ĠTable ĠT AB REF 16 , TA B REF 17 , TA B REF 18 Ġshow Ġsome Ġexamples . </s> For Ġexample , Ġreading Ġthe Ġfive Ġexample Ġsentences Ġcontaining Ġconsolidation , Ġone Ġcan Ġjudge Ġthat Ġconsolidation Ġis Ġa Ġtypical Ġmanifestation Ġof ĠCO VID - 19 . ĠThis Ġis Ġin Ġaccordance Ġwith Ġthe Ġconclusion Ġin ĠB IB REF 5 : ĠâĢ ľ Cons olid ation Ġbecomes Ġthe Ġdominant ĠCT Ġfindings Ġas Ġthe Ġdisease Ġprogresses ." ĠSimilarly , Ġthe Ġexample Ġsentences Ġof Ġple ural Ġeff usion , Ġground Ġglass Ġopacity , Ġthick ening , Ġfib rosis , Ġbron chi ect asis , Ġlymph aden opathy Ġshow Ġthat Ġthese Ġabnormalities Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 . ĠThis Ġis Ġconsistent Ġwith Ġthe Ġresults Ġreported Ġin Ġthe Ġliterature : </s> P le ural Ġeff usion : ĠâĢ ľ In Ġterms Ġof Ġple ural Ġchanges , ĠCT Ġshowed Ġthat Ġsix Ġ( 9 . 7 %) Ġhad Ġple ural Ġeff usion ." ĠB IB REF 6 </s> Ground Ġglass Ġopacity : ĠâĢ ľ The Ġpredominant Ġpattern Ġof Ġabnormalities Ġafter Ġsymptom Ġonset Ġwas Ġground - glass Ġopacity Ġ( 35 / 78 Ġ[ 45 %] Ġto Ġ49 / 79 Ġ[ 62 %] Ġin Ġdifferent Ġperiods ." ĠB IB REF 7 </s> Th ick ening : ĠâĢ ľ Furthermore , Ġground - glass Ġopacity Ġwas Ġsub c ategor ized Ġinto : Ġ( 1 ) Ġpure Ġground - glass Ġopacity ; Ġ( 2 ) Ġground - glass Ġopacity Ġwith Ġsmooth Ġinter l ob ular Ġse pt al Ġthick ening ." ĠB IB REF 7 </s> F ib rosis : ĠâĢ ľ In Ġfive Ġpatients , Ġfollow - up ĠCT Ġshowed Ġimprovement Ġwith Ġthe Ġappearance Ġof Ġfib rosis Ġand Ġresolution Ġof ĠG GO s .", ĠB IB REF 8 </s> Bron chi ect asis Ġand Ġlymph aden opathy : ĠâĢ ľ The Ġmost Ġcommon Ġpatterns Ġseen Ġon Ġchest ĠCT Ġwere Ġground - glass Ġopacity , Ġin Ġaddition Ġto Ġill - defined Ġmargins , Ġsmooth Ġor Ġirregular Ġinter l ob ular Ġse pt al Ġthick ening , Ġair Ġbron ch ogram Ġ, Ġcrazy - p aving Ġpattern , Ġand Ġthick ening Ġof Ġthe Ġadjacent Ġple ura . ĠLess Ġcommon ĠCT Ġfindings Ġwere Ġnod ules , Ġcy stic Ġchanges , Ġbron chio lect asis , Ġple ural Ġeff usion Ġ, Ġand Ġlymph aden opathy ." ĠB IB REF 9 </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġdevelop Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġautomatically Ġextract Ġunbiased Ġrad iological Ġfindings Ġof ĠCO VID - 19 . ĠWe Ġdevelop Ġa ĠB ERT - based Ġclass ifier Ġto Ġselect Ġsentences Ġthat Ġcontain ĠCO VID - related Ġrad iological Ġfindings Ġand Ġuse Ġboot stra pping Ġto Ġmine Ġhard Ġexamples Ġfor Ġreducing Ġfalse Ġpositives . ĠConst itu ent Ġparsing Ġis Ġused Ġto Ġextract Ġnoun Ġphrases Ġfrom Ġthe Ġpositive Ġsentences Ġand Ġthose Ġwith Ġtop Ġfrequencies Ġare Ġselected Ġfor Ġmedical Ġprofessionals Ġto Ġfurther Ġinvestigate . ĠFrom Ġthe ĠC ORD - 19 Ġdataset , Ġour Ġmethod Ġsuccessfully Ġdiscovers Ġrad iological Ġfindings Ġthat Ġare Ġclosely Ġrelated Ġwith ĠCO VID - 19 .