Unanswerable
NONE
1903.07406
Introduction </s> C ancer Ġis Ġone Ġof Ġthe Ġleading Ġcauses Ġof Ġdeath Ġin Ġthe Ġworld , Ġwith Ġover Ġ80 , 000 Ġdeaths Ġregistered Ġin ĠCanada Ġin Ġ2017 Ġ( Canadian ĠCancer ĠStatistics Ġ2017 ). ĠA Ġcomputer - a ided Ġsystem Ġfor Ġcancer Ġdiagnosis Ġusually Ġinvolves Ġa Ġpath ologist Ġrendering Ġa Ġdescriptive Ġreport Ġafter Ġexamining Ġthe Ġtissue Ġglass Ġslides Ġobtained Ġfrom Ġthe Ġbi opsy Ġof Ġa Ġpatient . ĠA Ġpathology Ġreport Ġcontains Ġspecific Ġanalysis Ġof Ġcells Ġand Ġtissues , Ġand Ġother Ġhist opath ological Ġindicators Ġthat Ġare Ġcrucial Ġfor Ġdiagn osing Ġmal ign ancies . ĠAn Ġaverage Ġsized Ġlaboratory Ġmay Ġproduces Ġa Ġlarge Ġquantity Ġof Ġpathology Ġreports Ġannually Ġ( e . g ., Ġin Ġexcess Ġof Ġ50 , 000 ), Ġbut Ġthese Ġreports Ġare Ġwritten Ġin Ġmostly Ġun struct ured Ġtext Ġand Ġwith Ġno Ġdirect Ġlink Ġto Ġthe Ġtissue Ġsample . ĠFurthermore , Ġthe Ġreport Ġfor Ġeach Ġpatient Ġis Ġa Ġpersonalized Ġdocument Ġand Ġoffers Ġvery Ġhigh Ġvariability Ġin Ġterminology Ġdue Ġto Ġlack Ġof Ġstandards Ġand Ġmay Ġeven Ġinclude Ġmiss pell ings Ġand Ġmissing Ġpunct uation , Ġclinical Ġdiagnoses Ġinter sp ersed Ġwith Ġcomplex Ġexplanations , Ġdifferent Ġterminology Ġto Ġlabel Ġthe Ġsame Ġmal ign ancy , Ġand Ġinformation Ġabout Ġmultiple Ġcarcin oma Ġappearances Ġincluded Ġin Ġa Ġsingle Ġreport ĠB IB REF 0 Ġ. </s> In ĠCanada , Ġeach ĠProvincial Ġand ĠTerrit orial ĠCancer ĠRegistry Ġ( P TC R ) Ġis Ġresponsible Ġfor Ġcollecting Ġthe Ġdata Ġabout Ġcancer Ġdiseases Ġand Ġreporting Ġthem Ġto ĠStatistics ĠCanada Ġ( Stat Can ). ĠEvery Ġyear , ĠCanadian ĠCancer ĠRegistry Ġ( CC R ) Ġuses Ġthe Ġinformation Ġsources Ġof ĠStat Can Ġto Ġcompile Ġan Ġannual Ġreport Ġon Ġcancer Ġand Ġtumor Ġdiseases . ĠMany Ġcountries Ġhave Ġtheir Ġown Ġcancer Ġregistry Ġprograms . ĠThese Ġprograms Ġrely Ġon Ġthe Ġacquisition Ġof Ġdiagnostic , Ġtreatment , Ġand Ġoutcome Ġinformation Ġthrough Ġmanual Ġprocessing Ġand Ġinterpretation Ġfrom Ġvarious Ġun struct ured Ġsources Ġ( e . g ., Ġpathology Ġreports , Ġautopsy / l abor atory Ġreports , Ġmedical Ġbilling Ġsumm aries ). ĠThe Ġmanual Ġclassification Ġof Ġcancer Ġpathology Ġreports Ġis Ġa Ġchallenging , Ġtime - consuming Ġtask Ġand Ġrequires Ġextensive Ġtraining ĠB IB REF 0 Ġ. </s> With Ġthe Ġcontinued Ġgrowth Ġin Ġthe Ġnumber Ġof Ġcancer Ġpatients , Ġand Ġthe Ġincrease Ġin Ġtreatment Ġcomplexity , Ġcancer Ġregist ries Ġface Ġa Ġsignificant Ġchallenge Ġin Ġmanually Ġreviewing Ġthe Ġlarge Ġquantity Ġof Ġreports ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġ. ĠIn Ġthis Ġsituation , ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġsystems Ġcan Ġoffer Ġa Ġunique Ġopportunity Ġto Ġautomatically Ġencode Ġthe Ġun struct ured Ġreports Ġinto Ġstructured Ġdata . ĠSince , Ġthe Ġregist ries Ġalready Ġhave Ġaccess Ġto Ġthe Ġlarge Ġquantity Ġof Ġhistorically Ġlabeled Ġand Ġencoded Ġreports , Ġa Ġsupervised Ġmachine Ġlearning Ġapproach Ġof Ġfeature Ġextraction Ġand Ġclassification Ġis Ġa Ġcompelling Ġdirection Ġfor Ġmaking Ġtheir Ġworkflow Ġmore Ġeffective Ġand Ġstreamlined . ĠIf Ġsuccessful , Ġsuch Ġa Ġsolution Ġwould Ġenable Ġprocessing Ġreports Ġin Ġmuch Ġlesser Ġtime Ġallowing Ġtrained Ġpersonnel Ġto Ġfocus Ġon Ġtheir Ġresearch Ġand Ġanalysis . ĠHowever , Ġdeveloping Ġan Ġautomated Ġsolution Ġwith Ġhigh Ġaccuracy Ġand Ġconsistency Ġacross Ġwide Ġvariety Ġof Ġreports Ġis Ġa Ġchallenging Ġproblem . </s> For Ġcancer Ġregist ries , Ġan Ġimportant Ġpiece Ġof Ġinformation Ġin Ġa Ġpathology Ġreport Ġis Ġthe Ġassociated ĠI CD - O Ġcode Ġwhich Ġdescribes Ġthe Ġpatient 's Ġhist ological Ġdiagnosis , Ġas Ġdescribed Ġby Ġthe ĠWorld ĠHealth ĠOrganization 's Ġ( WHO ) ĠInternational ĠClassification Ġof ĠDiseases Ġfor ĠOn c ology ĠB IB REF 2 Ġ. ĠPrediction Ġof Ġthe Ġprimary Ġdiagnosis Ġfrom Ġa Ġpathology Ġreport Ġprovides Ġa Ġvaluable Ġstarting Ġpoint Ġfor Ġexploration Ġof Ġmachine Ġlearning Ġtechniques Ġfor Ġautomated Ġcancer Ġsurveillance . ĠA Ġmajor Ġapplication Ġfor Ġthis Ġpurpose Ġwould Ġbe ĠâĢ ľ auto - reporting âĢ Ŀ Ġbased Ġon Ġanalysis Ġof Ġwhole Ġslide Ġimages , Ġthe Ġdigit ization Ġof Ġthe Ġbi opsy Ġglass Ġslides . ĠStruct ured , Ġsummarized Ġand Ġcategorized Ġreports Ġcan Ġbe Ġassociated Ġwith Ġthe Ġimage Ġcontent Ġwhen Ġsearching Ġin Ġlarge Ġarchives . ĠSuch Ġas Ġsystem Ġwould Ġbe Ġable Ġto Ġdrastically Ġincrease Ġthe Ġefficiency Ġof Ġdiagnostic Ġprocesses Ġfor Ġthe Ġmajority Ġof Ġcases Ġwhere Ġin Ġspite Ġof Ġobvious Ġprimary Ġdiagnosis , Ġstill Ġtime Ġand Ġeffort Ġis Ġrequired Ġfrom Ġthe Ġpath ologists Ġto Ġwrite Ġa Ġdescriptive Ġreport . </s> The Ġprimary Ġobjective Ġof Ġour Ġstudy Ġis Ġto Ġanalyze Ġthe Ġefficacy Ġof Ġexisting Ġmachine Ġlearning Ġapproaches Ġfor Ġthe Ġautomated Ġclassification Ġof Ġpathology Ġreports Ġinto Ġdifferent Ġdiagnosis Ġcategories . ĠWe Ġdemonstrate Ġthat ĠTF - ID F Ġfeature Ġvectors Ġcombined Ġwith Ġlinear ĠS VM Ġor ĠX GB o ost Ġclass ifier Ġcan Ġbe Ġan Ġeffective Ġmethod Ġfor Ġclassification Ġof Ġthe Ġreports , Ġachieving Ġup Ġto Ġ83 % Ġaccuracy . ĠWe Ġalso Ġshow Ġthat ĠTF - ID F Ġfeatures Ġare Ġcapable Ġof Ġidentifying Ġimportant Ġkeywords Ġwithin Ġa Ġpathology Ġreport . ĠFurthermore , Ġwe Ġhave Ġcreated Ġa Ġnew Ġdataset Ġconsisting Ġof Ġ1 , 949 Ġpathology Ġreports Ġacross Ġ37 Ġprimary Ġdiagnoses . ĠTaken Ġtogether , Ġour Ġexplor atory Ġexperiments Ġwith Ġa Ġnewly Ġintroduced Ġdataset Ġon Ġpathology Ġreports Ġopens Ġmany Ġnew Ġopportunities Ġfor Ġresearchers Ġto Ġdevelop Ġa Ġscalable Ġand Ġautomatic Ġinformation Ġextraction Ġfrom Ġun struct ured Ġpathology Ġreports . </s> Background </s> N LP Ġapproaches Ġfor Ġinformation Ġextraction Ġwithin Ġthe Ġbiomedical Ġresearch Ġareas Ġrange Ġfrom Ġrule - based Ġsystems ĠB IB REF 3 Ġ, Ġto Ġdomain - specific Ġsystems Ġusing Ġfeature - based Ġclassification ĠB IB REF 1 Ġ, Ġto Ġthe Ġrecent Ġdeep Ġnetworks Ġfor Ġend - to - end Ġfeature Ġextraction Ġand Ġclassification ĠB IB REF 0 Ġ. ĠN LP Ġhas Ġhad Ġvaried Ġdegree Ġof Ġsuccess Ġwith Ġfree - text Ġpathology Ġreports ĠB IB REF 4 Ġ. ĠVarious Ġstudies Ġhave Ġacknowledge Ġthe Ġsuccess Ġof ĠN LP Ġin Ġinterpreting Ġpathology Ġreports , Ġespecially Ġfor Ġclassification Ġtasks Ġor Ġextracting Ġa Ġsingle Ġattribute Ġfrom Ġa Ġreport ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. </s> The ĠCancer ĠText ĠInformation ĠExt raction ĠSystem Ġ( ca T IES ) ĠB IB REF 6 Ġis Ġa Ġframework Ġdeveloped Ġin Ġa Ġca BI G Ġproject Ġfocuses Ġon Ġinformation Ġextraction Ġfrom Ġpathology Ġreports . ĠSpecifically , Ġca T IES Ġextracts Ġinformation Ġfrom Ġsurgical Ġpathology Ġreports Ġ( SP R ) Ġwith Ġgood Ġprecision Ġas Ġwell Ġas Ġrecall . </s> Another Ġsystem Ġknown Ġas ĠOpen ĠRegistry ĠB IB REF 7 Ġis Ġcapable Ġof Ġfiltering Ġthe Ġreports Ġwith Ġdisease Ġcodes Ġcontaining Ġcancer . ĠIn ĠB IB REF 8 Ġ, Ġan Ġapproach Ġcalled ĠAutom ated ĠRet ri eval ĠConsole Ġ( ARC ) Ġis Ġproposed Ġwhich Ġuses Ġmachine Ġlearning Ġmodels Ġto Ġpredict Ġthe Ġdegree Ġof Ġassociation Ġof Ġa Ġgiven Ġpathology Ġor Ġrad iology Ġwith Ġthe Ġcancer . ĠThe Ġperformance Ġranges Ġfrom Ġan ĠF - me asure Ġof Ġ0 . 75 Ġfor Ġlung Ġcancer Ġto Ġ0 . 94 Ġfor Ġcol ore ct al Ġcancer . ĠHowever , ĠARC Ġuses Ġdomain - specific Ġrules Ġwhich Ġh iders Ġwith Ġthe Ġgeneral ization Ġof Ġthe Ġapproach Ġto Ġvariety Ġof Ġpathology Ġreports . </s> This Ġresearch Ġwork Ġis Ġinspired Ġby Ġthemes Ġemerging Ġin Ġmany Ġof Ġthe Ġabove Ġstudies . ĠSpecifically , Ġwe Ġare Ġevaluating Ġthe Ġtask Ġof Ġpredicting Ġthe Ġprimary Ġdiagnosis Ġfrom Ġthe Ġpathology Ġreport . ĠUnlike Ġprevious Ġapproaches , Ġthe Ġsystem Ġdoes Ġnot Ġrely Ġon Ġcustom Ġrule - based Ġknowledge , Ġdomain Ġspecific Ġfeatures , Ġbalanced Ġdataset Ġwith Ġfewer Ġnumber Ġof Ġclasses . </s> Materials Ġand ĠMethods </s> We Ġassembled Ġa Ġdataset Ġof Ġ1 , 949 Ġcleaned Ġpathology Ġreports . ĠEach Ġreport Ġis Ġassociated Ġwith Ġone Ġof Ġthe Ġ37 Ġdifferent Ġprimary Ġdiagnoses Ġbased Ġon ĠID C - O Ġcodes . ĠThe Ġreports Ġare Ġcollected Ġfrom Ġfour Ġdifferent Ġbody Ġparts Ġor Ġprimary Ġsites Ġfrom Ġmultiple Ġpatients . ĠThe Ġdistribution Ġof Ġreports Ġacross Ġdifferent Ġprimary Ġdiagnoses Ġand Ġprimary Ġsites Ġis Ġreported Ġin Ġtab : report - dist ribution . ĠThe Ġdataset Ġwas Ġdeveloped Ġin Ġthree Ġsteps Ġas Ġfollows . </s> Collect ing Ġpathology Ġreports : ĠThe Ġtotal Ġof Ġ11 , 112 Ġpathology Ġreports Ġwere Ġdownloaded Ġfrom ĠNC I 's ĠGen omic ĠData ĠCommons Ġ( G DC ) Ġdataset Ġin ĠPDF Ġformat ĠB IB REF 9 Ġ. ĠOut Ġof Ġall ĠPDF Ġfiles , Ġ1 , 949 Ġreports Ġwere Ġselected Ġacross Ġmultiple Ġpatients Ġfrom Ġfour Ġspecific Ġprimary Ġsites âĢĶ th ym us , Ġtest is , Ġlung , Ġand Ġkidney . ĠThe Ġselection Ġwas Ġprimarily Ġmade Ġbased Ġon Ġthe Ġquality Ġof ĠPDF Ġfiles . </s> C leaning Ġreports : ĠThe Ġnext Ġstep Ġwas Ġto Ġextract Ġthe Ġtext Ġcontent Ġfrom Ġthese Ġreports . ĠDue Ġto Ġthe Ġsignificant Ġtime Ġexpense Ġof Ġmanually Ġre - ty ping Ġall Ġthe Ġpathology Ġreports , Ġwe Ġdeveloped Ġa Ġnew Ġstrategy Ġto Ġprepare Ġour Ġdataset . ĠWe Ġapplied Ġan ĠOptical ĠCharacter ĠRecogn ition Ġ( OC R ) Ġsoftware Ġto Ġconvert Ġthe ĠPDF Ġreports Ġto Ġtext Ġfiles . ĠThen , Ġwe Ġmanually Ġinspected Ġall Ġgenerated Ġtext Ġfiles Ġto Ġfix Ġany Ġgrammar / spe lling Ġissues Ġand Ġirrelevant Ġcharacters Ġas Ġan Ġarte fact Ġproduced Ġby Ġthe ĠO CR Ġsystem . </s> Spl itting Ġinto Ġtraining - testing Ġdata : ĠWe Ġsplit Ġthe Ġcleaned Ġreports Ġinto Ġ70 % Ġand Ġ30 % Ġfor Ġtraining Ġand Ġtesting , Ġrespectively . ĠThis Ġsplit Ġresulted Ġin Ġ1 , 364 Ġtraining , Ġand Ġ5 85 Ġtesting Ġreports . </s> Pre - Process ing Ġof ĠReports </s> We Ġpre - process ed Ġthe Ġreports Ġby Ġsetting Ġtheir Ġtext Ġcontent Ġto Ġlower case Ġand Ġfiltering Ġout Ġany Ġnon - alph an umeric Ġcharacters . ĠWe Ġused ĠNL T K Ġlibrary Ġto Ġremove Ġstopping Ġwords , Ġe . g ., Ġ` the ', Ġ` an ', Ġ` was ', Ġ` if ' Ġand Ġso Ġon ĠB IB REF 10 Ġ. ĠWe Ġthen Ġanalyzed Ġthe Ġreports Ġto Ġfind Ġcommon Ġbig rams , Ġsuch Ġas ĠâĢ ľ l ung Ġp aren chy ma âĢ Ŀ , ĠâĢ ľ micro sc opic Ġexamination âĢ Ŀ , ĠâĢ ľ ly mph Ġnode âĢ Ŀ Ġetc . ĠWe Ġjoined Ġthe Ġbi agram s Ġwith Ġa Ġhyp hen , Ġconverting Ġthem Ġinto Ġa Ġsingle Ġword . ĠWe Ġfurther Ġremoved Ġthe Ġwords Ġthat Ġoccur Ġless Ġthan Ġ2 % Ġin Ġeach Ġof Ġthe Ġdiagnostic Ġcategory . ĠAs Ġwell , Ġwe Ġremoved Ġthe Ġwords Ġthat Ġoccur Ġmore Ġthan Ġ90 % Ġacross Ġall Ġthe Ġcategories . ĠWe Ġstored Ġeach Ġpre - process ed Ġreport Ġin Ġa Ġseparate Ġtext Ġfile . </s> TF - ID F Ġfeatures </s> TF - ID F Ġstands Ġfor ĠTerm ĠFrequency - In verse ĠDocument ĠFrequency , Ġand Ġit Ġis Ġa Ġuseful Ġweight ing Ġscheme Ġin Ġinformation Ġretrieval Ġand Ġtext Ġmining . ĠTF - ID F Ġsignifies Ġthe Ġimportance Ġof Ġa Ġterm Ġin Ġa Ġdocument Ġwithin Ġa Ġcorpus . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġa Ġdocument Ġhere Ġrefers Ġto Ġa Ġpathology Ġreport , Ġa Ġcorpus Ġrefers Ġto Ġthe Ġcollection Ġof Ġreports , Ġand Ġa Ġterm Ġrefers Ġto Ġa Ġsingle Ġword Ġin Ġa Ġreport . ĠThe ĠTF - ID F Ġweight Ġfor Ġa Ġterm ĠIN LINE FORM 0 Ġin Ġa Ġdocument ĠIN LINE FORM 1 Ġis Ġgiven Ġby ĠDIS PLAY FORM 0 </s> We Ġperformed Ġthe Ġfollowing Ġsteps Ġto Ġtransform Ġa Ġpathology Ġreport Ġinto Ġa Ġfeature Ġvector : </s> Create Ġa Ġset Ġof Ġvocabulary Ġcontaining Ġall Ġunique Ġwords Ġfrom Ġall Ġthe Ġpre - process ed Ġtraining Ġreports . </s> Create Ġa Ġzero Ġvector ĠIN LINE FORM 0 Ġof Ġthe Ġsame Ġlength Ġas Ġthe Ġvocabulary . </s> For Ġeach Ġword ĠIN LINE FORM 0 Ġin Ġa Ġreport ĠIN LINE FORM 1 Ġ, Ġset Ġthe Ġcorresponding Ġindex Ġin ĠIN LINE FORM 2 Ġto ĠIN LINE FORM 3 Ġ. </s> The Ġresultant ĠIN LINE FORM 0 Ġis Ġa Ġfeature Ġvector Ġfor Ġthe Ġreport ĠIN LINE FORM 1 Ġand Ġit Ġis Ġa Ġhighly Ġsparse Ġvector . </s> Key word Ġextraction Ġand Ġtopic Ġmodelling </s> The Ġkeyword Ġextraction Ġinvolves Ġidentifying Ġimportant Ġwords Ġwithin Ġreports Ġthat Ġsummarizes Ġits Ġcontent . ĠIn Ġcontrast , Ġthe Ġtopic Ġmodelling Ġallows Ġgrouping Ġthese Ġkeywords Ġusing Ġan Ġintelligent Ġscheme , Ġenabling Ġusers Ġto Ġfurther Ġfocus Ġon Ġcertain Ġaspects Ġof Ġa Ġdocument . ĠAll Ġthe Ġwords Ġin Ġa Ġpathology Ġreport Ġare Ġsorted Ġaccording Ġto Ġtheir ĠTF - ID F Ġweights . ĠThe Ġtop ĠIN LINE FORM 0 Ġsorted Ġwords Ġconstitute Ġthe Ġtop ĠIN LINE FORM 1 Ġkeywords Ġfor Ġthe Ġreport . ĠThe ĠIN LINE FORM 2 Ġis Ġempir ically Ġset Ġto Ġ50 Ġwithin Ġthis Ġresearch . ĠThe Ġextracted Ġkeywords Ġare Ġfurther Ġgrouped Ġinto Ġdifferent Ġtopics Ġby Ġusing Ġlatent ĠDir ich let Ġallocation Ġ( L DA ) ĠB IB REF 11 Ġ. ĠThe Ġkeywords Ġin Ġa Ġreport Ġare Ġhighlighted Ġusing Ġthe Ġcolor Ġtheme Ġbased Ġon Ġtheir Ġtopics . </s> E val uation Ġmetrics </s> Each Ġmodel Ġis Ġevaluated Ġusing Ġtwo Ġstandard ĠN LP Ġmetrics âĢĶ micro Ġand Ġmacro Ġaveraged ĠF - sc ores , Ġthe Ġharmonic Ġmean Ġof Ġrelated Ġmetrics Ġprecision Ġand Ġrecall . ĠFor Ġeach Ġdiagnostic Ġcategory ĠIN LINE FORM 0 Ġfrom Ġa Ġset Ġof Ġ37 Ġdifferent Ġclasses ĠIN LINE FORM 1 Ġ, Ġthe Ġnumber Ġof Ġtrue Ġpositives ĠIN LINE FORM 2 Ġ, Ġfalse Ġpositives ĠIN LINE FORM 3 Ġ, Ġand Ġfalse Ġnegatives ĠIN LINE FORM 4 Ġ, Ġthe Ġmicro ĠF - score Ġis Ġdefined Ġas ĠDIS PLAY FORM 0 </s> where as Ġmacro ĠF - score Ġis Ġgiven Ġby ĠDIS PLAY FORM 0 </s> In Ġsummary , Ġmicro - aver aged Ġmetrics Ġhave Ġclass Ġrepresentation Ġroughly Ġproportional Ġto Ġtheir Ġtest Ġset Ġrepresentation Ġ( same Ġas Ġaccuracy Ġfor Ġclassification Ġproblem Ġwith Ġa Ġsingle Ġlabel Ġper Ġdata Ġpoint ), Ġwhereas Ġmacro - aver aged Ġmetrics Ġare Ġaveraged Ġby Ġclass Ġwithout Ġweight ing Ġby Ġclass Ġprevalence ĠB IB REF 12 Ġ. </s> Exper imental Ġsetting </s> In Ġthis Ġstudy , Ġwe Ġperformed Ġtwo Ġdifferent Ġseries Ġof Ġexperiments : Ġi ) Ġevaluating Ġthe Ġperformance Ġof ĠTF - ID F Ġfeatures Ġand Ġvarious Ġmachine Ġlearning Ġclass ifiers Ġon Ġthe Ġtask Ġof Ġpredicting Ġprimary Ġdiagnosis Ġfrom Ġthe Ġtext Ġcontent Ġof Ġa Ġgiven Ġreport , Ġand Ġii ) Ġusing ĠTF - ID F Ġand ĠL DA Ġtechniques Ġto Ġhighlight Ġthe Ġimportant Ġkeywords Ġwithin Ġa Ġreport . ĠFor Ġthe Ġfirst Ġexperiment Ġseries , Ġtraining Ġreports Ġare Ġpre - process ed , Ġthen Ġtheir ĠTF - ID F Ġfeatures Ġare Ġextracted . ĠThe ĠTF - ID F Ġfeatures Ġand Ġthe Ġtraining Ġlabels Ġare Ġused Ġto Ġtrain Ġdifferent Ġclassification Ġmodels . ĠThese Ġdifferent Ġclassification Ġmodels Ġand Ġtheir Ġhyper - param eters Ġare Ġreported Ġin Ġtab : class ifier . ĠThe Ġperformance Ġof Ġclass ifiers Ġis Ġmeasured Ġquant itatively Ġon Ġthe Ġtest Ġdataset Ġusing Ġthe Ġevaluation Ġmetrics Ġdiscussed Ġin Ġthe Ġprevious Ġsection . ĠFor Ġthe Ġsecond Ġexperiment Ġseries , Ġa Ġrandom Ġreport Ġis Ġselected Ġand Ġits Ġtop Ġ50 Ġkeywords Ġare Ġextracted Ġusing ĠTF - ID F Ġweights . ĠThese Ġ50 Ġkeywords Ġare Ġhighlighted Ġusing Ġdifferent Ġcolors Ġbased Ġon Ġtheir Ġassociated Ġtopic , Ġwhich Ġare Ġextracted Ġthrough ĠL DA . ĠA Ġnon - ex pert Ġbased Ġqualitative Ġinspection Ġis Ġperformed Ġon Ġthe Ġextracted Ġkeywords Ġand Ġtheir Ġcorresponding Ġtopics . </s> Exper iment ĠSeries Ġ1 </s> A Ġclassification Ġmodel Ġis Ġtrained Ġto Ġpredict Ġthe Ġprimary Ġdiagnosis Ġgiven Ġthe Ġcontent Ġof Ġthe Ġcancer Ġpathology Ġreport . ĠThe Ġperformance Ġresults Ġon Ġthis Ġtask Ġare Ġreported Ġin Ġtab : results . ĠWe Ġcan Ġobserve Ġthat Ġthe ĠX GB o ost Ġclass ifier Ġoutper formed Ġall Ġother Ġmodels Ġfor Ġboth Ġthe Ġmicro ĠF - score Ġmetric , Ġwith Ġa Ġscore Ġof Ġ0 . 92 , Ġand Ġthe Ġmacro ĠF - score Ġmetric , Ġwith Ġa Ġscore Ġof Ġ0 . 31 . ĠThis Ġwas Ġan Ġimprovement Ġof Ġ7 % Ġfor Ġthe Ġmicro ĠF - score Ġover Ġthe Ġnext Ġbest Ġmodel , ĠS VM - L , Ġand Ġa Ġmarginal Ġimprovement Ġof Ġ5 % Ġfor Ġmacro ĠF - score . ĠIt Ġis Ġinteresting Ġto Ġnote Ġthat ĠS VM Ġwith Ġlinear Ġkernels Ġperforms Ġmuch Ġbetter Ġthan ĠS VM Ġwith ĠRB F Ġkernel , Ġscoring Ġ9 % Ġon Ġthe Ġmacro ĠF - score Ġand Ġ12 % Ġmore Ġon Ġthe Ġmicro ĠF - score . ĠIt Ġis Ġsuspected Ġthat Ġsince Ġwords Ġused Ġin Ġprimary Ġdiagnosis Ġitself Ġoccur Ġin Ġsome Ġreports , Ġthus Ġenabling Ġthe Ġlinear Ġmodels Ġto Ġoutper form Ġcomplex Ġmodels . </s> Exper iment ĠSeries Ġ2 </s> fig : key words Ġshows Ġthe Ġtop Ġ50 Ġkeywords Ġhighlighted Ġusing ĠTF - ID F Ġand ĠL DA . ĠThe Ġproposed Ġapproach Ġhas Ġperformed Ġwell Ġin Ġhighlighting Ġthe Ġimportant Ġregions , Ġfor Ġexample Ġthe Ġtopic Ġhighlighted Ġwith Ġa Ġred Ġcolor Ġcontaining ĠâĢ ľ pres ence Ġrange Ġtumor Ġnec rosis âĢ Ŀ Ġprovides Ġuseful Ġbiomark er Ġinformation Ġto Ġreaders . </s> Con clusions </s> We Ġproposed Ġa Ġsimple Ġyet Ġefficient ĠTF - ID F Ġmethod Ġto Ġextract Ġand Ġcorrobor ate Ġuseful Ġkeywords Ġfrom Ġpathology Ġcancer Ġreports . ĠEnc oding Ġa Ġpathology Ġreport Ġfor Ġcancer Ġand Ġtumor Ġsurveillance Ġis Ġa Ġlabor ious Ġtask , Ġand Ġsometimes Ġit Ġis Ġsubjected Ġto Ġhuman Ġerrors Ġand Ġvariability Ġin Ġthe Ġinterpretation . ĠOne Ġof Ġthe Ġmost Ġimportant Ġaspects Ġof Ġencoding Ġa Ġpathology Ġreport Ġinvolves Ġextracting Ġthe Ġprimary Ġdiagnosis . ĠThis Ġmay Ġbe Ġvery Ġuseful Ġfor Ġcontent - based Ġimage Ġretrieval Ġto Ġcombine Ġwith Ġvisual Ġinformation . ĠWe Ġused Ġexisting Ġclassification Ġmodel Ġand ĠTF - ID F Ġfeatures Ġto Ġpredict Ġthe Ġprimary Ġdiagnosis . ĠWe Ġachieved Ġup Ġto Ġ92 % Ġaccuracy Ġusing ĠX GB o ost Ġclass ifier . ĠThe Ġprediction Ġaccuracy Ġem powers Ġthe Ġadoption Ġof Ġmachine Ġlearning Ġmethods Ġfor Ġautomated Ġinformation Ġextraction Ġfrom Ġpathology Ġreports .
Introduction Cancer is one of the leading causes of death in the world, with over 80,000 deaths registered in Canada in 2017 (Canadian Cancer Statistics 2017). A computer-aided system for cancer diagnosis usually involves a pathologist rendering a descriptive report after examining the tissue glass slides obtained from the biopsy of a patient. A pathology report contains specific analysis of cells and tissues, and other histopathological indicators that are crucial for diagnosing malignancies. An average sized laboratory may produces a large quantity of pathology reports annually (e.g., in excess of 50,000), but these reports are written in mostly unstructured text and with no direct link to the tissue sample. Furthermore, the report for each patient is a personalized document and offers very high variability in terminology due to lack of standards and may even include misspellings and missing punctuation, clinical diagnoses interspersed with complex explanations, different terminology to label the same malignancy, and information about multiple carcinoma appearances included in a single report BIBREF0 . In Canada, each Provincial and Territorial Cancer Registry (PTCR) is responsible for collecting the data about cancer diseases and reporting them to Statistics Canada (StatCan). Every year, Canadian Cancer Registry (CCR) uses the information sources of StatCan to compile an annual report on cancer and tumor diseases. Many countries have their own cancer registry programs. These programs rely on the acquisition of diagnostic, treatment, and outcome information through manual processing and interpretation from various unstructured sources (e.g., pathology reports, autopsy/laboratory reports, medical billing summaries). The manual classification of cancer pathology reports is a challenging, time-consuming task and requires extensive training BIBREF0 . With the continued growth in the number of cancer patients, and the increase in treatment complexity, cancer registries face a significant challenge in manually reviewing the large quantity of reports BIBREF1 , BIBREF0 . In this situation, Natural Language Processing (NLP) systems can offer a unique opportunity to automatically encode the unstructured reports into structured data. Since, the registries already have access to the large quantity of historically labeled and encoded reports, a supervised machine learning approach of feature extraction and classification is a compelling direction for making their workflow more effective and streamlined. If successful, such a solution would enable processing reports in much lesser time allowing trained personnel to focus on their research and analysis. However, developing an automated solution with high accuracy and consistency across wide variety of reports is a challenging problem. For cancer registries, an important piece of information in a pathology report is the associated ICD-O code which describes the patient's histological diagnosis, as described by the World Health Organization's (WHO) International Classification of Diseases for Oncology BIBREF2 . Prediction of the primary diagnosis from a pathology report provides a valuable starting point for exploration of machine learning techniques for automated cancer surveillance. A major application for this purpose would be “auto-reporting” based on analysis of whole slide images, the digitization of the biopsy glass slides. Structured, summarized and categorized reports can be associated with the image content when searching in large archives. Such as system would be able to drastically increase the efficiency of diagnostic processes for the majority of cases where in spite of obvious primary diagnosis, still time and effort is required from the pathologists to write a descriptive report. The primary objective of our study is to analyze the efficacy of existing machine learning approaches for the automated classification of pathology reports into different diagnosis categories. We demonstrate that TF-IDF feature vectors combined with linear SVM or XGBoost classifier can be an effective method for classification of the reports, achieving up to 83% accuracy. We also show that TF-IDF features are capable of identifying important keywords within a pathology report. Furthermore, we have created a new dataset consisting of 1,949 pathology reports across 37 primary diagnoses. Taken together, our exploratory experiments with a newly introduced dataset on pathology reports opens many new opportunities for researchers to develop a scalable and automatic information extraction from unstructured pathology reports. Background NLP approaches for information extraction within the biomedical research areas range from rule-based systems BIBREF3 , to domain-specific systems using feature-based classification BIBREF1 , to the recent deep networks for end-to-end feature extraction and classification BIBREF0 . NLP has had varied degree of success with free-text pathology reports BIBREF4 . Various studies have acknowledge the success of NLP in interpreting pathology reports, especially for classification tasks or extracting a single attribute from a report BIBREF4 , BIBREF5 . The Cancer Text Information Extraction System (caTIES) BIBREF6 is a framework developed in a caBIG project focuses on information extraction from pathology reports. Specifically, caTIES extracts information from surgical pathology reports (SPR) with good precision as well as recall. Another system known as Open Registry BIBREF7 is capable of filtering the reports with disease codes containing cancer. In BIBREF8 , an approach called Automated Retrieval Console (ARC) is proposed which uses machine learning models to predict the degree of association of a given pathology or radiology with the cancer. The performance ranges from an F-measure of 0.75 for lung cancer to 0.94 for colorectal cancer. However, ARC uses domain-specific rules which hiders with the generalization of the approach to variety of pathology reports. This research work is inspired by themes emerging in many of the above studies. Specifically, we are evaluating the task of predicting the primary diagnosis from the pathology report. Unlike previous approaches, the system does not rely on custom rule-based knowledge, domain specific features, balanced dataset with fewer number of classes. Materials and Methods We assembled a dataset of 1,949 cleaned pathology reports. Each report is associated with one of the 37 different primary diagnoses based on IDC-O codes. The reports are collected from four different body parts or primary sites from multiple patients. The distribution of reports across different primary diagnoses and primary sites is reported in tab:report-distribution. The dataset was developed in three steps as follows. Collecting pathology reports: The total of 11,112 pathology reports were downloaded from NCI's Genomic Data Commons (GDC) dataset in PDF format BIBREF9 . Out of all PDF files, 1,949 reports were selected across multiple patients from four specific primary sites—thymus, testis, lung, and kidney. The selection was primarily made based on the quality of PDF files. Cleaning reports: The next step was to extract the text content from these reports. Due to the significant time expense of manually re-typing all the pathology reports, we developed a new strategy to prepare our dataset. We applied an Optical Character Recognition (OCR) software to convert the PDF reports to text files. Then, we manually inspected all generated text files to fix any grammar/spelling issues and irrelevant characters as an artefact produced by the OCR system. Splitting into training-testing data: We split the cleaned reports into 70% and 30% for training and testing, respectively. This split resulted in 1,364 training, and 585 testing reports. Pre-Processing of Reports We pre-processed the reports by setting their text content to lowercase and filtering out any non-alphanumeric characters. We used NLTK library to remove stopping words, e.g., `the', `an', `was', `if' and so on BIBREF10 . We then analyzed the reports to find common bigrams, such as “lung parenchyma”, “microscopic examination”, “lymph node” etc. We joined the biagrams with a hyphen, converting them into a single word. We further removed the words that occur less than 2% in each of the diagnostic category. As well, we removed the words that occur more than 90% across all the categories. We stored each pre-processed report in a separate text file. TF-IDF features TF-IDF stands for Term Frequency-Inverse Document Frequency, and it is a useful weighting scheme in information retrieval and text mining. TF-IDF signifies the importance of a term in a document within a corpus. It is important to note that a document here refers to a pathology report, a corpus refers to the collection of reports, and a term refers to a single word in a report. The TF-IDF weight for a term INLINEFORM0 in a document INLINEFORM1 is given by DISPLAYFORM0 We performed the following steps to transform a pathology report into a feature vector: Create a set of vocabulary containing all unique words from all the pre-processed training reports. Create a zero vector INLINEFORM0 of the same length as the vocabulary. For each word INLINEFORM0 in a report INLINEFORM1 , set the corresponding index in INLINEFORM2 to INLINEFORM3 . The resultant INLINEFORM0 is a feature vector for the report INLINEFORM1 and it is a highly sparse vector. Keyword extraction and topic modelling The keyword extraction involves identifying important words within reports that summarizes its content. In contrast, the topic modelling allows grouping these keywords using an intelligent scheme, enabling users to further focus on certain aspects of a document. All the words in a pathology report are sorted according to their TF-IDF weights. The top INLINEFORM0 sorted words constitute the top INLINEFORM1 keywords for the report. The INLINEFORM2 is empirically set to 50 within this research. The extracted keywords are further grouped into different topics by using latent Dirichlet allocation (LDA) BIBREF11 . The keywords in a report are highlighted using the color theme based on their topics. Evaluation metrics Each model is evaluated using two standard NLP metrics—micro and macro averaged F-scores, the harmonic mean of related metrics precision and recall. For each diagnostic category INLINEFORM0 from a set of 37 different classes INLINEFORM1 , the number of true positives INLINEFORM2 , false positives INLINEFORM3 , and false negatives INLINEFORM4 , the micro F-score is defined as DISPLAYFORM0 whereas macro F-score is given by DISPLAYFORM0 In summary, micro-averaged metrics have class representation roughly proportional to their test set representation (same as accuracy for classification problem with a single label per data point), whereas macro-averaged metrics are averaged by class without weighting by class prevalence BIBREF12 . Experimental setting In this study, we performed two different series of experiments: i) evaluating the performance of TF-IDF features and various machine learning classifiers on the task of predicting primary diagnosis from the text content of a given report, and ii) using TF-IDF and LDA techniques to highlight the important keywords within a report. For the first experiment series, training reports are pre-processed, then their TF-IDF features are extracted. The TF-IDF features and the training labels are used to train different classification models. These different classification models and their hyper-parameters are reported in tab:classifier. The performance of classifiers is measured quantitatively on the test dataset using the evaluation metrics discussed in the previous section. For the second experiment series, a random report is selected and its top 50 keywords are extracted using TF-IDF weights. These 50 keywords are highlighted using different colors based on their associated topic, which are extracted through LDA. A non-expert based qualitative inspection is performed on the extracted keywords and their corresponding topics. Experiment Series 1 A classification model is trained to predict the primary diagnosis given the content of the cancer pathology report. The performance results on this task are reported in tab:results. We can observe that the XGBoost classifier outperformed all other models for both the micro F-score metric, with a score of 0.92, and the macro F-score metric, with a score of 0.31. This was an improvement of 7% for the micro F-score over the next best model, SVM-L, and a marginal improvement of 5% for macro F-score. It is interesting to note that SVM with linear kernels performs much better than SVM with RBF kernel, scoring 9% on the macro F-score and 12% more on the micro F-score. It is suspected that since words used in primary diagnosis itself occur in some reports, thus enabling the linear models to outperform complex models. Experiment Series 2 fig:keywords shows the top 50 keywords highlighted using TF-IDF and LDA. The proposed approach has performed well in highlighting the important regions, for example the topic highlighted with a red color containing “presence range tumor necrosis” provides useful biomarker information to readers. Conclusions We proposed a simple yet efficient TF-IDF method to extract and corroborate useful keywords from pathology cancer reports. Encoding a pathology report for cancer and tumor surveillance is a laborious task, and sometimes it is subjected to human errors and variability in the interpretation. One of the most important aspects of encoding a pathology report involves extracting the primary diagnosis. This may be very useful for content-based image retrieval to combine with visual information. We used existing classification model and TF-IDF features to predict the primary diagnosis. We achieved up to 92% accuracy using XGBoost classifier. The prediction accuracy empowers the adoption of machine learning methods for automated information extraction from pathology reports.
What features are used?
1eef2d2c296fdd10b08bf7b4ff7792cccf177d3b
What Ġfeatures Ġare Ġused ?
Un answer able
<s> What Ġfeatures Ġare Ġused ? </s> Introduction </s> C ancer Ġis Ġone Ġof Ġthe Ġleading Ġcauses Ġof Ġdeath Ġin Ġthe Ġworld , Ġwith Ġover Ġ80 , 000 Ġdeaths Ġregistered Ġin ĠCanada Ġin Ġ2017 Ġ( Canadian ĠCancer ĠStatistics Ġ2017 ). ĠA Ġcomputer - a ided Ġsystem Ġfor Ġcancer Ġdiagnosis Ġusually Ġinvolves Ġa Ġpath ologist Ġrendering Ġa Ġdescriptive Ġreport Ġafter Ġexamining Ġthe Ġtissue Ġglass Ġslides Ġobtained Ġfrom Ġthe Ġbi opsy Ġof Ġa Ġpatient . ĠA Ġpathology Ġreport Ġcontains Ġspecific Ġanalysis Ġof Ġcells Ġand Ġtissues , Ġand Ġother Ġhist opath ological Ġindicators Ġthat Ġare Ġcrucial Ġfor Ġdiagn osing Ġmal ign ancies . ĠAn Ġaverage Ġsized Ġlaboratory Ġmay Ġproduces Ġa Ġlarge Ġquantity Ġof Ġpathology Ġreports Ġannually Ġ( e . g ., Ġin Ġexcess Ġof Ġ50 , 000 ), Ġbut Ġthese Ġreports Ġare Ġwritten Ġin Ġmostly Ġun struct ured Ġtext Ġand Ġwith Ġno Ġdirect Ġlink Ġto Ġthe Ġtissue Ġsample . ĠFurthermore , Ġthe Ġreport Ġfor Ġeach Ġpatient Ġis Ġa Ġpersonalized Ġdocument Ġand Ġoffers Ġvery Ġhigh Ġvariability Ġin Ġterminology Ġdue Ġto Ġlack Ġof Ġstandards Ġand Ġmay Ġeven Ġinclude Ġmiss pell ings Ġand Ġmissing Ġpunct uation , Ġclinical Ġdiagnoses Ġinter sp ersed Ġwith Ġcomplex Ġexplanations , Ġdifferent Ġterminology Ġto Ġlabel Ġthe Ġsame Ġmal ign ancy , Ġand Ġinformation Ġabout Ġmultiple Ġcarcin oma Ġappearances Ġincluded Ġin Ġa Ġsingle Ġreport ĠB IB REF 0 Ġ. </s> In ĠCanada , Ġeach ĠProvincial Ġand ĠTerrit orial ĠCancer ĠRegistry Ġ( P TC R ) Ġis Ġresponsible Ġfor Ġcollecting Ġthe Ġdata Ġabout Ġcancer Ġdiseases Ġand Ġreporting Ġthem Ġto ĠStatistics ĠCanada Ġ( Stat Can ). ĠEvery Ġyear , ĠCanadian ĠCancer ĠRegistry Ġ( CC R ) Ġuses Ġthe Ġinformation Ġsources Ġof ĠStat Can Ġto Ġcompile Ġan Ġannual Ġreport Ġon Ġcancer Ġand Ġtumor Ġdiseases . ĠMany Ġcountries Ġhave Ġtheir Ġown Ġcancer Ġregistry Ġprograms . ĠThese Ġprograms Ġrely Ġon Ġthe Ġacquisition Ġof Ġdiagnostic , Ġtreatment , Ġand Ġoutcome Ġinformation Ġthrough Ġmanual Ġprocessing Ġand Ġinterpretation Ġfrom Ġvarious Ġun struct ured Ġsources Ġ( e . g ., Ġpathology Ġreports , Ġautopsy / l abor atory Ġreports , Ġmedical Ġbilling Ġsumm aries ). ĠThe Ġmanual Ġclassification Ġof Ġcancer Ġpathology Ġreports Ġis Ġa Ġchallenging , Ġtime - consuming Ġtask Ġand Ġrequires Ġextensive Ġtraining ĠB IB REF 0 Ġ. </s> With Ġthe Ġcontinued Ġgrowth Ġin Ġthe Ġnumber Ġof Ġcancer Ġpatients , Ġand Ġthe Ġincrease Ġin Ġtreatment Ġcomplexity , Ġcancer Ġregist ries Ġface Ġa Ġsignificant Ġchallenge Ġin Ġmanually Ġreviewing Ġthe Ġlarge Ġquantity Ġof Ġreports ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġ. ĠIn Ġthis Ġsituation , ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġsystems Ġcan Ġoffer Ġa Ġunique Ġopportunity Ġto Ġautomatically Ġencode Ġthe Ġun struct ured Ġreports Ġinto Ġstructured Ġdata . ĠSince , Ġthe Ġregist ries Ġalready Ġhave Ġaccess Ġto Ġthe Ġlarge Ġquantity Ġof Ġhistorically Ġlabeled Ġand Ġencoded Ġreports , Ġa Ġsupervised Ġmachine Ġlearning Ġapproach Ġof Ġfeature Ġextraction Ġand Ġclassification Ġis Ġa Ġcompelling Ġdirection Ġfor Ġmaking Ġtheir Ġworkflow Ġmore Ġeffective Ġand Ġstreamlined . ĠIf Ġsuccessful , Ġsuch Ġa Ġsolution Ġwould Ġenable Ġprocessing Ġreports Ġin Ġmuch Ġlesser Ġtime Ġallowing Ġtrained Ġpersonnel Ġto Ġfocus Ġon Ġtheir Ġresearch Ġand Ġanalysis . ĠHowever , Ġdeveloping Ġan Ġautomated Ġsolution Ġwith Ġhigh Ġaccuracy Ġand Ġconsistency Ġacross Ġwide Ġvariety Ġof Ġreports Ġis Ġa Ġchallenging Ġproblem . </s> For Ġcancer Ġregist ries , Ġan Ġimportant Ġpiece Ġof Ġinformation Ġin Ġa Ġpathology Ġreport Ġis Ġthe Ġassociated ĠI CD - O Ġcode Ġwhich Ġdescribes Ġthe Ġpatient 's Ġhist ological Ġdiagnosis , Ġas Ġdescribed Ġby Ġthe ĠWorld ĠHealth ĠOrganization 's Ġ( WHO ) ĠInternational ĠClassification Ġof ĠDiseases Ġfor ĠOn c ology ĠB IB REF 2 Ġ. ĠPrediction Ġof Ġthe Ġprimary Ġdiagnosis Ġfrom Ġa Ġpathology Ġreport Ġprovides Ġa Ġvaluable Ġstarting Ġpoint Ġfor Ġexploration Ġof Ġmachine Ġlearning Ġtechniques Ġfor Ġautomated Ġcancer Ġsurveillance . ĠA Ġmajor Ġapplication Ġfor Ġthis Ġpurpose Ġwould Ġbe ĠâĢ ľ auto - reporting âĢ Ŀ Ġbased Ġon Ġanalysis Ġof Ġwhole Ġslide Ġimages , Ġthe Ġdigit ization Ġof Ġthe Ġbi opsy Ġglass Ġslides . ĠStruct ured , Ġsummarized Ġand Ġcategorized Ġreports Ġcan Ġbe Ġassociated Ġwith Ġthe Ġimage Ġcontent Ġwhen Ġsearching Ġin Ġlarge Ġarchives . ĠSuch Ġas Ġsystem Ġwould Ġbe Ġable Ġto Ġdrastically Ġincrease Ġthe Ġefficiency Ġof Ġdiagnostic Ġprocesses Ġfor Ġthe Ġmajority Ġof Ġcases Ġwhere Ġin Ġspite Ġof Ġobvious Ġprimary Ġdiagnosis , Ġstill Ġtime Ġand Ġeffort Ġis Ġrequired Ġfrom Ġthe Ġpath ologists Ġto Ġwrite Ġa Ġdescriptive Ġreport . </s> The Ġprimary Ġobjective Ġof Ġour Ġstudy Ġis Ġto Ġanalyze Ġthe Ġefficacy Ġof Ġexisting Ġmachine Ġlearning Ġapproaches Ġfor Ġthe Ġautomated Ġclassification Ġof Ġpathology Ġreports Ġinto Ġdifferent Ġdiagnosis Ġcategories . ĠWe Ġdemonstrate Ġthat ĠTF - ID F Ġfeature Ġvectors Ġcombined Ġwith Ġlinear ĠS VM Ġor ĠX GB o ost Ġclass ifier Ġcan Ġbe Ġan Ġeffective Ġmethod Ġfor Ġclassification Ġof Ġthe Ġreports , Ġachieving Ġup Ġto Ġ83 % Ġaccuracy . ĠWe Ġalso Ġshow Ġthat ĠTF - ID F Ġfeatures Ġare Ġcapable Ġof Ġidentifying Ġimportant Ġkeywords Ġwithin Ġa Ġpathology Ġreport . ĠFurthermore , Ġwe Ġhave Ġcreated Ġa Ġnew Ġdataset Ġconsisting Ġof Ġ1 , 949 Ġpathology Ġreports Ġacross Ġ37 Ġprimary Ġdiagnoses . ĠTaken Ġtogether , Ġour Ġexplor atory Ġexperiments Ġwith Ġa Ġnewly Ġintroduced Ġdataset Ġon Ġpathology Ġreports Ġopens Ġmany Ġnew Ġopportunities Ġfor Ġresearchers Ġto Ġdevelop Ġa Ġscalable Ġand Ġautomatic Ġinformation Ġextraction Ġfrom Ġun struct ured Ġpathology Ġreports . </s> Background </s> N LP Ġapproaches Ġfor Ġinformation Ġextraction Ġwithin Ġthe Ġbiomedical Ġresearch Ġareas Ġrange Ġfrom Ġrule - based Ġsystems ĠB IB REF 3 Ġ, Ġto Ġdomain - specific Ġsystems Ġusing Ġfeature - based Ġclassification ĠB IB REF 1 Ġ, Ġto Ġthe Ġrecent Ġdeep Ġnetworks Ġfor Ġend - to - end Ġfeature Ġextraction Ġand Ġclassification ĠB IB REF 0 Ġ. ĠN LP Ġhas Ġhad Ġvaried Ġdegree Ġof Ġsuccess Ġwith Ġfree - text Ġpathology Ġreports ĠB IB REF 4 Ġ. ĠVarious Ġstudies Ġhave Ġacknowledge Ġthe Ġsuccess Ġof ĠN LP Ġin Ġinterpreting Ġpathology Ġreports , Ġespecially Ġfor Ġclassification Ġtasks Ġor Ġextracting Ġa Ġsingle Ġattribute Ġfrom Ġa Ġreport ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. </s> The ĠCancer ĠText ĠInformation ĠExt raction ĠSystem Ġ( ca T IES ) ĠB IB REF 6 Ġis Ġa Ġframework Ġdeveloped Ġin Ġa Ġca BI G Ġproject Ġfocuses Ġon Ġinformation Ġextraction Ġfrom Ġpathology Ġreports . ĠSpecifically , Ġca T IES Ġextracts Ġinformation Ġfrom Ġsurgical Ġpathology Ġreports Ġ( SP R ) Ġwith Ġgood Ġprecision Ġas Ġwell Ġas Ġrecall . </s> Another Ġsystem Ġknown Ġas ĠOpen ĠRegistry ĠB IB REF 7 Ġis Ġcapable Ġof Ġfiltering Ġthe Ġreports Ġwith Ġdisease Ġcodes Ġcontaining Ġcancer . ĠIn ĠB IB REF 8 Ġ, Ġan Ġapproach Ġcalled ĠAutom ated ĠRet ri eval ĠConsole Ġ( ARC ) Ġis Ġproposed Ġwhich Ġuses Ġmachine Ġlearning Ġmodels Ġto Ġpredict Ġthe Ġdegree Ġof Ġassociation Ġof Ġa Ġgiven Ġpathology Ġor Ġrad iology Ġwith Ġthe Ġcancer . ĠThe Ġperformance Ġranges Ġfrom Ġan ĠF - me asure Ġof Ġ0 . 75 Ġfor Ġlung Ġcancer Ġto Ġ0 . 94 Ġfor Ġcol ore ct al Ġcancer . ĠHowever , ĠARC Ġuses Ġdomain - specific Ġrules Ġwhich Ġh iders Ġwith Ġthe Ġgeneral ization Ġof Ġthe Ġapproach Ġto Ġvariety Ġof Ġpathology Ġreports . </s> This Ġresearch Ġwork Ġis Ġinspired Ġby Ġthemes Ġemerging Ġin Ġmany Ġof Ġthe Ġabove Ġstudies . ĠSpecifically , Ġwe Ġare Ġevaluating Ġthe Ġtask Ġof Ġpredicting Ġthe Ġprimary Ġdiagnosis Ġfrom Ġthe Ġpathology Ġreport . ĠUnlike Ġprevious Ġapproaches , Ġthe Ġsystem Ġdoes Ġnot Ġrely Ġon Ġcustom Ġrule - based Ġknowledge , Ġdomain Ġspecific Ġfeatures , Ġbalanced Ġdataset Ġwith Ġfewer Ġnumber Ġof Ġclasses . </s> Materials Ġand ĠMethods </s> We Ġassembled Ġa Ġdataset Ġof Ġ1 , 949 Ġcleaned Ġpathology Ġreports . ĠEach Ġreport Ġis Ġassociated Ġwith Ġone Ġof Ġthe Ġ37 Ġdifferent Ġprimary Ġdiagnoses Ġbased Ġon ĠID C - O Ġcodes . ĠThe Ġreports Ġare Ġcollected Ġfrom Ġfour Ġdifferent Ġbody Ġparts Ġor Ġprimary Ġsites Ġfrom Ġmultiple Ġpatients . ĠThe Ġdistribution Ġof Ġreports Ġacross Ġdifferent Ġprimary Ġdiagnoses Ġand Ġprimary Ġsites Ġis Ġreported Ġin Ġtab : report - dist ribution . ĠThe Ġdataset Ġwas Ġdeveloped Ġin Ġthree Ġsteps Ġas Ġfollows . </s> Collect ing Ġpathology Ġreports : ĠThe Ġtotal Ġof Ġ11 , 112 Ġpathology Ġreports Ġwere Ġdownloaded Ġfrom ĠNC I 's ĠGen omic ĠData ĠCommons Ġ( G DC ) Ġdataset Ġin ĠPDF Ġformat ĠB IB REF 9 Ġ. ĠOut Ġof Ġall ĠPDF Ġfiles , Ġ1 , 949 Ġreports Ġwere Ġselected Ġacross Ġmultiple Ġpatients Ġfrom Ġfour Ġspecific Ġprimary Ġsites âĢĶ th ym us , Ġtest is , Ġlung , Ġand Ġkidney . ĠThe Ġselection Ġwas Ġprimarily Ġmade Ġbased Ġon Ġthe Ġquality Ġof ĠPDF Ġfiles . </s> C leaning Ġreports : ĠThe Ġnext Ġstep Ġwas Ġto Ġextract Ġthe Ġtext Ġcontent Ġfrom Ġthese Ġreports . ĠDue Ġto Ġthe Ġsignificant Ġtime Ġexpense Ġof Ġmanually Ġre - ty ping Ġall Ġthe Ġpathology Ġreports , Ġwe Ġdeveloped Ġa Ġnew Ġstrategy Ġto Ġprepare Ġour Ġdataset . ĠWe Ġapplied Ġan ĠOptical ĠCharacter ĠRecogn ition Ġ( OC R ) Ġsoftware Ġto Ġconvert Ġthe ĠPDF Ġreports Ġto Ġtext Ġfiles . ĠThen , Ġwe Ġmanually Ġinspected Ġall Ġgenerated Ġtext Ġfiles Ġto Ġfix Ġany Ġgrammar / spe lling Ġissues Ġand Ġirrelevant Ġcharacters Ġas Ġan Ġarte fact Ġproduced Ġby Ġthe ĠO CR Ġsystem . </s> Spl itting Ġinto Ġtraining - testing Ġdata : ĠWe Ġsplit Ġthe Ġcleaned Ġreports Ġinto Ġ70 % Ġand Ġ30 % Ġfor Ġtraining Ġand Ġtesting , Ġrespectively . ĠThis Ġsplit Ġresulted Ġin Ġ1 , 364 Ġtraining , Ġand Ġ5 85 Ġtesting Ġreports . </s> Pre - Process ing Ġof ĠReports </s> We Ġpre - process ed Ġthe Ġreports Ġby Ġsetting Ġtheir Ġtext Ġcontent Ġto Ġlower case Ġand Ġfiltering Ġout Ġany Ġnon - alph an umeric Ġcharacters . ĠWe Ġused ĠNL T K Ġlibrary Ġto Ġremove Ġstopping Ġwords , Ġe . g ., Ġ` the ', Ġ` an ', Ġ` was ', Ġ` if ' Ġand Ġso Ġon ĠB IB REF 10 Ġ. ĠWe Ġthen Ġanalyzed Ġthe Ġreports Ġto Ġfind Ġcommon Ġbig rams , Ġsuch Ġas ĠâĢ ľ l ung Ġp aren chy ma âĢ Ŀ , ĠâĢ ľ micro sc opic Ġexamination âĢ Ŀ , ĠâĢ ľ ly mph Ġnode âĢ Ŀ Ġetc . ĠWe Ġjoined Ġthe Ġbi agram s Ġwith Ġa Ġhyp hen , Ġconverting Ġthem Ġinto Ġa Ġsingle Ġword . ĠWe Ġfurther Ġremoved Ġthe Ġwords Ġthat Ġoccur Ġless Ġthan Ġ2 % Ġin Ġeach Ġof Ġthe Ġdiagnostic Ġcategory . ĠAs Ġwell , Ġwe Ġremoved Ġthe Ġwords Ġthat Ġoccur Ġmore Ġthan Ġ90 % Ġacross Ġall Ġthe Ġcategories . ĠWe Ġstored Ġeach Ġpre - process ed Ġreport Ġin Ġa Ġseparate Ġtext Ġfile . </s> TF - ID F Ġfeatures </s> TF - ID F Ġstands Ġfor ĠTerm ĠFrequency - In verse ĠDocument ĠFrequency , Ġand Ġit Ġis Ġa Ġuseful Ġweight ing Ġscheme Ġin Ġinformation Ġretrieval Ġand Ġtext Ġmining . ĠTF - ID F Ġsignifies Ġthe Ġimportance Ġof Ġa Ġterm Ġin Ġa Ġdocument Ġwithin Ġa Ġcorpus . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġa Ġdocument Ġhere Ġrefers Ġto Ġa Ġpathology Ġreport , Ġa Ġcorpus Ġrefers Ġto Ġthe Ġcollection Ġof Ġreports , Ġand Ġa Ġterm Ġrefers Ġto Ġa Ġsingle Ġword Ġin Ġa Ġreport . ĠThe ĠTF - ID F Ġweight Ġfor Ġa Ġterm ĠIN LINE FORM 0 Ġin Ġa Ġdocument ĠIN LINE FORM 1 Ġis Ġgiven Ġby ĠDIS PLAY FORM 0 </s> We Ġperformed Ġthe Ġfollowing Ġsteps Ġto Ġtransform Ġa Ġpathology Ġreport Ġinto Ġa Ġfeature Ġvector : </s> Create Ġa Ġset Ġof Ġvocabulary Ġcontaining Ġall Ġunique Ġwords Ġfrom Ġall Ġthe Ġpre - process ed Ġtraining Ġreports . </s> Create Ġa Ġzero Ġvector ĠIN LINE FORM 0 Ġof Ġthe Ġsame Ġlength Ġas Ġthe Ġvocabulary . </s> For Ġeach Ġword ĠIN LINE FORM 0 Ġin Ġa Ġreport ĠIN LINE FORM 1 Ġ, Ġset Ġthe Ġcorresponding Ġindex Ġin ĠIN LINE FORM 2 Ġto ĠIN LINE FORM 3 Ġ. </s> The Ġresultant ĠIN LINE FORM 0 Ġis Ġa Ġfeature Ġvector Ġfor Ġthe Ġreport ĠIN LINE FORM 1 Ġand Ġit Ġis Ġa Ġhighly Ġsparse Ġvector . </s> Key word Ġextraction Ġand Ġtopic Ġmodelling </s> The Ġkeyword Ġextraction Ġinvolves Ġidentifying Ġimportant Ġwords Ġwithin Ġreports Ġthat Ġsummarizes Ġits Ġcontent . ĠIn Ġcontrast , Ġthe Ġtopic Ġmodelling Ġallows Ġgrouping Ġthese Ġkeywords Ġusing Ġan Ġintelligent Ġscheme , Ġenabling Ġusers Ġto Ġfurther Ġfocus Ġon Ġcertain Ġaspects Ġof Ġa Ġdocument . ĠAll Ġthe Ġwords Ġin Ġa Ġpathology Ġreport Ġare Ġsorted Ġaccording Ġto Ġtheir ĠTF - ID F Ġweights . ĠThe Ġtop ĠIN LINE FORM 0 Ġsorted Ġwords Ġconstitute Ġthe Ġtop ĠIN LINE FORM 1 Ġkeywords Ġfor Ġthe Ġreport . ĠThe ĠIN LINE FORM 2 Ġis Ġempir ically Ġset Ġto Ġ50 Ġwithin Ġthis Ġresearch . ĠThe Ġextracted Ġkeywords Ġare Ġfurther Ġgrouped Ġinto Ġdifferent Ġtopics Ġby Ġusing Ġlatent ĠDir ich let Ġallocation Ġ( L DA ) ĠB IB REF 11 Ġ. ĠThe Ġkeywords Ġin Ġa Ġreport Ġare Ġhighlighted Ġusing Ġthe Ġcolor Ġtheme Ġbased Ġon Ġtheir Ġtopics . </s> E val uation Ġmetrics </s> Each Ġmodel Ġis Ġevaluated Ġusing Ġtwo Ġstandard ĠN LP Ġmetrics âĢĶ micro Ġand Ġmacro Ġaveraged ĠF - sc ores , Ġthe Ġharmonic Ġmean Ġof Ġrelated Ġmetrics Ġprecision Ġand Ġrecall . ĠFor Ġeach Ġdiagnostic Ġcategory ĠIN LINE FORM 0 Ġfrom Ġa Ġset Ġof Ġ37 Ġdifferent Ġclasses ĠIN LINE FORM 1 Ġ, Ġthe Ġnumber Ġof Ġtrue Ġpositives ĠIN LINE FORM 2 Ġ, Ġfalse Ġpositives ĠIN LINE FORM 3 Ġ, Ġand Ġfalse Ġnegatives ĠIN LINE FORM 4 Ġ, Ġthe Ġmicro ĠF - score Ġis Ġdefined Ġas ĠDIS PLAY FORM 0 </s> where as Ġmacro ĠF - score Ġis Ġgiven Ġby ĠDIS PLAY FORM 0 </s> In Ġsummary , Ġmicro - aver aged Ġmetrics Ġhave Ġclass Ġrepresentation Ġroughly Ġproportional Ġto Ġtheir Ġtest Ġset Ġrepresentation Ġ( same Ġas Ġaccuracy Ġfor Ġclassification Ġproblem Ġwith Ġa Ġsingle Ġlabel Ġper Ġdata Ġpoint ), Ġwhereas Ġmacro - aver aged Ġmetrics Ġare Ġaveraged Ġby Ġclass Ġwithout Ġweight ing Ġby Ġclass Ġprevalence ĠB IB REF 12 Ġ. </s> Exper imental Ġsetting </s> In Ġthis Ġstudy , Ġwe Ġperformed Ġtwo Ġdifferent Ġseries Ġof Ġexperiments : Ġi ) Ġevaluating Ġthe Ġperformance Ġof ĠTF - ID F Ġfeatures Ġand Ġvarious Ġmachine Ġlearning Ġclass ifiers Ġon Ġthe Ġtask Ġof Ġpredicting Ġprimary Ġdiagnosis Ġfrom Ġthe Ġtext Ġcontent Ġof Ġa Ġgiven Ġreport , Ġand Ġii ) Ġusing ĠTF - ID F Ġand ĠL DA Ġtechniques Ġto Ġhighlight Ġthe Ġimportant Ġkeywords Ġwithin Ġa Ġreport . ĠFor Ġthe Ġfirst Ġexperiment Ġseries , Ġtraining Ġreports Ġare Ġpre - process ed , Ġthen Ġtheir ĠTF - ID F Ġfeatures Ġare Ġextracted . ĠThe ĠTF - ID F Ġfeatures Ġand Ġthe Ġtraining Ġlabels Ġare Ġused Ġto Ġtrain Ġdifferent Ġclassification Ġmodels . ĠThese Ġdifferent Ġclassification Ġmodels Ġand Ġtheir Ġhyper - param eters Ġare Ġreported Ġin Ġtab : class ifier . ĠThe Ġperformance Ġof Ġclass ifiers Ġis Ġmeasured Ġquant itatively Ġon Ġthe Ġtest Ġdataset Ġusing Ġthe Ġevaluation Ġmetrics Ġdiscussed Ġin Ġthe Ġprevious Ġsection . ĠFor Ġthe Ġsecond Ġexperiment Ġseries , Ġa Ġrandom Ġreport Ġis Ġselected Ġand Ġits Ġtop Ġ50 Ġkeywords Ġare Ġextracted Ġusing ĠTF - ID F Ġweights . ĠThese Ġ50 Ġkeywords Ġare Ġhighlighted Ġusing Ġdifferent Ġcolors Ġbased Ġon Ġtheir Ġassociated Ġtopic , Ġwhich Ġare Ġextracted Ġthrough ĠL DA . ĠA Ġnon - ex pert Ġbased Ġqualitative Ġinspection Ġis Ġperformed Ġon Ġthe Ġextracted Ġkeywords Ġand Ġtheir Ġcorresponding Ġtopics . </s> Exper iment ĠSeries Ġ1 </s> A Ġclassification Ġmodel Ġis Ġtrained Ġto Ġpredict Ġthe Ġprimary Ġdiagnosis Ġgiven Ġthe Ġcontent Ġof Ġthe Ġcancer Ġpathology Ġreport . ĠThe Ġperformance Ġresults Ġon Ġthis Ġtask Ġare Ġreported Ġin Ġtab : results . ĠWe Ġcan Ġobserve Ġthat Ġthe ĠX GB o ost Ġclass ifier Ġoutper formed Ġall Ġother Ġmodels Ġfor Ġboth Ġthe Ġmicro ĠF - score Ġmetric , Ġwith Ġa Ġscore Ġof Ġ0 . 92 , Ġand Ġthe Ġmacro ĠF - score Ġmetric , Ġwith Ġa Ġscore Ġof Ġ0 . 31 . ĠThis Ġwas Ġan Ġimprovement Ġof Ġ7 % Ġfor Ġthe Ġmicro ĠF - score Ġover Ġthe Ġnext Ġbest Ġmodel , ĠS VM - L , Ġand Ġa Ġmarginal Ġimprovement Ġof Ġ5 % Ġfor Ġmacro ĠF - score . ĠIt Ġis Ġinteresting Ġto Ġnote Ġthat ĠS VM Ġwith Ġlinear Ġkernels Ġperforms Ġmuch Ġbetter Ġthan ĠS VM Ġwith ĠRB F Ġkernel , Ġscoring Ġ9 % Ġon Ġthe Ġmacro ĠF - score Ġand Ġ12 % Ġmore Ġon Ġthe Ġmicro ĠF - score . ĠIt Ġis Ġsuspected Ġthat Ġsince Ġwords Ġused Ġin Ġprimary Ġdiagnosis Ġitself Ġoccur Ġin Ġsome Ġreports , Ġthus Ġenabling Ġthe Ġlinear Ġmodels Ġto Ġoutper form Ġcomplex Ġmodels . </s> Exper iment ĠSeries Ġ2 </s> fig : key words Ġshows Ġthe Ġtop Ġ50 Ġkeywords Ġhighlighted Ġusing ĠTF - ID F Ġand ĠL DA . ĠThe Ġproposed Ġapproach Ġhas Ġperformed Ġwell Ġin Ġhighlighting Ġthe Ġimportant Ġregions , Ġfor Ġexample Ġthe Ġtopic Ġhighlighted Ġwith Ġa Ġred Ġcolor Ġcontaining ĠâĢ ľ pres ence Ġrange Ġtumor Ġnec rosis âĢ Ŀ Ġprovides Ġuseful Ġbiomark er Ġinformation Ġto Ġreaders . </s> Con clusions </s> We Ġproposed Ġa Ġsimple Ġyet Ġefficient ĠTF - ID F Ġmethod Ġto Ġextract Ġand Ġcorrobor ate Ġuseful Ġkeywords Ġfrom Ġpathology Ġcancer Ġreports . ĠEnc oding Ġa Ġpathology Ġreport Ġfor Ġcancer Ġand Ġtumor Ġsurveillance Ġis Ġa Ġlabor ious Ġtask , Ġand Ġsometimes Ġit Ġis Ġsubjected Ġto Ġhuman Ġerrors Ġand Ġvariability Ġin Ġthe Ġinterpretation . ĠOne Ġof Ġthe Ġmost Ġimportant Ġaspects Ġof Ġencoding Ġa Ġpathology Ġreport Ġinvolves Ġextracting Ġthe Ġprimary Ġdiagnosis . ĠThis Ġmay Ġbe Ġvery Ġuseful Ġfor Ġcontent - based Ġimage Ġretrieval Ġto Ġcombine Ġwith Ġvisual Ġinformation . ĠWe Ġused Ġexisting Ġclassification Ġmodel Ġand ĠTF - ID F Ġfeatures Ġto Ġpredict Ġthe Ġprimary Ġdiagnosis . ĠWe Ġachieved Ġup Ġto Ġ92 % Ġaccuracy Ġusing ĠX GB o ost Ġclass ifier . ĠThe Ġprediction Ġaccuracy Ġem powers Ġthe Ġadoption Ġof Ġmachine Ġlearning Ġmethods Ġfor Ġautomated Ġinformation Ġextraction Ġfrom Ġpathology Ġreports .