57,505 sentences
ABSTRACTIVE
57,505 sentences
ABSTRACTIVE
Our contributions in this work are summarized as follows. We define a task difficulty prediction task and show how this is related to, but distinct from, inter-worker agreement. We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task. We show that predicting annotation difficulty can be used to improve the task routing and model performance for a biomedical information extraction task. Our results open up a new direction for ensuring corpus quality. We believe that item difficulty prediction will likely be useful in other, non-specialized tasks as well, and that the most effective data collection in specialized domains requires research addressing the fundamental questions we examine here. An abstract may contain some `easy' and some `difficult' sentences. We thus perform our analysis at the sentence level. We split abstracts into sentences using spaCy. We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting. In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively.
An abstract may contain some `easy' and some `difficult' sentences. We thus perform our analysis at the sentence level. We split abstracts into sentences using spaCy. We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting. In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively.
1905.07791
Introduction </s> As semb ling Ġtraining Ġcorpor a Ġof Ġannot ated Ġnatural Ġlanguage Ġexamples Ġin Ġspecialized Ġdomains Ġsuch Ġas Ġbi omed ic ine Ġposes Ġconsiderable Ġchallenges . ĠExperts Ġwith Ġthe Ġrequisite Ġdomain Ġknowledge Ġto Ġperform Ġhigh - quality Ġannotation Ġtend Ġto Ġbe Ġexpensive , Ġwhile Ġlay Ġannot ators Ġmay Ġnot Ġhave Ġthe Ġnecessary Ġknowledge Ġto Ġprovide Ġhigh - quality Ġannotations . ĠA Ġpractical Ġapproach Ġfor Ġcollecting Ġa Ġsufficiently Ġlarge Ġcorpus Ġwould Ġbe Ġto Ġuse Ġcrowds ourcing Ġplatforms Ġlike ĠAmazon ĠMechanical ĠTurk Ġ( MT ur k ). ĠHowever , Ġcrowd Ġworkers Ġin Ġgeneral Ġare Ġlikely Ġto Ġprovide Ġnoisy Ġannotations ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġan Ġissue Ġexacerbated Ġby Ġthe Ġtechnical Ġnature Ġof Ġspecialized Ġcontent . ĠSome Ġof Ġthis Ġnoise Ġmay Ġreflect Ġworker Ġquality Ġand Ġcan Ġbe Ġmodeled ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġbut Ġfor Ġsome Ġinstances Ġlay Ġpeople Ġmay Ġsimply Ġlack Ġthe Ġdomain Ġknowledge Ġto Ġprovide Ġuseful Ġannotation . </s> In Ġthis Ġpaper Ġwe Ġreport Ġexperiments Ġon Ġthe ĠE BM - N LP Ġcorpus Ġcomprising Ġcrowds ourced Ġannotations Ġof Ġmedical Ġliterature ĠB IB REF 5 Ġ. ĠWe Ġoperational ize Ġthe Ġconcept Ġof Ġannotation Ġdifficulty Ġand Ġshow Ġhow Ġit Ġcan Ġbe Ġexploited Ġduring Ġtraining Ġto Ġimprove Ġinformation Ġextraction Ġmodels . ĠWe Ġthen Ġobtain Ġexpert Ġannotations Ġfor Ġthe Ġabstract s Ġpredicted Ġto Ġbe Ġmost Ġdifficult , Ġas Ġwell Ġas Ġfor Ġa Ġsimilar Ġnumber Ġof Ġrandomly Ġselected Ġabstract s . ĠThe Ġannotation Ġof Ġhighly Ġspecialized Ġdata Ġand Ġthe Ġuse Ġof Ġlay Ġand Ġexpert Ġannot ators Ġallow Ġus Ġto Ġexamine Ġthe Ġfollowing Ġkey Ġquestions Ġrelated Ġto Ġlay Ġand Ġexpert Ġannotations Ġin Ġspecialized Ġdomains : </s> Can Ġwe Ġpredict Ġitem Ġdifficulty ? ĠWe Ġdefine Ġa Ġtraining Ġinstance Ġas Ġdifficult Ġif Ġa Ġlay Ġannot ator Ġor Ġan Ġautomated Ġmodel Ġdisagree Ġon Ġits Ġlabeling . ĠWe Ġshow Ġthat Ġdifficulty Ġcan Ġbe Ġpredicted , Ġand Ġthat Ġit Ġis Ġdistinct Ġfrom Ġinter - annot ator Ġagreement . ĠFurther , Ġsuch Ġpredictions Ġcan Ġbe Ġused Ġduring Ġtraining Ġto Ġimprove Ġinformation Ġextraction Ġmodels . </s> Are Ġthere Ġsystematic Ġdifferences Ġbetween Ġexpert Ġand Ġlay Ġannotations ? ĠWe Ġobserve Ġdecidedly Ġlower Ġagreement Ġbetween Ġlay Ġworkers Ġas Ġcompared Ġto Ġdomain Ġexperts . ĠLay Ġannotations Ġhave Ġhigh Ġprecision Ġbut Ġlow Ġrecall Ġwith Ġrespect Ġto Ġexpert Ġannotations Ġin Ġthe Ġnew Ġdata Ġthat Ġwe Ġcollected . ĠMore Ġgenerally , Ġwe Ġexpect Ġlay Ġannotations Ġto Ġbe Ġlower Ġquality , Ġwhich Ġmay Ġtranslate Ġto Ġlower Ġprecision , Ġrecall , Ġor Ġboth , Ġcompared Ġto Ġexpert Ġannotations . ĠCan Ġone Ġrely Ġsolely Ġon Ġlay Ġannotations ? ĠRe asonable Ġmodels Ġcan Ġbe Ġtrained Ġusing Ġlay Ġannotations Ġalone , Ġbut Ġsimilar Ġperformance Ġcan Ġbe Ġachieved Ġusing Ġmarkedly Ġless Ġexpert Ġdata . ĠThis Ġsuggests Ġthat Ġthe Ġoptimal Ġratio Ġof Ġexpert Ġto Ġcrowd Ġannotations Ġfor Ġspecialized Ġtasks Ġwill Ġdepend Ġon Ġthe Ġcost Ġand Ġavailability Ġof Ġdomain Ġexperts . ĠExpert Ġannotations Ġare Ġpreferable Ġwhenever Ġits Ġcollection Ġis Ġpractical . ĠBut Ġin Ġreal - world Ġsettings , Ġa Ġcombination Ġof Ġexpert Ġand Ġlay Ġannotations Ġis Ġbetter Ġthan Ġusing Ġlay Ġdata Ġalone . </s> Does Ġit Ġmatter Ġwhat Ġdata Ġis Ġannot ated Ġby Ġexperts ? ĠWe Ġdemonstrate Ġthat Ġa Ġsystem Ġtrained Ġon Ġcombined Ġdata Ġachieves Ġbetter Ġpredictive Ġperformance Ġwhen Ġexperts Ġannot ate Ġdifficult Ġexamples Ġrather Ġthan Ġinstances Ġselected Ġat Ġi . i . d . Ġrandom . </s> Our Ġcontributions Ġin Ġthis Ġwork Ġare Ġsummarized Ġas Ġfollows . ĠWe Ġdefine Ġa Ġtask Ġdifficulty Ġprediction Ġtask Ġand Ġshow Ġhow Ġthis Ġis Ġrelated Ġto , Ġbut Ġdistinct Ġfrom , Ġinter - worker Ġagreement . ĠWe Ġintroduce Ġa Ġnew Ġmodel Ġfor Ġdifficulty Ġprediction Ġcombining Ġlearned Ġrepresentations Ġinduced Ġvia Ġa Ġpre - trained Ġ` universal ' Ġsentence Ġenc oder ĠB IB REF 6 Ġ, Ġand Ġa Ġsentence Ġenc oder Ġlearned Ġfrom Ġscratch Ġfor Ġthis Ġtask . ĠWe Ġshow Ġthat Ġpredicting Ġannotation Ġdifficulty Ġcan Ġbe Ġused Ġto Ġimprove Ġthe Ġtask Ġrouting Ġand Ġmodel Ġperformance Ġfor Ġa Ġbiomedical Ġinformation Ġextraction Ġtask . ĠOur Ġresults Ġopen Ġup Ġa Ġnew Ġdirection Ġfor Ġensuring Ġcorpus Ġquality . ĠWe Ġbelieve Ġthat Ġitem Ġdifficulty Ġprediction Ġwill Ġlikely Ġbe Ġuseful Ġin Ġother , Ġnon - special ized Ġtasks Ġas Ġwell , Ġand Ġthat Ġthe Ġmost Ġeffective Ġdata Ġcollection Ġin Ġspecialized Ġdomains Ġrequires Ġresearch Ġaddressing Ġthe Ġfundamental Ġquestions Ġwe Ġexamine Ġhere . </s> Related ĠWork </s> C rowd s ourcing Ġannotation Ġis Ġnow Ġa Ġwell - stud ied Ġproblem ĠB IB REF 7 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠDue Ġto Ġthe Ġnoise Ġinherent Ġin Ġsuch Ġannotations , Ġthere Ġhave Ġalso Ġbeen Ġconsiderable Ġefforts Ġto Ġdevelop Ġaggregation Ġmodels Ġthat Ġminimize Ġnoise ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. </s> There Ġare Ġalso Ġseveral Ġsurveys Ġof Ġcrowds ourcing Ġin Ġbi omed ic ine Ġspecifically ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠSome Ġwork Ġin Ġthis Ġspace Ġhas Ġcontrasted Ġmodel Ġperformance Ġachieved Ġusing Ġexpert Ġvs . Ġcrowd Ġannot ated Ġtraining Ġdata ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠDum it r ache Ġet Ġal . ĠDum it r ache : 2018 : C GT : 323 27 18 . 315 28 89 Ġconcluded Ġthat Ġperformance Ġis Ġsimilar Ġunder Ġthese Ġsupervision Ġtypes , Ġfinding Ġno Ġclear Ġadvantage Ġfrom Ġusing Ġexpert Ġannot ators . ĠThis Ġdiffers Ġfrom Ġour Ġfindings , Ġperhaps Ġowing Ġto Ġdifferences Ġin Ġdesign . ĠThe Ġexperts Ġwe Ġused Ġalready Ġhold Ġadvanced Ġmedical Ġdegrees , Ġfor Ġinstance , Ġwhile Ġthose Ġin Ġprior Ġwork Ġwere Ġmedical Ġstudents . ĠFurthermore , Ġthe Ġtask Ġconsidered Ġhere Ġwould Ġappear Ġto Ġbe Ġof Ġgreater Ġdifficulty : Ġeven Ġa Ġsystem Ġtrained Ġon Ġ$\ sim Ġ$ Ġ5 k Ġinstances Ġperforms Ġreasonably , Ġbut Ġfar Ġfrom Ġperfect . ĠBy Ġcontrast , Ġin Ġsome Ġof Ġthe Ġprior Ġwork Ġwhere Ġexperts Ġand Ġcrowd Ġannotations Ġwere Ġdeemed Ġequivalent , Ġa Ġclass ifier Ġtrained Ġon Ġ300 Ġexamples Ġcan Ġachieve Ġvery Ġhigh Ġaccuracy ĠB IB REF 12 Ġ. </s> More Ġrelevant Ġto Ġthis Ġpaper , Ġprior Ġwork Ġhas Ġinvestigated Ġmethods Ġfor Ġ` task Ġrouting ' Ġin Ġactive Ġlearning Ġscenarios Ġin Ġwhich Ġsupervision Ġis Ġprovided Ġby Ġheter ogeneous Ġlabel ers Ġwith Ġvarying Ġlevels Ġof Ġexpertise ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 14 Ġ. ĠThe Ġrelated Ġquestion Ġof Ġwhether Ġeffort Ġis Ġbetter Ġspent Ġcollecting Ġadditional Ġannotations Ġfor Ġalready Ġlabeled Ġ( but Ġpotentially Ġno is ily Ġso ) Ġexamples Ġor Ġnovel Ġinstances Ġhas Ġalso Ġbeen Ġaddressed ĠB IB REF 18 Ġ. ĠWhat Ġdistinguishes Ġthe Ġwork Ġhere Ġis Ġour Ġfocus Ġon Ġproviding Ġan Ġoperational Ġdefinition Ġof Ġinstance Ġdifficulty , Ġshowing Ġthat Ġthis Ġcan Ġbe Ġpredicted , Ġand Ġthen Ġusing Ġthis Ġto Ġinform Ġtask Ġrouting . </s> Application ĠDomain </s> Our Ġspecific Ġapplication Ġconcerns Ġannot ating Ġabstract s Ġof Ġarticles Ġthat Ġdescribe Ġthe Ġconduct Ġand Ġresults Ġof Ġrandomized Ġcontrolled Ġtrials Ġ( R CT s ). ĠExperiment ation Ġin Ġthis Ġdomain Ġhas Ġbecome Ġeasy Ġwith Ġthe Ġrecent Ġrelease Ġof Ġthe ĠE BM - N LP ĠB IB REF 5 Ġcorpus , Ġwhich Ġincludes Ġa Ġreasonably Ġlarge Ġtraining Ġdataset Ġannot ated Ġvia Ġcrowds ourcing , Ġand Ġa Ġmodest Ġtest Ġset Ġlabeled Ġby Ġindividuals Ġwith Ġadvanced Ġmedical Ġtraining . ĠMore Ġspecifically , Ġthe Ġtraining Ġset Ġcomprises Ġ4 , 7 41 Ġmedical Ġarticle Ġabstract s Ġwith Ġcrowds ourced Ġannotations Ġindicating Ġsnippets Ġ( sequ ences ) Ġthat Ġdescribe Ġthe ĠParticipants Ġ( p ), ĠInter ventions Ġ( i ), Ġand ĠOut come Ġ( o ) Ġelements Ġof Ġthe Ġrespective ĠR CT , Ġand Ġthe Ġtest Ġset Ġis Ġcomposed Ġof Ġ191 Ġabstract s Ġwith Ġp , Ġi , Ġo Ġsequence Ġannotations Ġfrom Ġthree Ġmedical Ġexperts . </s> Table Ġ1 Ġshows Ġan Ġexample Ġof Ġdifficult Ġand Ġeasy Ġexamples Ġaccording Ġto Ġour Ġdefinition Ġof Ġdifficulty . ĠThe Ġunder lined Ġtext Ġdem arc ates Ġthe Ġ( cons ensus ) Ġreference Ġlabel Ġprovided Ġby Ġdomain Ġexperts . ĠIn Ġthe Ġdifficult Ġexamples , Ġcrowd Ġworkers Ġmarked Ġtext Ġdistinct Ġfrom Ġthese Ġreference Ġannotations ; Ġwhereas Ġin Ġthe Ġeasy Ġcases Ġthey Ġreproduced Ġthem Ġwith Ġreasonable Ġfidelity . ĠThe Ġdifficult Ġsentences Ġusually Ġexhibit Ġcomplicated Ġstructure Ġand Ġfeature Ġjargon . </s> An Ġabstract Ġmay Ġcontain Ġsome Ġ` easy ' Ġand Ġsome Ġ` diff icult ' Ġsentences . ĠWe Ġthus Ġperform Ġour Ġanalysis Ġat Ġthe Ġsentence Ġlevel . ĠWe Ġsplit Ġabstract s Ġinto Ġsentences Ġusing Ġspa Cy . ĠWe Ġexcluded Ġsentences Ġthat Ġcomprise Ġfewer Ġthan Ġtwo Ġtokens , Ġas Ġthese Ġare Ġlikely Ġan Ġartifact Ġof Ġerrors Ġin Ġsentence Ġsplitting . ĠIn Ġtotal , Ġthis Ġresulted Ġin Ġ57 , 505 Ġand Ġ2 , 428 Ġsentences Ġin Ġthe Ġtrain Ġand Ġtest Ġset Ġabstract s , Ġrespectively . </s> Quant ifying ĠTask ĠDifficulty </s> The Ġtest Ġset Ġincludes Ġannotations Ġfrom Ġboth Ġcrowd Ġworkers Ġand Ġdomain Ġexperts . ĠWe Ġtreat Ġthe Ġlatter Ġas Ġground Ġtruth Ġand Ġthen Ġdefine Ġthe Ġdifficulty Ġof Ġsentences Ġin Ġterms Ġof Ġthe Ġobserved Ġagreement Ġbetween Ġexpert Ġand Ġlay Ġannot ators . ĠForm ally , Ġfor Ġannotation Ġtask Ġ$ t $ Ġand Ġinstance Ġ$ i $ Ġ: </s> $$ \ text { Diff iculty } _{ ti } Ġ= Ġ\ frac {\ sum Ġ_ { j = 1 } ^ n { f (\ text { label } _{ ij }, Ġy _ i }) }{ n } $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> where Ġ$ f $ Ġis Ġa Ġscoring Ġfunction Ġthat Ġmeasures Ġthe Ġquality Ġof Ġthe Ġlabel Ġfrom Ġworker Ġ$ j $ Ġfor Ġsentence Ġ$ i $ Ġ, Ġas Ġcompared Ġto Ġa Ġground Ġtruth Ġannotation , Ġ$ y _ i $ Ġ. ĠThe Ġdifficulty Ġscore Ġof Ġsentence Ġ$ i $ Ġis Ġtaken Ġas Ġan Ġaverage Ġover Ġthe Ġscores Ġfor Ġall Ġ$ n $ Ġlay workers . ĠWe Ġuse ĠSpe arm ans ' Ġcorrelation Ġcoefficient Ġas Ġa Ġscoring Ġfunction . ĠSpecifically , Ġfor Ġeach Ġsentence Ġwe Ġcreate Ġtwo Ġvectors Ġcomprising Ġcounts Ġof Ġhow Ġmany Ġtimes Ġeach Ġtoken Ġwas Ġannot ated Ġby Ġcrowd Ġand Ġexpert Ġworkers , Ġrespectively , Ġand Ġcalculate Ġthe Ġcorrelation Ġbetween Ġthese . ĠSent ences Ġwith Ġno Ġlabels Ġare Ġtreated Ġas Ġmaxim ally Ġeasy ; Ġthose Ġwith Ġonly Ġeither Ġcrowd Ġworker Ġor Ġexpert Ġlabel ( s ) Ġare Ġassumed Ġmaxim ally Ġdifficult . </s> The Ġtraining Ġset Ġcontains Ġonly Ġcrowds ourced Ġannotations . ĠTo Ġlabel Ġthe Ġtraining Ġdata , Ġwe Ġuse Ġa Ġ10 - fold Ġvalidation Ġlike Ġsetting . ĠWe Ġiter atively Ġret rain Ġthe ĠL ST M - CR F - Pattern Ġsequence Ġtag ger Ġof ĠPatel Ġet Ġal . Ġp atel 2018 sy nt actic Ġon Ġ9 Ġfolds Ġof Ġthe Ġtraining Ġdata Ġand Ġuse Ġthat Ġtrained Ġmodel Ġto Ġpredict Ġlabels Ġfor Ġthe Ġ10 th . ĠIn Ġthis Ġway Ġwe Ġobtain Ġpredictions Ġon Ġthe Ġfull Ġtraining Ġset . ĠWe Ġthen Ġuse Ġpredicted Ġspans Ġas Ġproxy Ġ` ground Ġtruth ' Ġannotations Ġto Ġcalculate Ġthe Ġdifficulty Ġscore Ġof Ġsentences Ġas Ġdescribed Ġabove ; Ġwe Ġnormal ize Ġthese Ġto Ġthe Ġ[ Ġ$ 0 , Ġ1 $ Ġ] Ġinterval . ĠWe Ġvalidate Ġthis Ġapproximation Ġby Ġcomparing Ġthe Ġproxy Ġscores Ġagainst Ġreference Ġscores Ġover Ġthe Ġtest Ġset , Ġthe ĠPearson 's Ġcorrelation Ġcoefficients Ġare Ġ0 . 57 Ġfor ĠPopulation , Ġ0 . 71 Ġfor ĠIntervention Ġand Ġ0 . 68 Ġfor ĠOut come . </s> There Ġexist Ġmany Ġsentences Ġthat Ġcontain Ġneither Ġmanual Ġnor Ġpredicted Ġannotations . ĠWe Ġtreat Ġthese Ġas Ġmaxim ally Ġeasy Ġsentences Ġ( with Ġdifficulty Ġscores Ġof Ġ0 ). ĠSuch Ġsentences Ġcomprise Ġ51 %, Ġ42 % Ġand Ġ36 % Ġfor ĠPopulation , ĠInter ventions Ġand ĠOut comes Ġdata Ġrespectively , Ġindicating Ġthat Ġit Ġis Ġeasier Ġto Ġidentify Ġsentences Ġthat Ġhave Ġno ĠPopulation Ġspans , Ġbut Ġharder Ġto Ġidentify Ġsentences Ġthat Ġhave Ġno ĠInter ventions Ġor ĠOut comes Ġspans . ĠThis Ġis Ġintuitive Ġas Ġdescriptions Ġof Ġthe Ġlatter Ġtwo Ġtend Ġto Ġbe Ġmore Ġtechnical Ġand Ġdense Ġwith Ġmedical Ġjargon . </s> We Ġshow Ġthe Ġdistribution Ġof Ġthe Ġautomatically Ġlabeled Ġscores Ġfor Ġsentences Ġthat Ġdo Ġcontain Ġspans Ġin ĠFigure Ġ1 Ġ. ĠThe Ġmean Ġof Ġthe ĠPopulation Ġ( p ) Ġsentence Ġscores Ġis Ġsignificantly Ġlower Ġthan Ġthat Ġfor Ġother Ġtypes Ġof Ġsentences Ġ( i Ġand Ġo ), Ġagain Ġindicating Ġthat Ġthey Ġare Ġeasier Ġon Ġaverage Ġto Ġannot ate . ĠThis Ġalign s Ġwith Ġa Ġprevious Ġfinding Ġthat Ġannot ating ĠInter ventions Ġand ĠOut comes Ġis Ġmore Ġdifficult Ġthan Ġannot ating ĠParticipants ĠB IB REF 5 Ġ. </s> Many Ġsentences Ġcontain Ġspans Ġtagged Ġby Ġthe ĠL ST M - CR F - Pattern Ġmodel , Ġbut Ġmissed Ġby Ġall Ġcrowd Ġworkers , Ġresulting Ġin Ġa Ġmaxim ally Ġdifficult Ġscore Ġ( 1 ). ĠInspection Ġof Ġsuch Ġsentences Ġrevealed Ġthat Ġsome Ġare Ġtruly Ġdifficult Ġexamples , Ġbut Ġothers Ġare Ġtagging Ġmodel Ġerrors . ĠIn Ġeither Ġcase , Ġsuch Ġsentences Ġhave Ġconfused Ġworkers Ġand / or Ġthe Ġmodel , Ġand Ġso Ġwe Ġretain Ġthem Ġall Ġas Ġ` diff icult ' Ġsentences . </s> Content Ġdescribing Ġthe Ġp , Ġi Ġand Ġo , Ġrespectively , Ġis Ġquite Ġdifferent . ĠAs Ġsuch , Ġone Ġsentence Ġusually Ġcontains Ġ( at Ġmost ) Ġonly Ġone Ġof Ġthese Ġthree Ġcontent Ġtypes . ĠWe Ġthus Ġtreat Ġdifficulty Ġprediction Ġfor Ġthe Ġrespective Ġlabel Ġtypes Ġas Ġseparate Ġtasks . </s> Diff iculty Ġis Ġnot ĠWorker ĠAgreement </s> Our Ġdefinition Ġof Ġdifficulty Ġis Ġderived Ġfrom Ġagreement Ġbetween Ġexpert Ġand Ġcrowd Ġannotations Ġfor Ġthe Ġtest Ġdata , Ġand Ġagreement Ġbetween Ġa Ġpredictive Ġmodel Ġand Ġcrowd Ġannotations Ġin Ġthe Ġtraining Ġdata . ĠIt Ġis Ġreasonable Ġto Ġask Ġif Ġthese Ġmeasures Ġare Ġrelated Ġto Ġinter - annot ator Ġagreement , Ġa Ġmetric Ġoften Ġused Ġin Ġlanguage Ġtechnology Ġresearch Ġto Ġidentify Ġambiguous Ġor Ġdifficult Ġitems . ĠHere Ġwe Ġexplicitly Ġverify Ġthat Ġour Ġdefinition Ġof Ġdifficulty Ġonly Ġweak ly Ġcorrelates Ġwith Ġinter - annot ator Ġagreement . </s> We Ġcalculate Ġinter - worker Ġagreement Ġbetween Ġcrowd Ġand Ġexpert Ġannot ators Ġusing ĠSpear man 's Ġcorrelation Ġcoefficient . ĠAs Ġshown Ġin ĠTable Ġ2 Ġ, Ġaverage Ġagreement Ġbetween Ġdomain Ġexperts Ġare Ġconsiderably Ġhigher Ġthan Ġagreements Ġbetween Ġcrowd Ġworkers Ġfor Ġall Ġthree Ġlabel Ġtypes . ĠThis Ġis Ġa Ġclear Ġindication Ġthat Ġthe Ġcrowd Ġannotations Ġare Ġno is ier . </s> Furthermore , Ġwe Ġcompare Ġthe Ġcorrelation Ġbetween Ġinter - annot ator Ġagreement Ġand Ġdifficulty Ġscores Ġin Ġthe Ġtraining Ġdata . ĠGiven Ġthat Ġthe Ġmajority Ġof Ġsentences Ġdo Ġnot Ġcontain Ġa ĠP ICO Ġspan , Ġwe Ġonly Ġinclude Ġin Ġthese Ġcalculations Ġthose Ġthat Ġcontain Ġa Ġreference Ġlabel . ĠPearson 's Ġr Ġare Ġ0 . 34 , Ġ0 . 30 Ġand Ġ0 . 31 Ġfor Ġp , Ġi Ġand Ġo , Ġrespectively , Ġconfirming Ġthat Ġinter - worker Ġagreement Ġand Ġour Ġproposed Ġdifficulty Ġscore Ġare Ġquite Ġdistinct . </s> P redict ing ĠAn notation ĠDifficulty </s> We Ġtreat Ġdifficulty Ġprediction Ġas Ġa Ġregression Ġproblem , Ġand Ġpropose Ġand Ġevaluate Ġneural Ġmodel Ġvariants Ġfor Ġthe Ġtask . ĠWe Ġfirst Ġtrain ĠR NN ĠB IB REF 19 Ġand ĠCNN ĠB IB REF 20 Ġmodels . </s> We Ġalso Ġuse Ġthe Ġuniversal Ġsentence Ġenc oder Ġ( USE ) ĠB IB REF 6 Ġto Ġinduce Ġsentence Ġrepresentations , Ġand Ġtrain Ġa Ġmodel Ġusing Ġthese Ġas Ġfeatures . ĠFollowing ĠB IB REF 6 Ġ, Ġwe Ġthen Ġexperiment Ġwith Ġan Ġensemble Ġmodel Ġthat Ġcombines Ġthe Ġ` universal ' Ġand Ġtask - specific Ġrepresentations Ġto Ġpredict Ġannotation Ġdifficulty . ĠWe Ġexpect Ġthese Ġuniversal Ġembed d ings Ġto Ġcapture Ġgeneral , Ġhigh - level Ġsemantics , Ġand Ġthe Ġtask Ġspecific Ġrepresentations Ġto Ġcapture Ġmore Ġgran ular Ġinformation . ĠFigure Ġ2 Ġdepicts Ġthe Ġmodel Ġarchitecture . ĠSent ences Ġare Ġfed Ġinto Ġboth Ġthe Ġuniversal Ġsentence Ġenc oder Ġand , Ġseparately , Ġa Ġtask Ġspecific Ġneural Ġenc oder , Ġyielding Ġtwo Ġrepresentations . ĠWe Ġconc aten ate Ġthese Ġand Ġpass Ġthe Ġcombined Ġvector Ġto Ġthe Ġregression Ġlayer . </s> Exper imental ĠSetup Ġand ĠResults </s> We Ġtrained Ġmodels Ġfor Ġeach Ġlabel Ġtype Ġseparately . ĠWord Ġembed d ings Ġwere Ġinitialized Ġto Ġ300 d ĠGl o Ve Ġvectors ĠB IB REF 21 Ġtrained Ġon Ġcommon Ġcrawl Ġdata ; Ġthese Ġare Ġfine - tun ed Ġduring Ġtraining . ĠWe Ġused Ġthe ĠAdam Ġoptim izer ĠB IB REF 22 Ġwith Ġlearning Ġrate Ġand Ġdecay Ġset Ġto Ġ0 . 001 Ġand Ġ0 . 99 , Ġrespectively . ĠWe Ġused Ġbatch Ġsizes Ġof Ġ16 . </s> We Ġused Ġthe Ġlarge Ġversion Ġof Ġthe Ġuniversal Ġsentence Ġenc oder Ġwith Ġa Ġtransformer ĠB IB REF 23 Ġ. ĠWe Ġdid Ġnot Ġupdate Ġthe Ġpret rained Ġsentence Ġenc oder Ġparameters Ġduring Ġtraining . ĠAll Ġhyper param aters Ġfor Ġall Ġmodels Ġ( including Ġhidden Ġlayers , Ġhidden Ġsizes , Ġand Ġdrop out ) Ġwere Ġtuned Ġusing ĠViz ier ĠB IB REF 24 Ġvia Ġ10 - fold Ġcross Ġvalidation Ġon Ġthe Ġtraining Ġset Ġmaximizing Ġfor ĠF 1 . </s> As Ġa Ġbaseline , Ġwe Ġalso Ġtrained Ġa Ġlinear ĠSupport - Vector ĠReg ression ĠB IB REF 25 Ġmodel Ġon Ġ$ n $ Ġ- gram Ġfeatures Ġ( Ġ$ n $ Ġranges Ġfrom Ġ1 Ġto Ġ3 ). </s> Table Ġ3 Ġreports ĠPearson Ġcorrelation Ġcoefficients Ġbetween Ġthe Ġpredictions Ġwith Ġeach Ġof Ġthe Ġneural Ġmodels Ġand Ġthe Ġground Ġtruth Ġdifficulty Ġscores . ĠR ows Ġ1 - 4 Ġcorrespond Ġto Ġindividual Ġmodels , Ġand Ġrow Ġ5 Ġreports Ġthe Ġensemble Ġperformance . ĠColumn s Ġcorrespond Ġto Ġlabel Ġtype . ĠResults Ġfrom Ġall Ġmodels Ġoutper form Ġthe Ġbaseline ĠS VR Ġmodel : ĠPearson 's Ġcorrelation Ġcoefficients Ġrange Ġfrom Ġ0 . 550 Ġto Ġ0 . 6 22 . ĠThe Ġregression Ġcorrelations Ġare Ġthe Ġlowest . </s> The ĠR NN Ġmodel Ġrealizes Ġthe Ġstrongest Ġperformance Ġamong Ġthe Ġstand - alone Ġ( non - ense mble ) Ġmodels , Ġoutper forming Ġvariants Ġthat Ġexploit ĠCNN Ġand ĠUSE Ġrepresentations . ĠComb ining Ġthe ĠR NN Ġand ĠUSE Ġfurther Ġimproves Ġresults . ĠWe Ġhypothes ize Ġthat Ġthis Ġis Ġdue Ġto Ġcomplementary Ġsentence Ġinformation Ġencoded Ġin Ġuniversal Ġrepresentations . </s> For Ġall Ġmodels , Ġcorrelations Ġfor ĠIntervention Ġand ĠOut comes Ġare Ġhigher Ġthan Ġfor ĠPopulation , Ġwhich Ġis Ġexpected Ġgiven Ġthe Ġdifficulty Ġdistributions Ġin ĠFigure Ġ1 Ġ. ĠIn Ġthese , Ġthe Ġsentences Ġare Ġmore Ġuniformly Ġdistributed , Ġwith Ġa Ġfair Ġnumber Ġof Ġdifficult Ġand Ġeasier Ġsentences . ĠBy Ġcontrast , Ġin ĠPopulation Ġthere Ġare Ġa Ġgreater Ġnumber Ġof Ġeasy Ġsentences Ġand Ġconsiderably Ġfewer Ġdifficult Ġsentences , Ġwhich Ġmakes Ġthe Ġdifficulty Ġranking Ġtask Ġparticularly Ġchallenging . </s> Better ĠIE Ġwith ĠDifficulty ĠPrediction </s> We Ġnext Ġpresent Ġexperiments Ġin Ġwhich Ġwe Ġattempt Ġto Ġuse Ġthe Ġpredicted Ġdifficulty Ġduring Ġtraining Ġto Ġimprove Ġmodels Ġfor Ġinformation Ġextraction Ġof Ġdescriptions Ġof ĠPopulation , ĠInter ventions Ġand ĠOut comes Ġfrom Ġmedical Ġarticle Ġabstract s . ĠWe Ġinvestigate Ġtwo Ġuses : Ġ( 1 ) Ġsimply Ġremoving Ġthe Ġmost Ġdifficult Ġsentences Ġfrom Ġthe Ġtraining Ġset , Ġand , Ġ( 2 ) Ġre - weight ing Ġthe Ġmost Ġdifficult Ġsentences . </s> We Ġagain Ġuse ĠL ST M - CR F - Pattern Ġas Ġthe Ġbase Ġmodel Ġand Ġexperimenting Ġon Ġthe ĠE BM - N LP Ġcorpus ĠB IB REF 5 Ġ. ĠThis Ġis Ġtrained Ġon Ġeither Ġ( 1 ) Ġthe Ġtraining Ġset Ġwith Ġdifficult Ġsentences Ġremoved , Ġor Ġ( 2 ) Ġthe Ġfull Ġtraining Ġset Ġbut Ġwith Ġinstances Ġre - weight ed Ġin Ġproportion Ġto Ġtheir Ġpredicted Ġdifficulty Ġscore . ĠFollowing ĠB IB REF 5 Ġ, Ġwe Ġuse Ġthe ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġof Ġ0 . 001 , Ġdecay Ġ0 . 9 , Ġbatch Ġsize Ġ20 Ġand Ġdrop out Ġ0 . 5 . ĠWe Ġuse Ġpret rained Ġ200 d ĠGl o Ve Ġvectors ĠB IB REF 21 Ġto Ġinitialize Ġword Ġembed d ings , Ġand Ġuse Ġ100 d Ġhidden Ġchar Ġrepresentations . ĠEach Ġword Ġis Ġthus Ġrepresented Ġwith Ġ300 Ġdimensions Ġin Ġtotal . ĠThe Ġhidden Ġsize Ġis Ġ100 Ġfor Ġthe ĠL ST M Ġin Ġthe Ġcharacter Ġrepresentation Ġcomponent , Ġand Ġ200 Ġfor Ġthe ĠL ST M Ġin Ġthe Ġinformation Ġextraction Ġcomponent . ĠWe Ġtrain Ġfor Ġ15 Ġepoch s , Ġsaving Ġparameters Ġthat Ġachieve Ġthe Ġbest ĠF 1 Ġscore Ġon Ġa Ġnested Ġdevelopment Ġset . </s> Rem oving ĠDiff icult ĠExamples </s> We Ġfirst Ġevaluate Ġchanges Ġin Ġperformance Ġinduced Ġby Ġtraining Ġthe Ġsequence Ġlabeling Ġmodel Ġusing Ġless Ġdata Ġby Ġremoving Ġdifficult Ġsentences Ġprior Ġto Ġtraining . ĠThe Ġhypothesis Ġhere Ġis Ġthat Ġthese Ġdifficult Ġinstances Ġare Ġlikely Ġto Ġintroduce Ġmore Ġnoise Ġthan Ġsignal . ĠWe Ġused Ġa Ġcross - fold Ġapproach Ġto Ġpredict Ġsentence Ġdifficulties , Ġtraining Ġon Ġ9 / 10 ths Ġof Ġthe Ġdata Ġand Ġscoring Ġthe Ġremaining Ġ1 / 10 th Ġat Ġa Ġtime . ĠWe Ġthen Ġsorted Ġsentences Ġby Ġpredicted Ġdifficulty Ġscores , Ġand Ġexperimented Ġwith Ġremoving Ġincreasing Ġnumbers Ġof Ġthese Ġ( in Ġorder Ġof Ġdifficulty ) Ġprior Ġto Ġtraining Ġthe ĠL ST M - CR F - Pattern Ġmodel . </s> Figure Ġ3 Ġshows Ġthe Ġresults Ġachieved Ġby Ġthe ĠL ST M - CR F - Pattern Ġmodel Ġafter Ġdisc arding Ġincreasing Ġamounts Ġof Ġthe Ġtraining Ġdata : Ġthe Ġ$ x $ Ġand Ġ$ y $ Ġaxes Ġcorrespond Ġto Ġthe Ġthe Ġpercentage Ġof Ġdata Ġremoved Ġand ĠF 1 Ġscores , Ġrespectively . ĠWe Ġcontrast Ġremoving Ġsentences Ġpredicted Ġto Ġbe Ġdifficult Ġwith Ġremoving Ġthem Ġ( a ) Ġrandomly Ġ( i . i . d .), Ġand , Ġ( b ) Ġin Ġinverse Ġorder Ġof Ġpredicted Ġinter - annot ator Ġagreement . ĠThe Ġagreement Ġprediction Ġmodel Ġis Ġtrained Ġexactly Ġthe Ġsame Ġlike Ġdifficult Ġprediction Ġmodel , Ġwith Ġsimply Ġchanging Ġthe Ġdifficult Ġscore Ġto Ġannotation Ġagreement . ĠF 1 Ġscores Ġactually Ġimprove Ġ( marg inally ) Ġwhen Ġwe Ġremove Ġthe Ġmost Ġdifficult Ġsentences , Ġup Ġuntil Ġwe Ġdrop Ġ4 % Ġof Ġthe Ġdata Ġfor ĠPopulation Ġand ĠInter ventions , Ġand Ġ6 % Ġfor ĠOut comes . ĠRem oving Ġtraining Ġpoints Ġat Ġi . i . d . Ġrandom Ġde grades Ġperformance , Ġas Ġexpected . ĠRem oving Ġsentences Ġin Ġorder Ġof Ġdisagreement Ġseems Ġto Ġhave Ġsimilar Ġeffect Ġas Ġremoving Ġthem Ġby Ġdifficulty Ġscore Ġwhen Ġremoving Ġsmall Ġamount Ġof Ġthe Ġdata , Ġbut Ġthe ĠF 1 Ġscores Ġdrop Ġmuch Ġfaster Ġwhen Ġremoving Ġmore Ġdata . ĠThese Ġfindings Ġindicate Ġthat Ġsentences Ġpredicted Ġto Ġbe Ġdifficult Ġare Ġindeed Ġnoisy , Ġto Ġthe Ġextent Ġthat Ġthey Ġdo Ġnot Ġseem Ġto Ġprovide Ġthe Ġmodel Ġuseful Ġsignal . </s> Re - weight ing Ġby ĠDifficulty </s> We Ġshowed Ġabove Ġthat Ġremoving Ġa Ġsmall Ġnumber Ġof Ġthe Ġmost Ġdifficult Ġsentences Ġdoes Ġnot Ġharm , Ġand Ġin Ġfact Ġmodest ly Ġimproves , Ġmedical ĠIE Ġmodel Ġperformance . ĠHowever , Ġusing Ġthe Ġavailable Ġdata Ġwe Ġare Ġunable Ġto Ġtest Ġif Ġthis Ġwill Ġbe Ġuseful Ġin Ġpractice , Ġas Ġwe Ġwould Ġneed Ġadditional Ġdata Ġto Ġdetermine Ġhow Ġmany Ġdifficult Ġsentences Ġshould Ġbe Ġdropped . </s> We Ġinstead Ġexplore Ġan Ġalternative , Ġpractical Ġmeans Ġof Ġexploiting Ġdifficulty Ġpredictions : Ġwe Ġre - weight Ġsentences Ġduring Ġtraining Ġin versely Ġto Ġtheir Ġpredicted Ġdifficulty . ĠForm ally , Ġwe Ġweight Ġsentence Ġ$ i $ Ġwith Ġdifficulty Ġscores Ġabove Ġ$\ t au Ġ$ Ġaccording Ġto : Ġ$ 1 - a \ cd ot Ġ( d _ i - \ t au Ġ) /( 1 - \ t au Ġ) $ Ġ, Ġwhere Ġ$ d _ i $ Ġis Ġthe Ġdifficulty Ġscore Ġfor Ġsentence Ġ$ i $ Ġ, Ġand Ġ$ a $ Ġis Ġa Ġparameter Ġcod ifying Ġthe Ġminimum Ġweight Ġvalue . ĠWe Ġset Ġ$\ t au Ġ$ Ġto Ġ0 . 8 Ġso Ġas Ġto Ġonly Ġre - weight Ġsentences Ġwith Ġdifficulty Ġin Ġthe Ġtop Ġ20 th Ġpercentile , Ġand Ġwe Ġset Ġ$ a $ Ġto Ġ0 . 5 . ĠThe Ġre - weight ing Ġis Ġequivalent Ġto Ġdown - sam pling Ġthe Ġdifficult Ġsentences . ĠL ST M - CR F - Pattern Ġis Ġour Ġbase Ġmodel . </s> Table Ġ4 Ġreports Ġthe Ġprecision , Ġrecall Ġand ĠF 1 Ġachieved Ġboth Ġwith Ġand Ġwithout Ġsentence Ġre - weight ing . ĠRe - weight ing Ġimproves Ġall Ġmetrics Ġmodest ly Ġbut Ġconsistently . ĠAll ĠF 1 Ġdifferences Ġare Ġstatistically Ġsignificant Ġunder Ġa Ġsign Ġtest Ġ( Ġ$ p < 0 . 01 $ Ġ). ĠThe Ġmodel Ġwith Ġbest Ġprecision Ġis Ġdifferent Ġfor ĠPatient , ĠIntervention Ġand ĠOut come Ġlabels . ĠHowever Ġre - weight ing Ġby Ġdifficulty Ġdoes Ġconsistently Ġyield Ġthe Ġbest Ġrecall Ġfor Ġall Ġthree Ġextraction Ġtypes , Ġwith Ġthe Ġmost Ġnotable Ġimprovement Ġfor Ġi Ġand Ġo , Ġwhere Ġrecall Ġimproved Ġby Ġ10 Ġpercentage Ġpoints . ĠThis Ġperformance Ġincrease Ġtranslated Ġto Ġimprovements Ġin ĠF 1 Ġacross Ġall Ġtypes , Ġas Ġcompared Ġto Ġthe Ġbase Ġmodel Ġand Ġto Ġre - weight ing Ġby Ġagreement . </s> In vol ving ĠExpert ĠAn not ators </s> The Ġpreceding Ġexperiments Ġdemonstrate Ġthat Ġre - weight ing Ġdifficult Ġsentences Ġannot ated Ġby Ġthe Ġcrowd Ġgenerally Ġimproves Ġthe Ġextraction Ġmodels . ĠPresumably Ġthe Ġperformance Ġis Ġinfluenced Ġby Ġthe Ġannotation Ġquality . </s> We Ġnow Ġexamine Ġthe Ġpossibility Ġthat Ġthe Ġhigher Ġquality Ġand Ġmore Ġconsistent Ġannotations Ġof Ġdomain Ġexperts Ġon Ġthe Ġdifficult Ġinstances Ġwill Ġbenefit Ġthe Ġextraction Ġmodel . ĠThis Ġsim ulates Ġan Ġannotation Ġstrategy Ġin Ġwhich Ġwe Ġroute Ġdifficult Ġinstances Ġto Ġdomain Ġexperts Ġand Ġeasier Ġones Ġto Ġcrowd Ġannot ators . ĠWe Ġalso Ġcontrast Ġthe Ġvalue Ġof Ġdifficult Ġdata Ġto Ġthat Ġof Ġan Ġi . i . d . Ġrandom Ġsample Ġof Ġthe Ġsame Ġsize , Ġboth Ġannot ated Ġby Ġexperts . </s> Ex pert Ġannotations Ġof ĠRandom Ġand ĠDiff icult ĠInst ances </s> We Ġre - annot ate Ġby Ġexperts Ġa Ġsubset Ġof Ġmost Ġdifficult Ġinstances Ġand Ġthe Ġsame Ġnumber Ġof Ġrandom Ġinstances . ĠAs Ġcollecting Ġannotations Ġfrom Ġexperts Ġis Ġslow Ġand Ġexpensive , Ġwe Ġonly Ġre - annot ate Ġthe Ġdifficult Ġinstances Ġfor Ġthe Ġinterventions Ġextraction Ġtask . ĠWe Ġre - annot ate Ġthe Ġabstract s Ġwhich Ġcover Ġthe Ġsentences Ġwith Ġpredicted Ġdifficulty Ġscores Ġin Ġthe Ġtop Ġ5 Ġpercentile . ĠWe Ġrank Ġthe Ġabstract s Ġfrom Ġthe Ġtraining Ġset Ġby Ġthe Ġcount Ġof Ġdifficult Ġsentences , Ġand Ġre - annot ate Ġthe Ġabstract s Ġthat Ġcontain Ġthe Ġmost Ġdifficult Ġsentences . ĠCon str ained Ġby Ġtime Ġand Ġbudget , Ġwe Ġselect Ġonly Ġ2000 Ġabstract s Ġfor Ġre - ann otation ; Ġ1000 Ġof Ġthese Ġare Ġtop - ranked , Ġand Ġ1000 Ġare Ġrandomly Ġsampled . ĠThis Ġre - ann otation Ġcost Ġ$ 3 , 000 . ĠWe Ġhave Ġreleased Ġthe Ġnew Ġannotation Ġdata Ġat : Ġhttps :// github . com / b ep ny e / E BM - N LP . </s> Following ĠB IB REF 5 Ġ, Ġwe Ġrecruited Ġfive Ġmedical Ġexperts Ġvia ĠUp - work Ġwith Ġadvanced Ġmedical Ġtraining Ġand Ġstrong Ġtechnical Ġreading / writing Ġskills . ĠThe Ġexpert Ġannot ator Ġwere Ġasked Ġto Ġread Ġthe Ġentire Ġabstract Ġand Ġhighlight , Ġusing Ġthe ĠBR AT Ġtool kit ĠB IB REF 26 Ġ, Ġall Ġspans Ġdescribing Ġmedical ĠInter ventions . ĠEach Ġabstract Ġis Ġonly Ġannot ated Ġby Ġone Ġexpert . ĠWe Ġexamined Ġ30 Ġre - annot ated Ġabstract s Ġto Ġensure Ġthe Ġannotation Ġquality Ġbefore Ġhiring Ġthe Ġannot ator . </s> Table Ġ5 Ġpresents Ġthe Ġresults Ġof ĠL ST M - CR F - Pattern Ġmodel Ġtrained Ġon Ġthe Ġre annot ated Ġdifficult Ġsubset Ġand Ġthe Ġrandom Ġsubset . ĠThe Ġfirst Ġtwo Ġrows Ġshow Ġthe Ġresults Ġfor Ġmodels Ġtrained Ġwith Ġexpert Ġannotations . ĠThe Ġmodel Ġtrained Ġon Ġrandom Ġdata Ġhas Ġa Ġslightly Ġbetter ĠF 1 Ġthan Ġthat Ġtrained Ġon Ġthe Ġsame Ġamount Ġof Ġdifficult Ġdata . ĠThe Ġmodel Ġtrained Ġon Ġrandom Ġdata Ġhas Ġhigher Ġprecision Ġbut Ġlower Ġrecall . </s> R ows Ġ3 Ġand Ġ4 Ġlist Ġthe Ġresults Ġfor Ġmodels Ġtrained Ġon Ġthe Ġsame Ġdata Ġbut Ġwith Ġcrowd Ġannotation . ĠModels Ġtrained Ġwith Ġexpert - annot ated Ġdata Ġare Ġclearly Ġsuperior Ġto Ġthose Ġtrained Ġwith Ġcrowd Ġlabels Ġwith Ġrespect Ġto ĠF 1 , Ġindicating Ġthat Ġthe Ġexperts Ġproduced Ġhigher Ġquality Ġannotations . ĠFor Ġcrowds ourced Ġannotations , Ġtraining Ġthe Ġmodel Ġwith Ġdata Ġsampled Ġat Ġi . i . d . Ġrandom Ġachieves Ġ2 % Ġhigher ĠF 1 Ġthan Ġwhen Ġdifficult Ġinstances Ġare Ġused . ĠWhen Ġexpert Ġannotations Ġare Ġused , Ġthis Ġdifference Ġis Ġless Ġthan Ġ1 %. ĠThis Ġtrend Ġin Ġperformance Ġmay Ġbe Ġexplained Ġby Ġdifferences Ġin Ġannotation Ġquality : Ġthe Ġrandomly Ġsampled Ġset Ġwas Ġmore Ġconsistently Ġannot ated Ġby Ġboth Ġexperts Ġand Ġcrowd Ġbecause Ġthe Ġdifficult Ġset Ġis Ġharder . ĠHowever , Ġin Ġboth Ġcases Ġexpert Ġannotations Ġare Ġbetter , Ġwith Ġa Ġbigger Ġdifference Ġbetween Ġthe Ġexpert Ġand Ġcrowd Ġmodels Ġon Ġthe Ġdifficult Ġset . </s> The Ġlast Ġrow Ġis Ġthe Ġmodel Ġtrained Ġon Ġall Ġ5 k Ġabstract s Ġwith Ġcrowd Ġannotations . ĠIts ĠF 1 Ġscore Ġis Ġlower Ġthan Ġeither Ġexpert Ġmodel Ġtrained Ġon Ġonly Ġ20 % Ġof Ġdata , Ġsuggesting Ġthat Ġexpert Ġannotations Ġshould Ġbe Ġcollected Ġwhenever Ġpossible . ĠAgain Ġthe Ġcrowd Ġmodel Ġon Ġcomplete Ġdata Ġhas Ġhigher Ġprecision Ġthan Ġexpert Ġmodels Ġbut Ġits Ġrecall Ġis Ġmuch Ġlower . </s> R outing ĠTo ĠExperts Ġor ĠCrowd </s> So Ġfar Ġa Ġsystem Ġwas Ġtrained Ġon Ġone Ġtype Ġof Ġdata , Ġeither Ġlabeled Ġby Ġcrowd Ġor Ġexperts . ĠWe Ġnow Ġexamine Ġthe Ġperformance Ġof Ġa Ġsystem Ġtrained Ġon Ġdata Ġthat Ġwas Ġrouted Ġto Ġeither Ġexperts Ġor Ġcrowd Ġannot ators Ġdepending Ġon Ġtheir Ġpredicted Ġdifficult . ĠGiven Ġthe Ġresults Ġpresented Ġso Ġfar Ġmixing Ġannot ators Ġmay Ġbe Ġbeneficial Ġgiven Ġtheir Ġrespective Ġtrade - offs Ġof Ġprecision Ġand Ġrecall . ĠWe Ġuse Ġthe Ġannotations Ġfrom Ġexperts Ġfor Ġan Ġabstract Ġif Ġit Ġexists Ġotherwise Ġuse Ġcrowd Ġannotations . ĠThe Ġresults Ġare Ġpresented Ġin ĠTable Ġ6 Ġ. </s> R ows Ġ1 Ġand Ġ2 Ġrepeat Ġthe Ġperformance Ġof Ġthe Ġmodels Ġtrained Ġon Ġdifficult Ġsubset Ġand Ġrandom Ġsubset Ġwith Ġexpert Ġannotations Ġonly Ġrespectively . ĠThe Ġthird Ġrow Ġis Ġthe Ġmodel Ġtrained Ġby Ġcombining Ġdifficult Ġand Ġrandom Ġsubs ets Ġwith Ġexpert Ġannotations . ĠThere Ġare Ġaround Ġ250 Ġabstract s Ġin Ġthe Ġoverlap Ġof Ġthese Ġtwo Ġsets , Ġso Ġthere Ġare Ġtotal Ġ1 . 75 k Ġabstract s Ġused Ġfor Ġtraining Ġthe ĠD + R Ġmodel . ĠR ows Ġ4 Ġto Ġ6 Ġare Ġthe Ġmodels Ġtrained Ġon Ġall Ġ5 k Ġabstract s Ġwith Ġmixed Ġannotations , Ġwhere ĠOther Ġmeans Ġthe Ġrest Ġof Ġthe Ġabstract s Ġwith Ġcrowd Ġannotation Ġonly . </s> The Ġresults Ġshow Ġadding Ġmore Ġtraining Ġdata Ġwith Ġcrowd Ġannotation Ġstill Ġimproves Ġat Ġleast Ġ1 Ġpoint ĠF 1 Ġscore Ġin Ġall Ġthree Ġextraction Ġtasks . ĠThe Ġimprovement Ġwhen Ġthe Ġdifficult Ġsubset Ġwith Ġexpert Ġannotations Ġis Ġmixed Ġwith Ġthe Ġremaining Ġcrowd Ġannotation Ġis Ġ3 . 5 ĠF 1 Ġscore , Ġmuch Ġlarger Ġthan Ġwhen Ġa Ġrandom Ġset Ġof Ġexpert Ġannotations Ġare Ġadded . ĠThe Ġmodel Ġtrained Ġwith Ġre - annot ating Ġthe Ġdifficult Ġsubset Ġ( D + Other ) Ġalso Ġoutper forms Ġthe Ġmodel Ġwith Ġre - annot ating Ġthe Ġrandom Ġsubset Ġ( R + Other ) Ġby Ġ2 Ġpoints Ġin ĠF 1 . ĠThe Ġmodel Ġtrained Ġwith Ġre - annot ating Ġboth Ġof Ġdifficult Ġand Ġrandom Ġsubs ets Ġ( D + R + Other ), Ġhowever , Ġachieves Ġonly Ġmarginally Ġhigher ĠF 1 Ġthan Ġthe Ġmodel Ġtrained Ġwith Ġthe Ġre - annot ated Ġdifficult Ġsubset Ġ( D + Other ). ĠIn Ġsum , Ġthe Ġresults Ġclearly Ġindicate Ġthat Ġmixing Ġexpert Ġand Ġcrowd Ġannotations Ġleads Ġto Ġbetter Ġmodels Ġthan Ġusing Ġsolely Ġcrowd Ġdata , Ġand Ġbetter Ġthan Ġusing Ġexpert Ġdata Ġalone . ĠMore Ġimportantly , Ġthere Ġis Ġgreater Ġgain Ġin Ġperformance Ġwhen Ġinstances Ġare Ġrouted Ġaccording Ġto Ġdifficulty , Ġas Ġcompared Ġto Ġrandomly Ġselecting Ġthe Ġdata Ġfor Ġexpert Ġannot ators . ĠThese Ġfindings Ġalign Ġwith Ġour Ġmotivating Ġhypothesis Ġthat Ġannotation Ġquality Ġfor Ġdifficult Ġinstances Ġis Ġimportant Ġfor Ġfinal Ġmodel Ġperformance . ĠThey Ġalso Ġindicate Ġthat Ġmixing Ġannotations Ġfrom Ġexpert Ġand Ġcrowd Ġcould Ġbe Ġan Ġeffective Ġway Ġto Ġachieve Ġacceptable Ġmodel Ġperformance Ġgiven Ġa Ġlimited Ġbudget . </s> How ĠMany ĠExpert ĠAnnotations ? </s> We Ġestablished Ġthat Ġcrowd Ġannotation Ġare Ġstill Ġuseful Ġin Ġsupplement ing Ġexpert Ġannotations Ġfor Ġmedical ĠIE . ĠOb taining Ġexpert Ġannotations Ġfor Ġthe Ġone Ġthousand Ġmost Ġdifficult Ġinstances Ġgreatly Ġimproved Ġthe Ġmodel Ġperformance . ĠHowever Ġthe Ġchoice Ġof Ġhow Ġmany Ġdifficult Ġinstances Ġto Ġannot ate Ġwas Ġan Ġunin formed Ġchoice . ĠHere Ġwe Ġcheck Ġif Ġless Ġexpert Ġdata Ġwould Ġhave Ġyielded Ġsimilar Ġgains . ĠFuture Ġwork Ġwill Ġneed Ġto Ġaddress Ġhow Ġbest Ġto Ġchoose Ġthis Ġparameter Ġfor Ġa Ġrouting Ġsystem . </s> We Ġsimulate Ġa Ġrouting Ġscenario Ġin Ġwhich Ġwe Ġsend Ġconsecutive Ġbatches Ġof Ġthe Ġmost Ġdifficult Ġexamples Ġto Ġthe Ġexperts Ġfor Ġannotation . ĠWe Ġtrack Ġchanges Ġin Ġperformance Ġas Ġwe Ġincrease Ġthe Ġnumber Ġof Ġmost - diff icult - articles Ġsent Ġto Ġdomain Ġexperts . ĠAs Ġshown Ġin ĠFigure Ġ4 Ġ, Ġadding Ġexpert Ġannotations Ġfor Ġdifficult Ġarticles Ġconsistently Ġincreases ĠF 1 Ġscores . ĠThe Ġperformance Ġgain Ġis Ġmostly Ġfrom Ġincreased Ġrecall ; Ġthe Ġprecision Ġchanges Ġonly Ġa Ġbit Ġwith Ġhigher Ġquality Ġannotation . ĠThis Ġobservation Ġimplies Ġthat Ġcrowd Ġworkers Ġoften Ġfail Ġto Ġmark Ġtarget Ġtokens , Ġbut Ġdo Ġnot Ġtend Ġto Ġproduce Ġlarge Ġnumbers Ġof Ġfalse Ġpositives . ĠWe Ġsuspect Ġsuch Ġfailures Ġto Ġidentify Ġrelevant Ġspans / t ok ens Ġare Ġdue Ġto Ġinsufficient Ġdomain Ġknowledge Ġpossessed Ġby Ġcrowd Ġworkers . </s> The ĠF 1 Ġscore Ġachieved Ġafter Ġre - annot ating Ġthe Ġ600 Ġmost - diff icult Ġarticles Ġreaches Ġ68 . 1 %, Ġwhich Ġis Ġclose Ġto Ġthe Ġperformance Ġwhen Ġre - annot ating Ġ1000 Ġrandom Ġarticles . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġrecognizing Ġdifficult Ġinstances . ĠThe Ġtrend Ġwhen Ġwe Ġuse Ġup Ġall Ġexpert Ġdata Ġis Ġstill Ġupward , Ġso Ġadding Ġeven Ġmore Ġexpert Ġdata Ġis Ġlikely Ġto Ġfurther Ġimprove Ġperformance . ĠUnfortunately Ġwe Ġexhausted Ġour Ġbudget Ġand Ġwere Ġnot Ġable Ġto Ġobtain Ġadditional Ġexpert Ġannotations . ĠIt Ġis Ġlikely Ġthat Ġas Ġthe Ġsize Ġof Ġthe Ġexpert Ġannotations Ġincreases , Ġthe Ġvalue Ġof Ġcrowd Ġannotations Ġwill Ġdiminish . ĠThis Ġinvestigation Ġis Ġleft Ġfor Ġfuture Ġwork . </s> Con clusions </s> We Ġhave Ġintroduced Ġthe Ġtask Ġof Ġpredicting Ġannotation Ġdifficulty Ġfor Ġbiomedical Ġinformation Ġextraction Ġ( IE ). ĠWe Ġtrained Ġneural Ġmodels Ġusing Ġdifferent Ġlearned Ġrepresentations Ġto Ġscore Ġtexts Ġin Ġterms Ġof Ġtheir Ġdifficulty . ĠResults Ġfrom Ġall Ġmodels Ġwere Ġstrong Ġwith ĠPearson âĢ Ļ s Ġcorrelation Ġcoefficients Ġhigher Ġthan Ġ0 . 45 Ġin Ġalmost Ġall Ġevaluations , Ġindicating Ġthe Ġfeasibility Ġof Ġthis Ġtask . ĠAn Ġensemble Ġmodel Ġcombining Ġuniversal Ġand Ġtask Ġspecific Ġfeature Ġsentence Ġvectors Ġyielded Ġthe Ġbest Ġresults . </s> Exper iments Ġon Ġbiomedical ĠIE Ġtasks Ġshow Ġthat Ġremoving Ġup Ġto Ġ$\ sim Ġ$ Ġ10 % Ġof Ġthe Ġsentences Ġpredicted Ġto Ġbe Ġmost Ġdifficult Ġdid Ġnot Ġdecrease Ġmodel Ġperformance , Ġand Ġthat Ġre - weight ing Ġsentences Ġin versely Ġto Ġtheir Ġdifficulty Ġscore Ġduring Ġtraining Ġimproves Ġpredictive Ġperformance . ĠSim ulations Ġin Ġwhich Ġdifficult Ġexamples Ġare Ġrouted Ġto Ġexperts Ġand Ġother Ġinstances Ġto Ġcrowd Ġannot ators Ġyields Ġthe Ġbest Ġresults , Ġoutper forming Ġthe Ġstrategy Ġof Ġrandomly Ġselecting Ġdata Ġfor Ġexpert Ġannotation , Ġand Ġsubstantially Ġimproving Ġupon Ġthe Ġapproach Ġof Ġrelying Ġexclusively Ġon Ġcrowd Ġannotations . ĠIn Ġfuture Ġwork , Ġrouting Ġstrategies Ġbased Ġon Ġinstance Ġdifficulty Ġcould Ġbe Ġfurther Ġinvestigated Ġfor Ġbudget - quality Ġtrade - off . </s> Acknowled gements </s> This Ġwork Ġhas Ġbeen Ġpartially Ġsupported Ġby ĠNS F 17 48 771 Ġgrant . ĠWallace Ġwas Ġsupport Ġin Ġpart Ġby ĠNIH / N LM ĠR 01 LM 01 20 86 .
Introduction Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges. Experts with the requisite domain knowledge to perform high-quality annotation tend to be expensive, while lay annotators may not have the necessary knowledge to provide high-quality annotations. A practical approach for collecting a sufficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk). However, crowd workers in general are likely to provide noisy annotations BIBREF0 , BIBREF1 , BIBREF2 , an issue exacerbated by the technical nature of specialized content. Some of this noise may reflect worker quality and can be modeled BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 , but for some instances lay people may simply lack the domain knowledge to provide useful annotation. In this paper we report experiments on the EBM-NLP corpus comprising crowdsourced annotations of medical literature BIBREF5 . We operationalize the concept of annotation difficulty and show how it can be exploited during training to improve information extraction models. We then obtain expert annotations for the abstracts predicted to be most difficult, as well as for a similar number of randomly selected abstracts. The annotation of highly specialized data and the use of lay and expert annotators allow us to examine the following key questions related to lay and expert annotations in specialized domains: Can we predict item difficulty? We define a training instance as difficult if a lay annotator or an automated model disagree on its labeling. We show that difficulty can be predicted, and that it is distinct from inter-annotator agreement. Further, such predictions can be used during training to improve information extraction models. Are there systematic differences between expert and lay annotations? We observe decidedly lower agreement between lay workers as compared to domain experts. Lay annotations have high precision but low recall with respect to expert annotations in the new data that we collected. More generally, we expect lay annotations to be lower quality, which may translate to lower precision, recall, or both, compared to expert annotations. Can one rely solely on lay annotations? Reasonable models can be trained using lay annotations alone, but similar performance can be achieved using markedly less expert data. This suggests that the optimal ratio of expert to crowd annotations for specialized tasks will depend on the cost and availability of domain experts. Expert annotations are preferable whenever its collection is practical. But in real-world settings, a combination of expert and lay annotations is better than using lay data alone. Does it matter what data is annotated by experts? We demonstrate that a system trained on combined data achieves better predictive performance when experts annotate difficult examples rather than instances selected at i.i.d. random. Our contributions in this work are summarized as follows. We define a task difficulty prediction task and show how this is related to, but distinct from, inter-worker agreement. We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task. We show that predicting annotation difficulty can be used to improve the task routing and model performance for a biomedical information extraction task. Our results open up a new direction for ensuring corpus quality. We believe that item difficulty prediction will likely be useful in other, non-specialized tasks as well, and that the most effective data collection in specialized domains requires research addressing the fundamental questions we examine here. Related Work Crowdsourcing annotation is now a well-studied problem BIBREF7 , BIBREF0 , BIBREF1 , BIBREF2 . Due to the noise inherent in such annotations, there have also been considerable efforts to develop aggregation models that minimize noise BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 . There are also several surveys of crowdsourcing in biomedicine specifically BIBREF8 , BIBREF9 , BIBREF10 . Some work in this space has contrasted model performance achieved using expert vs. crowd annotated training data BIBREF11 , BIBREF12 , BIBREF13 . Dumitrache et al. Dumitrache:2018:CGT:3232718.3152889 concluded that performance is similar under these supervision types, finding no clear advantage from using expert annotators. This differs from our findings, perhaps owing to differences in design. The experts we used already hold advanced medical degrees, for instance, while those in prior work were medical students. Furthermore, the task considered here would appear to be of greater difficulty: even a system trained on $\sim $ 5k instances performs reasonably, but far from perfect. By contrast, in some of the prior work where experts and crowd annotations were deemed equivalent, a classifier trained on 300 examples can achieve very high accuracy BIBREF12 . More relevant to this paper, prior work has investigated methods for `task routing' in active learning scenarios in which supervision is provided by heterogeneous labelers with varying levels of expertise BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF14 . The related question of whether effort is better spent collecting additional annotations for already labeled (but potentially noisily so) examples or novel instances has also been addressed BIBREF18 . What distinguishes the work here is our focus on providing an operational definition of instance difficulty, showing that this can be predicted, and then using this to inform task routing. Application Domain Our specific application concerns annotating abstracts of articles that describe the conduct and results of randomized controlled trials (RCTs). Experimentation in this domain has become easy with the recent release of the EBM-NLP BIBREF5 corpus, which includes a reasonably large training dataset annotated via crowdsourcing, and a modest test set labeled by individuals with advanced medical training. More specifically, the training set comprises 4,741 medical article abstracts with crowdsourced annotations indicating snippets (sequences) that describe the Participants (p), Interventions (i), and Outcome (o) elements of the respective RCT, and the test set is composed of 191 abstracts with p, i, o sequence annotations from three medical experts. Table 1 shows an example of difficult and easy examples according to our definition of difficulty. The underlined text demarcates the (consensus) reference label provided by domain experts. In the difficult examples, crowd workers marked text distinct from these reference annotations; whereas in the easy cases they reproduced them with reasonable fidelity. The difficult sentences usually exhibit complicated structure and feature jargon. An abstract may contain some `easy' and some `difficult' sentences. We thus perform our analysis at the sentence level. We split abstracts into sentences using spaCy. We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting. In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively. Quantifying Task Difficulty The test set includes annotations from both crowd workers and domain experts. We treat the latter as ground truth and then define the difficulty of sentences in terms of the observed agreement between expert and lay annotators. Formally, for annotation task $t$ and instance $i$ : $$\text{Difficulty}_{ti} = \frac{\sum _{j=1}^n{f(\text{label}_{ij}, y_i})}{n}$$   (Eq. 3) where $f$ is a scoring function that measures the quality of the label from worker $j$ for sentence $i$ , as compared to a ground truth annotation, $y_i$ . The difficulty score of sentence $i$ is taken as an average over the scores for all $n$ layworkers. We use Spearmans' correlation coefficient as a scoring function. Specifically, for each sentence we create two vectors comprising counts of how many times each token was annotated by crowd and expert workers, respectively, and calculate the correlation between these. Sentences with no labels are treated as maximally easy; those with only either crowd worker or expert label(s) are assumed maximally difficult. The training set contains only crowdsourced annotations. To label the training data, we use a 10-fold validation like setting. We iteratively retrain the LSTM-CRF-Pattern sequence tagger of Patel et al. patel2018syntactic on 9 folds of the training data and use that trained model to predict labels for the 10th. In this way we obtain predictions on the full training set. We then use predicted spans as proxy `ground truth' annotations to calculate the difficulty score of sentences as described above; we normalize these to the [ $0, 1$ ] interval. We validate this approximation by comparing the proxy scores against reference scores over the test set, the Pearson's correlation coefficients are 0.57 for Population, 0.71 for Intervention and 0.68 for Outcome. There exist many sentences that contain neither manual nor predicted annotations. We treat these as maximally easy sentences (with difficulty scores of 0). Such sentences comprise 51%, 42% and 36% for Population, Interventions and Outcomes data respectively, indicating that it is easier to identify sentences that have no Population spans, but harder to identify sentences that have no Interventions or Outcomes spans. This is intuitive as descriptions of the latter two tend to be more technical and dense with medical jargon. We show the distribution of the automatically labeled scores for sentences that do contain spans in Figure 1 . The mean of the Population (p) sentence scores is significantly lower than that for other types of sentences (i and o), again indicating that they are easier on average to annotate. This aligns with a previous finding that annotating Interventions and Outcomes is more difficult than annotating Participants BIBREF5 . Many sentences contain spans tagged by the LSTM-CRF-Pattern model, but missed by all crowd workers, resulting in a maximally difficult score (1). Inspection of such sentences revealed that some are truly difficult examples, but others are tagging model errors. In either case, such sentences have confused workers and/or the model, and so we retain them all as `difficult' sentences. Content describing the p, i and o, respectively, is quite different. As such, one sentence usually contains (at most) only one of these three content types. We thus treat difficulty prediction for the respective label types as separate tasks. Difficulty is not Worker Agreement Our definition of difficulty is derived from agreement between expert and crowd annotations for the test data, and agreement between a predictive model and crowd annotations in the training data. It is reasonable to ask if these measures are related to inter-annotator agreement, a metric often used in language technology research to identify ambiguous or difficult items. Here we explicitly verify that our definition of difficulty only weakly correlates with inter-annotator agreement. We calculate inter-worker agreement between crowd and expert annotators using Spearman's correlation coefficient. As shown in Table 2 , average agreement between domain experts are considerably higher than agreements between crowd workers for all three label types. This is a clear indication that the crowd annotations are noisier. Furthermore, we compare the correlation between inter-annotator agreement and difficulty scores in the training data. Given that the majority of sentences do not contain a PICO span, we only include in these calculations those that contain a reference label. Pearson's r are 0.34, 0.30 and 0.31 for p, i and o, respectively, confirming that inter-worker agreement and our proposed difficulty score are quite distinct. Predicting Annotation Difficulty We treat difficulty prediction as a regression problem, and propose and evaluate neural model variants for the task. We first train RNN BIBREF19 and CNN BIBREF20 models. We also use the universal sentence encoder (USE) BIBREF6 to induce sentence representations, and train a model using these as features. Following BIBREF6 , we then experiment with an ensemble model that combines the `universal' and task-specific representations to predict annotation difficulty. We expect these universal embeddings to capture general, high-level semantics, and the task specific representations to capture more granular information. Figure 2 depicts the model architecture. Sentences are fed into both the universal sentence encoder and, separately, a task specific neural encoder, yielding two representations. We concatenate these and pass the combined vector to the regression layer. Experimental Setup and Results We trained models for each label type separately. Word embeddings were initialized to 300d GloVe vectors BIBREF21 trained on common crawl data; these are fine-tuned during training. We used the Adam optimizer BIBREF22 with learning rate and decay set to 0.001 and 0.99, respectively. We used batch sizes of 16. We used the large version of the universal sentence encoder with a transformer BIBREF23 . We did not update the pretrained sentence encoder parameters during training. All hyperparamaters for all models (including hidden layers, hidden sizes, and dropout) were tuned using Vizier BIBREF24 via 10-fold cross validation on the training set maximizing for F1. As a baseline, we also trained a linear Support-Vector Regression BIBREF25 model on $n$ -gram features ( $n$ ranges from 1 to 3). Table 3 reports Pearson correlation coefficients between the predictions with each of the neural models and the ground truth difficulty scores. Rows 1-4 correspond to individual models, and row 5 reports the ensemble performance. Columns correspond to label type. Results from all models outperform the baseline SVR model: Pearson's correlation coefficients range from 0.550 to 0.622. The regression correlations are the lowest. The RNN model realizes the strongest performance among the stand-alone (non-ensemble) models, outperforming variants that exploit CNN and USE representations. Combining the RNN and USE further improves results. We hypothesize that this is due to complementary sentence information encoded in universal representations. For all models, correlations for Intervention and Outcomes are higher than for Population, which is expected given the difficulty distributions in Figure 1 . In these, the sentences are more uniformly distributed, with a fair number of difficult and easier sentences. By contrast, in Population there are a greater number of easy sentences and considerably fewer difficult sentences, which makes the difficulty ranking task particularly challenging. Better IE with Difficulty Prediction We next present experiments in which we attempt to use the predicted difficulty during training to improve models for information extraction of descriptions of Population, Interventions and Outcomes from medical article abstracts. We investigate two uses: (1) simply removing the most difficult sentences from the training set, and, (2) re-weighting the most difficult sentences. We again use LSTM-CRF-Pattern as the base model and experimenting on the EBM-NLP corpus BIBREF5 . This is trained on either (1) the training set with difficult sentences removed, or (2) the full training set but with instances re-weighted in proportion to their predicted difficulty score. Following BIBREF5 , we use the Adam optimizer with learning rate of 0.001, decay 0.9, batch size 20 and dropout 0.5. We use pretrained 200d GloVe vectors BIBREF21 to initialize word embeddings, and use 100d hidden char representations. Each word is thus represented with 300 dimensions in total. The hidden size is 100 for the LSTM in the character representation component, and 200 for the LSTM in the information extraction component. We train for 15 epochs, saving parameters that achieve the best F1 score on a nested development set. Removing Difficult Examples We first evaluate changes in performance induced by training the sequence labeling model using less data by removing difficult sentences prior to training. The hypothesis here is that these difficult instances are likely to introduce more noise than signal. We used a cross-fold approach to predict sentence difficulties, training on 9/10ths of the data and scoring the remaining 1/10th at a time. We then sorted sentences by predicted difficulty scores, and experimented with removing increasing numbers of these (in order of difficulty) prior to training the LSTM-CRF-Pattern model. Figure 3 shows the results achieved by the LSTM-CRF-Pattern model after discarding increasing amounts of the training data: the $x$ and $y$ axes correspond to the the percentage of data removed and F1 scores, respectively. We contrast removing sentences predicted to be difficult with removing them (a) randomly (i.i.d.), and, (b) in inverse order of predicted inter-annotator agreement. The agreement prediction model is trained exactly the same like difficult prediction model, with simply changing the difficult score to annotation agreement. F1 scores actually improve (marginally) when we remove the most difficult sentences, up until we drop 4% of the data for Population and Interventions, and 6% for Outcomes. Removing training points at i.i.d. random degrades performance, as expected. Removing sentences in order of disagreement seems to have similar effect as removing them by difficulty score when removing small amount of the data, but the F1 scores drop much faster when removing more data. These findings indicate that sentences predicted to be difficult are indeed noisy, to the extent that they do not seem to provide the model useful signal. Re-weighting by Difficulty We showed above that removing a small number of the most difficult sentences does not harm, and in fact modestly improves, medical IE model performance. However, using the available data we are unable to test if this will be useful in practice, as we would need additional data to determine how many difficult sentences should be dropped. We instead explore an alternative, practical means of exploiting difficulty predictions: we re-weight sentences during training inversely to their predicted difficulty. Formally, we weight sentence $i$ with difficulty scores above $\tau $ according to: $1-a\cdot (d_i-\tau )/(1-\tau )$ , where $d_i$ is the difficulty score for sentence $i$ , and $a$ is a parameter codifying the minimum weight value. We set $\tau $ to 0.8 so as to only re-weight sentences with difficulty in the top 20th percentile, and we set $a$ to 0.5. The re-weighting is equivalent to down-sampling the difficult sentences. LSTM-CRF-Pattern is our base model. Table 4 reports the precision, recall and F1 achieved both with and without sentence re-weighting. Re-weighting improves all metrics modestly but consistently. All F1 differences are statistically significant under a sign test ( $p<0.01$ ). The model with best precision is different for Patient, Intervention and Outcome labels. However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points. This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement. Involving Expert Annotators The preceding experiments demonstrate that re-weighting difficult sentences annotated by the crowd generally improves the extraction models. Presumably the performance is influenced by the annotation quality. We now examine the possibility that the higher quality and more consistent annotations of domain experts on the difficult instances will benefit the extraction model. This simulates an annotation strategy in which we route difficult instances to domain experts and easier ones to crowd annotators. We also contrast the value of difficult data to that of an i.i.d. random sample of the same size, both annotated by experts. Expert annotations of Random and Difficult Instances We re-annotate by experts a subset of most difficult instances and the same number of random instances. As collecting annotations from experts is slow and expensive, we only re-annotate the difficult instances for the interventions extraction task. We re-annotate the abstracts which cover the sentences with predicted difficulty scores in the top 5 percentile. We rank the abstracts from the training set by the count of difficult sentences, and re-annotate the abstracts that contain the most difficult sentences. Constrained by time and budget, we select only 2000 abstracts for re-annotation; 1000 of these are top-ranked, and 1000 are randomly sampled. This re-annotation cost $3,000. We have released the new annotation data at: https://github.com/bepnye/EBM-NLP. Following BIBREF5 , we recruited five medical experts via Up-work with advanced medical training and strong technical reading/writing skills. The expert annotator were asked to read the entire abstract and highlight, using the BRAT toolkit BIBREF26 , all spans describing medical Interventions. Each abstract is only annotated by one expert. We examined 30 re-annotated abstracts to ensure the annotation quality before hiring the annotator. Table 5 presents the results of LSTM-CRF-Pattern model trained on the reannotated difficult subset and the random subset. The first two rows show the results for models trained with expert annotations. The model trained on random data has a slightly better F1 than that trained on the same amount of difficult data. The model trained on random data has higher precision but lower recall. Rows 3 and 4 list the results for models trained on the same data but with crowd annotation. Models trained with expert-annotated data are clearly superior to those trained with crowd labels with respect to F1, indicating that the experts produced higher quality annotations. For crowdsourced annotations, training the model with data sampled at i.i.d. random achieves 2% higher F1 than when difficult instances are used. When expert annotations are used, this difference is less than 1%. This trend in performance may be explained by differences in annotation quality: the randomly sampled set was more consistently annotated by both experts and crowd because the difficult set is harder. However, in both cases expert annotations are better, with a bigger difference between the expert and crowd models on the difficult set. The last row is the model trained on all 5k abstracts with crowd annotations. Its F1 score is lower than either expert model trained on only 20% of data, suggesting that expert annotations should be collected whenever possible. Again the crowd model on complete data has higher precision than expert models but its recall is much lower. Routing To Experts or Crowd So far a system was trained on one type of data, either labeled by crowd or experts. We now examine the performance of a system trained on data that was routed to either experts or crowd annotators depending on their predicted difficult. Given the results presented so far mixing annotators may be beneficial given their respective trade-offs of precision and recall. We use the annotations from experts for an abstract if it exists otherwise use crowd annotations. The results are presented in Table 6 . Rows 1 and 2 repeat the performance of the models trained on difficult subset and random subset with expert annotations only respectively. The third row is the model trained by combining difficult and random subsets with expert annotations. There are around 250 abstracts in the overlap of these two sets, so there are total 1.75k abstracts used for training the D+R model. Rows 4 to 6 are the models trained on all 5k abstracts with mixed annotations, where Other means the rest of the abstracts with crowd annotation only. The results show adding more training data with crowd annotation still improves at least 1 point F1 score in all three extraction tasks. The improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3.5 F1 score, much larger than when a random set of expert annotations are added. The model trained with re-annotating the difficult subset (D+Other) also outperforms the model with re-annotating the random subset (R+Other) by 2 points in F1. The model trained with re-annotating both of difficult and random subsets (D+R+Other), however, achieves only marginally higher F1 than the model trained with the re-annotated difficult subset (D+Other). In sum, the results clearly indicate that mixing expert and crowd annotations leads to better models than using solely crowd data, and better than using expert data alone. More importantly, there is greater gain in performance when instances are routed according to difficulty, as compared to randomly selecting the data for expert annotators. These findings align with our motivating hypothesis that annotation quality for difficult instances is important for final model performance. They also indicate that mixing annotations from expert and crowd could be an effective way to achieve acceptable model performance given a limited budget. How Many Expert Annotations? We established that crowd annotation are still useful in supplementing expert annotations for medical IE. Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance. However the choice of how many difficult instances to annotate was an uninformed choice. Here we check if less expert data would have yielded similar gains. Future work will need to address how best to choose this parameter for a routing system. We simulate a routing scenario in which we send consecutive batches of the most difficult examples to the experts for annotation. We track changes in performance as we increase the number of most-difficult-articles sent to domain experts. As shown in Figure 4 , adding expert annotations for difficult articles consistently increases F1 scores. The performance gain is mostly from increased recall; the precision changes only a bit with higher quality annotation. This observation implies that crowd workers often fail to mark target tokens, but do not tend to produce large numbers of false positives. We suspect such failures to identify relevant spans/tokens are due to insufficient domain knowledge possessed by crowd workers. The F1 score achieved after re-annotating the 600 most-difficult articles reaches 68.1%, which is close to the performance when re-annotating 1000 random articles. This demonstrates the effectiveness of recognizing difficult instances. The trend when we use up all expert data is still upward, so adding even more expert data is likely to further improve performance. Unfortunately we exhausted our budget and were not able to obtain additional expert annotations. It is likely that as the size of the expert annotations increases, the value of crowd annotations will diminish. This investigation is left for future work. Conclusions We have introduced the task of predicting annotation difficulty for biomedical information extraction (IE). We trained neural models using different learned representations to score texts in terms of their difficulty. Results from all models were strong with Pearson’s correlation coefficients higher than 0.45 in almost all evaluations, indicating the feasibility of this task. An ensemble model combining universal and task specific feature sentence vectors yielded the best results. Experiments on biomedical IE tasks show that removing up to $\sim $ 10% of the sentences predicted to be most difficult did not decrease model performance, and that re-weighting sentences inversely to their difficulty score during training improves predictive performance. Simulations in which difficult examples are routed to experts and other instances to crowd annotators yields the best results, outperforming the strategy of randomly selecting data for expert annotation, and substantially improving upon the approach of relying exclusively on crowd annotations. In future work, routing strategies based on instance difficulty could be further investigated for budget-quality trade-off. Acknowledgements This work has been partially supported by NSF1748771 grant. Wallace was support in part by NIH/NLM R01LM012086.
How much data is needed to train the task-specific encoder?
8c48c726bb17a17d70ab29db4d65a93030dd5382
How Ġmuch Ġdata Ġis Ġneeded Ġto Ġtrain Ġthe Ġtask - specific Ġenc oder ?
57 , 505 Ġsentences
<s> How Ġmuch Ġdata Ġis Ġneeded Ġto Ġtrain Ġthe Ġtask - specific Ġenc oder ? </s> Introduction </s> As semb ling Ġtraining Ġcorpor a Ġof Ġannot ated Ġnatural Ġlanguage Ġexamples Ġin Ġspecialized Ġdomains Ġsuch Ġas Ġbi omed ic ine Ġposes Ġconsiderable Ġchallenges . ĠExperts Ġwith Ġthe Ġrequisite Ġdomain Ġknowledge Ġto Ġperform Ġhigh - quality Ġannotation Ġtend Ġto Ġbe Ġexpensive , Ġwhile Ġlay Ġannot ators Ġmay Ġnot Ġhave Ġthe Ġnecessary Ġknowledge Ġto Ġprovide Ġhigh - quality Ġannotations . ĠA Ġpractical Ġapproach Ġfor Ġcollecting Ġa Ġsufficiently Ġlarge Ġcorpus Ġwould Ġbe Ġto Ġuse Ġcrowds ourcing Ġplatforms Ġlike ĠAmazon ĠMechanical ĠTurk Ġ( MT ur k ). ĠHowever , Ġcrowd Ġworkers Ġin Ġgeneral Ġare Ġlikely Ġto Ġprovide Ġnoisy Ġannotations ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġan Ġissue Ġexacerbated Ġby Ġthe Ġtechnical Ġnature Ġof Ġspecialized Ġcontent . ĠSome Ġof Ġthis Ġnoise Ġmay Ġreflect Ġworker Ġquality Ġand Ġcan Ġbe Ġmodeled ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġbut Ġfor Ġsome Ġinstances Ġlay Ġpeople Ġmay Ġsimply Ġlack Ġthe Ġdomain Ġknowledge Ġto Ġprovide Ġuseful Ġannotation . </s> In Ġthis Ġpaper Ġwe Ġreport Ġexperiments Ġon Ġthe ĠE BM - N LP Ġcorpus Ġcomprising Ġcrowds ourced Ġannotations Ġof Ġmedical Ġliterature ĠB IB REF 5 Ġ. ĠWe Ġoperational ize Ġthe Ġconcept Ġof Ġannotation Ġdifficulty Ġand Ġshow Ġhow Ġit Ġcan Ġbe Ġexploited Ġduring Ġtraining Ġto Ġimprove Ġinformation Ġextraction Ġmodels . ĠWe Ġthen Ġobtain Ġexpert Ġannotations Ġfor Ġthe Ġabstract s Ġpredicted Ġto Ġbe Ġmost Ġdifficult , Ġas Ġwell Ġas Ġfor Ġa Ġsimilar Ġnumber Ġof Ġrandomly Ġselected Ġabstract s . ĠThe Ġannotation Ġof Ġhighly Ġspecialized Ġdata Ġand Ġthe Ġuse Ġof Ġlay Ġand Ġexpert Ġannot ators Ġallow Ġus Ġto Ġexamine Ġthe Ġfollowing Ġkey Ġquestions Ġrelated Ġto Ġlay Ġand Ġexpert Ġannotations Ġin Ġspecialized Ġdomains : </s> Can Ġwe Ġpredict Ġitem Ġdifficulty ? ĠWe Ġdefine Ġa Ġtraining Ġinstance Ġas Ġdifficult Ġif Ġa Ġlay Ġannot ator Ġor Ġan Ġautomated Ġmodel Ġdisagree Ġon Ġits Ġlabeling . ĠWe Ġshow Ġthat Ġdifficulty Ġcan Ġbe Ġpredicted , Ġand Ġthat Ġit Ġis Ġdistinct Ġfrom Ġinter - annot ator Ġagreement . ĠFurther , Ġsuch Ġpredictions Ġcan Ġbe Ġused Ġduring Ġtraining Ġto Ġimprove Ġinformation Ġextraction Ġmodels . </s> Are Ġthere Ġsystematic Ġdifferences Ġbetween Ġexpert Ġand Ġlay Ġannotations ? ĠWe Ġobserve Ġdecidedly Ġlower Ġagreement Ġbetween Ġlay Ġworkers Ġas Ġcompared Ġto Ġdomain Ġexperts . ĠLay Ġannotations Ġhave Ġhigh Ġprecision Ġbut Ġlow Ġrecall Ġwith Ġrespect Ġto Ġexpert Ġannotations Ġin Ġthe Ġnew Ġdata Ġthat Ġwe Ġcollected . ĠMore Ġgenerally , Ġwe Ġexpect Ġlay Ġannotations Ġto Ġbe Ġlower Ġquality , Ġwhich Ġmay Ġtranslate Ġto Ġlower Ġprecision , Ġrecall , Ġor Ġboth , Ġcompared Ġto Ġexpert Ġannotations . ĠCan Ġone Ġrely Ġsolely Ġon Ġlay Ġannotations ? ĠRe asonable Ġmodels Ġcan Ġbe Ġtrained Ġusing Ġlay Ġannotations Ġalone , Ġbut Ġsimilar Ġperformance Ġcan Ġbe Ġachieved Ġusing Ġmarkedly Ġless Ġexpert Ġdata . ĠThis Ġsuggests Ġthat Ġthe Ġoptimal Ġratio Ġof Ġexpert Ġto Ġcrowd Ġannotations Ġfor Ġspecialized Ġtasks Ġwill Ġdepend Ġon Ġthe Ġcost Ġand Ġavailability Ġof Ġdomain Ġexperts . ĠExpert Ġannotations Ġare Ġpreferable Ġwhenever Ġits Ġcollection Ġis Ġpractical . ĠBut Ġin Ġreal - world Ġsettings , Ġa Ġcombination Ġof Ġexpert Ġand Ġlay Ġannotations Ġis Ġbetter Ġthan Ġusing Ġlay Ġdata Ġalone . </s> Does Ġit Ġmatter Ġwhat Ġdata Ġis Ġannot ated Ġby Ġexperts ? ĠWe Ġdemonstrate Ġthat Ġa Ġsystem Ġtrained Ġon Ġcombined Ġdata Ġachieves Ġbetter Ġpredictive Ġperformance Ġwhen Ġexperts Ġannot ate Ġdifficult Ġexamples Ġrather Ġthan Ġinstances Ġselected Ġat Ġi . i . d . Ġrandom . </s> Our Ġcontributions Ġin Ġthis Ġwork Ġare Ġsummarized Ġas Ġfollows . ĠWe Ġdefine Ġa Ġtask Ġdifficulty Ġprediction Ġtask Ġand Ġshow Ġhow Ġthis Ġis Ġrelated Ġto , Ġbut Ġdistinct Ġfrom , Ġinter - worker Ġagreement . ĠWe Ġintroduce Ġa Ġnew Ġmodel Ġfor Ġdifficulty Ġprediction Ġcombining Ġlearned Ġrepresentations Ġinduced Ġvia Ġa Ġpre - trained Ġ` universal ' Ġsentence Ġenc oder ĠB IB REF 6 Ġ, Ġand Ġa Ġsentence Ġenc oder Ġlearned Ġfrom Ġscratch Ġfor Ġthis Ġtask . ĠWe Ġshow Ġthat Ġpredicting Ġannotation Ġdifficulty Ġcan Ġbe Ġused Ġto Ġimprove Ġthe Ġtask Ġrouting Ġand Ġmodel Ġperformance Ġfor Ġa Ġbiomedical Ġinformation Ġextraction Ġtask . ĠOur Ġresults Ġopen Ġup Ġa Ġnew Ġdirection Ġfor Ġensuring Ġcorpus Ġquality . ĠWe Ġbelieve Ġthat Ġitem Ġdifficulty Ġprediction Ġwill Ġlikely Ġbe Ġuseful Ġin Ġother , Ġnon - special ized Ġtasks Ġas Ġwell , Ġand Ġthat Ġthe Ġmost Ġeffective Ġdata Ġcollection Ġin Ġspecialized Ġdomains Ġrequires Ġresearch Ġaddressing Ġthe Ġfundamental Ġquestions Ġwe Ġexamine Ġhere . </s> Related ĠWork </s> C rowd s ourcing Ġannotation Ġis Ġnow Ġa Ġwell - stud ied Ġproblem ĠB IB REF 7 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠDue Ġto Ġthe Ġnoise Ġinherent Ġin Ġsuch Ġannotations , Ġthere Ġhave Ġalso Ġbeen Ġconsiderable Ġefforts Ġto Ġdevelop Ġaggregation Ġmodels Ġthat Ġminimize Ġnoise ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. </s> There Ġare Ġalso Ġseveral Ġsurveys Ġof Ġcrowds ourcing Ġin Ġbi omed ic ine Ġspecifically ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠSome Ġwork Ġin Ġthis Ġspace Ġhas Ġcontrasted Ġmodel Ġperformance Ġachieved Ġusing Ġexpert Ġvs . Ġcrowd Ġannot ated Ġtraining Ġdata ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠDum it r ache Ġet Ġal . ĠDum it r ache : 2018 : C GT : 323 27 18 . 315 28 89 Ġconcluded Ġthat Ġperformance Ġis Ġsimilar Ġunder Ġthese Ġsupervision Ġtypes , Ġfinding Ġno Ġclear Ġadvantage Ġfrom Ġusing Ġexpert Ġannot ators . ĠThis Ġdiffers Ġfrom Ġour Ġfindings , Ġperhaps Ġowing Ġto Ġdifferences Ġin Ġdesign . ĠThe Ġexperts Ġwe Ġused Ġalready Ġhold Ġadvanced Ġmedical Ġdegrees , Ġfor Ġinstance , Ġwhile Ġthose Ġin Ġprior Ġwork Ġwere Ġmedical Ġstudents . ĠFurthermore , Ġthe Ġtask Ġconsidered Ġhere Ġwould Ġappear Ġto Ġbe Ġof Ġgreater Ġdifficulty : Ġeven Ġa Ġsystem Ġtrained Ġon Ġ$\ sim Ġ$ Ġ5 k Ġinstances Ġperforms Ġreasonably , Ġbut Ġfar Ġfrom Ġperfect . ĠBy Ġcontrast , Ġin Ġsome Ġof Ġthe Ġprior Ġwork Ġwhere Ġexperts Ġand Ġcrowd Ġannotations Ġwere Ġdeemed Ġequivalent , Ġa Ġclass ifier Ġtrained Ġon Ġ300 Ġexamples Ġcan Ġachieve Ġvery Ġhigh Ġaccuracy ĠB IB REF 12 Ġ. </s> More Ġrelevant Ġto Ġthis Ġpaper , Ġprior Ġwork Ġhas Ġinvestigated Ġmethods Ġfor Ġ` task Ġrouting ' Ġin Ġactive Ġlearning Ġscenarios Ġin Ġwhich Ġsupervision Ġis Ġprovided Ġby Ġheter ogeneous Ġlabel ers Ġwith Ġvarying Ġlevels Ġof Ġexpertise ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 14 Ġ. ĠThe Ġrelated Ġquestion Ġof Ġwhether Ġeffort Ġis Ġbetter Ġspent Ġcollecting Ġadditional Ġannotations Ġfor Ġalready Ġlabeled Ġ( but Ġpotentially Ġno is ily Ġso ) Ġexamples Ġor Ġnovel Ġinstances Ġhas Ġalso Ġbeen Ġaddressed ĠB IB REF 18 Ġ. ĠWhat Ġdistinguishes Ġthe Ġwork Ġhere Ġis Ġour Ġfocus Ġon Ġproviding Ġan Ġoperational Ġdefinition Ġof Ġinstance Ġdifficulty , Ġshowing Ġthat Ġthis Ġcan Ġbe Ġpredicted , Ġand Ġthen Ġusing Ġthis Ġto Ġinform Ġtask Ġrouting . </s> Application ĠDomain </s> Our Ġspecific Ġapplication Ġconcerns Ġannot ating Ġabstract s Ġof Ġarticles Ġthat Ġdescribe Ġthe Ġconduct Ġand Ġresults Ġof Ġrandomized Ġcontrolled Ġtrials Ġ( R CT s ). ĠExperiment ation Ġin Ġthis Ġdomain Ġhas Ġbecome Ġeasy Ġwith Ġthe Ġrecent Ġrelease Ġof Ġthe ĠE BM - N LP ĠB IB REF 5 Ġcorpus , Ġwhich Ġincludes Ġa Ġreasonably Ġlarge Ġtraining Ġdataset Ġannot ated Ġvia Ġcrowds ourcing , Ġand Ġa Ġmodest Ġtest Ġset Ġlabeled Ġby Ġindividuals Ġwith Ġadvanced Ġmedical Ġtraining . ĠMore Ġspecifically , Ġthe Ġtraining Ġset Ġcomprises Ġ4 , 7 41 Ġmedical Ġarticle Ġabstract s Ġwith Ġcrowds ourced Ġannotations Ġindicating Ġsnippets Ġ( sequ ences ) Ġthat Ġdescribe Ġthe ĠParticipants Ġ( p ), ĠInter ventions Ġ( i ), Ġand ĠOut come Ġ( o ) Ġelements Ġof Ġthe Ġrespective ĠR CT , Ġand Ġthe Ġtest Ġset Ġis Ġcomposed Ġof Ġ191 Ġabstract s Ġwith Ġp , Ġi , Ġo Ġsequence Ġannotations Ġfrom Ġthree Ġmedical Ġexperts . </s> Table Ġ1 Ġshows Ġan Ġexample Ġof Ġdifficult Ġand Ġeasy Ġexamples Ġaccording Ġto Ġour Ġdefinition Ġof Ġdifficulty . ĠThe Ġunder lined Ġtext Ġdem arc ates Ġthe Ġ( cons ensus ) Ġreference Ġlabel Ġprovided Ġby Ġdomain Ġexperts . ĠIn Ġthe Ġdifficult Ġexamples , Ġcrowd Ġworkers Ġmarked Ġtext Ġdistinct Ġfrom Ġthese Ġreference Ġannotations ; Ġwhereas Ġin Ġthe Ġeasy Ġcases Ġthey Ġreproduced Ġthem Ġwith Ġreasonable Ġfidelity . ĠThe Ġdifficult Ġsentences Ġusually Ġexhibit Ġcomplicated Ġstructure Ġand Ġfeature Ġjargon . </s> An Ġabstract Ġmay Ġcontain Ġsome Ġ` easy ' Ġand Ġsome Ġ` diff icult ' Ġsentences . ĠWe Ġthus Ġperform Ġour Ġanalysis Ġat Ġthe Ġsentence Ġlevel . ĠWe Ġsplit Ġabstract s Ġinto Ġsentences Ġusing Ġspa Cy . ĠWe Ġexcluded Ġsentences Ġthat Ġcomprise Ġfewer Ġthan Ġtwo Ġtokens , Ġas Ġthese Ġare Ġlikely Ġan Ġartifact Ġof Ġerrors Ġin Ġsentence Ġsplitting . ĠIn Ġtotal , Ġthis Ġresulted Ġin Ġ57 , 505 Ġand Ġ2 , 428 Ġsentences Ġin Ġthe Ġtrain Ġand Ġtest Ġset Ġabstract s , Ġrespectively . </s> Quant ifying ĠTask ĠDifficulty </s> The Ġtest Ġset Ġincludes Ġannotations Ġfrom Ġboth Ġcrowd Ġworkers Ġand Ġdomain Ġexperts . ĠWe Ġtreat Ġthe Ġlatter Ġas Ġground Ġtruth Ġand Ġthen Ġdefine Ġthe Ġdifficulty Ġof Ġsentences Ġin Ġterms Ġof Ġthe Ġobserved Ġagreement Ġbetween Ġexpert Ġand Ġlay Ġannot ators . ĠForm ally , Ġfor Ġannotation Ġtask Ġ$ t $ Ġand Ġinstance Ġ$ i $ Ġ: </s> $$ \ text { Diff iculty } _{ ti } Ġ= Ġ\ frac {\ sum Ġ_ { j = 1 } ^ n { f (\ text { label } _{ ij }, Ġy _ i }) }{ n } $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> where Ġ$ f $ Ġis Ġa Ġscoring Ġfunction Ġthat Ġmeasures Ġthe Ġquality Ġof Ġthe Ġlabel Ġfrom Ġworker Ġ$ j $ Ġfor Ġsentence Ġ$ i $ Ġ, Ġas Ġcompared Ġto Ġa Ġground Ġtruth Ġannotation , Ġ$ y _ i $ Ġ. ĠThe Ġdifficulty Ġscore Ġof Ġsentence Ġ$ i $ Ġis Ġtaken Ġas Ġan Ġaverage Ġover Ġthe Ġscores Ġfor Ġall Ġ$ n $ Ġlay workers . ĠWe Ġuse ĠSpe arm ans ' Ġcorrelation Ġcoefficient Ġas Ġa Ġscoring Ġfunction . ĠSpecifically , Ġfor Ġeach Ġsentence Ġwe Ġcreate Ġtwo Ġvectors Ġcomprising Ġcounts Ġof Ġhow Ġmany Ġtimes Ġeach Ġtoken Ġwas Ġannot ated Ġby Ġcrowd Ġand Ġexpert Ġworkers , Ġrespectively , Ġand Ġcalculate Ġthe Ġcorrelation Ġbetween Ġthese . ĠSent ences Ġwith Ġno Ġlabels Ġare Ġtreated Ġas Ġmaxim ally Ġeasy ; Ġthose Ġwith Ġonly Ġeither Ġcrowd Ġworker Ġor Ġexpert Ġlabel ( s ) Ġare Ġassumed Ġmaxim ally Ġdifficult . </s> The Ġtraining Ġset Ġcontains Ġonly Ġcrowds ourced Ġannotations . ĠTo Ġlabel Ġthe Ġtraining Ġdata , Ġwe Ġuse Ġa Ġ10 - fold Ġvalidation Ġlike Ġsetting . ĠWe Ġiter atively Ġret rain Ġthe ĠL ST M - CR F - Pattern Ġsequence Ġtag ger Ġof ĠPatel Ġet Ġal . Ġp atel 2018 sy nt actic Ġon Ġ9 Ġfolds Ġof Ġthe Ġtraining Ġdata Ġand Ġuse Ġthat Ġtrained Ġmodel Ġto Ġpredict Ġlabels Ġfor Ġthe Ġ10 th . ĠIn Ġthis Ġway Ġwe Ġobtain Ġpredictions Ġon Ġthe Ġfull Ġtraining Ġset . ĠWe Ġthen Ġuse Ġpredicted Ġspans Ġas Ġproxy Ġ` ground Ġtruth ' Ġannotations Ġto Ġcalculate Ġthe Ġdifficulty Ġscore Ġof Ġsentences Ġas Ġdescribed Ġabove ; Ġwe Ġnormal ize Ġthese Ġto Ġthe Ġ[ Ġ$ 0 , Ġ1 $ Ġ] Ġinterval . ĠWe Ġvalidate Ġthis Ġapproximation Ġby Ġcomparing Ġthe Ġproxy Ġscores Ġagainst Ġreference Ġscores Ġover Ġthe Ġtest Ġset , Ġthe ĠPearson 's Ġcorrelation Ġcoefficients Ġare Ġ0 . 57 Ġfor ĠPopulation , Ġ0 . 71 Ġfor ĠIntervention Ġand Ġ0 . 68 Ġfor ĠOut come . </s> There Ġexist Ġmany Ġsentences Ġthat Ġcontain Ġneither Ġmanual Ġnor Ġpredicted Ġannotations . ĠWe Ġtreat Ġthese Ġas Ġmaxim ally Ġeasy Ġsentences Ġ( with Ġdifficulty Ġscores Ġof Ġ0 ). ĠSuch Ġsentences Ġcomprise Ġ51 %, Ġ42 % Ġand Ġ36 % Ġfor ĠPopulation , ĠInter ventions Ġand ĠOut comes Ġdata Ġrespectively , Ġindicating Ġthat Ġit Ġis Ġeasier Ġto Ġidentify Ġsentences Ġthat Ġhave Ġno ĠPopulation Ġspans , Ġbut Ġharder Ġto Ġidentify Ġsentences Ġthat Ġhave Ġno ĠInter ventions Ġor ĠOut comes Ġspans . ĠThis Ġis Ġintuitive Ġas Ġdescriptions Ġof Ġthe Ġlatter Ġtwo Ġtend Ġto Ġbe Ġmore Ġtechnical Ġand Ġdense Ġwith Ġmedical Ġjargon . </s> We Ġshow Ġthe Ġdistribution Ġof Ġthe Ġautomatically Ġlabeled Ġscores Ġfor Ġsentences Ġthat Ġdo Ġcontain Ġspans Ġin ĠFigure Ġ1 Ġ. ĠThe Ġmean Ġof Ġthe ĠPopulation Ġ( p ) Ġsentence Ġscores Ġis Ġsignificantly Ġlower Ġthan Ġthat Ġfor Ġother Ġtypes Ġof Ġsentences Ġ( i Ġand Ġo ), Ġagain Ġindicating Ġthat Ġthey Ġare Ġeasier Ġon Ġaverage Ġto Ġannot ate . ĠThis Ġalign s Ġwith Ġa Ġprevious Ġfinding Ġthat Ġannot ating ĠInter ventions Ġand ĠOut comes Ġis Ġmore Ġdifficult Ġthan Ġannot ating ĠParticipants ĠB IB REF 5 Ġ. </s> Many Ġsentences Ġcontain Ġspans Ġtagged Ġby Ġthe ĠL ST M - CR F - Pattern Ġmodel , Ġbut Ġmissed Ġby Ġall Ġcrowd Ġworkers , Ġresulting Ġin Ġa Ġmaxim ally Ġdifficult Ġscore Ġ( 1 ). ĠInspection Ġof Ġsuch Ġsentences Ġrevealed Ġthat Ġsome Ġare Ġtruly Ġdifficult Ġexamples , Ġbut Ġothers Ġare Ġtagging Ġmodel Ġerrors . ĠIn Ġeither Ġcase , Ġsuch Ġsentences Ġhave Ġconfused Ġworkers Ġand / or Ġthe Ġmodel , Ġand Ġso Ġwe Ġretain Ġthem Ġall Ġas Ġ` diff icult ' Ġsentences . </s> Content Ġdescribing Ġthe Ġp , Ġi Ġand Ġo , Ġrespectively , Ġis Ġquite Ġdifferent . ĠAs Ġsuch , Ġone Ġsentence Ġusually Ġcontains Ġ( at Ġmost ) Ġonly Ġone Ġof Ġthese Ġthree Ġcontent Ġtypes . ĠWe Ġthus Ġtreat Ġdifficulty Ġprediction Ġfor Ġthe Ġrespective Ġlabel Ġtypes Ġas Ġseparate Ġtasks . </s> Diff iculty Ġis Ġnot ĠWorker ĠAgreement </s> Our Ġdefinition Ġof Ġdifficulty Ġis Ġderived Ġfrom Ġagreement Ġbetween Ġexpert Ġand Ġcrowd Ġannotations Ġfor Ġthe Ġtest Ġdata , Ġand Ġagreement Ġbetween Ġa Ġpredictive Ġmodel Ġand Ġcrowd Ġannotations Ġin Ġthe Ġtraining Ġdata . ĠIt Ġis Ġreasonable Ġto Ġask Ġif Ġthese Ġmeasures Ġare Ġrelated Ġto Ġinter - annot ator Ġagreement , Ġa Ġmetric Ġoften Ġused Ġin Ġlanguage Ġtechnology Ġresearch Ġto Ġidentify Ġambiguous Ġor Ġdifficult Ġitems . ĠHere Ġwe Ġexplicitly Ġverify Ġthat Ġour Ġdefinition Ġof Ġdifficulty Ġonly Ġweak ly Ġcorrelates Ġwith Ġinter - annot ator Ġagreement . </s> We Ġcalculate Ġinter - worker Ġagreement Ġbetween Ġcrowd Ġand Ġexpert Ġannot ators Ġusing ĠSpear man 's Ġcorrelation Ġcoefficient . ĠAs Ġshown Ġin ĠTable Ġ2 Ġ, Ġaverage Ġagreement Ġbetween Ġdomain Ġexperts Ġare Ġconsiderably Ġhigher Ġthan Ġagreements Ġbetween Ġcrowd Ġworkers Ġfor Ġall Ġthree Ġlabel Ġtypes . ĠThis Ġis Ġa Ġclear Ġindication Ġthat Ġthe Ġcrowd Ġannotations Ġare Ġno is ier . </s> Furthermore , Ġwe Ġcompare Ġthe Ġcorrelation Ġbetween Ġinter - annot ator Ġagreement Ġand Ġdifficulty Ġscores Ġin Ġthe Ġtraining Ġdata . ĠGiven Ġthat Ġthe Ġmajority Ġof Ġsentences Ġdo Ġnot Ġcontain Ġa ĠP ICO Ġspan , Ġwe Ġonly Ġinclude Ġin Ġthese Ġcalculations Ġthose Ġthat Ġcontain Ġa Ġreference Ġlabel . ĠPearson 's Ġr Ġare Ġ0 . 34 , Ġ0 . 30 Ġand Ġ0 . 31 Ġfor Ġp , Ġi Ġand Ġo , Ġrespectively , Ġconfirming Ġthat Ġinter - worker Ġagreement Ġand Ġour Ġproposed Ġdifficulty Ġscore Ġare Ġquite Ġdistinct . </s> P redict ing ĠAn notation ĠDifficulty </s> We Ġtreat Ġdifficulty Ġprediction Ġas Ġa Ġregression Ġproblem , Ġand Ġpropose Ġand Ġevaluate Ġneural Ġmodel Ġvariants Ġfor Ġthe Ġtask . ĠWe Ġfirst Ġtrain ĠR NN ĠB IB REF 19 Ġand ĠCNN ĠB IB REF 20 Ġmodels . </s> We Ġalso Ġuse Ġthe Ġuniversal Ġsentence Ġenc oder Ġ( USE ) ĠB IB REF 6 Ġto Ġinduce Ġsentence Ġrepresentations , Ġand Ġtrain Ġa Ġmodel Ġusing Ġthese Ġas Ġfeatures . ĠFollowing ĠB IB REF 6 Ġ, Ġwe Ġthen Ġexperiment Ġwith Ġan Ġensemble Ġmodel Ġthat Ġcombines Ġthe Ġ` universal ' Ġand Ġtask - specific Ġrepresentations Ġto Ġpredict Ġannotation Ġdifficulty . ĠWe Ġexpect Ġthese Ġuniversal Ġembed d ings Ġto Ġcapture Ġgeneral , Ġhigh - level Ġsemantics , Ġand Ġthe Ġtask Ġspecific Ġrepresentations Ġto Ġcapture Ġmore Ġgran ular Ġinformation . ĠFigure Ġ2 Ġdepicts Ġthe Ġmodel Ġarchitecture . ĠSent ences Ġare Ġfed Ġinto Ġboth Ġthe Ġuniversal Ġsentence Ġenc oder Ġand , Ġseparately , Ġa Ġtask Ġspecific Ġneural Ġenc oder , Ġyielding Ġtwo Ġrepresentations . ĠWe Ġconc aten ate Ġthese Ġand Ġpass Ġthe Ġcombined Ġvector Ġto Ġthe Ġregression Ġlayer . </s> Exper imental ĠSetup Ġand ĠResults </s> We Ġtrained Ġmodels Ġfor Ġeach Ġlabel Ġtype Ġseparately . ĠWord Ġembed d ings Ġwere Ġinitialized Ġto Ġ300 d ĠGl o Ve Ġvectors ĠB IB REF 21 Ġtrained Ġon Ġcommon Ġcrawl Ġdata ; Ġthese Ġare Ġfine - tun ed Ġduring Ġtraining . ĠWe Ġused Ġthe ĠAdam Ġoptim izer ĠB IB REF 22 Ġwith Ġlearning Ġrate Ġand Ġdecay Ġset Ġto Ġ0 . 001 Ġand Ġ0 . 99 , Ġrespectively . ĠWe Ġused Ġbatch Ġsizes Ġof Ġ16 . </s> We Ġused Ġthe Ġlarge Ġversion Ġof Ġthe Ġuniversal Ġsentence Ġenc oder Ġwith Ġa Ġtransformer ĠB IB REF 23 Ġ. ĠWe Ġdid Ġnot Ġupdate Ġthe Ġpret rained Ġsentence Ġenc oder Ġparameters Ġduring Ġtraining . ĠAll Ġhyper param aters Ġfor Ġall Ġmodels Ġ( including Ġhidden Ġlayers , Ġhidden Ġsizes , Ġand Ġdrop out ) Ġwere Ġtuned Ġusing ĠViz ier ĠB IB REF 24 Ġvia Ġ10 - fold Ġcross Ġvalidation Ġon Ġthe Ġtraining Ġset Ġmaximizing Ġfor ĠF 1 . </s> As Ġa Ġbaseline , Ġwe Ġalso Ġtrained Ġa Ġlinear ĠSupport - Vector ĠReg ression ĠB IB REF 25 Ġmodel Ġon Ġ$ n $ Ġ- gram Ġfeatures Ġ( Ġ$ n $ Ġranges Ġfrom Ġ1 Ġto Ġ3 ). </s> Table Ġ3 Ġreports ĠPearson Ġcorrelation Ġcoefficients Ġbetween Ġthe Ġpredictions Ġwith Ġeach Ġof Ġthe Ġneural Ġmodels Ġand Ġthe Ġground Ġtruth Ġdifficulty Ġscores . ĠR ows Ġ1 - 4 Ġcorrespond Ġto Ġindividual Ġmodels , Ġand Ġrow Ġ5 Ġreports Ġthe Ġensemble Ġperformance . ĠColumn s Ġcorrespond Ġto Ġlabel Ġtype . ĠResults Ġfrom Ġall Ġmodels Ġoutper form Ġthe Ġbaseline ĠS VR Ġmodel : ĠPearson 's Ġcorrelation Ġcoefficients Ġrange Ġfrom Ġ0 . 550 Ġto Ġ0 . 6 22 . ĠThe Ġregression Ġcorrelations Ġare Ġthe Ġlowest . </s> The ĠR NN Ġmodel Ġrealizes Ġthe Ġstrongest Ġperformance Ġamong Ġthe Ġstand - alone Ġ( non - ense mble ) Ġmodels , Ġoutper forming Ġvariants Ġthat Ġexploit ĠCNN Ġand ĠUSE Ġrepresentations . ĠComb ining Ġthe ĠR NN Ġand ĠUSE Ġfurther Ġimproves Ġresults . ĠWe Ġhypothes ize Ġthat Ġthis Ġis Ġdue Ġto Ġcomplementary Ġsentence Ġinformation Ġencoded Ġin Ġuniversal Ġrepresentations . </s> For Ġall Ġmodels , Ġcorrelations Ġfor ĠIntervention Ġand ĠOut comes Ġare Ġhigher Ġthan Ġfor ĠPopulation , Ġwhich Ġis Ġexpected Ġgiven Ġthe Ġdifficulty Ġdistributions Ġin ĠFigure Ġ1 Ġ. ĠIn Ġthese , Ġthe Ġsentences Ġare Ġmore Ġuniformly Ġdistributed , Ġwith Ġa Ġfair Ġnumber Ġof Ġdifficult Ġand Ġeasier Ġsentences . ĠBy Ġcontrast , Ġin ĠPopulation Ġthere Ġare Ġa Ġgreater Ġnumber Ġof Ġeasy Ġsentences Ġand Ġconsiderably Ġfewer Ġdifficult Ġsentences , Ġwhich Ġmakes Ġthe Ġdifficulty Ġranking Ġtask Ġparticularly Ġchallenging . </s> Better ĠIE Ġwith ĠDifficulty ĠPrediction </s> We Ġnext Ġpresent Ġexperiments Ġin Ġwhich Ġwe Ġattempt Ġto Ġuse Ġthe Ġpredicted Ġdifficulty Ġduring Ġtraining Ġto Ġimprove Ġmodels Ġfor Ġinformation Ġextraction Ġof Ġdescriptions Ġof ĠPopulation , ĠInter ventions Ġand ĠOut comes Ġfrom Ġmedical Ġarticle Ġabstract s . ĠWe Ġinvestigate Ġtwo Ġuses : Ġ( 1 ) Ġsimply Ġremoving Ġthe Ġmost Ġdifficult Ġsentences Ġfrom Ġthe Ġtraining Ġset , Ġand , Ġ( 2 ) Ġre - weight ing Ġthe Ġmost Ġdifficult Ġsentences . </s> We Ġagain Ġuse ĠL ST M - CR F - Pattern Ġas Ġthe Ġbase Ġmodel Ġand Ġexperimenting Ġon Ġthe ĠE BM - N LP Ġcorpus ĠB IB REF 5 Ġ. ĠThis Ġis Ġtrained Ġon Ġeither Ġ( 1 ) Ġthe Ġtraining Ġset Ġwith Ġdifficult Ġsentences Ġremoved , Ġor Ġ( 2 ) Ġthe Ġfull Ġtraining Ġset Ġbut Ġwith Ġinstances Ġre - weight ed Ġin Ġproportion Ġto Ġtheir Ġpredicted Ġdifficulty Ġscore . ĠFollowing ĠB IB REF 5 Ġ, Ġwe Ġuse Ġthe ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġof Ġ0 . 001 , Ġdecay Ġ0 . 9 , Ġbatch Ġsize Ġ20 Ġand Ġdrop out Ġ0 . 5 . ĠWe Ġuse Ġpret rained Ġ200 d ĠGl o Ve Ġvectors ĠB IB REF 21 Ġto Ġinitialize Ġword Ġembed d ings , Ġand Ġuse Ġ100 d Ġhidden Ġchar Ġrepresentations . ĠEach Ġword Ġis Ġthus Ġrepresented Ġwith Ġ300 Ġdimensions Ġin Ġtotal . ĠThe Ġhidden Ġsize Ġis Ġ100 Ġfor Ġthe ĠL ST M Ġin Ġthe Ġcharacter Ġrepresentation Ġcomponent , Ġand Ġ200 Ġfor Ġthe ĠL ST M Ġin Ġthe Ġinformation Ġextraction Ġcomponent . ĠWe Ġtrain Ġfor Ġ15 Ġepoch s , Ġsaving Ġparameters Ġthat Ġachieve Ġthe Ġbest ĠF 1 Ġscore Ġon Ġa Ġnested Ġdevelopment Ġset . </s> Rem oving ĠDiff icult ĠExamples </s> We Ġfirst Ġevaluate Ġchanges Ġin Ġperformance Ġinduced Ġby Ġtraining Ġthe Ġsequence Ġlabeling Ġmodel Ġusing Ġless Ġdata Ġby Ġremoving Ġdifficult Ġsentences Ġprior Ġto Ġtraining . ĠThe Ġhypothesis Ġhere Ġis Ġthat Ġthese Ġdifficult Ġinstances Ġare Ġlikely Ġto Ġintroduce Ġmore Ġnoise Ġthan Ġsignal . ĠWe Ġused Ġa Ġcross - fold Ġapproach Ġto Ġpredict Ġsentence Ġdifficulties , Ġtraining Ġon Ġ9 / 10 ths Ġof Ġthe Ġdata Ġand Ġscoring Ġthe Ġremaining Ġ1 / 10 th Ġat Ġa Ġtime . ĠWe Ġthen Ġsorted Ġsentences Ġby Ġpredicted Ġdifficulty Ġscores , Ġand Ġexperimented Ġwith Ġremoving Ġincreasing Ġnumbers Ġof Ġthese Ġ( in Ġorder Ġof Ġdifficulty ) Ġprior Ġto Ġtraining Ġthe ĠL ST M - CR F - Pattern Ġmodel . </s> Figure Ġ3 Ġshows Ġthe Ġresults Ġachieved Ġby Ġthe ĠL ST M - CR F - Pattern Ġmodel Ġafter Ġdisc arding Ġincreasing Ġamounts Ġof Ġthe Ġtraining Ġdata : Ġthe Ġ$ x $ Ġand Ġ$ y $ Ġaxes Ġcorrespond Ġto Ġthe Ġthe Ġpercentage Ġof Ġdata Ġremoved Ġand ĠF 1 Ġscores , Ġrespectively . ĠWe Ġcontrast Ġremoving Ġsentences Ġpredicted Ġto Ġbe Ġdifficult Ġwith Ġremoving Ġthem Ġ( a ) Ġrandomly Ġ( i . i . d .), Ġand , Ġ( b ) Ġin Ġinverse Ġorder Ġof Ġpredicted Ġinter - annot ator Ġagreement . ĠThe Ġagreement Ġprediction Ġmodel Ġis Ġtrained Ġexactly Ġthe Ġsame Ġlike Ġdifficult Ġprediction Ġmodel , Ġwith Ġsimply Ġchanging Ġthe Ġdifficult Ġscore Ġto Ġannotation Ġagreement . ĠF 1 Ġscores Ġactually Ġimprove Ġ( marg inally ) Ġwhen Ġwe Ġremove Ġthe Ġmost Ġdifficult Ġsentences , Ġup Ġuntil Ġwe Ġdrop Ġ4 % Ġof Ġthe Ġdata Ġfor ĠPopulation Ġand ĠInter ventions , Ġand Ġ6 % Ġfor ĠOut comes . ĠRem oving Ġtraining Ġpoints Ġat Ġi . i . d . Ġrandom Ġde grades Ġperformance , Ġas Ġexpected . ĠRem oving Ġsentences Ġin Ġorder Ġof Ġdisagreement Ġseems Ġto Ġhave Ġsimilar Ġeffect Ġas Ġremoving Ġthem Ġby Ġdifficulty Ġscore Ġwhen Ġremoving Ġsmall Ġamount Ġof Ġthe Ġdata , Ġbut Ġthe ĠF 1 Ġscores Ġdrop Ġmuch Ġfaster Ġwhen Ġremoving Ġmore Ġdata . ĠThese Ġfindings Ġindicate Ġthat Ġsentences Ġpredicted Ġto Ġbe Ġdifficult Ġare Ġindeed Ġnoisy , Ġto Ġthe Ġextent Ġthat Ġthey Ġdo Ġnot Ġseem Ġto Ġprovide Ġthe Ġmodel Ġuseful Ġsignal . </s> Re - weight ing Ġby ĠDifficulty </s> We Ġshowed Ġabove Ġthat Ġremoving Ġa Ġsmall Ġnumber Ġof Ġthe Ġmost Ġdifficult Ġsentences Ġdoes Ġnot Ġharm , Ġand Ġin Ġfact Ġmodest ly Ġimproves , Ġmedical ĠIE Ġmodel Ġperformance . ĠHowever , Ġusing Ġthe Ġavailable Ġdata Ġwe Ġare Ġunable Ġto Ġtest Ġif Ġthis Ġwill Ġbe Ġuseful Ġin Ġpractice , Ġas Ġwe Ġwould Ġneed Ġadditional Ġdata Ġto Ġdetermine Ġhow Ġmany Ġdifficult Ġsentences Ġshould Ġbe Ġdropped . </s> We Ġinstead Ġexplore Ġan Ġalternative , Ġpractical Ġmeans Ġof Ġexploiting Ġdifficulty Ġpredictions : Ġwe Ġre - weight Ġsentences Ġduring Ġtraining Ġin versely Ġto Ġtheir Ġpredicted Ġdifficulty . ĠForm ally , Ġwe Ġweight Ġsentence Ġ$ i $ Ġwith Ġdifficulty Ġscores Ġabove Ġ$\ t au Ġ$ Ġaccording Ġto : Ġ$ 1 - a \ cd ot Ġ( d _ i - \ t au Ġ) /( 1 - \ t au Ġ) $ Ġ, Ġwhere Ġ$ d _ i $ Ġis Ġthe Ġdifficulty Ġscore Ġfor Ġsentence Ġ$ i $ Ġ, Ġand Ġ$ a $ Ġis Ġa Ġparameter Ġcod ifying Ġthe Ġminimum Ġweight Ġvalue . ĠWe Ġset Ġ$\ t au Ġ$ Ġto Ġ0 . 8 Ġso Ġas Ġto Ġonly Ġre - weight Ġsentences Ġwith Ġdifficulty Ġin Ġthe Ġtop Ġ20 th Ġpercentile , Ġand Ġwe Ġset Ġ$ a $ Ġto Ġ0 . 5 . ĠThe Ġre - weight ing Ġis Ġequivalent Ġto Ġdown - sam pling Ġthe Ġdifficult Ġsentences . ĠL ST M - CR F - Pattern Ġis Ġour Ġbase Ġmodel . </s> Table Ġ4 Ġreports Ġthe Ġprecision , Ġrecall Ġand ĠF 1 Ġachieved Ġboth Ġwith Ġand Ġwithout Ġsentence Ġre - weight ing . ĠRe - weight ing Ġimproves Ġall Ġmetrics Ġmodest ly Ġbut Ġconsistently . ĠAll ĠF 1 Ġdifferences Ġare Ġstatistically Ġsignificant Ġunder Ġa Ġsign Ġtest Ġ( Ġ$ p < 0 . 01 $ Ġ). ĠThe Ġmodel Ġwith Ġbest Ġprecision Ġis Ġdifferent Ġfor ĠPatient , ĠIntervention Ġand ĠOut come Ġlabels . ĠHowever Ġre - weight ing Ġby Ġdifficulty Ġdoes Ġconsistently Ġyield Ġthe Ġbest Ġrecall Ġfor Ġall Ġthree Ġextraction Ġtypes , Ġwith Ġthe Ġmost Ġnotable Ġimprovement Ġfor Ġi Ġand Ġo , Ġwhere Ġrecall Ġimproved Ġby Ġ10 Ġpercentage Ġpoints . ĠThis Ġperformance Ġincrease Ġtranslated Ġto Ġimprovements Ġin ĠF 1 Ġacross Ġall Ġtypes , Ġas Ġcompared Ġto Ġthe Ġbase Ġmodel Ġand Ġto Ġre - weight ing Ġby Ġagreement . </s> In vol ving ĠExpert ĠAn not ators </s> The Ġpreceding Ġexperiments Ġdemonstrate Ġthat Ġre - weight ing Ġdifficult Ġsentences Ġannot ated Ġby Ġthe Ġcrowd Ġgenerally Ġimproves Ġthe Ġextraction Ġmodels . ĠPresumably Ġthe Ġperformance Ġis Ġinfluenced Ġby Ġthe Ġannotation Ġquality . </s> We Ġnow Ġexamine Ġthe Ġpossibility Ġthat Ġthe Ġhigher Ġquality Ġand Ġmore Ġconsistent Ġannotations Ġof Ġdomain Ġexperts Ġon Ġthe Ġdifficult Ġinstances Ġwill Ġbenefit Ġthe Ġextraction Ġmodel . ĠThis Ġsim ulates Ġan Ġannotation Ġstrategy Ġin Ġwhich Ġwe Ġroute Ġdifficult Ġinstances Ġto Ġdomain Ġexperts Ġand Ġeasier Ġones Ġto Ġcrowd Ġannot ators . ĠWe Ġalso Ġcontrast Ġthe Ġvalue Ġof Ġdifficult Ġdata Ġto Ġthat Ġof Ġan Ġi . i . d . Ġrandom Ġsample Ġof Ġthe Ġsame Ġsize , Ġboth Ġannot ated Ġby Ġexperts . </s> Ex pert Ġannotations Ġof ĠRandom Ġand ĠDiff icult ĠInst ances </s> We Ġre - annot ate Ġby Ġexperts Ġa Ġsubset Ġof Ġmost Ġdifficult Ġinstances Ġand Ġthe Ġsame Ġnumber Ġof Ġrandom Ġinstances . ĠAs Ġcollecting Ġannotations Ġfrom Ġexperts Ġis Ġslow Ġand Ġexpensive , Ġwe Ġonly Ġre - annot ate Ġthe Ġdifficult Ġinstances Ġfor Ġthe Ġinterventions Ġextraction Ġtask . ĠWe Ġre - annot ate Ġthe Ġabstract s Ġwhich Ġcover Ġthe Ġsentences Ġwith Ġpredicted Ġdifficulty Ġscores Ġin Ġthe Ġtop Ġ5 Ġpercentile . ĠWe Ġrank Ġthe Ġabstract s Ġfrom Ġthe Ġtraining Ġset Ġby Ġthe Ġcount Ġof Ġdifficult Ġsentences , Ġand Ġre - annot ate Ġthe Ġabstract s Ġthat Ġcontain Ġthe Ġmost Ġdifficult Ġsentences . ĠCon str ained Ġby Ġtime Ġand Ġbudget , Ġwe Ġselect Ġonly Ġ2000 Ġabstract s Ġfor Ġre - ann otation ; Ġ1000 Ġof Ġthese Ġare Ġtop - ranked , Ġand Ġ1000 Ġare Ġrandomly Ġsampled . ĠThis Ġre - ann otation Ġcost Ġ$ 3 , 000 . ĠWe Ġhave Ġreleased Ġthe Ġnew Ġannotation Ġdata Ġat : Ġhttps :// github . com / b ep ny e / E BM - N LP . </s> Following ĠB IB REF 5 Ġ, Ġwe Ġrecruited Ġfive Ġmedical Ġexperts Ġvia ĠUp - work Ġwith Ġadvanced Ġmedical Ġtraining Ġand Ġstrong Ġtechnical Ġreading / writing Ġskills . ĠThe Ġexpert Ġannot ator Ġwere Ġasked Ġto Ġread Ġthe Ġentire Ġabstract Ġand Ġhighlight , Ġusing Ġthe ĠBR AT Ġtool kit ĠB IB REF 26 Ġ, Ġall Ġspans Ġdescribing Ġmedical ĠInter ventions . ĠEach Ġabstract Ġis Ġonly Ġannot ated Ġby Ġone Ġexpert . ĠWe Ġexamined Ġ30 Ġre - annot ated Ġabstract s Ġto Ġensure Ġthe Ġannotation Ġquality Ġbefore Ġhiring Ġthe Ġannot ator . </s> Table Ġ5 Ġpresents Ġthe Ġresults Ġof ĠL ST M - CR F - Pattern Ġmodel Ġtrained Ġon Ġthe Ġre annot ated Ġdifficult Ġsubset Ġand Ġthe Ġrandom Ġsubset . ĠThe Ġfirst Ġtwo Ġrows Ġshow Ġthe Ġresults Ġfor Ġmodels Ġtrained Ġwith Ġexpert Ġannotations . ĠThe Ġmodel Ġtrained Ġon Ġrandom Ġdata Ġhas Ġa Ġslightly Ġbetter ĠF 1 Ġthan Ġthat Ġtrained Ġon Ġthe Ġsame Ġamount Ġof Ġdifficult Ġdata . ĠThe Ġmodel Ġtrained Ġon Ġrandom Ġdata Ġhas Ġhigher Ġprecision Ġbut Ġlower Ġrecall . </s> R ows Ġ3 Ġand Ġ4 Ġlist Ġthe Ġresults Ġfor Ġmodels Ġtrained Ġon Ġthe Ġsame Ġdata Ġbut Ġwith Ġcrowd Ġannotation . ĠModels Ġtrained Ġwith Ġexpert - annot ated Ġdata Ġare Ġclearly Ġsuperior Ġto Ġthose Ġtrained Ġwith Ġcrowd Ġlabels Ġwith Ġrespect Ġto ĠF 1 , Ġindicating Ġthat Ġthe Ġexperts Ġproduced Ġhigher Ġquality Ġannotations . ĠFor Ġcrowds ourced Ġannotations , Ġtraining Ġthe Ġmodel Ġwith Ġdata Ġsampled Ġat Ġi . i . d . Ġrandom Ġachieves Ġ2 % Ġhigher ĠF 1 Ġthan Ġwhen Ġdifficult Ġinstances Ġare Ġused . ĠWhen Ġexpert Ġannotations Ġare Ġused , Ġthis Ġdifference Ġis Ġless Ġthan Ġ1 %. ĠThis Ġtrend Ġin Ġperformance Ġmay Ġbe Ġexplained Ġby Ġdifferences Ġin Ġannotation Ġquality : Ġthe Ġrandomly Ġsampled Ġset Ġwas Ġmore Ġconsistently Ġannot ated Ġby Ġboth Ġexperts Ġand Ġcrowd Ġbecause Ġthe Ġdifficult Ġset Ġis Ġharder . ĠHowever , Ġin Ġboth Ġcases Ġexpert Ġannotations Ġare Ġbetter , Ġwith Ġa Ġbigger Ġdifference Ġbetween Ġthe Ġexpert Ġand Ġcrowd Ġmodels Ġon Ġthe Ġdifficult Ġset . </s> The Ġlast Ġrow Ġis Ġthe Ġmodel Ġtrained Ġon Ġall Ġ5 k Ġabstract s Ġwith Ġcrowd Ġannotations . ĠIts ĠF 1 Ġscore Ġis Ġlower Ġthan Ġeither Ġexpert Ġmodel Ġtrained Ġon Ġonly Ġ20 % Ġof Ġdata , Ġsuggesting Ġthat Ġexpert Ġannotations Ġshould Ġbe Ġcollected Ġwhenever Ġpossible . ĠAgain Ġthe Ġcrowd Ġmodel Ġon Ġcomplete Ġdata Ġhas Ġhigher Ġprecision Ġthan Ġexpert Ġmodels Ġbut Ġits Ġrecall Ġis Ġmuch Ġlower . </s> R outing ĠTo ĠExperts Ġor ĠCrowd </s> So Ġfar Ġa Ġsystem Ġwas Ġtrained Ġon Ġone Ġtype Ġof Ġdata , Ġeither Ġlabeled Ġby Ġcrowd Ġor Ġexperts . ĠWe Ġnow Ġexamine Ġthe Ġperformance Ġof Ġa Ġsystem Ġtrained Ġon Ġdata Ġthat Ġwas Ġrouted Ġto Ġeither Ġexperts Ġor Ġcrowd Ġannot ators Ġdepending Ġon Ġtheir Ġpredicted Ġdifficult . ĠGiven Ġthe Ġresults Ġpresented Ġso Ġfar Ġmixing Ġannot ators Ġmay Ġbe Ġbeneficial Ġgiven Ġtheir Ġrespective Ġtrade - offs Ġof Ġprecision Ġand Ġrecall . ĠWe Ġuse Ġthe Ġannotations Ġfrom Ġexperts Ġfor Ġan Ġabstract Ġif Ġit Ġexists Ġotherwise Ġuse Ġcrowd Ġannotations . ĠThe Ġresults Ġare Ġpresented Ġin ĠTable Ġ6 Ġ. </s> R ows Ġ1 Ġand Ġ2 Ġrepeat Ġthe Ġperformance Ġof Ġthe Ġmodels Ġtrained Ġon Ġdifficult Ġsubset Ġand Ġrandom Ġsubset Ġwith Ġexpert Ġannotations Ġonly Ġrespectively . ĠThe Ġthird Ġrow Ġis Ġthe Ġmodel Ġtrained Ġby Ġcombining Ġdifficult Ġand Ġrandom Ġsubs ets Ġwith Ġexpert Ġannotations . ĠThere Ġare Ġaround Ġ250 Ġabstract s Ġin Ġthe Ġoverlap Ġof Ġthese Ġtwo Ġsets , Ġso Ġthere Ġare Ġtotal Ġ1 . 75 k Ġabstract s Ġused Ġfor Ġtraining Ġthe ĠD + R Ġmodel . ĠR ows Ġ4 Ġto Ġ6 Ġare Ġthe Ġmodels Ġtrained Ġon Ġall Ġ5 k Ġabstract s Ġwith Ġmixed Ġannotations , Ġwhere ĠOther Ġmeans Ġthe Ġrest Ġof Ġthe Ġabstract s Ġwith Ġcrowd Ġannotation Ġonly . </s> The Ġresults Ġshow Ġadding Ġmore Ġtraining Ġdata Ġwith Ġcrowd Ġannotation Ġstill Ġimproves Ġat Ġleast Ġ1 Ġpoint ĠF 1 Ġscore Ġin Ġall Ġthree Ġextraction Ġtasks . ĠThe Ġimprovement Ġwhen Ġthe Ġdifficult Ġsubset Ġwith Ġexpert Ġannotations Ġis Ġmixed Ġwith Ġthe Ġremaining Ġcrowd Ġannotation Ġis Ġ3 . 5 ĠF 1 Ġscore , Ġmuch Ġlarger Ġthan Ġwhen Ġa Ġrandom Ġset Ġof Ġexpert Ġannotations Ġare Ġadded . ĠThe Ġmodel Ġtrained Ġwith Ġre - annot ating Ġthe Ġdifficult Ġsubset Ġ( D + Other ) Ġalso Ġoutper forms Ġthe Ġmodel Ġwith Ġre - annot ating Ġthe Ġrandom Ġsubset Ġ( R + Other ) Ġby Ġ2 Ġpoints Ġin ĠF 1 . ĠThe Ġmodel Ġtrained Ġwith Ġre - annot ating Ġboth Ġof Ġdifficult Ġand Ġrandom Ġsubs ets Ġ( D + R + Other ), Ġhowever , Ġachieves Ġonly Ġmarginally Ġhigher ĠF 1 Ġthan Ġthe Ġmodel Ġtrained Ġwith Ġthe Ġre - annot ated Ġdifficult Ġsubset Ġ( D + Other ). ĠIn Ġsum , Ġthe Ġresults Ġclearly Ġindicate Ġthat Ġmixing Ġexpert Ġand Ġcrowd Ġannotations Ġleads Ġto Ġbetter Ġmodels Ġthan Ġusing Ġsolely Ġcrowd Ġdata , Ġand Ġbetter Ġthan Ġusing Ġexpert Ġdata Ġalone . ĠMore Ġimportantly , Ġthere Ġis Ġgreater Ġgain Ġin Ġperformance Ġwhen Ġinstances Ġare Ġrouted Ġaccording Ġto Ġdifficulty , Ġas Ġcompared Ġto Ġrandomly Ġselecting Ġthe Ġdata Ġfor Ġexpert Ġannot ators . ĠThese Ġfindings Ġalign Ġwith Ġour Ġmotivating Ġhypothesis Ġthat Ġannotation Ġquality Ġfor Ġdifficult Ġinstances Ġis Ġimportant Ġfor Ġfinal Ġmodel Ġperformance . ĠThey Ġalso Ġindicate Ġthat Ġmixing Ġannotations Ġfrom Ġexpert Ġand Ġcrowd Ġcould Ġbe Ġan Ġeffective Ġway Ġto Ġachieve Ġacceptable Ġmodel Ġperformance Ġgiven Ġa Ġlimited Ġbudget . </s> How ĠMany ĠExpert ĠAnnotations ? </s> We Ġestablished Ġthat Ġcrowd Ġannotation Ġare Ġstill Ġuseful Ġin Ġsupplement ing Ġexpert Ġannotations Ġfor Ġmedical ĠIE . ĠOb taining Ġexpert Ġannotations Ġfor Ġthe Ġone Ġthousand Ġmost Ġdifficult Ġinstances Ġgreatly Ġimproved Ġthe Ġmodel Ġperformance . ĠHowever Ġthe Ġchoice Ġof Ġhow Ġmany Ġdifficult Ġinstances Ġto Ġannot ate Ġwas Ġan Ġunin formed Ġchoice . ĠHere Ġwe Ġcheck Ġif Ġless Ġexpert Ġdata Ġwould Ġhave Ġyielded Ġsimilar Ġgains . ĠFuture Ġwork Ġwill Ġneed Ġto Ġaddress Ġhow Ġbest Ġto Ġchoose Ġthis Ġparameter Ġfor Ġa Ġrouting Ġsystem . </s> We Ġsimulate Ġa Ġrouting Ġscenario Ġin Ġwhich Ġwe Ġsend Ġconsecutive Ġbatches Ġof Ġthe Ġmost Ġdifficult Ġexamples Ġto Ġthe Ġexperts Ġfor Ġannotation . ĠWe Ġtrack Ġchanges Ġin Ġperformance Ġas Ġwe Ġincrease Ġthe Ġnumber Ġof Ġmost - diff icult - articles Ġsent Ġto Ġdomain Ġexperts . ĠAs Ġshown Ġin ĠFigure Ġ4 Ġ, Ġadding Ġexpert Ġannotations Ġfor Ġdifficult Ġarticles Ġconsistently Ġincreases ĠF 1 Ġscores . ĠThe Ġperformance Ġgain Ġis Ġmostly Ġfrom Ġincreased Ġrecall ; Ġthe Ġprecision Ġchanges Ġonly Ġa Ġbit Ġwith Ġhigher Ġquality Ġannotation . ĠThis Ġobservation Ġimplies Ġthat Ġcrowd Ġworkers Ġoften Ġfail Ġto Ġmark Ġtarget Ġtokens , Ġbut Ġdo Ġnot Ġtend Ġto Ġproduce Ġlarge Ġnumbers Ġof Ġfalse Ġpositives . ĠWe Ġsuspect Ġsuch Ġfailures Ġto Ġidentify Ġrelevant Ġspans / t ok ens Ġare Ġdue Ġto Ġinsufficient Ġdomain Ġknowledge Ġpossessed Ġby Ġcrowd Ġworkers . </s> The ĠF 1 Ġscore Ġachieved Ġafter Ġre - annot ating Ġthe Ġ600 Ġmost - diff icult Ġarticles Ġreaches Ġ68 . 1 %, Ġwhich Ġis Ġclose Ġto Ġthe Ġperformance Ġwhen Ġre - annot ating Ġ1000 Ġrandom Ġarticles . ĠThis Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġrecognizing Ġdifficult Ġinstances . ĠThe Ġtrend Ġwhen Ġwe Ġuse Ġup Ġall Ġexpert Ġdata Ġis Ġstill Ġupward , Ġso Ġadding Ġeven Ġmore Ġexpert Ġdata Ġis Ġlikely Ġto Ġfurther Ġimprove Ġperformance . ĠUnfortunately Ġwe Ġexhausted Ġour Ġbudget Ġand Ġwere Ġnot Ġable Ġto Ġobtain Ġadditional Ġexpert Ġannotations . ĠIt Ġis Ġlikely Ġthat Ġas Ġthe Ġsize Ġof Ġthe Ġexpert Ġannotations Ġincreases , Ġthe Ġvalue Ġof Ġcrowd Ġannotations Ġwill Ġdiminish . ĠThis Ġinvestigation Ġis Ġleft Ġfor Ġfuture Ġwork . </s> Con clusions </s> We Ġhave Ġintroduced Ġthe Ġtask Ġof Ġpredicting Ġannotation Ġdifficulty Ġfor Ġbiomedical Ġinformation Ġextraction Ġ( IE ). ĠWe Ġtrained Ġneural Ġmodels Ġusing Ġdifferent Ġlearned Ġrepresentations Ġto Ġscore Ġtexts Ġin Ġterms Ġof Ġtheir Ġdifficulty . ĠResults Ġfrom Ġall Ġmodels Ġwere Ġstrong Ġwith ĠPearson âĢ Ļ s Ġcorrelation Ġcoefficients Ġhigher Ġthan Ġ0 . 45 Ġin Ġalmost Ġall Ġevaluations , Ġindicating Ġthe Ġfeasibility Ġof Ġthis Ġtask . ĠAn Ġensemble Ġmodel Ġcombining Ġuniversal Ġand Ġtask Ġspecific Ġfeature Ġsentence Ġvectors Ġyielded Ġthe Ġbest Ġresults . </s> Exper iments Ġon Ġbiomedical ĠIE Ġtasks Ġshow Ġthat Ġremoving Ġup Ġto Ġ$\ sim Ġ$ Ġ10 % Ġof Ġthe Ġsentences Ġpredicted Ġto Ġbe Ġmost Ġdifficult Ġdid Ġnot Ġdecrease Ġmodel Ġperformance , Ġand Ġthat Ġre - weight ing Ġsentences Ġin versely Ġto Ġtheir Ġdifficulty Ġscore Ġduring Ġtraining Ġimproves Ġpredictive Ġperformance . ĠSim ulations Ġin Ġwhich Ġdifficult Ġexamples Ġare Ġrouted Ġto Ġexperts Ġand Ġother Ġinstances Ġto Ġcrowd Ġannot ators Ġyields Ġthe Ġbest Ġresults , Ġoutper forming Ġthe Ġstrategy Ġof Ġrandomly Ġselecting Ġdata Ġfor Ġexpert Ġannotation , Ġand Ġsubstantially Ġimproving Ġupon Ġthe Ġapproach Ġof Ġrelying Ġexclusively Ġon Ġcrowd Ġannotations . ĠIn Ġfuture Ġwork , Ġrouting Ġstrategies Ġbased Ġon Ġinstance Ġdifficulty Ġcould Ġbe Ġfurther Ġinvestigated Ġfor Ġbudget - quality Ġtrade - off . </s> Acknowled gements </s> This Ġwork Ġhas Ġbeen Ġpartially Ġsupported Ġby ĠNS F 17 48 771 Ġgrant . ĠWallace Ġwas Ġsupport Ġin Ġpart Ġby ĠNIH / N LM ĠR 01 LM 01 20 86 .