BF, BA, SFU and Sherlock
EXTRACTIVE
BioScope Abstracts, SFU, and BioScope Full Papers
ABSTRACTIVE
We use a default train-validation-test split of 70-15-15 for each dataset, and use all 4 datasets (BF, BA, SFU and Sherlock). The results for BERT are taken from BIBREF12. The results for XLNet and RoBERTa are averaged across 5 runs for statistical significance. Figure FIGREF14 contains results for negation cue detection and scope resolution. We report state-of-the-art results on negation scope resolution on BF, BA and SFU datasets. Contrary to popular opinion, we observe that XLNet is better than RoBERTa for the cue detection and scope resolution tasks. A few possible reasons for this trend are:
FLOAT SELECTED: Fig. 5: Results for Joint Training on Multiple Datasets
2001.02885
Introduction </s> The Ġtask Ġof Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġis Ġcritical Ġin Ġdistinguishing Ġfactual Ġinformation Ġfrom Ġspeculative Ġinformation . ĠThis Ġhas Ġmultiple Ġuse - cases , Ġlike Ġsystems Ġthat Ġdetermine Ġthe Ġver acity Ġof Ġinformation , Ġand Ġthose Ġthat Ġinvolve Ġrequirement Ġanalysis . ĠThis Ġtask Ġis Ġparticularly Ġimportant Ġto Ġthe Ġbiomedical Ġdomain , Ġwhere Ġpatient Ġreports Ġand Ġmedical Ġarticles Ġoften Ġuse Ġthis Ġfeature Ġof Ġnatural Ġlanguage . ĠThis Ġtask Ġis Ġcommonly Ġbroken Ġdown Ġinto Ġtwo Ġsubt asks : Ġthe Ġfirst Ġsubt ask , Ġspeculation Ġcue Ġdetection , Ġis Ġto Ġidentify Ġthe Ġuncertainty Ġcue Ġin Ġa Ġsentence , Ġwhile Ġthe Ġsecond Ġsubt ask : Ġscope Ġresolution , Ġis Ġto Ġidentify Ġthe Ġscope Ġof Ġthat Ġcue . ĠFor Ġinstance , Ġconsider Ġthe Ġexample : </s> It Ġmight Ġrain Ġtomorrow . </s> The Ġspeculation Ġcue Ġin Ġthe Ġsentence Ġabove Ġis ĠâĢ ĺ might âĢ Ļ Ġand Ġthe Ġscope Ġof Ġthe Ġcue ĠâĢ ĺ might âĢ Ļ Ġis ĠâĢ ĺ rain Ġtomorrow âĢ Ļ . ĠThus , Ġthe Ġspeculation Ġcue Ġis Ġthe Ġword Ġthat Ġexpresses Ġthe Ġspeculation , Ġwhile Ġthe Ġwords Ġaffected Ġby Ġthe Ġspeculation Ġare Ġin Ġthe Ġscope Ġof Ġthat Ġcue . </s> This Ġtask Ġwas Ġthe ĠCo N LL - 2010 ĠShared ĠTask Ġ( BI B REF 0 ), Ġwhich Ġhad Ġ3 Ġdifferent Ġsubt asks . ĠTask Ġ1 B Ġwas Ġspeculation Ġcue Ġdetection Ġon Ġthe ĠBio Scope ĠCorpus , ĠTask Ġ1 W Ġwas Ġwe asel Ġidentification Ġfrom ĠWikipedia Ġarticles , Ġand ĠTask Ġ2 Ġwas Ġspeculation Ġscope Ġresolution Ġfrom Ġthe ĠBio Scope ĠCorpus . ĠFor Ġeach Ġtask , Ġthe Ġparticipants Ġwere Ġprovided Ġthe Ġtrain Ġand Ġtest Ġset , Ġwhich Ġis Ġhence forth Ġreferred Ġto Ġas ĠTask Ġ1 B ĠCo N LL Ġand ĠTask Ġ2 ĠCo N LL Ġthroughout Ġthis Ġpaper . </s> For Ġour Ġexperimentation , Ġwe Ġuse Ġthe Ġsub Ġcorpor a Ġof Ġthe ĠBio Scope ĠCorpus Ġ( BI B REF 1 ), Ġnamely Ġthe ĠBio Scope ĠAbstract s Ġsub Ġcorpor a , Ġwhich Ġis Ġreferred Ġto Ġas ĠBA , Ġand Ġthe ĠBio Scope ĠFull ĠPapers Ġsub Ġcorpor a , Ġwhich Ġis Ġreferred Ġto Ġas ĠBF . ĠWe Ġalso Ġuse Ġthe ĠSF U ĠReview ĠCorpus Ġ( BI B REF 2 ), Ġwhich Ġis Ġreferred Ġto Ġas ĠSF U . </s> This Ġsubt ask Ġof Ġnatural Ġlanguage Ġprocessing , Ġalong Ġwith Ġanother Ġsimilar Ġsubt ask , Ġneg ation Ġdetection Ġand Ġscope Ġresolution , Ġhave Ġbeen Ġthe Ġsubject Ġof Ġa Ġbody Ġof Ġwork Ġover Ġthe Ġyears . ĠThe Ġapproaches Ġused Ġto Ġsolve Ġthem Ġhave Ġevolved Ġfrom Ġsimple Ġrule - based Ġsystems Ġ( BI B REF 3 ) Ġbased Ġon Ġlinguistic Ġinformation Ġextracted Ġfrom Ġthe Ġsentences , Ġto Ġmodern Ġdeep - learning Ġbased Ġmethods . ĠThe ĠMachine ĠLearning Ġtechniques Ġused Ġvaried Ġfrom ĠMaximum ĠEnt ropy ĠClass ifiers Ġ( BI B REF 4 ) Ġto ĠSupport ĠVector ĠMachines Ġ( BI B REF 5 , BI B REF 6 , BI B REF 7 , BI B REF 8 ), Ġwhile Ġthe Ġdeep Ġlearning Ġapproaches Ġincluded ĠRec ursive ĠNeural ĠNetworks Ġ( BI B REF 9 , BI B REF 10 ), ĠConv olution al ĠNeural ĠNetworks Ġ( BI B REF 11 ) Ġand Ġmost Ġrecently Ġtransfer Ġlearning - based Ġarchitectures Ġlike ĠBid irection al ĠEnc oder ĠRepresent ation Ġfrom ĠTransformers Ġ( BER T ) Ġ( BI B REF 12 ). ĠFigures ĠFIG REF 1 Ġand ĠFIG REF 1 Ġcontain Ġa Ġsummary Ġof Ġthe Ġpapers Ġaddressing Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġ( BI B REF 13 , ĠB IB REF 5 , ĠB IB REF 9 , ĠB IB REF 3 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 6 , ĠB IB REF 11 , ĠB IB REF 18 , ĠB IB REF 10 , ĠB IB REF 19 , ĠB IB REF 7 , ĠB IB REF 4 , ĠB IB REF 8 ). </s> Insp ired Ġby Ġthe Ġmost Ġrecent Ġapproach Ġof Ġapplying ĠB ERT Ġto Ġneg ation Ġdetection Ġand Ġscope Ġresolution Ġ( BI B REF 12 ), Ġwe Ġtake Ġthis Ġapproach Ġone Ġstep Ġfurther Ġby Ġperforming Ġa Ġcomparative Ġanalysis Ġof Ġthree Ġpopular Ġtransformer - based Ġarchitectures : ĠB ERT Ġ( BI B REF 20 ), ĠXL Net Ġ( BI B REF 21 ) Ġand ĠRo BER Ta Ġ( BI B REF 22 ), Ġapplied Ġto Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠWe Ġevaluate Ġthe Ġperformance Ġof Ġeach Ġmodel Ġacross Ġall Ġdatasets Ġvia Ġthe Ġsingle Ġdataset Ġtraining Ġapproach , Ġand Ġreport Ġall Ġscores Ġincluding Ġinter - dat as et Ġscores Ġ( i . e . Ġtrain Ġon Ġone Ġdataset , Ġevaluate Ġon Ġanother ) Ġto Ġtest Ġthe Ġgeneral iz ability Ġof Ġthe Ġmodels . ĠThis Ġapproach Ġoutper forms Ġall Ġexisting Ġsystems Ġon Ġthe Ġtask Ġof Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠFurther , Ġwe Ġjointly Ġtrain Ġon Ġmultiple Ġdatasets Ġand Ġobtain Ġimprovements Ġover Ġthe Ġsingle Ġdataset Ġtraining Ġapproach Ġon Ġmost Ġdatasets . </s> Cont rary Ġto Ġresults Ġobserved Ġon Ġbenchmark ĠGL UE Ġtasks , Ġwe Ġobserve ĠXL Net Ġconsistently Ġoutper forming ĠRo BER Ta . ĠTo Ġconfirm Ġthis Ġobservation , Ġwe Ġapply Ġthese Ġmodels Ġto Ġthe Ġneg ation Ġdetection Ġand Ġscope Ġresolution Ġtask , Ġand Ġobserve Ġa Ġcontinuity Ġin Ġthis Ġtrend , Ġreporting Ġstate - of - the - art Ġresults Ġon Ġthree Ġof Ġfour Ġdatasets Ġon Ġthe Ġneg ation Ġscope Ġresolution Ġtask . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows : ĠIn ĠSection Ġ2 , Ġwe Ġprovide Ġa Ġdetailed Ġdescription Ġof Ġour Ġmethodology Ġand Ġelaborate Ġon Ġthe Ġexperimentation Ġdetails . ĠIn ĠSection Ġ3 , Ġwe Ġpresent Ġour Ġresults Ġand Ġanalysis Ġon Ġthe Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġtask , Ġusing Ġthe Ġsingle Ġdataset Ġand Ġthe Ġmultiple Ġdataset Ġtraining Ġapproach . ĠIn ĠSection Ġ4 , Ġwe Ġshow Ġthe Ġresults Ġof Ġapplying ĠXL Net Ġand ĠRo BER Ta Ġon Ġneg ation Ġdetection Ġand Ġscope Ġresolution Ġand Ġpropose Ġa Ġfew Ġreasons Ġto Ġexplain Ġwhy ĠXL Net Ġperforms Ġbetter Ġthan ĠRo BER Ta . ĠFinally , Ġthe Ġfuture Ġscope Ġand Ġconclusion Ġis Ġmentioned Ġin ĠSection Ġ5 . </s> Method ology Ġand ĠExperimental ĠSetup </s> We Ġuse Ġthe Ġmethodology Ġby ĠK hand el wal Ġand ĠSaw ant Ġ( BI B REF 12 ), Ġand Ġmodify Ġit Ġto Ġsupport Ġexperimentation Ġwith Ġmultiple Ġmodels . </s> For ĠSpe culation ĠCue ĠDetection : </s> Input ĠSent ence : ĠIt Ġmight Ġrain Ġtomorrow . </s> True ĠLab els : ĠNot - A - C ue , ĠCue , ĠNot - A - C ue , ĠNot - A - C ue . </s> First , Ġthis Ġsentence Ġis Ġpre process ed Ġto Ġget Ġthe Ġtarget Ġlabels Ġas Ġper Ġthe Ġfollowing Ġannotation Ġschema : </s> 1 ĠâĢĵ ĠNormal ĠCue Ġ2 ĠâĢĵ ĠMulti word ĠCue Ġ3 ĠâĢĵ ĠNot Ġa Ġcue Ġ4 ĠâĢĵ ĠPad Ġtoken </s> Thus , Ġthe Ġpre process ed Ġsequence Ġis Ġas Ġfollows : </s> Input ĠSent ence : Ġ[ It , Ġmight , Ġrain , Ġtomorrow ] </s> True ĠLab els : Ġ[ 3 , 1 , 3 , 3 ] </s> Then , Ġwe Ġpre process Ġthe Ġinput Ġusing Ġthe Ġtoken izer Ġfor Ġthe Ġmodel Ġbeing Ġused Ġ( BER T , ĠXL Net Ġor ĠRo BER Ta ): Ġsplitting Ġeach Ġword Ġinto Ġone Ġor Ġmore Ġtokens , Ġand Ġconverting Ġeach Ġtoken Ġto Ġit âĢ Ļ s Ġcorresponding Ġtoken ID , Ġand Ġpadding Ġit Ġto Ġthe Ġmaximum Ġinput Ġlength Ġof Ġthe Ġmodel . ĠThus , </s> Input ĠSent ence : Ġ[ w tt ( It ), Ġw tt ( might ), Ġw tt ( rain ), Ġw tt ( tom ), Ġw tt ( ## Ġor ), Ġw tt ( ## Ġrow ), Ġw tt ( ãĢ Ī pad Ġ ãĢ ī ), w tt ( ãĢ Ī pad Ġ ãĢ ī ) ...] </s> True ĠLab els : Ġ[ 3 , 1 , 3 , 3 , 3 , 3 , 4 , 4 , 4 , 4 , ...] </s> The Ġword ĠâĢ ĺ tom orrow ' Ġhas Ġbeen Ġsplit Ġinto Ġ3 Ġtokens , ĠâĢ ĺ tom ', ĠâĢ ĺ ## or ' Ġand ĠâĢ ĺ ## row '. ĠThe Ġfunction Ġto Ġconvert Ġthe Ġword Ġto Ġtoken ID Ġis Ġrepresented Ġby Ġw tt . </s> For ĠSpe culation ĠScope ĠResolution : </s> If Ġa Ġsentence Ġhas Ġmultiple Ġcues , Ġeach Ġcue 's Ġscope Ġwill Ġbe Ġresolved Ġindividually . </s> Input ĠSent ence : ĠIt Ġmight Ġrain Ġtomorrow . </s> True ĠLab els : ĠOut - Of - Scope , ĠOut - Of - Scope , ĠIn - Scope , ĠIn - Scope . </s> First , Ġthis Ġsentence Ġis Ġpre process ed Ġto Ġget Ġthe Ġtarget Ġlabels Ġas Ġper Ġthe Ġfollowing Ġannotation Ġschema : </s> 0 ĠâĢĵ ĠOut - Of - Scope Ġ1 ĠâĢĵ ĠIn - Scope </s> Thus , Ġthe Ġpre process ed Ġsequence Ġis Ġas Ġfollows : </s> True ĠScope ĠLab els : Ġ[ 0 , 0 , 1 , 1 ] </s> As Ġfor Ġcue Ġdetection , Ġwe Ġpre process Ġthe Ġinput Ġusing Ġthe Ġtoken izer Ġfor Ġthe Ġmodel Ġbeing Ġused . ĠAdditionally , Ġwe Ġneed Ġto Ġindicate Ġwhich Ġcue 's Ġscope Ġwe Ġwant Ġto Ġfind Ġin Ġthe Ġinput Ġsentence . ĠWe Ġdo Ġthis Ġby Ġinserting Ġa Ġspecial Ġtoken Ġrepresenting Ġthe Ġtoken Ġtype Ġ( according Ġto Ġthe Ġcue Ġdetection Ġannotation Ġschema ) Ġbefore Ġthe Ġcue Ġword Ġwhose Ġscope Ġis Ġbeing Ġresolved . ĠHere , Ġwe Ġwant Ġto Ġfind Ġthe Ġscope Ġof Ġthe Ġcue ĠâĢ ĺ might '. ĠThus , </s> Input ĠSent ence : Ġ[ w tt ( It ), Ġw tt ( ãĢ Ī token [ 1 ] ãĢ ī ), Ġw tt ( might ), Ġw tt ( rain ), Ġw tt ( tom ), Ġw tt ( ## or ), Ġw tt ( ## row ), Ġw tt ( ãĢ Ī pad ãĢ ī ), Ġw tt ( ãĢ Ī pad ãĢ ī ) ...] </s> True ĠScope ĠLab els : Ġ[ 0 , 0 , 1 , 1 , 1 , 1 , 0 , 0 , 0 , 0 , ...] </s> Now , Ġthe Ġpre process ed Ġinput Ġfor Ġcue Ġdetection Ġand Ġsimilarly Ġfor Ġscope Ġdetection Ġis Ġfed Ġas Ġinput Ġto Ġour Ġmodel Ġas Ġfollows : </s> X Ġ= ĠModel Ġ( Input ) </s> Y Ġ= ĠW * X Ġ+ Ġb </s> The ĠW Ġmatrix Ġis Ġa Ġmatrix Ġof Ġsize Ġn _ hidden Ġx Ġnum _ classes Ġ( n _ hidden Ġis Ġthe Ġsize Ġof Ġthe Ġrepresentation Ġof Ġa Ġtoken Ġwithin Ġthe Ġmodel ). ĠThese Ġlog its Ġare Ġfed Ġto Ġthe Ġloss Ġfunction . </s> We Ġuse Ġthe Ġfollowing Ġvariants Ġof Ġeach Ġmodel : </s> BER T : Ġb ert - base - unc ased s 3 . amazon aws . com / models . h ug ging face . co / bert / bert - base - unc ased . tar . gz Ġ( The Ġmodel Ġused Ġby ĠB IB REF 12 ) </s> Ro BER Ta : Ġro bert a - b ases 3 . amazon aws . com / models . h ug ging face . co / bert / ro bert a - base - py tor ch _ model . bin Ġ( Ro BER Ta - base Ġdoes Ġnot Ġhave Ġan Ġunc ased Ġvariant ) </s> XL Net : Ġx l net - base - c ased s 3 . amazon aws . com / models . h ug ging face . co / bert / x l net - base - c ased - py tor ch _ model . bin Ġ( XL Net - base Ġdoes Ġnot Ġhave Ġan Ġunc ased Ġvariant ) </s> The Ġoutput Ġof Ġthe Ġmodel Ġis Ġa Ġvector Ġof Ġprobabilities Ġper Ġtoken . ĠThe Ġloss Ġis Ġcalculated Ġfor Ġeach Ġtoken , Ġby Ġusing Ġthe Ġoutput Ġvector Ġand Ġthe Ġtrue Ġlabel Ġfor Ġthat Ġtoken . ĠWe Ġuse Ġclass Ġweights Ġfor Ġthe Ġloss Ġfunction , Ġsetting Ġthe Ġweight Ġfor Ġlabel Ġ4 Ġto Ġ0 Ġand Ġall Ġother Ġlabels Ġto Ġ1 Ġ( for Ġcue Ġdetection Ġonly ) Ġto Ġavoid Ġtraining Ġon Ġpadding Ġtoken âĢ Ļ s Ġoutput . </s> We Ġcalculate Ġthe Ġscores Ġ( Pre cision , ĠRecall , ĠF 1 ) Ġfor Ġthe Ġmodel Ġper Ġword Ġof Ġthe Ġinput Ġsentence , Ġnot Ġper Ġtoken Ġthat Ġwas Ġfed Ġto Ġthe Ġmodel , Ġas Ġthe Ġtokens Ġcould Ġbe Ġdifferent Ġfor Ġdifferent Ġmodels Ġleading Ġto Ġinaccurate Ġscores . ĠFor Ġthe Ġabove Ġexample , Ġwe Ġcalculate Ġthe Ġoutput Ġlabel Ġfor Ġthe Ġword ĠâĢ ĺ tom orrow ', Ġnot Ġfor Ġeach Ġtoken Ġit Ġwas Ġsplit Ġinto Ġ( âĢ ĺ tom ', ĠâĢ ĺ ## or ' Ġand ĠâĢ ĺ ## row '). ĠTo Ġfind Ġthe Ġlabel Ġfor Ġeach Ġword Ġfrom Ġthe Ġtokens Ġit Ġwas Ġsplit Ġinto , Ġwe Ġexperiment Ġwith Ġ2 Ġmethods : </s> Average : ĠWe Ġaverage Ġthe Ġoutput Ġvectors Ġ( soft max Ġprobabilities ) Ġfor Ġeach Ġtoken Ġthat Ġthe Ġword Ġwas Ġsplit Ġinto Ġby Ġthe Ġmodel 's Ġtoken izer . ĠIn Ġthe Ġexample Ġabove , Ġwe Ġaverage Ġthe Ġoutput Ġof ĠâĢ ĺ tom ', ĠâĢ ĺ ## or ' Ġand ĠâĢ ĺ ## row ' Ġto Ġget Ġthe Ġoutput Ġfor ĠâĢ ĺ tom orrow '. ĠThen , Ġwe Ġtake Ġan Ġarg max Ġover Ġthe Ġresultant Ġvector . ĠThis Ġis Ġthen Ġcompared Ġwith Ġthe Ġtrue Ġlabel Ġfor Ġthe Ġoriginal Ġword . </s> First ĠToken : ĠHere , Ġwe Ġonly Ġconsider Ġthe Ġfirst Ġtoken 's Ġprobability Ġvector Ġ( among Ġall Ġtokens Ġthe Ġword Ġwas Ġsplit Ġinto ) Ġas Ġthe Ġoutput Ġfor Ġthat Ġword , Ġand Ġget Ġthe Ġlabel Ġby Ġan Ġarg max Ġover Ġthis Ġvector . ĠIn Ġthe Ġexample Ġabove , Ġwe Ġwould Ġconsider Ġthe Ġoutput Ġvector Ġcorresponding Ġto Ġthe Ġtoken ĠâĢ ĺ tom ' Ġas Ġthe Ġoutput Ġfor Ġthe Ġword ĠâĢ ĺ tom orrow '. </s> For Ġcue Ġdetection , Ġthe Ġresults Ġare Ġreported Ġfor Ġthe ĠAverage Ġmethod Ġonly , Ġwhile Ġwe Ġreport Ġthe Ġscores Ġfor Ġboth ĠAverage Ġand ĠFirst ĠToken Ġfor ĠScope ĠResolution . </s> For Ġfair Ġcomparison , Ġwe Ġuse Ġthe Ġsame Ġhyper param eters Ġfor Ġthe Ġentire Ġarchitecture Ġfor Ġall Ġ3 Ġmodels . ĠOnly Ġthe Ġtoken izer Ġand Ġthe Ġmodel Ġare Ġchanged Ġfor Ġeach Ġmodel . ĠAll Ġother Ġhyper param eters Ġare Ġkept Ġsame . ĠWe Ġfin et une Ġthe Ġmodels Ġfor Ġ60 Ġepoch s , Ġusing Ġearly Ġstopping Ġwith Ġa Ġpatience Ġof Ġ6 Ġon Ġthe ĠF 1 Ġscore Ġ( word Ġlevel ) Ġon Ġthe Ġvalidation Ġdataset . ĠWe Ġuse Ġan Ġinitial Ġlearning Ġrate Ġof Ġ3 e - 5 , Ġwith Ġa Ġbatch Ġsize Ġof Ġ8 . ĠWe Ġuse Ġthe ĠC ateg orical ĠCross ĠEnt ropy Ġloss Ġfunction . </s> We Ġuse Ġthe ĠHug ging face âĢ Ļ s ĠPy tor ch ĠTrans former Ġlibrary Ġ( BI B REF 23 ) Ġfor Ġthe Ġmodels Ġand Ġtrain Ġall Ġour Ġmodels Ġon ĠGoogle ĠCol abor atory . </s> Results : ĠSpe culation ĠCue ĠDetection Ġand ĠScope ĠResolution </s> We Ġuse Ġa Ġdefault Ġtrain - valid ation - test Ġsplit Ġof Ġ70 - 15 - 15 Ġfor Ġeach Ġdataset . ĠFor Ġthe Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġsubt asks Ġusing Ġsingle - dat as et Ġtraining , Ġwe Ġreport Ġthe Ġresults Ġas Ġan Ġaverage Ġof Ġ5 Ġruns Ġof Ġthe Ġmodel . ĠFor Ġtraining Ġthe Ġmodel Ġon Ġmultiple Ġdatasets , Ġwe Ġperform Ġa Ġ70 - 15 - 15 Ġsplit Ġof Ġeach Ġtraining Ġdataset , Ġafter Ġwhich Ġthe Ġtrain Ġand Ġvalidation Ġpart Ġof Ġthe Ġindividual Ġdatasets Ġare Ġmerged Ġwhile Ġthe Ġscores Ġare Ġreported Ġfor Ġthe Ġtest Ġpart Ġof Ġthe Ġindividual Ġdatasets , Ġwhich Ġis Ġnot Ġused Ġfor Ġtraining Ġor Ġvalidation . ĠWe Ġreport Ġthe Ġresults Ġas Ġan Ġaverage Ġof Ġ3 Ġruns Ġof Ġthe Ġmodel . ĠFigure ĠFIG REF 8 Ġcontains Ġresults Ġfor Ġspeculation Ġcue Ġdetection Ġand Ġscope Ġresolution Ġwhen Ġtrained Ġon Ġa Ġsingle Ġdataset . ĠAll Ġmodels Ġperform Ġthe Ġbest Ġwhen Ġtrained Ġon Ġthe Ġsame Ġdataset Ġas Ġthey Ġare Ġevaluated Ġon , Ġexcept Ġfor ĠBF , Ġwhich Ġgets Ġthe Ġbest Ġresults Ġwhen Ġtrained Ġon ĠBA . ĠThis Ġis Ġbecause Ġof Ġthe Ġtransfer Ġlearning Ġcapabilities Ġof Ġthe Ġmodels Ġand Ġthe Ġfact Ġthat ĠBF Ġis Ġa Ġsmaller Ġdataset Ġthan ĠBA Ġ( BF : Ġ26 70 Ġsentences , ĠBA : Ġ118 71 Ġsentences ). ĠFor Ġspeculation Ġcue Ġdetection , Ġthere Ġis Ġlesser Ġgeneral iz ability Ġfor Ġmodels Ġtrained Ġon ĠBF Ġor ĠBA , Ġwhile Ġthere Ġis Ġmore Ġgeneral iz ability Ġfor Ġmodels Ġtrained Ġon ĠSF U . ĠThis Ġcould Ġbe Ġbecause Ġof Ġthe Ġdifferent Ġnature Ġof Ġthe Ġbiomedical Ġdomain . </s> Figure ĠFIG REF 11 Ġcontains Ġthe Ġresults Ġfor Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġfor Ġmodels Ġtrained Ġjointly Ġon Ġmultiple Ġdatasets . ĠWe Ġobserve Ġthat Ġtraining Ġon Ġmultiple Ġdatasets Ġhelps Ġthe Ġperformance Ġof Ġall Ġmodels Ġon Ġeach Ġdataset , Ġas Ġthe Ġquantity Ġof Ġdata Ġavailable Ġto Ġtrain Ġthe Ġmodel Ġincreases . ĠWe Ġalso Ġobserve Ġthat ĠXL Net Ġconsistently Ġoutper forms ĠB ERT Ġand ĠRo BER Ta . ĠTo Ġconfirm Ġthis Ġobservation , Ġwe Ġapply Ġthe Ġ2 Ġmodels Ġto Ġthe Ġrelated Ġtask Ġof Ġneg ation Ġdetection Ġand Ġscope Ġresolution </s> Neg ation ĠCue ĠDetection Ġand ĠScope ĠResolution </s> We Ġuse Ġa Ġdefault Ġtrain - valid ation - test Ġsplit Ġof Ġ70 - 15 - 15 Ġfor Ġeach Ġdataset , Ġand Ġuse Ġall Ġ4 Ġdatasets Ġ( BF , ĠBA , ĠSF U Ġand ĠSherlock ). ĠThe Ġresults Ġfor ĠB ERT Ġare Ġtaken Ġfrom ĠB IB REF 12 . ĠThe Ġresults Ġfor ĠXL Net Ġand ĠRo BER Ta Ġare Ġaveraged Ġacross Ġ5 Ġruns Ġfor Ġstatistical Ġsignificance . ĠFigure ĠFIG REF 14 Ġcontains Ġresults Ġfor Ġneg ation Ġcue Ġdetection Ġand Ġscope Ġresolution . ĠWe Ġreport Ġstate - of - the - art Ġresults Ġon Ġneg ation Ġscope Ġresolution Ġon ĠBF , ĠBA Ġand ĠSF U Ġdatasets . ĠContrary Ġto Ġpopular Ġopinion , Ġwe Ġobserve Ġthat ĠXL Net Ġis Ġbetter Ġthan ĠRo BER Ta Ġfor Ġthe Ġcue Ġdetection Ġand Ġscope Ġresolution Ġtasks . ĠA Ġfew Ġpossible Ġreasons Ġfor Ġthis Ġtrend Ġare : </s> Domain Ġspecificity , Ġas Ġboth Ġneg ation Ġand Ġspeculation Ġare Ġclosely Ġrelated Ġsubt asks . ĠFurther Ġexperimentation Ġon Ġdifferent Ġtasks Ġis Ġneeded Ġto Ġverify Ġthis . </s> Most Ġbenchmark Ġtasks Ġare Ġsentence Ġclassification Ġtasks , Ġwhereas Ġthe Ġsubt asks Ġwe Ġexperiment Ġon Ġare Ġsequence Ġlab elling Ġtasks . ĠGiven Ġthe Ġpre - training Ġobjective Ġof ĠXL Net Ġ( training Ġon Ġperm utations Ġof Ġthe Ġinput ), Ġit Ġmay Ġbe Ġable Ġto Ġcapture Ġlong - term Ġdependencies Ġbetter , Ġessential Ġfor Ġsequence Ġlab elling Ġtasks . </s> We Ġwork Ġwith Ġthe Ġbase Ġvariants Ġof Ġthe Ġmodels , Ġwhile Ġmost Ġresults Ġare Ġreported Ġwith Ġthe Ġlarge Ġvariants Ġof Ġthe Ġmodels . </s> Conclusion Ġand ĠFuture ĠScope </s> In Ġthis Ġpaper , Ġwe Ġexpanded Ġon Ġthe Ġwork Ġof ĠK hand el wal Ġand ĠSaw ant Ġ( BI B REF 12 ) Ġby Ġlooking Ġat Ġalternative Ġtransfer - learning Ġmodels Ġand Ġexperimented Ġwith Ġtraining Ġon Ġmultiple Ġdatasets . ĠOn Ġthe Ġspeculation Ġdetection Ġtask , Ġwe Ġobtained Ġa Ġgain Ġof Ġ0 . 42 ĠF 1 Ġpoints Ġon ĠBF , Ġ1 . 98 ĠF 1 Ġpoints Ġon ĠBA Ġand Ġ0 . 29 ĠF 1 Ġpoints Ġon ĠSF U , Ġwhile Ġon Ġthe Ġscope Ġresolution Ġtask , Ġwe Ġobtained Ġa Ġgain Ġof Ġ8 . 06 ĠF 1 Ġpoints Ġon ĠBF , Ġ4 . 27 ĠF 1 Ġpoints Ġon ĠBA Ġand Ġ11 . 87 ĠF 1 Ġpoints Ġon ĠSF U , Ġwhen Ġtrained Ġon Ġa Ġsingle Ġdataset . ĠWhile Ġtraining Ġon Ġmultiple Ġdatasets , Ġwe Ġobserved Ġa Ġgain Ġof Ġ10 . 6 ĠF 1 Ġpoints Ġon ĠBF Ġand Ġ1 . 94 ĠF 1 Ġpoints Ġon ĠBA Ġon Ġthe Ġspeculation Ġdetection Ġtask Ġand Ġ2 . 16 ĠF 1 Ġpoints Ġon ĠBF Ġand Ġ0 . 25 ĠF 1 Ġpoints Ġon ĠSF U Ġon Ġthe Ġscope Ġresolution Ġtask Ġover Ġthe Ġsingle Ġdataset Ġtraining Ġapproach . ĠWe Ġthus Ġsignificantly Ġadvance Ġthe Ġstate - of - the - art Ġfor Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠOn Ġthe Ġneg ation Ġscope Ġresolution Ġtask , Ġwe Ġapplied Ġthe ĠXL Net Ġand ĠRo BER Ta Ġand Ġobtained Ġa Ġgain Ġof Ġ3 . 16 ĠF 1 Ġpoints Ġon ĠBF , Ġ0 . 06 ĠF 1 Ġpoints Ġon ĠBA Ġand Ġ0 . 3 ĠF 1 Ġpoints Ġon ĠSF U . ĠThus , Ġwe Ġdemonstrated Ġthe Ġusefulness Ġof Ġtransformer - based Ġarchitectures Ġin Ġthe Ġfield Ġof Ġneg ation Ġand Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠWe Ġbelieve Ġthat Ġa Ġlarger Ġand Ġmore Ġgeneral Ġdataset Ġwould Ġgo Ġa Ġlong Ġway Ġin Ġbolst ering Ġfuture Ġresearch Ġand Ġwould Ġhelp Ġcreate Ġbetter Ġsystems Ġthat Ġare Ġnot Ġdomain - specific .
Introduction The task of speculation detection and scope resolution is critical in distinguishing factual information from speculative information. This has multiple use-cases, like systems that determine the veracity of information, and those that involve requirement analysis. This task is particularly important to the biomedical domain, where patient reports and medical articles often use this feature of natural language. This task is commonly broken down into two subtasks: the first subtask, speculation cue detection, is to identify the uncertainty cue in a sentence, while the second subtask: scope resolution, is to identify the scope of that cue. For instance, consider the example: It might rain tomorrow. The speculation cue in the sentence above is ‘might’ and the scope of the cue ‘might’ is ‘rain tomorrow’. Thus, the speculation cue is the word that expresses the speculation, while the words affected by the speculation are in the scope of that cue. This task was the CoNLL-2010 Shared Task (BIBREF0), which had 3 different subtasks. Task 1B was speculation cue detection on the BioScope Corpus, Task 1W was weasel identification from Wikipedia articles, and Task 2 was speculation scope resolution from the BioScope Corpus. For each task, the participants were provided the train and test set, which is henceforth referred to as Task 1B CoNLL and Task 2 CoNLL throughout this paper. For our experimentation, we use the sub corpora of the BioScope Corpus (BIBREF1), namely the BioScope Abstracts sub corpora, which is referred to as BA, and the BioScope Full Papers sub corpora, which is referred to as BF. We also use the SFU Review Corpus (BIBREF2), which is referred to as SFU. This subtask of natural language processing, along with another similar subtask, negation detection and scope resolution, have been the subject of a body of work over the years. The approaches used to solve them have evolved from simple rule-based systems (BIBREF3) based on linguistic information extracted from the sentences, to modern deep-learning based methods. The Machine Learning techniques used varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), while the deep learning approaches included Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12). Figures FIGREF1 and FIGREF1 contain a summary of the papers addressing speculation detection and scope resolution (BIBREF13, BIBREF5, BIBREF9, BIBREF3, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF6, BIBREF11, BIBREF18, BIBREF10, BIBREF19, BIBREF7, BIBREF4, BIBREF8). Inspired by the most recent approach of applying BERT to negation detection and scope resolution (BIBREF12), we take this approach one step further by performing a comparative analysis of three popular transformer-based architectures: BERT (BIBREF20), XLNet (BIBREF21) and RoBERTa (BIBREF22), applied to speculation detection and scope resolution. We evaluate the performance of each model across all datasets via the single dataset training approach, and report all scores including inter-dataset scores (i.e. train on one dataset, evaluate on another) to test the generalizability of the models. This approach outperforms all existing systems on the task of speculation detection and scope resolution. Further, we jointly train on multiple datasets and obtain improvements over the single dataset training approach on most datasets. Contrary to results observed on benchmark GLUE tasks, we observe XLNet consistently outperforming RoBERTa. To confirm this observation, we apply these models to the negation detection and scope resolution task, and observe a continuity in this trend, reporting state-of-the-art results on three of four datasets on the negation scope resolution task. The rest of the paper is organized as follows: In Section 2, we provide a detailed description of our methodology and elaborate on the experimentation details. In Section 3, we present our results and analysis on the speculation detection and scope resolution task, using the single dataset and the multiple dataset training approach. In Section 4, we show the results of applying XLNet and RoBERTa on negation detection and scope resolution and propose a few reasons to explain why XLNet performs better than RoBERTa. Finally, the future scope and conclusion is mentioned in Section 5. Methodology and Experimental Setup We use the methodology by Khandelwal and Sawant (BIBREF12), and modify it to support experimentation with multiple models. For Speculation Cue Detection: Input Sentence: It might rain tomorrow. True Labels: Not-A-Cue, Cue, Not-A-Cue, Not-A-Cue. First, this sentence is preprocessed to get the target labels as per the following annotation schema: 1 – Normal Cue 2 – Multiword Cue 3 – Not a cue 4 – Pad token Thus, the preprocessed sequence is as follows: Input Sentence: [It, might, rain, tomorrow] True Labels: [3,1,3,3] Then, we preprocess the input using the tokenizer for the model being used (BERT, XLNet or RoBERTa): splitting each word into one or more tokens, and converting each token to it’s corresponding tokenID, and padding it to the maximum input length of the model. Thus, Input Sentence: [wtt(It), wtt(might), wtt(rain), wtt(tom), wtt(## or), wtt(## row), wtt(〈pad 〉),wtt(〈pad 〉)...] True Labels: [3,1,3,3,3,3,4,4,4,4,...] The word ‘tomorrow' has been split into 3 tokens, ‘tom', ‘##or' and ‘##row'. The function to convert the word to tokenID is represented by wtt. For Speculation Scope Resolution: If a sentence has multiple cues, each cue's scope will be resolved individually. Input Sentence: It might rain tomorrow. True Labels: Out-Of-Scope, Out-Of-Scope, In-Scope, In-Scope. First, this sentence is preprocessed to get the target labels as per the following annotation schema: 0 – Out-Of-Scope 1 – In-Scope Thus, the preprocessed sequence is as follows: True Scope Labels: [0,0,1,1] As for cue detection, we preprocess the input using the tokenizer for the model being used. Additionally, we need to indicate which cue's scope we want to find in the input sentence. We do this by inserting a special token representing the token type (according to the cue detection annotation schema) before the cue word whose scope is being resolved. Here, we want to find the scope of the cue ‘might'. Thus, Input Sentence: [wtt(It), wtt(〈token[1]〉), wtt(might), wtt(rain), wtt(tom), wtt(##or), wtt(##row), wtt(〈pad〉), wtt(〈pad〉)...] True Scope Labels: [0,0,1,1,1,1,0,0,0,0,...] Now, the preprocessed input for cue detection and similarly for scope detection is fed as input to our model as follows: X = Model (Input) Y = W*X + b The W matrix is a matrix of size n_hidden x num_classes (n_hidden is the size of the representation of a token within the model). These logits are fed to the loss function. We use the following variants of each model: BERT: bert-base-uncaseds3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz (The model used by BIBREF12) RoBERTa: roberta-bases3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin (RoBERTa-base does not have an uncased variant) XLNet: xlnet-base-caseds3.amazonaws.com/models.huggingface.co/bert/xlnet-base-cased-pytorch_model.bin (XLNet-base does not have an uncased variant) The output of the model is a vector of probabilities per token. The loss is calculated for each token, by using the output vector and the true label for that token. We use class weights for the loss function, setting the weight for label 4 to 0 and all other labels to 1 (for cue detection only) to avoid training on padding token’s output. We calculate the scores (Precision, Recall, F1) for the model per word of the input sentence, not per token that was fed to the model, as the tokens could be different for different models leading to inaccurate scores. For the above example, we calculate the output label for the word ‘tomorrow', not for each token it was split into (‘tom', ‘##or' and ‘##row'). To find the label for each word from the tokens it was split into, we experiment with 2 methods: Average: We average the output vectors (softmax probabilities) for each token that the word was split into by the model's tokenizer. In the example above, we average the output of ‘tom', ‘##or' and ‘##row' to get the output for ‘tomorrow'. Then, we take an argmax over the resultant vector. This is then compared with the true label for the original word. First Token: Here, we only consider the first token's probability vector (among all tokens the word was split into) as the output for that word, and get the label by an argmax over this vector. In the example above, we would consider the output vector corresponding to the token ‘tom' as the output for the word ‘tomorrow'. For cue detection, the results are reported for the Average method only, while we report the scores for both Average and First Token for Scope Resolution. For fair comparison, we use the same hyperparameters for the entire architecture for all 3 models. Only the tokenizer and the model are changed for each model. All other hyperparameters are kept same. We finetune the models for 60 epochs, using early stopping with a patience of 6 on the F1 score (word level) on the validation dataset. We use an initial learning rate of 3e-5, with a batch size of 8. We use the Categorical Cross Entropy loss function. We use the Huggingface’s Pytorch Transformer library (BIBREF23) for the models and train all our models on Google Colaboratory. Results: Speculation Cue Detection and Scope Resolution We use a default train-validation-test split of 70-15-15 for each dataset. For the speculation detection and scope resolution subtasks using single-dataset training, we report the results as an average of 5 runs of the model. For training the model on multiple datasets, we perform a 70-15-15 split of each training dataset, after which the train and validation part of the individual datasets are merged while the scores are reported for the test part of the individual datasets, which is not used for training or validation. We report the results as an average of 3 runs of the model. Figure FIGREF8 contains results for speculation cue detection and scope resolution when trained on a single dataset. All models perform the best when trained on the same dataset as they are evaluated on, except for BF, which gets the best results when trained on BA. This is because of the transfer learning capabilities of the models and the fact that BF is a smaller dataset than BA (BF: 2670 sentences, BA: 11871 sentences). For speculation cue detection, there is lesser generalizability for models trained on BF or BA, while there is more generalizability for models trained on SFU. This could be because of the different nature of the biomedical domain. Figure FIGREF11 contains the results for speculation detection and scope resolution for models trained jointly on multiple datasets. We observe that training on multiple datasets helps the performance of all models on each dataset, as the quantity of data available to train the model increases. We also observe that XLNet consistently outperforms BERT and RoBERTa. To confirm this observation, we apply the 2 models to the related task of negation detection and scope resolution Negation Cue Detection and Scope Resolution We use a default train-validation-test split of 70-15-15 for each dataset, and use all 4 datasets (BF, BA, SFU and Sherlock). The results for BERT are taken from BIBREF12. The results for XLNet and RoBERTa are averaged across 5 runs for statistical significance. Figure FIGREF14 contains results for negation cue detection and scope resolution. We report state-of-the-art results on negation scope resolution on BF, BA and SFU datasets. Contrary to popular opinion, we observe that XLNet is better than RoBERTa for the cue detection and scope resolution tasks. A few possible reasons for this trend are: Domain specificity, as both negation and speculation are closely related subtasks. Further experimentation on different tasks is needed to verify this. Most benchmark tasks are sentence classification tasks, whereas the subtasks we experiment on are sequence labelling tasks. Given the pre-training objective of XLNet (training on permutations of the input), it may be able to capture long-term dependencies better, essential for sequence labelling tasks. We work with the base variants of the models, while most results are reported with the large variants of the models. Conclusion and Future Scope In this paper, we expanded on the work of Khandelwal and Sawant (BIBREF12) by looking at alternative transfer-learning models and experimented with training on multiple datasets. On the speculation detection task, we obtained a gain of 0.42 F1 points on BF, 1.98 F1 points on BA and 0.29 F1 points on SFU, while on the scope resolution task, we obtained a gain of 8.06 F1 points on BF, 4.27 F1 points on BA and 11.87 F1 points on SFU, when trained on a single dataset. While training on multiple datasets, we observed a gain of 10.6 F1 points on BF and 1.94 F1 points on BA on the speculation detection task and 2.16 F1 points on BF and 0.25 F1 points on SFU on the scope resolution task over the single dataset training approach. We thus significantly advance the state-of-the-art for speculation detection and scope resolution. On the negation scope resolution task, we applied the XLNet and RoBERTa and obtained a gain of 3.16 F1 points on BF, 0.06 F1 points on BA and 0.3 F1 points on SFU. Thus, we demonstrated the usefulness of transformer-based architectures in the field of negation and speculation detection and scope resolution. We believe that a larger and more general dataset would go a long way in bolstering future research and would help create better systems that are not domain-specific.
Which multiple datasets did they train on during joint training?
511517efc96edcd3e91e7783821c9d6d5a6562af
Which Ġmultiple Ġdatasets Ġdid Ġthey Ġtrain Ġon Ġduring Ġjoint Ġtraining ?
BF , ĠBA , ĠSF U Ġand ĠSherlock
<s> Which Ġmultiple Ġdatasets Ġdid Ġthey Ġtrain Ġon Ġduring Ġjoint Ġtraining ? </s> Introduction </s> The Ġtask Ġof Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġis Ġcritical Ġin Ġdistinguishing Ġfactual Ġinformation Ġfrom Ġspeculative Ġinformation . ĠThis Ġhas Ġmultiple Ġuse - cases , Ġlike Ġsystems Ġthat Ġdetermine Ġthe Ġver acity Ġof Ġinformation , Ġand Ġthose Ġthat Ġinvolve Ġrequirement Ġanalysis . ĠThis Ġtask Ġis Ġparticularly Ġimportant Ġto Ġthe Ġbiomedical Ġdomain , Ġwhere Ġpatient Ġreports Ġand Ġmedical Ġarticles Ġoften Ġuse Ġthis Ġfeature Ġof Ġnatural Ġlanguage . ĠThis Ġtask Ġis Ġcommonly Ġbroken Ġdown Ġinto Ġtwo Ġsubt asks : Ġthe Ġfirst Ġsubt ask , Ġspeculation Ġcue Ġdetection , Ġis Ġto Ġidentify Ġthe Ġuncertainty Ġcue Ġin Ġa Ġsentence , Ġwhile Ġthe Ġsecond Ġsubt ask : Ġscope Ġresolution , Ġis Ġto Ġidentify Ġthe Ġscope Ġof Ġthat Ġcue . ĠFor Ġinstance , Ġconsider Ġthe Ġexample : </s> It Ġmight Ġrain Ġtomorrow . </s> The Ġspeculation Ġcue Ġin Ġthe Ġsentence Ġabove Ġis ĠâĢ ĺ might âĢ Ļ Ġand Ġthe Ġscope Ġof Ġthe Ġcue ĠâĢ ĺ might âĢ Ļ Ġis ĠâĢ ĺ rain Ġtomorrow âĢ Ļ . ĠThus , Ġthe Ġspeculation Ġcue Ġis Ġthe Ġword Ġthat Ġexpresses Ġthe Ġspeculation , Ġwhile Ġthe Ġwords Ġaffected Ġby Ġthe Ġspeculation Ġare Ġin Ġthe Ġscope Ġof Ġthat Ġcue . </s> This Ġtask Ġwas Ġthe ĠCo N LL - 2010 ĠShared ĠTask Ġ( BI B REF 0 ), Ġwhich Ġhad Ġ3 Ġdifferent Ġsubt asks . ĠTask Ġ1 B Ġwas Ġspeculation Ġcue Ġdetection Ġon Ġthe ĠBio Scope ĠCorpus , ĠTask Ġ1 W Ġwas Ġwe asel Ġidentification Ġfrom ĠWikipedia Ġarticles , Ġand ĠTask Ġ2 Ġwas Ġspeculation Ġscope Ġresolution Ġfrom Ġthe ĠBio Scope ĠCorpus . ĠFor Ġeach Ġtask , Ġthe Ġparticipants Ġwere Ġprovided Ġthe Ġtrain Ġand Ġtest Ġset , Ġwhich Ġis Ġhence forth Ġreferred Ġto Ġas ĠTask Ġ1 B ĠCo N LL Ġand ĠTask Ġ2 ĠCo N LL Ġthroughout Ġthis Ġpaper . </s> For Ġour Ġexperimentation , Ġwe Ġuse Ġthe Ġsub Ġcorpor a Ġof Ġthe ĠBio Scope ĠCorpus Ġ( BI B REF 1 ), Ġnamely Ġthe ĠBio Scope ĠAbstract s Ġsub Ġcorpor a , Ġwhich Ġis Ġreferred Ġto Ġas ĠBA , Ġand Ġthe ĠBio Scope ĠFull ĠPapers Ġsub Ġcorpor a , Ġwhich Ġis Ġreferred Ġto Ġas ĠBF . ĠWe Ġalso Ġuse Ġthe ĠSF U ĠReview ĠCorpus Ġ( BI B REF 2 ), Ġwhich Ġis Ġreferred Ġto Ġas ĠSF U . </s> This Ġsubt ask Ġof Ġnatural Ġlanguage Ġprocessing , Ġalong Ġwith Ġanother Ġsimilar Ġsubt ask , Ġneg ation Ġdetection Ġand Ġscope Ġresolution , Ġhave Ġbeen Ġthe Ġsubject Ġof Ġa Ġbody Ġof Ġwork Ġover Ġthe Ġyears . ĠThe Ġapproaches Ġused Ġto Ġsolve Ġthem Ġhave Ġevolved Ġfrom Ġsimple Ġrule - based Ġsystems Ġ( BI B REF 3 ) Ġbased Ġon Ġlinguistic Ġinformation Ġextracted Ġfrom Ġthe Ġsentences , Ġto Ġmodern Ġdeep - learning Ġbased Ġmethods . ĠThe ĠMachine ĠLearning Ġtechniques Ġused Ġvaried Ġfrom ĠMaximum ĠEnt ropy ĠClass ifiers Ġ( BI B REF 4 ) Ġto ĠSupport ĠVector ĠMachines Ġ( BI B REF 5 , BI B REF 6 , BI B REF 7 , BI B REF 8 ), Ġwhile Ġthe Ġdeep Ġlearning Ġapproaches Ġincluded ĠRec ursive ĠNeural ĠNetworks Ġ( BI B REF 9 , BI B REF 10 ), ĠConv olution al ĠNeural ĠNetworks Ġ( BI B REF 11 ) Ġand Ġmost Ġrecently Ġtransfer Ġlearning - based Ġarchitectures Ġlike ĠBid irection al ĠEnc oder ĠRepresent ation Ġfrom ĠTransformers Ġ( BER T ) Ġ( BI B REF 12 ). ĠFigures ĠFIG REF 1 Ġand ĠFIG REF 1 Ġcontain Ġa Ġsummary Ġof Ġthe Ġpapers Ġaddressing Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġ( BI B REF 13 , ĠB IB REF 5 , ĠB IB REF 9 , ĠB IB REF 3 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 6 , ĠB IB REF 11 , ĠB IB REF 18 , ĠB IB REF 10 , ĠB IB REF 19 , ĠB IB REF 7 , ĠB IB REF 4 , ĠB IB REF 8 ). </s> Insp ired Ġby Ġthe Ġmost Ġrecent Ġapproach Ġof Ġapplying ĠB ERT Ġto Ġneg ation Ġdetection Ġand Ġscope Ġresolution Ġ( BI B REF 12 ), Ġwe Ġtake Ġthis Ġapproach Ġone Ġstep Ġfurther Ġby Ġperforming Ġa Ġcomparative Ġanalysis Ġof Ġthree Ġpopular Ġtransformer - based Ġarchitectures : ĠB ERT Ġ( BI B REF 20 ), ĠXL Net Ġ( BI B REF 21 ) Ġand ĠRo BER Ta Ġ( BI B REF 22 ), Ġapplied Ġto Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠWe Ġevaluate Ġthe Ġperformance Ġof Ġeach Ġmodel Ġacross Ġall Ġdatasets Ġvia Ġthe Ġsingle Ġdataset Ġtraining Ġapproach , Ġand Ġreport Ġall Ġscores Ġincluding Ġinter - dat as et Ġscores Ġ( i . e . Ġtrain Ġon Ġone Ġdataset , Ġevaluate Ġon Ġanother ) Ġto Ġtest Ġthe Ġgeneral iz ability Ġof Ġthe Ġmodels . ĠThis Ġapproach Ġoutper forms Ġall Ġexisting Ġsystems Ġon Ġthe Ġtask Ġof Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠFurther , Ġwe Ġjointly Ġtrain Ġon Ġmultiple Ġdatasets Ġand Ġobtain Ġimprovements Ġover Ġthe Ġsingle Ġdataset Ġtraining Ġapproach Ġon Ġmost Ġdatasets . </s> Cont rary Ġto Ġresults Ġobserved Ġon Ġbenchmark ĠGL UE Ġtasks , Ġwe Ġobserve ĠXL Net Ġconsistently Ġoutper forming ĠRo BER Ta . ĠTo Ġconfirm Ġthis Ġobservation , Ġwe Ġapply Ġthese Ġmodels Ġto Ġthe Ġneg ation Ġdetection Ġand Ġscope Ġresolution Ġtask , Ġand Ġobserve Ġa Ġcontinuity Ġin Ġthis Ġtrend , Ġreporting Ġstate - of - the - art Ġresults Ġon Ġthree Ġof Ġfour Ġdatasets Ġon Ġthe Ġneg ation Ġscope Ġresolution Ġtask . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows : ĠIn ĠSection Ġ2 , Ġwe Ġprovide Ġa Ġdetailed Ġdescription Ġof Ġour Ġmethodology Ġand Ġelaborate Ġon Ġthe Ġexperimentation Ġdetails . ĠIn ĠSection Ġ3 , Ġwe Ġpresent Ġour Ġresults Ġand Ġanalysis Ġon Ġthe Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġtask , Ġusing Ġthe Ġsingle Ġdataset Ġand Ġthe Ġmultiple Ġdataset Ġtraining Ġapproach . ĠIn ĠSection Ġ4 , Ġwe Ġshow Ġthe Ġresults Ġof Ġapplying ĠXL Net Ġand ĠRo BER Ta Ġon Ġneg ation Ġdetection Ġand Ġscope Ġresolution Ġand Ġpropose Ġa Ġfew Ġreasons Ġto Ġexplain Ġwhy ĠXL Net Ġperforms Ġbetter Ġthan ĠRo BER Ta . ĠFinally , Ġthe Ġfuture Ġscope Ġand Ġconclusion Ġis Ġmentioned Ġin ĠSection Ġ5 . </s> Method ology Ġand ĠExperimental ĠSetup </s> We Ġuse Ġthe Ġmethodology Ġby ĠK hand el wal Ġand ĠSaw ant Ġ( BI B REF 12 ), Ġand Ġmodify Ġit Ġto Ġsupport Ġexperimentation Ġwith Ġmultiple Ġmodels . </s> For ĠSpe culation ĠCue ĠDetection : </s> Input ĠSent ence : ĠIt Ġmight Ġrain Ġtomorrow . </s> True ĠLab els : ĠNot - A - C ue , ĠCue , ĠNot - A - C ue , ĠNot - A - C ue . </s> First , Ġthis Ġsentence Ġis Ġpre process ed Ġto Ġget Ġthe Ġtarget Ġlabels Ġas Ġper Ġthe Ġfollowing Ġannotation Ġschema : </s> 1 ĠâĢĵ ĠNormal ĠCue Ġ2 ĠâĢĵ ĠMulti word ĠCue Ġ3 ĠâĢĵ ĠNot Ġa Ġcue Ġ4 ĠâĢĵ ĠPad Ġtoken </s> Thus , Ġthe Ġpre process ed Ġsequence Ġis Ġas Ġfollows : </s> Input ĠSent ence : Ġ[ It , Ġmight , Ġrain , Ġtomorrow ] </s> True ĠLab els : Ġ[ 3 , 1 , 3 , 3 ] </s> Then , Ġwe Ġpre process Ġthe Ġinput Ġusing Ġthe Ġtoken izer Ġfor Ġthe Ġmodel Ġbeing Ġused Ġ( BER T , ĠXL Net Ġor ĠRo BER Ta ): Ġsplitting Ġeach Ġword Ġinto Ġone Ġor Ġmore Ġtokens , Ġand Ġconverting Ġeach Ġtoken Ġto Ġit âĢ Ļ s Ġcorresponding Ġtoken ID , Ġand Ġpadding Ġit Ġto Ġthe Ġmaximum Ġinput Ġlength Ġof Ġthe Ġmodel . ĠThus , </s> Input ĠSent ence : Ġ[ w tt ( It ), Ġw tt ( might ), Ġw tt ( rain ), Ġw tt ( tom ), Ġw tt ( ## Ġor ), Ġw tt ( ## Ġrow ), Ġw tt ( ãĢ Ī pad Ġ ãĢ ī ), w tt ( ãĢ Ī pad Ġ ãĢ ī ) ...] </s> True ĠLab els : Ġ[ 3 , 1 , 3 , 3 , 3 , 3 , 4 , 4 , 4 , 4 , ...] </s> The Ġword ĠâĢ ĺ tom orrow ' Ġhas Ġbeen Ġsplit Ġinto Ġ3 Ġtokens , ĠâĢ ĺ tom ', ĠâĢ ĺ ## or ' Ġand ĠâĢ ĺ ## row '. ĠThe Ġfunction Ġto Ġconvert Ġthe Ġword Ġto Ġtoken ID Ġis Ġrepresented Ġby Ġw tt . </s> For ĠSpe culation ĠScope ĠResolution : </s> If Ġa Ġsentence Ġhas Ġmultiple Ġcues , Ġeach Ġcue 's Ġscope Ġwill Ġbe Ġresolved Ġindividually . </s> Input ĠSent ence : ĠIt Ġmight Ġrain Ġtomorrow . </s> True ĠLab els : ĠOut - Of - Scope , ĠOut - Of - Scope , ĠIn - Scope , ĠIn - Scope . </s> First , Ġthis Ġsentence Ġis Ġpre process ed Ġto Ġget Ġthe Ġtarget Ġlabels Ġas Ġper Ġthe Ġfollowing Ġannotation Ġschema : </s> 0 ĠâĢĵ ĠOut - Of - Scope Ġ1 ĠâĢĵ ĠIn - Scope </s> Thus , Ġthe Ġpre process ed Ġsequence Ġis Ġas Ġfollows : </s> True ĠScope ĠLab els : Ġ[ 0 , 0 , 1 , 1 ] </s> As Ġfor Ġcue Ġdetection , Ġwe Ġpre process Ġthe Ġinput Ġusing Ġthe Ġtoken izer Ġfor Ġthe Ġmodel Ġbeing Ġused . ĠAdditionally , Ġwe Ġneed Ġto Ġindicate Ġwhich Ġcue 's Ġscope Ġwe Ġwant Ġto Ġfind Ġin Ġthe Ġinput Ġsentence . ĠWe Ġdo Ġthis Ġby Ġinserting Ġa Ġspecial Ġtoken Ġrepresenting Ġthe Ġtoken Ġtype Ġ( according Ġto Ġthe Ġcue Ġdetection Ġannotation Ġschema ) Ġbefore Ġthe Ġcue Ġword Ġwhose Ġscope Ġis Ġbeing Ġresolved . ĠHere , Ġwe Ġwant Ġto Ġfind Ġthe Ġscope Ġof Ġthe Ġcue ĠâĢ ĺ might '. ĠThus , </s> Input ĠSent ence : Ġ[ w tt ( It ), Ġw tt ( ãĢ Ī token [ 1 ] ãĢ ī ), Ġw tt ( might ), Ġw tt ( rain ), Ġw tt ( tom ), Ġw tt ( ## or ), Ġw tt ( ## row ), Ġw tt ( ãĢ Ī pad ãĢ ī ), Ġw tt ( ãĢ Ī pad ãĢ ī ) ...] </s> True ĠScope ĠLab els : Ġ[ 0 , 0 , 1 , 1 , 1 , 1 , 0 , 0 , 0 , 0 , ...] </s> Now , Ġthe Ġpre process ed Ġinput Ġfor Ġcue Ġdetection Ġand Ġsimilarly Ġfor Ġscope Ġdetection Ġis Ġfed Ġas Ġinput Ġto Ġour Ġmodel Ġas Ġfollows : </s> X Ġ= ĠModel Ġ( Input ) </s> Y Ġ= ĠW * X Ġ+ Ġb </s> The ĠW Ġmatrix Ġis Ġa Ġmatrix Ġof Ġsize Ġn _ hidden Ġx Ġnum _ classes Ġ( n _ hidden Ġis Ġthe Ġsize Ġof Ġthe Ġrepresentation Ġof Ġa Ġtoken Ġwithin Ġthe Ġmodel ). ĠThese Ġlog its Ġare Ġfed Ġto Ġthe Ġloss Ġfunction . </s> We Ġuse Ġthe Ġfollowing Ġvariants Ġof Ġeach Ġmodel : </s> BER T : Ġb ert - base - unc ased s 3 . amazon aws . com / models . h ug ging face . co / bert / bert - base - unc ased . tar . gz Ġ( The Ġmodel Ġused Ġby ĠB IB REF 12 ) </s> Ro BER Ta : Ġro bert a - b ases 3 . amazon aws . com / models . h ug ging face . co / bert / ro bert a - base - py tor ch _ model . bin Ġ( Ro BER Ta - base Ġdoes Ġnot Ġhave Ġan Ġunc ased Ġvariant ) </s> XL Net : Ġx l net - base - c ased s 3 . amazon aws . com / models . h ug ging face . co / bert / x l net - base - c ased - py tor ch _ model . bin Ġ( XL Net - base Ġdoes Ġnot Ġhave Ġan Ġunc ased Ġvariant ) </s> The Ġoutput Ġof Ġthe Ġmodel Ġis Ġa Ġvector Ġof Ġprobabilities Ġper Ġtoken . ĠThe Ġloss Ġis Ġcalculated Ġfor Ġeach Ġtoken , Ġby Ġusing Ġthe Ġoutput Ġvector Ġand Ġthe Ġtrue Ġlabel Ġfor Ġthat Ġtoken . ĠWe Ġuse Ġclass Ġweights Ġfor Ġthe Ġloss Ġfunction , Ġsetting Ġthe Ġweight Ġfor Ġlabel Ġ4 Ġto Ġ0 Ġand Ġall Ġother Ġlabels Ġto Ġ1 Ġ( for Ġcue Ġdetection Ġonly ) Ġto Ġavoid Ġtraining Ġon Ġpadding Ġtoken âĢ Ļ s Ġoutput . </s> We Ġcalculate Ġthe Ġscores Ġ( Pre cision , ĠRecall , ĠF 1 ) Ġfor Ġthe Ġmodel Ġper Ġword Ġof Ġthe Ġinput Ġsentence , Ġnot Ġper Ġtoken Ġthat Ġwas Ġfed Ġto Ġthe Ġmodel , Ġas Ġthe Ġtokens Ġcould Ġbe Ġdifferent Ġfor Ġdifferent Ġmodels Ġleading Ġto Ġinaccurate Ġscores . ĠFor Ġthe Ġabove Ġexample , Ġwe Ġcalculate Ġthe Ġoutput Ġlabel Ġfor Ġthe Ġword ĠâĢ ĺ tom orrow ', Ġnot Ġfor Ġeach Ġtoken Ġit Ġwas Ġsplit Ġinto Ġ( âĢ ĺ tom ', ĠâĢ ĺ ## or ' Ġand ĠâĢ ĺ ## row '). ĠTo Ġfind Ġthe Ġlabel Ġfor Ġeach Ġword Ġfrom Ġthe Ġtokens Ġit Ġwas Ġsplit Ġinto , Ġwe Ġexperiment Ġwith Ġ2 Ġmethods : </s> Average : ĠWe Ġaverage Ġthe Ġoutput Ġvectors Ġ( soft max Ġprobabilities ) Ġfor Ġeach Ġtoken Ġthat Ġthe Ġword Ġwas Ġsplit Ġinto Ġby Ġthe Ġmodel 's Ġtoken izer . ĠIn Ġthe Ġexample Ġabove , Ġwe Ġaverage Ġthe Ġoutput Ġof ĠâĢ ĺ tom ', ĠâĢ ĺ ## or ' Ġand ĠâĢ ĺ ## row ' Ġto Ġget Ġthe Ġoutput Ġfor ĠâĢ ĺ tom orrow '. ĠThen , Ġwe Ġtake Ġan Ġarg max Ġover Ġthe Ġresultant Ġvector . ĠThis Ġis Ġthen Ġcompared Ġwith Ġthe Ġtrue Ġlabel Ġfor Ġthe Ġoriginal Ġword . </s> First ĠToken : ĠHere , Ġwe Ġonly Ġconsider Ġthe Ġfirst Ġtoken 's Ġprobability Ġvector Ġ( among Ġall Ġtokens Ġthe Ġword Ġwas Ġsplit Ġinto ) Ġas Ġthe Ġoutput Ġfor Ġthat Ġword , Ġand Ġget Ġthe Ġlabel Ġby Ġan Ġarg max Ġover Ġthis Ġvector . ĠIn Ġthe Ġexample Ġabove , Ġwe Ġwould Ġconsider Ġthe Ġoutput Ġvector Ġcorresponding Ġto Ġthe Ġtoken ĠâĢ ĺ tom ' Ġas Ġthe Ġoutput Ġfor Ġthe Ġword ĠâĢ ĺ tom orrow '. </s> For Ġcue Ġdetection , Ġthe Ġresults Ġare Ġreported Ġfor Ġthe ĠAverage Ġmethod Ġonly , Ġwhile Ġwe Ġreport Ġthe Ġscores Ġfor Ġboth ĠAverage Ġand ĠFirst ĠToken Ġfor ĠScope ĠResolution . </s> For Ġfair Ġcomparison , Ġwe Ġuse Ġthe Ġsame Ġhyper param eters Ġfor Ġthe Ġentire Ġarchitecture Ġfor Ġall Ġ3 Ġmodels . ĠOnly Ġthe Ġtoken izer Ġand Ġthe Ġmodel Ġare Ġchanged Ġfor Ġeach Ġmodel . ĠAll Ġother Ġhyper param eters Ġare Ġkept Ġsame . ĠWe Ġfin et une Ġthe Ġmodels Ġfor Ġ60 Ġepoch s , Ġusing Ġearly Ġstopping Ġwith Ġa Ġpatience Ġof Ġ6 Ġon Ġthe ĠF 1 Ġscore Ġ( word Ġlevel ) Ġon Ġthe Ġvalidation Ġdataset . ĠWe Ġuse Ġan Ġinitial Ġlearning Ġrate Ġof Ġ3 e - 5 , Ġwith Ġa Ġbatch Ġsize Ġof Ġ8 . ĠWe Ġuse Ġthe ĠC ateg orical ĠCross ĠEnt ropy Ġloss Ġfunction . </s> We Ġuse Ġthe ĠHug ging face âĢ Ļ s ĠPy tor ch ĠTrans former Ġlibrary Ġ( BI B REF 23 ) Ġfor Ġthe Ġmodels Ġand Ġtrain Ġall Ġour Ġmodels Ġon ĠGoogle ĠCol abor atory . </s> Results : ĠSpe culation ĠCue ĠDetection Ġand ĠScope ĠResolution </s> We Ġuse Ġa Ġdefault Ġtrain - valid ation - test Ġsplit Ġof Ġ70 - 15 - 15 Ġfor Ġeach Ġdataset . ĠFor Ġthe Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġsubt asks Ġusing Ġsingle - dat as et Ġtraining , Ġwe Ġreport Ġthe Ġresults Ġas Ġan Ġaverage Ġof Ġ5 Ġruns Ġof Ġthe Ġmodel . ĠFor Ġtraining Ġthe Ġmodel Ġon Ġmultiple Ġdatasets , Ġwe Ġperform Ġa Ġ70 - 15 - 15 Ġsplit Ġof Ġeach Ġtraining Ġdataset , Ġafter Ġwhich Ġthe Ġtrain Ġand Ġvalidation Ġpart Ġof Ġthe Ġindividual Ġdatasets Ġare Ġmerged Ġwhile Ġthe Ġscores Ġare Ġreported Ġfor Ġthe Ġtest Ġpart Ġof Ġthe Ġindividual Ġdatasets , Ġwhich Ġis Ġnot Ġused Ġfor Ġtraining Ġor Ġvalidation . ĠWe Ġreport Ġthe Ġresults Ġas Ġan Ġaverage Ġof Ġ3 Ġruns Ġof Ġthe Ġmodel . ĠFigure ĠFIG REF 8 Ġcontains Ġresults Ġfor Ġspeculation Ġcue Ġdetection Ġand Ġscope Ġresolution Ġwhen Ġtrained Ġon Ġa Ġsingle Ġdataset . ĠAll Ġmodels Ġperform Ġthe Ġbest Ġwhen Ġtrained Ġon Ġthe Ġsame Ġdataset Ġas Ġthey Ġare Ġevaluated Ġon , Ġexcept Ġfor ĠBF , Ġwhich Ġgets Ġthe Ġbest Ġresults Ġwhen Ġtrained Ġon ĠBA . ĠThis Ġis Ġbecause Ġof Ġthe Ġtransfer Ġlearning Ġcapabilities Ġof Ġthe Ġmodels Ġand Ġthe Ġfact Ġthat ĠBF Ġis Ġa Ġsmaller Ġdataset Ġthan ĠBA Ġ( BF : Ġ26 70 Ġsentences , ĠBA : Ġ118 71 Ġsentences ). ĠFor Ġspeculation Ġcue Ġdetection , Ġthere Ġis Ġlesser Ġgeneral iz ability Ġfor Ġmodels Ġtrained Ġon ĠBF Ġor ĠBA , Ġwhile Ġthere Ġis Ġmore Ġgeneral iz ability Ġfor Ġmodels Ġtrained Ġon ĠSF U . ĠThis Ġcould Ġbe Ġbecause Ġof Ġthe Ġdifferent Ġnature Ġof Ġthe Ġbiomedical Ġdomain . </s> Figure ĠFIG REF 11 Ġcontains Ġthe Ġresults Ġfor Ġspeculation Ġdetection Ġand Ġscope Ġresolution Ġfor Ġmodels Ġtrained Ġjointly Ġon Ġmultiple Ġdatasets . ĠWe Ġobserve Ġthat Ġtraining Ġon Ġmultiple Ġdatasets Ġhelps Ġthe Ġperformance Ġof Ġall Ġmodels Ġon Ġeach Ġdataset , Ġas Ġthe Ġquantity Ġof Ġdata Ġavailable Ġto Ġtrain Ġthe Ġmodel Ġincreases . ĠWe Ġalso Ġobserve Ġthat ĠXL Net Ġconsistently Ġoutper forms ĠB ERT Ġand ĠRo BER Ta . ĠTo Ġconfirm Ġthis Ġobservation , Ġwe Ġapply Ġthe Ġ2 Ġmodels Ġto Ġthe Ġrelated Ġtask Ġof Ġneg ation Ġdetection Ġand Ġscope Ġresolution </s> Neg ation ĠCue ĠDetection Ġand ĠScope ĠResolution </s> We Ġuse Ġa Ġdefault Ġtrain - valid ation - test Ġsplit Ġof Ġ70 - 15 - 15 Ġfor Ġeach Ġdataset , Ġand Ġuse Ġall Ġ4 Ġdatasets Ġ( BF , ĠBA , ĠSF U Ġand ĠSherlock ). ĠThe Ġresults Ġfor ĠB ERT Ġare Ġtaken Ġfrom ĠB IB REF 12 . ĠThe Ġresults Ġfor ĠXL Net Ġand ĠRo BER Ta Ġare Ġaveraged Ġacross Ġ5 Ġruns Ġfor Ġstatistical Ġsignificance . ĠFigure ĠFIG REF 14 Ġcontains Ġresults Ġfor Ġneg ation Ġcue Ġdetection Ġand Ġscope Ġresolution . ĠWe Ġreport Ġstate - of - the - art Ġresults Ġon Ġneg ation Ġscope Ġresolution Ġon ĠBF , ĠBA Ġand ĠSF U Ġdatasets . ĠContrary Ġto Ġpopular Ġopinion , Ġwe Ġobserve Ġthat ĠXL Net Ġis Ġbetter Ġthan ĠRo BER Ta Ġfor Ġthe Ġcue Ġdetection Ġand Ġscope Ġresolution Ġtasks . ĠA Ġfew Ġpossible Ġreasons Ġfor Ġthis Ġtrend Ġare : </s> Domain Ġspecificity , Ġas Ġboth Ġneg ation Ġand Ġspeculation Ġare Ġclosely Ġrelated Ġsubt asks . ĠFurther Ġexperimentation Ġon Ġdifferent Ġtasks Ġis Ġneeded Ġto Ġverify Ġthis . </s> Most Ġbenchmark Ġtasks Ġare Ġsentence Ġclassification Ġtasks , Ġwhereas Ġthe Ġsubt asks Ġwe Ġexperiment Ġon Ġare Ġsequence Ġlab elling Ġtasks . ĠGiven Ġthe Ġpre - training Ġobjective Ġof ĠXL Net Ġ( training Ġon Ġperm utations Ġof Ġthe Ġinput ), Ġit Ġmay Ġbe Ġable Ġto Ġcapture Ġlong - term Ġdependencies Ġbetter , Ġessential Ġfor Ġsequence Ġlab elling Ġtasks . </s> We Ġwork Ġwith Ġthe Ġbase Ġvariants Ġof Ġthe Ġmodels , Ġwhile Ġmost Ġresults Ġare Ġreported Ġwith Ġthe Ġlarge Ġvariants Ġof Ġthe Ġmodels . </s> Conclusion Ġand ĠFuture ĠScope </s> In Ġthis Ġpaper , Ġwe Ġexpanded Ġon Ġthe Ġwork Ġof ĠK hand el wal Ġand ĠSaw ant Ġ( BI B REF 12 ) Ġby Ġlooking Ġat Ġalternative Ġtransfer - learning Ġmodels Ġand Ġexperimented Ġwith Ġtraining Ġon Ġmultiple Ġdatasets . ĠOn Ġthe Ġspeculation Ġdetection Ġtask , Ġwe Ġobtained Ġa Ġgain Ġof Ġ0 . 42 ĠF 1 Ġpoints Ġon ĠBF , Ġ1 . 98 ĠF 1 Ġpoints Ġon ĠBA Ġand Ġ0 . 29 ĠF 1 Ġpoints Ġon ĠSF U , Ġwhile Ġon Ġthe Ġscope Ġresolution Ġtask , Ġwe Ġobtained Ġa Ġgain Ġof Ġ8 . 06 ĠF 1 Ġpoints Ġon ĠBF , Ġ4 . 27 ĠF 1 Ġpoints Ġon ĠBA Ġand Ġ11 . 87 ĠF 1 Ġpoints Ġon ĠSF U , Ġwhen Ġtrained Ġon Ġa Ġsingle Ġdataset . ĠWhile Ġtraining Ġon Ġmultiple Ġdatasets , Ġwe Ġobserved Ġa Ġgain Ġof Ġ10 . 6 ĠF 1 Ġpoints Ġon ĠBF Ġand Ġ1 . 94 ĠF 1 Ġpoints Ġon ĠBA Ġon Ġthe Ġspeculation Ġdetection Ġtask Ġand Ġ2 . 16 ĠF 1 Ġpoints Ġon ĠBF Ġand Ġ0 . 25 ĠF 1 Ġpoints Ġon ĠSF U Ġon Ġthe Ġscope Ġresolution Ġtask Ġover Ġthe Ġsingle Ġdataset Ġtraining Ġapproach . ĠWe Ġthus Ġsignificantly Ġadvance Ġthe Ġstate - of - the - art Ġfor Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠOn Ġthe Ġneg ation Ġscope Ġresolution Ġtask , Ġwe Ġapplied Ġthe ĠXL Net Ġand ĠRo BER Ta Ġand Ġobtained Ġa Ġgain Ġof Ġ3 . 16 ĠF 1 Ġpoints Ġon ĠBF , Ġ0 . 06 ĠF 1 Ġpoints Ġon ĠBA Ġand Ġ0 . 3 ĠF 1 Ġpoints Ġon ĠSF U . ĠThus , Ġwe Ġdemonstrated Ġthe Ġusefulness Ġof Ġtransformer - based Ġarchitectures Ġin Ġthe Ġfield Ġof Ġneg ation Ġand Ġspeculation Ġdetection Ġand Ġscope Ġresolution . ĠWe Ġbelieve Ġthat Ġa Ġlarger Ġand Ġmore Ġgeneral Ġdataset Ġwould Ġgo Ġa Ġlong Ġway Ġin Ġbolst ering Ġfuture Ġresearch Ġand Ġwould Ġhelp Ġcreate Ġbetter Ġsystems Ġthat Ġare Ġnot Ġdomain - specific .