Restrictivity , Factivity , Coreference 
EXTRACTIVE
semantics-altering grammatical modifiers
EXTRACTIVE
We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis. As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question). Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer. Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions.
Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards. Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal.
2003.04642
Introduction </s> There Ġis Ġa Ġrecent Ġspark Ġof Ġinterest Ġin Ġthe Ġtask Ġof ĠQuestion ĠAn sw ering Ġ( Q A ) Ġover Ġun struct ured Ġtextual Ġdata , Ġalso Ġreferred Ġto Ġas ĠMachine ĠReading ĠComp rehens ion Ġ( M RC ). ĠThis Ġis Ġmostly Ġdue Ġto Ġwide - spread Ġsuccess Ġof Ġadvances Ġin Ġvarious Ġfacets Ġof Ġdeep Ġlearning Ġrelated Ġresearch , Ġsuch Ġas Ġnovel Ġarchitectures ĠB IB REF 0 , ĠB IB REF 1 Ġthat Ġallow Ġfor Ġefficient Ġoptim isation Ġof Ġneural Ġnetworks Ġconsisting Ġof Ġmultiple Ġlayers , Ġhardware Ġdesigned Ġfor Ġdeep Ġlearning Ġpurposes Ġand Ġsoftware Ġframeworks ĠB IB REF 2 , ĠB IB REF 3 Ġthat Ġallow Ġefficient Ġdevelopment Ġand Ġtesting Ġof Ġnovel Ġapproaches . ĠThese Ġfactors Ġenable Ġresearchers Ġto Ġproduce Ġmodels Ġthat Ġare Ġpre - trained Ġon Ġlarge Ġscale Ġcorpor a Ġand Ġprovide Ġcontextual ised Ġword Ġrepresentations ĠB IB REF 4 Ġthat Ġare Ġshown Ġto Ġbe Ġa Ġvital Ġcomponent Ġtowards Ġsolutions Ġfor Ġa Ġvariety Ġof Ġnatural Ġlanguage Ġunderstanding Ġtasks , Ġincluding ĠM RC ĠB IB REF 5 . ĠAnother Ġimportant Ġfactor Ġthat Ġled Ġto Ġthe Ġrecent Ġsuccess Ġin ĠM RC - related Ġtasks Ġis Ġthe Ġwidespread Ġavailability Ġof Ġvarious Ġlarge Ġdatasets , Ġe . g ., ĠS Qu AD ĠB IB REF 6 , Ġthat Ġprovide Ġsufficient Ġexamples Ġfor Ġoptim ising Ġstatistical Ġmodels . ĠThe Ġcombination Ġof Ġthese Ġfactors Ġyields Ġnotable Ġresults , Ġeven Ġsurpass ing Ġhuman Ġperformance ĠB IB REF 7 . </s> M RC Ġis Ġa Ġgeneric Ġtask Ġformat Ġthat Ġcan Ġbe Ġused Ġto Ġprobe Ġfor Ġvarious Ġnatural Ġlanguage Ġunderstanding Ġcapabilities ĠB IB REF 8 . ĠTherefore Ġit Ġis Ġcru cially Ġimportant Ġto Ġestablish Ġa Ġrigorous Ġevaluation Ġmethodology Ġin Ġorder Ġto Ġbe Ġable Ġto Ġdraw Ġreliable Ġconclusions Ġfrom Ġconducted Ġexperiments . ĠWhile Ġincreasing Ġeffort Ġis Ġput Ġinto Ġthe Ġevaluation Ġof Ġnovel Ġarchitectures , Ġsuch Ġas Ġkeeping Ġthe Ġevaluation Ġdata Ġfrom Ġpublic Ġaccess Ġto Ġprevent Ġunintentional Ġover fitting Ġto Ġtest Ġdata , Ġperforming Ġab lation Ġand Ġerror Ġstudies Ġand Ġintroducing Ġnovel Ġmetrics ĠB IB REF 9 , Ġsurprisingly Ġlittle Ġis Ġdone Ġto Ġestablish Ġthe Ġquality Ġof Ġthe Ġdata Ġitself . ĠAdditionally , Ġrecent Ġresearch Ġarrived Ġat Ġworrisome Ġfindings : Ġthe Ġdata Ġof Ġthose Ġgold Ġstandards , Ġwhich Ġis Ġusually Ġgathered Ġinvolving Ġa Ġcrowd - s ourcing Ġstep , Ġsuffers Ġfrom Ġflaws Ġin Ġdesign ĠB IB REF 10 Ġor Ġcontains Ġoverly Ġspecific Ġkeywords ĠB IB REF 11 . ĠFurthermore , Ġthese Ġgold Ġstandards Ġcontain ĠâĢ ľ ann otation Ġarte facts âĢ Ŀ , Ġcues Ġthat Ġlead Ġmodels Ġinto Ġfocusing Ġon Ġsuperficial Ġaspects Ġof Ġtext , Ġsuch Ġas Ġlex ical Ġoverlap Ġand Ġword Ġorder , Ġinstead Ġof Ġactual Ġlanguage Ġunderstanding ĠB IB REF 12 , ĠB IB REF 13 . ĠThese Ġweaknesses Ġcast Ġsome Ġdoubt Ġon Ġwhether Ġthe Ġdata Ġcan Ġreliably Ġevaluate Ġthe Ġreading Ġcomprehension Ġperformance Ġof Ġthe Ġmodels Ġthey Ġevaluate , Ġi . e . Ġif Ġthe Ġmodels Ġare Ġindeed Ġbeing Ġassessed Ġfor Ġtheir Ġcapability Ġto Ġread . </s> Figure ĠFIG REF 3 Ġshows Ġan Ġexample Ġfrom ĠHot pot Q A ĠB IB REF 14 , Ġa Ġdataset Ġthat Ġexhibits Ġthe Ġlast Ġkind Ġof Ġweakness Ġmentioned Ġabove , Ġi . e ., Ġthe Ġpresence Ġof Ġunique Ġkeywords Ġin Ġboth Ġthe Ġquestion Ġand Ġthe Ġpassage Ġ( in Ġclose Ġproximity Ġto Ġthe Ġexpected Ġanswer ). </s> An Ġevaluation Ġmethodology Ġis Ġvital Ġto Ġthe Ġfine - gr ained Ġunderstanding Ġof Ġchallenges Ġassociated Ġwith Ġa Ġsingle Ġgold Ġstandard , Ġin Ġorder Ġto Ġunderstand Ġin Ġgreater Ġdetail Ġwhich Ġcapabilities Ġof ĠM RC Ġmodels Ġit Ġevaluates . ĠMore Ġimportantly , Ġit Ġallows Ġto Ġdraw Ġcomparisons Ġbetween Ġmultiple Ġgold Ġstandards Ġand Ġbetween Ġthe Ġresults Ġof Ġrespective Ġstate - of - the - art Ġmodels Ġthat Ġare Ġevaluated Ġon Ġthem . </s> In Ġthis Ġwork , Ġwe Ġtake Ġa Ġstep Ġback Ġand Ġpropose Ġa Ġframework Ġto Ġsystematically Ġanalyse ĠM RC Ġevaluation Ġdata , Ġtypically Ġa Ġset Ġof Ġquestions Ġand Ġexpected Ġanswers Ġto Ġbe Ġderived Ġfrom Ġaccompanying Ġpassages . ĠCon crete ly , Ġwe Ġintroduce Ġa Ġmethodology Ġto Ġcategor ise Ġthe Ġlinguistic Ġcomplexity Ġof Ġthe Ġtextual Ġdata Ġand Ġthe Ġreasoning Ġand Ġpotential Ġexternal Ġknowledge Ġrequired Ġto Ġobtain Ġthe Ġexpected Ġanswer . ĠAdditionally Ġwe Ġpropose Ġto Ġtake Ġa Ġcloser Ġlook Ġat Ġthe Ġfactual Ġcorrectness Ġof Ġthe Ġexpected Ġanswers , Ġa Ġquality Ġdimension Ġthat Ġappears Ġunder - expl ored Ġin Ġliterature . </s> We Ġdemonstrate Ġthe Ġusefulness Ġof Ġthe Ġproposed Ġframework Ġby Ġapplying Ġit Ġto Ġprecisely Ġdescribe Ġand Ġcompare Ġsix Ġcontemporary ĠM RC Ġdatasets . ĠOur Ġfindings Ġreveal Ġconcerns Ġabout Ġtheir Ġfactual Ġcorrectness , Ġthe Ġpresence Ġof Ġlex ical Ġcues Ġthat Ġsimplify Ġthe Ġtask Ġof Ġreading Ġcomprehension Ġand Ġthe Ġlack Ġof Ġsemantic Ġaltering Ġgram matical Ġmodifiers . ĠWe Ġrelease Ġthe Ġraw Ġdata Ġcomprised Ġof Ġ300 Ġparagraphs , Ġquestions Ġand Ġanswers Ġrich ly Ġannot ated Ġunder Ġthe Ġproposed Ġframework Ġas Ġa Ġresource Ġfor Ġresearchers Ġdeveloping Ġnatural Ġlanguage Ġunderstanding Ġmodels Ġand Ġdatasets Ġto Ġutil ise Ġfurther . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge Ġthis Ġis Ġthe Ġfirst Ġattempt Ġto Ġintroduce Ġa Ġcommon Ġevaluation Ġmethodology Ġfor ĠM RC Ġgold Ġstandards Ġand Ġthe Ġfirst Ġacross - the - board Ġqualitative Ġevaluation Ġof ĠM RC Ġdatasets Ġwith Ġrespect Ġto Ġthe Ġproposed Ġcategories . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠProblem Ġdefinition </s> We Ġdefine Ġthe Ġtask Ġof Ġmachine Ġreading Ġcomprehension , Ġthe Ġtarget Ġapplication Ġof Ġthe Ġproposed Ġmethodology Ġas Ġfollows : ĠGiven Ġa Ġparagraph Ġ$ P $ Ġthat Ġconsists Ġof Ġtokens Ġ( words ) Ġ$ p _ 1 , Ġ\ ld ots Ġ, Ġp _{ n _ P } $ Ġand Ġa Ġquestion Ġ$ Q $ Ġthat Ġconsists Ġof Ġtokens Ġ$ q _ 1 Ġ\ ld ots Ġq _{ n _ Q } $, Ġthe Ġgoal Ġis Ġto Ġretrieve Ġan Ġanswer Ġ$ A $ Ġwith Ġtokens Ġ$ a _ 1 Ġ\ ld ots Ġa _{ n _ A } $. Ġ$ A $ Ġis Ġcommonly Ġconstrained Ġto Ġbe Ġone Ġof Ġthe Ġfollowing Ġcases ĠB IB REF 15 , Ġillustrated Ġin ĠFigure ĠFIG REF 9 : </s> Multiple Ġchoice , Ġwhere Ġthe Ġgoal Ġis Ġto Ġpredict Ġ$ A $ Ġfrom Ġa Ġgiven Ġset Ġof Ġchoices Ġ$\ math cal Ġ{ A } $. </s> Cl o ze - style , Ġwhere Ġ$ S $ Ġis Ġa Ġsentence , Ġand Ġ$ A $ Ġand Ġ$ Q $ Ġare Ġobtained Ġby Ġremoving Ġa Ġsequence Ġof Ġwords Ġsuch Ġthat Ġ$ Q Ġ= ĠS Ġ- ĠA $. ĠThe Ġtask Ġis Ġto Ġfill Ġin Ġthe Ġresulting Ġgap Ġin Ġ$ Q $ Ġwith Ġthe Ġexpected Ġanswer Ġ$ A $ Ġto Ġform Ġ$ S $. </s> Sp an , Ġwhere Ġis Ġa Ġcontinuous Ġsubsequ ence Ġof Ġtokens Ġfrom Ġthe Ġparagraph Ġ($ A Ġ\ sub set eq ĠP $ ). ĠFlav ours Ġinclude Ġmultiple Ġspans Ġas Ġthe Ġcorrect Ġanswer Ġor Ġ$ A Ġ\ sub set eq ĠQ $. </s> Free Ġform , Ġwhere Ġ$ A $ Ġis Ġan Ġuncon str ained Ġnatural Ġlanguage Ġstring . </s> A Ġgold Ġstandard Ġ$ G $ Ġis Ġcomposed Ġof Ġ$ m $ Ġentries Ġ$( Q _ i , ĠA _ i , ĠP _ i ) _{ i \ in Ġ\ l brace Ġ1 , \ ld ots Ġ, Ġm \ r brace Ġ} $. </s> The Ġperformance Ġof Ġan Ġapproach Ġis Ġestablished Ġby Ġcomparing Ġits Ġanswer Ġpredictions Ġ$ A ^ * _{ i } $ Ġon Ġthe Ġgiven Ġinput Ġ$( Q _ i , ĠT _ i ) $ Ġ( and Ġ$\ math cal Ġ{ A } _ i $ Ġfor Ġthe Ġmultiple Ġchoice Ġsetting ) Ġagainst Ġthe Ġexpected Ġanswer Ġ$ A _ i $ Ġfor Ġall Ġ$ i \ in Ġ\ l brace Ġ1 , \ ld ots Ġ, Ġm \ r brace Ġ$ Ġunder Ġa Ġperformance Ġmetric . ĠTypical Ġperformance Ġmetrics Ġare Ġexact Ġmatch Ġ( EM ) Ġor Ġaccuracy , Ġi . e . Ġthe Ġpercentage Ġof Ġexactly Ġpredicted Ġanswers , Ġand Ġthe ĠF 1 Ġscore ĠâĢĵ Ġthe Ġharmonic Ġmean Ġbetween Ġthe Ġprecision Ġand Ġthe Ġrecall Ġof Ġthe Ġpredicted Ġtokens Ġcompared Ġto Ġexpected Ġanswer Ġtokens . ĠThe Ġoverall ĠF 1 Ġscore Ġcan Ġeither Ġbe Ġcomputed Ġby Ġaveraging Ġthe ĠF 1 Ġscores Ġfor Ġevery Ġinstance Ġor Ġby Ġfirst Ġaveraging Ġthe Ġprecision Ġand Ġrecall Ġand Ġthen Ġcomputing Ġthe ĠF 1 Ġscore Ġfrom Ġthose Ġaverages Ġ( mac ro ĠF 1 ). ĠFree - text Ġanswers , Ġmeanwhile , Ġare Ġevaluated Ġby Ġmeans Ġof Ġtext Ġgeneration Ġand Ġsummar isation Ġmetrics Ġsuch Ġas ĠB LE U ĠB IB REF 16 Ġor ĠR OU GE - L ĠB IB REF 17 . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest </s> In Ġthis Ġsection Ġwe Ġdescribe Ġa Ġmethodology Ġto Ġcategor ise Ġgold Ġstandards Ġaccording Ġto Ġlinguistic Ġcomplexity , Ġrequired Ġreasoning Ġand Ġbackground Ġknowledge , Ġand Ġtheir Ġfactual Ġcorrectness . ĠSpecifically , Ġwe Ġuse Ġthose Ġdimensions Ġas Ġhigh - level Ġcategories Ġof Ġa Ġqualitative Ġannotation Ġschema Ġfor Ġannot ating Ġquestion , Ġexpected Ġanswer Ġand Ġthe Ġcorresponding Ġcontext . ĠWe Ġfurther Ġenrich Ġthe Ġqualitative Ġannotations Ġby Ġa Ġmetric Ġbased Ġon Ġlex ical Ġcues Ġin Ġorder Ġto Ġapproximate Ġa Ġlower Ġbound Ġfor Ġthe Ġcomplexity Ġof Ġthe Ġreading Ġcomprehension Ġtask . ĠBy Ġsampling Ġentries Ġfrom Ġeach Ġgold Ġstandard Ġand Ġannot ating Ġthem , Ġwe Ġobtain Ġmeasurable Ġresults Ġand Ġthus Ġare Ġable Ġto Ġmake Ġobservations Ġabout Ġthe Ġchallenges Ġpresent Ġin Ġthat Ġgold Ġstandard Ġdata . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠProblem Ġsetting </s> We Ġare Ġinterested Ġin Ġdifferent Ġtypes Ġof Ġthe Ġexpected Ġanswer . ĠWe Ġdifferentiate Ġbetween ĠSpan , Ġwhere Ġan Ġanswer Ġis Ġa Ġcontinuous Ġspan Ġtaken Ġfrom Ġthe Ġpassage , ĠPar aph r asing , Ġwhere Ġthe Ġanswer Ġis Ġa Ġparaph rase Ġof Ġa Ġtext Ġspan , ĠUn answer able , Ġwhere Ġthere Ġis Ġno Ġanswer Ġpresent Ġin Ġthe Ġcontext , Ġand ĠGener ated , Ġif Ġit Ġdoes Ġnot Ġfall Ġinto Ġany Ġof Ġthe Ġother Ġcategories . ĠIt Ġis Ġnot Ġsufficient Ġfor Ġan Ġanswer Ġto Ġrest ate Ġthe Ġquestion Ġor Ġcombine Ġmultiple ĠSpan Ġor ĠPar aph r asing Ġanswers Ġto Ġbe Ġannot ated Ġas ĠGener ated . ĠIt Ġis Ġworth Ġmentioning Ġthat Ġwe Ġfocus Ġour Ġinvestigations Ġon Ġanswer able Ġquestions . ĠFor Ġa Ġcomplementary Ġqualitative Ġanalysis Ġthat Ġcategor ises Ġun answer able Ġquestions , Ġthe Ġreader Ġis Ġreferred Ġto ĠY ats kar 2019 . </s> Furthermore , Ġwe Ġmark Ġa Ġsentence Ġas ĠSupporting ĠFact Ġif Ġit Ġcontains Ġevidence Ġrequired Ġto Ġproduce Ġthe Ġexpected Ġanswer , Ġas Ġthey Ġare Ġused Ġfurther Ġin Ġthe Ġcomplexity Ġanalysis . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠFact ual ĠCorrect ness </s> An Ġimportant Ġfactor Ġfor Ġthe Ġquality Ġof Ġa Ġbenchmark Ġis Ġits Ġfactual Ġcorrectness , Ġbecause Ġon Ġthe Ġone Ġhand , Ġthe Ġpresence Ġof Ġfact ually Ġwrong Ġor Ġdeb atable Ġexamples Ġintroduces Ġan Ġupper Ġbound Ġfor Ġthe Ġachievable Ġperformance Ġof Ġmodels Ġon Ġthose Ġgold Ġstandards . ĠOn Ġthe Ġother Ġhand , Ġit Ġis Ġhard Ġto Ġdraw Ġconclusions Ġabout Ġthe Ġcorrectness Ġof Ġanswers Ġproduced Ġby Ġa Ġmodel Ġthat Ġis Ġevaluated Ġon Ġpartially Ġincorrect Ġdata . </s> One Ġway Ġby Ġwhich Ġdevelopers Ġof Ġmodern Ġcrowd - s ourced Ġgold Ġstandards Ġensure Ġquality Ġis Ġby Ġhaving Ġthe Ġsame Ġentry Ġannot ated Ġby Ġmultiple Ġworkers ĠB IB REF 18 Ġand Ġkeeping Ġonly Ġthose Ġwith Ġhigh Ġagreement . ĠWe Ġinvestigate Ġwhether Ġthis Ġmethod Ġis Ġenough Ġto Ġestablish Ġa Ġsound Ġground Ġtruth Ġanswer Ġthat Ġis Ġunamb ig uously Ġcorrect . ĠCon crete ly Ġwe Ġannot ate Ġan Ġanswer Ġas ĠDe bat able Ġwhen Ġthe Ġpassage Ġfeatures Ġmultiple Ġplausible Ġanswers , Ġwhen Ġmultiple Ġexpected Ġanswers Ġcontradict Ġeach Ġother , Ġor Ġan Ġanswer Ġis Ġnot Ġspecific Ġenough Ġwith Ġrespect Ġto Ġthe Ġquestion Ġand Ġa Ġmore Ġspecific Ġanswer Ġis Ġpresent . ĠWe Ġannot ate Ġan Ġanswer Ġas ĠWrong Ġwhen Ġit Ġis Ġfact ually Ġwrong Ġand Ġa Ġcorrect Ġanswer Ġis Ġpresent Ġin Ġthe Ġcontext . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠRequired ĠReason ing </s> It Ġis Ġimportant Ġto Ġunderstand Ġwhat Ġtypes Ġof Ġreasoning Ġthe Ġbenchmark Ġevaluates , Ġin Ġorder Ġto Ġbe Ġable Ġto Ġacc redit Ġvarious Ġreasoning Ġcapabilities Ġto Ġthe Ġmodels Ġit Ġevaluates . ĠOur Ġproposed Ġreasoning Ġcategories Ġare Ġinspired Ġby Ġthose Ġfound Ġin Ġscientific Ġquestion Ġanswering Ġliterature ĠB IB REF 19 , ĠB IB REF 20 , Ġas Ġresearch Ġin Ġthis Ġarea Ġfocuses Ġon Ġunderstanding Ġthe Ġrequired Ġreasoning Ġcapabilities . ĠWe Ġinclude Ġreasoning Ġabout Ġthe ĠTem poral Ġsuccession Ġof Ġevents , ĠSp atial Ġreasoning Ġabout Ġdirections Ġand Ġenvironment , Ġand ĠCa usal Ġreasoning Ġabout Ġthe Ġcause - effect Ġrelationship Ġbetween Ġevents . ĠWe Ġfurther Ġannot ate Ġ( multiple - choice ) Ġanswers Ġthat Ġcan Ġonly Ġbe Ġanswered ĠBy ĠEx clusion Ġof Ġevery Ġother Ġalternative . </s> We Ġfurther Ġextend Ġthe Ġreasoning Ġcategories Ġby Ġoperational Ġlogic , Ġsimilar Ġto Ġthose Ġrequired Ġin Ġsemantic Ġparsing Ġtasks ĠB IB REF 21 , Ġas Ġsolving Ġthose Ġtasks Ġtypically Ġrequires ĠâĢ ľ multi - hop âĢ Ŀ Ġreasoning ĠB IB REF 14 , ĠB IB REF 22 . ĠWhen Ġan Ġanswer Ġcan Ġonly Ġbe Ġobtained Ġby Ġcombining Ġinformation Ġfrom Ġdifferent Ġsentences Ġjoined Ġby Ġmentioning Ġa Ġcommon Ġentity , Ġconcept , Ġdate , Ġfact Ġor Ġevent Ġ( from Ġhere Ġon Ġcalled Ġentity ), Ġwe Ġannot ate Ġit Ġas ĠBridge . ĠWe Ġfurther Ġannot ate Ġthe Ġcases , Ġwhen Ġthe Ġanswer Ġis Ġa Ġconcrete Ġentity Ġthat Ġsatisfies Ġa ĠCon str aint Ġspecified Ġin Ġthe Ġquestion , Ġwhen Ġit Ġis Ġrequired Ġto Ġdraw Ġa ĠComparison Ġof Ġmultiple Ġentities ' Ġproperties Ġor Ġwhen Ġthe Ġexpected Ġanswer Ġis Ġan ĠInter section Ġof Ġtheir Ġproperties Ġ( e . g . ĠâĢ ľ What Ġdo ĠPerson ĠA Ġand ĠPerson ĠB Ġhave Ġin Ġcommon ? âĢ Ŀ ) </s> We Ġare Ġinterested Ġin Ġthe Ġlinguistic Ġreasoning Ġcapabilities Ġprob ed Ġby Ġa Ġgold Ġstandard , Ġtherefore Ġwe Ġinclude Ġthe Ġappropriate Ġcategory Ġused Ġby ĠWang 2019 . ĠSpecifically , Ġwe Ġannot ate Ġoccurrences Ġthat Ġrequire Ġunderstanding Ġof ĠNeg ation , ĠQuant ifiers Ġ( such Ġas ĠâĢ ľ every âĢ Ŀ , ĠâĢ ľ some âĢ Ŀ , Ġor ĠâĢ ľ all âĢ Ŀ ), ĠCond itional Ġ( âĢ ľ if Ġ... then âĢ Ŀ ) Ġstatements Ġand Ġthe Ġlogical Ġimplications Ġof ĠCon - / Dis j unction Ġ( i . e . ĠâĢ ľ and âĢ Ŀ Ġand ĠâĢ ľ or âĢ Ŀ ) Ġin Ġorder Ġto Ġderive Ġthe Ġexpected Ġanswer . </s> Finally , Ġwe Ġinvestigate Ġwhether Ġarithmetic Ġreasoning Ġrequirements Ġemerge Ġin ĠM RC Ġgold Ġstandards Ġas Ġthis Ġcan Ġprobe Ġfor Ġreasoning Ġthat Ġis Ġnot Ġevaluated Ġby Ġsimple Ġanswer Ġretrieval ĠB IB REF 23 . ĠTo Ġthis Ġend , Ġwe Ġannot ate Ġthe Ġpresence Ġof Ġof ĠAdd ition Ġand ĠSub t raction , Ġanswers Ġthat Ġrequire ĠOrder ing Ġof Ġnumerical Ġvalues , ĠCount ing Ġand ĠOther Ġoccurrences Ġof Ġsimple Ġmathematical Ġoperations . </s> An Ġexample Ġcan Ġexhibit Ġmultiple Ġforms Ġof Ġreasoning . ĠNotably , Ġwe Ġdo Ġnot Ġannot ate Ġany Ġof Ġthe Ġcategories Ġmentioned Ġabove Ġif Ġthe Ġexpected Ġanswer Ġis Ġdirectly Ġstated Ġin Ġthe Ġpassage . ĠFor Ġexample , Ġif Ġthe Ġquestion Ġasks ĠâĢ ľ How Ġmany Ġtotal Ġpoints Ġwere Ġscored Ġin Ġthe Ġgame ? âĢ Ŀ Ġand Ġthe Ġpassage Ġcontains Ġa Ġsentence Ġsimilar Ġto ĠâĢ ľ The Ġtotal Ġscore Ġof Ġthe Ġgame Ġwas Ġ51 Ġpoints âĢ Ŀ , Ġit Ġdoes Ġnot Ġrequire Ġany Ġreasoning , Ġin Ġwhich Ġcase Ġwe Ġannot ate Ġit Ġas ĠRet ri eval . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠKnowledge </s> W orth while Ġknowing Ġis Ġwhether Ġthe Ġinformation Ġpresented Ġin Ġthe Ġcontext Ġis Ġsufficient Ġto Ġanswer Ġthe Ġquestion , Ġas Ġthere Ġis Ġan Ġincrease Ġof Ġbenchmarks Ġdeliberately Ġdesigned Ġto Ġprobe Ġa Ġmodel 's Ġreliance Ġon Ġsome Ġsort Ġof Ġbackground Ġknowledge ĠB IB REF 24 . ĠWe Ġseek Ġto Ġcategor ise Ġthe Ġtype Ġof Ġknowledge Ġrequired . ĠSimilar Ġto ĠWang 2019 , Ġon Ġthe Ġone Ġhand Ġwe Ġannot ate Ġthe Ġreliance Ġon Ġfactual Ġknowledge , Ġthat Ġis Ġ( Ge o ) political / Legal , ĠCultural / Hist oric , ĠTechnical / Scient ific Ġand ĠOther ĠDomain ĠSpecific Ġknowledge Ġabout Ġthe Ġworld Ġthat Ġcan Ġbe Ġexpressed Ġas Ġa Ġset Ġof Ġfacts . ĠOn Ġthe Ġother Ġhand , Ġwe Ġdenote ĠInt uitive Ġknowledge Ġrequirements , Ġwhich Ġis Ġchallenging Ġto Ġexpress Ġas Ġa Ġset Ġof Ġfacts , Ġsuch Ġas Ġthe Ġknowledge Ġthat Ġa Ġparent hetic Ġnumerical Ġexpression Ġnext Ġto Ġa Ġperson 's Ġname Ġin Ġa Ġbiography Ġusually Ġdenotes Ġhis Ġlife Ġspan . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠL ingu istic ĠComplex ity </s> Another Ġdimension Ġof Ġinterest Ġis Ġthe Ġevaluation Ġof Ġvarious Ġlinguistic Ġcapabilities Ġof ĠM RC Ġmodels ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 27 . ĠWe Ġaim Ġto Ġestablish Ġwhich Ġlinguistic Ġphenomena Ġare Ġprob ed Ġby Ġgold Ġstandards Ġand Ġto Ġwhich Ġdegree . ĠTo Ġthat Ġend , Ġwe Ġdraw Ġinspiration Ġfrom Ġthe Ġannotation Ġschema Ġused Ġby ĠWang 2019 , Ġand Ġadapt Ġit Ġaround Ġlex ical Ġsemantics Ġand Ġsyntax . </s> More Ġspecifically , Ġwe Ġannot ate Ġfeatures Ġthat Ġintroduce Ġvariance Ġbetween Ġthe Ġsupporting Ġfacts Ġand Ġthe Ġquestion . ĠWith Ġregard Ġto Ġlex ical Ġsemantics , Ġwe Ġfocus Ġon Ġthe Ġuse Ġof Ġredundant Ġwords Ġthat Ġdo Ġnot Ġalter Ġthe Ġmeaning Ġof Ġa Ġsentence Ġfor Ġthe Ġtask Ġof Ġretrieving Ġthe Ġexpected Ġanswer Ġ( Red und ancy ), Ġrequirements Ġon Ġthe Ġunderstanding Ġof Ġwords ' Ġsemantic Ġfields Ġ( Lex ical ĠEnt ail ment ) Ġand Ġthe Ġuse Ġof ĠSyn onyms Ġand ĠPar aph r ases Ġwith Ġrespect Ġto Ġthe Ġquestion Ġwording . ĠFurthermore Ġwe Ġannot ate Ġcases Ġwhere Ġsupporting Ġfacts Ġcontain ĠAb bre vi ations Ġof Ġconcepts Ġintroduced Ġin Ġthe Ġquestion Ġ( and Ġvice Ġversa ) Ġand Ġwhen Ġa ĠD ative Ġcase Ġsubstitutes Ġthe Ġuse Ġof Ġa Ġpre position Ġ( e . g . ĠâĢ ľ I Ġbought Ġher Ġa Ġgift âĢ Ŀ Ġvs ĠâĢ ľ I Ġbought Ġa Ġgift Ġfor Ġher âĢ Ŀ ). ĠRegarding Ġsyntax , Ġwe Ġannot ate Ġchanges Ġfrom Ġpassive Ġto Ġactive ĠVoice , Ġthe Ġsubstitution Ġof Ġa ĠGen itive Ġcase Ġwith Ġa Ġpre position Ġ( e . g . ĠâĢ ľ of âĢ Ŀ ) Ġand Ġchanges Ġfrom Ġnominal Ġto Ġverbal Ġstyle Ġand Ġvice Ġversa Ġ( N om inal isation ). </s> We Ġrecognise Ġfeatures Ġthat Ġadd Ġambiguity Ġto Ġthe Ġsupporting Ġfacts , Ġfor Ġexample Ġwhen Ġinformation Ġis Ġonly Ġexpressed Ġimplicitly Ġby Ġusing Ġan ĠEll ips is . ĠAs Ġopposed Ġto Ġredundant Ġwords , Ġwe Ġannot ate ĠRestrict ivity Ġand ĠFact ivity Ġmodifiers , Ġwords Ġand Ġphrases Ġwhose Ġpresence Ġdoes Ġchange Ġthe Ġmeaning Ġof Ġa Ġsentence Ġwith Ġregard Ġto Ġthe Ġexpected Ġanswer , Ġand Ġoccurrences Ġof Ġintra - Ġor Ġinter - sent ence ĠCore ference Ġin Ġsupporting Ġfacts Ġ( that Ġis Ġrelevant Ġto Ġthe Ġquestion ). ĠLastly , Ġwe Ġmark Ġambiguous Ġsynt actic Ġfeatures , Ġwhen Ġtheir Ġresolution Ġis Ġrequired Ġin Ġorder Ġto Ġobtain Ġthe Ġanswer . ĠCon crete ly , Ġwe Ġmark Ġargument Ġcollection Ġwith Ġcon - Ġand Ġdis jun ctions Ġ( List ing ) Ġand Ġambiguous ĠPre pos itions , ĠCoord ination ĠScope Ġand ĠRelative Ġclauses / Ad verb ial Ġphrases / App os itions . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠComplex ity </s> Finally , Ġwe Ġwant Ġto Ġapproximate Ġthe Ġpresence Ġof Ġlex ical Ġcues Ġthat Ġmight Ġsimplify Ġthe Ġreading Ġrequired Ġin Ġorder Ġto Ġarrive Ġat Ġthe Ġanswer . ĠQuant ifying Ġthis Ġallows Ġfor Ġmore Ġreliable Ġstatements Ġabout Ġand Ġcomparison Ġof Ġthe Ġcomplexity Ġof Ġgold Ġstandards , Ġparticularly Ġregarding Ġthe Ġevaluation Ġof Ġcomprehension Ġthat Ġgoes Ġbeyond Ġsimple Ġlex ical Ġmatching . ĠWe Ġpropose Ġthe Ġuse Ġof Ġcoarse Ġmetrics Ġbased Ġon Ġlex ical Ġoverlap Ġbetween Ġquestion Ġand Ġcontext Ġsentences . ĠInt uitive ly , Ġwe Ġaim Ġto Ġquantify Ġhow Ġmuch Ġsupporting Ġfacts ĠâĢ ľ stand Ġout âĢ Ŀ Ġfrom Ġtheir Ġsurrounding Ġpassage Ġcontext . ĠThis Ġcan Ġbe Ġused Ġas Ġproxy Ġfor Ġthe Ġcapability Ġto Ġretrieve Ġthe Ġanswer ĠB IB REF 10 . ĠSpecifically , Ġwe Ġmeasure Ġ( i ) Ġthe Ġnumber Ġof Ġwords Ġjointly Ġoccurring Ġin Ġa Ġquestion Ġand Ġa Ġsentence , Ġ( ii ) Ġthe Ġlength Ġof Ġthe Ġlongest Ġn - gram Ġshared Ġby Ġquestion Ġand Ġsentence Ġand Ġ( iii ) Ġwhether Ġa Ġword Ġor Ġn - gram Ġfrom Ġthe Ġquestion Ġuniquely Ġappears Ġin Ġa Ġsentence . </s> The Ġresulting Ġtax onomy Ġof Ġthe Ġframework Ġis Ġshown Ġin ĠFigure ĠFIG REF 10 . ĠThe Ġfull Ġcatalogue Ġof Ġfeatures , Ġtheir Ġdescription , Ġdetailed Ġannotation Ġguideline Ġas Ġwell Ġas Ġillustrating Ġexamples Ġcan Ġbe Ġfound Ġin ĠAppendix Ġ. </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠCandidate ĠDat as ets </s> We Ġselect Ġcontemporary ĠM RC Ġbenchmarks Ġto Ġrepresent Ġall Ġfour Ġcommonly Ġused Ġproblem Ġdefinitions ĠB IB REF 15 . ĠIn Ġselecting Ġrelevant Ġdatasets , Ġwe Ġdo Ġnot Ġconsider Ġthose Ġthat Ġare Ġconsidered ĠâĢ ľ s olved âĢ Ŀ , Ġi . e . Ġwhere Ġthe Ġstate Ġof Ġthe Ġart Ġperformance Ġsurpass es Ġhuman Ġperformance , Ġas Ġis Ġthe Ġcase Ġwith ĠS Qu AD ĠB IB REF 28 , ĠB IB REF 7 . ĠCon crete ly , Ġwe Ġselected Ġgold Ġstandards Ġthat Ġfit Ġour Ġproblem Ġdefinition Ġand Ġwere Ġpublished Ġin Ġthe Ġyears Ġ2016 Ġto Ġ2019 , Ġhave Ġat Ġleast Ġ$( 2019 Ġ- Ġpublication \ Ġyear ) Ġ\ times Ġ20 $ Ġcitations , Ġand Ġbucket Ġthem Ġaccording Ġto Ġthe Ġanswer Ġselection Ġstyles Ġas Ġdescribed Ġin ĠSection ĠSEC REF 4 ĠWe Ġrandomly Ġdraw Ġone Ġfrom Ġeach Ġbucket Ġand Ġadd Ġtwo Ġrandomly Ġdrawn Ġdatasets Ġfrom Ġthe Ġcandidate Ġpool . ĠThis Ġleaves Ġus Ġwith Ġthe Ġdatasets Ġdescribed Ġin ĠTable ĠT AB REF 19 . ĠFor Ġa Ġmore Ġdetailed Ġdescription , Ġwe Ġrefer Ġto ĠAppendix Ġ. </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠAn notation ĠTask </s> We Ġrandomly Ġselect Ġ50 Ġdistinct Ġquestion , Ġanswer Ġand Ġpassage Ġtri ples Ġfrom Ġthe Ġpublicly Ġavailable Ġdevelopment Ġsets Ġof Ġthe Ġdescribed Ġdatasets . ĠTraining , Ġdevelopment Ġand Ġthe Ġ( hidden ) Ġtest Ġset Ġare Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġdefined Ġby Ġthe Ġdata Ġcollection Ġmethod Ġof Ġthe Ġrespective Ġdataset . ĠFor Ġthose Ġcollections Ġthat Ġcontain Ġmultiple Ġquestions Ġover Ġa Ġsingle Ġpassage , Ġwe Ġensure Ġthat Ġwe Ġare Ġsampling Ġunique Ġparagraphs Ġin Ġorder Ġto Ġincrease Ġthe Ġvariety Ġof Ġinvestigated Ġtexts . </s> The Ġsamples Ġwere Ġannot ated Ġby Ġthe Ġfirst Ġauthor Ġof Ġthis Ġpaper , Ġusing Ġthe Ġproposed Ġschema . ĠIn Ġorder Ġto Ġvalidate Ġour Ġfindings , Ġwe Ġfurther Ġtake Ġ20 % Ġof Ġthe Ġannot ated Ġsamples Ġand Ġpresent Ġthem Ġto Ġa Ġsecond Ġannot ator Ġ( second Ġauthor ). ĠSince Ġat Ġits Ġcore , Ġthe Ġannotation Ġis Ġa Ġmulti - label Ġtask , Ġwe Ġreport Ġthe Ġinter - annot ator Ġagreement Ġby Ġcomputing Ġthe Ġ( micro - aver aged ) ĠF 1 Ġscore , Ġwhere Ġwe Ġtreat Ġthe Ġfirst Ġannot ator 's Ġlabels Ġas Ġgold . ĠTable ĠT AB REF 21 Ġreports Ġthe Ġagreement Ġscores , Ġthe Ġoverall Ġ( micro ) Ġaverage ĠF 1 Ġscore Ġof Ġthe Ġannotations Ġis Ġ0 . 82 , Ġwhich Ġmeans Ġthat Ġon Ġaverage , Ġmore Ġthan Ġtwo Ġthirds Ġof Ġthe Ġoverall Ġannot ated Ġlabels Ġwere Ġagreed Ġon Ġby Ġboth Ġannot ators . ĠWe Ġdeem Ġthis Ġsatisfactory , Ġgiven Ġthe Ġcomplexity Ġof Ġthe Ġannotation Ġschema . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis </s> We Ġpresent Ġa Ġconcise Ġview Ġof Ġthe Ġannotation Ġresults Ġin ĠFigure ĠFIG REF 23 . ĠThe Ġfull Ġannotation Ġresults Ġcan Ġbe Ġfound Ġin ĠAppendix Ġ. ĠWe Ġcentre Ġour Ġdiscussion Ġaround Ġthe Ġfollowing Ġmain Ġpoints : </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠL ingu istic ĠFeatures </s> As Ġobserved Ġin ĠFigure ĠFIG REF 23 Ġthe Ġgold Ġstandards Ġfeature Ġa Ġhigh Ġdegree Ġof ĠRed und ancy , Ġpe aking Ġat Ġ76 % Ġof Ġthe Ġannot ated ĠHot pot Q A Ġsamples Ġand Ġsyn onyms Ġand Ġparaph r ases Ġ( lab elled ĠSyn onym ), Ġwith ĠRe Co R d Ġsamples Ġcontaining Ġ58 % Ġof Ġthem , Ġlikely Ġto Ġbe Ġattributed Ġto Ġthe Ġelabor ating Ġtype Ġof Ġdiscourse Ġof Ġthe Ġdataset Ġsources Ġ( ency cl op edia Ġand Ġnews wire ). ĠThis Ġis , Ġhowever , Ġnot Ġsurprising , Ġas Ġit Ġis Ġfairly Ġwell Ġunderstood Ġin Ġthe Ġliterature Ġthat Ġcurrent Ġstate - of - the - art Ġmodels Ġperform Ġwell Ġon Ġdistinguishing Ġrelevant Ġwords Ġand Ġphrases Ġfrom Ġredundant Ġones ĠB IB REF 32 . ĠAdditionally , Ġthe Ġrepresent ational Ġcapability Ġof Ġsyn onym Ġrelationships Ġof Ġword Ġembed d ings Ġhas Ġbeen Ġinvestigated Ġand Ġis Ġwell Ġknown ĠB IB REF 33 . ĠFinally , Ġwe Ġobserve Ġthe Ġpresence Ġof Ġsynt actic Ġfeatures , Ġsuch Ġas Ġambiguous Ġrelative Ġclauses , Ġapp os itions Ġand Ġad verb ial Ġphrases , Ġ( Rel Adv App Ġ40 % Ġin ĠHot pot Q A Ġand ĠRe Co R d ) Ġand Ġthose Ġintroducing Ġvariance , Ġconcrete ly Ġswitching Ġbetween Ġverbal Ġand Ġnominal Ġstyles Ġ( e . g . ĠNom inal isation Ġ10 % Ġin ĠHot pot Q A ) Ġand Ġfrom Ġpassive Ġto Ġactive Ġvoice Ġ( Voice , Ġ8 % Ġin ĠHot pot Q A ). </s> Sy nt actic Ġfeatures Ġcontributing Ġto Ġvariety Ġand Ġambiguity Ġthat Ġwe Ġdid Ġnot Ġobserve Ġin Ġour Ġsamples Ġare Ġthe Ġexploitation Ġof Ġverb Ġsymmetry , Ġthe Ġuse Ġof Ġd ative Ġand Ġgen itive Ġcases Ġor Ġambiguous Ġpre pos itions Ġand Ġcoordination Ġscope Ġ( respect ively ĠSy mm etry , ĠD ative , ĠGen itive , ĠPre pos itions , ĠScope ). ĠTherefore Ġwe Ġcannot Ġestablish Ġwhether Ġmodels Ġare Ġcapable Ġof Ġdealing Ġwith Ġthose Ġfeatures Ġby Ġevaluating Ġthem Ġon Ġthose Ġgold Ġstandards . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠFact ual ĠCorrect ness </s> We Ġidentify Ġthree Ġcommon Ġsources Ġthat Ġsurface Ġin Ġdifferent Ġproblems Ġregarding Ġan Ġanswer 's Ġfactual Ġcorrectness , Ġas Ġreported Ġin ĠFigure ĠFIG REF 23 Ġand Ġillustrate Ġtheir Ġinstant iations Ġin ĠTable ĠT AB REF 31 : </s> Design ĠCon str aints : ĠCho osing Ġthe Ġtask Ġdesign Ġand Ġthe Ġdata Ġcollection Ġmethod Ġintroduces Ġsome Ġconstraints Ġthat Ġlead Ġto Ġfact ually Ġdeb atable Ġexamples . ĠFor Ġexample , Ġa Ġspan Ġmight Ġhave Ġbeen Ġarbitrarily Ġselected Ġfrom Ġmultiple Ġspans Ġthat Ġpotentially Ġanswer Ġa Ġquestion , Ġbut Ġonly Ġa Ġsingle Ġcontinuous Ġanswer Ġspan Ġper Ġquestion Ġis Ġallowed Ġby Ġdesign , Ġas Ġobserved Ġin Ġthe ĠNews Q A Ġand ĠMs Marco Ġsamples Ġ( 32 % Ġand Ġ34 % Ġexamples Ġannot ated Ġas ĠDe bat able Ġwith Ġ16 % Ġand Ġ53 % Ġthereof Ġexhibiting Ġarbitrary Ġselection , Ġrespectively ). ĠSometimes , Ġwhen Ġadditional Ġpassages Ġare Ġadded Ġafter Ġthe Ġannotation Ġstep , Ġthey Ġcan Ġby Ġchance Ġcontain Ġpassages Ġthat Ġanswer Ġthe Ġquestion Ġmore Ġprecisely Ġthan Ġthe Ġoriginal Ġspan , Ġas Ġseen Ġin ĠHot pot Q A Ġ( 16 % ĠDe bat able Ġsamples , Ġ25 % Ġof Ġthem Ġdue Ġto Ġarbitrary Ġselection ). ĠIn Ġthe Ġcase Ġof ĠMulti RC Ġit Ġappears Ġto Ġbe Ġinconsistent , Ġwhether Ġmultiple Ġcorrect Ġanswer Ġchoices Ġare Ġexpected Ġto Ġbe Ġcorrect Ġin Ġisolation Ġor Ġin Ġconjunction Ġ( 28 % ĠDe bat able Ġwith Ġ29 % Ġof Ġthem Ġexhibiting Ġthis Ġproblem ). ĠThis Ġmight Ġprovide Ġan Ġexplanation Ġto Ġits Ġrelatively Ġweak Ġhuman Ġbaseline Ġperformance Ġof Ġ84 % ĠF 1 Ġscore ĠB IB REF 31 . </s> Weak ĠQuality Ġassurance : ĠWhen Ġthe Ġ( typically Ġcrowd - s ourced ) Ġannotations Ġare Ġnot Ġappropriately Ġvalidated , Ġincorrect Ġexamples Ġwill Ġfind Ġtheir Ġway Ġinto Ġthe Ġgold Ġstandards . ĠThis Ġtypically Ġresults Ġin Ġfact ually Ġwrong Ġexpected Ġanswers Ġ( i . e . Ġwhen Ġa Ġmore Ġcorrect Ġanswer Ġis Ġpresent Ġin Ġthe Ġcontext ) Ġor Ġa Ġquestion Ġis Ġexpected Ġto Ġbe ĠUn answer able , Ġbut Ġis Ġactually Ġanswer able Ġfrom Ġthe Ġprovided Ġcontext . ĠThe Ġlatter Ġis Ġobserved Ġin ĠMs Marco Ġ( 83 % Ġof Ġexamples Ġannot ated Ġas ĠWrong ) Ġand ĠNews Q A , Ġwhere Ġ60 % Ġof Ġthe Ġexamples Ġannot ated Ġas ĠWrong Ġare ĠUn answer able Ġwith Ġan Ġanswer Ġpresent . </s> Ar bit rary ĠPrecision : ĠThere Ġappears Ġto Ġbe Ġno Ġclear Ġguideline Ġon Ġhow Ġprecise Ġthe Ġanswer Ġis Ġexpected Ġto Ġbe , Ġwhen Ġthe Ġpassage Ġexpresses Ġthe Ġanswer Ġin Ġvarying Ġgran ular ities . ĠWe Ġannot ated Ġinstances Ġas ĠDe bat able Ġwhen Ġthe Ġexpected Ġanswer Ġwas Ġnot Ġthe Ġmost Ġprecise Ġgiven Ġthe Ġcontext Ġ( 44 % Ġand Ġ29 % Ġof ĠDe bat able Ġinstances Ġin ĠNews Q A Ġand ĠMulti RC , Ġrespectively ). </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠSem antics - al tering Ġgram matical Ġmodifiers </s> We Ġtook Ġinterest Ġin Ġwhether Ġany Ġof Ġthe Ġbenchmarks Ġcontain Ġwhat Ġwe Ġcall Ġdistracting Ġlex ical Ġfeatures Ġ( or Ġdistract ors ): Ġgram matical Ġmodifiers Ġthat Ġalter Ġthe Ġsemantics Ġof Ġa Ġsentence Ġfor Ġthe Ġfinal Ġtask Ġof Ġanswering Ġthe Ġgiven Ġquestion Ġwhile Ġpreserving Ġa Ġsimilar Ġlex ical Ġform . ĠAn Ġexample Ġof Ġsuch Ġfeatures Ġare Ġcues Ġfor Ġ( double ) ĠNeg ation Ġ( e . g ., ĠâĢ ľ no âĢ Ŀ , ĠâĢ ľ not âĢ Ŀ ), Ġwhich Ġwhen Ġintroduced Ġin Ġa Ġsentence , Ġreverse Ġits Ġmeaning . ĠOther Ġexamples Ġinclude Ġmodifiers Ġden oting ĠRestrict ivity , ĠFact ivity Ġand ĠReason ing Ġ( such Ġas ĠMon oton icity Ġand ĠCond itional Ġcues ). ĠExamples Ġof Ġquestion - answer Ġpairs Ġcontaining Ġa Ġdistract or Ġare Ġshown Ġin ĠTable ĠFIG REF 37 . </s> We Ġposit Ġthat Ġthe Ġpresence Ġof Ġsuch Ġdistract ors Ġwould Ġallow Ġfor Ġevaluating Ġreading Ġcomprehension Ġbeyond Ġpotential Ġsimple Ġword Ġmatching . ĠHowever , Ġwe Ġobserve Ġno Ġpresence Ġof Ġsuch Ġfeatures Ġin Ġthe Ġbenchmarks Ġ( be yond ĠNeg ation Ġin ĠDR OP , ĠRe Co R d Ġand ĠHot pot Q A , Ġwith Ġ4 %, Ġ4 % Ġand Ġ2 % Ġrespectively ). ĠThis Ġresults Ġin Ġgold Ġstandards Ġthat Ġclearly Ġexpress Ġthe Ġevidence Ġrequired Ġto Ġobtain Ġthe Ġanswer , Ġlacking Ġmore Ġchallenging , Ġi . e ., Ġdistracting , Ġsentences Ġthat Ġcan Ġassess Ġwhether Ġa Ġmodel Ġcan Ġtruly Ġunderstand Ġmeaning . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠOther </s> In Ġthe ĠFigure ĠFIG REF 23 Ġwe Ġobserve Ġthat ĠOper ational Ġand ĠAr ithmetic Ġreasoning Ġmoderately Ġ( 6 % Ġto Ġ8 % Ġcombined ) Ġappears ĠâĢ ľ in Ġthe Ġwild âĢ Ŀ , Ġi . e . Ġwhen Ġnot Ġenforced Ġby Ġthe Ġdata Ġdesign Ġas Ġis Ġthe Ġcase Ġwith ĠHot pot Q A Ġ( 80 % ĠOperations Ġcombined ) Ġor ĠDR OP Ġ( 68 % ĠAr ithmetic Ġcombined ). ĠCa usal Ġreasoning Ġis Ġ( ex clusively ) Ġpresent Ġin ĠMulti RC Ġ( 32 %), Ġwhereas ĠTem poral Ġand ĠSp atial Ġreasoning Ġrequirements Ġseem Ġto Ġnot Ġnaturally Ġemerge Ġin Ġgold Ġstandards . ĠIn ĠRe Co R d , Ġa Ġfraction Ġof Ġ38 % Ġquestions Ġcan Ġonly Ġbe Ġanswered ĠBy ĠEx clusion Ġof Ġevery Ġother Ġcandidate , Ġdue Ġto Ġthe Ġdesign Ġchoice Ġof Ġallowing Ġquestions Ġwhere Ġthe Ġrequired Ġinformation Ġto Ġanswer Ġthem Ġis Ġnot Ġfully Ġexpressed Ġin Ġthe Ġaccompanying Ġparagraph . </s> Therefore , Ġit Ġis Ġalso Ġa Ġlittle Ġsurprising Ġto Ġobserve Ġthat ĠRe Co R d Ġrequires Ġexternal Ġresources Ġwith Ġregard Ġto Ġknowledge , Ġas Ġseen Ġin ĠFigure ĠFIG REF 23 . ĠMulti RC Ġrequires Ġtechnical Ġor Ġmore Ġprecisely Ġbasic Ġscientific Ġknowledge Ġ( 6 % ĠTechnical / Scient ific ), Ġas Ġa Ġportion Ġof Ġparagraphs Ġis Ġextracted Ġfrom Ġelementary Ġschool Ġscience Ġtextbooks ĠB IB REF 31 . ĠOther Ġbenchmarks Ġmoderately Ġprobe Ġfor Ġfactual Ġknowledge Ġ( 0 % Ġto Ġ4 % Ġacross Ġall Ġcategories ), Ġwhile ĠInt uitive Ġknowledge Ġis Ġrequired Ġto Ġderive Ġanswers Ġin Ġeach Ġgold Ġstandard . </s> It Ġis Ġalso Ġworth Ġpointing Ġout , Ġas Ġdone Ġin ĠFigure ĠFIG REF 23 , Ġthat Ġalthough ĠMulti RC Ġand ĠMs Marco Ġare Ġnot Ġmod elled Ġas Ġa Ġspan Ġselection Ġproblem , Ġtheir Ġsamples Ġstill Ġcontain Ġ50 % Ġand Ġ66 % Ġof Ġanswers Ġthat Ġare Ġdirectly Ġtaken Ġfrom Ġthe Ġcontext . ĠDR OP Ġcontains Ġthe Ġbiggest Ġfraction Ġof Ġgenerated Ġanswers Ġ( 60 %), Ġdue Ġto Ġthe Ġrequirement Ġof Ġarithmetic Ġoperations . </s> To Ġconclude Ġour Ġanalysis , Ġwe Ġobserve Ġsimilar Ġdistributions Ġof Ġlinguistic Ġfeatures Ġand Ġreasoning Ġpatterns , Ġexcept Ġwhere Ġthere Ġare Ġconstraints Ġenforced Ġby Ġdataset Ġdesign , Ġannotation Ġguidelines Ġor Ġsource Ġtext Ġchoice . ĠFurthermore , Ġcareful Ġconsideration Ġof Ġdesign Ġchoices Ġ( such Ġas Ġsingle - span Ġanswers ) Ġis Ġrequired , Ġto Ġavoid Ġimpair ing Ġthe Ġfactual Ġcorrectness Ġof Ġdatasets , Ġas Ġpure Ġcrowd - worker Ġagreement Ġseems Ġnot Ġsufficient Ġin Ġmultiple Ġcases . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQuant itative ĠResults Ġ: :: ĠLex ical Ġoverlap </s> We Ġused Ġthe Ġscores Ġassigned Ġby Ġour Ġproposed Ġset Ġof Ġmetrics Ġ( disc ussed Ġin ĠSection ĠSEC REF 11 ĠDimensions Ġof ĠInterest : ĠComplex ity ) Ġto Ġpredict Ġthe Ġsupporting Ġfacts Ġin Ġthe Ġgold Ġstandard Ġsamples Ġ( that Ġwe Ġincluded Ġin Ġour Ġmanual Ġannotation ). ĠCon crete ly , Ġwe Ġused Ġthe Ġfollowing Ġfive Ġfeatures Ġcapturing Ġlex ical Ġoverlap : Ġ( i ) Ġthe Ġnumber Ġof Ġwords Ġoccurring Ġin Ġsentence Ġand Ġquestion , Ġ( ii ) Ġthe Ġlength Ġof Ġthe Ġlongest Ġn - gram Ġshared Ġby Ġsentence Ġand Ġquestion , Ġwhether Ġa Ġ( iii ) Ġun i - Ġand Ġ( iv ) Ġbig ram Ġfrom Ġthe Ġquestion Ġis Ġunique Ġto Ġa Ġsentence , Ġand Ġ( v ) Ġthe Ġsentence Ġindex , Ġas Ġinput Ġto Ġa Ġlog istic Ġregression Ġclass ifier . ĠWe Ġoptim ised Ġon Ġeach Ġsample Ġleaving Ġone Ġexample Ġfor Ġevaluation . ĠWe Ġcompute Ġthe Ġaverage ĠPrecision , ĠRecall Ġand ĠF 1 Ġscore Ġby Ġmeans Ġof Ġleave - one - out Ġvalidation Ġwith Ġevery Ġsample Ġentry . ĠThe Ġaveraged Ġresults Ġafter Ġ5 Ġruns Ġare Ġreported Ġin ĠTable ĠT AB REF 41 . </s> We Ġobserve Ġthat Ġeven Ġby Ġusing Ġonly Ġour Ġfive Ġfeatures Ġbased Ġlex ical Ġoverlap , Ġthe Ġsimple Ġlog istic Ġregression Ġbaseline Ġis Ġable Ġto Ġseparate Ġout Ġthe Ġsupporting Ġfacts Ġfrom Ġthe Ġcontext Ġto Ġa Ġvarying Ġdegree . ĠThis Ġis Ġin Ġline Ġwith Ġthe Ġlack Ġof Ġsemantics - al tering Ġgram matical Ġmodifiers Ġdiscussed Ġin Ġthe Ġqualitative Ġanalysis Ġsection Ġabove . ĠThe Ġclass ifier Ġperforms Ġbest Ġon ĠDR OP Ġ( 66 % ĠF 1 ) Ġand ĠMulti RC Ġ( 40 % ĠF 1 ), Ġwhich Ġmeans Ġthat Ġlex ical Ġcues Ġcan Ġconsiderably Ġfacilitate Ġthe Ġsearch Ġfor Ġthe Ġanswer Ġin Ġthose Ġgold Ġstandards . ĠOn ĠMulti RC , Ġy ad av 2019 quick Ġcome Ġto Ġa Ġsimilar Ġconclusion , Ġby Ġusing Ġa Ġmore Ġsophisticated Ġapproach Ġbased Ġon Ġoverlap Ġbetween Ġquestion , Ġsentence Ġand Ġanswer Ġchoices . </s> Sur prisingly , Ġthe Ġclass ifier Ġis Ġable Ġto Ġpick Ġup Ġa Ġsignal Ġfrom Ġsupporting Ġfacts Ġeven Ġon Ġdata Ġthat Ġhas Ġbeen Ġpr uned Ġagainst Ġlex ical Ġoverlap Ġhe ur istics Ġby Ġpop ulating Ġthe Ġcontext Ġwith Ġadditional Ġdocuments Ġthat Ġhave Ġhigh Ġoverlap Ġscores Ġwith Ġthe Ġquestion . ĠThis Ġresults Ġin Ġsignificantly Ġhigher Ġscores Ġthan Ġwhen Ġguessing Ġrandomly Ġ( Hot pot Q A Ġ26 % ĠF 1 , Ġand ĠMs Marco Ġ11 % ĠF 1 ). ĠWe Ġobserve Ġsimilar Ġresults Ġin Ġthe Ġcase Ġthe Ġlength Ġof Ġthe Ġquestion Ġleaves Ġfew Ġcandidates Ġto Ġcompute Ġoverlap Ġwith Ġ$ 6 . 3 $ Ġand Ġ$ 7 . 3 $ Ġtokens Ġon Ġaverage Ġfor ĠMs Marco Ġand ĠNews Q A Ġ( 26 % ĠF 1 ), Ġcompared Ġto Ġ$ 16 . 9 $ Ġtokens Ġon Ġaverage Ġfor Ġthe Ġremaining Ġfour Ġdataset Ġsamples . </s> Finally , Ġit Ġis Ġworth Ġmentioning Ġthat Ġalthough Ġthe Ġqueries Ġin ĠRe Co R d Ġare Ġexplicitly Ġindependent Ġfrom Ġthe Ġpassage , Ġthe Ġlinear Ġclass ifier Ġis Ġstill Ġcapable Ġof Ġachieving Ġ34 % ĠF 1 Ġscore Ġin Ġpredicting Ġthe Ġsupporting Ġfacts . </s> However , Ġneural Ġnetworks Ġperform Ġsignificantly Ġbetter Ġthan Ġour Ġadmittedly Ġcrude Ġbaseline Ġ( e . g . Ġ66 % ĠF 1 Ġfor Ġsupporting Ġfacts Ġclassification Ġon ĠHot pot Q A ĠB IB REF 14 ), Ġalbeit Ġutil ising Ġmore Ġtraining Ġexamples , Ġand Ġa Ġricher Ġsentence Ġrepresentation . ĠThis Ġfacts Ġimplies Ġthat Ġthose Ġneural Ġmodels Ġare Ġcapable Ġof Ġsolving Ġmore Ġchallenging Ġproblems Ġthan Ġsimple ĠâĢ ľ text Ġmatching âĢ Ŀ Ġas Ġperformed Ġby Ġthe Ġlog istic Ġregression Ġbaseline . ĠHowever , Ġthey Ġstill Ġcircumvent Ġactual Ġreading Ġcomprehension Ġas Ġthe Ġrespective Ġgold Ġstandards Ġare Ġof Ġlimited Ġsuit ability Ġto Ġevaluate Ġthis ĠB IB REF 34 , ĠB IB REF 35 . ĠThis Ġsuggests Ġan Ġexciting Ġfuture Ġresearch Ġdirection , Ġthat Ġis Ġcategor ising Ġthe Ġscale Ġbetween Ġtext Ġmatching Ġand Ġreading Ġcomprehension Ġmore Ġprecisely Ġand Ġrespectively Ġpositioning Ġstate - of - the - art Ġmodels Ġthere on . </s> Related ĠWork </s> Although Ġnot Ġas Ġprominent Ġas Ġthe Ġresearch Ġon Ġnovel Ġarchitecture , Ġthere Ġhas Ġbeen Ġsteady Ġprogress Ġin Ġcritically Ġinvestigating Ġthe Ġdata Ġand Ġevaluation Ġaspects Ġof ĠN LP Ġand Ġmachine Ġlearning Ġin Ġgeneral Ġand ĠM RC Ġin Ġparticular . </s> Related ĠWork Ġ: :: ĠAd vers arial ĠEvaluation </s> The Ġauthors Ġof Ġthe ĠAdd Sent Ġalgorithm ĠB IB REF 11 Ġshow Ġthat ĠM RC Ġmodels Ġtrained Ġand Ġevaluated Ġon Ġthe ĠS Qu AD Ġdataset Ġpay Ġtoo Ġlittle Ġattention Ġto Ġdetails Ġthat Ġmight Ġchange Ġthe Ġsemantics Ġof Ġa Ġsentence , Ġand Ġpropose Ġa Ġcrowd - s ourcing Ġbased Ġmethod Ġto Ġgenerate Ġadversary Ġexamples Ġto Ġexploit Ġthat Ġweakness . ĠThis Ġmethod Ġwas Ġfurther Ġadapted Ġto Ġbe Ġfully Ġautomated ĠB IB REF 36 Ġand Ġapplied Ġto Ġdifferent Ġgold Ġstandards ĠB IB REF 35 . ĠOur Ġproposed Ġapproach Ġdiffers Ġin Ġthat Ġwe Ġaim Ġto Ġprovide Ġqualitative Ġjust ifications Ġfor Ġthose Ġquant itatively Ġmeasured Ġissues . </s> Related ĠWork Ġ: :: ĠSan ity ĠBas elines </s> Another Ġline Ġof Ġresearch Ġestablishes Ġsane Ġbas elines Ġto Ġprovide Ġmore Ġmeaningful Ġcontext Ġto Ġthe Ġraw Ġperformance Ġscores Ġof Ġevaluated Ġmodels . ĠWhen Ġremoving Ġintegral Ġparts Ġof Ġthe Ġtask Ġformulation Ġsuch Ġas Ġquestion , Ġthe Ġtextual Ġpassage Ġor Ġparts Ġthereof ĠB IB REF 37 Ġor Ġrestricting Ġmodel Ġcomplexity Ġby Ġdesign Ġin Ġorder Ġto Ġsuppress Ġsome Ġrequired Ġform Ġof Ġreasoning ĠB IB REF 38 , Ġmodels Ġare Ġstill Ġable Ġto Ġperform Ġcompar ably Ġto Ġthe Ġstate - of - the - art . ĠThis Ġraises Ġconcerns Ġabout Ġthe Ġperceived Ġbenchmark Ġcomplexity Ġand Ġis Ġrelated Ġto Ġour Ġwork Ġin Ġa Ġbroader Ġsense Ġas Ġone Ġof Ġour Ġgoals Ġis Ġto Ġestimate Ġthe Ġcomplexity Ġof Ġbenchmarks . </s> Related ĠWork Ġ: :: ĠBench mark Ġevaluation Ġin ĠN LP </s> Beyond ĠM RC , Ġefforts Ġsimilar Ġto Ġours Ġthat Ġpursue Ġthe Ġgoal Ġof Ġanalys ing Ġthe Ġevaluation Ġof Ġestablished Ġdatasets Ġexist Ġin ĠNatural ĠLanguage ĠIn ference ĠB IB REF 13 , ĠB IB REF 12 . ĠTheir Ġanalyses Ġreveal Ġthe Ġexistence Ġof Ġbiases Ġin Ġtraining Ġand Ġevaluation Ġdata Ġthat Ġcan Ġbe Ġapprox imated Ġwith Ġsimple Ġmajority - based Ġhe ur istics . ĠBecause Ġof Ġthese Ġbiases , Ġtrained Ġmodels Ġfail Ġto Ġextract Ġthe Ġsemantics Ġthat Ġare Ġrequired Ġfor Ġthe Ġcorrect Ġinference . ĠFurthermore , Ġa Ġfair Ġshare Ġof Ġwork Ġwas Ġdone Ġto Ġreveal Ġgender Ġbias Ġin Ġcore ference Ġresolution Ġdatasets Ġand Ġmodels ĠB IB REF 39 , ĠB IB REF 40 , ĠB IB REF 41 . </s> Related ĠWork Ġ: :: ĠAn notation ĠTax onom ies </s> Finally , Ġrelated Ġto Ġour Ġframework Ġare Ġworks Ġthat Ġintroduce Ġannotation Ġcategories Ġfor Ġgold Ġstandards Ġevaluation . ĠCon crete ly , Ġwe Ġbuild Ġour Ġannotation Ġframework Ġaround Ġlinguistic Ġfeatures Ġthat Ġwere Ġintroduced Ġin Ġthe ĠGL UE Ġsuite ĠB IB REF 42 Ġand Ġthe Ġreasoning Ġcategories Ġintroduced Ġin Ġthe ĠWorld Tree Ġdataset ĠB IB REF 19 . ĠA Ġqualitative Ġanalysis Ġcomplementary Ġto Ġours , Ġwith Ġfocus Ġon Ġthe Ġun answer ability Ġpatterns Ġin Ġdatasets Ġthat Ġfeature Ġun answer able Ġquestions Ġwas Ġdone Ġby ĠY ats kar 2019 . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġintroduce Ġa Ġnovel Ġframework Ġto Ġcharacter ise Ġmachine Ġreading Ġcomprehension Ġgold Ġstandards . ĠThis Ġframework Ġhas Ġpotential Ġapplications Ġwhen Ġcomparing Ġdifferent Ġgold Ġstandards , Ġconsidering Ġthe Ġdesign Ġchoices Ġfor Ġa Ġnew Ġgold Ġstandard Ġand Ġperforming Ġqualitative Ġerror Ġanalyses Ġfor Ġa Ġproposed Ġapproach . </s> Furthermore Ġwe Ġapplied Ġthe Ġframework Ġto Ġanalyse Ġpopular Ġstate - of - the - art Ġgold Ġstandards Ġfor Ġmachine Ġreading Ġcomprehension : ĠWe Ġreveal Ġissues Ġwith Ġtheir Ġfactual Ġcorrectness , Ġshow Ġthe Ġpresence Ġof Ġlex ical Ġcues Ġand Ġwe Ġobserve Ġthat Ġsemantics - al tering Ġgram matical Ġmodifiers Ġare Ġmissing Ġin Ġall Ġof Ġthe Ġinvestigated Ġgold Ġstandards . ĠStud ying Ġhow Ġto Ġintroduce Ġthose Ġmodifiers Ġinto Ġgold Ġstandards Ġand Ġobserving Ġwhether Ġstate - of - the - art ĠM RC Ġmodels Ġare Ġcapable Ġof Ġperforming Ġreading Ġcomprehension Ġon Ġtext Ġcontaining Ġthem , Ġis Ġa Ġfuture Ġresearch Ġgoal . </s> A Ġfuture Ġline Ġof Ġresearch Ġis Ġto Ġextend Ġthe Ġframework Ġto Ġbe Ġable Ġto Ġidentify Ġthe Ġdifferent Ġtypes Ġof Ġexplo itable Ġcues Ġsuch Ġas Ġquestion Ġor Ġentity Ġtyping Ġand Ġconcrete Ġoverlap Ġpatterns . ĠThis Ġwill Ġallow Ġthe Ġframework Ġto Ġserve Ġas Ġan Ġinterpret able Ġestimate Ġof Ġreading Ġcomprehension Ġcomplexity Ġof Ġgold Ġstandards . ĠFinally , Ġinvestigating Ġgold Ġstandards Ġunder Ġthis Ġframework Ġwhere ĠM RC Ġmodels Ġoutper form Ġthe Ġhuman Ġbaseline Ġ( e . g . ĠS Qu AD ) Ġwill Ġcontribute Ġto Ġa Ġdeeper Ġunderstanding Ġof Ġthe Ġseemingly Ġsuperb Ġperformance Ġof Ġdeep Ġlearning Ġapproaches Ġon Ġthem .
Introduction There is a recent spark of interest in the task of Question Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC). This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures BIBREF0, BIBREF1 that allow for efficient optimisation of neural networks consisting of multiple layers, hardware designed for deep learning purposes and software frameworks BIBREF2, BIBREF3 that allow efficient development and testing of novel approaches. These factors enable researchers to produce models that are pre-trained on large scale corpora and provide contextualised word representations BIBREF4 that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC BIBREF5. Another important factor that led to the recent success in MRC-related tasks is the widespread availability of various large datasets, e.g., SQuAD BIBREF6, that provide sufficient examples for optimising statistical models. The combination of these factors yields notable results, even surpassing human performance BIBREF7. MRC is a generic task format that can be used to probe for various natural language understanding capabilities BIBREF8. Therefore it is crucially important to establish a rigorous evaluation methodology in order to be able to draw reliable conclusions from conducted experiments. While increasing effort is put into the evaluation of novel architectures, such as keeping the evaluation data from public access to prevent unintentional overfitting to test data, performing ablation and error studies and introducing novel metrics BIBREF9, surprisingly little is done to establish the quality of the data itself. Additionally, recent research arrived at worrisome findings: the data of those gold standards, which is usually gathered involving a crowd-sourcing step, suffers from flaws in design BIBREF10 or contains overly specific keywords BIBREF11. Furthermore, these gold standards contain “annotation artefacts”, cues that lead models into focusing on superficial aspects of text, such as lexical overlap and word order, instead of actual language understanding BIBREF12, BIBREF13. These weaknesses cast some doubt on whether the data can reliably evaluate the reading comprehension performance of the models they evaluate, i.e. if the models are indeed being assessed for their capability to read. Figure FIGREF3 shows an example from HotpotQA BIBREF14, a dataset that exhibits the last kind of weakness mentioned above, i.e., the presence of unique keywords in both the question and the passage (in close proximity to the expected answer). An evaluation methodology is vital to the fine-grained understanding of challenges associated with a single gold standard, in order to understand in greater detail which capabilities of MRC models it evaluates. More importantly, it allows to draw comparisons between multiple gold standards and between the results of respective state-of-the-art models that are evaluated on them. In this work, we take a step back and propose a framework to systematically analyse MRC evaluation data, typically a set of questions and expected answers to be derived from accompanying passages. Concretely, we introduce a methodology to categorise the linguistic complexity of the textual data and the reasoning and potential external knowledge required to obtain the expected answer. Additionally we propose to take a closer look at the factual correctness of the expected answers, a quality dimension that appears under-explored in literature. We demonstrate the usefulness of the proposed framework by applying it to precisely describe and compare six contemporary MRC datasets. Our findings reveal concerns about their factual correctness, the presence of lexical cues that simplify the task of reading comprehension and the lack of semantic altering grammatical modifiers. We release the raw data comprised of 300 paragraphs, questions and answers richly annotated under the proposed framework as a resource for researchers developing natural language understanding models and datasets to utilise further. To the best of our knowledge this is the first attempt to introduce a common evaluation methodology for MRC gold standards and the first across-the-board qualitative evaluation of MRC datasets with respect to the proposed categories. Framework for MRC Gold Standard Analysis ::: Problem definition We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \ldots a_{n_A}$. $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9: Multiple choice, where the goal is to predict $A$ from a given set of choices $\mathcal {A}$. Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$. The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$. Span, where is a continuous subsequence of tokens from the paragraph ($A \subseteq P$). Flavours include multiple spans as the correct answer or $A \subseteq Q$. Free form, where $A$ is an unconstrained natural language string. A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\in \lbrace 1,\ldots , m\rbrace }$. The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\in \lbrace 1,\ldots , m\rbrace $ under a performance metric. Typical performance metrics are exact match (EM) or accuracy, i.e. the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens. The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1). Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17. Framework for MRC Gold Standard Analysis ::: Dimensions of Interest In this section we describe a methodology to categorise gold standards according to linguistic complexity, required reasoning and background knowledge, and their factual correctness. Specifically, we use those dimensions as high-level categories of a qualitative annotation schema for annotating question, expected answer and the corresponding context. We further enrich the qualitative annotations by a metric based on lexical cues in order to approximate a lower bound for the complexity of the reading comprehension task. By sampling entries from each gold standard and annotating them, we obtain measurable results and thus are able to make observations about the challenges present in that gold standard data. Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting We are interested in different types of the expected answer. We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories. It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated. It is worth mentioning that we focus our investigations on answerable questions. For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019. Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis. Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards. On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data. One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement. We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct. Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present. We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context. Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates. Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities. We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events. We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative. We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22. When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge. We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e.g. “What do Person A and Person B have in common?”) We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019. Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if ...then”) statements and the logical implications of Con-/Disjunction (i.e. “and” and “or”) in order to derive the expected answer. Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23. To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations. An example can exhibit multiple forms of reasoning. Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage. For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval. Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Knowledge Worthwhile knowing is whether the information presented in the context is sufficient to answer the question, as there is an increase of benchmarks deliberately designed to probe a model's reliance on some sort of background knowledge BIBREF24. We seek to categorise the type of knowledge required. Similar to Wang2019, on the one hand we annotate the reliance on factual knowledge, that is (Geo)political/Legal, Cultural/Historic, Technical/Scientific and Other Domain Specific knowledge about the world that can be expressed as a set of facts. On the other hand, we denote Intuitive knowledge requirements, which is challenging to express as a set of facts, such as the knowledge that a parenthetic numerical expression next to a person's name in a biography usually denotes his life span. Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27. We aim to establish which linguistic phenomena are probed by gold standards and to which degree. To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax. More specifically, we annotate features that introduce variance between the supporting facts and the question. With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording. Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e.g. “I bought her a gift” vs “I bought a gift for her”). Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e.g. “of”) and changes from nominal to verbal style and vice versa (Nominalisation). We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis. As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question). Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer. Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions. Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Complexity Finally, we want to approximate the presence of lexical cues that might simplify the reading required in order to arrive at the answer. Quantifying this allows for more reliable statements about and comparison of the complexity of gold standards, particularly regarding the evaluation of comprehension that goes beyond simple lexical matching. We propose the use of coarse metrics based on lexical overlap between question and context sentences. Intuitively, we aim to quantify how much supporting facts “stand out” from their surrounding passage context. This can be used as proxy for the capability to retrieve the answer BIBREF10. Specifically, we measure (i) the number of words jointly occurring in a question and a sentence, (ii) the length of the longest n-gram shared by question and sentence and (iii) whether a word or n-gram from the question uniquely appears in a sentence. The resulting taxonomy of the framework is shown in Figure FIGREF10. The full catalogue of features, their description, detailed annotation guideline as well as illustrating examples can be found in Appendix . Application of the Framework ::: Candidate Datasets We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15. In selecting relevant datasets, we do not consider those that are considered “solved”, i.e. where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7. Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool. This leaves us with the datasets described in Table TABREF19. For a more detailed description, we refer to Appendix . Application of the Framework ::: Annotation Task We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets. Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset. For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts. The samples were annotated by the first author of this paper, using the proposed schema. In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author). Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold. Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0.82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators. We deem this satisfactory, given the complexity of the annotation schema. Application of the Framework ::: Qualitative Analysis We present a concise view of the annotation results in Figure FIGREF23. The full annotation results can be found in Appendix . We centre our discussion around the following main points: Application of the Framework ::: Qualitative Analysis ::: Linguistic Features As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire). This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32. Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33. Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e.g. Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA). Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope). Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards. Application of the Framework ::: Qualitative Analysis ::: Factual Correctness We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31: Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples. For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively). Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection). In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem). This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31. Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards. This typically results in factually wrong expected answers (i.e. when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context. The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present. Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities. We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively). Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form. An example of such features are cues for (double) Negation (e.g., “no”, “not”), which when introduced in a sentence, reverse its meaning. Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues). Examples of question-answer pairs containing a distractor are shown in Table FIGREF37. We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching. However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively). This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i.e., distracting, sentences that can assess whether a model can truly understand meaning. Application of the Framework ::: Qualitative Analysis ::: Other In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i.e. when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined). Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards. In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph. Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23. MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31. Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard. It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context. DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations. To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice. Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases. Application of the Framework ::: Quantitative Results ::: Lexical overlap We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation). Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier. We optimised on each sample leaving one example for evaluation. We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry. The averaged results after 5 runs are reported in Table TABREF41. We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree. This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above. The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards. On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices. Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question. This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1). We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6.3$ and $7.3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16.9$ tokens on average for the remaining four dataset samples. Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts. However, neural networks perform significantly better than our admittedly crude baseline (e.g. 66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation. This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline. However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35. This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon. Related Work Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular. Related Work ::: Adversarial Evaluation The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness. This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35. Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues. Related Work ::: Sanity Baselines Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models. When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art. This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks. Related Work ::: Benchmark evaluation in NLP Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12. Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics. Because of these biases, trained models fail to extract the semantics that are required for the correct inference. Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41. Related Work ::: Annotation Taxonomies Finally, related to our framework are works that introduce annotation categories for gold standards evaluation. Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19. A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019. Conclusion In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards. This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach. Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards. Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal. A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns. This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards. Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e.g. SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them.
What features are absent from MRC gold standards that can result in potential lexical ambiguity?
71bd5db79635d48a0730163a9f2e8ef19a86cd66
What Ġfeatures Ġare Ġabsent Ġfrom ĠM RC Ġgold Ġstandards Ġthat Ġcan Ġresult Ġin Ġpotential Ġlex ical Ġambiguity ?
Rest rict ivity Ġ, ĠFact ivity Ġ, ĠCore ference Ġ
<s> What Ġfeatures Ġare Ġabsent Ġfrom ĠM RC Ġgold Ġstandards Ġthat Ġcan Ġresult Ġin Ġpotential Ġlex ical Ġambiguity ? </s> Introduction </s> There Ġis Ġa Ġrecent Ġspark Ġof Ġinterest Ġin Ġthe Ġtask Ġof ĠQuestion ĠAn sw ering Ġ( Q A ) Ġover Ġun struct ured Ġtextual Ġdata , Ġalso Ġreferred Ġto Ġas ĠMachine ĠReading ĠComp rehens ion Ġ( M RC ). ĠThis Ġis Ġmostly Ġdue Ġto Ġwide - spread Ġsuccess Ġof Ġadvances Ġin Ġvarious Ġfacets Ġof Ġdeep Ġlearning Ġrelated Ġresearch , Ġsuch Ġas Ġnovel Ġarchitectures ĠB IB REF 0 , ĠB IB REF 1 Ġthat Ġallow Ġfor Ġefficient Ġoptim isation Ġof Ġneural Ġnetworks Ġconsisting Ġof Ġmultiple Ġlayers , Ġhardware Ġdesigned Ġfor Ġdeep Ġlearning Ġpurposes Ġand Ġsoftware Ġframeworks ĠB IB REF 2 , ĠB IB REF 3 Ġthat Ġallow Ġefficient Ġdevelopment Ġand Ġtesting Ġof Ġnovel Ġapproaches . ĠThese Ġfactors Ġenable Ġresearchers Ġto Ġproduce Ġmodels Ġthat Ġare Ġpre - trained Ġon Ġlarge Ġscale Ġcorpor a Ġand Ġprovide Ġcontextual ised Ġword Ġrepresentations ĠB IB REF 4 Ġthat Ġare Ġshown Ġto Ġbe Ġa Ġvital Ġcomponent Ġtowards Ġsolutions Ġfor Ġa Ġvariety Ġof Ġnatural Ġlanguage Ġunderstanding Ġtasks , Ġincluding ĠM RC ĠB IB REF 5 . ĠAnother Ġimportant Ġfactor Ġthat Ġled Ġto Ġthe Ġrecent Ġsuccess Ġin ĠM RC - related Ġtasks Ġis Ġthe Ġwidespread Ġavailability Ġof Ġvarious Ġlarge Ġdatasets , Ġe . g ., ĠS Qu AD ĠB IB REF 6 , Ġthat Ġprovide Ġsufficient Ġexamples Ġfor Ġoptim ising Ġstatistical Ġmodels . ĠThe Ġcombination Ġof Ġthese Ġfactors Ġyields Ġnotable Ġresults , Ġeven Ġsurpass ing Ġhuman Ġperformance ĠB IB REF 7 . </s> M RC Ġis Ġa Ġgeneric Ġtask Ġformat Ġthat Ġcan Ġbe Ġused Ġto Ġprobe Ġfor Ġvarious Ġnatural Ġlanguage Ġunderstanding Ġcapabilities ĠB IB REF 8 . ĠTherefore Ġit Ġis Ġcru cially Ġimportant Ġto Ġestablish Ġa Ġrigorous Ġevaluation Ġmethodology Ġin Ġorder Ġto Ġbe Ġable Ġto Ġdraw Ġreliable Ġconclusions Ġfrom Ġconducted Ġexperiments . ĠWhile Ġincreasing Ġeffort Ġis Ġput Ġinto Ġthe Ġevaluation Ġof Ġnovel Ġarchitectures , Ġsuch Ġas Ġkeeping Ġthe Ġevaluation Ġdata Ġfrom Ġpublic Ġaccess Ġto Ġprevent Ġunintentional Ġover fitting Ġto Ġtest Ġdata , Ġperforming Ġab lation Ġand Ġerror Ġstudies Ġand Ġintroducing Ġnovel Ġmetrics ĠB IB REF 9 , Ġsurprisingly Ġlittle Ġis Ġdone Ġto Ġestablish Ġthe Ġquality Ġof Ġthe Ġdata Ġitself . ĠAdditionally , Ġrecent Ġresearch Ġarrived Ġat Ġworrisome Ġfindings : Ġthe Ġdata Ġof Ġthose Ġgold Ġstandards , Ġwhich Ġis Ġusually Ġgathered Ġinvolving Ġa Ġcrowd - s ourcing Ġstep , Ġsuffers Ġfrom Ġflaws Ġin Ġdesign ĠB IB REF 10 Ġor Ġcontains Ġoverly Ġspecific Ġkeywords ĠB IB REF 11 . ĠFurthermore , Ġthese Ġgold Ġstandards Ġcontain ĠâĢ ľ ann otation Ġarte facts âĢ Ŀ , Ġcues Ġthat Ġlead Ġmodels Ġinto Ġfocusing Ġon Ġsuperficial Ġaspects Ġof Ġtext , Ġsuch Ġas Ġlex ical Ġoverlap Ġand Ġword Ġorder , Ġinstead Ġof Ġactual Ġlanguage Ġunderstanding ĠB IB REF 12 , ĠB IB REF 13 . ĠThese Ġweaknesses Ġcast Ġsome Ġdoubt Ġon Ġwhether Ġthe Ġdata Ġcan Ġreliably Ġevaluate Ġthe Ġreading Ġcomprehension Ġperformance Ġof Ġthe Ġmodels Ġthey Ġevaluate , Ġi . e . Ġif Ġthe Ġmodels Ġare Ġindeed Ġbeing Ġassessed Ġfor Ġtheir Ġcapability Ġto Ġread . </s> Figure ĠFIG REF 3 Ġshows Ġan Ġexample Ġfrom ĠHot pot Q A ĠB IB REF 14 , Ġa Ġdataset Ġthat Ġexhibits Ġthe Ġlast Ġkind Ġof Ġweakness Ġmentioned Ġabove , Ġi . e ., Ġthe Ġpresence Ġof Ġunique Ġkeywords Ġin Ġboth Ġthe Ġquestion Ġand Ġthe Ġpassage Ġ( in Ġclose Ġproximity Ġto Ġthe Ġexpected Ġanswer ). </s> An Ġevaluation Ġmethodology Ġis Ġvital Ġto Ġthe Ġfine - gr ained Ġunderstanding Ġof Ġchallenges Ġassociated Ġwith Ġa Ġsingle Ġgold Ġstandard , Ġin Ġorder Ġto Ġunderstand Ġin Ġgreater Ġdetail Ġwhich Ġcapabilities Ġof ĠM RC Ġmodels Ġit Ġevaluates . ĠMore Ġimportantly , Ġit Ġallows Ġto Ġdraw Ġcomparisons Ġbetween Ġmultiple Ġgold Ġstandards Ġand Ġbetween Ġthe Ġresults Ġof Ġrespective Ġstate - of - the - art Ġmodels Ġthat Ġare Ġevaluated Ġon Ġthem . </s> In Ġthis Ġwork , Ġwe Ġtake Ġa Ġstep Ġback Ġand Ġpropose Ġa Ġframework Ġto Ġsystematically Ġanalyse ĠM RC Ġevaluation Ġdata , Ġtypically Ġa Ġset Ġof Ġquestions Ġand Ġexpected Ġanswers Ġto Ġbe Ġderived Ġfrom Ġaccompanying Ġpassages . ĠCon crete ly , Ġwe Ġintroduce Ġa Ġmethodology Ġto Ġcategor ise Ġthe Ġlinguistic Ġcomplexity Ġof Ġthe Ġtextual Ġdata Ġand Ġthe Ġreasoning Ġand Ġpotential Ġexternal Ġknowledge Ġrequired Ġto Ġobtain Ġthe Ġexpected Ġanswer . ĠAdditionally Ġwe Ġpropose Ġto Ġtake Ġa Ġcloser Ġlook Ġat Ġthe Ġfactual Ġcorrectness Ġof Ġthe Ġexpected Ġanswers , Ġa Ġquality Ġdimension Ġthat Ġappears Ġunder - expl ored Ġin Ġliterature . </s> We Ġdemonstrate Ġthe Ġusefulness Ġof Ġthe Ġproposed Ġframework Ġby Ġapplying Ġit Ġto Ġprecisely Ġdescribe Ġand Ġcompare Ġsix Ġcontemporary ĠM RC Ġdatasets . ĠOur Ġfindings Ġreveal Ġconcerns Ġabout Ġtheir Ġfactual Ġcorrectness , Ġthe Ġpresence Ġof Ġlex ical Ġcues Ġthat Ġsimplify Ġthe Ġtask Ġof Ġreading Ġcomprehension Ġand Ġthe Ġlack Ġof Ġsemantic Ġaltering Ġgram matical Ġmodifiers . ĠWe Ġrelease Ġthe Ġraw Ġdata Ġcomprised Ġof Ġ300 Ġparagraphs , Ġquestions Ġand Ġanswers Ġrich ly Ġannot ated Ġunder Ġthe Ġproposed Ġframework Ġas Ġa Ġresource Ġfor Ġresearchers Ġdeveloping Ġnatural Ġlanguage Ġunderstanding Ġmodels Ġand Ġdatasets Ġto Ġutil ise Ġfurther . </s> To Ġthe Ġbest Ġof Ġour Ġknowledge Ġthis Ġis Ġthe Ġfirst Ġattempt Ġto Ġintroduce Ġa Ġcommon Ġevaluation Ġmethodology Ġfor ĠM RC Ġgold Ġstandards Ġand Ġthe Ġfirst Ġacross - the - board Ġqualitative Ġevaluation Ġof ĠM RC Ġdatasets Ġwith Ġrespect Ġto Ġthe Ġproposed Ġcategories . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠProblem Ġdefinition </s> We Ġdefine Ġthe Ġtask Ġof Ġmachine Ġreading Ġcomprehension , Ġthe Ġtarget Ġapplication Ġof Ġthe Ġproposed Ġmethodology Ġas Ġfollows : ĠGiven Ġa Ġparagraph Ġ$ P $ Ġthat Ġconsists Ġof Ġtokens Ġ( words ) Ġ$ p _ 1 , Ġ\ ld ots Ġ, Ġp _{ n _ P } $ Ġand Ġa Ġquestion Ġ$ Q $ Ġthat Ġconsists Ġof Ġtokens Ġ$ q _ 1 Ġ\ ld ots Ġq _{ n _ Q } $, Ġthe Ġgoal Ġis Ġto Ġretrieve Ġan Ġanswer Ġ$ A $ Ġwith Ġtokens Ġ$ a _ 1 Ġ\ ld ots Ġa _{ n _ A } $. Ġ$ A $ Ġis Ġcommonly Ġconstrained Ġto Ġbe Ġone Ġof Ġthe Ġfollowing Ġcases ĠB IB REF 15 , Ġillustrated Ġin ĠFigure ĠFIG REF 9 : </s> Multiple Ġchoice , Ġwhere Ġthe Ġgoal Ġis Ġto Ġpredict Ġ$ A $ Ġfrom Ġa Ġgiven Ġset Ġof Ġchoices Ġ$\ math cal Ġ{ A } $. </s> Cl o ze - style , Ġwhere Ġ$ S $ Ġis Ġa Ġsentence , Ġand Ġ$ A $ Ġand Ġ$ Q $ Ġare Ġobtained Ġby Ġremoving Ġa Ġsequence Ġof Ġwords Ġsuch Ġthat Ġ$ Q Ġ= ĠS Ġ- ĠA $. ĠThe Ġtask Ġis Ġto Ġfill Ġin Ġthe Ġresulting Ġgap Ġin Ġ$ Q $ Ġwith Ġthe Ġexpected Ġanswer Ġ$ A $ Ġto Ġform Ġ$ S $. </s> Sp an , Ġwhere Ġis Ġa Ġcontinuous Ġsubsequ ence Ġof Ġtokens Ġfrom Ġthe Ġparagraph Ġ($ A Ġ\ sub set eq ĠP $ ). ĠFlav ours Ġinclude Ġmultiple Ġspans Ġas Ġthe Ġcorrect Ġanswer Ġor Ġ$ A Ġ\ sub set eq ĠQ $. </s> Free Ġform , Ġwhere Ġ$ A $ Ġis Ġan Ġuncon str ained Ġnatural Ġlanguage Ġstring . </s> A Ġgold Ġstandard Ġ$ G $ Ġis Ġcomposed Ġof Ġ$ m $ Ġentries Ġ$( Q _ i , ĠA _ i , ĠP _ i ) _{ i \ in Ġ\ l brace Ġ1 , \ ld ots Ġ, Ġm \ r brace Ġ} $. </s> The Ġperformance Ġof Ġan Ġapproach Ġis Ġestablished Ġby Ġcomparing Ġits Ġanswer Ġpredictions Ġ$ A ^ * _{ i } $ Ġon Ġthe Ġgiven Ġinput Ġ$( Q _ i , ĠT _ i ) $ Ġ( and Ġ$\ math cal Ġ{ A } _ i $ Ġfor Ġthe Ġmultiple Ġchoice Ġsetting ) Ġagainst Ġthe Ġexpected Ġanswer Ġ$ A _ i $ Ġfor Ġall Ġ$ i \ in Ġ\ l brace Ġ1 , \ ld ots Ġ, Ġm \ r brace Ġ$ Ġunder Ġa Ġperformance Ġmetric . ĠTypical Ġperformance Ġmetrics Ġare Ġexact Ġmatch Ġ( EM ) Ġor Ġaccuracy , Ġi . e . Ġthe Ġpercentage Ġof Ġexactly Ġpredicted Ġanswers , Ġand Ġthe ĠF 1 Ġscore ĠâĢĵ Ġthe Ġharmonic Ġmean Ġbetween Ġthe Ġprecision Ġand Ġthe Ġrecall Ġof Ġthe Ġpredicted Ġtokens Ġcompared Ġto Ġexpected Ġanswer Ġtokens . ĠThe Ġoverall ĠF 1 Ġscore Ġcan Ġeither Ġbe Ġcomputed Ġby Ġaveraging Ġthe ĠF 1 Ġscores Ġfor Ġevery Ġinstance Ġor Ġby Ġfirst Ġaveraging Ġthe Ġprecision Ġand Ġrecall Ġand Ġthen Ġcomputing Ġthe ĠF 1 Ġscore Ġfrom Ġthose Ġaverages Ġ( mac ro ĠF 1 ). ĠFree - text Ġanswers , Ġmeanwhile , Ġare Ġevaluated Ġby Ġmeans Ġof Ġtext Ġgeneration Ġand Ġsummar isation Ġmetrics Ġsuch Ġas ĠB LE U ĠB IB REF 16 Ġor ĠR OU GE - L ĠB IB REF 17 . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest </s> In Ġthis Ġsection Ġwe Ġdescribe Ġa Ġmethodology Ġto Ġcategor ise Ġgold Ġstandards Ġaccording Ġto Ġlinguistic Ġcomplexity , Ġrequired Ġreasoning Ġand Ġbackground Ġknowledge , Ġand Ġtheir Ġfactual Ġcorrectness . ĠSpecifically , Ġwe Ġuse Ġthose Ġdimensions Ġas Ġhigh - level Ġcategories Ġof Ġa Ġqualitative Ġannotation Ġschema Ġfor Ġannot ating Ġquestion , Ġexpected Ġanswer Ġand Ġthe Ġcorresponding Ġcontext . ĠWe Ġfurther Ġenrich Ġthe Ġqualitative Ġannotations Ġby Ġa Ġmetric Ġbased Ġon Ġlex ical Ġcues Ġin Ġorder Ġto Ġapproximate Ġa Ġlower Ġbound Ġfor Ġthe Ġcomplexity Ġof Ġthe Ġreading Ġcomprehension Ġtask . ĠBy Ġsampling Ġentries Ġfrom Ġeach Ġgold Ġstandard Ġand Ġannot ating Ġthem , Ġwe Ġobtain Ġmeasurable Ġresults Ġand Ġthus Ġare Ġable Ġto Ġmake Ġobservations Ġabout Ġthe Ġchallenges Ġpresent Ġin Ġthat Ġgold Ġstandard Ġdata . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠProblem Ġsetting </s> We Ġare Ġinterested Ġin Ġdifferent Ġtypes Ġof Ġthe Ġexpected Ġanswer . ĠWe Ġdifferentiate Ġbetween ĠSpan , Ġwhere Ġan Ġanswer Ġis Ġa Ġcontinuous Ġspan Ġtaken Ġfrom Ġthe Ġpassage , ĠPar aph r asing , Ġwhere Ġthe Ġanswer Ġis Ġa Ġparaph rase Ġof Ġa Ġtext Ġspan , ĠUn answer able , Ġwhere Ġthere Ġis Ġno Ġanswer Ġpresent Ġin Ġthe Ġcontext , Ġand ĠGener ated , Ġif Ġit Ġdoes Ġnot Ġfall Ġinto Ġany Ġof Ġthe Ġother Ġcategories . ĠIt Ġis Ġnot Ġsufficient Ġfor Ġan Ġanswer Ġto Ġrest ate Ġthe Ġquestion Ġor Ġcombine Ġmultiple ĠSpan Ġor ĠPar aph r asing Ġanswers Ġto Ġbe Ġannot ated Ġas ĠGener ated . ĠIt Ġis Ġworth Ġmentioning Ġthat Ġwe Ġfocus Ġour Ġinvestigations Ġon Ġanswer able Ġquestions . ĠFor Ġa Ġcomplementary Ġqualitative Ġanalysis Ġthat Ġcategor ises Ġun answer able Ġquestions , Ġthe Ġreader Ġis Ġreferred Ġto ĠY ats kar 2019 . </s> Furthermore , Ġwe Ġmark Ġa Ġsentence Ġas ĠSupporting ĠFact Ġif Ġit Ġcontains Ġevidence Ġrequired Ġto Ġproduce Ġthe Ġexpected Ġanswer , Ġas Ġthey Ġare Ġused Ġfurther Ġin Ġthe Ġcomplexity Ġanalysis . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠFact ual ĠCorrect ness </s> An Ġimportant Ġfactor Ġfor Ġthe Ġquality Ġof Ġa Ġbenchmark Ġis Ġits Ġfactual Ġcorrectness , Ġbecause Ġon Ġthe Ġone Ġhand , Ġthe Ġpresence Ġof Ġfact ually Ġwrong Ġor Ġdeb atable Ġexamples Ġintroduces Ġan Ġupper Ġbound Ġfor Ġthe Ġachievable Ġperformance Ġof Ġmodels Ġon Ġthose Ġgold Ġstandards . ĠOn Ġthe Ġother Ġhand , Ġit Ġis Ġhard Ġto Ġdraw Ġconclusions Ġabout Ġthe Ġcorrectness Ġof Ġanswers Ġproduced Ġby Ġa Ġmodel Ġthat Ġis Ġevaluated Ġon Ġpartially Ġincorrect Ġdata . </s> One Ġway Ġby Ġwhich Ġdevelopers Ġof Ġmodern Ġcrowd - s ourced Ġgold Ġstandards Ġensure Ġquality Ġis Ġby Ġhaving Ġthe Ġsame Ġentry Ġannot ated Ġby Ġmultiple Ġworkers ĠB IB REF 18 Ġand Ġkeeping Ġonly Ġthose Ġwith Ġhigh Ġagreement . ĠWe Ġinvestigate Ġwhether Ġthis Ġmethod Ġis Ġenough Ġto Ġestablish Ġa Ġsound Ġground Ġtruth Ġanswer Ġthat Ġis Ġunamb ig uously Ġcorrect . ĠCon crete ly Ġwe Ġannot ate Ġan Ġanswer Ġas ĠDe bat able Ġwhen Ġthe Ġpassage Ġfeatures Ġmultiple Ġplausible Ġanswers , Ġwhen Ġmultiple Ġexpected Ġanswers Ġcontradict Ġeach Ġother , Ġor Ġan Ġanswer Ġis Ġnot Ġspecific Ġenough Ġwith Ġrespect Ġto Ġthe Ġquestion Ġand Ġa Ġmore Ġspecific Ġanswer Ġis Ġpresent . ĠWe Ġannot ate Ġan Ġanswer Ġas ĠWrong Ġwhen Ġit Ġis Ġfact ually Ġwrong Ġand Ġa Ġcorrect Ġanswer Ġis Ġpresent Ġin Ġthe Ġcontext . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠRequired ĠReason ing </s> It Ġis Ġimportant Ġto Ġunderstand Ġwhat Ġtypes Ġof Ġreasoning Ġthe Ġbenchmark Ġevaluates , Ġin Ġorder Ġto Ġbe Ġable Ġto Ġacc redit Ġvarious Ġreasoning Ġcapabilities Ġto Ġthe Ġmodels Ġit Ġevaluates . ĠOur Ġproposed Ġreasoning Ġcategories Ġare Ġinspired Ġby Ġthose Ġfound Ġin Ġscientific Ġquestion Ġanswering Ġliterature ĠB IB REF 19 , ĠB IB REF 20 , Ġas Ġresearch Ġin Ġthis Ġarea Ġfocuses Ġon Ġunderstanding Ġthe Ġrequired Ġreasoning Ġcapabilities . ĠWe Ġinclude Ġreasoning Ġabout Ġthe ĠTem poral Ġsuccession Ġof Ġevents , ĠSp atial Ġreasoning Ġabout Ġdirections Ġand Ġenvironment , Ġand ĠCa usal Ġreasoning Ġabout Ġthe Ġcause - effect Ġrelationship Ġbetween Ġevents . ĠWe Ġfurther Ġannot ate Ġ( multiple - choice ) Ġanswers Ġthat Ġcan Ġonly Ġbe Ġanswered ĠBy ĠEx clusion Ġof Ġevery Ġother Ġalternative . </s> We Ġfurther Ġextend Ġthe Ġreasoning Ġcategories Ġby Ġoperational Ġlogic , Ġsimilar Ġto Ġthose Ġrequired Ġin Ġsemantic Ġparsing Ġtasks ĠB IB REF 21 , Ġas Ġsolving Ġthose Ġtasks Ġtypically Ġrequires ĠâĢ ľ multi - hop âĢ Ŀ Ġreasoning ĠB IB REF 14 , ĠB IB REF 22 . ĠWhen Ġan Ġanswer Ġcan Ġonly Ġbe Ġobtained Ġby Ġcombining Ġinformation Ġfrom Ġdifferent Ġsentences Ġjoined Ġby Ġmentioning Ġa Ġcommon Ġentity , Ġconcept , Ġdate , Ġfact Ġor Ġevent Ġ( from Ġhere Ġon Ġcalled Ġentity ), Ġwe Ġannot ate Ġit Ġas ĠBridge . ĠWe Ġfurther Ġannot ate Ġthe Ġcases , Ġwhen Ġthe Ġanswer Ġis Ġa Ġconcrete Ġentity Ġthat Ġsatisfies Ġa ĠCon str aint Ġspecified Ġin Ġthe Ġquestion , Ġwhen Ġit Ġis Ġrequired Ġto Ġdraw Ġa ĠComparison Ġof Ġmultiple Ġentities ' Ġproperties Ġor Ġwhen Ġthe Ġexpected Ġanswer Ġis Ġan ĠInter section Ġof Ġtheir Ġproperties Ġ( e . g . ĠâĢ ľ What Ġdo ĠPerson ĠA Ġand ĠPerson ĠB Ġhave Ġin Ġcommon ? âĢ Ŀ ) </s> We Ġare Ġinterested Ġin Ġthe Ġlinguistic Ġreasoning Ġcapabilities Ġprob ed Ġby Ġa Ġgold Ġstandard , Ġtherefore Ġwe Ġinclude Ġthe Ġappropriate Ġcategory Ġused Ġby ĠWang 2019 . ĠSpecifically , Ġwe Ġannot ate Ġoccurrences Ġthat Ġrequire Ġunderstanding Ġof ĠNeg ation , ĠQuant ifiers Ġ( such Ġas ĠâĢ ľ every âĢ Ŀ , ĠâĢ ľ some âĢ Ŀ , Ġor ĠâĢ ľ all âĢ Ŀ ), ĠCond itional Ġ( âĢ ľ if Ġ... then âĢ Ŀ ) Ġstatements Ġand Ġthe Ġlogical Ġimplications Ġof ĠCon - / Dis j unction Ġ( i . e . ĠâĢ ľ and âĢ Ŀ Ġand ĠâĢ ľ or âĢ Ŀ ) Ġin Ġorder Ġto Ġderive Ġthe Ġexpected Ġanswer . </s> Finally , Ġwe Ġinvestigate Ġwhether Ġarithmetic Ġreasoning Ġrequirements Ġemerge Ġin ĠM RC Ġgold Ġstandards Ġas Ġthis Ġcan Ġprobe Ġfor Ġreasoning Ġthat Ġis Ġnot Ġevaluated Ġby Ġsimple Ġanswer Ġretrieval ĠB IB REF 23 . ĠTo Ġthis Ġend , Ġwe Ġannot ate Ġthe Ġpresence Ġof Ġof ĠAdd ition Ġand ĠSub t raction , Ġanswers Ġthat Ġrequire ĠOrder ing Ġof Ġnumerical Ġvalues , ĠCount ing Ġand ĠOther Ġoccurrences Ġof Ġsimple Ġmathematical Ġoperations . </s> An Ġexample Ġcan Ġexhibit Ġmultiple Ġforms Ġof Ġreasoning . ĠNotably , Ġwe Ġdo Ġnot Ġannot ate Ġany Ġof Ġthe Ġcategories Ġmentioned Ġabove Ġif Ġthe Ġexpected Ġanswer Ġis Ġdirectly Ġstated Ġin Ġthe Ġpassage . ĠFor Ġexample , Ġif Ġthe Ġquestion Ġasks ĠâĢ ľ How Ġmany Ġtotal Ġpoints Ġwere Ġscored Ġin Ġthe Ġgame ? âĢ Ŀ Ġand Ġthe Ġpassage Ġcontains Ġa Ġsentence Ġsimilar Ġto ĠâĢ ľ The Ġtotal Ġscore Ġof Ġthe Ġgame Ġwas Ġ51 Ġpoints âĢ Ŀ , Ġit Ġdoes Ġnot Ġrequire Ġany Ġreasoning , Ġin Ġwhich Ġcase Ġwe Ġannot ate Ġit Ġas ĠRet ri eval . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠKnowledge </s> W orth while Ġknowing Ġis Ġwhether Ġthe Ġinformation Ġpresented Ġin Ġthe Ġcontext Ġis Ġsufficient Ġto Ġanswer Ġthe Ġquestion , Ġas Ġthere Ġis Ġan Ġincrease Ġof Ġbenchmarks Ġdeliberately Ġdesigned Ġto Ġprobe Ġa Ġmodel 's Ġreliance Ġon Ġsome Ġsort Ġof Ġbackground Ġknowledge ĠB IB REF 24 . ĠWe Ġseek Ġto Ġcategor ise Ġthe Ġtype Ġof Ġknowledge Ġrequired . ĠSimilar Ġto ĠWang 2019 , Ġon Ġthe Ġone Ġhand Ġwe Ġannot ate Ġthe Ġreliance Ġon Ġfactual Ġknowledge , Ġthat Ġis Ġ( Ge o ) political / Legal , ĠCultural / Hist oric , ĠTechnical / Scient ific Ġand ĠOther ĠDomain ĠSpecific Ġknowledge Ġabout Ġthe Ġworld Ġthat Ġcan Ġbe Ġexpressed Ġas Ġa Ġset Ġof Ġfacts . ĠOn Ġthe Ġother Ġhand , Ġwe Ġdenote ĠInt uitive Ġknowledge Ġrequirements , Ġwhich Ġis Ġchallenging Ġto Ġexpress Ġas Ġa Ġset Ġof Ġfacts , Ġsuch Ġas Ġthe Ġknowledge Ġthat Ġa Ġparent hetic Ġnumerical Ġexpression Ġnext Ġto Ġa Ġperson 's Ġname Ġin Ġa Ġbiography Ġusually Ġdenotes Ġhis Ġlife Ġspan . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠL ingu istic ĠComplex ity </s> Another Ġdimension Ġof Ġinterest Ġis Ġthe Ġevaluation Ġof Ġvarious Ġlinguistic Ġcapabilities Ġof ĠM RC Ġmodels ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 27 . ĠWe Ġaim Ġto Ġestablish Ġwhich Ġlinguistic Ġphenomena Ġare Ġprob ed Ġby Ġgold Ġstandards Ġand Ġto Ġwhich Ġdegree . ĠTo Ġthat Ġend , Ġwe Ġdraw Ġinspiration Ġfrom Ġthe Ġannotation Ġschema Ġused Ġby ĠWang 2019 , Ġand Ġadapt Ġit Ġaround Ġlex ical Ġsemantics Ġand Ġsyntax . </s> More Ġspecifically , Ġwe Ġannot ate Ġfeatures Ġthat Ġintroduce Ġvariance Ġbetween Ġthe Ġsupporting Ġfacts Ġand Ġthe Ġquestion . ĠWith Ġregard Ġto Ġlex ical Ġsemantics , Ġwe Ġfocus Ġon Ġthe Ġuse Ġof Ġredundant Ġwords Ġthat Ġdo Ġnot Ġalter Ġthe Ġmeaning Ġof Ġa Ġsentence Ġfor Ġthe Ġtask Ġof Ġretrieving Ġthe Ġexpected Ġanswer Ġ( Red und ancy ), Ġrequirements Ġon Ġthe Ġunderstanding Ġof Ġwords ' Ġsemantic Ġfields Ġ( Lex ical ĠEnt ail ment ) Ġand Ġthe Ġuse Ġof ĠSyn onyms Ġand ĠPar aph r ases Ġwith Ġrespect Ġto Ġthe Ġquestion Ġwording . ĠFurthermore Ġwe Ġannot ate Ġcases Ġwhere Ġsupporting Ġfacts Ġcontain ĠAb bre vi ations Ġof Ġconcepts Ġintroduced Ġin Ġthe Ġquestion Ġ( and Ġvice Ġversa ) Ġand Ġwhen Ġa ĠD ative Ġcase Ġsubstitutes Ġthe Ġuse Ġof Ġa Ġpre position Ġ( e . g . ĠâĢ ľ I Ġbought Ġher Ġa Ġgift âĢ Ŀ Ġvs ĠâĢ ľ I Ġbought Ġa Ġgift Ġfor Ġher âĢ Ŀ ). ĠRegarding Ġsyntax , Ġwe Ġannot ate Ġchanges Ġfrom Ġpassive Ġto Ġactive ĠVoice , Ġthe Ġsubstitution Ġof Ġa ĠGen itive Ġcase Ġwith Ġa Ġpre position Ġ( e . g . ĠâĢ ľ of âĢ Ŀ ) Ġand Ġchanges Ġfrom Ġnominal Ġto Ġverbal Ġstyle Ġand Ġvice Ġversa Ġ( N om inal isation ). </s> We Ġrecognise Ġfeatures Ġthat Ġadd Ġambiguity Ġto Ġthe Ġsupporting Ġfacts , Ġfor Ġexample Ġwhen Ġinformation Ġis Ġonly Ġexpressed Ġimplicitly Ġby Ġusing Ġan ĠEll ips is . ĠAs Ġopposed Ġto Ġredundant Ġwords , Ġwe Ġannot ate ĠRestrict ivity Ġand ĠFact ivity Ġmodifiers , Ġwords Ġand Ġphrases Ġwhose Ġpresence Ġdoes Ġchange Ġthe Ġmeaning Ġof Ġa Ġsentence Ġwith Ġregard Ġto Ġthe Ġexpected Ġanswer , Ġand Ġoccurrences Ġof Ġintra - Ġor Ġinter - sent ence ĠCore ference Ġin Ġsupporting Ġfacts Ġ( that Ġis Ġrelevant Ġto Ġthe Ġquestion ). ĠLastly , Ġwe Ġmark Ġambiguous Ġsynt actic Ġfeatures , Ġwhen Ġtheir Ġresolution Ġis Ġrequired Ġin Ġorder Ġto Ġobtain Ġthe Ġanswer . ĠCon crete ly , Ġwe Ġmark Ġargument Ġcollection Ġwith Ġcon - Ġand Ġdis jun ctions Ġ( List ing ) Ġand Ġambiguous ĠPre pos itions , ĠCoord ination ĠScope Ġand ĠRelative Ġclauses / Ad verb ial Ġphrases / App os itions . </s> Fram ework Ġfor ĠM RC ĠGold ĠStandard ĠAnalysis Ġ: :: ĠDimensions Ġof ĠInterest Ġ: :: ĠComplex ity </s> Finally , Ġwe Ġwant Ġto Ġapproximate Ġthe Ġpresence Ġof Ġlex ical Ġcues Ġthat Ġmight Ġsimplify Ġthe Ġreading Ġrequired Ġin Ġorder Ġto Ġarrive Ġat Ġthe Ġanswer . ĠQuant ifying Ġthis Ġallows Ġfor Ġmore Ġreliable Ġstatements Ġabout Ġand Ġcomparison Ġof Ġthe Ġcomplexity Ġof Ġgold Ġstandards , Ġparticularly Ġregarding Ġthe Ġevaluation Ġof Ġcomprehension Ġthat Ġgoes Ġbeyond Ġsimple Ġlex ical Ġmatching . ĠWe Ġpropose Ġthe Ġuse Ġof Ġcoarse Ġmetrics Ġbased Ġon Ġlex ical Ġoverlap Ġbetween Ġquestion Ġand Ġcontext Ġsentences . ĠInt uitive ly , Ġwe Ġaim Ġto Ġquantify Ġhow Ġmuch Ġsupporting Ġfacts ĠâĢ ľ stand Ġout âĢ Ŀ Ġfrom Ġtheir Ġsurrounding Ġpassage Ġcontext . ĠThis Ġcan Ġbe Ġused Ġas Ġproxy Ġfor Ġthe Ġcapability Ġto Ġretrieve Ġthe Ġanswer ĠB IB REF 10 . ĠSpecifically , Ġwe Ġmeasure Ġ( i ) Ġthe Ġnumber Ġof Ġwords Ġjointly Ġoccurring Ġin Ġa Ġquestion Ġand Ġa Ġsentence , Ġ( ii ) Ġthe Ġlength Ġof Ġthe Ġlongest Ġn - gram Ġshared Ġby Ġquestion Ġand Ġsentence Ġand Ġ( iii ) Ġwhether Ġa Ġword Ġor Ġn - gram Ġfrom Ġthe Ġquestion Ġuniquely Ġappears Ġin Ġa Ġsentence . </s> The Ġresulting Ġtax onomy Ġof Ġthe Ġframework Ġis Ġshown Ġin ĠFigure ĠFIG REF 10 . ĠThe Ġfull Ġcatalogue Ġof Ġfeatures , Ġtheir Ġdescription , Ġdetailed Ġannotation Ġguideline Ġas Ġwell Ġas Ġillustrating Ġexamples Ġcan Ġbe Ġfound Ġin ĠAppendix Ġ. </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠCandidate ĠDat as ets </s> We Ġselect Ġcontemporary ĠM RC Ġbenchmarks Ġto Ġrepresent Ġall Ġfour Ġcommonly Ġused Ġproblem Ġdefinitions ĠB IB REF 15 . ĠIn Ġselecting Ġrelevant Ġdatasets , Ġwe Ġdo Ġnot Ġconsider Ġthose Ġthat Ġare Ġconsidered ĠâĢ ľ s olved âĢ Ŀ , Ġi . e . Ġwhere Ġthe Ġstate Ġof Ġthe Ġart Ġperformance Ġsurpass es Ġhuman Ġperformance , Ġas Ġis Ġthe Ġcase Ġwith ĠS Qu AD ĠB IB REF 28 , ĠB IB REF 7 . ĠCon crete ly , Ġwe Ġselected Ġgold Ġstandards Ġthat Ġfit Ġour Ġproblem Ġdefinition Ġand Ġwere Ġpublished Ġin Ġthe Ġyears Ġ2016 Ġto Ġ2019 , Ġhave Ġat Ġleast Ġ$( 2019 Ġ- Ġpublication \ Ġyear ) Ġ\ times Ġ20 $ Ġcitations , Ġand Ġbucket Ġthem Ġaccording Ġto Ġthe Ġanswer Ġselection Ġstyles Ġas Ġdescribed Ġin ĠSection ĠSEC REF 4 ĠWe Ġrandomly Ġdraw Ġone Ġfrom Ġeach Ġbucket Ġand Ġadd Ġtwo Ġrandomly Ġdrawn Ġdatasets Ġfrom Ġthe Ġcandidate Ġpool . ĠThis Ġleaves Ġus Ġwith Ġthe Ġdatasets Ġdescribed Ġin ĠTable ĠT AB REF 19 . ĠFor Ġa Ġmore Ġdetailed Ġdescription , Ġwe Ġrefer Ġto ĠAppendix Ġ. </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠAn notation ĠTask </s> We Ġrandomly Ġselect Ġ50 Ġdistinct Ġquestion , Ġanswer Ġand Ġpassage Ġtri ples Ġfrom Ġthe Ġpublicly Ġavailable Ġdevelopment Ġsets Ġof Ġthe Ġdescribed Ġdatasets . ĠTraining , Ġdevelopment Ġand Ġthe Ġ( hidden ) Ġtest Ġset Ġare Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġdefined Ġby Ġthe Ġdata Ġcollection Ġmethod Ġof Ġthe Ġrespective Ġdataset . ĠFor Ġthose Ġcollections Ġthat Ġcontain Ġmultiple Ġquestions Ġover Ġa Ġsingle Ġpassage , Ġwe Ġensure Ġthat Ġwe Ġare Ġsampling Ġunique Ġparagraphs Ġin Ġorder Ġto Ġincrease Ġthe Ġvariety Ġof Ġinvestigated Ġtexts . </s> The Ġsamples Ġwere Ġannot ated Ġby Ġthe Ġfirst Ġauthor Ġof Ġthis Ġpaper , Ġusing Ġthe Ġproposed Ġschema . ĠIn Ġorder Ġto Ġvalidate Ġour Ġfindings , Ġwe Ġfurther Ġtake Ġ20 % Ġof Ġthe Ġannot ated Ġsamples Ġand Ġpresent Ġthem Ġto Ġa Ġsecond Ġannot ator Ġ( second Ġauthor ). ĠSince Ġat Ġits Ġcore , Ġthe Ġannotation Ġis Ġa Ġmulti - label Ġtask , Ġwe Ġreport Ġthe Ġinter - annot ator Ġagreement Ġby Ġcomputing Ġthe Ġ( micro - aver aged ) ĠF 1 Ġscore , Ġwhere Ġwe Ġtreat Ġthe Ġfirst Ġannot ator 's Ġlabels Ġas Ġgold . ĠTable ĠT AB REF 21 Ġreports Ġthe Ġagreement Ġscores , Ġthe Ġoverall Ġ( micro ) Ġaverage ĠF 1 Ġscore Ġof Ġthe Ġannotations Ġis Ġ0 . 82 , Ġwhich Ġmeans Ġthat Ġon Ġaverage , Ġmore Ġthan Ġtwo Ġthirds Ġof Ġthe Ġoverall Ġannot ated Ġlabels Ġwere Ġagreed Ġon Ġby Ġboth Ġannot ators . ĠWe Ġdeem Ġthis Ġsatisfactory , Ġgiven Ġthe Ġcomplexity Ġof Ġthe Ġannotation Ġschema . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis </s> We Ġpresent Ġa Ġconcise Ġview Ġof Ġthe Ġannotation Ġresults Ġin ĠFigure ĠFIG REF 23 . ĠThe Ġfull Ġannotation Ġresults Ġcan Ġbe Ġfound Ġin ĠAppendix Ġ. ĠWe Ġcentre Ġour Ġdiscussion Ġaround Ġthe Ġfollowing Ġmain Ġpoints : </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠL ingu istic ĠFeatures </s> As Ġobserved Ġin ĠFigure ĠFIG REF 23 Ġthe Ġgold Ġstandards Ġfeature Ġa Ġhigh Ġdegree Ġof ĠRed und ancy , Ġpe aking Ġat Ġ76 % Ġof Ġthe Ġannot ated ĠHot pot Q A Ġsamples Ġand Ġsyn onyms Ġand Ġparaph r ases Ġ( lab elled ĠSyn onym ), Ġwith ĠRe Co R d Ġsamples Ġcontaining Ġ58 % Ġof Ġthem , Ġlikely Ġto Ġbe Ġattributed Ġto Ġthe Ġelabor ating Ġtype Ġof Ġdiscourse Ġof Ġthe Ġdataset Ġsources Ġ( ency cl op edia Ġand Ġnews wire ). ĠThis Ġis , Ġhowever , Ġnot Ġsurprising , Ġas Ġit Ġis Ġfairly Ġwell Ġunderstood Ġin Ġthe Ġliterature Ġthat Ġcurrent Ġstate - of - the - art Ġmodels Ġperform Ġwell Ġon Ġdistinguishing Ġrelevant Ġwords Ġand Ġphrases Ġfrom Ġredundant Ġones ĠB IB REF 32 . ĠAdditionally , Ġthe Ġrepresent ational Ġcapability Ġof Ġsyn onym Ġrelationships Ġof Ġword Ġembed d ings Ġhas Ġbeen Ġinvestigated Ġand Ġis Ġwell Ġknown ĠB IB REF 33 . ĠFinally , Ġwe Ġobserve Ġthe Ġpresence Ġof Ġsynt actic Ġfeatures , Ġsuch Ġas Ġambiguous Ġrelative Ġclauses , Ġapp os itions Ġand Ġad verb ial Ġphrases , Ġ( Rel Adv App Ġ40 % Ġin ĠHot pot Q A Ġand ĠRe Co R d ) Ġand Ġthose Ġintroducing Ġvariance , Ġconcrete ly Ġswitching Ġbetween Ġverbal Ġand Ġnominal Ġstyles Ġ( e . g . ĠNom inal isation Ġ10 % Ġin ĠHot pot Q A ) Ġand Ġfrom Ġpassive Ġto Ġactive Ġvoice Ġ( Voice , Ġ8 % Ġin ĠHot pot Q A ). </s> Sy nt actic Ġfeatures Ġcontributing Ġto Ġvariety Ġand Ġambiguity Ġthat Ġwe Ġdid Ġnot Ġobserve Ġin Ġour Ġsamples Ġare Ġthe Ġexploitation Ġof Ġverb Ġsymmetry , Ġthe Ġuse Ġof Ġd ative Ġand Ġgen itive Ġcases Ġor Ġambiguous Ġpre pos itions Ġand Ġcoordination Ġscope Ġ( respect ively ĠSy mm etry , ĠD ative , ĠGen itive , ĠPre pos itions , ĠScope ). ĠTherefore Ġwe Ġcannot Ġestablish Ġwhether Ġmodels Ġare Ġcapable Ġof Ġdealing Ġwith Ġthose Ġfeatures Ġby Ġevaluating Ġthem Ġon Ġthose Ġgold Ġstandards . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠFact ual ĠCorrect ness </s> We Ġidentify Ġthree Ġcommon Ġsources Ġthat Ġsurface Ġin Ġdifferent Ġproblems Ġregarding Ġan Ġanswer 's Ġfactual Ġcorrectness , Ġas Ġreported Ġin ĠFigure ĠFIG REF 23 Ġand Ġillustrate Ġtheir Ġinstant iations Ġin ĠTable ĠT AB REF 31 : </s> Design ĠCon str aints : ĠCho osing Ġthe Ġtask Ġdesign Ġand Ġthe Ġdata Ġcollection Ġmethod Ġintroduces Ġsome Ġconstraints Ġthat Ġlead Ġto Ġfact ually Ġdeb atable Ġexamples . ĠFor Ġexample , Ġa Ġspan Ġmight Ġhave Ġbeen Ġarbitrarily Ġselected Ġfrom Ġmultiple Ġspans Ġthat Ġpotentially Ġanswer Ġa Ġquestion , Ġbut Ġonly Ġa Ġsingle Ġcontinuous Ġanswer Ġspan Ġper Ġquestion Ġis Ġallowed Ġby Ġdesign , Ġas Ġobserved Ġin Ġthe ĠNews Q A Ġand ĠMs Marco Ġsamples Ġ( 32 % Ġand Ġ34 % Ġexamples Ġannot ated Ġas ĠDe bat able Ġwith Ġ16 % Ġand Ġ53 % Ġthereof Ġexhibiting Ġarbitrary Ġselection , Ġrespectively ). ĠSometimes , Ġwhen Ġadditional Ġpassages Ġare Ġadded Ġafter Ġthe Ġannotation Ġstep , Ġthey Ġcan Ġby Ġchance Ġcontain Ġpassages Ġthat Ġanswer Ġthe Ġquestion Ġmore Ġprecisely Ġthan Ġthe Ġoriginal Ġspan , Ġas Ġseen Ġin ĠHot pot Q A Ġ( 16 % ĠDe bat able Ġsamples , Ġ25 % Ġof Ġthem Ġdue Ġto Ġarbitrary Ġselection ). ĠIn Ġthe Ġcase Ġof ĠMulti RC Ġit Ġappears Ġto Ġbe Ġinconsistent , Ġwhether Ġmultiple Ġcorrect Ġanswer Ġchoices Ġare Ġexpected Ġto Ġbe Ġcorrect Ġin Ġisolation Ġor Ġin Ġconjunction Ġ( 28 % ĠDe bat able Ġwith Ġ29 % Ġof Ġthem Ġexhibiting Ġthis Ġproblem ). ĠThis Ġmight Ġprovide Ġan Ġexplanation Ġto Ġits Ġrelatively Ġweak Ġhuman Ġbaseline Ġperformance Ġof Ġ84 % ĠF 1 Ġscore ĠB IB REF 31 . </s> Weak ĠQuality Ġassurance : ĠWhen Ġthe Ġ( typically Ġcrowd - s ourced ) Ġannotations Ġare Ġnot Ġappropriately Ġvalidated , Ġincorrect Ġexamples Ġwill Ġfind Ġtheir Ġway Ġinto Ġthe Ġgold Ġstandards . ĠThis Ġtypically Ġresults Ġin Ġfact ually Ġwrong Ġexpected Ġanswers Ġ( i . e . Ġwhen Ġa Ġmore Ġcorrect Ġanswer Ġis Ġpresent Ġin Ġthe Ġcontext ) Ġor Ġa Ġquestion Ġis Ġexpected Ġto Ġbe ĠUn answer able , Ġbut Ġis Ġactually Ġanswer able Ġfrom Ġthe Ġprovided Ġcontext . ĠThe Ġlatter Ġis Ġobserved Ġin ĠMs Marco Ġ( 83 % Ġof Ġexamples Ġannot ated Ġas ĠWrong ) Ġand ĠNews Q A , Ġwhere Ġ60 % Ġof Ġthe Ġexamples Ġannot ated Ġas ĠWrong Ġare ĠUn answer able Ġwith Ġan Ġanswer Ġpresent . </s> Ar bit rary ĠPrecision : ĠThere Ġappears Ġto Ġbe Ġno Ġclear Ġguideline Ġon Ġhow Ġprecise Ġthe Ġanswer Ġis Ġexpected Ġto Ġbe , Ġwhen Ġthe Ġpassage Ġexpresses Ġthe Ġanswer Ġin Ġvarying Ġgran ular ities . ĠWe Ġannot ated Ġinstances Ġas ĠDe bat able Ġwhen Ġthe Ġexpected Ġanswer Ġwas Ġnot Ġthe Ġmost Ġprecise Ġgiven Ġthe Ġcontext Ġ( 44 % Ġand Ġ29 % Ġof ĠDe bat able Ġinstances Ġin ĠNews Q A Ġand ĠMulti RC , Ġrespectively ). </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠSem antics - al tering Ġgram matical Ġmodifiers </s> We Ġtook Ġinterest Ġin Ġwhether Ġany Ġof Ġthe Ġbenchmarks Ġcontain Ġwhat Ġwe Ġcall Ġdistracting Ġlex ical Ġfeatures Ġ( or Ġdistract ors ): Ġgram matical Ġmodifiers Ġthat Ġalter Ġthe Ġsemantics Ġof Ġa Ġsentence Ġfor Ġthe Ġfinal Ġtask Ġof Ġanswering Ġthe Ġgiven Ġquestion Ġwhile Ġpreserving Ġa Ġsimilar Ġlex ical Ġform . ĠAn Ġexample Ġof Ġsuch Ġfeatures Ġare Ġcues Ġfor Ġ( double ) ĠNeg ation Ġ( e . g ., ĠâĢ ľ no âĢ Ŀ , ĠâĢ ľ not âĢ Ŀ ), Ġwhich Ġwhen Ġintroduced Ġin Ġa Ġsentence , Ġreverse Ġits Ġmeaning . ĠOther Ġexamples Ġinclude Ġmodifiers Ġden oting ĠRestrict ivity , ĠFact ivity Ġand ĠReason ing Ġ( such Ġas ĠMon oton icity Ġand ĠCond itional Ġcues ). ĠExamples Ġof Ġquestion - answer Ġpairs Ġcontaining Ġa Ġdistract or Ġare Ġshown Ġin ĠTable ĠFIG REF 37 . </s> We Ġposit Ġthat Ġthe Ġpresence Ġof Ġsuch Ġdistract ors Ġwould Ġallow Ġfor Ġevaluating Ġreading Ġcomprehension Ġbeyond Ġpotential Ġsimple Ġword Ġmatching . ĠHowever , Ġwe Ġobserve Ġno Ġpresence Ġof Ġsuch Ġfeatures Ġin Ġthe Ġbenchmarks Ġ( be yond ĠNeg ation Ġin ĠDR OP , ĠRe Co R d Ġand ĠHot pot Q A , Ġwith Ġ4 %, Ġ4 % Ġand Ġ2 % Ġrespectively ). ĠThis Ġresults Ġin Ġgold Ġstandards Ġthat Ġclearly Ġexpress Ġthe Ġevidence Ġrequired Ġto Ġobtain Ġthe Ġanswer , Ġlacking Ġmore Ġchallenging , Ġi . e ., Ġdistracting , Ġsentences Ġthat Ġcan Ġassess Ġwhether Ġa Ġmodel Ġcan Ġtruly Ġunderstand Ġmeaning . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQual itative ĠAnalysis Ġ: :: ĠOther </s> In Ġthe ĠFigure ĠFIG REF 23 Ġwe Ġobserve Ġthat ĠOper ational Ġand ĠAr ithmetic Ġreasoning Ġmoderately Ġ( 6 % Ġto Ġ8 % Ġcombined ) Ġappears ĠâĢ ľ in Ġthe Ġwild âĢ Ŀ , Ġi . e . Ġwhen Ġnot Ġenforced Ġby Ġthe Ġdata Ġdesign Ġas Ġis Ġthe Ġcase Ġwith ĠHot pot Q A Ġ( 80 % ĠOperations Ġcombined ) Ġor ĠDR OP Ġ( 68 % ĠAr ithmetic Ġcombined ). ĠCa usal Ġreasoning Ġis Ġ( ex clusively ) Ġpresent Ġin ĠMulti RC Ġ( 32 %), Ġwhereas ĠTem poral Ġand ĠSp atial Ġreasoning Ġrequirements Ġseem Ġto Ġnot Ġnaturally Ġemerge Ġin Ġgold Ġstandards . ĠIn ĠRe Co R d , Ġa Ġfraction Ġof Ġ38 % Ġquestions Ġcan Ġonly Ġbe Ġanswered ĠBy ĠEx clusion Ġof Ġevery Ġother Ġcandidate , Ġdue Ġto Ġthe Ġdesign Ġchoice Ġof Ġallowing Ġquestions Ġwhere Ġthe Ġrequired Ġinformation Ġto Ġanswer Ġthem Ġis Ġnot Ġfully Ġexpressed Ġin Ġthe Ġaccompanying Ġparagraph . </s> Therefore , Ġit Ġis Ġalso Ġa Ġlittle Ġsurprising Ġto Ġobserve Ġthat ĠRe Co R d Ġrequires Ġexternal Ġresources Ġwith Ġregard Ġto Ġknowledge , Ġas Ġseen Ġin ĠFigure ĠFIG REF 23 . ĠMulti RC Ġrequires Ġtechnical Ġor Ġmore Ġprecisely Ġbasic Ġscientific Ġknowledge Ġ( 6 % ĠTechnical / Scient ific ), Ġas Ġa Ġportion Ġof Ġparagraphs Ġis Ġextracted Ġfrom Ġelementary Ġschool Ġscience Ġtextbooks ĠB IB REF 31 . ĠOther Ġbenchmarks Ġmoderately Ġprobe Ġfor Ġfactual Ġknowledge Ġ( 0 % Ġto Ġ4 % Ġacross Ġall Ġcategories ), Ġwhile ĠInt uitive Ġknowledge Ġis Ġrequired Ġto Ġderive Ġanswers Ġin Ġeach Ġgold Ġstandard . </s> It Ġis Ġalso Ġworth Ġpointing Ġout , Ġas Ġdone Ġin ĠFigure ĠFIG REF 23 , Ġthat Ġalthough ĠMulti RC Ġand ĠMs Marco Ġare Ġnot Ġmod elled Ġas Ġa Ġspan Ġselection Ġproblem , Ġtheir Ġsamples Ġstill Ġcontain Ġ50 % Ġand Ġ66 % Ġof Ġanswers Ġthat Ġare Ġdirectly Ġtaken Ġfrom Ġthe Ġcontext . ĠDR OP Ġcontains Ġthe Ġbiggest Ġfraction Ġof Ġgenerated Ġanswers Ġ( 60 %), Ġdue Ġto Ġthe Ġrequirement Ġof Ġarithmetic Ġoperations . </s> To Ġconclude Ġour Ġanalysis , Ġwe Ġobserve Ġsimilar Ġdistributions Ġof Ġlinguistic Ġfeatures Ġand Ġreasoning Ġpatterns , Ġexcept Ġwhere Ġthere Ġare Ġconstraints Ġenforced Ġby Ġdataset Ġdesign , Ġannotation Ġguidelines Ġor Ġsource Ġtext Ġchoice . ĠFurthermore , Ġcareful Ġconsideration Ġof Ġdesign Ġchoices Ġ( such Ġas Ġsingle - span Ġanswers ) Ġis Ġrequired , Ġto Ġavoid Ġimpair ing Ġthe Ġfactual Ġcorrectness Ġof Ġdatasets , Ġas Ġpure Ġcrowd - worker Ġagreement Ġseems Ġnot Ġsufficient Ġin Ġmultiple Ġcases . </s> Application Ġof Ġthe ĠFramework Ġ: :: ĠQuant itative ĠResults Ġ: :: ĠLex ical Ġoverlap </s> We Ġused Ġthe Ġscores Ġassigned Ġby Ġour Ġproposed Ġset Ġof Ġmetrics Ġ( disc ussed Ġin ĠSection ĠSEC REF 11 ĠDimensions Ġof ĠInterest : ĠComplex ity ) Ġto Ġpredict Ġthe Ġsupporting Ġfacts Ġin Ġthe Ġgold Ġstandard Ġsamples Ġ( that Ġwe Ġincluded Ġin Ġour Ġmanual Ġannotation ). ĠCon crete ly , Ġwe Ġused Ġthe Ġfollowing Ġfive Ġfeatures Ġcapturing Ġlex ical Ġoverlap : Ġ( i ) Ġthe Ġnumber Ġof Ġwords Ġoccurring Ġin Ġsentence Ġand Ġquestion , Ġ( ii ) Ġthe Ġlength Ġof Ġthe Ġlongest Ġn - gram Ġshared Ġby Ġsentence Ġand Ġquestion , Ġwhether Ġa Ġ( iii ) Ġun i - Ġand Ġ( iv ) Ġbig ram Ġfrom Ġthe Ġquestion Ġis Ġunique Ġto Ġa Ġsentence , Ġand Ġ( v ) Ġthe Ġsentence Ġindex , Ġas Ġinput Ġto Ġa Ġlog istic Ġregression Ġclass ifier . ĠWe Ġoptim ised Ġon Ġeach Ġsample Ġleaving Ġone Ġexample Ġfor Ġevaluation . ĠWe Ġcompute Ġthe Ġaverage ĠPrecision , ĠRecall Ġand ĠF 1 Ġscore Ġby Ġmeans Ġof Ġleave - one - out Ġvalidation Ġwith Ġevery Ġsample Ġentry . ĠThe Ġaveraged Ġresults Ġafter Ġ5 Ġruns Ġare Ġreported Ġin ĠTable ĠT AB REF 41 . </s> We Ġobserve Ġthat Ġeven Ġby Ġusing Ġonly Ġour Ġfive Ġfeatures Ġbased Ġlex ical Ġoverlap , Ġthe Ġsimple Ġlog istic Ġregression Ġbaseline Ġis Ġable Ġto Ġseparate Ġout Ġthe Ġsupporting Ġfacts Ġfrom Ġthe Ġcontext Ġto Ġa Ġvarying Ġdegree . ĠThis Ġis Ġin Ġline Ġwith Ġthe Ġlack Ġof Ġsemantics - al tering Ġgram matical Ġmodifiers Ġdiscussed Ġin Ġthe Ġqualitative Ġanalysis Ġsection Ġabove . ĠThe Ġclass ifier Ġperforms Ġbest Ġon ĠDR OP Ġ( 66 % ĠF 1 ) Ġand ĠMulti RC Ġ( 40 % ĠF 1 ), Ġwhich Ġmeans Ġthat Ġlex ical Ġcues Ġcan Ġconsiderably Ġfacilitate Ġthe Ġsearch Ġfor Ġthe Ġanswer Ġin Ġthose Ġgold Ġstandards . ĠOn ĠMulti RC , Ġy ad av 2019 quick Ġcome Ġto Ġa Ġsimilar Ġconclusion , Ġby Ġusing Ġa Ġmore Ġsophisticated Ġapproach Ġbased Ġon Ġoverlap Ġbetween Ġquestion , Ġsentence Ġand Ġanswer Ġchoices . </s> Sur prisingly , Ġthe Ġclass ifier Ġis Ġable Ġto Ġpick Ġup Ġa Ġsignal Ġfrom Ġsupporting Ġfacts Ġeven Ġon Ġdata Ġthat Ġhas Ġbeen Ġpr uned Ġagainst Ġlex ical Ġoverlap Ġhe ur istics Ġby Ġpop ulating Ġthe Ġcontext Ġwith Ġadditional Ġdocuments Ġthat Ġhave Ġhigh Ġoverlap Ġscores Ġwith Ġthe Ġquestion . ĠThis Ġresults Ġin Ġsignificantly Ġhigher Ġscores Ġthan Ġwhen Ġguessing Ġrandomly Ġ( Hot pot Q A Ġ26 % ĠF 1 , Ġand ĠMs Marco Ġ11 % ĠF 1 ). ĠWe Ġobserve Ġsimilar Ġresults Ġin Ġthe Ġcase Ġthe Ġlength Ġof Ġthe Ġquestion Ġleaves Ġfew Ġcandidates Ġto Ġcompute Ġoverlap Ġwith Ġ$ 6 . 3 $ Ġand Ġ$ 7 . 3 $ Ġtokens Ġon Ġaverage Ġfor ĠMs Marco Ġand ĠNews Q A Ġ( 26 % ĠF 1 ), Ġcompared Ġto Ġ$ 16 . 9 $ Ġtokens Ġon Ġaverage Ġfor Ġthe Ġremaining Ġfour Ġdataset Ġsamples . </s> Finally , Ġit Ġis Ġworth Ġmentioning Ġthat Ġalthough Ġthe Ġqueries Ġin ĠRe Co R d Ġare Ġexplicitly Ġindependent Ġfrom Ġthe Ġpassage , Ġthe Ġlinear Ġclass ifier Ġis Ġstill Ġcapable Ġof Ġachieving Ġ34 % ĠF 1 Ġscore Ġin Ġpredicting Ġthe Ġsupporting Ġfacts . </s> However , Ġneural Ġnetworks Ġperform Ġsignificantly Ġbetter Ġthan Ġour Ġadmittedly Ġcrude Ġbaseline Ġ( e . g . Ġ66 % ĠF 1 Ġfor Ġsupporting Ġfacts Ġclassification Ġon ĠHot pot Q A ĠB IB REF 14 ), Ġalbeit Ġutil ising Ġmore Ġtraining Ġexamples , Ġand Ġa Ġricher Ġsentence Ġrepresentation . ĠThis Ġfacts Ġimplies Ġthat Ġthose Ġneural Ġmodels Ġare Ġcapable Ġof Ġsolving Ġmore Ġchallenging Ġproblems Ġthan Ġsimple ĠâĢ ľ text Ġmatching âĢ Ŀ Ġas Ġperformed Ġby Ġthe Ġlog istic Ġregression Ġbaseline . ĠHowever , Ġthey Ġstill Ġcircumvent Ġactual Ġreading Ġcomprehension Ġas Ġthe Ġrespective Ġgold Ġstandards Ġare Ġof Ġlimited Ġsuit ability Ġto Ġevaluate Ġthis ĠB IB REF 34 , ĠB IB REF 35 . ĠThis Ġsuggests Ġan Ġexciting Ġfuture Ġresearch Ġdirection , Ġthat Ġis Ġcategor ising Ġthe Ġscale Ġbetween Ġtext Ġmatching Ġand Ġreading Ġcomprehension Ġmore Ġprecisely Ġand Ġrespectively Ġpositioning Ġstate - of - the - art Ġmodels Ġthere on . </s> Related ĠWork </s> Although Ġnot Ġas Ġprominent Ġas Ġthe Ġresearch Ġon Ġnovel Ġarchitecture , Ġthere Ġhas Ġbeen Ġsteady Ġprogress Ġin Ġcritically Ġinvestigating Ġthe Ġdata Ġand Ġevaluation Ġaspects Ġof ĠN LP Ġand Ġmachine Ġlearning Ġin Ġgeneral Ġand ĠM RC Ġin Ġparticular . </s> Related ĠWork Ġ: :: ĠAd vers arial ĠEvaluation </s> The Ġauthors Ġof Ġthe ĠAdd Sent Ġalgorithm ĠB IB REF 11 Ġshow Ġthat ĠM RC Ġmodels Ġtrained Ġand Ġevaluated Ġon Ġthe ĠS Qu AD Ġdataset Ġpay Ġtoo Ġlittle Ġattention Ġto Ġdetails Ġthat Ġmight Ġchange Ġthe Ġsemantics Ġof Ġa Ġsentence , Ġand Ġpropose Ġa Ġcrowd - s ourcing Ġbased Ġmethod Ġto Ġgenerate Ġadversary Ġexamples Ġto Ġexploit Ġthat Ġweakness . ĠThis Ġmethod Ġwas Ġfurther Ġadapted Ġto Ġbe Ġfully Ġautomated ĠB IB REF 36 Ġand Ġapplied Ġto Ġdifferent Ġgold Ġstandards ĠB IB REF 35 . ĠOur Ġproposed Ġapproach Ġdiffers Ġin Ġthat Ġwe Ġaim Ġto Ġprovide Ġqualitative Ġjust ifications Ġfor Ġthose Ġquant itatively Ġmeasured Ġissues . </s> Related ĠWork Ġ: :: ĠSan ity ĠBas elines </s> Another Ġline Ġof Ġresearch Ġestablishes Ġsane Ġbas elines Ġto Ġprovide Ġmore Ġmeaningful Ġcontext Ġto Ġthe Ġraw Ġperformance Ġscores Ġof Ġevaluated Ġmodels . ĠWhen Ġremoving Ġintegral Ġparts Ġof Ġthe Ġtask Ġformulation Ġsuch Ġas Ġquestion , Ġthe Ġtextual Ġpassage Ġor Ġparts Ġthereof ĠB IB REF 37 Ġor Ġrestricting Ġmodel Ġcomplexity Ġby Ġdesign Ġin Ġorder Ġto Ġsuppress Ġsome Ġrequired Ġform Ġof Ġreasoning ĠB IB REF 38 , Ġmodels Ġare Ġstill Ġable Ġto Ġperform Ġcompar ably Ġto Ġthe Ġstate - of - the - art . ĠThis Ġraises Ġconcerns Ġabout Ġthe Ġperceived Ġbenchmark Ġcomplexity Ġand Ġis Ġrelated Ġto Ġour Ġwork Ġin Ġa Ġbroader Ġsense Ġas Ġone Ġof Ġour Ġgoals Ġis Ġto Ġestimate Ġthe Ġcomplexity Ġof Ġbenchmarks . </s> Related ĠWork Ġ: :: ĠBench mark Ġevaluation Ġin ĠN LP </s> Beyond ĠM RC , Ġefforts Ġsimilar Ġto Ġours Ġthat Ġpursue Ġthe Ġgoal Ġof Ġanalys ing Ġthe Ġevaluation Ġof Ġestablished Ġdatasets Ġexist Ġin ĠNatural ĠLanguage ĠIn ference ĠB IB REF 13 , ĠB IB REF 12 . ĠTheir Ġanalyses Ġreveal Ġthe Ġexistence Ġof Ġbiases Ġin Ġtraining Ġand Ġevaluation Ġdata Ġthat Ġcan Ġbe Ġapprox imated Ġwith Ġsimple Ġmajority - based Ġhe ur istics . ĠBecause Ġof Ġthese Ġbiases , Ġtrained Ġmodels Ġfail Ġto Ġextract Ġthe Ġsemantics Ġthat Ġare Ġrequired Ġfor Ġthe Ġcorrect Ġinference . ĠFurthermore , Ġa Ġfair Ġshare Ġof Ġwork Ġwas Ġdone Ġto Ġreveal Ġgender Ġbias Ġin Ġcore ference Ġresolution Ġdatasets Ġand Ġmodels ĠB IB REF 39 , ĠB IB REF 40 , ĠB IB REF 41 . </s> Related ĠWork Ġ: :: ĠAn notation ĠTax onom ies </s> Finally , Ġrelated Ġto Ġour Ġframework Ġare Ġworks Ġthat Ġintroduce Ġannotation Ġcategories Ġfor Ġgold Ġstandards Ġevaluation . ĠCon crete ly , Ġwe Ġbuild Ġour Ġannotation Ġframework Ġaround Ġlinguistic Ġfeatures Ġthat Ġwere Ġintroduced Ġin Ġthe ĠGL UE Ġsuite ĠB IB REF 42 Ġand Ġthe Ġreasoning Ġcategories Ġintroduced Ġin Ġthe ĠWorld Tree Ġdataset ĠB IB REF 19 . ĠA Ġqualitative Ġanalysis Ġcomplementary Ġto Ġours , Ġwith Ġfocus Ġon Ġthe Ġun answer ability Ġpatterns Ġin Ġdatasets Ġthat Ġfeature Ġun answer able Ġquestions Ġwas Ġdone Ġby ĠY ats kar 2019 . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġintroduce Ġa Ġnovel Ġframework Ġto Ġcharacter ise Ġmachine Ġreading Ġcomprehension Ġgold Ġstandards . ĠThis Ġframework Ġhas Ġpotential Ġapplications Ġwhen Ġcomparing Ġdifferent Ġgold Ġstandards , Ġconsidering Ġthe Ġdesign Ġchoices Ġfor Ġa Ġnew Ġgold Ġstandard Ġand Ġperforming Ġqualitative Ġerror Ġanalyses Ġfor Ġa Ġproposed Ġapproach . </s> Furthermore Ġwe Ġapplied Ġthe Ġframework Ġto Ġanalyse Ġpopular Ġstate - of - the - art Ġgold Ġstandards Ġfor Ġmachine Ġreading Ġcomprehension : ĠWe Ġreveal Ġissues Ġwith Ġtheir Ġfactual Ġcorrectness , Ġshow Ġthe Ġpresence Ġof Ġlex ical Ġcues Ġand Ġwe Ġobserve Ġthat Ġsemantics - al tering Ġgram matical Ġmodifiers Ġare Ġmissing Ġin Ġall Ġof Ġthe Ġinvestigated Ġgold Ġstandards . ĠStud ying Ġhow Ġto Ġintroduce Ġthose Ġmodifiers Ġinto Ġgold Ġstandards Ġand Ġobserving Ġwhether Ġstate - of - the - art ĠM RC Ġmodels Ġare Ġcapable Ġof Ġperforming Ġreading Ġcomprehension Ġon Ġtext Ġcontaining Ġthem , Ġis Ġa Ġfuture Ġresearch Ġgoal . </s> A Ġfuture Ġline Ġof Ġresearch Ġis Ġto Ġextend Ġthe Ġframework Ġto Ġbe Ġable Ġto Ġidentify Ġthe Ġdifferent Ġtypes Ġof Ġexplo itable Ġcues Ġsuch Ġas Ġquestion Ġor Ġentity Ġtyping Ġand Ġconcrete Ġoverlap Ġpatterns . ĠThis Ġwill Ġallow Ġthe Ġframework Ġto Ġserve Ġas Ġan Ġinterpret able Ġestimate Ġof Ġreading Ġcomprehension Ġcomplexity Ġof Ġgold Ġstandards . ĠFinally , Ġinvestigating Ġgold Ġstandards Ġunder Ġthis Ġframework Ġwhere ĠM RC Ġmodels Ġoutper form Ġthe Ġhuman Ġbaseline Ġ( e . g . ĠS Qu AD ) Ġwill Ġcontribute Ġto Ġa Ġdeeper Ġunderstanding Ġof Ġthe Ġseemingly Ġsuperb Ġperformance Ġof Ġdeep Ġlearning Ġapproaches Ġon Ġthem .