More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. 13% of the questions are not answerable.  Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based).  The final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%).
ABSTRACTIVE
Distribution of category labels, number of answerable-not answerable questions, number of text-based and script-based questions, average text, question, and answer length, number of questions per text
ABSTRACTIVE
More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. For 13% of the questions, the workers did not agree on one of the 4 categories with a 3 out of 5 majority, so we did not include these questions in our dataset. The distribution of category labels on the remaining 87% is shown in Table TABREF10 . 14,074 (52%) questions could be answered. Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based). After removing 135 questions during the validation, the final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%). This ratio was manually verified based on a random sample of questions.
The distribution of category labels on the remaining 87% is shown in Table TABREF10 . 14,074 (52%) questions could be answered. Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based). After removing 135 questions during the validation, the final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%). This ratio was manually verified based on a random sample of questions. We split the dataset into training (9,731 questions on 1,470 texts), development (1,411 questions on 219 texts), and test set (2,797 questions on 430 texts). Each text appears only in one of the three sets. The complete set of texts for 5 scenarios was held out for the test set. The average text, question, and answer length is 196.0 words, 7.8 words, and 3.6 words, respectively. On average, there are 6.7 questions per text.
1803.05223
Introduction </s> Amb ig uity Ġand Ġimplicit ness Ġare Ġinherent Ġproperties Ġof Ġnatural Ġlanguage Ġthat Ġcause Ġchallenges Ġfor Ġcomputational Ġmodels Ġof Ġlanguage Ġunderstanding . ĠIn Ġeveryday Ġcommunication , Ġpeople Ġassume Ġa Ġshared Ġcommon Ġground Ġwhich Ġforms Ġa Ġbasis Ġfor Ġefficiently Ġresolving Ġambig u ities Ġand Ġfor Ġinfer ring Ġimplicit Ġinformation . ĠThus , Ġrecover able Ġinformation Ġis Ġoften Ġleft Ġun mentioned Ġor Ġunders pec ified . ĠSuch Ġinformation Ġmay Ġinclude Ġen cycl oped ic Ġand Ġcommons ense Ġknowledge . ĠThis Ġwork Ġfocuses Ġon Ġcommons ense Ġknowledge Ġabout Ġeveryday Ġactivities , Ġso - called Ġscripts . </s> This Ġpaper Ġintroduces Ġa Ġdataset Ġto Ġevaluate Ġnatural Ġlanguage Ġunderstanding Ġapproaches Ġwith Ġa Ġfocus Ġon Ġinterpretation Ġprocesses Ġrequiring Ġinference Ġbased Ġon Ġcommons ense Ġknowledge . ĠIn Ġparticular , Ġwe Ġpresent ĠMC Script , Ġa Ġdataset Ġfor Ġassessing Ġthe Ġcontribution Ġof Ġscript Ġknowledge Ġto Ġmachine Ġcomprehension . ĠScript s Ġare Ġsequences Ġof Ġevents Ġdescribing Ġstereotypical Ġhuman Ġactivities Ġ( also Ġcalled Ġscenarios ), Ġfor Ġexample Ġbaking Ġa Ġcake Ġor Ġtaking Ġa Ġbus ĠB IB REF 0 Ġ. ĠTo Ġillustrate Ġthe Ġimportance Ġof Ġscript Ġknowledge , Ġconsider ĠExample Ġ( ĠSEC REF 1 Ġ): </s> Without Ġusing Ġcommons ense Ġknowledge , Ġit Ġmay Ġbe Ġdifficult Ġto Ġtell Ġwho Ġate Ġthe Ġfood : ĠRachel Ġor Ġthe Ġwaitress . ĠIn Ġcontrast , Ġif Ġwe Ġutilize Ġcommons ense Ġknowledge , Ġin Ġparticular , Ġscript Ġknowledge Ġabout Ġthe Ġeating Ġin Ġa Ġrestaurant Ġscenario , Ġwe Ġcan Ġmake Ġthe Ġfollowing Ġinf erences : ĠRachel Ġis Ġmost Ġlikely Ġa Ġcustomer , Ġsince Ġshe Ġreceived Ġan Ġorder . ĠIt Ġis Ġusually Ġthe Ġcustomer , Ġand Ġnot Ġthe Ġwaitress , Ġwho Ġeats Ġthe Ġordered Ġfood . ĠSo ĠShe Ġmost Ġlikely Ġrefers Ġto ĠRachel . </s> Various Ġapproaches Ġfor Ġscript Ġknowledge Ġextraction Ġand Ġprocessing Ġhave Ġbeen Ġproposed Ġin Ġrecent Ġyears . ĠHowever , Ġsystems Ġhave Ġbeen Ġevaluated Ġfor Ġspecific Ġaspects Ġof Ġscript Ġknowledge Ġonly , Ġsuch Ġas Ġevent Ġordering ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġevent Ġparaph r asing ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġor Ġevent Ġprediction Ġ( name ly , Ġthe Ġnarrative Ġclo ze Ġtask ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ). ĠThese Ġevaluation Ġmethods Ġlack Ġa Ġclear Ġconnection Ġto Ġreal - world Ġtasks . ĠOur ĠMC Script Ġdataset Ġprovides Ġan Ġextr ins ic Ġevaluation Ġframework , Ġbased Ġon Ġtext Ġcomprehension Ġinvolving Ġcommons ense Ġknowledge . ĠThis Ġframework Ġmakes Ġit Ġpossible Ġto Ġassess Ġsystem Ġperformance Ġin Ġa Ġmultiple - choice Ġquestion Ġanswering Ġsetting , Ġwithout Ġimposing Ġany Ġspecific Ġstructural Ġor Ġmethod ical Ġrequirements . </s> MC Script Ġis Ġa Ġcollection Ġof Ġ( 1 ) Ġnarrative Ġtexts , Ġ( 2 ) Ġquestions Ġof Ġvarious Ġtypes Ġreferring Ġto Ġthese Ġtexts , Ġand Ġ( 3 ) Ġpairs Ġof Ġanswer Ġcandidates Ġfor Ġeach Ġquestion . ĠIt Ġcomprises Ġapprox . Ġ2 , 100 Ġtexts Ġand Ġa Ġtotal Ġof Ġapprox . Ġ14 , 000 Ġquestions . ĠAn sw ering Ġa Ġsubstantial Ġsubset Ġof Ġquestions Ġrequires Ġknowledge Ġbeyond Ġthe Ġfacts Ġmentioned Ġin Ġthe Ġtext , Ġi . e . Ġit Ġrequires Ġinference Ġusing Ġcommons ense Ġknowledge Ġabout Ġeveryday Ġactivities . ĠAn Ġexample Ġis Ġgiven Ġin ĠFigure ĠFIG REF 2 Ġ. ĠFor Ġboth Ġquestions , Ġthe Ġcorrect Ġchoice Ġfor Ġan Ġanswer Ġrequires Ġcommons ense Ġknowledge Ġabout Ġthe Ġactivity Ġof Ġplanting Ġa Ġtree , Ġwhich Ġgoes Ġbeyond Ġwhat Ġis Ġmentioned Ġin Ġthe Ġtext . ĠText s , Ġquestions , Ġand Ġanswers Ġwere Ġobtained Ġthrough Ġcrowds ourcing . ĠIn Ġorder Ġto Ġensure Ġhigh Ġquality , Ġwe Ġmanually Ġvalidated Ġand Ġfiltered Ġthe Ġdataset . ĠDue Ġto Ġour Ġdesign Ġof Ġthe Ġdata Ġacquisition Ġprocess , Ġwe Ġended Ġup Ġwith Ġa Ġsubstantial Ġsubset Ġof Ġquestions Ġthat Ġrequire Ġcommons ense Ġinference Ġ( 27 . 4 %). </s> Corp us </s> Machine Ġcomprehension Ġdatasets Ġconsist Ġof Ġthree Ġmain Ġcomponents : Ġtexts , Ġquestions Ġand Ġanswers . ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġdata Ġcollection Ġfor Ġthese Ġ3 Ġcomponents . ĠWe Ġfirst Ġdescribe Ġa Ġseries Ġof Ġpilot Ġstudies Ġthat Ġwe Ġconducted Ġin Ġorder Ġto Ġcollect Ġcommons ense Ġinference Ġquestions Ġ( Section ĠSEC REF 4 Ġ). ĠIn ĠSection ĠSEC REF 5 Ġ, Ġwe Ġdiscuss Ġthe Ġresulting Ġdata Ġcollection Ġof Ġquestions , Ġtexts Ġand Ġanswers Ġvia Ġcrowds ourcing Ġon ĠAmazon ĠMechanical ĠTurk Ġ( hen ce forth ĠM Tur k ). ĠSection ĠSEC REF 17 Ġgives Ġinformation Ġabout Ġsome Ġnecessary Ġpost processing Ġsteps Ġand Ġthe Ġdataset Ġvalidation . ĠLastly , ĠSection ĠSEC REF 19 Ġgives Ġstatistics Ġabout Ġthe Ġfinal Ġdataset . </s> P ilot ĠStudy </s> As Ġa Ġstarting Ġpoint Ġfor Ġour Ġpilots , Ġwe Ġmade Ġuse Ġof Ġtexts Ġfrom Ġthe ĠIn Script Ġcorpus ĠB IB REF 10 Ġ, Ġwhich Ġprovides Ġstories Ġcentered Ġaround Ġeveryday Ġsituations Ġ( see ĠSection ĠSEC REF 7 Ġ). ĠWe Ġconducted Ġthree Ġdifferent Ġpilot Ġstudies Ġto Ġdetermine Ġthe Ġbest Ġway Ġof Ġcollecting Ġquestions Ġthat Ġrequire Ġinference Ġover Ġcommons ense Ġknowledge : </s> The Ġmost Ġintuitive Ġway Ġof Ġcollecting Ġreading Ġcomprehension Ġquestions Ġis Ġto Ġshow Ġtexts Ġto Ġworkers Ġand Ġlet Ġthem Ġformulate Ġquestions Ġand Ġanswers Ġon Ġthe Ġtexts , Ġwhich Ġis Ġwhat Ġwe Ġtried Ġinternally Ġin Ġa Ġfirst Ġpilot . ĠSince Ġour Ġfocus Ġis Ġto Ġprovide Ġan Ġevaluation Ġframework Ġfor Ġinference Ġover Ġcommons ense Ġknowledge , Ġwe Ġmanually Ġassessed Ġthe Ġnumber Ġof Ġquestions Ġthat Ġindeed Ġrequire Ġcommon Ġsense Ġknowledge . ĠWe Ġfound Ġtoo Ġmany Ġquestions Ġand Ġanswers Ġcollected Ġin Ġthis Ġmanner Ġto Ġbe Ġlex ically Ġclose Ġto Ġthe Ġtext . </s> In Ġa Ġsecond Ġpilot , Ġwe Ġinvestigated Ġthe Ġoption Ġto Ġtake Ġthe Ġquestions Ġcollected Ġfor Ġone Ġtext Ġand Ġshow Ġthem Ġas Ġquestions Ġfor Ġanother Ġtext Ġof Ġthe Ġsame Ġscenario . ĠWhile Ġthis Ġmethod Ġresulted Ġin Ġa Ġlarger Ġnumber Ġof Ġquestions Ġthat Ġrequired Ġinference , Ġwe Ġfound Ġthe Ġmajority Ġof Ġquestions Ġto Ġnot Ġmake Ġsense Ġat Ġall Ġwhen Ġpaired Ġwith Ġanother Ġtext . ĠMany Ġquestions Ġwere Ġspecific Ġto Ġa Ġtext Ġ( and Ġnot Ġto Ġa Ġscenario ), Ġrequiring Ġdetails Ġthat Ġcould Ġnot Ġbe Ġanswered Ġfrom Ġother Ġtexts . ĠSince Ġthe Ġtwo Ġprevious Ġpilot Ġsetups Ġresulted Ġin Ġquestions Ġthat Ġcentered Ġaround Ġthe Ġtexts Ġthemselves , Ġwe Ġdecided Ġfor Ġa Ġthird Ġpilot Ġto Ġnot Ġshow Ġworkers Ġany Ġspecific Ġtexts Ġat Ġall . ĠInstead , Ġwe Ġasked Ġfor Ġquestions Ġthat Ġcentered Ġaround Ġa Ġspecific Ġscript Ġscenario Ġ( e . g . Ġeating Ġin Ġa Ġrestaurant ). ĠWe Ġfound Ġthis Ġmode Ġof Ġcollection Ġto Ġresult Ġin Ġquestions Ġthat Ġhave Ġthe Ġright Ġlevel Ġof Ġspecificity Ġfor Ġour Ġpurposes : Ġnamely , Ġquestions Ġthat Ġare Ġrelated Ġto Ġa Ġscenario Ġand Ġthat Ġcan Ġbe Ġanswered Ġfrom Ġdifferent Ġtexts Ġ( about Ġthat Ġscenario ), Ġbut Ġfor Ġwhich Ġa Ġtext Ġdoes Ġnot Ġneed Ġto Ġprovide Ġthe Ġanswer Ġexplicitly . </s> The Ġnext Ġsection Ġwill Ġdescribe Ġthe Ġmode Ġof Ġcollection Ġchosen Ġfor Ġthe Ġfinal Ġdataset , Ġbased Ġon Ġthe Ġthird Ġpilot , Ġin Ġmore Ġdetail . </s> Data ĠCollection </s> As Ġmentioned Ġin Ġthe Ġprevious Ġsection , Ġwe Ġdecided Ġto Ġbase Ġthe Ġquestion Ġcollection Ġon Ġscript Ġscenarios Ġrather Ġthan Ġspecific Ġtexts . ĠAs Ġa Ġstarting Ġpoint Ġfor Ġour Ġdata Ġcollection , Ġwe Ġuse Ġscenarios Ġfrom Ġthree Ġscript Ġdata Ġcollections ĠB IB REF 3 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠTogether , Ġthese Ġresources Ġcontain Ġmore Ġthan Ġ200 Ġscenarios . ĠTo Ġmake Ġsure Ġthat Ġscenarios Ġhave Ġdifferent Ġcomplexity Ġand Ġcontent , Ġwe Ġselected Ġ80 Ġof Ġthem Ġand Ġcame Ġup Ġwith Ġ20 Ġnew Ġscenarios . ĠTogether Ġwith Ġthe Ġ10 Ġscenarios Ġfrom ĠIn Script , Ġwe Ġend Ġup Ġwith Ġa Ġtotal Ġof Ġ110 Ġscenarios . </s> For Ġthe Ġcollection Ġof Ġtexts , Ġwe Ġfollowed Ġmod i ins cript , Ġwhere Ġworkers Ġwere Ġasked Ġto Ġwrite Ġa Ġstory Ġabout Ġa Ġgiven Ġactivity ĠâĢ ľ as Ġif Ġexplaining Ġit Ġto Ġa Ġchild âĢ Ŀ . ĠThis Ġresults Ġin Ġelaborate Ġand Ġexplicit Ġtexts Ġthat Ġare Ġcentered Ġaround Ġa Ġsingle Ġscenario . ĠConsequently , Ġthe Ġtexts Ġare Ġsynt act ically Ġsimple , Ġfacilitating Ġmachine Ġcomprehension Ġmodels Ġto Ġfocus Ġon Ġsemantic Ġchallenges Ġand Ġinference . ĠWe Ġcollected Ġ20 Ġtexts Ġfor Ġeach Ġscenario . ĠEach Ġparticipant Ġwas Ġallowed Ġto Ġwrite Ġonly Ġone Ġstory Ġper Ġscenario , Ġbut Ġwork Ġon Ġas Ġmany Ġscenarios Ġas Ġthey Ġliked . ĠFor Ġeach Ġof Ġthe Ġ10 Ġscenarios Ġfrom ĠIn Script , Ġwe Ġrandomly Ġselected Ġ20 Ġexisting Ġtexts Ġfrom Ġthat Ġresource . </s> For Ġcollecting Ġquestions , Ġworkers Ġwere Ġinstructed Ġto ĠâĢ ľ im agine Ġthey Ġtold Ġa Ġstory Ġabout Ġa Ġcertain Ġscenario Ġto Ġa Ġchild Ġand Ġwant Ġto Ġtest Ġif Ġthe Ġchild Ġunderstood Ġeverything Ġcorrectly âĢ Ŀ . ĠThis Ġinstruction Ġalso Ġensured Ġthat Ġquestions Ġare Ġlingu istically Ġsimple , Ġelaborate Ġand Ġexplicit . ĠWorkers Ġwere Ġasked Ġto Ġformulate Ġquestions Ġabout Ġdetails Ġof Ġsuch Ġa Ġsituation , Ġi . e . Ġindependent Ġof Ġa Ġconcrete Ġnarrative . ĠThis Ġresulted Ġin Ġquestions , Ġthe Ġanswer Ġto Ġwhich Ġis Ġnot Ġliterally Ġmentioned Ġin Ġthe Ġtext . ĠTo Ġcover Ġa Ġbroad Ġrange Ġof Ġquestion Ġtypes , Ġwe Ġasked Ġparticipants Ġto Ġwrite Ġ3 Ġtemporal Ġquestions Ġ( asking Ġabout Ġtime Ġpoints Ġand Ġevent Ġorder ), Ġ3 Ġcontent Ġquestions Ġ( asking Ġabout Ġpersons Ġor Ġdetails Ġin Ġthe Ġscenario ) Ġand Ġ3 Ġreasoning Ġquestions Ġ( asking Ġhow Ġor Ġwhy Ġsomething Ġhappened ). ĠThey Ġwere Ġalso Ġasked Ġto Ġformulate Ġ6 Ġfree Ġquestions , Ġwhich Ġresulted Ġin Ġa Ġtotal Ġof Ġ15 Ġquestions . ĠAs king Ġeach Ġworker Ġfor Ġa Ġhigh Ġnumber Ġof Ġquestions Ġenforced Ġthat Ġmore Ġcreative Ġquestions Ġwere Ġformulated , Ġwhich Ġgo Ġbeyond Ġobvious Ġquestions Ġfor Ġa Ġscenario . ĠSince Ġparticipants Ġwere Ġnot Ġshown Ġa Ġconcrete Ġstory , Ġwe Ġasked Ġthem Ġto Ġuse Ġthe Ġneutral Ġpronoun ĠâĢ ľ they âĢ Ŀ Ġto Ġaddress Ġthe Ġprotagonist Ġof Ġthe Ġstory . ĠWe Ġpermitted Ġparticipants Ġto Ġwork Ġon Ġas Ġmany Ġscenarios Ġas Ġdesired Ġand Ġwe Ġcollected Ġquestions Ġfrom Ġ10 Ġparticipants Ġper Ġscenario . </s> Our Ġmode Ġof Ġquestion Ġcollection Ġresults Ġin Ġquestions Ġthat Ġare Ġnot Ġassociated Ġwith Ġspecific Ġtexts . ĠFor Ġeach Ġtext , Ġwe Ġcollected Ġanswers Ġfor Ġ15 Ġquestions Ġthat Ġwere Ġrandomly Ġselected Ġfrom Ġthe Ġsame Ġscenario . ĠSince Ġquestions Ġand Ġtexts Ġwere Ġcollected Ġindependently , Ġanswering Ġa Ġrandom Ġquestion Ġis Ġnot Ġalways Ġpossible Ġfor Ġa Ġgiven Ġtext . ĠTherefore , Ġwe Ġcarried Ġout Ġanswer Ġcollection Ġin Ġtwo Ġsteps . ĠIn Ġthe Ġfirst Ġstep , Ġwe Ġasked Ġparticipants Ġto Ġassign Ġa Ġcategory Ġto Ġeach Ġtext âĢĵ question Ġpair . </s> We Ġdistinguish Ġtwo Ġcategories Ġof Ġanswer able Ġquestions : ĠThe Ġcategory Ġtext - based Ġwas Ġassigned Ġto Ġquestions Ġthat Ġcan Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġdirectly . ĠIf Ġthe Ġanswer Ġcould Ġonly Ġbe Ġinferred Ġby Ġusing Ġcommons ense Ġknowledge , Ġthe Ġcategory Ġscript - based Ġwas Ġassigned . ĠMaking Ġthis Ġdistinction Ġis Ġinteresting Ġfor Ġevaluation Ġpurposes , Ġsince Ġit Ġenables Ġus Ġto Ġestimate Ġthe Ġnumber Ġof Ġcommons ense Ġinference Ġquestions . ĠFor Ġquestions Ġthat Ġdid Ġnot Ġmake Ġsense Ġat Ġall Ġgiven Ġa Ġtext , Ġunf itting Ġwas Ġassigned . ĠIf Ġa Ġquestion Ġmade Ġsense Ġfor Ġa Ġtext , Ġbut Ġit Ġwas Ġimpossible Ġto Ġfind Ġan Ġanswer , Ġthe Ġlabel Ġunknown Ġwas Ġused . </s> In Ġa Ġsecond Ġstep , Ġwe Ġtold Ġparticipants Ġto Ġformulate Ġa Ġplausible Ġcorrect Ġand Ġa Ġplausible Ġincorrect Ġanswer Ġcandidate Ġto Ġanswer able Ġquestions Ġ( text - based Ġor Ġscript - based ). ĠTo Ġlevel Ġout Ġthe Ġeffort Ġbetween Ġanswer able Ġand Ġnon - answer able Ġquestions , Ġparticipants Ġhad Ġto Ġwrite Ġa Ġnew Ġquestion Ġwhen Ġselecting Ġunknown Ġor Ġunf itting . ĠIn Ġorder Ġto Ġget Ġreliable Ġjudgments Ġabout Ġwhether Ġor Ġnot Ġa Ġquestion Ġcan Ġbe Ġanswered , Ġwe Ġcollected Ġdata Ġfrom Ġ5 Ġparticipants Ġfor Ġeach Ġquestion Ġand Ġdecided Ġon Ġthe Ġfinal Ġcategory Ġvia Ġmajority Ġvote Ġ( at Ġleast Ġ3 Ġout Ġof Ġ5 ). ĠConsequently , Ġfor Ġeach Ġquestion Ġwith Ġa Ġmajority Ġvote Ġon Ġeither Ġtext - based Ġor Ġscript - based , Ġthere Ġare Ġ3 Ġto Ġ5 Ġcorrect Ġand Ġincorrect Ġanswer Ġcandidates , Ġone Ġfrom Ġeach Ġparticipant Ġwho Ġagreed Ġon Ġthe Ġcategory . ĠQuestions Ġwithout Ġa Ġclear Ġmajority Ġvote Ġor Ġwith Ġties Ġwere Ġnot Ġincluded Ġin Ġthe Ġdataset . </s> We Ġperformed Ġfour Ġpost - processing Ġsteps Ġon Ġthe Ġcollected Ġdata . </s> We Ġmanually Ġfiltered Ġout Ġtexts Ġthat Ġwere Ġinstructional Ġrather Ġthan Ġnarrative . </s> All Ġtexts , Ġquestions Ġand Ġanswers Ġwere Ġspell checked Ġby Ġrunning Ġa Spell Ġand Ġmanually Ġinspecting Ġall Ġcorrections Ġproposed Ġby Ġthe Ġspell check er . </s> We Ġfound Ġthat Ġsome Ġparticipants Ġdid Ġnot Ġuse ĠâĢ ľ they âĢ Ŀ Ġwhen Ġreferring Ġto Ġthe Ġprotagonist . ĠWe Ġidentified ĠâĢ ľ I âĢ Ŀ , ĠâĢ ľ you âĢ Ŀ , ĠâĢ ľ he âĢ Ŀ , ĠâĢ ľ she âĢ Ŀ , ĠâĢ ľ my âĢ Ŀ , ĠâĢ ľ your âĢ Ŀ , ĠâĢ ľ his âĢ Ŀ , ĠâĢ ľ her âĢ Ŀ Ġand ĠâĢ ľ the Ġperson âĢ Ŀ Ġas Ġmost Ġcommon Ġalternatives Ġand Ġreplaced Ġeach Ġappearance Ġmanually Ġwith ĠâĢ ľ they âĢ Ŀ Ġor ĠâĢ ľ their âĢ Ŀ , Ġif Ġappropriate . </s> We Ġmanually Ġfiltered Ġout Ġinvalid Ġquestions , Ġe . g . Ġquestions Ġthat Ġare Ġsuggestive Ġ( âĢ ľ Should Ġyou Ġask Ġan Ġadult Ġbefore Ġusing Ġa Ġknife ? âĢ Ŀ ) Ġor Ġthat Ġask Ġfor Ġthe Ġpersonal Ġopinion Ġof Ġthe Ġreader Ġ( âĢ ľ Do Ġyou Ġthink Ġgoing Ġto Ġthe Ġmuseum Ġwas Ġa Ġgood Ġidea ? âĢ Ŀ ). </s> Answer ĠSelection Ġand ĠVal idation </s> We Ġfinalized Ġthe Ġdataset Ġby Ġselecting Ġone Ġcorrect Ġand Ġone Ġincorrect Ġanswer Ġfor Ġeach Ġquestion âĢĵ text Ġpair . ĠTo Ġincrease Ġthe Ġproportion Ġof Ġnon - t riv ial Ġinference Ġcases , Ġwe Ġchose Ġthe Ġcandidate Ġwith Ġthe Ġlowest Ġlex ical Ġoverlap Ġwith Ġthe Ġtext Ġfrom Ġthe Ġset Ġof Ġcorrect Ġanswer Ġcandidates Ġas Ġcorrect Ġanswer . ĠUsing Ġthis Ġprinciple Ġalso Ġfor Ġincorrect Ġanswers Ġleads Ġto Ġproblems . ĠWe Ġfound Ġthat Ġmany Ġincorrect Ġcandidates Ġwere Ġnot Ġplausible Ġanswers Ġto Ġa Ġgiven Ġquestion . ĠInstead Ġof Ġselecting Ġa Ġcandidate Ġbased Ġon Ġoverlap , Ġwe Ġhence Ġdecided Ġto Ġrely Ġon Ġmajority Ġvote Ġand Ġselected Ġthe Ġcandidate Ġfrom Ġthe Ġset Ġof Ġincorrect Ġanswers Ġthat Ġwas Ġmost Ġoften Ġmentioned . </s> For Ġthis Ġstep , Ġwe Ġnormalized Ġeach Ġcandidate Ġby Ġlower c asing , Ġdeleting Ġpunct uation Ġand Ġstop Ġwords Ġ( articles , Ġand , Ġto Ġand Ġor ), Ġand Ġtransforming Ġall Ġnumber Ġwords Ġinto Ġdigits , Ġusing Ġtext 2 num . ĠWe Ġmerged Ġall Ġanswers Ġthat Ġwere Ġstring - ident ical , Ġcontained Ġanother Ġanswer , Ġor Ġhad Ġa ĠLe ven sh tein Ġdistance ĠB IB REF 13 Ġof Ġ3 Ġor Ġless Ġto Ġanother Ġanswer . ĠThe ĠâĢ ľ most Ġfrequent Ġanswer âĢ Ŀ Ġwas Ġthen Ġselected Ġbased Ġon Ġhow Ġmany Ġother Ġanswers Ġit Ġwas Ġmerged Ġwith . ĠOnly Ġif Ġthere Ġwas Ġno Ġmajority , Ġwe Ġselected Ġthe Ġcandidate Ġwith Ġthe Ġhighest Ġoverlap Ġwith Ġthe Ġtext Ġas Ġa Ġfall back . ĠDue Ġto Ġannotation Ġmistakes , Ġwe Ġfound Ġa Ġsmall Ġnumber Ġof Ġchosen Ġcorrect Ġand Ġincorrect Ġanswers Ġto Ġbe Ġinappropriate , Ġthat Ġis , Ġsome ĠâĢ ľ correct âĢ Ŀ Ġanswers Ġwere Ġactually Ġincorrect Ġand Ġvice Ġversa . ĠTherefore , Ġwe Ġmanually Ġvalidated Ġthe Ġcomplete Ġdataset Ġin Ġa Ġfinal Ġstep . ĠWe Ġasked Ġannot ators Ġto Ġread Ġall Ġtexts , Ġquestions , Ġand Ġanswers , Ġand Ġto Ġmark Ġfor Ġeach Ġquestion Ġwhether Ġthe Ġcorrect Ġand Ġincorrect Ġanswers Ġwere Ġappropriate . ĠIf Ġan Ġanswer Ġwas Ġinappropriate Ġor Ġcontained Ġany Ġerrors , Ġthey Ġselected Ġa Ġdifferent Ġanswer Ġfrom Ġthe Ġset Ġof Ġcollected Ġcandidates . ĠFor Ġapproximately Ġ11 . 5 % Ġof Ġthe Ġquestions , Ġat Ġleast Ġone Ġanswer Ġwas Ġreplaced . Ġ135 Ġquestions Ġ( app rox . Ġ1 %) Ġwere Ġexcluded Ġfrom Ġthe Ġdataset Ġbecause Ġno Ġappropriate Ġcorrect Ġor Ġincorrect Ġanswer Ġcould Ġbe Ġfound . </s> Data ĠStatistics </s> For Ġall Ġexperiments , Ġwe Ġadmitted Ġonly Ġexperienced ĠM Tur k Ġworkers Ġwho Ġare Ġbased Ġin Ġthe ĠUS . ĠOne ĠHIT Ġconsisted Ġof Ġwriting Ġone Ġtext Ġfor Ġthe Ġtext Ġcollection , Ġform ulating Ġ15 Ġquestions Ġfor Ġthe Ġquestion Ġcollection , Ġor Ġfinding Ġ15 Ġpairs Ġof Ġanswers Ġfor Ġthe Ġanswer Ġcollection . ĠWe Ġpaid Ġ$ 0 . 50 Ġper ĠHIT Ġfor Ġthe Ġtext Ġand Ġquestion Ġcollection , Ġand Ġ$ 0 . 60 Ġper ĠHIT Ġfor Ġthe Ġanswer Ġcollection . </s> More Ġthan Ġ2 , 100 Ġtexts Ġwere Ġpaired Ġwith Ġ15 Ġquestions Ġeach , Ġresulting Ġin Ġa Ġtotal Ġnumber Ġof Ġapprox . Ġ32 , 000 Ġannot ated Ġquestions . ĠFor Ġ13 % Ġof Ġthe Ġquestions , Ġthe Ġworkers Ġdid Ġnot Ġagree Ġon Ġone Ġof Ġthe Ġ4 Ġcategories Ġwith Ġa Ġ3 Ġout Ġof Ġ5 Ġmajority , Ġso Ġwe Ġdid Ġnot Ġinclude Ġthese Ġquestions Ġin Ġour Ġdataset . </s> The Ġdistribution Ġof Ġcategory Ġlabels Ġon Ġthe Ġremaining Ġ87 % Ġis Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. Ġ14 , 07 4 Ġ( 52 %) Ġquestions Ġcould Ġbe Ġanswered . ĠOut Ġof Ġthe Ġanswer able Ġquestions , Ġ10 , 160 Ġcould Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġdirectly Ġ( text - based ) Ġand Ġ3 , 9 14 Ġquestions Ġrequired Ġthe Ġuse Ġof Ġcommons ense Ġknowledge Ġ( script - based ). ĠAfter Ġremoving Ġ135 Ġquestions Ġduring Ġthe Ġvalidation , Ġthe Ġfinal Ġdataset Ġcomprises Ġ13 , 9 39 Ġquestions , Ġ3 , 8 27 Ġof Ġwhich Ġrequire Ġcommons ense Ġknowledge Ġ( i . e . Ġ27 . 4 %). ĠThis Ġratio Ġwas Ġmanually Ġverified Ġbased Ġon Ġa Ġrandom Ġsample Ġof Ġquestions . </s> We Ġsplit Ġthe Ġdataset Ġinto Ġtraining Ġ( 9 , 7 31 Ġquestions Ġon Ġ1 , 470 Ġtexts ), Ġdevelopment Ġ( 1 , 411 Ġquestions Ġon Ġ219 Ġtexts ), Ġand Ġtest Ġset Ġ( 2 , 797 Ġquestions Ġon Ġ430 Ġtexts ). ĠEach Ġtext Ġappears Ġonly Ġin Ġone Ġof Ġthe Ġthree Ġsets . ĠThe Ġcomplete Ġset Ġof Ġtexts Ġfor Ġ5 Ġscenarios Ġwas Ġheld Ġout Ġfor Ġthe Ġtest Ġset . ĠThe Ġaverage Ġtext , Ġquestion , Ġand Ġanswer Ġlength Ġis Ġ196 . 0 Ġwords , Ġ7 . 8 Ġwords , Ġand Ġ3 . 6 Ġwords , Ġrespectively . ĠOn Ġaverage , Ġthere Ġare Ġ6 . 7 Ġquestions Ġper Ġtext . </s> Figure ĠFIG REF 21 Ġshows Ġthe Ġdistribution Ġof Ġquestion Ġtypes Ġin Ġthe Ġdataset , Ġwhich Ġwe Ġidentified Ġusing Ġsimple Ġhe ur istics Ġbased Ġon Ġthe Ġfirst Ġwords Ġof Ġa Ġquestion : ĠYes / no Ġquestions Ġwere Ġidentified Ġas Ġquestions Ġstarting Ġwith Ġan Ġauxiliary Ġor Ġmod al Ġverb , Ġall Ġother Ġquestion Ġtypes Ġwere Ġdetermined Ġbased Ġon Ġthe Ġquestion Ġword . </s> We Ġfound Ġthat Ġ29 % Ġof Ġall Ġquestions Ġare Ġyes / no Ġquestions . ĠQuestions Ġabout Ġdetails Ġof Ġa Ġsituation Ġ( such Ġas Ġwhat / Ġwhich Ġand Ġwho ) Ġform Ġthe Ġsecond Ġmost Ġfrequent Ġquestion Ġcategory . ĠTem poral Ġquestions Ġ( when Ġand Ġhow Ġlong / often ) Ġform Ġapprox . Ġ11 % Ġof Ġall Ġquestions . ĠWe Ġleave Ġa Ġmore Ġdetailed Ġanalysis Ġof Ġquestion Ġtypes Ġfor Ġfuture Ġwork . </s> Data ĠAnalysis </s> As Ġcan Ġbe Ġseen Ġfrom Ġthe Ġdata Ġstatistics , Ġour Ġmode Ġof Ġcollection Ġleads Ġto Ġa Ġsubstantial Ġproportion Ġof Ġquestions Ġthat Ġrequire Ġinference Ġusing Ġcommons ense Ġknowledge . ĠStill , Ġthe Ġdataset Ġcontains Ġa Ġlarge Ġnumber Ġof Ġquestions Ġin Ġwhich Ġthe Ġanswer Ġis Ġexplicitly Ġcontained Ġor Ġimplied Ġby Ġthe Ġtext : ĠFigure ĠFIG REF 22 Ġshows Ġpassages Ġfrom Ġan Ġexample Ġtext Ġof Ġthe Ġdataset Ġtogether Ġwith Ġtwo Ġsuch Ġquestions . ĠFor Ġquestion ĠQ 1 , Ġthe Ġanswer Ġis Ġgiven Ġliterally Ġin Ġthe Ġtext . ĠAn sw ering Ġquestion ĠQ 2 Ġis Ġnot Ġas Ġsimple ; Ġit Ġcan Ġbe Ġsolved , Ġhowever , Ġvia Ġstandard Ġsemantic Ġrelated ness Ġinformation Ġ( ch icken Ġand Ġhot dogs Ġare Ġmeat ; Ġwater , Ġsoda Ġand Ġjuice Ġare Ġdrinks ). </s> The Ġfollowing Ġcases Ġrequire Ġcommons ense Ġinference Ġto Ġbe Ġdecided . ĠIn Ġall Ġthese Ġcases , Ġthe Ġanswers Ġare Ġnot Ġovertly Ġcontained Ġnor Ġeasily Ġderiv able Ġfrom Ġthe Ġrespective Ġtexts . ĠWe Ġdo Ġnot Ġshow Ġthe Ġfull Ġtexts , Ġbut Ġonly Ġthe Ġscenario Ġnames Ġfor Ġeach Ġquestion . </s> Example ĠUID 23 Ġrefers Ġto Ġa Ġlibrary Ġsetting . ĠScript Ġknowledge Ġhelps Ġin Ġassessing Ġthat Ġusually , Ġpaying Ġis Ġnot Ġan Ġevent Ġwhen Ġborrowing Ġa Ġbook , Ġwhich Ġanswers Ġthe Ġquestion . ĠSimilarly , Ġevent Ġinformation Ġhelps Ġin Ġanswering Ġthe Ġquestions Ġin ĠExamples ĠUID 24 Ġand ĠUID 25 Ġ. ĠIn ĠExample ĠUID 26 Ġ, Ġknowledge Ġabout Ġthe Ġtypical Ġrole Ġof Ġparents Ġin Ġthe Ġpreparation Ġof Ġa Ġpicnic Ġwill Ġenable Ġa Ġplaus ibility Ġdecision . ĠSimilarly , Ġin ĠExample ĠUID 27 Ġ, Ġit Ġis Ġcommons ense Ġknowledge Ġthat Ġshowers Ġusually Ġtake Ġa Ġfew Ġminutes Ġrather Ġthan Ġhours . </s> There Ġare Ġalso Ġcases Ġin Ġwhich Ġthe Ġanswer Ġcan Ġbe Ġinferred Ġfrom Ġthe Ġtext , Ġbut Ġwhere Ġcommons ense Ġknowledge Ġis Ġstill Ġbeneficial : ĠThe Ġtext Ġfor Ġexample ĠUID 28 Ġdoes Ġnot Ġcontain Ġthe Ġinformation Ġthat Ġbreakfast Ġis Ġeaten Ġin Ġthe Ġmorning , Ġbut Ġit Ġcould Ġstill Ġbe Ġinferred Ġfrom Ġmany Ġpointers Ġin Ġthe Ġtext Ġ( e . g . Ġphrases Ġsuch Ġas ĠI Ġwoke Ġup ), Ġor Ġfrom Ġcommons ense Ġknowledge . </s> These Ġfew Ġexamples Ġillustrate Ġthat Ġour Ġdataset Ġcovers Ġquestions Ġwith Ġa Ġwide Ġspectrum Ġof Ġdifficulty , Ġfrom Ġrather Ġsimple Ġquestions Ġthat Ġcan Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġto Ġchallenging Ġinference Ġproblems . </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġassess Ġthe Ġperformance Ġof Ġbaseline Ġmodels Ġon ĠMC Script , Ġusing Ġaccuracy Ġas Ġthe Ġevaluation Ġmeasure . ĠWe Ġemploy Ġmodels Ġof Ġdiffering Ġcomplexity : Ġtwo Ġun super vised Ġmodels Ġusing Ġonly Ġword Ġinformation Ġand Ġdistribution al Ġinformation , Ġrespectively , Ġand Ġtwo Ġsupervised Ġneural Ġmodels . ĠWe Ġassess Ġperformance Ġon Ġtwo Ġdimensions : ĠOne , Ġwe Ġshow Ġhow Ġwell Ġthe Ġmodels Ġperform Ġon Ġtext - based Ġquestions Ġas Ġcompared Ġto Ġquestions Ġthat Ġrequire Ġcommon Ġsense Ġfor Ġfinding Ġthe Ġcorrect Ġanswer . ĠTwo , Ġwe Ġevaluate Ġeach Ġmodel Ġfor Ġeach Ġdifferent Ġquestion Ġtype . </s> Mod els </s> We Ġfirst Ġuse Ġa Ġsimple Ġword Ġmatching Ġbaseline , Ġby Ġselecting Ġthe Ġanswer Ġthat Ġhas Ġthe Ġhighest Ġliteral Ġoverlap Ġwith Ġthe Ġtext . ĠIn Ġcase Ġof Ġa Ġtie , Ġwe Ġrandomly Ġselect Ġone Ġof Ġthe Ġanswers . </s> The Ġsecond Ġbaseline Ġis Ġa Ġsliding Ġwindow Ġapproach Ġthat Ġlooks Ġat Ġwindows Ġof ĠIN LINE FORM 0 Ġtokens Ġon Ġthe Ġtext . ĠEach Ġtext Ġand Ġeach Ġanswer Ġare Ġrepresented Ġas Ġa Ġsequence Ġof Ġword Ġembed d ings . ĠThe Ġembed d ings Ġfor Ġeach Ġwindow Ġof Ġsize ĠIN LINE FORM 1 Ġand Ġeach Ġanswer Ġare Ġthen Ġaveraged Ġto Ġderive Ġwindow Ġand Ġanswer Ġrepresentations , Ġrespectively . ĠThe Ġanswer Ġwith Ġthe Ġlowest Ġcos ine Ġdistance Ġto Ġone Ġof Ġthe Ġwindows Ġof Ġthe Ġtext Ġis Ġthen Ġselected Ġas Ġcorrect . </s> We Ġemploy Ġa Ġsimple Ġneural Ġmodel Ġas Ġa Ġthird Ġbaseline . ĠIn Ġthis Ġmodel , Ġeach Ġtext , Ġquestion , Ġand Ġanswer Ġis Ġrepresented Ġby Ġa Ġvector . ĠFor Ġa Ġgiven Ġsequence Ġof Ġwords ĠIN LINE FORM 0 Ġ, Ġwe Ġcompute Ġthis Ġrepresentation Ġby Ġaveraging Ġover Ġthe Ġcomponents Ġof Ġthe Ġword Ġembed d ings ĠIN LINE FORM 1 Ġthat Ġcorrespond Ġto Ġa Ġword ĠIN LINE FORM 2 Ġ, Ġand Ġthen Ġapply Ġa Ġlinear Ġtransformation Ġusing Ġa Ġweight Ġmatrix . ĠThis Ġprocedure Ġis Ġapplied Ġto Ġeach Ġanswer ĠIN LINE FORM 3 Ġto Ġderive Ġan Ġanswer Ġrepresentation ĠIN LINE FORM 4 Ġ. ĠThe Ġrepresentation Ġof Ġa Ġtext ĠIN LINE FORM 5 Ġand Ġof Ġa Ġquestion ĠIN LINE FORM 6 Ġare Ġcomputed Ġin Ġthe Ġsame Ġway . ĠWe Ġuse Ġdifferent Ġweight Ġmat rices Ġfor ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġrespectively . ĠA Ġcombined Ġrepresentation ĠIN LINE FORM 10 Ġfor Ġthe Ġtext âĢĵ question Ġpair Ġis Ġthen Ġconstructed Ġusing Ġa Ġbil in ear Ġtransformation Ġmatrix ĠIN LINE FORM 11 Ġ: ĠDIS PLAY FORM 0 </s> We Ġcompute Ġa Ġscore Ġfor Ġeach Ġanswer Ġby Ġusing Ġthe Ġdot Ġproduct Ġand Ġpass Ġthe Ġscores Ġfor Ġboth Ġanswers Ġthrough Ġa Ġsoft max Ġlayer Ġfor Ġprediction . ĠThe Ġprobability ĠIN LINE FORM 0 Ġfor Ġan Ġanswer ĠIN LINE FORM 1 Ġto Ġbe Ġcorrect Ġis Ġthus Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> The Ġattentive Ġreader Ġis Ġa Ġwell - established Ġmachine Ġcomprehension Ġmodel Ġthat Ġreaches Ġgood Ġperformance Ġe . g . Ġon Ġthe ĠCNN / Daily ĠMail Ġcorpus ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠWe Ġuse Ġthe Ġmodel Ġformulation Ġby Ġc hen 2016 th orough Ġand Ġl ai 2017 race , Ġwho Ġemploy Ġbil in ear Ġweight Ġfunctions Ġto Ġcompute Ġboth Ġattention Ġand Ġanswer - text Ġfit . ĠBi - direction al ĠGR Us Ġare Ġused Ġto Ġencode Ġquestions , Ġtexts Ġand Ġanswers Ġinto Ġhidden Ġrepresentations . ĠFor Ġa Ġquestion ĠIN LINE FORM 0 Ġand Ġan Ġanswer ĠIN LINE FORM 1 Ġ, Ġthe Ġlast Ġstate Ġof Ġthe ĠGR Us , ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġare Ġused Ġas Ġrepresentations , Ġwhile Ġthe Ġtext Ġis Ġencoded Ġas Ġa Ġsequence Ġof Ġhidden Ġstates ĠIN LINE FORM 4 Ġ. ĠWe Ġthen Ġcompute Ġan Ġattention Ġscore ĠIN LINE FORM 5 Ġfor Ġeach Ġhidden Ġstate ĠIN LINE FORM 6 Ġusing Ġthe Ġquestion Ġrepresentation ĠIN LINE FORM 7 Ġ, Ġa Ġweight Ġmatrix ĠIN LINE FORM 8 Ġ, Ġand Ġan Ġattention Ġbias ĠIN LINE FORM 9 Ġ. ĠLast , Ġa Ġtext Ġrepresentation ĠIN LINE FORM 10 Ġis Ġcomputed Ġas Ġa Ġweighted Ġaverage Ġof Ġthe Ġhidden Ġrepresentations : ĠDIS PLAY FORM 0 </s> The Ġprobability ĠIN LINE FORM 0 Ġof Ġanswer ĠIN LINE FORM 1 Ġbeing Ġcorrect Ġis Ġthen Ġpredicted Ġusing Ġanother Ġbil in ear Ġweight Ġmatrix ĠIN LINE FORM 2 Ġ, Ġfollowed Ġby Ġan Ġapplication Ġof Ġthe Ġsoft max Ġfunction Ġover Ġboth Ġanswer Ġoptions Ġfor Ġthe Ġquestion : ĠDIS PLAY FORM 0 </s> Im plementation ĠDetails </s> Text s , Ġquestions Ġand Ġanswers Ġwere Ġtoken ized Ġusing ĠNL T K Ġand Ġlower c ased . ĠWe Ġused Ġ100 - dimensional ĠGl o Ve Ġvectors ĠB IB REF 16 Ġto Ġembed Ġeach Ġtoken . ĠFor Ġthe Ġneural Ġmodels , Ġthe Ġembed d ings Ġare Ġused Ġto Ġinitialize Ġthe Ġtoken Ġrepresentations , Ġand Ġare Ġrefined Ġduring Ġtraining . ĠFor Ġthe Ġsliding Ġsimilarity Ġwindow Ġapproach , Ġwe Ġset ĠIN LINE FORM 0 Ġ. </s> The Ġvocabulary Ġof Ġthe Ġneural Ġmodels Ġwas Ġextracted Ġfrom Ġtraining Ġand Ġdevelopment Ġdata . ĠFor Ġoptimizing Ġthe Ġbil in ear Ġmodel Ġand Ġthe Ġattentive Ġreader , Ġwe Ġused Ġvanilla Ġsto ch astic Ġgradient Ġdescent Ġwith Ġgradient Ġclipping , Ġif Ġthe Ġnorm Ġof Ġgrad ients Ġexceeds Ġ10 . ĠThe Ġsize Ġof Ġthe Ġhidden Ġlayers Ġwas Ġtuned Ġto Ġ64 , Ġwith Ġa Ġlearning Ġrate Ġof ĠIN LINE FORM 0 Ġ, Ġfor Ġboth Ġmodels . ĠWe Ġapply Ġa Ġdrop out Ġof ĠIN LINE FORM 1 Ġto Ġthe Ġword Ġembed d ings . ĠB atch Ġsize Ġwas Ġset Ġto Ġ25 Ġand Ġall Ġmodels Ġwere Ġtrained Ġfor Ġ150 Ġepoch s . ĠDuring Ġtraining , Ġwe Ġmeasured Ġperformance Ġon Ġthe Ġdevelopment Ġset , Ġand Ġwe Ġselected Ġthe Ġmodel Ġfrom Ġthe Ġbest Ġperforming Ġepoch Ġfor Ġtesting . </s> Results Ġand ĠEvaluation </s> As Ġan Ġupper Ġbound Ġfor Ġmodel Ġperformance , Ġwe Ġassess Ġhow Ġwell Ġhumans Ġcan Ġsolve Ġour Ġtask . ĠTwo Ġtrained Ġannot ators Ġlabeled Ġthe Ġcorrect Ġanswer Ġon Ġall Ġinstances Ġof Ġthe Ġtest Ġset . ĠThey Ġagreed Ġwith Ġthe Ġgold Ġstandard Ġin Ġ98 . 2 Ġ% Ġof Ġcases . ĠThis Ġresult Ġshows Ġthat Ġhumans Ġhave Ġno Ġdifficulty Ġin Ġfinding Ġthe Ġcorrect Ġanswer , Ġirrespective Ġof Ġthe Ġquestion Ġtype . </s> Table ĠT AB REF 37 Ġshows Ġthe Ġperformance Ġof Ġthe Ġbaseline Ġmodels Ġas Ġcompared Ġto Ġthe Ġhuman Ġupper Ġbound Ġand Ġa Ġrandom Ġbaseline . ĠAs Ġcan Ġbe Ġseen , Ġneural Ġmodels Ġhave Ġa Ġclear Ġadvantage Ġover Ġthe Ġpure Ġword Ġoverlap Ġbaseline , Ġwhich Ġperforms Ġworst , Ġwith Ġan Ġaccuracy Ġof ĠIN LINE FORM 0 Ġ. </s> The Ġlow Ġaccuracy Ġis Ġmostly Ġdue Ġto Ġthe Ġnature Ġof Ġcorrect Ġanswers Ġin Ġour Ġdata : ĠEach Ġcorrect Ġanswer Ġhas Ġa Ġlow Ġoverlap Ġwith Ġthe Ġtext Ġby Ġdesign . ĠSince Ġthe Ġoverlap Ġmodel Ġselects Ġthe Ġanswer Ġwith Ġa Ġhigh Ġoverlap Ġto Ġthe Ġtext , Ġit Ġdoes Ġnot Ġperform Ġwell . ĠIn Ġparticular , Ġthis Ġalso Ġexplains Ġthe Ġvery Ġbad Ġresult Ġon Ġtext - based Ġquestions . ĠThe Ġsliding Ġsimilarity Ġwindow Ġmodel Ġdoes Ġnot Ġoutper form Ġthe Ġsimple Ġword Ġoverlap Ġmodel Ġby Ġa Ġlarge Ġmargin : ĠDistribution al Ġinformation Ġalone Ġis Ġinsufficient Ġto Ġhandle Ġcomplex Ġquestions Ġin Ġthe Ġdataset . </s> Both Ġneural Ġmodels Ġoutper form Ġthe Ġun super vised Ġbas elines Ġby Ġa Ġlarge Ġmargin . ĠWhen Ġcomparing Ġthe Ġtwo Ġmodels , Ġthe Ġattentive Ġreader Ġis Ġable Ġto Ġbeat Ġthe Ġbil in ear Ġmodel Ġby Ġonly ĠIN LINE FORM 0 Ġ. ĠA Ġpossible Ġexplanation Ġfor Ġthis Ġis Ġthat Ġthe Ġattentive Ġreader Ġonly Ġattends Ġto Ġthe Ġtext . ĠSince Ġmany Ġquestions Ġcannot Ġbe Ġdirectly Ġanswered Ġfrom Ġthe Ġtext , Ġthe Ġattentive Ġreader Ġis Ġnot Ġable Ġto Ġperform Ġsignificantly Ġbetter Ġthan Ġa Ġsimpler Ġneural Ġmodel . </s> What Ġis Ġsurprising Ġis Ġthat Ġthe Ġattentive Ġreader Ġworks Ġbetter Ġon Ġcommons ense - based Ġquestions Ġthan Ġon Ġtext Ġquestions . ĠThis Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġmany Ġcommons ense Ġquestions Ġdo Ġhave Ġprototyp ical Ġanswers Ġwithin Ġa Ġscenario , Ġirrespective Ġof Ġthe Ġtext . ĠThe Ġattentive Ġreader Ġis Ġapparently Ġable Ġto Ġjust Ġmemor ize Ġthese Ġprototyp ical Ġanswers , Ġthus Ġachieving Ġhigher Ġaccuracy . </s> In spect ing Ġattention Ġvalues Ġof Ġthe Ġattentive Ġreader , Ġwe Ġfound Ġthat Ġin Ġmost Ġcases , Ġthe Ġmodel Ġis Ġunable Ġto Ġproperly Ġattend Ġto Ġthe Ġrelevant Ġparts Ġof Ġthe Ġtext , Ġeven Ġwhen Ġthe Ġanswer Ġis Ġliterally Ġgiven Ġin Ġthe Ġtext . ĠA Ġpossible Ġexplanation Ġis Ġthat Ġthe Ġmodel Ġis Ġconfused Ġby Ġthe Ġlarge Ġamount Ġof Ġquestions Ġthat Ġcannot Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġdirectly , Ġwhich Ġmight Ġconf ound Ġthe Ġcomputation Ġof Ġattention Ġvalues . </s> Also , Ġthe Ġattentive Ġreader Ġwas Ġoriginally Ġconstructed Ġfor Ġreconstruct ing Ġliteral Ġtext Ġspans Ġas Ġanswers . ĠOur Ġmode Ġof Ġanswer Ġcollection , Ġhowever , Ġresults Ġin Ġmany Ġcorrect Ġanswers Ġthat Ġcannot Ġbe Ġfound Ġver bat im Ġin Ġthe Ġtext . ĠThis Ġpresents Ġdifficulties Ġfor Ġthe Ġattention Ġmechanism . </s> The Ġfact Ġthat Ġan Ġattention Ġmodel Ġoutper forms Ġa Ġsimple Ġbil in ear Ġbaseline Ġonly Ġmarginally Ġshows Ġthat ĠMC Script Ġposes Ġa Ġnew Ġchallenge Ġto Ġmachine Ġcomprehension Ġsystems . ĠModels Ġconcentrating Ġsolely Ġon Ġthe Ġtext Ġare Ġinsufficient Ġto Ġperform Ġwell Ġon Ġthe Ġdata . </s> Figure ĠFIG REF 39 Ġgives Ġaccuracy Ġvalues Ġof Ġall Ġbaseline Ġsystems Ġon Ġthe Ġmost Ġfrequent Ġquestion Ġtypes Ġ( app earing Ġ> 25 Ġtimes Ġin Ġthe Ġtest Ġdata ), Ġas Ġdetermined Ġbased Ġon Ġthe Ġquestion Ġwords Ġ( see ĠSection ĠSEC REF 19 Ġ). ĠThe Ġnumbers Ġdepicted Ġon Ġthe Ġleft - hand Ġside Ġof Ġthe Ġy - axis Ġrepresent Ġmodel Ġaccuracy . ĠThe Ġright - hand Ġside Ġof Ġthe Ġy - axis Ġindicates Ġthe Ġnumber Ġof Ġtimes Ġa Ġquestion Ġtype Ġappears Ġin Ġthe Ġtest Ġdata . </s> The Ġneural Ġmodels Ġunsurprisingly Ġoutper form Ġthe Ġother Ġmodels Ġin Ġmost Ġcases , Ġand Ġthe Ġdifference Ġfor Ġwho Ġquestions Ġis Ġlargest . ĠA Ġlarge Ġnumber Ġof Ġthese Ġquestions Ġask Ġfor Ġthe Ġnarrator Ġof Ġthe Ġstory , Ġwho Ġis Ġusually Ġnot Ġmentioned Ġliterally Ġin Ġthe Ġtext , Ġsince Ġmost Ġstories Ġare Ġwritten Ġin Ġthe Ġfirst Ġperson . </s> It Ġis Ġalso Ġapparent Ġthat Ġall Ġmodels Ġperform Ġrather Ġbadly Ġon Ġyes / no Ġquestions . ĠEach Ġmodel Ġbasically Ġcompares Ġthe Ġanswer Ġto Ġsome Ġrepresentation Ġof Ġthe Ġtext . ĠFor Ġyes / no Ġquestions , Ġthis Ġmakes Ġsense Ġfor Ġless Ġthan Ġhalf Ġof Ġall Ġcases . ĠFor Ġthe Ġmajority Ġof Ġyes / no Ġquestions , Ġhowever , Ġanswers Ġconsist Ġonly Ġof Ġyes Ġor Ġno , Ġwithout Ġfurther Ġcontent Ġwords . </s> Related ĠWork </s> In Ġrecent Ġyears , Ġa Ġnumber Ġof Ġreading Ġcomprehension Ġdatasets Ġhave Ġbeen Ġproposed , Ġincluding ĠM CT est ĠB IB REF 17 Ġ, ĠB Ab I ĠB IB REF 18 Ġ, Ġthe ĠChildren 's ĠBook ĠTest Ġ( C BT , Ġhill 2015 gold il ocks ), ĠCNN / Daily ĠMail ĠB IB REF 14 Ġ, Ġthe ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD , Ġr aj pur kar 2016 squ ad ), Ġand ĠR ACE ĠB IB REF 19 Ġ. ĠThese Ġdatasets Ġdiffer Ġwith Ġrespect Ġto Ġtext Ġtype Ġ( Wikipedia Ġtexts , Ġexamination Ġtexts , Ġetc .), Ġmode Ġof Ġanswer Ġselection Ġ( span - based , Ġmultiple Ġchoice , Ġetc .) Ġand Ġtest Ġsystems Ġregarding Ġdifferent Ġaspects Ġof Ġlanguage Ġunderstanding , Ġbut Ġthey Ġdo Ġnot Ġexplicitly Ġaddress Ġcommons ense Ġknowledge . </s> Two Ġnotable Ġexceptions Ġare Ġthe ĠNews Q A Ġand ĠTri via Q A Ġdatasets . ĠNews Q A ĠB IB REF 20 Ġis Ġa Ġdataset Ġof Ġnews wire Ġtexts Ġfrom ĠCNN Ġwith Ġquestions Ġand Ġanswers Ġwritten Ġby Ġcrowds ourcing Ġworkers . ĠNews Q A Ġclosely Ġresembles Ġour Ġown Ġdata Ġcollection Ġwith Ġrespect Ġto Ġthe Ġmethod Ġof Ġdata Ġacquisition . ĠAs Ġfor Ġour Ġdata Ġcollection , Ġfull Ġtexts Ġwere Ġnot Ġshown Ġto Ġworkers Ġas Ġa Ġbasis Ġfor Ġquestion Ġformulation , Ġbut Ġonly Ġthe Ġtext 's Ġtitle Ġand Ġa Ġshort Ġsummary , Ġto Ġavoid Ġliteral Ġrepet itions Ġand Ġsupport Ġthe Ġgeneration Ġof Ġnon - t riv ial Ġquestions Ġrequiring Ġbackground Ġknowledge . ĠThe ĠNews Q A Ġtext Ġcollection Ġdiffers Ġfrom Ġours Ġin Ġdomain Ġand Ġgenre Ġ( new sw ire Ġtexts Ġvs . Ġnarrative Ġstories Ġabout Ġeveryday Ġevents ). ĠKnowledge Ġrequired Ġto Ġanswer Ġthe Ġquestions Ġis Ġmostly Ġfactual Ġknowledge Ġand Ġscript Ġknowledge Ġis Ġonly Ġmarginally Ġrelevant . ĠAlso , Ġthe Ġtask Ġis Ġnot Ġexactly Ġquestion Ġanswering , Ġbut Ġidentification Ġof Ġdocument Ġpassages Ġcontaining Ġthe Ġanswer . </s> Trivia Q A ĠB IB REF 21 Ġis Ġa Ġcorpus Ġthat Ġcontains Ġautomatically Ġcollected Ġquestion - answer Ġpairs Ġfrom Ġ14 Ġtrivia Ġand Ġquiz - league Ġwebsites , Ġtogether Ġwith Ġweb - c rawled Ġevidence Ġdocuments Ġfrom ĠWikipedia Ġand ĠBing . ĠWhile Ġa Ġmajority Ġof Ġquestions Ġrequire Ġworld Ġknowledge Ġfor Ġfinding Ġthe Ġcorrect Ġanswer , Ġit Ġis Ġmostly Ġfactual Ġknowledge . </s> Summary </s> We Ġpresent Ġa Ġnew Ġdataset Ġfor Ġthe Ġtask Ġof Ġmachine Ġcomprehension Ġfoc uss ing Ġon Ġcommons ense Ġknowledge . ĠQuestions Ġwere Ġcollected Ġbased Ġon Ġscript Ġscenarios , Ġrather Ġthan Ġindividual Ġtexts , Ġwhich Ġresulted Ġin Ġquestion âĢĵ answer Ġpairs Ġthat Ġexplicitly Ġinvolve Ġcommons ense Ġknowledge . ĠIn Ġcontrast Ġto Ġprevious Ġevaluation Ġtasks , Ġthis Ġsetup Ġallows Ġus Ġfor Ġthe Ġfirst Ġtime Ġto Ġassess Ġthe Ġcontribution Ġof Ġscript Ġknowledge Ġfor Ġcomputational Ġmodels Ġof Ġlanguage Ġunderstanding Ġin Ġa Ġreal - world Ġevaluation Ġscenario . </s> We Ġexpect Ġour Ġdataset Ġto Ġbecome Ġa Ġstandard Ġbenchmark Ġfor Ġtesting Ġmodels Ġof Ġcommons ense Ġand Ġscript Ġknowledge . ĠHuman Ġperformance Ġshows Ġthat Ġthe Ġdataset Ġis Ġhighly Ġreliable . ĠThe Ġresults Ġof Ġseveral Ġbas elines , Ġin Ġcontrast , Ġillustrate Ġthat Ġour Ġtask Ġprovides Ġchallenging Ġtest Ġcases Ġfor Ġthe Ġbroader Ġnatural Ġlanguage Ġprocessing Ġcommunity . ĠMC Script Ġforms Ġthe Ġbasis Ġof Ġa Ġshared Ġtask Ġorganized Ġat ĠSem E val Ġ2018 . ĠThe Ġdataset Ġis Ġavailable Ġat Ġhttp :// www . s fb 110 2 . uni - sa ar land . de /? page _ id = 25 82 . </s> Acknowled gements </s> We Ġthank Ġthe Ġreviewers Ġfor Ġtheir Ġhelpful Ġcomments . ĠWe Ġalso Ġthank ĠFlor ian ĠP us se Ġfor Ġthe Ġhelp Ġwith Ġthe ĠM Tur k Ġexperiments Ġand Ġour Ġstudent Ġassistants ĠChristine ĠSch Ãĥ Â ¤ fer , ĠDam y ana ĠGate va , ĠLeon ie ĠHar ter , ĠSarah ĠM ame che , ĠStefan ĠGr Ãĥ Â ¼ new ald Ġand ĠTat iana ĠAn ik ina Ġfor Ġhelp Ġwith Ġthe Ġannotations . ĠThis Ġresearch Ġwas Ġfunded Ġby Ġthe ĠGerman ĠResearch ĠFoundation Ġ( DF G ) Ġas Ġpart Ġof ĠSF B Ġ110 2 Ġ` Information ĠD ensity Ġand ĠL ingu istic ĠEnc oding ' Ġand ĠEX C Ġ284 Ġ` Mult im od al ĠComputing Ġand ĠInter action '.
Introduction Ambiguity and implicitness are inherent properties of natural language that cause challenges for computational models of language understanding. In everyday communication, people assume a shared common ground which forms a basis for efficiently resolving ambiguities and for inferring implicit information. Thus, recoverable information is often left unmentioned or underspecified. Such information may include encyclopedic and commonsense knowledge. This work focuses on commonsense knowledge about everyday activities, so-called scripts. This paper introduces a dataset to evaluate natural language understanding approaches with a focus on interpretation processes requiring inference based on commonsense knowledge. In particular, we present MCScript, a dataset for assessing the contribution of script knowledge to machine comprehension. Scripts are sequences of events describing stereotypical human activities (also called scenarios), for example baking a cake or taking a bus BIBREF0 . To illustrate the importance of script knowledge, consider Example ( SECREF1 ): Without using commonsense knowledge, it may be difficult to tell who ate the food: Rachel or the waitress. In contrast, if we utilize commonsense knowledge, in particular, script knowledge about the eating in a restaurant scenario, we can make the following inferences: Rachel is most likely a customer, since she received an order. It is usually the customer, and not the waitress, who eats the ordered food. So She most likely refers to Rachel. Various approaches for script knowledge extraction and processing have been proposed in recent years. However, systems have been evaluated for specific aspects of script knowledge only, such as event ordering BIBREF1 , BIBREF2 , event paraphrasing BIBREF3 , BIBREF4 or event prediction (namely, the narrative cloze task BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 ). These evaluation methods lack a clear connection to real-world tasks. Our MCScript dataset provides an extrinsic evaluation framework, based on text comprehension involving commonsense knowledge. This framework makes it possible to assess system performance in a multiple-choice question answering setting, without imposing any specific structural or methodical requirements. MCScript is a collection of (1) narrative texts, (2) questions of various types referring to these texts, and (3) pairs of answer candidates for each question. It comprises approx. 2,100 texts and a total of approx. 14,000 questions. Answering a substantial subset of questions requires knowledge beyond the facts mentioned in the text, i.e. it requires inference using commonsense knowledge about everyday activities. An example is given in Figure FIGREF2 . For both questions, the correct choice for an answer requires commonsense knowledge about the activity of planting a tree, which goes beyond what is mentioned in the text. Texts, questions, and answers were obtained through crowdsourcing. In order to ensure high quality, we manually validated and filtered the dataset. Due to our design of the data acquisition process, we ended up with a substantial subset of questions that require commonsense inference (27.4%). Corpus Machine comprehension datasets consist of three main components: texts, questions and answers. In this section, we describe our data collection for these 3 components. We first describe a series of pilot studies that we conducted in order to collect commonsense inference questions (Section SECREF4 ). In Section SECREF5 , we discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk (henceforth MTurk). Section SECREF17 gives information about some necessary postprocessing steps and the dataset validation. Lastly, Section SECREF19 gives statistics about the final dataset. Pilot Study As a starting point for our pilots, we made use of texts from the InScript corpus BIBREF10 , which provides stories centered around everyday situations (see Section SECREF7 ). We conducted three different pilot studies to determine the best way of collecting questions that require inference over commonsense knowledge: The most intuitive way of collecting reading comprehension questions is to show texts to workers and let them formulate questions and answers on the texts, which is what we tried internally in a first pilot. Since our focus is to provide an evaluation framework for inference over commonsense knowledge, we manually assessed the number of questions that indeed require common sense knowledge. We found too many questions and answers collected in this manner to be lexically close to the text. In a second pilot, we investigated the option to take the questions collected for one text and show them as questions for another text of the same scenario. While this method resulted in a larger number of questions that required inference, we found the majority of questions to not make sense at all when paired with another text. Many questions were specific to a text (and not to a scenario), requiring details that could not be answered from other texts. Since the two previous pilot setups resulted in questions that centered around the texts themselves, we decided for a third pilot to not show workers any specific texts at all. Instead, we asked for questions that centered around a specific script scenario (e.g. eating in a restaurant). We found this mode of collection to result in questions that have the right level of specificity for our purposes: namely, questions that are related to a scenario and that can be answered from different texts (about that scenario), but for which a text does not need to provide the answer explicitly. The next section will describe the mode of collection chosen for the final dataset, based on the third pilot, in more detail. Data Collection As mentioned in the previous section, we decided to base the question collection on script scenarios rather than specific texts. As a starting point for our data collection, we use scenarios from three script data collections BIBREF3 , BIBREF11 , BIBREF12 . Together, these resources contain more than 200 scenarios. To make sure that scenarios have different complexity and content, we selected 80 of them and came up with 20 new scenarios. Together with the 10 scenarios from InScript, we end up with a total of 110 scenarios. For the collection of texts, we followed modiinscript, where workers were asked to write a story about a given activity “as if explaining it to a child”. This results in elaborate and explicit texts that are centered around a single scenario. Consequently, the texts are syntactically simple, facilitating machine comprehension models to focus on semantic challenges and inference. We collected 20 texts for each scenario. Each participant was allowed to write only one story per scenario, but work on as many scenarios as they liked. For each of the 10 scenarios from InScript, we randomly selected 20 existing texts from that resource. For collecting questions, workers were instructed to “imagine they told a story about a certain scenario to a child and want to test if the child understood everything correctly”. This instruction also ensured that questions are linguistically simple, elaborate and explicit. Workers were asked to formulate questions about details of such a situation, i.e. independent of a concrete narrative. This resulted in questions, the answer to which is not literally mentioned in the text. To cover a broad range of question types, we asked participants to write 3 temporal questions (asking about time points and event order), 3 content questions (asking about persons or details in the scenario) and 3 reasoning questions (asking how or why something happened). They were also asked to formulate 6 free questions, which resulted in a total of 15 questions. Asking each worker for a high number of questions enforced that more creative questions were formulated, which go beyond obvious questions for a scenario. Since participants were not shown a concrete story, we asked them to use the neutral pronoun “they” to address the protagonist of the story. We permitted participants to work on as many scenarios as desired and we collected questions from 10 participants per scenario. Our mode of question collection results in questions that are not associated with specific texts. For each text, we collected answers for 15 questions that were randomly selected from the same scenario. Since questions and texts were collected independently, answering a random question is not always possible for a given text. Therefore, we carried out answer collection in two steps. In the first step, we asked participants to assign a category to each text–question pair. We distinguish two categories of answerable questions: The category text-based was assigned to questions that can be answered from the text directly. If the answer could only be inferred by using commonsense knowledge, the category script-based was assigned. Making this distinction is interesting for evaluation purposes, since it enables us to estimate the number of commonsense inference questions. For questions that did not make sense at all given a text, unfitting was assigned. If a question made sense for a text, but it was impossible to find an answer, the label unknown was used. In a second step, we told participants to formulate a plausible correct and a plausible incorrect answer candidate to answerable questions (text-based or script-based). To level out the effort between answerable and non-answerable questions, participants had to write a new question when selecting unknown or unfitting. In order to get reliable judgments about whether or not a question can be answered, we collected data from 5 participants for each question and decided on the final category via majority vote (at least 3 out of 5). Consequently, for each question with a majority vote on either text-based or script-based, there are 3 to 5 correct and incorrect answer candidates, one from each participant who agreed on the category. Questions without a clear majority vote or with ties were not included in the dataset. We performed four post-processing steps on the collected data. We manually filtered out texts that were instructional rather than narrative. All texts, questions and answers were spellchecked by running aSpell and manually inspecting all corrections proposed by the spellchecker. We found that some participants did not use “they” when referring to the protagonist. We identified “I”, “you”, “he”, “she”, “my”, “your”, “his”, “her” and “the person” as most common alternatives and replaced each appearance manually with “they” or “their”, if appropriate. We manually filtered out invalid questions, e.g. questions that are suggestive (“Should you ask an adult before using a knife?”) or that ask for the personal opinion of the reader (“Do you think going to the museum was a good idea?”). Answer Selection and Validation We finalized the dataset by selecting one correct and one incorrect answer for each question–text pair. To increase the proportion of non-trivial inference cases, we chose the candidate with the lowest lexical overlap with the text from the set of correct answer candidates as correct answer. Using this principle also for incorrect answers leads to problems. We found that many incorrect candidates were not plausible answers to a given question. Instead of selecting a candidate based on overlap, we hence decided to rely on majority vote and selected the candidate from the set of incorrect answers that was most often mentioned. For this step, we normalized each candidate by lowercasing, deleting punctuation and stop words (articles, and, to and or), and transforming all number words into digits, using text2num. We merged all answers that were string-identical, contained another answer, or had a Levenshtein distance BIBREF13 of 3 or less to another answer. The “most frequent answer” was then selected based on how many other answers it was merged with. Only if there was no majority, we selected the candidate with the highest overlap with the text as a fallback. Due to annotation mistakes, we found a small number of chosen correct and incorrect answers to be inappropriate, that is, some “correct” answers were actually incorrect and vice versa. Therefore, we manually validated the complete dataset in a final step. We asked annotators to read all texts, questions, and answers, and to mark for each question whether the correct and incorrect answers were appropriate. If an answer was inappropriate or contained any errors, they selected a different answer from the set of collected candidates. For approximately 11.5% of the questions, at least one answer was replaced. 135 questions (approx. 1%) were excluded from the dataset because no appropriate correct or incorrect answer could be found. Data Statistics For all experiments, we admitted only experienced MTurk workers who are based in the US. One HIT consisted of writing one text for the text collection, formulating 15 questions for the question collection, or finding 15 pairs of answers for the answer collection. We paid $0.50 per HIT for the text and question collection, and $0.60 per HIT for the answer collection. More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. For 13% of the questions, the workers did not agree on one of the 4 categories with a 3 out of 5 majority, so we did not include these questions in our dataset. The distribution of category labels on the remaining 87% is shown in Table TABREF10 . 14,074 (52%) questions could be answered. Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based). After removing 135 questions during the validation, the final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%). This ratio was manually verified based on a random sample of questions. We split the dataset into training (9,731 questions on 1,470 texts), development (1,411 questions on 219 texts), and test set (2,797 questions on 430 texts). Each text appears only in one of the three sets. The complete set of texts for 5 scenarios was held out for the test set. The average text, question, and answer length is 196.0 words, 7.8 words, and 3.6 words, respectively. On average, there are 6.7 questions per text. Figure FIGREF21 shows the distribution of question types in the dataset, which we identified using simple heuristics based on the first words of a question: Yes/no questions were identified as questions starting with an auxiliary or modal verb, all other question types were determined based on the question word. We found that 29% of all questions are yes/no questions. Questions about details of a situation (such as what/ which and who) form the second most frequent question category. Temporal questions (when and how long/often) form approx. 11% of all questions. We leave a more detailed analysis of question types for future work. Data Analysis As can be seen from the data statistics, our mode of collection leads to a substantial proportion of questions that require inference using commonsense knowledge. Still, the dataset contains a large number of questions in which the answer is explicitly contained or implied by the text: Figure FIGREF22 shows passages from an example text of the dataset together with two such questions. For question Q1, the answer is given literally in the text. Answering question Q2 is not as simple; it can be solved, however, via standard semantic relatedness information (chicken and hotdogs are meat; water, soda and juice are drinks). The following cases require commonsense inference to be decided. In all these cases, the answers are not overtly contained nor easily derivable from the respective texts. We do not show the full texts, but only the scenario names for each question. Example UID23 refers to a library setting. Script knowledge helps in assessing that usually, paying is not an event when borrowing a book, which answers the question. Similarly, event information helps in answering the questions in Examples UID24 and UID25 . In Example UID26 , knowledge about the typical role of parents in the preparation of a picnic will enable a plausibility decision. Similarly, in Example UID27 , it is commonsense knowledge that showers usually take a few minutes rather than hours. There are also cases in which the answer can be inferred from the text, but where commonsense knowledge is still beneficial: The text for example UID28 does not contain the information that breakfast is eaten in the morning, but it could still be inferred from many pointers in the text (e.g. phrases such as I woke up), or from commonsense knowledge. These few examples illustrate that our dataset covers questions with a wide spectrum of difficulty, from rather simple questions that can be answered from the text to challenging inference problems. Experiments In this section, we assess the performance of baseline models on MCScript, using accuracy as the evaluation measure. We employ models of differing complexity: two unsupervised models using only word information and distributional information, respectively, and two supervised neural models. We assess performance on two dimensions: One, we show how well the models perform on text-based questions as compared to questions that require common sense for finding the correct answer. Two, we evaluate each model for each different question type. Models We first use a simple word matching baseline, by selecting the answer that has the highest literal overlap with the text. In case of a tie, we randomly select one of the answers. The second baseline is a sliding window approach that looks at windows of INLINEFORM0 tokens on the text. Each text and each answer are represented as a sequence of word embeddings. The embeddings for each window of size INLINEFORM1 and each answer are then averaged to derive window and answer representations, respectively. The answer with the lowest cosine distance to one of the windows of the text is then selected as correct. We employ a simple neural model as a third baseline. In this model, each text, question, and answer is represented by a vector. For a given sequence of words INLINEFORM0 , we compute this representation by averaging over the components of the word embeddings INLINEFORM1 that correspond to a word INLINEFORM2 , and then apply a linear transformation using a weight matrix. This procedure is applied to each answer INLINEFORM3 to derive an answer representation INLINEFORM4 . The representation of a text INLINEFORM5 and of a question INLINEFORM6 are computed in the same way. We use different weight matrices for INLINEFORM7 , INLINEFORM8 and INLINEFORM9 , respectively. A combined representation INLINEFORM10 for the text–question pair is then constructed using a bilinear transformation matrix INLINEFORM11 : DISPLAYFORM0 We compute a score for each answer by using the dot product and pass the scores for both answers through a softmax layer for prediction. The probability INLINEFORM0 for an answer INLINEFORM1 to be correct is thus defined as: DISPLAYFORM0 The attentive reader is a well-established machine comprehension model that reaches good performance e.g. on the CNN/Daily Mail corpus BIBREF14 , BIBREF15 . We use the model formulation by chen2016thorough and lai2017race, who employ bilinear weight functions to compute both attention and answer-text fit. Bi-directional GRUs are used to encode questions, texts and answers into hidden representations. For a question INLINEFORM0 and an answer INLINEFORM1 , the last state of the GRUs, INLINEFORM2 and INLINEFORM3 , are used as representations, while the text is encoded as a sequence of hidden states INLINEFORM4 . We then compute an attention score INLINEFORM5 for each hidden state INLINEFORM6 using the question representation INLINEFORM7 , a weight matrix INLINEFORM8 , and an attention bias INLINEFORM9 . Last, a text representation INLINEFORM10 is computed as a weighted average of the hidden representations: DISPLAYFORM0 The probability INLINEFORM0 of answer INLINEFORM1 being correct is then predicted using another bilinear weight matrix INLINEFORM2 , followed by an application of the softmax function over both answer options for the question: DISPLAYFORM0 Implementation Details Texts, questions and answers were tokenized using NLTK and lowercased. We used 100-dimensional GloVe vectors BIBREF16 to embed each token. For the neural models, the embeddings are used to initialize the token representations, and are refined during training. For the sliding similarity window approach, we set INLINEFORM0 . The vocabulary of the neural models was extracted from training and development data. For optimizing the bilinear model and the attentive reader, we used vanilla stochastic gradient descent with gradient clipping, if the norm of gradients exceeds 10. The size of the hidden layers was tuned to 64, with a learning rate of INLINEFORM0 , for both models. We apply a dropout of INLINEFORM1 to the word embeddings. Batch size was set to 25 and all models were trained for 150 epochs. During training, we measured performance on the development set, and we selected the model from the best performing epoch for testing. Results and Evaluation As an upper bound for model performance, we assess how well humans can solve our task. Two trained annotators labeled the correct answer on all instances of the test set. They agreed with the gold standard in 98.2 % of cases. This result shows that humans have no difficulty in finding the correct answer, irrespective of the question type. Table TABREF37 shows the performance of the baseline models as compared to the human upper bound and a random baseline. As can be seen, neural models have a clear advantage over the pure word overlap baseline, which performs worst, with an accuracy of INLINEFORM0 . The low accuracy is mostly due to the nature of correct answers in our data: Each correct answer has a low overlap with the text by design. Since the overlap model selects the answer with a high overlap to the text, it does not perform well. In particular, this also explains the very bad result on text-based questions. The sliding similarity window model does not outperform the simple word overlap model by a large margin: Distributional information alone is insufficient to handle complex questions in the dataset. Both neural models outperform the unsupervised baselines by a large margin. When comparing the two models, the attentive reader is able to beat the bilinear model by only INLINEFORM0 . A possible explanation for this is that the attentive reader only attends to the text. Since many questions cannot be directly answered from the text, the attentive reader is not able to perform significantly better than a simpler neural model. What is surprising is that the attentive reader works better on commonsense-based questions than on text questions. This can be explained by the fact that many commonsense questions do have prototypical answers within a scenario, irrespective of the text. The attentive reader is apparently able to just memorize these prototypical answers, thus achieving higher accuracy. Inspecting attention values of the attentive reader, we found that in most cases, the model is unable to properly attend to the relevant parts of the text, even when the answer is literally given in the text. A possible explanation is that the model is confused by the large amount of questions that cannot be answered from the text directly, which might confound the computation of attention values. Also, the attentive reader was originally constructed for reconstructing literal text spans as answers. Our mode of answer collection, however, results in many correct answers that cannot be found verbatim in the text. This presents difficulties for the attention mechanism. The fact that an attention model outperforms a simple bilinear baseline only marginally shows that MCScript poses a new challenge to machine comprehension systems. Models concentrating solely on the text are insufficient to perform well on the data. Figure FIGREF39 gives accuracy values of all baseline systems on the most frequent question types (appearing >25 times in the test data), as determined based on the question words (see Section SECREF19 ). The numbers depicted on the left-hand side of the y-axis represent model accuracy. The right-hand side of the y-axis indicates the number of times a question type appears in the test data. The neural models unsurprisingly outperform the other models in most cases, and the difference for who questions is largest. A large number of these questions ask for the narrator of the story, who is usually not mentioned literally in the text, since most stories are written in the first person. It is also apparent that all models perform rather badly on yes/no questions. Each model basically compares the answer to some representation of the text. For yes/no questions, this makes sense for less than half of all cases. For the majority of yes/no questions, however, answers consist only of yes or no, without further content words. Related Work In recent years, a number of reading comprehension datasets have been proposed, including MCTest BIBREF17 , BAbI BIBREF18 , the Children's Book Test (CBT, hill2015goldilocks), CNN/Daily Mail BIBREF14 , the Stanford Question Answering Dataset (SQuAD, rajpurkar2016squad), and RACE BIBREF19 . These datasets differ with respect to text type (Wikipedia texts, examination texts, etc.), mode of answer selection (span-based, multiple choice, etc.) and test systems regarding different aspects of language understanding, but they do not explicitly address commonsense knowledge. Two notable exceptions are the NewsQA and TriviaQA datasets. NewsQA BIBREF20 is a dataset of newswire texts from CNN with questions and answers written by crowdsourcing workers. NewsQA closely resembles our own data collection with respect to the method of data acquisition. As for our data collection, full texts were not shown to workers as a basis for question formulation, but only the text's title and a short summary, to avoid literal repetitions and support the generation of non-trivial questions requiring background knowledge. The NewsQA text collection differs from ours in domain and genre (newswire texts vs. narrative stories about everyday events). Knowledge required to answer the questions is mostly factual knowledge and script knowledge is only marginally relevant. Also, the task is not exactly question answering, but identification of document passages containing the answer. TriviaQA BIBREF21 is a corpus that contains automatically collected question-answer pairs from 14 trivia and quiz-league websites, together with web-crawled evidence documents from Wikipedia and Bing. While a majority of questions require world knowledge for finding the correct answer, it is mostly factual knowledge. Summary We present a new dataset for the task of machine comprehension focussing on commonsense knowledge. Questions were collected based on script scenarios, rather than individual texts, which resulted in question–answer pairs that explicitly involve commonsense knowledge. In contrast to previous evaluation tasks, this setup allows us for the first time to assess the contribution of script knowledge for computational models of language understanding in a real-world evaluation scenario. We expect our dataset to become a standard benchmark for testing models of commonsense and script knowledge. Human performance shows that the dataset is highly reliable. The results of several baselines, in contrast, illustrate that our task provides challenging test cases for the broader natural language processing community. MCScript forms the basis of a shared task organized at SemEval 2018. The dataset is available at http://www.sfb1102.uni-saarland.de/?page_id=2582. Acknowledgements We thank the reviewers for their helpful comments. We also thank Florian Pusse for the help with the MTurk experiments and our student assistants Christine SchÃ¤fer, Damyana Gateva, Leonie Harter, Sarah Mameche, Stefan GrÃ¼newald and Tatiana Anikina for help with the annotations. This research was funded by the German Research Foundation (DFG) as part of SFB 1102 `Information Density and Linguistic Encoding' and EXC 284 `Multimodal Computing and Interaction'.
what dataset statistics are provided?
ad1be65c4f0655ac5c902d17f05454c0d4c4a15d
what Ġdataset Ġstatistics Ġare Ġprovided ?
More Ġthan Ġ2 , 100 Ġtexts Ġwere Ġpaired Ġwith Ġ15 Ġquestions Ġeach , Ġresulting Ġin Ġa Ġtotal Ġnumber Ġof Ġapprox . Ġ32 , 000 Ġannot ated Ġquestions . Ġ13 % Ġof Ġthe Ġquestions Ġare Ġnot Ġanswer able . Ġ ĠOut Ġof Ġthe Ġanswer able Ġquestions , Ġ10 , 160 Ġcould Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġdirectly Ġ( text - based ) Ġand Ġ3 , 9 14 Ġquestions Ġrequired Ġthe Ġuse Ġof Ġcommons ense Ġknowledge Ġ( script - based ). Ġ ĠThe Ġfinal Ġdataset Ġcomprises Ġ13 , 9 39 Ġquestions , Ġ3 , 8 27 Ġof Ġwhich Ġrequire Ġcommons ense Ġknowledge Ġ( i . e . Ġ27 . 4 %).
<s> what Ġdataset Ġstatistics Ġare Ġprovided ? </s> Introduction </s> Amb ig uity Ġand Ġimplicit ness Ġare Ġinherent Ġproperties Ġof Ġnatural Ġlanguage Ġthat Ġcause Ġchallenges Ġfor Ġcomputational Ġmodels Ġof Ġlanguage Ġunderstanding . ĠIn Ġeveryday Ġcommunication , Ġpeople Ġassume Ġa Ġshared Ġcommon Ġground Ġwhich Ġforms Ġa Ġbasis Ġfor Ġefficiently Ġresolving Ġambig u ities Ġand Ġfor Ġinfer ring Ġimplicit Ġinformation . ĠThus , Ġrecover able Ġinformation Ġis Ġoften Ġleft Ġun mentioned Ġor Ġunders pec ified . ĠSuch Ġinformation Ġmay Ġinclude Ġen cycl oped ic Ġand Ġcommons ense Ġknowledge . ĠThis Ġwork Ġfocuses Ġon Ġcommons ense Ġknowledge Ġabout Ġeveryday Ġactivities , Ġso - called Ġscripts . </s> This Ġpaper Ġintroduces Ġa Ġdataset Ġto Ġevaluate Ġnatural Ġlanguage Ġunderstanding Ġapproaches Ġwith Ġa Ġfocus Ġon Ġinterpretation Ġprocesses Ġrequiring Ġinference Ġbased Ġon Ġcommons ense Ġknowledge . ĠIn Ġparticular , Ġwe Ġpresent ĠMC Script , Ġa Ġdataset Ġfor Ġassessing Ġthe Ġcontribution Ġof Ġscript Ġknowledge Ġto Ġmachine Ġcomprehension . ĠScript s Ġare Ġsequences Ġof Ġevents Ġdescribing Ġstereotypical Ġhuman Ġactivities Ġ( also Ġcalled Ġscenarios ), Ġfor Ġexample Ġbaking Ġa Ġcake Ġor Ġtaking Ġa Ġbus ĠB IB REF 0 Ġ. ĠTo Ġillustrate Ġthe Ġimportance Ġof Ġscript Ġknowledge , Ġconsider ĠExample Ġ( ĠSEC REF 1 Ġ): </s> Without Ġusing Ġcommons ense Ġknowledge , Ġit Ġmay Ġbe Ġdifficult Ġto Ġtell Ġwho Ġate Ġthe Ġfood : ĠRachel Ġor Ġthe Ġwaitress . ĠIn Ġcontrast , Ġif Ġwe Ġutilize Ġcommons ense Ġknowledge , Ġin Ġparticular , Ġscript Ġknowledge Ġabout Ġthe Ġeating Ġin Ġa Ġrestaurant Ġscenario , Ġwe Ġcan Ġmake Ġthe Ġfollowing Ġinf erences : ĠRachel Ġis Ġmost Ġlikely Ġa Ġcustomer , Ġsince Ġshe Ġreceived Ġan Ġorder . ĠIt Ġis Ġusually Ġthe Ġcustomer , Ġand Ġnot Ġthe Ġwaitress , Ġwho Ġeats Ġthe Ġordered Ġfood . ĠSo ĠShe Ġmost Ġlikely Ġrefers Ġto ĠRachel . </s> Various Ġapproaches Ġfor Ġscript Ġknowledge Ġextraction Ġand Ġprocessing Ġhave Ġbeen Ġproposed Ġin Ġrecent Ġyears . ĠHowever , Ġsystems Ġhave Ġbeen Ġevaluated Ġfor Ġspecific Ġaspects Ġof Ġscript Ġknowledge Ġonly , Ġsuch Ġas Ġevent Ġordering ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġevent Ġparaph r asing ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġor Ġevent Ġprediction Ġ( name ly , Ġthe Ġnarrative Ġclo ze Ġtask ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ). ĠThese Ġevaluation Ġmethods Ġlack Ġa Ġclear Ġconnection Ġto Ġreal - world Ġtasks . ĠOur ĠMC Script Ġdataset Ġprovides Ġan Ġextr ins ic Ġevaluation Ġframework , Ġbased Ġon Ġtext Ġcomprehension Ġinvolving Ġcommons ense Ġknowledge . ĠThis Ġframework Ġmakes Ġit Ġpossible Ġto Ġassess Ġsystem Ġperformance Ġin Ġa Ġmultiple - choice Ġquestion Ġanswering Ġsetting , Ġwithout Ġimposing Ġany Ġspecific Ġstructural Ġor Ġmethod ical Ġrequirements . </s> MC Script Ġis Ġa Ġcollection Ġof Ġ( 1 ) Ġnarrative Ġtexts , Ġ( 2 ) Ġquestions Ġof Ġvarious Ġtypes Ġreferring Ġto Ġthese Ġtexts , Ġand Ġ( 3 ) Ġpairs Ġof Ġanswer Ġcandidates Ġfor Ġeach Ġquestion . ĠIt Ġcomprises Ġapprox . Ġ2 , 100 Ġtexts Ġand Ġa Ġtotal Ġof Ġapprox . Ġ14 , 000 Ġquestions . ĠAn sw ering Ġa Ġsubstantial Ġsubset Ġof Ġquestions Ġrequires Ġknowledge Ġbeyond Ġthe Ġfacts Ġmentioned Ġin Ġthe Ġtext , Ġi . e . Ġit Ġrequires Ġinference Ġusing Ġcommons ense Ġknowledge Ġabout Ġeveryday Ġactivities . ĠAn Ġexample Ġis Ġgiven Ġin ĠFigure ĠFIG REF 2 Ġ. ĠFor Ġboth Ġquestions , Ġthe Ġcorrect Ġchoice Ġfor Ġan Ġanswer Ġrequires Ġcommons ense Ġknowledge Ġabout Ġthe Ġactivity Ġof Ġplanting Ġa Ġtree , Ġwhich Ġgoes Ġbeyond Ġwhat Ġis Ġmentioned Ġin Ġthe Ġtext . ĠText s , Ġquestions , Ġand Ġanswers Ġwere Ġobtained Ġthrough Ġcrowds ourcing . ĠIn Ġorder Ġto Ġensure Ġhigh Ġquality , Ġwe Ġmanually Ġvalidated Ġand Ġfiltered Ġthe Ġdataset . ĠDue Ġto Ġour Ġdesign Ġof Ġthe Ġdata Ġacquisition Ġprocess , Ġwe Ġended Ġup Ġwith Ġa Ġsubstantial Ġsubset Ġof Ġquestions Ġthat Ġrequire Ġcommons ense Ġinference Ġ( 27 . 4 %). </s> Corp us </s> Machine Ġcomprehension Ġdatasets Ġconsist Ġof Ġthree Ġmain Ġcomponents : Ġtexts , Ġquestions Ġand Ġanswers . ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġdata Ġcollection Ġfor Ġthese Ġ3 Ġcomponents . ĠWe Ġfirst Ġdescribe Ġa Ġseries Ġof Ġpilot Ġstudies Ġthat Ġwe Ġconducted Ġin Ġorder Ġto Ġcollect Ġcommons ense Ġinference Ġquestions Ġ( Section ĠSEC REF 4 Ġ). ĠIn ĠSection ĠSEC REF 5 Ġ, Ġwe Ġdiscuss Ġthe Ġresulting Ġdata Ġcollection Ġof Ġquestions , Ġtexts Ġand Ġanswers Ġvia Ġcrowds ourcing Ġon ĠAmazon ĠMechanical ĠTurk Ġ( hen ce forth ĠM Tur k ). ĠSection ĠSEC REF 17 Ġgives Ġinformation Ġabout Ġsome Ġnecessary Ġpost processing Ġsteps Ġand Ġthe Ġdataset Ġvalidation . ĠLastly , ĠSection ĠSEC REF 19 Ġgives Ġstatistics Ġabout Ġthe Ġfinal Ġdataset . </s> P ilot ĠStudy </s> As Ġa Ġstarting Ġpoint Ġfor Ġour Ġpilots , Ġwe Ġmade Ġuse Ġof Ġtexts Ġfrom Ġthe ĠIn Script Ġcorpus ĠB IB REF 10 Ġ, Ġwhich Ġprovides Ġstories Ġcentered Ġaround Ġeveryday Ġsituations Ġ( see ĠSection ĠSEC REF 7 Ġ). ĠWe Ġconducted Ġthree Ġdifferent Ġpilot Ġstudies Ġto Ġdetermine Ġthe Ġbest Ġway Ġof Ġcollecting Ġquestions Ġthat Ġrequire Ġinference Ġover Ġcommons ense Ġknowledge : </s> The Ġmost Ġintuitive Ġway Ġof Ġcollecting Ġreading Ġcomprehension Ġquestions Ġis Ġto Ġshow Ġtexts Ġto Ġworkers Ġand Ġlet Ġthem Ġformulate Ġquestions Ġand Ġanswers Ġon Ġthe Ġtexts , Ġwhich Ġis Ġwhat Ġwe Ġtried Ġinternally Ġin Ġa Ġfirst Ġpilot . ĠSince Ġour Ġfocus Ġis Ġto Ġprovide Ġan Ġevaluation Ġframework Ġfor Ġinference Ġover Ġcommons ense Ġknowledge , Ġwe Ġmanually Ġassessed Ġthe Ġnumber Ġof Ġquestions Ġthat Ġindeed Ġrequire Ġcommon Ġsense Ġknowledge . ĠWe Ġfound Ġtoo Ġmany Ġquestions Ġand Ġanswers Ġcollected Ġin Ġthis Ġmanner Ġto Ġbe Ġlex ically Ġclose Ġto Ġthe Ġtext . </s> In Ġa Ġsecond Ġpilot , Ġwe Ġinvestigated Ġthe Ġoption Ġto Ġtake Ġthe Ġquestions Ġcollected Ġfor Ġone Ġtext Ġand Ġshow Ġthem Ġas Ġquestions Ġfor Ġanother Ġtext Ġof Ġthe Ġsame Ġscenario . ĠWhile Ġthis Ġmethod Ġresulted Ġin Ġa Ġlarger Ġnumber Ġof Ġquestions Ġthat Ġrequired Ġinference , Ġwe Ġfound Ġthe Ġmajority Ġof Ġquestions Ġto Ġnot Ġmake Ġsense Ġat Ġall Ġwhen Ġpaired Ġwith Ġanother Ġtext . ĠMany Ġquestions Ġwere Ġspecific Ġto Ġa Ġtext Ġ( and Ġnot Ġto Ġa Ġscenario ), Ġrequiring Ġdetails Ġthat Ġcould Ġnot Ġbe Ġanswered Ġfrom Ġother Ġtexts . ĠSince Ġthe Ġtwo Ġprevious Ġpilot Ġsetups Ġresulted Ġin Ġquestions Ġthat Ġcentered Ġaround Ġthe Ġtexts Ġthemselves , Ġwe Ġdecided Ġfor Ġa Ġthird Ġpilot Ġto Ġnot Ġshow Ġworkers Ġany Ġspecific Ġtexts Ġat Ġall . ĠInstead , Ġwe Ġasked Ġfor Ġquestions Ġthat Ġcentered Ġaround Ġa Ġspecific Ġscript Ġscenario Ġ( e . g . Ġeating Ġin Ġa Ġrestaurant ). ĠWe Ġfound Ġthis Ġmode Ġof Ġcollection Ġto Ġresult Ġin Ġquestions Ġthat Ġhave Ġthe Ġright Ġlevel Ġof Ġspecificity Ġfor Ġour Ġpurposes : Ġnamely , Ġquestions Ġthat Ġare Ġrelated Ġto Ġa Ġscenario Ġand Ġthat Ġcan Ġbe Ġanswered Ġfrom Ġdifferent Ġtexts Ġ( about Ġthat Ġscenario ), Ġbut Ġfor Ġwhich Ġa Ġtext Ġdoes Ġnot Ġneed Ġto Ġprovide Ġthe Ġanswer Ġexplicitly . </s> The Ġnext Ġsection Ġwill Ġdescribe Ġthe Ġmode Ġof Ġcollection Ġchosen Ġfor Ġthe Ġfinal Ġdataset , Ġbased Ġon Ġthe Ġthird Ġpilot , Ġin Ġmore Ġdetail . </s> Data ĠCollection </s> As Ġmentioned Ġin Ġthe Ġprevious Ġsection , Ġwe Ġdecided Ġto Ġbase Ġthe Ġquestion Ġcollection Ġon Ġscript Ġscenarios Ġrather Ġthan Ġspecific Ġtexts . ĠAs Ġa Ġstarting Ġpoint Ġfor Ġour Ġdata Ġcollection , Ġwe Ġuse Ġscenarios Ġfrom Ġthree Ġscript Ġdata Ġcollections ĠB IB REF 3 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠTogether , Ġthese Ġresources Ġcontain Ġmore Ġthan Ġ200 Ġscenarios . ĠTo Ġmake Ġsure Ġthat Ġscenarios Ġhave Ġdifferent Ġcomplexity Ġand Ġcontent , Ġwe Ġselected Ġ80 Ġof Ġthem Ġand Ġcame Ġup Ġwith Ġ20 Ġnew Ġscenarios . ĠTogether Ġwith Ġthe Ġ10 Ġscenarios Ġfrom ĠIn Script , Ġwe Ġend Ġup Ġwith Ġa Ġtotal Ġof Ġ110 Ġscenarios . </s> For Ġthe Ġcollection Ġof Ġtexts , Ġwe Ġfollowed Ġmod i ins cript , Ġwhere Ġworkers Ġwere Ġasked Ġto Ġwrite Ġa Ġstory Ġabout Ġa Ġgiven Ġactivity ĠâĢ ľ as Ġif Ġexplaining Ġit Ġto Ġa Ġchild âĢ Ŀ . ĠThis Ġresults Ġin Ġelaborate Ġand Ġexplicit Ġtexts Ġthat Ġare Ġcentered Ġaround Ġa Ġsingle Ġscenario . ĠConsequently , Ġthe Ġtexts Ġare Ġsynt act ically Ġsimple , Ġfacilitating Ġmachine Ġcomprehension Ġmodels Ġto Ġfocus Ġon Ġsemantic Ġchallenges Ġand Ġinference . ĠWe Ġcollected Ġ20 Ġtexts Ġfor Ġeach Ġscenario . ĠEach Ġparticipant Ġwas Ġallowed Ġto Ġwrite Ġonly Ġone Ġstory Ġper Ġscenario , Ġbut Ġwork Ġon Ġas Ġmany Ġscenarios Ġas Ġthey Ġliked . ĠFor Ġeach Ġof Ġthe Ġ10 Ġscenarios Ġfrom ĠIn Script , Ġwe Ġrandomly Ġselected Ġ20 Ġexisting Ġtexts Ġfrom Ġthat Ġresource . </s> For Ġcollecting Ġquestions , Ġworkers Ġwere Ġinstructed Ġto ĠâĢ ľ im agine Ġthey Ġtold Ġa Ġstory Ġabout Ġa Ġcertain Ġscenario Ġto Ġa Ġchild Ġand Ġwant Ġto Ġtest Ġif Ġthe Ġchild Ġunderstood Ġeverything Ġcorrectly âĢ Ŀ . ĠThis Ġinstruction Ġalso Ġensured Ġthat Ġquestions Ġare Ġlingu istically Ġsimple , Ġelaborate Ġand Ġexplicit . ĠWorkers Ġwere Ġasked Ġto Ġformulate Ġquestions Ġabout Ġdetails Ġof Ġsuch Ġa Ġsituation , Ġi . e . Ġindependent Ġof Ġa Ġconcrete Ġnarrative . ĠThis Ġresulted Ġin Ġquestions , Ġthe Ġanswer Ġto Ġwhich Ġis Ġnot Ġliterally Ġmentioned Ġin Ġthe Ġtext . ĠTo Ġcover Ġa Ġbroad Ġrange Ġof Ġquestion Ġtypes , Ġwe Ġasked Ġparticipants Ġto Ġwrite Ġ3 Ġtemporal Ġquestions Ġ( asking Ġabout Ġtime Ġpoints Ġand Ġevent Ġorder ), Ġ3 Ġcontent Ġquestions Ġ( asking Ġabout Ġpersons Ġor Ġdetails Ġin Ġthe Ġscenario ) Ġand Ġ3 Ġreasoning Ġquestions Ġ( asking Ġhow Ġor Ġwhy Ġsomething Ġhappened ). ĠThey Ġwere Ġalso Ġasked Ġto Ġformulate Ġ6 Ġfree Ġquestions , Ġwhich Ġresulted Ġin Ġa Ġtotal Ġof Ġ15 Ġquestions . ĠAs king Ġeach Ġworker Ġfor Ġa Ġhigh Ġnumber Ġof Ġquestions Ġenforced Ġthat Ġmore Ġcreative Ġquestions Ġwere Ġformulated , Ġwhich Ġgo Ġbeyond Ġobvious Ġquestions Ġfor Ġa Ġscenario . ĠSince Ġparticipants Ġwere Ġnot Ġshown Ġa Ġconcrete Ġstory , Ġwe Ġasked Ġthem Ġto Ġuse Ġthe Ġneutral Ġpronoun ĠâĢ ľ they âĢ Ŀ Ġto Ġaddress Ġthe Ġprotagonist Ġof Ġthe Ġstory . ĠWe Ġpermitted Ġparticipants Ġto Ġwork Ġon Ġas Ġmany Ġscenarios Ġas Ġdesired Ġand Ġwe Ġcollected Ġquestions Ġfrom Ġ10 Ġparticipants Ġper Ġscenario . </s> Our Ġmode Ġof Ġquestion Ġcollection Ġresults Ġin Ġquestions Ġthat Ġare Ġnot Ġassociated Ġwith Ġspecific Ġtexts . ĠFor Ġeach Ġtext , Ġwe Ġcollected Ġanswers Ġfor Ġ15 Ġquestions Ġthat Ġwere Ġrandomly Ġselected Ġfrom Ġthe Ġsame Ġscenario . ĠSince Ġquestions Ġand Ġtexts Ġwere Ġcollected Ġindependently , Ġanswering Ġa Ġrandom Ġquestion Ġis Ġnot Ġalways Ġpossible Ġfor Ġa Ġgiven Ġtext . ĠTherefore , Ġwe Ġcarried Ġout Ġanswer Ġcollection Ġin Ġtwo Ġsteps . ĠIn Ġthe Ġfirst Ġstep , Ġwe Ġasked Ġparticipants Ġto Ġassign Ġa Ġcategory Ġto Ġeach Ġtext âĢĵ question Ġpair . </s> We Ġdistinguish Ġtwo Ġcategories Ġof Ġanswer able Ġquestions : ĠThe Ġcategory Ġtext - based Ġwas Ġassigned Ġto Ġquestions Ġthat Ġcan Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġdirectly . ĠIf Ġthe Ġanswer Ġcould Ġonly Ġbe Ġinferred Ġby Ġusing Ġcommons ense Ġknowledge , Ġthe Ġcategory Ġscript - based Ġwas Ġassigned . ĠMaking Ġthis Ġdistinction Ġis Ġinteresting Ġfor Ġevaluation Ġpurposes , Ġsince Ġit Ġenables Ġus Ġto Ġestimate Ġthe Ġnumber Ġof Ġcommons ense Ġinference Ġquestions . ĠFor Ġquestions Ġthat Ġdid Ġnot Ġmake Ġsense Ġat Ġall Ġgiven Ġa Ġtext , Ġunf itting Ġwas Ġassigned . ĠIf Ġa Ġquestion Ġmade Ġsense Ġfor Ġa Ġtext , Ġbut Ġit Ġwas Ġimpossible Ġto Ġfind Ġan Ġanswer , Ġthe Ġlabel Ġunknown Ġwas Ġused . </s> In Ġa Ġsecond Ġstep , Ġwe Ġtold Ġparticipants Ġto Ġformulate Ġa Ġplausible Ġcorrect Ġand Ġa Ġplausible Ġincorrect Ġanswer Ġcandidate Ġto Ġanswer able Ġquestions Ġ( text - based Ġor Ġscript - based ). ĠTo Ġlevel Ġout Ġthe Ġeffort Ġbetween Ġanswer able Ġand Ġnon - answer able Ġquestions , Ġparticipants Ġhad Ġto Ġwrite Ġa Ġnew Ġquestion Ġwhen Ġselecting Ġunknown Ġor Ġunf itting . ĠIn Ġorder Ġto Ġget Ġreliable Ġjudgments Ġabout Ġwhether Ġor Ġnot Ġa Ġquestion Ġcan Ġbe Ġanswered , Ġwe Ġcollected Ġdata Ġfrom Ġ5 Ġparticipants Ġfor Ġeach Ġquestion Ġand Ġdecided Ġon Ġthe Ġfinal Ġcategory Ġvia Ġmajority Ġvote Ġ( at Ġleast Ġ3 Ġout Ġof Ġ5 ). ĠConsequently , Ġfor Ġeach Ġquestion Ġwith Ġa Ġmajority Ġvote Ġon Ġeither Ġtext - based Ġor Ġscript - based , Ġthere Ġare Ġ3 Ġto Ġ5 Ġcorrect Ġand Ġincorrect Ġanswer Ġcandidates , Ġone Ġfrom Ġeach Ġparticipant Ġwho Ġagreed Ġon Ġthe Ġcategory . ĠQuestions Ġwithout Ġa Ġclear Ġmajority Ġvote Ġor Ġwith Ġties Ġwere Ġnot Ġincluded Ġin Ġthe Ġdataset . </s> We Ġperformed Ġfour Ġpost - processing Ġsteps Ġon Ġthe Ġcollected Ġdata . </s> We Ġmanually Ġfiltered Ġout Ġtexts Ġthat Ġwere Ġinstructional Ġrather Ġthan Ġnarrative . </s> All Ġtexts , Ġquestions Ġand Ġanswers Ġwere Ġspell checked Ġby Ġrunning Ġa Spell Ġand Ġmanually Ġinspecting Ġall Ġcorrections Ġproposed Ġby Ġthe Ġspell check er . </s> We Ġfound Ġthat Ġsome Ġparticipants Ġdid Ġnot Ġuse ĠâĢ ľ they âĢ Ŀ Ġwhen Ġreferring Ġto Ġthe Ġprotagonist . ĠWe Ġidentified ĠâĢ ľ I âĢ Ŀ , ĠâĢ ľ you âĢ Ŀ , ĠâĢ ľ he âĢ Ŀ , ĠâĢ ľ she âĢ Ŀ , ĠâĢ ľ my âĢ Ŀ , ĠâĢ ľ your âĢ Ŀ , ĠâĢ ľ his âĢ Ŀ , ĠâĢ ľ her âĢ Ŀ Ġand ĠâĢ ľ the Ġperson âĢ Ŀ Ġas Ġmost Ġcommon Ġalternatives Ġand Ġreplaced Ġeach Ġappearance Ġmanually Ġwith ĠâĢ ľ they âĢ Ŀ Ġor ĠâĢ ľ their âĢ Ŀ , Ġif Ġappropriate . </s> We Ġmanually Ġfiltered Ġout Ġinvalid Ġquestions , Ġe . g . Ġquestions Ġthat Ġare Ġsuggestive Ġ( âĢ ľ Should Ġyou Ġask Ġan Ġadult Ġbefore Ġusing Ġa Ġknife ? âĢ Ŀ ) Ġor Ġthat Ġask Ġfor Ġthe Ġpersonal Ġopinion Ġof Ġthe Ġreader Ġ( âĢ ľ Do Ġyou Ġthink Ġgoing Ġto Ġthe Ġmuseum Ġwas Ġa Ġgood Ġidea ? âĢ Ŀ ). </s> Answer ĠSelection Ġand ĠVal idation </s> We Ġfinalized Ġthe Ġdataset Ġby Ġselecting Ġone Ġcorrect Ġand Ġone Ġincorrect Ġanswer Ġfor Ġeach Ġquestion âĢĵ text Ġpair . ĠTo Ġincrease Ġthe Ġproportion Ġof Ġnon - t riv ial Ġinference Ġcases , Ġwe Ġchose Ġthe Ġcandidate Ġwith Ġthe Ġlowest Ġlex ical Ġoverlap Ġwith Ġthe Ġtext Ġfrom Ġthe Ġset Ġof Ġcorrect Ġanswer Ġcandidates Ġas Ġcorrect Ġanswer . ĠUsing Ġthis Ġprinciple Ġalso Ġfor Ġincorrect Ġanswers Ġleads Ġto Ġproblems . ĠWe Ġfound Ġthat Ġmany Ġincorrect Ġcandidates Ġwere Ġnot Ġplausible Ġanswers Ġto Ġa Ġgiven Ġquestion . ĠInstead Ġof Ġselecting Ġa Ġcandidate Ġbased Ġon Ġoverlap , Ġwe Ġhence Ġdecided Ġto Ġrely Ġon Ġmajority Ġvote Ġand Ġselected Ġthe Ġcandidate Ġfrom Ġthe Ġset Ġof Ġincorrect Ġanswers Ġthat Ġwas Ġmost Ġoften Ġmentioned . </s> For Ġthis Ġstep , Ġwe Ġnormalized Ġeach Ġcandidate Ġby Ġlower c asing , Ġdeleting Ġpunct uation Ġand Ġstop Ġwords Ġ( articles , Ġand , Ġto Ġand Ġor ), Ġand Ġtransforming Ġall Ġnumber Ġwords Ġinto Ġdigits , Ġusing Ġtext 2 num . ĠWe Ġmerged Ġall Ġanswers Ġthat Ġwere Ġstring - ident ical , Ġcontained Ġanother Ġanswer , Ġor Ġhad Ġa ĠLe ven sh tein Ġdistance ĠB IB REF 13 Ġof Ġ3 Ġor Ġless Ġto Ġanother Ġanswer . ĠThe ĠâĢ ľ most Ġfrequent Ġanswer âĢ Ŀ Ġwas Ġthen Ġselected Ġbased Ġon Ġhow Ġmany Ġother Ġanswers Ġit Ġwas Ġmerged Ġwith . ĠOnly Ġif Ġthere Ġwas Ġno Ġmajority , Ġwe Ġselected Ġthe Ġcandidate Ġwith Ġthe Ġhighest Ġoverlap Ġwith Ġthe Ġtext Ġas Ġa Ġfall back . ĠDue Ġto Ġannotation Ġmistakes , Ġwe Ġfound Ġa Ġsmall Ġnumber Ġof Ġchosen Ġcorrect Ġand Ġincorrect Ġanswers Ġto Ġbe Ġinappropriate , Ġthat Ġis , Ġsome ĠâĢ ľ correct âĢ Ŀ Ġanswers Ġwere Ġactually Ġincorrect Ġand Ġvice Ġversa . ĠTherefore , Ġwe Ġmanually Ġvalidated Ġthe Ġcomplete Ġdataset Ġin Ġa Ġfinal Ġstep . ĠWe Ġasked Ġannot ators Ġto Ġread Ġall Ġtexts , Ġquestions , Ġand Ġanswers , Ġand Ġto Ġmark Ġfor Ġeach Ġquestion Ġwhether Ġthe Ġcorrect Ġand Ġincorrect Ġanswers Ġwere Ġappropriate . ĠIf Ġan Ġanswer Ġwas Ġinappropriate Ġor Ġcontained Ġany Ġerrors , Ġthey Ġselected Ġa Ġdifferent Ġanswer Ġfrom Ġthe Ġset Ġof Ġcollected Ġcandidates . ĠFor Ġapproximately Ġ11 . 5 % Ġof Ġthe Ġquestions , Ġat Ġleast Ġone Ġanswer Ġwas Ġreplaced . Ġ135 Ġquestions Ġ( app rox . Ġ1 %) Ġwere Ġexcluded Ġfrom Ġthe Ġdataset Ġbecause Ġno Ġappropriate Ġcorrect Ġor Ġincorrect Ġanswer Ġcould Ġbe Ġfound . </s> Data ĠStatistics </s> For Ġall Ġexperiments , Ġwe Ġadmitted Ġonly Ġexperienced ĠM Tur k Ġworkers Ġwho Ġare Ġbased Ġin Ġthe ĠUS . ĠOne ĠHIT Ġconsisted Ġof Ġwriting Ġone Ġtext Ġfor Ġthe Ġtext Ġcollection , Ġform ulating Ġ15 Ġquestions Ġfor Ġthe Ġquestion Ġcollection , Ġor Ġfinding Ġ15 Ġpairs Ġof Ġanswers Ġfor Ġthe Ġanswer Ġcollection . ĠWe Ġpaid Ġ$ 0 . 50 Ġper ĠHIT Ġfor Ġthe Ġtext Ġand Ġquestion Ġcollection , Ġand Ġ$ 0 . 60 Ġper ĠHIT Ġfor Ġthe Ġanswer Ġcollection . </s> More Ġthan Ġ2 , 100 Ġtexts Ġwere Ġpaired Ġwith Ġ15 Ġquestions Ġeach , Ġresulting Ġin Ġa Ġtotal Ġnumber Ġof Ġapprox . Ġ32 , 000 Ġannot ated Ġquestions . ĠFor Ġ13 % Ġof Ġthe Ġquestions , Ġthe Ġworkers Ġdid Ġnot Ġagree Ġon Ġone Ġof Ġthe Ġ4 Ġcategories Ġwith Ġa Ġ3 Ġout Ġof Ġ5 Ġmajority , Ġso Ġwe Ġdid Ġnot Ġinclude Ġthese Ġquestions Ġin Ġour Ġdataset . </s> The Ġdistribution Ġof Ġcategory Ġlabels Ġon Ġthe Ġremaining Ġ87 % Ġis Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. Ġ14 , 07 4 Ġ( 52 %) Ġquestions Ġcould Ġbe Ġanswered . ĠOut Ġof Ġthe Ġanswer able Ġquestions , Ġ10 , 160 Ġcould Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġdirectly Ġ( text - based ) Ġand Ġ3 , 9 14 Ġquestions Ġrequired Ġthe Ġuse Ġof Ġcommons ense Ġknowledge Ġ( script - based ). ĠAfter Ġremoving Ġ135 Ġquestions Ġduring Ġthe Ġvalidation , Ġthe Ġfinal Ġdataset Ġcomprises Ġ13 , 9 39 Ġquestions , Ġ3 , 8 27 Ġof Ġwhich Ġrequire Ġcommons ense Ġknowledge Ġ( i . e . Ġ27 . 4 %). ĠThis Ġratio Ġwas Ġmanually Ġverified Ġbased Ġon Ġa Ġrandom Ġsample Ġof Ġquestions . </s> We Ġsplit Ġthe Ġdataset Ġinto Ġtraining Ġ( 9 , 7 31 Ġquestions Ġon Ġ1 , 470 Ġtexts ), Ġdevelopment Ġ( 1 , 411 Ġquestions Ġon Ġ219 Ġtexts ), Ġand Ġtest Ġset Ġ( 2 , 797 Ġquestions Ġon Ġ430 Ġtexts ). ĠEach Ġtext Ġappears Ġonly Ġin Ġone Ġof Ġthe Ġthree Ġsets . ĠThe Ġcomplete Ġset Ġof Ġtexts Ġfor Ġ5 Ġscenarios Ġwas Ġheld Ġout Ġfor Ġthe Ġtest Ġset . ĠThe Ġaverage Ġtext , Ġquestion , Ġand Ġanswer Ġlength Ġis Ġ196 . 0 Ġwords , Ġ7 . 8 Ġwords , Ġand Ġ3 . 6 Ġwords , Ġrespectively . ĠOn Ġaverage , Ġthere Ġare Ġ6 . 7 Ġquestions Ġper Ġtext . </s> Figure ĠFIG REF 21 Ġshows Ġthe Ġdistribution Ġof Ġquestion Ġtypes Ġin Ġthe Ġdataset , Ġwhich Ġwe Ġidentified Ġusing Ġsimple Ġhe ur istics Ġbased Ġon Ġthe Ġfirst Ġwords Ġof Ġa Ġquestion : ĠYes / no Ġquestions Ġwere Ġidentified Ġas Ġquestions Ġstarting Ġwith Ġan Ġauxiliary Ġor Ġmod al Ġverb , Ġall Ġother Ġquestion Ġtypes Ġwere Ġdetermined Ġbased Ġon Ġthe Ġquestion Ġword . </s> We Ġfound Ġthat Ġ29 % Ġof Ġall Ġquestions Ġare Ġyes / no Ġquestions . ĠQuestions Ġabout Ġdetails Ġof Ġa Ġsituation Ġ( such Ġas Ġwhat / Ġwhich Ġand Ġwho ) Ġform Ġthe Ġsecond Ġmost Ġfrequent Ġquestion Ġcategory . ĠTem poral Ġquestions Ġ( when Ġand Ġhow Ġlong / often ) Ġform Ġapprox . Ġ11 % Ġof Ġall Ġquestions . ĠWe Ġleave Ġa Ġmore Ġdetailed Ġanalysis Ġof Ġquestion Ġtypes Ġfor Ġfuture Ġwork . </s> Data ĠAnalysis </s> As Ġcan Ġbe Ġseen Ġfrom Ġthe Ġdata Ġstatistics , Ġour Ġmode Ġof Ġcollection Ġleads Ġto Ġa Ġsubstantial Ġproportion Ġof Ġquestions Ġthat Ġrequire Ġinference Ġusing Ġcommons ense Ġknowledge . ĠStill , Ġthe Ġdataset Ġcontains Ġa Ġlarge Ġnumber Ġof Ġquestions Ġin Ġwhich Ġthe Ġanswer Ġis Ġexplicitly Ġcontained Ġor Ġimplied Ġby Ġthe Ġtext : ĠFigure ĠFIG REF 22 Ġshows Ġpassages Ġfrom Ġan Ġexample Ġtext Ġof Ġthe Ġdataset Ġtogether Ġwith Ġtwo Ġsuch Ġquestions . ĠFor Ġquestion ĠQ 1 , Ġthe Ġanswer Ġis Ġgiven Ġliterally Ġin Ġthe Ġtext . ĠAn sw ering Ġquestion ĠQ 2 Ġis Ġnot Ġas Ġsimple ; Ġit Ġcan Ġbe Ġsolved , Ġhowever , Ġvia Ġstandard Ġsemantic Ġrelated ness Ġinformation Ġ( ch icken Ġand Ġhot dogs Ġare Ġmeat ; Ġwater , Ġsoda Ġand Ġjuice Ġare Ġdrinks ). </s> The Ġfollowing Ġcases Ġrequire Ġcommons ense Ġinference Ġto Ġbe Ġdecided . ĠIn Ġall Ġthese Ġcases , Ġthe Ġanswers Ġare Ġnot Ġovertly Ġcontained Ġnor Ġeasily Ġderiv able Ġfrom Ġthe Ġrespective Ġtexts . ĠWe Ġdo Ġnot Ġshow Ġthe Ġfull Ġtexts , Ġbut Ġonly Ġthe Ġscenario Ġnames Ġfor Ġeach Ġquestion . </s> Example ĠUID 23 Ġrefers Ġto Ġa Ġlibrary Ġsetting . ĠScript Ġknowledge Ġhelps Ġin Ġassessing Ġthat Ġusually , Ġpaying Ġis Ġnot Ġan Ġevent Ġwhen Ġborrowing Ġa Ġbook , Ġwhich Ġanswers Ġthe Ġquestion . ĠSimilarly , Ġevent Ġinformation Ġhelps Ġin Ġanswering Ġthe Ġquestions Ġin ĠExamples ĠUID 24 Ġand ĠUID 25 Ġ. ĠIn ĠExample ĠUID 26 Ġ, Ġknowledge Ġabout Ġthe Ġtypical Ġrole Ġof Ġparents Ġin Ġthe Ġpreparation Ġof Ġa Ġpicnic Ġwill Ġenable Ġa Ġplaus ibility Ġdecision . ĠSimilarly , Ġin ĠExample ĠUID 27 Ġ, Ġit Ġis Ġcommons ense Ġknowledge Ġthat Ġshowers Ġusually Ġtake Ġa Ġfew Ġminutes Ġrather Ġthan Ġhours . </s> There Ġare Ġalso Ġcases Ġin Ġwhich Ġthe Ġanswer Ġcan Ġbe Ġinferred Ġfrom Ġthe Ġtext , Ġbut Ġwhere Ġcommons ense Ġknowledge Ġis Ġstill Ġbeneficial : ĠThe Ġtext Ġfor Ġexample ĠUID 28 Ġdoes Ġnot Ġcontain Ġthe Ġinformation Ġthat Ġbreakfast Ġis Ġeaten Ġin Ġthe Ġmorning , Ġbut Ġit Ġcould Ġstill Ġbe Ġinferred Ġfrom Ġmany Ġpointers Ġin Ġthe Ġtext Ġ( e . g . Ġphrases Ġsuch Ġas ĠI Ġwoke Ġup ), Ġor Ġfrom Ġcommons ense Ġknowledge . </s> These Ġfew Ġexamples Ġillustrate Ġthat Ġour Ġdataset Ġcovers Ġquestions Ġwith Ġa Ġwide Ġspectrum Ġof Ġdifficulty , Ġfrom Ġrather Ġsimple Ġquestions Ġthat Ġcan Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġto Ġchallenging Ġinference Ġproblems . </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġassess Ġthe Ġperformance Ġof Ġbaseline Ġmodels Ġon ĠMC Script , Ġusing Ġaccuracy Ġas Ġthe Ġevaluation Ġmeasure . ĠWe Ġemploy Ġmodels Ġof Ġdiffering Ġcomplexity : Ġtwo Ġun super vised Ġmodels Ġusing Ġonly Ġword Ġinformation Ġand Ġdistribution al Ġinformation , Ġrespectively , Ġand Ġtwo Ġsupervised Ġneural Ġmodels . ĠWe Ġassess Ġperformance Ġon Ġtwo Ġdimensions : ĠOne , Ġwe Ġshow Ġhow Ġwell Ġthe Ġmodels Ġperform Ġon Ġtext - based Ġquestions Ġas Ġcompared Ġto Ġquestions Ġthat Ġrequire Ġcommon Ġsense Ġfor Ġfinding Ġthe Ġcorrect Ġanswer . ĠTwo , Ġwe Ġevaluate Ġeach Ġmodel Ġfor Ġeach Ġdifferent Ġquestion Ġtype . </s> Mod els </s> We Ġfirst Ġuse Ġa Ġsimple Ġword Ġmatching Ġbaseline , Ġby Ġselecting Ġthe Ġanswer Ġthat Ġhas Ġthe Ġhighest Ġliteral Ġoverlap Ġwith Ġthe Ġtext . ĠIn Ġcase Ġof Ġa Ġtie , Ġwe Ġrandomly Ġselect Ġone Ġof Ġthe Ġanswers . </s> The Ġsecond Ġbaseline Ġis Ġa Ġsliding Ġwindow Ġapproach Ġthat Ġlooks Ġat Ġwindows Ġof ĠIN LINE FORM 0 Ġtokens Ġon Ġthe Ġtext . ĠEach Ġtext Ġand Ġeach Ġanswer Ġare Ġrepresented Ġas Ġa Ġsequence Ġof Ġword Ġembed d ings . ĠThe Ġembed d ings Ġfor Ġeach Ġwindow Ġof Ġsize ĠIN LINE FORM 1 Ġand Ġeach Ġanswer Ġare Ġthen Ġaveraged Ġto Ġderive Ġwindow Ġand Ġanswer Ġrepresentations , Ġrespectively . ĠThe Ġanswer Ġwith Ġthe Ġlowest Ġcos ine Ġdistance Ġto Ġone Ġof Ġthe Ġwindows Ġof Ġthe Ġtext Ġis Ġthen Ġselected Ġas Ġcorrect . </s> We Ġemploy Ġa Ġsimple Ġneural Ġmodel Ġas Ġa Ġthird Ġbaseline . ĠIn Ġthis Ġmodel , Ġeach Ġtext , Ġquestion , Ġand Ġanswer Ġis Ġrepresented Ġby Ġa Ġvector . ĠFor Ġa Ġgiven Ġsequence Ġof Ġwords ĠIN LINE FORM 0 Ġ, Ġwe Ġcompute Ġthis Ġrepresentation Ġby Ġaveraging Ġover Ġthe Ġcomponents Ġof Ġthe Ġword Ġembed d ings ĠIN LINE FORM 1 Ġthat Ġcorrespond Ġto Ġa Ġword ĠIN LINE FORM 2 Ġ, Ġand Ġthen Ġapply Ġa Ġlinear Ġtransformation Ġusing Ġa Ġweight Ġmatrix . ĠThis Ġprocedure Ġis Ġapplied Ġto Ġeach Ġanswer ĠIN LINE FORM 3 Ġto Ġderive Ġan Ġanswer Ġrepresentation ĠIN LINE FORM 4 Ġ. ĠThe Ġrepresentation Ġof Ġa Ġtext ĠIN LINE FORM 5 Ġand Ġof Ġa Ġquestion ĠIN LINE FORM 6 Ġare Ġcomputed Ġin Ġthe Ġsame Ġway . ĠWe Ġuse Ġdifferent Ġweight Ġmat rices Ġfor ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġrespectively . ĠA Ġcombined Ġrepresentation ĠIN LINE FORM 10 Ġfor Ġthe Ġtext âĢĵ question Ġpair Ġis Ġthen Ġconstructed Ġusing Ġa Ġbil in ear Ġtransformation Ġmatrix ĠIN LINE FORM 11 Ġ: ĠDIS PLAY FORM 0 </s> We Ġcompute Ġa Ġscore Ġfor Ġeach Ġanswer Ġby Ġusing Ġthe Ġdot Ġproduct Ġand Ġpass Ġthe Ġscores Ġfor Ġboth Ġanswers Ġthrough Ġa Ġsoft max Ġlayer Ġfor Ġprediction . ĠThe Ġprobability ĠIN LINE FORM 0 Ġfor Ġan Ġanswer ĠIN LINE FORM 1 Ġto Ġbe Ġcorrect Ġis Ġthus Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> The Ġattentive Ġreader Ġis Ġa Ġwell - established Ġmachine Ġcomprehension Ġmodel Ġthat Ġreaches Ġgood Ġperformance Ġe . g . Ġon Ġthe ĠCNN / Daily ĠMail Ġcorpus ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠWe Ġuse Ġthe Ġmodel Ġformulation Ġby Ġc hen 2016 th orough Ġand Ġl ai 2017 race , Ġwho Ġemploy Ġbil in ear Ġweight Ġfunctions Ġto Ġcompute Ġboth Ġattention Ġand Ġanswer - text Ġfit . ĠBi - direction al ĠGR Us Ġare Ġused Ġto Ġencode Ġquestions , Ġtexts Ġand Ġanswers Ġinto Ġhidden Ġrepresentations . ĠFor Ġa Ġquestion ĠIN LINE FORM 0 Ġand Ġan Ġanswer ĠIN LINE FORM 1 Ġ, Ġthe Ġlast Ġstate Ġof Ġthe ĠGR Us , ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġare Ġused Ġas Ġrepresentations , Ġwhile Ġthe Ġtext Ġis Ġencoded Ġas Ġa Ġsequence Ġof Ġhidden Ġstates ĠIN LINE FORM 4 Ġ. ĠWe Ġthen Ġcompute Ġan Ġattention Ġscore ĠIN LINE FORM 5 Ġfor Ġeach Ġhidden Ġstate ĠIN LINE FORM 6 Ġusing Ġthe Ġquestion Ġrepresentation ĠIN LINE FORM 7 Ġ, Ġa Ġweight Ġmatrix ĠIN LINE FORM 8 Ġ, Ġand Ġan Ġattention Ġbias ĠIN LINE FORM 9 Ġ. ĠLast , Ġa Ġtext Ġrepresentation ĠIN LINE FORM 10 Ġis Ġcomputed Ġas Ġa Ġweighted Ġaverage Ġof Ġthe Ġhidden Ġrepresentations : ĠDIS PLAY FORM 0 </s> The Ġprobability ĠIN LINE FORM 0 Ġof Ġanswer ĠIN LINE FORM 1 Ġbeing Ġcorrect Ġis Ġthen Ġpredicted Ġusing Ġanother Ġbil in ear Ġweight Ġmatrix ĠIN LINE FORM 2 Ġ, Ġfollowed Ġby Ġan Ġapplication Ġof Ġthe Ġsoft max Ġfunction Ġover Ġboth Ġanswer Ġoptions Ġfor Ġthe Ġquestion : ĠDIS PLAY FORM 0 </s> Im plementation ĠDetails </s> Text s , Ġquestions Ġand Ġanswers Ġwere Ġtoken ized Ġusing ĠNL T K Ġand Ġlower c ased . ĠWe Ġused Ġ100 - dimensional ĠGl o Ve Ġvectors ĠB IB REF 16 Ġto Ġembed Ġeach Ġtoken . ĠFor Ġthe Ġneural Ġmodels , Ġthe Ġembed d ings Ġare Ġused Ġto Ġinitialize Ġthe Ġtoken Ġrepresentations , Ġand Ġare Ġrefined Ġduring Ġtraining . ĠFor Ġthe Ġsliding Ġsimilarity Ġwindow Ġapproach , Ġwe Ġset ĠIN LINE FORM 0 Ġ. </s> The Ġvocabulary Ġof Ġthe Ġneural Ġmodels Ġwas Ġextracted Ġfrom Ġtraining Ġand Ġdevelopment Ġdata . ĠFor Ġoptimizing Ġthe Ġbil in ear Ġmodel Ġand Ġthe Ġattentive Ġreader , Ġwe Ġused Ġvanilla Ġsto ch astic Ġgradient Ġdescent Ġwith Ġgradient Ġclipping , Ġif Ġthe Ġnorm Ġof Ġgrad ients Ġexceeds Ġ10 . ĠThe Ġsize Ġof Ġthe Ġhidden Ġlayers Ġwas Ġtuned Ġto Ġ64 , Ġwith Ġa Ġlearning Ġrate Ġof ĠIN LINE FORM 0 Ġ, Ġfor Ġboth Ġmodels . ĠWe Ġapply Ġa Ġdrop out Ġof ĠIN LINE FORM 1 Ġto Ġthe Ġword Ġembed d ings . ĠB atch Ġsize Ġwas Ġset Ġto Ġ25 Ġand Ġall Ġmodels Ġwere Ġtrained Ġfor Ġ150 Ġepoch s . ĠDuring Ġtraining , Ġwe Ġmeasured Ġperformance Ġon Ġthe Ġdevelopment Ġset , Ġand Ġwe Ġselected Ġthe Ġmodel Ġfrom Ġthe Ġbest Ġperforming Ġepoch Ġfor Ġtesting . </s> Results Ġand ĠEvaluation </s> As Ġan Ġupper Ġbound Ġfor Ġmodel Ġperformance , Ġwe Ġassess Ġhow Ġwell Ġhumans Ġcan Ġsolve Ġour Ġtask . ĠTwo Ġtrained Ġannot ators Ġlabeled Ġthe Ġcorrect Ġanswer Ġon Ġall Ġinstances Ġof Ġthe Ġtest Ġset . ĠThey Ġagreed Ġwith Ġthe Ġgold Ġstandard Ġin Ġ98 . 2 Ġ% Ġof Ġcases . ĠThis Ġresult Ġshows Ġthat Ġhumans Ġhave Ġno Ġdifficulty Ġin Ġfinding Ġthe Ġcorrect Ġanswer , Ġirrespective Ġof Ġthe Ġquestion Ġtype . </s> Table ĠT AB REF 37 Ġshows Ġthe Ġperformance Ġof Ġthe Ġbaseline Ġmodels Ġas Ġcompared Ġto Ġthe Ġhuman Ġupper Ġbound Ġand Ġa Ġrandom Ġbaseline . ĠAs Ġcan Ġbe Ġseen , Ġneural Ġmodels Ġhave Ġa Ġclear Ġadvantage Ġover Ġthe Ġpure Ġword Ġoverlap Ġbaseline , Ġwhich Ġperforms Ġworst , Ġwith Ġan Ġaccuracy Ġof ĠIN LINE FORM 0 Ġ. </s> The Ġlow Ġaccuracy Ġis Ġmostly Ġdue Ġto Ġthe Ġnature Ġof Ġcorrect Ġanswers Ġin Ġour Ġdata : ĠEach Ġcorrect Ġanswer Ġhas Ġa Ġlow Ġoverlap Ġwith Ġthe Ġtext Ġby Ġdesign . ĠSince Ġthe Ġoverlap Ġmodel Ġselects Ġthe Ġanswer Ġwith Ġa Ġhigh Ġoverlap Ġto Ġthe Ġtext , Ġit Ġdoes Ġnot Ġperform Ġwell . ĠIn Ġparticular , Ġthis Ġalso Ġexplains Ġthe Ġvery Ġbad Ġresult Ġon Ġtext - based Ġquestions . ĠThe Ġsliding Ġsimilarity Ġwindow Ġmodel Ġdoes Ġnot Ġoutper form Ġthe Ġsimple Ġword Ġoverlap Ġmodel Ġby Ġa Ġlarge Ġmargin : ĠDistribution al Ġinformation Ġalone Ġis Ġinsufficient Ġto Ġhandle Ġcomplex Ġquestions Ġin Ġthe Ġdataset . </s> Both Ġneural Ġmodels Ġoutper form Ġthe Ġun super vised Ġbas elines Ġby Ġa Ġlarge Ġmargin . ĠWhen Ġcomparing Ġthe Ġtwo Ġmodels , Ġthe Ġattentive Ġreader Ġis Ġable Ġto Ġbeat Ġthe Ġbil in ear Ġmodel Ġby Ġonly ĠIN LINE FORM 0 Ġ. ĠA Ġpossible Ġexplanation Ġfor Ġthis Ġis Ġthat Ġthe Ġattentive Ġreader Ġonly Ġattends Ġto Ġthe Ġtext . ĠSince Ġmany Ġquestions Ġcannot Ġbe Ġdirectly Ġanswered Ġfrom Ġthe Ġtext , Ġthe Ġattentive Ġreader Ġis Ġnot Ġable Ġto Ġperform Ġsignificantly Ġbetter Ġthan Ġa Ġsimpler Ġneural Ġmodel . </s> What Ġis Ġsurprising Ġis Ġthat Ġthe Ġattentive Ġreader Ġworks Ġbetter Ġon Ġcommons ense - based Ġquestions Ġthan Ġon Ġtext Ġquestions . ĠThis Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġmany Ġcommons ense Ġquestions Ġdo Ġhave Ġprototyp ical Ġanswers Ġwithin Ġa Ġscenario , Ġirrespective Ġof Ġthe Ġtext . ĠThe Ġattentive Ġreader Ġis Ġapparently Ġable Ġto Ġjust Ġmemor ize Ġthese Ġprototyp ical Ġanswers , Ġthus Ġachieving Ġhigher Ġaccuracy . </s> In spect ing Ġattention Ġvalues Ġof Ġthe Ġattentive Ġreader , Ġwe Ġfound Ġthat Ġin Ġmost Ġcases , Ġthe Ġmodel Ġis Ġunable Ġto Ġproperly Ġattend Ġto Ġthe Ġrelevant Ġparts Ġof Ġthe Ġtext , Ġeven Ġwhen Ġthe Ġanswer Ġis Ġliterally Ġgiven Ġin Ġthe Ġtext . ĠA Ġpossible Ġexplanation Ġis Ġthat Ġthe Ġmodel Ġis Ġconfused Ġby Ġthe Ġlarge Ġamount Ġof Ġquestions Ġthat Ġcannot Ġbe Ġanswered Ġfrom Ġthe Ġtext Ġdirectly , Ġwhich Ġmight Ġconf ound Ġthe Ġcomputation Ġof Ġattention Ġvalues . </s> Also , Ġthe Ġattentive Ġreader Ġwas Ġoriginally Ġconstructed Ġfor Ġreconstruct ing Ġliteral Ġtext Ġspans Ġas Ġanswers . ĠOur Ġmode Ġof Ġanswer Ġcollection , Ġhowever , Ġresults Ġin Ġmany Ġcorrect Ġanswers Ġthat Ġcannot Ġbe Ġfound Ġver bat im Ġin Ġthe Ġtext . ĠThis Ġpresents Ġdifficulties Ġfor Ġthe Ġattention Ġmechanism . </s> The Ġfact Ġthat Ġan Ġattention Ġmodel Ġoutper forms Ġa Ġsimple Ġbil in ear Ġbaseline Ġonly Ġmarginally Ġshows Ġthat ĠMC Script Ġposes Ġa Ġnew Ġchallenge Ġto Ġmachine Ġcomprehension Ġsystems . ĠModels Ġconcentrating Ġsolely Ġon Ġthe Ġtext Ġare Ġinsufficient Ġto Ġperform Ġwell Ġon Ġthe Ġdata . </s> Figure ĠFIG REF 39 Ġgives Ġaccuracy Ġvalues Ġof Ġall Ġbaseline Ġsystems Ġon Ġthe Ġmost Ġfrequent Ġquestion Ġtypes Ġ( app earing Ġ> 25 Ġtimes Ġin Ġthe Ġtest Ġdata ), Ġas Ġdetermined Ġbased Ġon Ġthe Ġquestion Ġwords Ġ( see ĠSection ĠSEC REF 19 Ġ). ĠThe Ġnumbers Ġdepicted Ġon Ġthe Ġleft - hand Ġside Ġof Ġthe Ġy - axis Ġrepresent Ġmodel Ġaccuracy . ĠThe Ġright - hand Ġside Ġof Ġthe Ġy - axis Ġindicates Ġthe Ġnumber Ġof Ġtimes Ġa Ġquestion Ġtype Ġappears Ġin Ġthe Ġtest Ġdata . </s> The Ġneural Ġmodels Ġunsurprisingly Ġoutper form Ġthe Ġother Ġmodels Ġin Ġmost Ġcases , Ġand Ġthe Ġdifference Ġfor Ġwho Ġquestions Ġis Ġlargest . ĠA Ġlarge Ġnumber Ġof Ġthese Ġquestions Ġask Ġfor Ġthe Ġnarrator Ġof Ġthe Ġstory , Ġwho Ġis Ġusually Ġnot Ġmentioned Ġliterally Ġin Ġthe Ġtext , Ġsince Ġmost Ġstories Ġare Ġwritten Ġin Ġthe Ġfirst Ġperson . </s> It Ġis Ġalso Ġapparent Ġthat Ġall Ġmodels Ġperform Ġrather Ġbadly Ġon Ġyes / no Ġquestions . ĠEach Ġmodel Ġbasically Ġcompares Ġthe Ġanswer Ġto Ġsome Ġrepresentation Ġof Ġthe Ġtext . ĠFor Ġyes / no Ġquestions , Ġthis Ġmakes Ġsense Ġfor Ġless Ġthan Ġhalf Ġof Ġall Ġcases . ĠFor Ġthe Ġmajority Ġof Ġyes / no Ġquestions , Ġhowever , Ġanswers Ġconsist Ġonly Ġof Ġyes Ġor Ġno , Ġwithout Ġfurther Ġcontent Ġwords . </s> Related ĠWork </s> In Ġrecent Ġyears , Ġa Ġnumber Ġof Ġreading Ġcomprehension Ġdatasets Ġhave Ġbeen Ġproposed , Ġincluding ĠM CT est ĠB IB REF 17 Ġ, ĠB Ab I ĠB IB REF 18 Ġ, Ġthe ĠChildren 's ĠBook ĠTest Ġ( C BT , Ġhill 2015 gold il ocks ), ĠCNN / Daily ĠMail ĠB IB REF 14 Ġ, Ġthe ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD , Ġr aj pur kar 2016 squ ad ), Ġand ĠR ACE ĠB IB REF 19 Ġ. ĠThese Ġdatasets Ġdiffer Ġwith Ġrespect Ġto Ġtext Ġtype Ġ( Wikipedia Ġtexts , Ġexamination Ġtexts , Ġetc .), Ġmode Ġof Ġanswer Ġselection Ġ( span - based , Ġmultiple Ġchoice , Ġetc .) Ġand Ġtest Ġsystems Ġregarding Ġdifferent Ġaspects Ġof Ġlanguage Ġunderstanding , Ġbut Ġthey Ġdo Ġnot Ġexplicitly Ġaddress Ġcommons ense Ġknowledge . </s> Two Ġnotable Ġexceptions Ġare Ġthe ĠNews Q A Ġand ĠTri via Q A Ġdatasets . ĠNews Q A ĠB IB REF 20 Ġis Ġa Ġdataset Ġof Ġnews wire Ġtexts Ġfrom ĠCNN Ġwith Ġquestions Ġand Ġanswers Ġwritten Ġby Ġcrowds ourcing Ġworkers . ĠNews Q A Ġclosely Ġresembles Ġour Ġown Ġdata Ġcollection Ġwith Ġrespect Ġto Ġthe Ġmethod Ġof Ġdata Ġacquisition . ĠAs Ġfor Ġour Ġdata Ġcollection , Ġfull Ġtexts Ġwere Ġnot Ġshown Ġto Ġworkers Ġas Ġa Ġbasis Ġfor Ġquestion Ġformulation , Ġbut Ġonly Ġthe Ġtext 's Ġtitle Ġand Ġa Ġshort Ġsummary , Ġto Ġavoid Ġliteral Ġrepet itions Ġand Ġsupport Ġthe Ġgeneration Ġof Ġnon - t riv ial Ġquestions Ġrequiring Ġbackground Ġknowledge . ĠThe ĠNews Q A Ġtext Ġcollection Ġdiffers Ġfrom Ġours Ġin Ġdomain Ġand Ġgenre Ġ( new sw ire Ġtexts Ġvs . Ġnarrative Ġstories Ġabout Ġeveryday Ġevents ). ĠKnowledge Ġrequired Ġto Ġanswer Ġthe Ġquestions Ġis Ġmostly Ġfactual Ġknowledge Ġand Ġscript Ġknowledge Ġis Ġonly Ġmarginally Ġrelevant . ĠAlso , Ġthe Ġtask Ġis Ġnot Ġexactly Ġquestion Ġanswering , Ġbut Ġidentification Ġof Ġdocument Ġpassages Ġcontaining Ġthe Ġanswer . </s> Trivia Q A ĠB IB REF 21 Ġis Ġa Ġcorpus Ġthat Ġcontains Ġautomatically Ġcollected Ġquestion - answer Ġpairs Ġfrom Ġ14 Ġtrivia Ġand Ġquiz - league Ġwebsites , Ġtogether Ġwith Ġweb - c rawled Ġevidence Ġdocuments Ġfrom ĠWikipedia Ġand ĠBing . ĠWhile Ġa Ġmajority Ġof Ġquestions Ġrequire Ġworld Ġknowledge Ġfor Ġfinding Ġthe Ġcorrect Ġanswer , Ġit Ġis Ġmostly Ġfactual Ġknowledge . </s> Summary </s> We Ġpresent Ġa Ġnew Ġdataset Ġfor Ġthe Ġtask Ġof Ġmachine Ġcomprehension Ġfoc uss ing Ġon Ġcommons ense Ġknowledge . ĠQuestions Ġwere Ġcollected Ġbased Ġon Ġscript Ġscenarios , Ġrather Ġthan Ġindividual Ġtexts , Ġwhich Ġresulted Ġin Ġquestion âĢĵ answer Ġpairs Ġthat Ġexplicitly Ġinvolve Ġcommons ense Ġknowledge . ĠIn Ġcontrast Ġto Ġprevious Ġevaluation Ġtasks , Ġthis Ġsetup Ġallows Ġus Ġfor Ġthe Ġfirst Ġtime Ġto Ġassess Ġthe Ġcontribution Ġof Ġscript Ġknowledge Ġfor Ġcomputational Ġmodels Ġof Ġlanguage Ġunderstanding Ġin Ġa Ġreal - world Ġevaluation Ġscenario . </s> We Ġexpect Ġour Ġdataset Ġto Ġbecome Ġa Ġstandard Ġbenchmark Ġfor Ġtesting Ġmodels Ġof Ġcommons ense Ġand Ġscript Ġknowledge . ĠHuman Ġperformance Ġshows Ġthat Ġthe Ġdataset Ġis Ġhighly Ġreliable . ĠThe Ġresults Ġof Ġseveral Ġbas elines , Ġin Ġcontrast , Ġillustrate Ġthat Ġour Ġtask Ġprovides Ġchallenging Ġtest Ġcases Ġfor Ġthe Ġbroader Ġnatural Ġlanguage Ġprocessing Ġcommunity . ĠMC Script Ġforms Ġthe Ġbasis Ġof Ġa Ġshared Ġtask Ġorganized Ġat ĠSem E val Ġ2018 . ĠThe Ġdataset Ġis Ġavailable Ġat Ġhttp :// www . s fb 110 2 . uni - sa ar land . de /? page _ id = 25 82 . </s> Acknowled gements </s> We Ġthank Ġthe Ġreviewers Ġfor Ġtheir Ġhelpful Ġcomments . ĠWe Ġalso Ġthank ĠFlor ian ĠP us se Ġfor Ġthe Ġhelp Ġwith Ġthe ĠM Tur k Ġexperiments Ġand Ġour Ġstudent Ġassistants ĠChristine ĠSch Ãĥ Â ¤ fer , ĠDam y ana ĠGate va , ĠLeon ie ĠHar ter , ĠSarah ĠM ame che , ĠStefan ĠGr Ãĥ Â ¼ new ald Ġand ĠTat iana ĠAn ik ina Ġfor Ġhelp Ġwith Ġthe Ġannotations . ĠThis Ġresearch Ġwas Ġfunded Ġby Ġthe ĠGerman ĠResearch ĠFoundation Ġ( DF G ) Ġas Ġpart Ġof ĠSF B Ġ110 2 Ġ` Information ĠD ensity Ġand ĠL ingu istic ĠEnc oding ' Ġand ĠEX C Ġ284 Ġ` Mult im od al ĠComputing Ġand ĠInter action '.