Unanswerable
NONE
English
EXTRACTIVE
In order to generate more paraphrases of the questions, we used a backtranslation method to enrich them. The idea is to use a translation model that will translate our human-generated questions into another language, and then translate them back to English. This double translation will introduce rewordings of the questions that we will be able to integrate into this dataset. This approach has been used in BIBREF7 to perform data augmentation on the training set. For this purpose, we have trained a fairseq BIBREF19 model to translate sentences from English to French and for French to English. In order to preserve the quality of the sentences we have so far, we only keep the most probable translation of each original sentence. Indeed a beam search is used during the translation to predict the most probable translations which mean that we each translation comes with an associated probability. By selecting only the first translations, we almost double the number of questions without degrading the quality of the questions proposed in the dataset.
1810.12196
Introduction </s> A Ġlarge Ġmajority Ġof Ġthe Ġhuman Ġknowledge Ġis Ġrecorded Ġthrough Ġtext Ġdocuments . ĠThat Ġis Ġwhy Ġability Ġfor Ġa Ġsystem Ġto Ġautomatically Ġinfer Ġinformation Ġfrom Ġtext Ġwithout Ġany Ġstructured Ġdata Ġhas Ġbecome Ġa Ġmajor Ġchallenge . ĠAn sw ering Ġquestions Ġabout Ġa Ġgiven Ġdocument Ġis Ġa Ġrelevant Ġproxy Ġtask Ġthat Ġhas Ġbeen Ġproposed Ġas Ġa Ġway Ġto Ġevaluate Ġthe Ġreading Ġability Ġof Ġa Ġgiven Ġmodel . ĠIn Ġthis Ġconfiguration , Ġa Ġtext Ġdocument Ġsuch Ġas Ġa Ġnews Ġarticle , Ġa Ġdocument Ġfrom ĠWikipedia Ġor Ġany Ġtype Ġof Ġtext Ġis Ġpresented Ġto Ġa Ġmachine Ġwith Ġan Ġassociated Ġset Ġof Ġquestions . ĠThe Ġsystem Ġis Ġthen Ġexpected Ġto Ġanswer Ġthese Ġquestions Ġand Ġevaluated Ġby Ġits Ġaccuracy Ġon Ġthis Ġtask . ĠThe Ġmachine Ġreading Ġframework Ġis Ġvery Ġgeneral Ġand Ġwe Ġcan Ġimagine Ġa Ġlarge Ġpanel Ġof Ġquestions Ġthat Ġcan Ġpossibly Ġhandle Ġmost Ġof Ġthe Ġstandard Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠFor Ġexample , Ġthe Ġtask Ġof Ġnamed Ġentities Ġrecognition Ġcan Ġbe Ġformulated Ġas Ġa Ġmachine Ġreading Ġone Ġwhere Ġyour Ġdocument Ġis Ġthe Ġsentence Ġand Ġthe Ġquestion Ġwould Ġbe Ġ' What Ġare Ġthe Ġnamed Ġentities Ġmentioned Ġin Ġthis Ġsentence ? '. ĠThese Ġnatural Ġlanguage Ġinteractions Ġare Ġan Ġimportant Ġobjective Ġfor Ġreading Ġsystems . </s> Recently , Ġmany Ġdatasets Ġhave Ġbeen Ġproposed Ġto Ġbuild Ġand Ġevaluate Ġreading Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠFrom Ġclo ze Ġstyle Ġquestions ĠB IB REF 2 Ġto Ġopen Ġquestions ĠB IB REF 3 Ġ, Ġfrom Ġsynthetic Ġdata ĠB IB REF 4 Ġto Ġhuman Ġwritten Ġarticles ĠB IB REF 5 Ġ, Ġmany Ġstyles Ġof Ġdocuments Ġand Ġquestions Ġhave Ġbeen Ġproposed Ġto Ġchallenge Ġreading Ġmodels . ĠThe Ġcorrect Ġanswer Ġto Ġthe Ġquestions Ġproposed Ġin Ġmost Ġof Ġthese Ġdatasets Ġis Ġa Ġspan Ġof Ġtext Ġof Ġthe Ġsource Ġdocument , Ġwhich Ġcan Ġbe Ġrestricted Ġto Ġa Ġsingle Ġword Ġin Ġseveral Ġcases . ĠIt Ġmeans Ġthat Ġthe Ġanswer Ġshould Ġexplicitly Ġbe Ġpresent Ġin Ġthe Ġsource Ġdocument Ġand Ġthat Ġthe Ġmodel Ġshould Ġbe Ġable Ġto Ġlocate Ġit . </s> Different Ġmodels Ġhave Ġalready Ġshown Ġsuperhuman Ġperformance Ġon Ġseveral Ġof Ġthese Ġdatasets Ġand Ġparticularly Ġon Ġthe ĠS Qu AD Ġdataset Ġcomposed Ġof ĠWikipedia Ġarticles ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġsome Ġlimits Ġof Ġsuch Ġmodels Ġhave Ġbeen Ġhighlighted Ġwhen Ġthey Ġencounter Ġpert urb ations Ġinto Ġthe Ġinput Ġdocuments ĠB IB REF 8 Ġ. ĠIndeed Ġalmost Ġall Ġof Ġthe Ġstate Ġof Ġthe Ġart Ġmodels Ġon Ġthe ĠS Qu AD Ġdataset Ġsuffer Ġfrom Ġa Ġlack Ġof Ġrobust ness Ġagainst Ġadvers arial Ġexamples . ĠOnce Ġthe Ġmodel Ġis Ġtrained , Ġa Ġmeaningless Ġsentence Ġadded Ġat Ġthe Ġend Ġof Ġthe Ġtext Ġdocument Ġcan Ġcompletely Ġdisturb Ġthe Ġreading Ġsystem . ĠConversely , Ġthese Ġadvers arial Ġexamples Ġdo Ġnot Ġseem Ġto Ġfool Ġa Ġhuman Ġreader Ġwho Ġwill Ġbe Ġcapable Ġof Ġanswering Ġthe Ġquestions Ġas Ġwell Ġas Ġwithout Ġthis Ġpert urb ation . ĠOne Ġpossible Ġexplanation Ġof Ġthis Ġphenomenon Ġis Ġthat Ġcomputers Ġare Ġgood Ġat Ġextracting Ġpatterns Ġin Ġthe Ġdocument Ġthat Ġmatch Ġthe Ġrepresentation Ġof Ġthe Ġquestion . ĠIf Ġmultiple Ġspans Ġof Ġthe Ġdocuments Ġlook Ġsimilar Ġto Ġthe Ġquestions , Ġthe Ġreader Ġmight Ġnot Ġbe Ġable Ġto Ġdecide Ġwhich Ġone Ġis Ġrelevant . ĠMoreover , ĠWikipedia Ġarticles Ġtend Ġto Ġbe Ġwritten Ġwith Ġthe Ġsame Ġstandard Ġwriting Ġstyle , Ġfactual , Ġunamb iguous . ĠSuch Ġwriting Ġstyle Ġtends Ġto Ġfavor Ġthe Ġpattern Ġmatching Ġbetween Ġthe Ġquestions Ġand Ġthe Ġdocuments . ĠThis Ġformat Ġof Ġdocuments / quest ions Ġhas Ġcertainly Ġinfluenced Ġthe Ġdesign Ġof Ġthe Ġcomprehension Ġmodels Ġthat Ġhave Ġbeen Ġproposed Ġso Ġfar . ĠMost Ġof Ġthem Ġare Ġcomposed Ġof Ġstacked Ġattention Ġlayers Ġthat Ġmatch Ġquestion Ġand Ġdocument Ġrepresentations . </s> Following Ġconcepts Ġproposed Ġin Ġthe Ġ20 Ġb Ab I Ġtasks ĠB IB REF 4 Ġor Ġin Ġthe Ġvisual Ġquestion - ans w ering Ġdataset ĠCLE VR ĠB IB REF 9 Ġ, Ġwe Ġthink Ġthat Ġthe Ġchallenge , Ġlimited Ġto Ġthe Ġdetection Ġof Ġrelevant Ġpassages Ġin Ġa Ġdocument , Ġis Ġonly Ġthe Ġfirst Ġstep Ġin Ġbuilding Ġsystems Ġthat Ġtruly Ġunderstand Ġtext . ĠThe Ġsecond Ġstep Ġis Ġthe Ġability Ġof Ġreasoning Ġwith Ġthe Ġrelevant Ġinformation Ġextracted Ġfrom Ġa Ġdocument . ĠTo Ġset Ġup Ġthis Ġchallenge , Ġwe Ġpropose Ġto Ġleverage Ġon Ġa Ġhotel Ġreviews Ġcorpus Ġthat Ġrequires Ġreasoning Ġskills Ġto Ġanswer Ġnatural Ġlanguage Ġquestions . ĠThe Ġreviews Ġwe Ġused Ġhave Ġbeen Ġextracted Ġfrom ĠTrip Ad visor Ġand Ġoriginally Ġproposed Ġin ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠIn Ġthe Ġoriginal Ġdata , Ġeach Ġreview Ġcomes Ġwith Ġa Ġset Ġof Ġrated Ġaspects Ġamong Ġthe Ġseventh Ġavailable : ĠBusiness Ġservice , ĠCheck Ġin Ġ/ ĠFront ĠDesk , ĠClean liness , ĠLocation , ĠRoom , ĠSleep ĠQuality , ĠValue Ġand Ġfor Ġall Ġthe Ġreviews Ġan ĠOverall Ġrating . ĠIn Ġthis Ġarticles Ġwe Ġpropose Ġto Ġexploit Ġthese Ġdata Ġto Ġcreate Ġa Ġdataset Ġof Ġquestion - ans w ering Ġthat Ġwill Ġchallenge Ġ8 Ġcompet encies Ġof Ġthe Ġreader . </s> Our Ġcontributions Ġcan Ġbe Ġsummarized Ġas Ġfollow : </s> Machine Ġcomprehension Ġdatasets </s> Review Q A Ġis Ġproposed Ġas Ġa Ġnovel Ġdataset Ġregarding Ġthe Ġcollection Ġof Ġthe Ġexisting Ġones . ĠIndeed Ġa Ġlarge Ġpanel Ġof Ġavailable Ġdatasets , Ġthat Ġevaluate Ġmodels Ġon Ġdifferent Ġtypes Ġof Ġdocuments , Ġcan Ġonly Ġbe Ġvaluable Ġfor Ġdesigning Ġefficient Ġmodels Ġand Ġlearning Ġprotocols . ĠIn Ġthis Ġfollowing Ġpart , Ġwe Ġdescribe Ġseveral Ġof Ġthese Ġdatasets . </s> S Qu AD : ĠThe ĠStand ford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD ) Ġintroduced Ġin ĠB IB REF 0 Ġis Ġa Ġlarge Ġdataset Ġof Ġnatural Ġquestions Ġover Ġthe Ġ500 Ġmost Ġpopular Ġarticles Ġof ĠWikipedia . ĠAll Ġthe Ġquestions Ġhave Ġbeen Ġcrowds ourced Ġand Ġanswers Ġare Ġspans Ġof Ġtext Ġextracted Ġfrom Ġsource Ġdocuments . ĠThis Ġdataset Ġhas Ġbeen Ġvery Ġpopular Ġthese Ġlast Ġtwo Ġyears Ġand Ġthe Ġperformance Ġof Ġthe Ġarchitectures Ġthat Ġhave Ġbeen Ġproposed Ġhave Ġrapidly Ġincreased Ġuntil Ġseveral Ġmodels Ġsurpass Ġthe Ġhuman Ġscore . ĠIndeed , Ġin Ġthe Ġoriginal Ġpaper Ġhuman Ġperformance Ġhas Ġbeen Ġmeasured Ġat Ġ82 . 304 Ġpoints Ġfor Ġthe Ġexact Ġmatch Ġmetric Ġand Ġat Ġthe Ġtime Ġwe Ġare Ġwriting Ġthis Ġpaper Ġfour Ġmodels Ġhave Ġalready Ġa Ġhigher Ġscore . ĠIn Ġanother Ġhand ĠB IB REF 8 Ġhas Ġshown Ġthat Ġthese Ġmodels Ġsuffer Ġfrom Ġa Ġlack Ġof Ġrobust ness Ġagainst Ġadvers arial Ġexamples Ġthat Ġare Ġmeaningless Ġfrom Ġa Ġhuman Ġpoint Ġof Ġview . ĠThis Ġsuggests Ġthe Ġneed Ġfor Ġa Ġmore Ġchallenging Ġdataset Ġthat Ġwill Ġallow Ġdeveloping Ġstrongest Ġreasoning Ġarchitectures . </s> News Q A : ĠNews Q A ĠB IB REF 1 Ġis Ġa Ġdataset Ġvery Ġsimilar Ġto ĠS Qu AD . ĠIt Ġcontains Ġ120 . 000 Ġhuman Ġgenerated Ġquestions Ġover Ġ12 . 000 Ġarticles Ġform ĠCNN Ġoriginally Ġintroduced Ġin ĠB IB REF 5 Ġ. ĠIt Ġhas Ġbeen Ġdesigned Ġto Ġbe Ġmore Ġchallenging Ġthan ĠS Qu AD Ġwith Ġquestions Ġthat Ġmight Ġrequire Ġto Ġextract Ġmultiple Ġspans Ġof Ġtext Ġor Ġnot Ġbe Ġanswer able . </s> Wiki Hop Ġand ĠMed Hop : ĠThese Ġare Ġtwo Ġrecent Ġdatasets Ġintroduced Ġin ĠB IB REF 13 Ġ. ĠUnlike ĠS Qu AD Ġand ĠNews Q A , Ġimportant Ġfacts Ġare Ġspread Ġout Ġacross Ġmultiple Ġdocuments Ġand , Ġin Ġorder Ġto Ġanswer Ġa Ġquestion , Ġit Ġis Ġnecessary Ġto Ġjump Ġover Ġa Ġset Ġof Ġpassages Ġto Ġcollect Ġthe Ġrequired Ġinformation . ĠThe Ġrelevant Ġpassages Ġare Ġnot Ġexplicitly Ġmentioned Ġin Ġthe Ġdata Ġso Ġthis Ġdataset Ġmeasures Ġthe Ġability Ġthat Ġa Ġmodel Ġhas Ġto Ġnavigate Ġacross Ġmultiple Ġdocuments . ĠThe Ġquestions Ġcome Ġwith Ġa Ġset Ġof Ġcandidates Ġwhich Ġare Ġall Ġpresent Ġin Ġthe Ġtext . </s> MS ĠMarco : ĠThis Ġdataset Ġhas Ġbeen Ġreleased Ġin ĠB IB REF 14 Ġ. ĠThe Ġdocuments Ġcome Ġfrom Ġthe Ġinternet Ġand Ġthe Ġquestions Ġare Ġreal Ġuser Ġqueries Ġasked Ġthrough Ġthe Ġb ing Ġsearch Ġengine . ĠThe Ġdataset Ġcontains Ġaround Ġ100 . 000 Ġqueries Ġand Ġeach Ġof Ġthem Ġcomes Ġwith Ġa Ġset Ġof Ġapprox im atively Ġ10 Ġrelevant Ġpassages . ĠLike Ġin ĠS Qu AD , Ġseveral Ġmodels Ġare Ġalready Ġdoing Ġsuperhuman Ġperformances Ġon Ġthis Ġdataset . </s> Facebook Ġb Ab I Ġtasks : ĠThis Ġis Ġa Ġset Ġof Ġ20 Ġtoy Ġtasks Ġproposed Ġin ĠB IB REF 4 Ġand Ġdesigned Ġto Ġmeasure Ġtext Ġunderstanding . ĠEach Ġtask Ġrequires Ġa Ġcertain Ġcapability Ġto Ġbe Ġcompleted Ġlike Ġinduction , Ġdeduction Ġand Ġmore . ĠDocuments Ġare Ġsynthetic Ġstories , Ġcomposed Ġof Ġfew Ġsentences Ġthat Ġdescribe Ġa Ġset Ġof Ġactions . ĠThis Ġdataset Ġwas Ġone Ġof Ġthe Ġfirst Ġattempt Ġto Ġintroduce Ġa Ġgeneral Ġset Ġof Ġprerequisite Ġcapabilities Ġrequired Ġfor Ġthe Ġreading Ġtask . ĠAlthough Ġit Ġhas Ġbeen Ġa Ġvery Ġchallenging Ġframework , Ġbeneficial Ġto Ġthe Ġemergence Ġof Ġthe Ġattention Ġmechanism Ġinside Ġthe Ġreading Ġarchitectures , Ġa ĠG ated Ġend - to - end Ġmemory Ġnetwork ĠB IB REF 15 Ġnow Ġsucceed Ġin Ġalmost Ġall Ġof Ġthe Ġ20 Ġtasks . ĠOne Ġof Ġthe Ġpossible Ġreason Ġis Ġthat Ġthe Ġdata Ġare Ġsynthetic Ġdata , Ġwithout Ġnoise Ġor Ġambiguity . ĠWe Ġpropose Ġa Ġcomparable Ġframework Ġwith Ġunderstanding Ġand Ġreasoning Ġtasks Ġbased Ġon Ġuser - generated Ġcomments Ġthat Ġare Ġmuch Ġmore Ġrealistic Ġand Ġthat Ġrequired Ġlanguage Ġcompet encies Ġto Ġbe Ġunderstood . </s> CLE VR : ĠBeyond Ġtextual Ġquestion - ans w ering , ĠVisual ĠQuestion - An sw ering Ġ( V Q A ) Ġhas Ġbeen Ġlargely Ġstudied Ġduring Ġthe Ġlast Ġcouple Ġof Ġyears . ĠMore Ġrecently , Ġthe Ġproblem Ġof Ġrelational Ġreasoning Ġhas Ġbeen Ġintroduced Ġthrough Ġthis Ġdataset ĠB IB REF 9 Ġ. ĠThe Ġmain Ġoriginal Ġidea Ġwas Ġto Ġintroduce Ġrelational Ġreasoning Ġquestions Ġover Ġobject Ġshapes Ġand Ġplac ements . ĠThis Ġdataset Ġhas Ġalready Ġmotivated Ġthe Ġdevelopment Ġof Ġoriginal Ġdeep Ġmodels . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġnatural Ġlanguage Ġquestion - ans w ering Ġcorpus Ġhas Ġbeen Ġdesigned Ġto Ġinvestigate Ġsuch Ġcapabilities . ĠAs Ġwe Ġwill Ġpresent Ġin Ġthe Ġfollowing Ġof Ġthis Ġpaper , Ġwe Ġthink Ġsentiment Ġanalysis Ġis Ġparticularly Ġsuited Ġfor Ġthis Ġtask Ġand Ġwe Ġwill Ġintroduce Ġa Ġnovel Ġmachine Ġreading Ġcorpus Ġwith Ġsuch Ġcapability Ġrequirements . </s> Att ention - based Ġmodels Ġfor Ġaspect - based Ġsentiment Ġanalysis </s> Sent iment Ġanalysis Ġis Ġone Ġof Ġthe Ġhistorical Ġtasks Ġof ĠNatural ĠLanguage ĠProcessing . ĠIt Ġis Ġan Ġimportant Ġchallenge Ġfor Ġcompanies , Ġrestaurants , Ġhotels Ġthat Ġaim Ġto Ġanalyze Ġcustomer Ġsatisfaction Ġregarding Ġproducts Ġand Ġquality Ġof Ġservices . ĠGiven Ġa Ġtext Ġdocument , Ġthe Ġobjective Ġis Ġto Ġpredict Ġits Ġoverall Ġpol arity . ĠGenerally , Ġit Ġcan Ġbe Ġpositive , Ġnegative Ġor Ġneutral . ĠThis Ġanalysis Ġgives Ġa Ġquick Ġoverview Ġof Ġa Ġgeneral Ġsentiment Ġover Ġa Ġset Ġof Ġdocuments , Ġbut Ġthis Ġframework Ġtends Ġto Ġbe Ġrestrictive . ĠIndeed , Ġone Ġdocument Ġtends Ġto Ġexpress Ġmultiple Ġopinions Ġof Ġdifferent Ġaspects . ĠFor Ġinstance , Ġin Ġthe Ġsentence : ĠThe Ġfish Ġwas Ġvery Ġgood Ġbut Ġthe Ġservice Ġwas Ġterrible , Ġthere Ġis Ġnot Ġa Ġgeneral Ġdominant Ġsentiment , Ġand Ġa Ġfiner Ġanalysis Ġis Ġneeded . ĠThe Ġtask Ġof Ġaspect - based Ġsentiment Ġanalysis Ġaims Ġto Ġpredict Ġa Ġpol arity Ġof Ġa Ġsentence Ġregarding Ġa Ġgiven Ġaspect . ĠIn Ġthe Ġprevious Ġexample Ġa Ġpositive Ġpol arity Ġshould Ġbe Ġassociated Ġto Ġthe Ġaspect Ġfood , Ġand Ġon Ġthe Ġcontrary , Ġa Ġnegative Ġsentiment Ġis Ġexpressed Ġregarding Ġthe Ġquality Ġof Ġthe Ġservice . </s> The Ġidea Ġof Ġusing Ġmodels Ġoriginally Ġdesigned Ġfor Ġquestion - ans w ering , Ġfor Ġthe Ġsentiment Ġanalysis Ġtask Ġhas Ġbeen Ġintroduced Ġin ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠIn Ġthese Ġpapers , Ġseveral Ġadaptations Ġof Ġthe Ġend - to - end Ġmemory Ġnetwork Ġ( Mem N 2 N ) ĠB IB REF 18 Ġare Ġused Ġto Ġpredict Ġthe Ġpol arity Ġof Ġa Ġreview Ġregarding Ġa Ġgiven Ġaspect . ĠIn Ġthat Ġconfiguration , Ġthe Ġreview Ġis Ġencoded Ġinto Ġthe Ġmemory Ġcells Ġand Ġthe Ġcontroller , Ġusually Ġinitialized Ġwith Ġa Ġrepresentation Ġof Ġthe Ġquestion , Ġis Ġinitialized Ġwith Ġa Ġrepresentation Ġof Ġthe Ġaspect . ĠThe Ġanalysis Ġof Ġthe Ġattention Ġbetween Ġthe Ġvalues Ġof Ġthe Ġcontroller Ġand Ġthe Ġdocument Ġhas Ġshown Ġinteresting Ġresults , Ġby Ġhighlighting Ġrelevant Ġpart Ġof Ġa Ġdocument Ġregarding Ġan Ġaspect . </s> Review Q A Ġdataset </s> We Ġthink Ġthat Ġevaluating Ġthe Ġtask Ġof Ġsentiment Ġanalysis Ġthrough Ġthe Ġsetup Ġof Ġquestion - ans w ering Ġis Ġa Ġrelevant Ġplayground Ġfor Ġmachine Ġreading Ġresearch . ĠIndeed Ġnatural Ġlanguage Ġquestions Ġabout Ġthe Ġdifferent Ġaspects Ġof Ġthe Ġtargeted Ġvenues Ġare Ġtypical Ġkind Ġof Ġquestions Ġwe Ġwant Ġto Ġbe Ġable Ġto Ġask Ġto Ġa Ġsystem . ĠIn Ġthis Ġcontext , Ġwe Ġintroduce Ġa Ġset Ġof Ġreasoning Ġquestions Ġtypes Ġover Ġthe Ġrelationships Ġbetween Ġaspects . ĠWe Ġpropose ĠReview Q A , Ġa Ġdataset Ġof Ġnatural Ġlanguage Ġquestions Ġover Ġhotel Ġreviews . ĠThese Ġquestions Ġare Ġdivided Ġinto Ġ8 Ġgroups , Ġregarding Ġthe Ġcompet ency Ġrequired Ġto Ġbe Ġanswered . ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġeach Ġtask Ġand Ġthe Ġprocess Ġfollowed Ġto Ġgenerate Ġthis Ġdataset . </s> Original Ġdata </s> We Ġused Ġa Ġset Ġof Ġreviews Ġextracted Ġfrom Ġthe ĠTrip Ad visor Ġwebsite Ġand Ġoriginally Ġproposed Ġin ĠB IB REF 10 Ġand ĠB IB REF 11 Ġ. ĠThis Ġcorpus Ġis Ġavailable Ġat Ġhttp :// www . cs . v irgin ia . edu / ~ hw 5 x / Data / L ARA / T rip Ad visor / T rip Ad visor J son . tar . b z 2 . ĠEach Ġreview Ġcomes Ġwith Ġthe Ġname Ġof Ġthe Ġassociated Ġhotel , Ġa Ġtitle , Ġan Ġoverall Ġrating , Ġa Ġcomment Ġand Ġa Ġlist Ġof Ġrated Ġaspects . ĠFrom Ġ0 Ġto Ġ7 Ġaspects , Ġamong Ġvalue , Ġroom , Ġlocation , Ġclean liness , Ġcheck - in / front Ġdesk , Ġservice , Ġbusiness Ġservice , Ġcan Ġpossibly Ġbe Ġrated Ġin Ġa Ġreview . ĠFigure ĠFIG REF 8 Ġdisplays Ġa Ġreview Ġextracted Ġfrom Ġthis Ġdataset . </s> Rel ational Ġreasoning Ġcompet encies </s> Object ive : ĠStarting Ġwith Ġthe Ġoriginal Ġcorpus , Ġwe Ġaim Ġat Ġbuilding Ġa Ġmachine Ġreading Ġtask Ġwhere Ġnatural Ġlanguage Ġquestions Ġwill Ġchallenge Ġthe Ġmodel Ġon Ġits Ġunderstanding Ġof Ġthe Ġreviews . ĠIndeed Ġlearning Ġrelational Ġreasoning Ġcompet encies Ġover Ġnatural Ġlanguage Ġdocuments Ġis Ġa Ġmajor Ġchallenge Ġof Ġthe Ġcurrent Ġreading Ġmodels . ĠThese Ġoriginal Ġraw Ġdata Ġallow Ġus Ġto Ġgenerate Ġrelational Ġquestions Ġthat Ġcan Ġpossibly Ġrequire Ġa Ġglobal Ġunderstanding Ġof Ġthe Ġcomment Ġand Ġreasoning Ġskills Ġto Ġbe Ġtreated . ĠFor Ġexample , Ġasking Ġa Ġquestion Ġlike ĠWhat Ġis Ġthe Ġbest Ġaspect Ġrated Ġin Ġthis Ġcomment Ġ? Ġis Ġnot Ġan Ġeasy Ġquestion Ġthat Ġcan Ġbe Ġanswered Ġwithout Ġa Ġdeep Ġunderstanding Ġof Ġthe Ġreview . ĠIt Ġis Ġnecessary Ġto Ġcapture Ġall Ġthe Ġaspects Ġmentioned Ġin Ġthe Ġtext , Ġto Ġpredict Ġtheir Ġrating Ġand Ġfinally Ġto Ġselect Ġthe Ġbest Ġone . ĠThe Ġtasks Ġand Ġthe Ġdataset Ġwe Ġpropose Ġare Ġpublicly Ġavailable Ġat Ġhttp :// www . euro pe . na ver l abs . com / Blog / Review Q A - A - no vel - rel ational - as pect - based - op inion - dat as et - for - machine - reading </s> We Ġintroduce Ġa Ġlist Ġof Ġ8 Ġdifferent Ġcompet encies Ġthat Ġa Ġreading Ġsystem Ġshould Ġmaster Ġin Ġorder Ġto Ġprocess Ġreviews Ġand Ġtext Ġdocuments Ġin Ġgeneral . ĠThese Ġ8 Ġtasks Ġrequire Ġdifferent Ġcompet encies Ġand Ġa Ġdifferent Ġlevel Ġof Ġunderstanding Ġof Ġthe Ġdocument Ġto Ġbe Ġwell Ġanswered . ĠFor Ġinstance , Ġdetecting Ġif Ġan Ġaspect Ġis Ġmentioned Ġin Ġa Ġreview Ġwill Ġrequire Ġless Ġunderstanding Ġof Ġthe Ġreview Ġthan Ġpredicting Ġexplicitly Ġthe Ġrating Ġof Ġthis Ġaspect . ĠTable ĠT AB REF 10 Ġpresents Ġthe Ġ8 Ġtasks Ġwe Ġhave Ġintroduced Ġin Ġthis Ġdataset Ġwith Ġan Ġexample Ġof Ġa Ġquestion Ġthat Ġcorresponds Ġto Ġeach Ġtask . ĠWe Ġalso Ġprovide Ġthe Ġexpected Ġtype Ġof Ġthe Ġanswer Ġ( Yes / No Ġquestion , Ġrating Ġquestion ... ). ĠIt Ġcan Ġbe Ġan Ġadditional Ġtool Ġto Ġanalyze Ġthe Ġerrors Ġof Ġthe Ġreaders . </s> Construction Ġof Ġthe Ġdataset </s> We Ġsample Ġ100 . 000 Ġreviews Ġfrom Ġthe Ġoriginal Ġcorpus . ĠFigure ĠFIG REF 12 Ġpresents Ġthe Ġdistribution Ġof Ġthe Ġnumber Ġof Ġwords Ġof Ġthe Ġreviews Ġin Ġthe Ġdataset . ĠWe Ġexplicitly Ġfavor Ġreviews Ġwhich Ġcontain Ġan Ġimportant Ġnumber Ġof Ġwords . ĠIn Ġaverage , Ġa Ġreview Ġcontains Ġ200 Ġwords . ĠIndeed Ġthese Ġlong Ġreviews Ġare Ġmost Ġlikely Ġto Ġcontain Ġchallenging Ġrelations Ġbetween Ġdifferent Ġaspects . ĠA Ġshort Ġreview Ġwhich Ġdeals Ġwith Ġonly Ġa Ġfew Ġaspects Ġis Ġmore Ġlikely Ġto Ġnot Ġbe Ġvery Ġrelevant Ġto Ġthe Ġchallenge Ġwe Ġwant Ġto Ġpropose Ġin Ġthis Ġdataset . ĠFigure ĠFIG REF 14 Ġdisplays Ġthe Ġdistribution Ġof Ġthe Ġratings Ġper Ġaspects Ġin Ġthe Ġ100 . 000 Ġreviews Ġwe Ġbased Ġour Ġdataset . ĠWe Ġcan Ġsee Ġthat Ġthe Ġaverage Ġvalues Ġof Ġthese Ġratings Ġtend Ġto Ġbe Ġquite Ġhigh . ĠIt Ġcould Ġhave Ġintroduced Ġbias Ġif Ġit Ġwas Ġnot Ġthe Ġcase Ġfor Ġall Ġthe Ġaspects . ĠFor Ġexample , Ġwe Ġdo Ġnot Ġwant Ġthat Ġthe Ġmodel Ġlearns Ġthat Ġin Ġgeneral , Ġthe Ġservice Ġis Ġrated Ġbetter Ġthan Ġthe Ġlocation Ġand Ġthem Ġanswer Ġwithout Ġlooking Ġat Ġthe Ġdocument . ĠSince Ġthis Ġsituation Ġis Ġthe Ġsame Ġfor Ġall Ġthe Ġaspects , Ġthe Ġrelational Ġtasks Ġintroduced Ġin Ġthis Ġdataset Ġremains Ġextremely Ġrelevant . </s> Then Ġwe Ġrandomly Ġselect Ġ6 Ġtasks Ġfor Ġeach Ġreview Ġ( the Ġsame Ġtask Ġcan Ġbe Ġselected Ġmultiple Ġtimes ) Ġand Ġrandomly Ġselect Ġa Ġnatural Ġlanguage Ġquestion Ġthat Ġcorresponds Ġto Ġthis Ġtask . ĠThe Ġquestions Ġare Ġhuman - generated Ġpatterns Ġthat Ġwe Ġhave Ġcrowds ourced Ġin Ġorder Ġto Ġproduce Ġa Ġdataset Ġas Ġrich Ġas Ġpossible . ĠTo Ġthis Ġend , Ġwe Ġhave Ġgenerated Ġseveral Ġpatterns Ġthat Ġcorrespond Ġto Ġthe Ġcapabilities Ġwe Ġwanted Ġto Ġexpress Ġin Ġa Ġgiven Ġquestion Ġand Ġwe Ġhave Ġcrowds ourced Ġre ph r asing Ġof Ġthese Ġpatterns . </s> The Ġfinal Ġdataset Ġwe Ġpropose Ġis Ġcomposed Ġof Ġmore Ġthan Ġ500 . 000 Ġquestions Ġabout Ġ100 . 000 Ġreviews . ĠTable ĠT AB REF 13 Ġshows Ġthe Ġrep art ition Ġof Ġthe Ġdocuments Ġand Ġqueries Ġinto Ġthe Ġtrain Ġand Ġtest Ġset . ĠEach Ġreview Ġcontains Ġa Ġmaximum Ġof Ġ6 Ġquestions . ĠSometimes Ġless Ġwhen Ġit Ġis Ġnot Ġpossible Ġto Ġgenerate Ġall . ĠFor Ġexample , Ġif Ġonly Ġtwo Ġor Ġthree Ġaspects Ġare Ġmentioned Ġin Ġa Ġreview , Ġwe Ġwill Ġbe Ġable Ġto Ġgenerate Ġonly Ġa Ġlittle Ġset Ġof Ġrelational Ġquestions . ĠFigure ĠFIG REF 15 Ġdepicts Ġthe Ġrep art ition Ġof Ġthe Ġanswers Ġin Ġthe Ġgenerated Ġdataset . ĠA Ġmajority Ġof Ġthe Ġtasks Ġwe Ġintroduced , Ġeven Ġif Ġthey Ġpossibly Ġrequire Ġa Ġhigh Ġlevel Ġof Ġunderstanding Ġof Ġthe Ġdocument Ġand Ġthe Ġquestion , Ġare Ġbinary Ġquestions . ĠIt Ġmeans Ġthat Ġin Ġthe Ġgenerated Ġdataset Ġthe Ġanswers Ġyes Ġand Ġno Ġtend Ġto Ġbe Ġmore Ġpresent Ġthan Ġthe Ġothers . ĠTo Ġbalance Ġin Ġa Ġbetter Ġway Ġthe Ġdistribution Ġof Ġthe Ġanswers , Ġwe Ġchose Ġto Ġaffect Ġa Ġhigher Ġprobability Ġof Ġsampling Ġto Ġthe Ġtask Ġ5 , Ġ6 , Ġ7 . 1 , Ġ8 . ĠIndeed , Ġthese Ġtasks Ġare Ġnot Ġbinary Ġquestions Ġand Ġrequired Ġan Ġaspect Ġname Ġas Ġthe Ġanswer . ĠFigure ĠFIG REF 17 Ġrepresents Ġthe Ġrep art ition Ġof Ġquestion Ġtypes Ġin Ġour Ġdataset . ĠFinally , Ġfigure ĠFIG REF 15 Ġshows Ġthe Ġrep art ition Ġof Ġthe Ġanswers Ġin Ġthe Ġdataset . </s> Par aph rase Ġaug mentation Ġusing Ġback translation </s> In Ġorder Ġto Ġgenerate Ġmore Ġparaph r ases Ġof Ġthe Ġquestions , Ġwe Ġused Ġa Ġback translation Ġmethod Ġto Ġenrich Ġthem . ĠThe Ġidea Ġis Ġto Ġuse Ġa Ġtranslation Ġmodel Ġthat Ġwill Ġtranslate Ġour Ġhuman - generated Ġquestions Ġinto Ġanother Ġlanguage , Ġand Ġthen Ġtranslate Ġthem Ġback Ġto ĠEnglish . ĠThis Ġdouble Ġtranslation Ġwill Ġintroduce Ġre word ings Ġof Ġthe Ġquestions Ġthat Ġwe Ġwill Ġbe Ġable Ġto Ġintegrate Ġinto Ġthis Ġdataset . ĠThis Ġapproach Ġhas Ġbeen Ġused Ġin ĠB IB REF 7 Ġto Ġperform Ġdata Ġaug mentation Ġon Ġthe Ġtraining Ġset . ĠFor Ġthis Ġpurpose , Ġwe Ġhave Ġtrained Ġa Ġfair seq ĠB IB REF 19 Ġmodel Ġto Ġtranslate Ġsentences Ġfrom ĠEnglish Ġto ĠFrench Ġand Ġfor ĠFrench Ġto ĠEnglish . ĠIn Ġorder Ġto Ġpreserve Ġthe Ġquality Ġof Ġthe Ġsentences Ġwe Ġhave Ġso Ġfar , Ġwe Ġonly Ġkeep Ġthe Ġmost Ġprobable Ġtranslation Ġof Ġeach Ġoriginal Ġsentence . ĠIndeed Ġa Ġbeam Ġsearch Ġis Ġused Ġduring Ġthe Ġtranslation Ġto Ġpredict Ġthe Ġmost Ġprobable Ġtranslations Ġwhich Ġmean Ġthat Ġwe Ġeach Ġtranslation Ġcomes Ġwith Ġan Ġassociated Ġprobability . ĠBy Ġselecting Ġonly Ġthe Ġfirst Ġtranslations , Ġwe Ġalmost Ġdouble Ġthe Ġnumber Ġof Ġquestions Ġwithout Ġdegrading Ġthe Ġquality Ġof Ġthe Ġquestions Ġproposed Ġin Ġthe Ġdataset . </s> Mod els </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġperformance Ġof Ġfour Ġdifferent Ġmodels Ġon Ġour Ġdataset : Ġa Ġlog istic Ġregression Ġand Ġthree Ġneural Ġmodels . ĠThe Ġfirst Ġone Ġis Ġa Ġbasic ĠL ST M ĠB IB REF 20 Ġ, Ġthe Ġsecond Ġa ĠMem N 2 N ĠB IB REF 18 Ġand Ġthe Ġthird Ġone Ġis Ġa Ġmodel Ġof Ġour Ġown Ġdesign . ĠThis Ġfourth Ġmodel Ġre uses Ġthe Ġencoding Ġlayers Ġof Ġthe ĠR - net ĠB IB REF 12 Ġand Ġwe Ġmodify Ġthe Ġfinal Ġlayers Ġwith Ġa Ġprojection Ġlayer Ġthat Ġwill Ġbe Ġable Ġto Ġselect Ġthe Ġanswer Ġamong Ġthe Ġset Ġof Ġcandidates Ġinstead Ġof Ġpointing Ġthe Ġansw erer Ġdirectly Ġinto Ġthe Ġsource Ġdocument . </s> Log istic Ġregression : ĠTo Ġproduce Ġthe Ġrepresentation Ġof Ġthe Ġinput , Ġwe Ġconc aten ate Ġthe ĠBag - Of - Words Ġrepresentation Ġof Ġthe Ġdocument Ġwith Ġthe ĠBag - Of - Words Ġrepresentation Ġof Ġthe Ġquestion . ĠIt Ġproduces Ġan Ġarray Ġof Ġsize ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġvocabulary Ġsize . ĠThen Ġwe Ġuse Ġa Ġlog istic Ġregression Ġto Ġselect Ġthe Ġmost Ġprobable Ġanswer Ġamong Ġthe ĠIN LINE FORM 2 Ġpossibilities . </s> L ST M : ĠWe Ġstart Ġwith Ġa Ġconc aten ation Ġof Ġthe Ġsequence Ġof Ġindexes Ġof Ġthe Ġdocument Ġwith Ġthe Ġsequence Ġof Ġindexes Ġof Ġthe Ġquestion . ĠThem Ġwe Ġfeed Ġan ĠL ST M Ġnetwork Ġwith Ġthis Ġvector Ġand Ġuse Ġthe Ġfinal Ġstate Ġas Ġthe Ġrepresentation Ġof Ġthe Ġinput . ĠFinally , Ġwe Ġapply Ġa Ġlog istic Ġregression Ġover Ġthis Ġrepresentation Ġto Ġproduce Ġthe Ġfinal Ġdecision . </s> End - to - end Ġmemory Ġnetworks : ĠThis Ġarchitecture Ġis Ġbased Ġon Ġtwo Ġdifferent Ġmemory Ġcells Ġ( input Ġand Ġoutput ) Ġthat Ġcontain Ġa Ġrepresentation Ġof Ġthe Ġdocument . ĠA Ġcontroller , Ġinitialized Ġwith Ġthe Ġencoding Ġof Ġthe Ġquestion , Ġis Ġused Ġto Ġcalculate Ġan Ġattention Ġbetween Ġthis Ġcontroller Ġand Ġthe Ġrepresentation Ġof Ġthe Ġdocument Ġin Ġthe Ġinput Ġmemory . ĠThis Ġattention Ġis Ġthem Ġused Ġto Ġre - weight Ġthe Ġrepresentation Ġof Ġthe Ġdocument Ġin Ġthe Ġoutput Ġmemory . ĠThis Ġresponse Ġfrom Ġthe Ġoutput Ġmemory Ġis Ġthem Ġutilized Ġto Ġupdate Ġthe Ġcontroller . ĠAfter Ġthat , Ġeither Ġa Ġmatrix Ġis Ġused Ġto Ġproject Ġthis Ġrepresentation Ġinto Ġthe Ġanswer Ġspace Ġeither Ġthe Ġcontroller Ġis Ġused Ġto Ġgo Ġthrough Ġan Ġover Ġhop Ġof Ġmemory . ĠThis Ġarchitecture Ġallows Ġthe Ġmodel Ġto Ġsequ entially Ġlook Ġinto Ġthe Ġinitial Ġdocument Ġseeking Ġfor Ġimportant Ġinformation Ġregarding Ġthe Ġcurrent Ġstate Ġof Ġits Ġcontroller . ĠThis Ġmodel Ġachieves Ġvery Ġgood Ġperformances Ġon Ġthe Ġ20 Ġb Ab I Ġtasks Ġdataset . </s> Deep Ġproject ive Ġreader : ĠThis Ġis Ġa Ġmodel Ġof Ġour Ġown Ġdesign , Ġlargely Ġinspired Ġby Ġthe Ġefficient ĠR - net Ġreader ĠB IB REF 12 Ġ. ĠThe Ġoverall Ġarchitecture Ġis Ġcomposed Ġof Ġ4 Ġstacked Ġlayers : Ġan Ġencoding Ġlayer , Ġa Ġquestion / document Ġattention , Ġa Ġself - att ention Ġlayer Ġand Ġa Ġprojection Ġlayer . ĠThe Ġfollowing Ġparagraphs Ġbriefly Ġdescribe Ġthe Ġoverall Ġutility Ġof Ġeach Ġof Ġthese Ġlayers . </s> Enc oding : ĠThe Ġsentence Ġis Ġtoken ized Ġby Ġwords . ĠEach Ġtoken Ġis Ġrepresented Ġby Ġthe Ġconc aten ation Ġof Ġits Ġembed ding Ġvector Ġand Ġthe Ġfinal Ġstate Ġof Ġa Ġbid irection al Ġrecurrent Ġnetwork Ġover Ġthe Ġcharacters Ġof Ġthis Ġword . ĠFinally , Ġanother Ġbid irection al ĠR NN Ġon Ġthe Ġtop Ġof Ġthis Ġrepresentation Ġproduce Ġthe Ġencoding Ġof Ġthe Ġdocument Ġand Ġthe Ġquestion . </s> Question / document Ġattention : ĠWe Ġapply Ġa Ġquestion / document Ġattention Ġlayer Ġthat Ġmatches Ġthe Ġrepresentation Ġof Ġthe Ġquestion Ġwith Ġeach Ġtoken Ġof Ġthe Ġdocument Ġindividually Ġto Ġoutput Ġan Ġattention Ġthat Ġgives Ġmore Ġweight Ġto Ġthe Ġimportant Ġtokens Ġof Ġthe Ġdocument Ġregarding Ġthe Ġquestion . </s> Self - att ention Ġlayer : ĠThe Ġprevious Ġlayer Ġhas Ġbuilt Ġa Ġquestion - aware Ġrepresentation Ġof Ġthe Ġdocument . ĠOne Ġproblem Ġwith Ġsuch Ġrepresentation Ġis Ġthat Ġform Ġthe Ġmoment Ġeach Ġtoken Ġhas Ġonly Ġa Ġgood Ġknowledge Ġof Ġits Ġclosest Ġneighbors . ĠTo Ġtackle Ġthis Ġproblem , ĠB IB REF 12 Ġhave Ġproposed Ġto Ġuse Ġa Ġself - att ention Ġlayer Ġthat Ġmatches Ġeach Ġindividual Ġtoken Ġwith Ġall Ġthe Ġother Ġtokens Ġof Ġthe Ġdocument . ĠDoing Ġthat , Ġeach Ġtoken Ġis Ġnow Ġaware Ġof Ġa Ġlarger Ġcontext . </s> Output Ġlayer : ĠA Ġbid irection al ĠR NN Ġis Ġapplied Ġon Ġthe Ġtop Ġof Ġthe Ġlast Ġlayer Ġand Ġwe Ġuse Ġits Ġfinal Ġstate Ġas Ġthe Ġrepresentation Ġof Ġthe Ġinput . ĠWe Ġuse Ġa Ġprojection Ġmatrix Ġto Ġproject Ġthis Ġrepresentation Ġinto Ġthe Ġanswer Ġspace Ġand Ġselect Ġthe Ġmost Ġprobable Ġone </s> Training Ġdetails </s> We Ġpropose Ġto Ġtrain Ġthese Ġmodels Ġon Ġthe Ġentire Ġset Ġof Ġtasks Ġand Ġthem Ġto Ġmeasure Ġthe Ġoverall Ġperformance Ġand Ġthe Ġaccuracy Ġof Ġeach Ġindividual Ġtask . ĠIn Ġall Ġthe Ġmodels , Ġwe Ġuse Ġthe ĠAdam Ġoptim izer ĠB IB REF 21 Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 01 Ġand Ġthe Ġbatch Ġsize Ġis Ġset Ġto Ġ64 . ĠAll Ġthe Ġparameter Ġare Ġinitialized Ġfrom Ġa ĠGa ussian Ġdistribution Ġwith Ġmean Ġ0 Ġand Ġa Ġstandard Ġdeviation Ġof Ġ0 . 01 . ĠThe Ġdimension Ġof Ġthe Ġword Ġembed d ings Ġin Ġthe Ġproject ive Ġdeep Ġreading Ġmodel Ġand Ġthe ĠL ST M Ġmodel Ġis Ġ300 Ġand Ġwe Ġuse ĠGl ove Ġpre - trained Ġvectors Ġ( ĠB IB REF 22 Ġ). ĠWe Ġuse Ġa ĠMem N 2 N Ġwith Ġ5 Ġmemory Ġhops Ġand Ġa Ġlinear Ġstart Ġof Ġ5 Ġepoch s . ĠThe Ġreviews Ġare Ġsplit Ġby Ġsentence Ġand Ġeach Ġmemory Ġblock Ġcorresponds Ġto Ġone Ġsentence . ĠEach Ġsentence Ġis Ġrepresented Ġby Ġits Ġbag - of - word Ġrepresentation Ġaugmented Ġwith Ġtemporal Ġencoding Ġas Ġit Ġis Ġsuggested Ġin ĠB IB REF 18 Ġ. </s> Model Ġperformance </s> Table ĠT AB REF 19 Ġdisplays Ġthe Ġperformance Ġof Ġthe Ġ4 Ġbas elines Ġon Ġthe ĠReview Q A 's Ġtest Ġset . ĠThese Ġresults Ġare Ġthe Ġperformance Ġachieved Ġby Ġour Ġown Ġimplementation Ġof Ġthese Ġ4 Ġmodels . ĠAccording Ġto Ġour Ġresults , Ġthe Ġsimple ĠL ST M Ġnetwork Ġand Ġthe ĠMem N 2 N Ġperform Ġvery Ġpoorly Ġon Ġthis Ġdataset . ĠEspecially Ġon Ġthe Ġmost Ġadvanced Ġreasoning Ġtasks . ĠIndeed , Ġthe Ġtask Ġ5 Ġwhich Ġcorresponds Ġto Ġthe Ġprediction Ġof Ġthe Ġexact Ġrating Ġof Ġan Ġaspect Ġseems Ġto Ġbe Ġvery Ġchallenging Ġfor Ġthese Ġmodel . ĠMaybe Ġthe Ġtoken ization Ġby Ġsentence Ġto Ġcreate Ġthe Ġmemory Ġblocks Ġof Ġthe ĠMem N 2 N , Ġwhich Ġis Ġappropriated Ġin Ġthe Ġcase Ġof Ġthe Ġb Ab I Ġtasks , Ġis Ġnot Ġa Ġgood Ġrepresentation Ġof Ġthe Ġdocuments Ġwhen Ġit Ġhas Ġto Ġhandle Ġhuman Ġgenerated Ġcomments . ĠHowever , Ġthe Ġlog istic Ġregression Ġachieves Ġreasonable Ġperformance Ġon Ġthese Ġtasks , Ġand Ġdo Ġnot Ġsuffer Ġfrom Ġcatastrophic Ġperformance Ġon Ġany Ġtasks . ĠIts Ġworst Ġresult Ġcomes Ġon Ġtask Ġ6 Ġand Ġone Ġof Ġthe Ġreason Ġis Ġprobably Ġthat Ġthis Ġarchitecture Ġis Ġnot Ġdesigned Ġto Ġpredict Ġa Ġlist Ġof Ġanswers . ĠOn Ġthe Ġcontrary , Ġthe Ġdeep Ġproject ive Ġreader Ġachieves Ġencouraging Ġon Ġthis Ġdataset . ĠIt Ġoutper forms Ġall Ġthe Ġother Ġbas elines , Ġwith Ġvery Ġgood Ġscores Ġon Ġthe Ġfirst Ġfourth Ġtasks . ĠThe Ġquestion / document Ġand Ġdocument / document Ġattention Ġlayers Ġproposed Ġin ĠB IB REF 12 Ġseem Ġonce Ġagain Ġto Ġproduce Ġrich Ġenc od ings Ġof Ġthe Ġinputs Ġwhich Ġare Ġrelevant Ġfor Ġour Ġprojection Ġlayer . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġformal ize Ġthe Ġsentiment Ġanalysis Ġtask Ġthrough Ġthe Ġframework Ġof Ġmachine Ġreading Ġand Ġrelease ĠReview Q A , Ġa Ġrelational Ġquestion - ans w ering Ġcorpus . ĠThis Ġdataset Ġallows Ġevaluating Ġa Ġset Ġof Ġrelational Ġreasoning Ġskills Ġthrough Ġnatural Ġlanguage Ġquestions . ĠIt Ġis Ġcomposed Ġof Ġa Ġlarge Ġpanel Ġof Ġhuman - generated Ġquestions . ĠMoreover , Ġwe Ġpropose Ġto Ġaugment Ġthe Ġdataset Ġwith Ġback trans lated Ġreform ulations Ġof Ġthese Ġquestions . ĠFinally , Ġwe Ġevaluate Ġ4 Ġmodels Ġon Ġthis Ġdataset , Ġincluding Ġa Ġproject ive Ġmodel Ġof Ġour Ġown Ġdesign Ġthat Ġseems Ġto Ġbe Ġa Ġstrong Ġbaseline Ġfor Ġthis Ġdataset . ĠWe Ġexpect Ġthat Ġthis Ġlarge Ġdataset Ġwill Ġencourage Ġthe Ġresearch Ġcommunity Ġto Ġdevelop Ġreasoning Ġmodels Ġand Ġevaluate Ġtheir Ġmodels Ġon Ġthis Ġset Ġof Ġtasks . </s> Acknowled gment </s> We Ġthank ĠV ass il ina ĠNik oul ina Ġand ĠSt Ã© ph ane ĠCl inch ant Ġfor Ġthe Ġhelp Ġregarding Ġthe Ġback translation Ġre w ording Ġof Ġthe Ġquestions .
Introduction A large majority of the human knowledge is recorded through text documents. That is why ability for a system to automatically infer information from text without any structured data has become a major challenge. Answering questions about a given document is a relevant proxy task that has been proposed as a way to evaluate the reading ability of a given model. In this configuration, a text document such as a news article, a document from Wikipedia or any type of text is presented to a machine with an associated set of questions. The system is then expected to answer these questions and evaluated by its accuracy on this task. The machine reading framework is very general and we can imagine a large panel of questions that can possibly handle most of the standard natural language processing tasks. For example, the task of named entities recognition can be formulated as a machine reading one where your document is the sentence and the question would be 'What are the named entities mentioned in this sentence?'. These natural language interactions are an important objective for reading systems. Recently, many datasets have been proposed to build and evaluate reading models BIBREF0 , BIBREF1 . From cloze style questions BIBREF2 to open questions BIBREF3 , from synthetic data BIBREF4 to human written articles BIBREF5 , many styles of documents and questions have been proposed to challenge reading models. The correct answer to the questions proposed in most of these datasets is a span of text of the source document, which can be restricted to a single word in several cases. It means that the answer should explicitly be present in the source document and that the model should be able to locate it. Different models have already shown superhuman performance on several of these datasets and particularly on the SQuAD dataset composed of Wikipedia articles BIBREF6 , BIBREF7 . However, some limits of such models have been highlighted when they encounter perturbations into the input documents BIBREF8 . Indeed almost all of the state of the art models on the SQuAD dataset suffer from a lack of robustness against adversarial examples. Once the model is trained, a meaningless sentence added at the end of the text document can completely disturb the reading system. Conversely, these adversarial examples do not seem to fool a human reader who will be capable of answering the questions as well as without this perturbation. One possible explanation of this phenomenon is that computers are good at extracting patterns in the document that match the representation of the question. If multiple spans of the documents look similar to the questions, the reader might not be able to decide which one is relevant. Moreover, Wikipedia articles tend to be written with the same standard writing style, factual, unambiguous. Such writing style tends to favor the pattern matching between the questions and the documents. This format of documents/questions has certainly influenced the design of the comprehension models that have been proposed so far. Most of them are composed of stacked attention layers that match question and document representations. Following concepts proposed in the 20 bAbI tasks BIBREF4 or in the visual question-answering dataset CLEVR BIBREF9 , we think that the challenge, limited to the detection of relevant passages in a document, is only the first step in building systems that truly understand text. The second step is the ability of reasoning with the relevant information extracted from a document. To set up this challenge, we propose to leverage on a hotel reviews corpus that requires reasoning skills to answer natural language questions. The reviews we used have been extracted from TripAdvisor and originally proposed in BIBREF10 , BIBREF11 . In the original data, each review comes with a set of rated aspects among the seventh available: Business service, Check in / Front Desk, Cleanliness, Location, Room, Sleep Quality, Value and for all the reviews an Overall rating. In this articles we propose to exploit these data to create a dataset of question-answering that will challenge 8 competencies of the reader. Our contributions can be summarized as follow: Machine comprehension datasets ReviewQA is proposed as a novel dataset regarding the collection of the existing ones. Indeed a large panel of available datasets, that evaluate models on different types of documents, can only be valuable for designing efficient models and learning protocols. In this following part, we describe several of these datasets. SQuAD: The Standford Question Answering Dataset (SQuAD) introduced in BIBREF0 is a large dataset of natural questions over the 500 most popular articles of Wikipedia. All the questions have been crowdsourced and answers are spans of text extracted from source documents. This dataset has been very popular these last two years and the performance of the architectures that have been proposed have rapidly increased until several models surpass the human score. Indeed, in the original paper human performance has been measured at 82.304 points for the exact match metric and at the time we are writing this paper four models have already a higher score. In another hand BIBREF8 has shown that these models suffer from a lack of robustness against adversarial examples that are meaningless from a human point of view. This suggests the need for a more challenging dataset that will allow developing strongest reasoning architectures. NewsQA: NewsQA BIBREF1 is a dataset very similar to SQuAD. It contains 120.000 human generated questions over 12.000 articles form CNN originally introduced in BIBREF5 . It has been designed to be more challenging than SQuAD with questions that might require to extract multiple spans of text or not be answerable. WikiHop and MedHop: These are two recent datasets introduced in BIBREF13 . Unlike SQuAD and NewsQA, important facts are spread out across multiple documents and, in order to answer a question, it is necessary to jump over a set of passages to collect the required information. The relevant passages are not explicitly mentioned in the data so this dataset measures the ability that a model has to navigate across multiple documents. The questions come with a set of candidates which are all present in the text. MS Marco: This dataset has been released in BIBREF14 . The documents come from the internet and the questions are real user queries asked through the bing search engine. The dataset contains around 100.000 queries and each of them comes with a set of approximatively 10 relevant passages. Like in SQuAD, several models are already doing superhuman performances on this dataset. Facebook bAbI tasks: This is a set of 20 toy tasks proposed in BIBREF4 and designed to measure text understanding. Each task requires a certain capability to be completed like induction, deduction and more. Documents are synthetic stories, composed of few sentences that describe a set of actions. This dataset was one of the first attempt to introduce a general set of prerequisite capabilities required for the reading task. Although it has been a very challenging framework, beneficial to the emergence of the attention mechanism inside the reading architectures, a Gated end-to-end memory network BIBREF15 now succeed in almost all of the 20 tasks. One of the possible reason is that the data are synthetic data, without noise or ambiguity. We propose a comparable framework with understanding and reasoning tasks based on user-generated comments that are much more realistic and that required language competencies to be understood. CLEVR: Beyond textual question-answering, Visual Question-Answering (VQA) has been largely studied during the last couple of years. More recently, the problem of relational reasoning has been introduced through this dataset BIBREF9 . The main original idea was to introduce relational reasoning questions over object shapes and placements. This dataset has already motivated the development of original deep models. To the best of our knowledge, no natural language question-answering corpus has been designed to investigate such capabilities. As we will present in the following of this paper, we think sentiment analysis is particularly suited for this task and we will introduce a novel machine reading corpus with such capability requirements. Attention-based models for aspect-based sentiment analysis Sentiment analysis is one of the historical tasks of Natural Language Processing. It is an important challenge for companies, restaurants, hotels that aim to analyze customer satisfaction regarding products and quality of services. Given a text document, the objective is to predict its overall polarity. Generally, it can be positive, negative or neutral. This analysis gives a quick overview of a general sentiment over a set of documents, but this framework tends to be restrictive. Indeed, one document tends to express multiple opinions of different aspects. For instance, in the sentence: The fish was very good but the service was terrible, there is not a general dominant sentiment, and a finer analysis is needed. The task of aspect-based sentiment analysis aims to predict a polarity of a sentence regarding a given aspect. In the previous example a positive polarity should be associated to the aspect food, and on the contrary, a negative sentiment is expressed regarding the quality of the service. The idea of using models originally designed for question-answering, for the sentiment analysis task has been introduced in BIBREF16 , BIBREF17 . In these papers, several adaptations of the end-to-end memory network (MemN2N) BIBREF18 are used to predict the polarity of a review regarding a given aspect. In that configuration, the review is encoded into the memory cells and the controller, usually initialized with a representation of the question, is initialized with a representation of the aspect. The analysis of the attention between the values of the controller and the document has shown interesting results, by highlighting relevant part of a document regarding an aspect. ReviewQA dataset We think that evaluating the task of sentiment analysis through the setup of question-answering is a relevant playground for machine reading research. Indeed natural language questions about the different aspects of the targeted venues are typical kind of questions we want to be able to ask to a system. In this context, we introduce a set of reasoning questions types over the relationships between aspects. We propose ReviewQA, a dataset of natural language questions over hotel reviews. These questions are divided into 8 groups, regarding the competency required to be answered. In this section, we describe each task and the process followed to generate this dataset. Original data We used a set of reviews extracted from the TripAdvisor website and originally proposed in BIBREF10 and BIBREF11 . This corpus is available at http://www.cs.virginia.edu/~hw5x/Data/LARA/TripAdvisor/TripAdvisorJson.tar.bz2. Each review comes with the name of the associated hotel, a title, an overall rating, a comment and a list of rated aspects. From 0 to 7 aspects, among value, room, location, cleanliness, check-in/front desk, service, business service, can possibly be rated in a review. Figure FIGREF8 displays a review extracted from this dataset. Relational reasoning competencies Objective: Starting with the original corpus, we aim at building a machine reading task where natural language questions will challenge the model on its understanding of the reviews. Indeed learning relational reasoning competencies over natural language documents is a major challenge of the current reading models. These original raw data allow us to generate relational questions that can possibly require a global understanding of the comment and reasoning skills to be treated. For example, asking a question like What is the best aspect rated in this comment ? is not an easy question that can be answered without a deep understanding of the review. It is necessary to capture all the aspects mentioned in the text, to predict their rating and finally to select the best one. The tasks and the dataset we propose are publicly available at http://www.europe.naverlabs.com/Blog/ReviewQA-A-novel-relational-aspect-based-opinion-dataset-for-machine-reading We introduce a list of 8 different competencies that a reading system should master in order to process reviews and text documents in general. These 8 tasks require different competencies and a different level of understanding of the document to be well answered. For instance, detecting if an aspect is mentioned in a review will require less understanding of the review than predicting explicitly the rating of this aspect. Table TABREF10 presents the 8 tasks we have introduced in this dataset with an example of a question that corresponds to each task. We also provide the expected type of the answer (Yes/No question, rating question...). It can be an additional tool to analyze the errors of the readers. Construction of the dataset We sample 100.000 reviews from the original corpus. Figure FIGREF12 presents the distribution of the number of words of the reviews in the dataset. We explicitly favor reviews which contain an important number of words. In average, a review contains 200 words. Indeed these long reviews are most likely to contain challenging relations between different aspects. A short review which deals with only a few aspects is more likely to not be very relevant to the challenge we want to propose in this dataset. Figure FIGREF14 displays the distribution of the ratings per aspects in the 100.000 reviews we based our dataset. We can see that the average values of these ratings tend to be quite high. It could have introduced bias if it was not the case for all the aspects. For example, we do not want that the model learns that in general, the service is rated better than the location and them answer without looking at the document. Since this situation is the same for all the aspects, the relational tasks introduced in this dataset remains extremely relevant. Then we randomly select 6 tasks for each review (the same task can be selected multiple times) and randomly select a natural language question that corresponds to this task. The questions are human-generated patterns that we have crowdsourced in order to produce a dataset as rich as possible. To this end, we have generated several patterns that correspond to the capabilities we wanted to express in a given question and we have crowdsourced rephrasing of these patterns. The final dataset we propose is composed of more than 500.000 questions about 100.000 reviews. Table TABREF13 shows the repartition of the documents and queries into the train and test set. Each review contains a maximum of 6 questions. Sometimes less when it is not possible to generate all. For example, if only two or three aspects are mentioned in a review, we will be able to generate only a little set of relational questions. Figure FIGREF15 depicts the repartition of the answers in the generated dataset. A majority of the tasks we introduced, even if they possibly require a high level of understanding of the document and the question, are binary questions. It means that in the generated dataset the answers yes and no tend to be more present than the others. To balance in a better way the distribution of the answers, we chose to affect a higher probability of sampling to the task 5, 6, 7.1, 8. Indeed, these tasks are not binary questions and required an aspect name as the answer. Figure FIGREF17 represents the repartition of question types in our dataset. Finally, figure FIGREF15 shows the repartition of the answers in the dataset. Paraphrase augmentation using backtranslation In order to generate more paraphrases of the questions, we used a backtranslation method to enrich them. The idea is to use a translation model that will translate our human-generated questions into another language, and then translate them back to English. This double translation will introduce rewordings of the questions that we will be able to integrate into this dataset. This approach has been used in BIBREF7 to perform data augmentation on the training set. For this purpose, we have trained a fairseq BIBREF19 model to translate sentences from English to French and for French to English. In order to preserve the quality of the sentences we have so far, we only keep the most probable translation of each original sentence. Indeed a beam search is used during the translation to predict the most probable translations which mean that we each translation comes with an associated probability. By selecting only the first translations, we almost double the number of questions without degrading the quality of the questions proposed in the dataset. Models In this section, we present the performance of four different models on our dataset: a logistic regression and three neural models. The first one is a basic LSTM BIBREF20 , the second a MemN2N BIBREF18 and the third one is a model of our own design. This fourth model reuses the encoding layers of the R-net BIBREF12 and we modify the final layers with a projection layer that will be able to select the answer among the set of candidates instead of pointing the answerer directly into the source document. Logistic regression: To produce the representation of the input, we concatenate the Bag-Of-Words representation of the document with the Bag-Of-Words representation of the question. It produces an array of size INLINEFORM0 where INLINEFORM1 is the vocabulary size. Then we use a logistic regression to select the most probable answer among the INLINEFORM2 possibilities. LSTM: We start with a concatenation of the sequence of indexes of the document with the sequence of indexes of the question. Them we feed an LSTM network with this vector and use the final state as the representation of the input. Finally, we apply a logistic regression over this representation to produce the final decision. End-to-end memory networks: This architecture is based on two different memory cells (input and output) that contain a representation of the document. A controller, initialized with the encoding of the question, is used to calculate an attention between this controller and the representation of the document in the input memory. This attention is them used to re-weight the representation of the document in the output memory. This response from the output memory is them utilized to update the controller. After that, either a matrix is used to project this representation into the answer space either the controller is used to go through an over hop of memory. This architecture allows the model to sequentially look into the initial document seeking for important information regarding the current state of its controller. This model achieves very good performances on the 20 bAbI tasks dataset. Deep projective reader: This is a model of our own design, largely inspired by the efficient R-net reader BIBREF12 . The overall architecture is composed of 4 stacked layers: an encoding layer, a question/document attention, a self-attention layer and a projection layer. The following paragraphs briefly describe the overall utility of each of these layers. Encoding: The sentence is tokenized by words. Each token is represented by the concatenation of its embedding vector and the final state of a bidirectional recurrent network over the characters of this word. Finally, another bidirectional RNN on the top of this representation produce the encoding of the document and the question. Question/document attention: We apply a question/document attention layer that matches the representation of the question with each token of the document individually to output an attention that gives more weight to the important tokens of the document regarding the question. Self-attention layer: The previous layer has built a question-aware representation of the document. One problem with such representation is that form the moment each token has only a good knowledge of its closest neighbors. To tackle this problem, BIBREF12 have proposed to use a self-attention layer that matches each individual token with all the other tokens of the document. Doing that, each token is now aware of a larger context. Output layer: A bidirectional RNN is applied on the top of the last layer and we use its final state as the representation of the input. We use a projection matrix to project this representation into the answer space and select the most probable one Training details We propose to train these models on the entire set of tasks and them to measure the overall performance and the accuracy of each individual task. In all the models, we use the Adam optimizer BIBREF21 with a learning rate of 0.01 and the batch size is set to 64. All the parameter are initialized from a Gaussian distribution with mean 0 and a standard deviation of 0.01. The dimension of the word embeddings in the projective deep reading model and the LSTM model is 300 and we use Glove pre-trained vectors ( BIBREF22 ). We use a MemN2N with 5 memory hops and a linear start of 5 epochs. The reviews are split by sentence and each memory block corresponds to one sentence. Each sentence is represented by its bag-of-word representation augmented with temporal encoding as it is suggested in BIBREF18 . Model performance Table TABREF19 displays the performance of the 4 baselines on the ReviewQA's test set. These results are the performance achieved by our own implementation of these 4 models. According to our results, the simple LSTM network and the MemN2N perform very poorly on this dataset. Especially on the most advanced reasoning tasks. Indeed, the task 5 which corresponds to the prediction of the exact rating of an aspect seems to be very challenging for these model. Maybe the tokenization by sentence to create the memory blocks of the MemN2N, which is appropriated in the case of the bAbI tasks, is not a good representation of the documents when it has to handle human generated comments. However, the logistic regression achieves reasonable performance on these tasks, and do not suffer from catastrophic performance on any tasks. Its worst result comes on task 6 and one of the reason is probably that this architecture is not designed to predict a list of answers. On the contrary, the deep projective reader achieves encouraging on this dataset. It outperforms all the other baselines, with very good scores on the first fourth tasks. The question/document and document/document attention layers proposed in BIBREF12 seem once again to produce rich encodings of the inputs which are relevant for our projection layer. Conclusion In this paper, we formalize the sentiment analysis task through the framework of machine reading and release ReviewQA, a relational question-answering corpus. This dataset allows evaluating a set of relational reasoning skills through natural language questions. It is composed of a large panel of human-generated questions. Moreover, we propose to augment the dataset with backtranslated reformulations of these questions. Finally, we evaluate 4 models on this dataset, including a projective model of our own design that seems to be a strong baseline for this dataset. We expect that this large dataset will encourage the research community to develop reasoning models and evaluate their models on this set of tasks. Acknowledgment We thank Vassilina Nikoulina and Stéphane Clinchant for the help regarding the backtranslation rewording of the questions.
What language are the reviews in?
2236386729105f5cf42f73cc055ce3acdea2d452
What Ġlanguage Ġare Ġthe Ġreviews Ġin ?
Un answer able
<s> What Ġlanguage Ġare Ġthe Ġreviews Ġin ? </s> Introduction </s> A Ġlarge Ġmajority Ġof Ġthe Ġhuman Ġknowledge Ġis Ġrecorded Ġthrough Ġtext Ġdocuments . ĠThat Ġis Ġwhy Ġability Ġfor Ġa Ġsystem Ġto Ġautomatically Ġinfer Ġinformation Ġfrom Ġtext Ġwithout Ġany Ġstructured Ġdata Ġhas Ġbecome Ġa Ġmajor Ġchallenge . ĠAn sw ering Ġquestions Ġabout Ġa Ġgiven Ġdocument Ġis Ġa Ġrelevant Ġproxy Ġtask Ġthat Ġhas Ġbeen Ġproposed Ġas Ġa Ġway Ġto Ġevaluate Ġthe Ġreading Ġability Ġof Ġa Ġgiven Ġmodel . ĠIn Ġthis Ġconfiguration , Ġa Ġtext Ġdocument Ġsuch Ġas Ġa Ġnews Ġarticle , Ġa Ġdocument Ġfrom ĠWikipedia Ġor Ġany Ġtype Ġof Ġtext Ġis Ġpresented Ġto Ġa Ġmachine Ġwith Ġan Ġassociated Ġset Ġof Ġquestions . ĠThe Ġsystem Ġis Ġthen Ġexpected Ġto Ġanswer Ġthese Ġquestions Ġand Ġevaluated Ġby Ġits Ġaccuracy Ġon Ġthis Ġtask . ĠThe Ġmachine Ġreading Ġframework Ġis Ġvery Ġgeneral Ġand Ġwe Ġcan Ġimagine Ġa Ġlarge Ġpanel Ġof Ġquestions Ġthat Ġcan Ġpossibly Ġhandle Ġmost Ġof Ġthe Ġstandard Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠFor Ġexample , Ġthe Ġtask Ġof Ġnamed Ġentities Ġrecognition Ġcan Ġbe Ġformulated Ġas Ġa Ġmachine Ġreading Ġone Ġwhere Ġyour Ġdocument Ġis Ġthe Ġsentence Ġand Ġthe Ġquestion Ġwould Ġbe Ġ' What Ġare Ġthe Ġnamed Ġentities Ġmentioned Ġin Ġthis Ġsentence ? '. ĠThese Ġnatural Ġlanguage Ġinteractions Ġare Ġan Ġimportant Ġobjective Ġfor Ġreading Ġsystems . </s> Recently , Ġmany Ġdatasets Ġhave Ġbeen Ġproposed Ġto Ġbuild Ġand Ġevaluate Ġreading Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠFrom Ġclo ze Ġstyle Ġquestions ĠB IB REF 2 Ġto Ġopen Ġquestions ĠB IB REF 3 Ġ, Ġfrom Ġsynthetic Ġdata ĠB IB REF 4 Ġto Ġhuman Ġwritten Ġarticles ĠB IB REF 5 Ġ, Ġmany Ġstyles Ġof Ġdocuments Ġand Ġquestions Ġhave Ġbeen Ġproposed Ġto Ġchallenge Ġreading Ġmodels . ĠThe Ġcorrect Ġanswer Ġto Ġthe Ġquestions Ġproposed Ġin Ġmost Ġof Ġthese Ġdatasets Ġis Ġa Ġspan Ġof Ġtext Ġof Ġthe Ġsource Ġdocument , Ġwhich Ġcan Ġbe Ġrestricted Ġto Ġa Ġsingle Ġword Ġin Ġseveral Ġcases . ĠIt Ġmeans Ġthat Ġthe Ġanswer Ġshould Ġexplicitly Ġbe Ġpresent Ġin Ġthe Ġsource Ġdocument Ġand Ġthat Ġthe Ġmodel Ġshould Ġbe Ġable Ġto Ġlocate Ġit . </s> Different Ġmodels Ġhave Ġalready Ġshown Ġsuperhuman Ġperformance Ġon Ġseveral Ġof Ġthese Ġdatasets Ġand Ġparticularly Ġon Ġthe ĠS Qu AD Ġdataset Ġcomposed Ġof ĠWikipedia Ġarticles ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġsome Ġlimits Ġof Ġsuch Ġmodels Ġhave Ġbeen Ġhighlighted Ġwhen Ġthey Ġencounter Ġpert urb ations Ġinto Ġthe Ġinput Ġdocuments ĠB IB REF 8 Ġ. ĠIndeed Ġalmost Ġall Ġof Ġthe Ġstate Ġof Ġthe Ġart Ġmodels Ġon Ġthe ĠS Qu AD Ġdataset Ġsuffer Ġfrom Ġa Ġlack Ġof Ġrobust ness Ġagainst Ġadvers arial Ġexamples . ĠOnce Ġthe Ġmodel Ġis Ġtrained , Ġa Ġmeaningless Ġsentence Ġadded Ġat Ġthe Ġend Ġof Ġthe Ġtext Ġdocument Ġcan Ġcompletely Ġdisturb Ġthe Ġreading Ġsystem . ĠConversely , Ġthese Ġadvers arial Ġexamples Ġdo Ġnot Ġseem Ġto Ġfool Ġa Ġhuman Ġreader Ġwho Ġwill Ġbe Ġcapable Ġof Ġanswering Ġthe Ġquestions Ġas Ġwell Ġas Ġwithout Ġthis Ġpert urb ation . ĠOne Ġpossible Ġexplanation Ġof Ġthis Ġphenomenon Ġis Ġthat Ġcomputers Ġare Ġgood Ġat Ġextracting Ġpatterns Ġin Ġthe Ġdocument Ġthat Ġmatch Ġthe Ġrepresentation Ġof Ġthe Ġquestion . ĠIf Ġmultiple Ġspans Ġof Ġthe Ġdocuments Ġlook Ġsimilar Ġto Ġthe Ġquestions , Ġthe Ġreader Ġmight Ġnot Ġbe Ġable Ġto Ġdecide Ġwhich Ġone Ġis Ġrelevant . ĠMoreover , ĠWikipedia Ġarticles Ġtend Ġto Ġbe Ġwritten Ġwith Ġthe Ġsame Ġstandard Ġwriting Ġstyle , Ġfactual , Ġunamb iguous . ĠSuch Ġwriting Ġstyle Ġtends Ġto Ġfavor Ġthe Ġpattern Ġmatching Ġbetween Ġthe Ġquestions Ġand Ġthe Ġdocuments . ĠThis Ġformat Ġof Ġdocuments / quest ions Ġhas Ġcertainly Ġinfluenced Ġthe Ġdesign Ġof Ġthe Ġcomprehension Ġmodels Ġthat Ġhave Ġbeen Ġproposed Ġso Ġfar . ĠMost Ġof Ġthem Ġare Ġcomposed Ġof Ġstacked Ġattention Ġlayers Ġthat Ġmatch Ġquestion Ġand Ġdocument Ġrepresentations . </s> Following Ġconcepts Ġproposed Ġin Ġthe Ġ20 Ġb Ab I Ġtasks ĠB IB REF 4 Ġor Ġin Ġthe Ġvisual Ġquestion - ans w ering Ġdataset ĠCLE VR ĠB IB REF 9 Ġ, Ġwe Ġthink Ġthat Ġthe Ġchallenge , Ġlimited Ġto Ġthe Ġdetection Ġof Ġrelevant Ġpassages Ġin Ġa Ġdocument , Ġis Ġonly Ġthe Ġfirst Ġstep Ġin Ġbuilding Ġsystems Ġthat Ġtruly Ġunderstand Ġtext . ĠThe Ġsecond Ġstep Ġis Ġthe Ġability Ġof Ġreasoning Ġwith Ġthe Ġrelevant Ġinformation Ġextracted Ġfrom Ġa Ġdocument . ĠTo Ġset Ġup Ġthis Ġchallenge , Ġwe Ġpropose Ġto Ġleverage Ġon Ġa Ġhotel Ġreviews Ġcorpus Ġthat Ġrequires Ġreasoning Ġskills Ġto Ġanswer Ġnatural Ġlanguage Ġquestions . ĠThe Ġreviews Ġwe Ġused Ġhave Ġbeen Ġextracted Ġfrom ĠTrip Ad visor Ġand Ġoriginally Ġproposed Ġin ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠIn Ġthe Ġoriginal Ġdata , Ġeach Ġreview Ġcomes Ġwith Ġa Ġset Ġof Ġrated Ġaspects Ġamong Ġthe Ġseventh Ġavailable : ĠBusiness Ġservice , ĠCheck Ġin Ġ/ ĠFront ĠDesk , ĠClean liness , ĠLocation , ĠRoom , ĠSleep ĠQuality , ĠValue Ġand Ġfor Ġall Ġthe Ġreviews Ġan ĠOverall Ġrating . ĠIn Ġthis Ġarticles Ġwe Ġpropose Ġto Ġexploit Ġthese Ġdata Ġto Ġcreate Ġa Ġdataset Ġof Ġquestion - ans w ering Ġthat Ġwill Ġchallenge Ġ8 Ġcompet encies Ġof Ġthe Ġreader . </s> Our Ġcontributions Ġcan Ġbe Ġsummarized Ġas Ġfollow : </s> Machine Ġcomprehension Ġdatasets </s> Review Q A Ġis Ġproposed Ġas Ġa Ġnovel Ġdataset Ġregarding Ġthe Ġcollection Ġof Ġthe Ġexisting Ġones . ĠIndeed Ġa Ġlarge Ġpanel Ġof Ġavailable Ġdatasets , Ġthat Ġevaluate Ġmodels Ġon Ġdifferent Ġtypes Ġof Ġdocuments , Ġcan Ġonly Ġbe Ġvaluable Ġfor Ġdesigning Ġefficient Ġmodels Ġand Ġlearning Ġprotocols . ĠIn Ġthis Ġfollowing Ġpart , Ġwe Ġdescribe Ġseveral Ġof Ġthese Ġdatasets . </s> S Qu AD : ĠThe ĠStand ford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD ) Ġintroduced Ġin ĠB IB REF 0 Ġis Ġa Ġlarge Ġdataset Ġof Ġnatural Ġquestions Ġover Ġthe Ġ500 Ġmost Ġpopular Ġarticles Ġof ĠWikipedia . ĠAll Ġthe Ġquestions Ġhave Ġbeen Ġcrowds ourced Ġand Ġanswers Ġare Ġspans Ġof Ġtext Ġextracted Ġfrom Ġsource Ġdocuments . ĠThis Ġdataset Ġhas Ġbeen Ġvery Ġpopular Ġthese Ġlast Ġtwo Ġyears Ġand Ġthe Ġperformance Ġof Ġthe Ġarchitectures Ġthat Ġhave Ġbeen Ġproposed Ġhave Ġrapidly Ġincreased Ġuntil Ġseveral Ġmodels Ġsurpass Ġthe Ġhuman Ġscore . ĠIndeed , Ġin Ġthe Ġoriginal Ġpaper Ġhuman Ġperformance Ġhas Ġbeen Ġmeasured Ġat Ġ82 . 304 Ġpoints Ġfor Ġthe Ġexact Ġmatch Ġmetric Ġand Ġat Ġthe Ġtime Ġwe Ġare Ġwriting Ġthis Ġpaper Ġfour Ġmodels Ġhave Ġalready Ġa Ġhigher Ġscore . ĠIn Ġanother Ġhand ĠB IB REF 8 Ġhas Ġshown Ġthat Ġthese Ġmodels Ġsuffer Ġfrom Ġa Ġlack Ġof Ġrobust ness Ġagainst Ġadvers arial Ġexamples Ġthat Ġare Ġmeaningless Ġfrom Ġa Ġhuman Ġpoint Ġof Ġview . ĠThis Ġsuggests Ġthe Ġneed Ġfor Ġa Ġmore Ġchallenging Ġdataset Ġthat Ġwill Ġallow Ġdeveloping Ġstrongest Ġreasoning Ġarchitectures . </s> News Q A : ĠNews Q A ĠB IB REF 1 Ġis Ġa Ġdataset Ġvery Ġsimilar Ġto ĠS Qu AD . ĠIt Ġcontains Ġ120 . 000 Ġhuman Ġgenerated Ġquestions Ġover Ġ12 . 000 Ġarticles Ġform ĠCNN Ġoriginally Ġintroduced Ġin ĠB IB REF 5 Ġ. ĠIt Ġhas Ġbeen Ġdesigned Ġto Ġbe Ġmore Ġchallenging Ġthan ĠS Qu AD Ġwith Ġquestions Ġthat Ġmight Ġrequire Ġto Ġextract Ġmultiple Ġspans Ġof Ġtext Ġor Ġnot Ġbe Ġanswer able . </s> Wiki Hop Ġand ĠMed Hop : ĠThese Ġare Ġtwo Ġrecent Ġdatasets Ġintroduced Ġin ĠB IB REF 13 Ġ. ĠUnlike ĠS Qu AD Ġand ĠNews Q A , Ġimportant Ġfacts Ġare Ġspread Ġout Ġacross Ġmultiple Ġdocuments Ġand , Ġin Ġorder Ġto Ġanswer Ġa Ġquestion , Ġit Ġis Ġnecessary Ġto Ġjump Ġover Ġa Ġset Ġof Ġpassages Ġto Ġcollect Ġthe Ġrequired Ġinformation . ĠThe Ġrelevant Ġpassages Ġare Ġnot Ġexplicitly Ġmentioned Ġin Ġthe Ġdata Ġso Ġthis Ġdataset Ġmeasures Ġthe Ġability Ġthat Ġa Ġmodel Ġhas Ġto Ġnavigate Ġacross Ġmultiple Ġdocuments . ĠThe Ġquestions Ġcome Ġwith Ġa Ġset Ġof Ġcandidates Ġwhich Ġare Ġall Ġpresent Ġin Ġthe Ġtext . </s> MS ĠMarco : ĠThis Ġdataset Ġhas Ġbeen Ġreleased Ġin ĠB IB REF 14 Ġ. ĠThe Ġdocuments Ġcome Ġfrom Ġthe Ġinternet Ġand Ġthe Ġquestions Ġare Ġreal Ġuser Ġqueries Ġasked Ġthrough Ġthe Ġb ing Ġsearch Ġengine . ĠThe Ġdataset Ġcontains Ġaround Ġ100 . 000 Ġqueries Ġand Ġeach Ġof Ġthem Ġcomes Ġwith Ġa Ġset Ġof Ġapprox im atively Ġ10 Ġrelevant Ġpassages . ĠLike Ġin ĠS Qu AD , Ġseveral Ġmodels Ġare Ġalready Ġdoing Ġsuperhuman Ġperformances Ġon Ġthis Ġdataset . </s> Facebook Ġb Ab I Ġtasks : ĠThis Ġis Ġa Ġset Ġof Ġ20 Ġtoy Ġtasks Ġproposed Ġin ĠB IB REF 4 Ġand Ġdesigned Ġto Ġmeasure Ġtext Ġunderstanding . ĠEach Ġtask Ġrequires Ġa Ġcertain Ġcapability Ġto Ġbe Ġcompleted Ġlike Ġinduction , Ġdeduction Ġand Ġmore . ĠDocuments Ġare Ġsynthetic Ġstories , Ġcomposed Ġof Ġfew Ġsentences Ġthat Ġdescribe Ġa Ġset Ġof Ġactions . ĠThis Ġdataset Ġwas Ġone Ġof Ġthe Ġfirst Ġattempt Ġto Ġintroduce Ġa Ġgeneral Ġset Ġof Ġprerequisite Ġcapabilities Ġrequired Ġfor Ġthe Ġreading Ġtask . ĠAlthough Ġit Ġhas Ġbeen Ġa Ġvery Ġchallenging Ġframework , Ġbeneficial Ġto Ġthe Ġemergence Ġof Ġthe Ġattention Ġmechanism Ġinside Ġthe Ġreading Ġarchitectures , Ġa ĠG ated Ġend - to - end Ġmemory Ġnetwork ĠB IB REF 15 Ġnow Ġsucceed Ġin Ġalmost Ġall Ġof Ġthe Ġ20 Ġtasks . ĠOne Ġof Ġthe Ġpossible Ġreason Ġis Ġthat Ġthe Ġdata Ġare Ġsynthetic Ġdata , Ġwithout Ġnoise Ġor Ġambiguity . ĠWe Ġpropose Ġa Ġcomparable Ġframework Ġwith Ġunderstanding Ġand Ġreasoning Ġtasks Ġbased Ġon Ġuser - generated Ġcomments Ġthat Ġare Ġmuch Ġmore Ġrealistic Ġand Ġthat Ġrequired Ġlanguage Ġcompet encies Ġto Ġbe Ġunderstood . </s> CLE VR : ĠBeyond Ġtextual Ġquestion - ans w ering , ĠVisual ĠQuestion - An sw ering Ġ( V Q A ) Ġhas Ġbeen Ġlargely Ġstudied Ġduring Ġthe Ġlast Ġcouple Ġof Ġyears . ĠMore Ġrecently , Ġthe Ġproblem Ġof Ġrelational Ġreasoning Ġhas Ġbeen Ġintroduced Ġthrough Ġthis Ġdataset ĠB IB REF 9 Ġ. ĠThe Ġmain Ġoriginal Ġidea Ġwas Ġto Ġintroduce Ġrelational Ġreasoning Ġquestions Ġover Ġobject Ġshapes Ġand Ġplac ements . ĠThis Ġdataset Ġhas Ġalready Ġmotivated Ġthe Ġdevelopment Ġof Ġoriginal Ġdeep Ġmodels . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġnatural Ġlanguage Ġquestion - ans w ering Ġcorpus Ġhas Ġbeen Ġdesigned Ġto Ġinvestigate Ġsuch Ġcapabilities . ĠAs Ġwe Ġwill Ġpresent Ġin Ġthe Ġfollowing Ġof Ġthis Ġpaper , Ġwe Ġthink Ġsentiment Ġanalysis Ġis Ġparticularly Ġsuited Ġfor Ġthis Ġtask Ġand Ġwe Ġwill Ġintroduce Ġa Ġnovel Ġmachine Ġreading Ġcorpus Ġwith Ġsuch Ġcapability Ġrequirements . </s> Att ention - based Ġmodels Ġfor Ġaspect - based Ġsentiment Ġanalysis </s> Sent iment Ġanalysis Ġis Ġone Ġof Ġthe Ġhistorical Ġtasks Ġof ĠNatural ĠLanguage ĠProcessing . ĠIt Ġis Ġan Ġimportant Ġchallenge Ġfor Ġcompanies , Ġrestaurants , Ġhotels Ġthat Ġaim Ġto Ġanalyze Ġcustomer Ġsatisfaction Ġregarding Ġproducts Ġand Ġquality Ġof Ġservices . ĠGiven Ġa Ġtext Ġdocument , Ġthe Ġobjective Ġis Ġto Ġpredict Ġits Ġoverall Ġpol arity . ĠGenerally , Ġit Ġcan Ġbe Ġpositive , Ġnegative Ġor Ġneutral . ĠThis Ġanalysis Ġgives Ġa Ġquick Ġoverview Ġof Ġa Ġgeneral Ġsentiment Ġover Ġa Ġset Ġof Ġdocuments , Ġbut Ġthis Ġframework Ġtends Ġto Ġbe Ġrestrictive . ĠIndeed , Ġone Ġdocument Ġtends Ġto Ġexpress Ġmultiple Ġopinions Ġof Ġdifferent Ġaspects . ĠFor Ġinstance , Ġin Ġthe Ġsentence : ĠThe Ġfish Ġwas Ġvery Ġgood Ġbut Ġthe Ġservice Ġwas Ġterrible , Ġthere Ġis Ġnot Ġa Ġgeneral Ġdominant Ġsentiment , Ġand Ġa Ġfiner Ġanalysis Ġis Ġneeded . ĠThe Ġtask Ġof Ġaspect - based Ġsentiment Ġanalysis Ġaims Ġto Ġpredict Ġa Ġpol arity Ġof Ġa Ġsentence Ġregarding Ġa Ġgiven Ġaspect . ĠIn Ġthe Ġprevious Ġexample Ġa Ġpositive Ġpol arity Ġshould Ġbe Ġassociated Ġto Ġthe Ġaspect Ġfood , Ġand Ġon Ġthe Ġcontrary , Ġa Ġnegative Ġsentiment Ġis Ġexpressed Ġregarding Ġthe Ġquality Ġof Ġthe Ġservice . </s> The Ġidea Ġof Ġusing Ġmodels Ġoriginally Ġdesigned Ġfor Ġquestion - ans w ering , Ġfor Ġthe Ġsentiment Ġanalysis Ġtask Ġhas Ġbeen Ġintroduced Ġin ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠIn Ġthese Ġpapers , Ġseveral Ġadaptations Ġof Ġthe Ġend - to - end Ġmemory Ġnetwork Ġ( Mem N 2 N ) ĠB IB REF 18 Ġare Ġused Ġto Ġpredict Ġthe Ġpol arity Ġof Ġa Ġreview Ġregarding Ġa Ġgiven Ġaspect . ĠIn Ġthat Ġconfiguration , Ġthe Ġreview Ġis Ġencoded Ġinto Ġthe Ġmemory Ġcells Ġand Ġthe Ġcontroller , Ġusually Ġinitialized Ġwith Ġa Ġrepresentation Ġof Ġthe Ġquestion , Ġis Ġinitialized Ġwith Ġa Ġrepresentation Ġof Ġthe Ġaspect . ĠThe Ġanalysis Ġof Ġthe Ġattention Ġbetween Ġthe Ġvalues Ġof Ġthe Ġcontroller Ġand Ġthe Ġdocument Ġhas Ġshown Ġinteresting Ġresults , Ġby Ġhighlighting Ġrelevant Ġpart Ġof Ġa Ġdocument Ġregarding Ġan Ġaspect . </s> Review Q A Ġdataset </s> We Ġthink Ġthat Ġevaluating Ġthe Ġtask Ġof Ġsentiment Ġanalysis Ġthrough Ġthe Ġsetup Ġof Ġquestion - ans w ering Ġis Ġa Ġrelevant Ġplayground Ġfor Ġmachine Ġreading Ġresearch . ĠIndeed Ġnatural Ġlanguage Ġquestions Ġabout Ġthe Ġdifferent Ġaspects Ġof Ġthe Ġtargeted Ġvenues Ġare Ġtypical Ġkind Ġof Ġquestions Ġwe Ġwant Ġto Ġbe Ġable Ġto Ġask Ġto Ġa Ġsystem . ĠIn Ġthis Ġcontext , Ġwe Ġintroduce Ġa Ġset Ġof Ġreasoning Ġquestions Ġtypes Ġover Ġthe Ġrelationships Ġbetween Ġaspects . ĠWe Ġpropose ĠReview Q A , Ġa Ġdataset Ġof Ġnatural Ġlanguage Ġquestions Ġover Ġhotel Ġreviews . ĠThese Ġquestions Ġare Ġdivided Ġinto Ġ8 Ġgroups , Ġregarding Ġthe Ġcompet ency Ġrequired Ġto Ġbe Ġanswered . ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġeach Ġtask Ġand Ġthe Ġprocess Ġfollowed Ġto Ġgenerate Ġthis Ġdataset . </s> Original Ġdata </s> We Ġused Ġa Ġset Ġof Ġreviews Ġextracted Ġfrom Ġthe ĠTrip Ad visor Ġwebsite Ġand Ġoriginally Ġproposed Ġin ĠB IB REF 10 Ġand ĠB IB REF 11 Ġ. ĠThis Ġcorpus Ġis Ġavailable Ġat Ġhttp :// www . cs . v irgin ia . edu / ~ hw 5 x / Data / L ARA / T rip Ad visor / T rip Ad visor J son . tar . b z 2 . ĠEach Ġreview Ġcomes Ġwith Ġthe Ġname Ġof Ġthe Ġassociated Ġhotel , Ġa Ġtitle , Ġan Ġoverall Ġrating , Ġa Ġcomment Ġand Ġa Ġlist Ġof Ġrated Ġaspects . ĠFrom Ġ0 Ġto Ġ7 Ġaspects , Ġamong Ġvalue , Ġroom , Ġlocation , Ġclean liness , Ġcheck - in / front Ġdesk , Ġservice , Ġbusiness Ġservice , Ġcan Ġpossibly Ġbe Ġrated Ġin Ġa Ġreview . ĠFigure ĠFIG REF 8 Ġdisplays Ġa Ġreview Ġextracted Ġfrom Ġthis Ġdataset . </s> Rel ational Ġreasoning Ġcompet encies </s> Object ive : ĠStarting Ġwith Ġthe Ġoriginal Ġcorpus , Ġwe Ġaim Ġat Ġbuilding Ġa Ġmachine Ġreading Ġtask Ġwhere Ġnatural Ġlanguage Ġquestions Ġwill Ġchallenge Ġthe Ġmodel Ġon Ġits Ġunderstanding Ġof Ġthe Ġreviews . ĠIndeed Ġlearning Ġrelational Ġreasoning Ġcompet encies Ġover Ġnatural Ġlanguage Ġdocuments Ġis Ġa Ġmajor Ġchallenge Ġof Ġthe Ġcurrent Ġreading Ġmodels . ĠThese Ġoriginal Ġraw Ġdata Ġallow Ġus Ġto Ġgenerate Ġrelational Ġquestions Ġthat Ġcan Ġpossibly Ġrequire Ġa Ġglobal Ġunderstanding Ġof Ġthe Ġcomment Ġand Ġreasoning Ġskills Ġto Ġbe Ġtreated . ĠFor Ġexample , Ġasking Ġa Ġquestion Ġlike ĠWhat Ġis Ġthe Ġbest Ġaspect Ġrated Ġin Ġthis Ġcomment Ġ? Ġis Ġnot Ġan Ġeasy Ġquestion Ġthat Ġcan Ġbe Ġanswered Ġwithout Ġa Ġdeep Ġunderstanding Ġof Ġthe Ġreview . ĠIt Ġis Ġnecessary Ġto Ġcapture Ġall Ġthe Ġaspects Ġmentioned Ġin Ġthe Ġtext , Ġto Ġpredict Ġtheir Ġrating Ġand Ġfinally Ġto Ġselect Ġthe Ġbest Ġone . ĠThe Ġtasks Ġand Ġthe Ġdataset Ġwe Ġpropose Ġare Ġpublicly Ġavailable Ġat Ġhttp :// www . euro pe . na ver l abs . com / Blog / Review Q A - A - no vel - rel ational - as pect - based - op inion - dat as et - for - machine - reading </s> We Ġintroduce Ġa Ġlist Ġof Ġ8 Ġdifferent Ġcompet encies Ġthat Ġa Ġreading Ġsystem Ġshould Ġmaster Ġin Ġorder Ġto Ġprocess Ġreviews Ġand Ġtext Ġdocuments Ġin Ġgeneral . ĠThese Ġ8 Ġtasks Ġrequire Ġdifferent Ġcompet encies Ġand Ġa Ġdifferent Ġlevel Ġof Ġunderstanding Ġof Ġthe Ġdocument Ġto Ġbe Ġwell Ġanswered . ĠFor Ġinstance , Ġdetecting Ġif Ġan Ġaspect Ġis Ġmentioned Ġin Ġa Ġreview Ġwill Ġrequire Ġless Ġunderstanding Ġof Ġthe Ġreview Ġthan Ġpredicting Ġexplicitly Ġthe Ġrating Ġof Ġthis Ġaspect . ĠTable ĠT AB REF 10 Ġpresents Ġthe Ġ8 Ġtasks Ġwe Ġhave Ġintroduced Ġin Ġthis Ġdataset Ġwith Ġan Ġexample Ġof Ġa Ġquestion Ġthat Ġcorresponds Ġto Ġeach Ġtask . ĠWe Ġalso Ġprovide Ġthe Ġexpected Ġtype Ġof Ġthe Ġanswer Ġ( Yes / No Ġquestion , Ġrating Ġquestion ... ). ĠIt Ġcan Ġbe Ġan Ġadditional Ġtool Ġto Ġanalyze Ġthe Ġerrors Ġof Ġthe Ġreaders . </s> Construction Ġof Ġthe Ġdataset </s> We Ġsample Ġ100 . 000 Ġreviews Ġfrom Ġthe Ġoriginal Ġcorpus . ĠFigure ĠFIG REF 12 Ġpresents Ġthe Ġdistribution Ġof Ġthe Ġnumber Ġof Ġwords Ġof Ġthe Ġreviews Ġin Ġthe Ġdataset . ĠWe Ġexplicitly Ġfavor Ġreviews Ġwhich Ġcontain Ġan Ġimportant Ġnumber Ġof Ġwords . ĠIn Ġaverage , Ġa Ġreview Ġcontains Ġ200 Ġwords . ĠIndeed Ġthese Ġlong Ġreviews Ġare Ġmost Ġlikely Ġto Ġcontain Ġchallenging Ġrelations Ġbetween Ġdifferent Ġaspects . ĠA Ġshort Ġreview Ġwhich Ġdeals Ġwith Ġonly Ġa Ġfew Ġaspects Ġis Ġmore Ġlikely Ġto Ġnot Ġbe Ġvery Ġrelevant Ġto Ġthe Ġchallenge Ġwe Ġwant Ġto Ġpropose Ġin Ġthis Ġdataset . ĠFigure ĠFIG REF 14 Ġdisplays Ġthe Ġdistribution Ġof Ġthe Ġratings Ġper Ġaspects Ġin Ġthe Ġ100 . 000 Ġreviews Ġwe Ġbased Ġour Ġdataset . ĠWe Ġcan Ġsee Ġthat Ġthe Ġaverage Ġvalues Ġof Ġthese Ġratings Ġtend Ġto Ġbe Ġquite Ġhigh . ĠIt Ġcould Ġhave Ġintroduced Ġbias Ġif Ġit Ġwas Ġnot Ġthe Ġcase Ġfor Ġall Ġthe Ġaspects . ĠFor Ġexample , Ġwe Ġdo Ġnot Ġwant Ġthat Ġthe Ġmodel Ġlearns Ġthat Ġin Ġgeneral , Ġthe Ġservice Ġis Ġrated Ġbetter Ġthan Ġthe Ġlocation Ġand Ġthem Ġanswer Ġwithout Ġlooking Ġat Ġthe Ġdocument . ĠSince Ġthis Ġsituation Ġis Ġthe Ġsame Ġfor Ġall Ġthe Ġaspects , Ġthe Ġrelational Ġtasks Ġintroduced Ġin Ġthis Ġdataset Ġremains Ġextremely Ġrelevant . </s> Then Ġwe Ġrandomly Ġselect Ġ6 Ġtasks Ġfor Ġeach Ġreview Ġ( the Ġsame Ġtask Ġcan Ġbe Ġselected Ġmultiple Ġtimes ) Ġand Ġrandomly Ġselect Ġa Ġnatural Ġlanguage Ġquestion Ġthat Ġcorresponds Ġto Ġthis Ġtask . ĠThe Ġquestions Ġare Ġhuman - generated Ġpatterns Ġthat Ġwe Ġhave Ġcrowds ourced Ġin Ġorder Ġto Ġproduce Ġa Ġdataset Ġas Ġrich Ġas Ġpossible . ĠTo Ġthis Ġend , Ġwe Ġhave Ġgenerated Ġseveral Ġpatterns Ġthat Ġcorrespond Ġto Ġthe Ġcapabilities Ġwe Ġwanted Ġto Ġexpress Ġin Ġa Ġgiven Ġquestion Ġand Ġwe Ġhave Ġcrowds ourced Ġre ph r asing Ġof Ġthese Ġpatterns . </s> The Ġfinal Ġdataset Ġwe Ġpropose Ġis Ġcomposed Ġof Ġmore Ġthan Ġ500 . 000 Ġquestions Ġabout Ġ100 . 000 Ġreviews . ĠTable ĠT AB REF 13 Ġshows Ġthe Ġrep art ition Ġof Ġthe Ġdocuments Ġand Ġqueries Ġinto Ġthe Ġtrain Ġand Ġtest Ġset . ĠEach Ġreview Ġcontains Ġa Ġmaximum Ġof Ġ6 Ġquestions . ĠSometimes Ġless Ġwhen Ġit Ġis Ġnot Ġpossible Ġto Ġgenerate Ġall . ĠFor Ġexample , Ġif Ġonly Ġtwo Ġor Ġthree Ġaspects Ġare Ġmentioned Ġin Ġa Ġreview , Ġwe Ġwill Ġbe Ġable Ġto Ġgenerate Ġonly Ġa Ġlittle Ġset Ġof Ġrelational Ġquestions . ĠFigure ĠFIG REF 15 Ġdepicts Ġthe Ġrep art ition Ġof Ġthe Ġanswers Ġin Ġthe Ġgenerated Ġdataset . ĠA Ġmajority Ġof Ġthe Ġtasks Ġwe Ġintroduced , Ġeven Ġif Ġthey Ġpossibly Ġrequire Ġa Ġhigh Ġlevel Ġof Ġunderstanding Ġof Ġthe Ġdocument Ġand Ġthe Ġquestion , Ġare Ġbinary Ġquestions . ĠIt Ġmeans Ġthat Ġin Ġthe Ġgenerated Ġdataset Ġthe Ġanswers Ġyes Ġand Ġno Ġtend Ġto Ġbe Ġmore Ġpresent Ġthan Ġthe Ġothers . ĠTo Ġbalance Ġin Ġa Ġbetter Ġway Ġthe Ġdistribution Ġof Ġthe Ġanswers , Ġwe Ġchose Ġto Ġaffect Ġa Ġhigher Ġprobability Ġof Ġsampling Ġto Ġthe Ġtask Ġ5 , Ġ6 , Ġ7 . 1 , Ġ8 . ĠIndeed , Ġthese Ġtasks Ġare Ġnot Ġbinary Ġquestions Ġand Ġrequired Ġan Ġaspect Ġname Ġas Ġthe Ġanswer . ĠFigure ĠFIG REF 17 Ġrepresents Ġthe Ġrep art ition Ġof Ġquestion Ġtypes Ġin Ġour Ġdataset . ĠFinally , Ġfigure ĠFIG REF 15 Ġshows Ġthe Ġrep art ition Ġof Ġthe Ġanswers Ġin Ġthe Ġdataset . </s> Par aph rase Ġaug mentation Ġusing Ġback translation </s> In Ġorder Ġto Ġgenerate Ġmore Ġparaph r ases Ġof Ġthe Ġquestions , Ġwe Ġused Ġa Ġback translation Ġmethod Ġto Ġenrich Ġthem . ĠThe Ġidea Ġis Ġto Ġuse Ġa Ġtranslation Ġmodel Ġthat Ġwill Ġtranslate Ġour Ġhuman - generated Ġquestions Ġinto Ġanother Ġlanguage , Ġand Ġthen Ġtranslate Ġthem Ġback Ġto ĠEnglish . ĠThis Ġdouble Ġtranslation Ġwill Ġintroduce Ġre word ings Ġof Ġthe Ġquestions Ġthat Ġwe Ġwill Ġbe Ġable Ġto Ġintegrate Ġinto Ġthis Ġdataset . ĠThis Ġapproach Ġhas Ġbeen Ġused Ġin ĠB IB REF 7 Ġto Ġperform Ġdata Ġaug mentation Ġon Ġthe Ġtraining Ġset . ĠFor Ġthis Ġpurpose , Ġwe Ġhave Ġtrained Ġa Ġfair seq ĠB IB REF 19 Ġmodel Ġto Ġtranslate Ġsentences Ġfrom ĠEnglish Ġto ĠFrench Ġand Ġfor ĠFrench Ġto ĠEnglish . ĠIn Ġorder Ġto Ġpreserve Ġthe Ġquality Ġof Ġthe Ġsentences Ġwe Ġhave Ġso Ġfar , Ġwe Ġonly Ġkeep Ġthe Ġmost Ġprobable Ġtranslation Ġof Ġeach Ġoriginal Ġsentence . ĠIndeed Ġa Ġbeam Ġsearch Ġis Ġused Ġduring Ġthe Ġtranslation Ġto Ġpredict Ġthe Ġmost Ġprobable Ġtranslations Ġwhich Ġmean Ġthat Ġwe Ġeach Ġtranslation Ġcomes Ġwith Ġan Ġassociated Ġprobability . ĠBy Ġselecting Ġonly Ġthe Ġfirst Ġtranslations , Ġwe Ġalmost Ġdouble Ġthe Ġnumber Ġof Ġquestions Ġwithout Ġdegrading Ġthe Ġquality Ġof Ġthe Ġquestions Ġproposed Ġin Ġthe Ġdataset . </s> Mod els </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġperformance Ġof Ġfour Ġdifferent Ġmodels Ġon Ġour Ġdataset : Ġa Ġlog istic Ġregression Ġand Ġthree Ġneural Ġmodels . ĠThe Ġfirst Ġone Ġis Ġa Ġbasic ĠL ST M ĠB IB REF 20 Ġ, Ġthe Ġsecond Ġa ĠMem N 2 N ĠB IB REF 18 Ġand Ġthe Ġthird Ġone Ġis Ġa Ġmodel Ġof Ġour Ġown Ġdesign . ĠThis Ġfourth Ġmodel Ġre uses Ġthe Ġencoding Ġlayers Ġof Ġthe ĠR - net ĠB IB REF 12 Ġand Ġwe Ġmodify Ġthe Ġfinal Ġlayers Ġwith Ġa Ġprojection Ġlayer Ġthat Ġwill Ġbe Ġable Ġto Ġselect Ġthe Ġanswer Ġamong Ġthe Ġset Ġof Ġcandidates Ġinstead Ġof Ġpointing Ġthe Ġansw erer Ġdirectly Ġinto Ġthe Ġsource Ġdocument . </s> Log istic Ġregression : ĠTo Ġproduce Ġthe Ġrepresentation Ġof Ġthe Ġinput , Ġwe Ġconc aten ate Ġthe ĠBag - Of - Words Ġrepresentation Ġof Ġthe Ġdocument Ġwith Ġthe ĠBag - Of - Words Ġrepresentation Ġof Ġthe Ġquestion . ĠIt Ġproduces Ġan Ġarray Ġof Ġsize ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġvocabulary Ġsize . ĠThen Ġwe Ġuse Ġa Ġlog istic Ġregression Ġto Ġselect Ġthe Ġmost Ġprobable Ġanswer Ġamong Ġthe ĠIN LINE FORM 2 Ġpossibilities . </s> L ST M : ĠWe Ġstart Ġwith Ġa Ġconc aten ation Ġof Ġthe Ġsequence Ġof Ġindexes Ġof Ġthe Ġdocument Ġwith Ġthe Ġsequence Ġof Ġindexes Ġof Ġthe Ġquestion . ĠThem Ġwe Ġfeed Ġan ĠL ST M Ġnetwork Ġwith Ġthis Ġvector Ġand Ġuse Ġthe Ġfinal Ġstate Ġas Ġthe Ġrepresentation Ġof Ġthe Ġinput . ĠFinally , Ġwe Ġapply Ġa Ġlog istic Ġregression Ġover Ġthis Ġrepresentation Ġto Ġproduce Ġthe Ġfinal Ġdecision . </s> End - to - end Ġmemory Ġnetworks : ĠThis Ġarchitecture Ġis Ġbased Ġon Ġtwo Ġdifferent Ġmemory Ġcells Ġ( input Ġand Ġoutput ) Ġthat Ġcontain Ġa Ġrepresentation Ġof Ġthe Ġdocument . ĠA Ġcontroller , Ġinitialized Ġwith Ġthe Ġencoding Ġof Ġthe Ġquestion , Ġis Ġused Ġto Ġcalculate Ġan Ġattention Ġbetween Ġthis Ġcontroller Ġand Ġthe Ġrepresentation Ġof Ġthe Ġdocument Ġin Ġthe Ġinput Ġmemory . ĠThis Ġattention Ġis Ġthem Ġused Ġto Ġre - weight Ġthe Ġrepresentation Ġof Ġthe Ġdocument Ġin Ġthe Ġoutput Ġmemory . ĠThis Ġresponse Ġfrom Ġthe Ġoutput Ġmemory Ġis Ġthem Ġutilized Ġto Ġupdate Ġthe Ġcontroller . ĠAfter Ġthat , Ġeither Ġa Ġmatrix Ġis Ġused Ġto Ġproject Ġthis Ġrepresentation Ġinto Ġthe Ġanswer Ġspace Ġeither Ġthe Ġcontroller Ġis Ġused Ġto Ġgo Ġthrough Ġan Ġover Ġhop Ġof Ġmemory . ĠThis Ġarchitecture Ġallows Ġthe Ġmodel Ġto Ġsequ entially Ġlook Ġinto Ġthe Ġinitial Ġdocument Ġseeking Ġfor Ġimportant Ġinformation Ġregarding Ġthe Ġcurrent Ġstate Ġof Ġits Ġcontroller . ĠThis Ġmodel Ġachieves Ġvery Ġgood Ġperformances Ġon Ġthe Ġ20 Ġb Ab I Ġtasks Ġdataset . </s> Deep Ġproject ive Ġreader : ĠThis Ġis Ġa Ġmodel Ġof Ġour Ġown Ġdesign , Ġlargely Ġinspired Ġby Ġthe Ġefficient ĠR - net Ġreader ĠB IB REF 12 Ġ. ĠThe Ġoverall Ġarchitecture Ġis Ġcomposed Ġof Ġ4 Ġstacked Ġlayers : Ġan Ġencoding Ġlayer , Ġa Ġquestion / document Ġattention , Ġa Ġself - att ention Ġlayer Ġand Ġa Ġprojection Ġlayer . ĠThe Ġfollowing Ġparagraphs Ġbriefly Ġdescribe Ġthe Ġoverall Ġutility Ġof Ġeach Ġof Ġthese Ġlayers . </s> Enc oding : ĠThe Ġsentence Ġis Ġtoken ized Ġby Ġwords . ĠEach Ġtoken Ġis Ġrepresented Ġby Ġthe Ġconc aten ation Ġof Ġits Ġembed ding Ġvector Ġand Ġthe Ġfinal Ġstate Ġof Ġa Ġbid irection al Ġrecurrent Ġnetwork Ġover Ġthe Ġcharacters Ġof Ġthis Ġword . ĠFinally , Ġanother Ġbid irection al ĠR NN Ġon Ġthe Ġtop Ġof Ġthis Ġrepresentation Ġproduce Ġthe Ġencoding Ġof Ġthe Ġdocument Ġand Ġthe Ġquestion . </s> Question / document Ġattention : ĠWe Ġapply Ġa Ġquestion / document Ġattention Ġlayer Ġthat Ġmatches Ġthe Ġrepresentation Ġof Ġthe Ġquestion Ġwith Ġeach Ġtoken Ġof Ġthe Ġdocument Ġindividually Ġto Ġoutput Ġan Ġattention Ġthat Ġgives Ġmore Ġweight Ġto Ġthe Ġimportant Ġtokens Ġof Ġthe Ġdocument Ġregarding Ġthe Ġquestion . </s> Self - att ention Ġlayer : ĠThe Ġprevious Ġlayer Ġhas Ġbuilt Ġa Ġquestion - aware Ġrepresentation Ġof Ġthe Ġdocument . ĠOne Ġproblem Ġwith Ġsuch Ġrepresentation Ġis Ġthat Ġform Ġthe Ġmoment Ġeach Ġtoken Ġhas Ġonly Ġa Ġgood Ġknowledge Ġof Ġits Ġclosest Ġneighbors . ĠTo Ġtackle Ġthis Ġproblem , ĠB IB REF 12 Ġhave Ġproposed Ġto Ġuse Ġa Ġself - att ention Ġlayer Ġthat Ġmatches Ġeach Ġindividual Ġtoken Ġwith Ġall Ġthe Ġother Ġtokens Ġof Ġthe Ġdocument . ĠDoing Ġthat , Ġeach Ġtoken Ġis Ġnow Ġaware Ġof Ġa Ġlarger Ġcontext . </s> Output Ġlayer : ĠA Ġbid irection al ĠR NN Ġis Ġapplied Ġon Ġthe Ġtop Ġof Ġthe Ġlast Ġlayer Ġand Ġwe Ġuse Ġits Ġfinal Ġstate Ġas Ġthe Ġrepresentation Ġof Ġthe Ġinput . ĠWe Ġuse Ġa Ġprojection Ġmatrix Ġto Ġproject Ġthis Ġrepresentation Ġinto Ġthe Ġanswer Ġspace Ġand Ġselect Ġthe Ġmost Ġprobable Ġone </s> Training Ġdetails </s> We Ġpropose Ġto Ġtrain Ġthese Ġmodels Ġon Ġthe Ġentire Ġset Ġof Ġtasks Ġand Ġthem Ġto Ġmeasure Ġthe Ġoverall Ġperformance Ġand Ġthe Ġaccuracy Ġof Ġeach Ġindividual Ġtask . ĠIn Ġall Ġthe Ġmodels , Ġwe Ġuse Ġthe ĠAdam Ġoptim izer ĠB IB REF 21 Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 01 Ġand Ġthe Ġbatch Ġsize Ġis Ġset Ġto Ġ64 . ĠAll Ġthe Ġparameter Ġare Ġinitialized Ġfrom Ġa ĠGa ussian Ġdistribution Ġwith Ġmean Ġ0 Ġand Ġa Ġstandard Ġdeviation Ġof Ġ0 . 01 . ĠThe Ġdimension Ġof Ġthe Ġword Ġembed d ings Ġin Ġthe Ġproject ive Ġdeep Ġreading Ġmodel Ġand Ġthe ĠL ST M Ġmodel Ġis Ġ300 Ġand Ġwe Ġuse ĠGl ove Ġpre - trained Ġvectors Ġ( ĠB IB REF 22 Ġ). ĠWe Ġuse Ġa ĠMem N 2 N Ġwith Ġ5 Ġmemory Ġhops Ġand Ġa Ġlinear Ġstart Ġof Ġ5 Ġepoch s . ĠThe Ġreviews Ġare Ġsplit Ġby Ġsentence Ġand Ġeach Ġmemory Ġblock Ġcorresponds Ġto Ġone Ġsentence . ĠEach Ġsentence Ġis Ġrepresented Ġby Ġits Ġbag - of - word Ġrepresentation Ġaugmented Ġwith Ġtemporal Ġencoding Ġas Ġit Ġis Ġsuggested Ġin ĠB IB REF 18 Ġ. </s> Model Ġperformance </s> Table ĠT AB REF 19 Ġdisplays Ġthe Ġperformance Ġof Ġthe Ġ4 Ġbas elines Ġon Ġthe ĠReview Q A 's Ġtest Ġset . ĠThese Ġresults Ġare Ġthe Ġperformance Ġachieved Ġby Ġour Ġown Ġimplementation Ġof Ġthese Ġ4 Ġmodels . ĠAccording Ġto Ġour Ġresults , Ġthe Ġsimple ĠL ST M Ġnetwork Ġand Ġthe ĠMem N 2 N Ġperform Ġvery Ġpoorly Ġon Ġthis Ġdataset . ĠEspecially Ġon Ġthe Ġmost Ġadvanced Ġreasoning Ġtasks . ĠIndeed , Ġthe Ġtask Ġ5 Ġwhich Ġcorresponds Ġto Ġthe Ġprediction Ġof Ġthe Ġexact Ġrating Ġof Ġan Ġaspect Ġseems Ġto Ġbe Ġvery Ġchallenging Ġfor Ġthese Ġmodel . ĠMaybe Ġthe Ġtoken ization Ġby Ġsentence Ġto Ġcreate Ġthe Ġmemory Ġblocks Ġof Ġthe ĠMem N 2 N , Ġwhich Ġis Ġappropriated Ġin Ġthe Ġcase Ġof Ġthe Ġb Ab I Ġtasks , Ġis Ġnot Ġa Ġgood Ġrepresentation Ġof Ġthe Ġdocuments Ġwhen Ġit Ġhas Ġto Ġhandle Ġhuman Ġgenerated Ġcomments . ĠHowever , Ġthe Ġlog istic Ġregression Ġachieves Ġreasonable Ġperformance Ġon Ġthese Ġtasks , Ġand Ġdo Ġnot Ġsuffer Ġfrom Ġcatastrophic Ġperformance Ġon Ġany Ġtasks . ĠIts Ġworst Ġresult Ġcomes Ġon Ġtask Ġ6 Ġand Ġone Ġof Ġthe Ġreason Ġis Ġprobably Ġthat Ġthis Ġarchitecture Ġis Ġnot Ġdesigned Ġto Ġpredict Ġa Ġlist Ġof Ġanswers . ĠOn Ġthe Ġcontrary , Ġthe Ġdeep Ġproject ive Ġreader Ġachieves Ġencouraging Ġon Ġthis Ġdataset . ĠIt Ġoutper forms Ġall Ġthe Ġother Ġbas elines , Ġwith Ġvery Ġgood Ġscores Ġon Ġthe Ġfirst Ġfourth Ġtasks . ĠThe Ġquestion / document Ġand Ġdocument / document Ġattention Ġlayers Ġproposed Ġin ĠB IB REF 12 Ġseem Ġonce Ġagain Ġto Ġproduce Ġrich Ġenc od ings Ġof Ġthe Ġinputs Ġwhich Ġare Ġrelevant Ġfor Ġour Ġprojection Ġlayer . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġformal ize Ġthe Ġsentiment Ġanalysis Ġtask Ġthrough Ġthe Ġframework Ġof Ġmachine Ġreading Ġand Ġrelease ĠReview Q A , Ġa Ġrelational Ġquestion - ans w ering Ġcorpus . ĠThis Ġdataset Ġallows Ġevaluating Ġa Ġset Ġof Ġrelational Ġreasoning Ġskills Ġthrough Ġnatural Ġlanguage Ġquestions . ĠIt Ġis Ġcomposed Ġof Ġa Ġlarge Ġpanel Ġof Ġhuman - generated Ġquestions . ĠMoreover , Ġwe Ġpropose Ġto Ġaugment Ġthe Ġdataset Ġwith Ġback trans lated Ġreform ulations Ġof Ġthese Ġquestions . ĠFinally , Ġwe Ġevaluate Ġ4 Ġmodels Ġon Ġthis Ġdataset , Ġincluding Ġa Ġproject ive Ġmodel Ġof Ġour Ġown Ġdesign Ġthat Ġseems Ġto Ġbe Ġa Ġstrong Ġbaseline Ġfor Ġthis Ġdataset . ĠWe Ġexpect Ġthat Ġthis Ġlarge Ġdataset Ġwill Ġencourage Ġthe Ġresearch Ġcommunity Ġto Ġdevelop Ġreasoning Ġmodels Ġand Ġevaluate Ġtheir Ġmodels Ġon Ġthis Ġset Ġof Ġtasks . </s> Acknowled gment </s> We Ġthank ĠV ass il ina ĠNik oul ina Ġand ĠSt Ã© ph ane ĠCl inch ant Ġfor Ġthe Ġhelp Ġregarding Ġthe Ġback translation Ġre w ording Ġof Ġthe Ġquestions .