Yes
BOOLEAN
No
BOOLEAN
If there exists a sentence whose INLINEFORM0 , the paragraph consisting of that sentence is considered the silver-standard answer passage. Table TABREF3 shows how robust these silver-standard passages are based on human judgement ( INLINEFORM1 ) and how many passages are collected ( INLINEFORM2 ) for INLINEFORM3 , where the human judgement is performed on 50 random samples for each case. For answer retrieval, a dataset is created by INLINEFORM4 , which gives INLINEFORM5 accuracy and INLINEFORM6 coverage, respectively. Finally, each question is queried to Lucene and the top- INLINEFORM7 paragraphs are retrieved from the entire Wikipedia. If the answer sentence exists within those retrieved paragraphs according to the silver-standard, it is considered correct.
1801.02073
Introduction </s> Question Ġanswering Ġ( Q A ) Ġhas Ġbeen Ġa Ġblo oming Ġresearch Ġfield Ġfor Ġthe Ġlast Ġdecade . ĠSelection - based ĠQ A Ġimplies Ġa Ġfamily Ġof Ġtasks Ġthat Ġfind Ġanswer Ġcontexts Ġfrom Ġlarge Ġdata Ġgiven Ġquestions Ġin Ġnatural Ġlanguage . ĠThree Ġtasks Ġhave Ġbeen Ġproposed Ġfor Ġselection - based ĠQ A . ĠGiven Ġa Ġdocument , Ġanswer Ġextraction ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġfinds Ġanswer Ġphrases Ġwhereas Ġanswer Ġselection ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġand Ġanswer Ġtriggering ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġfind Ġanswer Ġsentences Ġinstead , Ġalthough Ġthe Ġpresence Ġof Ġthe Ġanswer Ġcontext Ġis Ġnot Ġassumed Ġwithin Ġthe Ġprovided Ġdocument Ġfor Ġanswer Ġtriggering Ġbut Ġit Ġis Ġfor Ġthe Ġother Ġtwo Ġtasks . ĠRecently , Ġvarious ĠQ A Ġtasks Ġthat Ġare Ġnot Ġselection - based Ġhave Ġbeen Ġproposed ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ; Ġhowever , Ġselection - based ĠQ A Ġremains Ġstill Ġimportant Ġbecause Ġof Ġits Ġpractical Ġvalue Ġto Ġreal Ġapplications Ġ( e . g ., ĠIBM ĠWatson , ĠMIT ĠStart ). </s> Several Ġdatasets Ġhave Ġbeen Ġreleased Ġfor Ġselection - based ĠQ A . Ġw ang : 07 a Ġcreated Ġthe ĠQ AS ent Ġdataset Ġconsisting Ġof Ġ277 Ġquestions , Ġwhich Ġhas Ġbeen Ġwidely Ġused Ġfor Ġbenchmark ing Ġthe Ġanswer Ġselection Ġtask . Ġf eng : 15 a Ġpresented ĠInsurance Q A Ġcomprising Ġ16 K + Ġquestions Ġon Ġinsurance Ġcontexts . Ġy ang : 15 a Ġintroduced ĠWiki Q A Ġfor Ġanswer Ġselection Ġand Ġtriggering . Ġjur c zyk : 16 Ġcreated ĠSel Q A Ġfor Ġlarge Ġreal - scale Ġanswer Ġtriggering . Ġr aj pur kar 2016 squ ad Ġpresented ĠS Qu AD Ġfor Ġanswer Ġextraction Ġand Ġselection Ġas Ġwell Ġas Ġfor Ġreading Ġcomprehension . ĠFinally , Ġmor ales - E t Al : 2016 : EM N LP 2016 Ġprovided ĠInf ob ox Q A Ġfor Ġanswer Ġselection . </s> These Ġcorpor a Ġmake Ġit Ġpossible Ġto Ġevaluate Ġthe Ġrobust ness Ġof Ġstatistical Ġquestion Ġanswering Ġlearning . ĠAlthough Ġall Ġof Ġthese Ġcorpor a Ġtarget Ġon Ġselection - based ĠQ A , Ġthey Ġare Ġdesigned Ġfor Ġdifferent Ġpurposes Ġsuch Ġthat Ġit Ġis Ġimportant Ġto Ġunderstand Ġthe Ġnature Ġof Ġthese Ġcorpor a Ġso Ġa Ġbetter Ġuse Ġof Ġthem Ġcan Ġbe Ġmade . ĠIn Ġthis Ġpaper , Ġwe Ġmake Ġboth Ġintrinsic Ġand Ġextr ins ic Ġanalyses Ġof Ġfour Ġlatest Ġcorpor a Ġbased Ġon ĠWikipedia , ĠWiki Q A , ĠSel Q A , ĠS Qu AD , Ġand ĠInf ob ox Q A . ĠWe Ġfirst Ġgive Ġa Ġthorough Ġintrinsic Ġanalysis Ġregarding Ġcontextual Ġsimilarities , Ġquestion Ġtypes , Ġand Ġanswer Ġcategories Ġ( Section ĠSEC REF 2 Ġ). ĠWe Ġthen Ġmap Ġquestions Ġin Ġall Ġcorpor a Ġto Ġthe Ġcurrent Ġversion Ġof ĠEnglish ĠWikipedia Ġand Ġbenchmark Ġanother Ġselection - based ĠQ A Ġtask , Ġanswer Ġretrieval Ġ( Section ĠSEC REF 3 Ġ). ĠFinally , Ġwe Ġpresent Ġan Ġextr ins ic Ġanalysis Ġthrough Ġa Ġset Ġof Ġexperiments Ġcross - testing Ġthese Ġcorpor a Ġusing Ġa Ġconv olution al Ġneural Ġnetwork Ġarchitecture Ġ( Section ĠSEC REF 4 Ġ). </s> Int r ins ic ĠAnalysis </s> Four Ġpublicly Ġavailable Ġcorpor a Ġare Ġselected Ġfor Ġour Ġanalysis . ĠThese Ġcorpor a Ġare Ġbased Ġon ĠWikipedia , Ġso Ġmore Ġcomparable Ġthan Ġthe Ġothers , Ġand Ġhave Ġalready Ġbeen Ġused Ġfor Ġthe Ġevaluation Ġof Ġseveral ĠQ A Ġsystems . </s> Wiki Q A ĠB IB REF 6 Ġcomprises Ġquestions Ġselected Ġfrom Ġthe ĠBing Ġsearch Ġqueries , Ġwhere Ġuser Ġclick Ġdata Ġgive Ġthe Ġquestions Ġand Ġtheir Ġcorresponding ĠWikipedia Ġarticles . ĠThe Ġabstract s Ġof Ġthese Ġarticles Ġare Ġthen Ġextracted Ġto Ġcreate Ġanswer Ġcandidates . ĠThe Ġassumption Ġis Ġmade Ġthat Ġif Ġmany Ġqueries Ġlead Ġto Ġthe Ġsame Ġarticle , Ġit Ġmust Ġcontain Ġthe Ġanswer Ġcontext ; Ġhowever , Ġthis Ġassumption Ġfails Ġfor Ġsome Ġoccasions , Ġwhich Ġmakes Ġthis Ġdataset Ġmore Ġchallenging . ĠSince Ġthe Ġexistence Ġof Ġanswer Ġcontexts Ġis Ġnot Ġguaranteed Ġin Ġthis Ġtask , Ġit Ġis Ġcalled Ġanswer Ġtriggering Ġinstead Ġof Ġanswer Ġselection . </s> Sel Q A ĠB IB REF 7 Ġis Ġa Ġproduct Ġof Ġfive Ġannotation Ġtasks Ġthrough Ġcrowds ourcing . ĠIt Ġconsists Ġof Ġabout Ġ8 K Ġquestions Ġwhere Ġa Ġhalf Ġof Ġthe Ġquestions Ġare Ġparaph r ased Ġfrom Ġthe Ġother Ġhalf , Ġaiming Ġto Ġreduce Ġcontextual Ġsimilarities Ġbetween Ġquestions Ġand Ġanswers . ĠEach Ġquestion Ġis Ġassociated Ġwith Ġa Ġsection Ġin ĠWikipedia Ġwhere Ġthe Ġanswer Ġcontext Ġis Ġguaranteed , Ġand Ġalso Ġwith Ġfive Ġsections Ġselected Ġfrom Ġthe Ġentire ĠWikipedia Ġwhere Ġthe Ġselection Ġis Ġmade Ġby Ġthe ĠLuc ene Ġsearch Ġengine . ĠThis Ġsecond Ġdataset Ġdoes Ġnot Ġassume Ġthe Ġexistence Ġof Ġthe Ġanswer Ġcontext , Ġso Ġcan Ġbe Ġused Ġfor Ġthe Ġevaluation Ġof Ġanswer Ġtriggering . </s> S Qu AD ĠB IB REF 12 Ġpresents Ġ107 K + Ġcrowds ourced Ġquestions Ġon Ġ5 36 ĠWikipedia Ġarticles , Ġwhere Ġthe Ġanswer Ġcontexts Ġare Ġguaranteed Ġto Ġexist Ġwithin Ġthe Ġprovided Ġparagraph . ĠIt Ġcontains Ġannotation Ġof Ġanswer Ġphrases Ġas Ġwell Ġas Ġthe Ġpointers Ġto Ġthe Ġsentences Ġincluding Ġthe Ġanswer Ġphrases ; Ġthus , Ġit Ġcan Ġbe Ġused Ġfor Ġboth Ġanswer Ġextraction Ġand Ġselection . ĠThis Ġcorpus Ġalso Ġprovides Ġhuman Ġaccuracy Ġon Ġthose Ġquestions , Ġsetting Ġup Ġa Ġreasonable Ġupper Ġbound Ġfor Ġmachines . ĠTo Ġavoid Ġover fitting , Ġthe Ġevaluation Ġset Ġis Ġnot Ġpublicly Ġavailable Ġalthough Ġsystem Ġoutputs Ġcan Ġbe Ġevaluated Ġby Ġtheir Ġprovided Ġscript . </s> Inf ob ox Q A ĠB IB REF 13 Ġgives Ġ15 K + Ġquestions Ġbased Ġon Ġthe Ġinf ob ox es Ġfrom Ġ150 Ġarticles Ġin ĠWikipedia . ĠEach Ġquestion Ġis Ġcrowds ourced Ġand Ġassociated Ġwith Ġan Ġinf ob ox , Ġwhere Ġeach Ġline Ġof Ġthe Ġinf ob ox Ġis Ġconsidered Ġan Ġanswer Ġcandidate . ĠThis Ġcorpus Ġemphasizes Ġthe Ġgravity Ġof Ġinf ob ox es , Ġwhich Ġsummary Ġarguably Ġthe Ġmost Ġcommonly Ġasked Ġinformation Ġabout Ġthose Ġarticles . ĠAlthough Ġthe Ġnature Ġof Ġthis Ġcorpus Ġis Ġdifferent Ġfrom Ġthe Ġothers , Ġit Ġcan Ġalso Ġbe Ġused Ġto Ġevaluate Ġanswer Ġselection . </s> Analysis </s> All Ġcorpor a Ġprovide Ġdatasets / spl its Ġfor Ġanswer Ġselection , Ġwhereas Ġonly Ġ( Wiki Q A , ĠS Qu AD ) Ġand Ġ( Wiki Q A , ĠSel Q A ) Ġprovide Ġdatasets Ġfor Ġanswer Ġextraction Ġand Ġanswer Ġtriggering , Ġrespectively . ĠS Qu AD Ġis Ġmuch Ġlarger Ġin Ġsize Ġalthough Ġquestions Ġin Ġthis Ġcorpus Ġare Ġoften Ġparaph r ased Ġmultiple Ġtimes . ĠOn Ġthe Ġcontrary , ĠS Qu AD 's Ġaverage Ġcandidates Ġper Ġquestion Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġthe Ġsmallest Ġbecause ĠS Qu AD Ġextracts Ġanswer Ġcandidates Ġfrom Ġparagraphs Ġwhereas Ġthe Ġothers Ġextract Ġthem Ġfrom Ġsections Ġor Ġinf ob ox es Ġthat Ġconsist Ġof Ġbigger Ġcontexts . ĠAlthough ĠInf ob ox Q A Ġis Ġlarger Ġthan ĠWiki Q A Ġor ĠSel Q A , Ġthe Ġnumber Ġof Ġtoken Ġtypes Ġ( ĠIN LINE FORM 1 Ġ) Ġin ĠInf ob ox Q A Ġis Ġsmaller Ġthan Ġthose Ġtwo , Ġdue Ġto Ġthe Ġrepetitive Ġnature Ġof Ġinf ob ox es . </s> All Ġcorpor a Ġshow Ġsimilar Ġaverage Ġanswer Ġcandidate Ġlengths Ġ( ĠIN LINE FORM 0 Ġ), Ġexcept Ġfor ĠInf ob ox Q A Ġwhere Ġeach Ġline Ġin Ġthe Ġinf ob ox Ġis Ġconsidered Ġa Ġcandidate . ĠSel Q A Ġand ĠS Qu AD Ġshow Ġsimilar Ġaverage Ġquestion Ġlengths Ġ( ĠIN LINE FORM 1 Ġ) Ġbecause Ġof Ġthe Ġsimilarity Ġbetween Ġtheir Ġannotation Ġschemes . ĠIt Ġis Ġnot Ġsurprising Ġthat ĠWiki Q A 's Ġaverage Ġquestion Ġlength Ġis Ġthe Ġsmallest , Ġconsidering Ġtheir Ġquestions Ġare Ġtaken Ġfrom Ġsearch Ġqueries . ĠInf ob ox Q A 's Ġaverage Ġquestion Ġlength Ġis Ġrelatively Ġsmall , Ġdue Ġto Ġthe Ġrestricted Ġinformation Ġthat Ġcan Ġbe Ġasked Ġfrom Ġthe Ġinf ob ox es . ĠInf ob ox Q A Ġand ĠWiki Q A Ġshow Ġthe Ġleast Ġquestion - answer Ġword Ġoverl aps Ġover Ġquestions Ġand Ġanswers Ġ( ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġin ĠTable ĠT AB REF 2 Ġ), Ġrespectively . ĠIn Ġterms Ġof Ġthe ĠF 1 - score Ġfor Ġoverlapping Ġwords Ġ( ĠIN LINE FORM 4 Ġ), ĠS Qu AD Ġgives Ġthe Ġleast Ġportion Ġof Ġoverl aps Ġbetween Ġquestion - answer Ġpairs Ġalthough ĠWiki Q A Ġcomes Ġvery Ġclose . </s> Fig . ĠFIG REF 4 Ġshows Ġthe Ġdistributions Ġof Ġseven Ġquestion Ġtypes Ġgrouped Ġdeter minist ically Ġfrom Ġthe Ġlex icons . ĠAlthough Ġthese Ġcorpor a Ġhave Ġbeen Ġindependently Ġdeveloped , Ġa Ġgeneral Ġtrend Ġis Ġfound , Ġwhere Ġthe Ġwhat Ġquestion Ġtype Ġdominates , Ġfollowed Ġby Ġhow Ġand Ġwho , Ġfollowed Ġby Ġwhen Ġand Ġwhere , Ġand Ġso Ġon . </s> Fig . ĠFIG REF 6 Ġshows Ġthe Ġdistributions Ġof Ġanswer Ġcategories Ġautomatically Ġclassified Ġby Ġour ĠConv olution al ĠNeural ĠNetwork Ġmodel Ġtrained Ġon Ġthe Ġdata Ġdistributed Ġby Ġli : 02 a . ĠInterestingly , Ġeach Ġcorpus Ġfocuses Ġon Ġdifferent Ġcategories , ĠN umeric Ġfor ĠWiki Q A Ġand ĠSel Q A , ĠEntity Ġfor ĠS Qu AD , Ġand ĠPerson Ġfor ĠInf ob ox Q A , Ġwhich Ġgives Ġenough Ġdivers ities Ġfor Ġstatistical Ġlearning Ġto Ġbuild Ġrobust Ġmodels . </s> Answer ĠRet ri eval </s> This Ġsection Ġdescribes Ġanother Ġselection - based ĠQ A Ġtask , Ġcalled Ġanswer Ġretrieval , Ġthat Ġfinds Ġthe Ġanswer Ġcontext Ġfrom Ġa Ġlarger Ġdataset , Ġthe Ġentire ĠWikipedia . ĠS Qu AD Ġprovides Ġno Ġmapping Ġof Ġthe Ġanswer Ġcontexts Ġto ĠWikipedia , Ġwhereas ĠWiki Q A Ġand ĠSel Q A Ġprovide Ġm appings ; Ġhowever , Ġtheir Ġdata Ġdo Ġnot Ġcome Ġfrom Ġthe Ġsame Ġversion Ġof ĠWikipedia . ĠWe Ġpropose Ġan Ġautomatic Ġway Ġof Ġmapping Ġthe Ġanswer Ġcontexts Ġfrom Ġall Ġcorpor a Ġto Ġthe Ġsame Ġversion Ġof ĠWik ip eda Ġso Ġthey Ġcan Ġbe Ġcoherent ly Ġused Ġfor Ġanswer Ġretrieval . </s> Each Ġparagraph Ġin ĠWikipedia Ġis Ġfirst Ġindexed Ġby ĠLuc ene Ġusing Ġ{ 1 , 2 , 3 } - gram s , Ġwhere Ġthe Ġparagraphs Ġare Ġseparated Ġby ĠWiki Ext ractor Ġand Ġsegment ed Ġby ĠN LP 4 J Ġ( 28 . 7 M + Ġparagraphs Ġare Ġindexed ). ĠEach Ġanswer Ġsentence Ġfrom Ġthe Ġcorpor a Ġin ĠTable ĠT AB REF 3 Ġis Ġthen Ġquer ied Ġto ĠLuc ene , Ġand Ġthe Ġtop - 5 Ġranked Ġparagraphs Ġare Ġretrieved . ĠThe Ġcos ine Ġsimilarity Ġbetween Ġeach Ġsentence Ġin Ġthese Ġparagraphs Ġand Ġthe Ġanswer Ġsentence Ġis Ġmeasured Ġfor ĠIN LINE FORM 0 Ġ- gram s , Ġsay ĠIN LINE FORM 1 Ġ. ĠA Ġweight Ġis Ġassigned Ġto Ġeach ĠIN LINE FORM 2 Ġ- gram Ġscore , Ġsay ĠIN LINE FORM 3 Ġ, Ġand Ġthe Ġweighted Ġsum Ġis Ġmeasured : ĠIN LINE FORM 4 Ġ. ĠThe Ġfixed Ġweights Ġof ĠIN LINE FORM 5 Ġare Ġused Ġfor Ġour Ġexperiments , Ġwhich Ġcan Ġbe Ġimproved . </s> If Ġthere Ġexists Ġa Ġsentence Ġwhose ĠIN LINE FORM 0 Ġ, Ġthe Ġparagraph Ġconsisting Ġof Ġthat Ġsentence Ġis Ġconsidered Ġthe Ġsilver - standard Ġanswer Ġpassage . ĠTable ĠT AB REF 3 Ġshows Ġhow Ġrobust Ġthese Ġsilver - standard Ġpassages Ġare Ġbased Ġon Ġhuman Ġjudgement Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġhow Ġmany Ġpassages Ġare Ġcollected Ġ( ĠIN LINE FORM 2 Ġ) Ġfor ĠIN LINE FORM 3 Ġ, Ġwhere Ġthe Ġhuman Ġjudgement Ġis Ġperformed Ġon Ġ50 Ġrandom Ġsamples Ġfor Ġeach Ġcase . ĠFor Ġanswer Ġretrieval , Ġa Ġdataset Ġis Ġcreated Ġby ĠIN LINE FORM 4 Ġ, Ġwhich Ġgives ĠIN LINE FORM 5 Ġaccuracy Ġand ĠIN LINE FORM 6 Ġcoverage , Ġrespectively . ĠFinally , Ġeach Ġquestion Ġis Ġquer ied Ġto ĠLuc ene Ġand Ġthe Ġtop - ĠIN LINE FORM 7 Ġparagraphs Ġare Ġretrieved Ġfrom Ġthe Ġentire ĠWikipedia . ĠIf Ġthe Ġanswer Ġsentence Ġexists Ġwithin Ġthose Ġretrieved Ġparagraphs Ġaccording Ġto Ġthe Ġsilver - standard , Ġit Ġis Ġconsidered Ġcorrect . </s> Finding Ġa Ġparagraph Ġthat Ġincludes Ġthe Ġanswer Ġcontext Ġout Ġof Ġthe Ġentire ĠWikipedia Ġis Ġan Ġextremely Ġdifficult Ġtask Ġ( 128 . 7 M ). ĠThe Ġlast Ġrow Ġof ĠTable ĠT AB REF 3 Ġshows Ġresults Ġfrom Ġanswer Ġretrieval . ĠGiven ĠIN LINE FORM 0 Ġ, ĠSel Q A Ġand ĠS Qu AD Ġshow Ġabout Ġ34 % Ġand Ġ35 % Ġaccuracy , Ġwhich Ġare Ġreasonable . ĠHowever , ĠWiki Q A Ġshows Ġa Ġsignificantly Ġlower Ġaccuracy Ġof Ġ12 . 47 %; Ġthis Ġis Ġbecause Ġthe Ġquestions Ġin ĠWiki Q A Ġis Ġabout Ġtwice Ġshorter Ġthan Ġthe Ġquestions Ġin Ġthe Ġother Ġcorpor a Ġsuch Ġthat Ġnot Ġenough Ġlex icons Ġcan Ġbe Ġextracted Ġfrom Ġthese Ġquestions Ġfor Ġthe ĠLuc ene Ġsearch . </s> Answer ĠSelection </s> Answer Ġselection Ġis Ġevaluated Ġby Ġtwo Ġmetrics , Ġmean Ġaverage Ġprecision Ġ( MAP ) Ġand Ġmean Ġreciprocal Ġrank Ġ( MR R ). ĠThe Ġbig ram ĠCNN Ġintroduced Ġby Ġy u : 14 a Ġis Ġused Ġto Ġgenerate Ġall Ġthe Ġresults Ġin ĠTable ĠT AB REF 11 Ġ, Ġwhere Ġmodels Ġare Ġtrained Ġon Ġeither Ġsingle Ġor Ġcombined Ġdatasets . ĠClearly , Ġthe Ġquestions Ġin ĠWiki Q A Ġare Ġthe Ġmost Ġchallenging , Ġand Ġadding Ġmore Ġtraining Ġdata Ġfrom Ġthe Ġother Ġcorpor a Ġhurts Ġaccuracy Ġdue Ġto Ġthe Ġuniqueness Ġof Ġquery - based Ġquestions Ġin Ġthis Ġcorpus . ĠThe Ġbest Ġmodel Ġis Ġachieved Ġby Ġtraining Ġon ĠW + S + Q Ġfor ĠSel Q A ; Ġadding ĠInf ob ox Q A Ġhurts Ġaccuracy Ġfor ĠSel Q A Ġalthough Ġit Ġgives Ġa Ġmarginal Ġgain Ġfor ĠS Qu AD . ĠJust Ġlike ĠWiki Q A , ĠInf ob ox Q A Ġperforms Ġthe Ġbest Ġwhen Ġit Ġis Ġtrained Ġon Ġonly Ġitself . ĠFrom Ġour Ġanalysis , Ġwe Ġsuggest Ġthat Ġto Ġuse Ġmodels Ġtrained Ġon ĠWiki Q A Ġand ĠInf ob ox Q A Ġfor Ġshort Ġquery - like Ġquestions , Ġwhereas Ġto Ġuse Ġones Ġtrained Ġon ĠSel Q A Ġand ĠS Qu AD Ġfor Ġlong Ġnatural Ġquestions . </s> Answer ĠTrigger ing </s> The Ġresults Ġof ĠIN LINE FORM 0 Ġfrom Ġthe Ġanswer Ġretrieval Ġtask Ġin ĠSection ĠSEC REF 13 Ġare Ġused Ġto Ġcreate Ġthe Ġdatasets Ġfor Ġanswer Ġtriggering , Ġwhere Ġabout Ġ65 % Ġof Ġthe Ġquestions Ġare Ġnot Ġexpected Ġto Ġfind Ġtheir Ġanswer Ġcontexts Ġfrom Ġthe Ġprovided Ġparagraphs Ġfor ĠSel Q A Ġand ĠS Qu AD Ġand Ġ87 . 5 % Ġare Ġnot Ġexpected Ġfor ĠWiki Q A . ĠAnswer Ġtriggering Ġis Ġevaluated Ġby Ġthe ĠF 1 Ġscores Ġas Ġpresented Ġin ĠTable ĠT AB REF 11 Ġ, Ġwhere Ġthree Ġcorpor a Ġare Ġcross Ġvalidated . ĠThe Ġresults Ġon ĠWiki Q A Ġare Ġpretty Ġlow Ġas Ġexpected Ġfrom Ġthe Ġpoor Ġaccuracy Ġon Ġthe Ġanswer Ġretrieval Ġtask . ĠTraining Ġon ĠSel Q A Ġgives Ġthe Ġbest Ġmodels Ġfor Ġboth ĠWiki Q A Ġand ĠSel Q A . ĠTraining Ġon ĠS Qu AD Ġgives Ġthe Ġbest Ġmodel Ġfor ĠS Qu AD Ġalthough Ġthe Ġmodel Ġtrained Ġon ĠSel Q A Ġis Ġcomparable . ĠSince Ġthe Ġanswer Ġtriggering Ġdatasets Ġare Ġabout Ġ5 Ġtimes Ġlarger Ġthan Ġthe Ġanswer Ġselection Ġdatasets , Ġit Ġis Ġcomput ationally Ġtoo Ġexpensive Ġto Ġcombine Ġall Ġdata Ġfor Ġtraining . ĠWe Ġplan Ġto Ġfind Ġa Ġstrong Ġmachine Ġto Ġperform Ġthis Ġexperiment Ġin Ġnear Ġfuture . </s> Related Ġwork </s> L ately , Ġseveral Ġdeep Ġlearning Ġapproaches Ġhave Ġbeen Ġproposed Ġfor Ġquestion Ġanswering . Ġy u : 14 a Ġpresented Ġa ĠCNN Ġmodel Ġthat Ġrecognizes Ġthe Ġsemantic Ġsimilarity Ġbetween Ġtwo Ġsentences . Ġw ang - ny berg : 2015 : AC L - IJ CN LP Ġpresented Ġa Ġstacked Ġbid irection al ĠL ST M Ġapproach Ġto Ġread Ġwords Ġin Ġsequence , Ġthen Ġoutputs Ġtheir Ġsimilarity Ġscores . Ġf eng : 15 a Ġapplied Ġa Ġgeneral Ġdeep Ġlearning Ġframework Ġto Ġnon - fact oid Ġquestion Ġanswering . Ġs ant os : 16 a Ġintroduced Ġan Ġattentive Ġpool ing Ġmechanism Ġthat Ġled Ġto Ġfurther Ġimprovements Ġin Ġselection - based ĠQ A . </s> Conclusion </s> We Ġpresent Ġa Ġcomprehensive Ġcomparison Ġstudy Ġof Ġthe Ġexisting Ġcorpor a Ġfor Ġselection - based Ġquestion Ġanswering . ĠOur Ġintrinsic Ġanalysis Ġprovides Ġa Ġbetter Ġunderstanding Ġof Ġthe Ġuniqueness Ġor Ġsimilarity Ġbetween Ġthese Ġcorpor a . ĠOur Ġextr ins ic Ġanalysis Ġshows Ġthe Ġstrength Ġor Ġweakness Ġof Ġcombining Ġthese Ġcorpor a Ġtogether Ġfor Ġstatistical Ġlearning . ĠAdditionally , Ġwe Ġcreate Ġa Ġsilver - standard Ġdataset Ġfor Ġanswer Ġretrieval Ġand Ġtriggering , Ġwhich Ġwill Ġbe Ġpublicly Ġavailable . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġexplore Ġdifferent Ġways Ġof Ġimproving Ġthe Ġquality Ġof Ġour Ġsilver - standard Ġdatasets Ġby Ġfine - tun ing Ġthe Ġhyper - param eters .
Introduction Question answering (QA) has been a blooming research field for the last decade. Selection-based QA implies a family of tasks that find answer contexts from large data given questions in natural language. Three tasks have been proposed for selection-based QA. Given a document, answer extraction BIBREF0 , BIBREF1 finds answer phrases whereas answer selection BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 and answer triggering BIBREF6 , BIBREF7 find answer sentences instead, although the presence of the answer context is not assumed within the provided document for answer triggering but it is for the other two tasks. Recently, various QA tasks that are not selection-based have been proposed BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 ; however, selection-based QA remains still important because of its practical value to real applications (e.g., IBM Watson, MIT Start). Several datasets have been released for selection-based QA. wang:07a created the QASent dataset consisting of 277 questions, which has been widely used for benchmarking the answer selection task. feng:15a presented InsuranceQA comprising 16K+ questions on insurance contexts. yang:15a introduced WikiQA for answer selection and triggering. jurczyk:16 created SelQA for large real-scale answer triggering. rajpurkar2016squad presented SQuAD for answer extraction and selection as well as for reading comprehension. Finally, morales-EtAl:2016:EMNLP2016 provided InfoboxQA for answer selection. These corpora make it possible to evaluate the robustness of statistical question answering learning. Although all of these corpora target on selection-based QA, they are designed for different purposes such that it is important to understand the nature of these corpora so a better use of them can be made. In this paper, we make both intrinsic and extrinsic analyses of four latest corpora based on Wikipedia, WikiQA, SelQA, SQuAD, and InfoboxQA. We first give a thorough intrinsic analysis regarding contextual similarities, question types, and answer categories (Section SECREF2 ). We then map questions in all corpora to the current version of English Wikipedia and benchmark another selection-based QA task, answer retrieval (Section SECREF3 ). Finally, we present an extrinsic analysis through a set of experiments cross-testing these corpora using a convolutional neural network architecture (Section SECREF4 ). Intrinsic Analysis Four publicly available corpora are selected for our analysis. These corpora are based on Wikipedia, so more comparable than the others, and have already been used for the evaluation of several QA systems. WikiQA BIBREF6 comprises questions selected from the Bing search queries, where user click data give the questions and their corresponding Wikipedia articles. The abstracts of these articles are then extracted to create answer candidates. The assumption is made that if many queries lead to the same article, it must contain the answer context; however, this assumption fails for some occasions, which makes this dataset more challenging. Since the existence of answer contexts is not guaranteed in this task, it is called answer triggering instead of answer selection. SelQA BIBREF7 is a product of five annotation tasks through crowdsourcing. It consists of about 8K questions where a half of the questions are paraphrased from the other half, aiming to reduce contextual similarities between questions and answers. Each question is associated with a section in Wikipedia where the answer context is guaranteed, and also with five sections selected from the entire Wikipedia where the selection is made by the Lucene search engine. This second dataset does not assume the existence of the answer context, so can be used for the evaluation of answer triggering. SQuAD BIBREF12 presents 107K+ crowdsourced questions on 536 Wikipedia articles, where the answer contexts are guaranteed to exist within the provided paragraph. It contains annotation of answer phrases as well as the pointers to the sentences including the answer phrases; thus, it can be used for both answer extraction and selection. This corpus also provides human accuracy on those questions, setting up a reasonable upper bound for machines. To avoid overfitting, the evaluation set is not publicly available although system outputs can be evaluated by their provided script. InfoboxQA BIBREF13 gives 15K+ questions based on the infoboxes from 150 articles in Wikipedia. Each question is crowdsourced and associated with an infobox, where each line of the infobox is considered an answer candidate. This corpus emphasizes the gravity of infoboxes, which summary arguably the most commonly asked information about those articles. Although the nature of this corpus is different from the others, it can also be used to evaluate answer selection. Analysis All corpora provide datasets/splits for answer selection, whereas only (WikiQA, SQuAD) and (WikiQA, SelQA) provide datasets for answer extraction and answer triggering, respectively. SQuAD is much larger in size although questions in this corpus are often paraphrased multiple times. On the contrary, SQuAD's average candidates per question ( INLINEFORM0 ) is the smallest because SQuAD extracts answer candidates from paragraphs whereas the others extract them from sections or infoboxes that consist of bigger contexts. Although InfoboxQA is larger than WikiQA or SelQA, the number of token types ( INLINEFORM1 ) in InfoboxQA is smaller than those two, due to the repetitive nature of infoboxes. All corpora show similar average answer candidate lengths ( INLINEFORM0 ), except for InfoboxQA where each line in the infobox is considered a candidate. SelQA and SQuAD show similar average question lengths ( INLINEFORM1 ) because of the similarity between their annotation schemes. It is not surprising that WikiQA's average question length is the smallest, considering their questions are taken from search queries. InfoboxQA's average question length is relatively small, due to the restricted information that can be asked from the infoboxes. InfoboxQA and WikiQA show the least question-answer word overlaps over questions and answers ( INLINEFORM2 and INLINEFORM3 in Table TABREF2 ), respectively. In terms of the F1-score for overlapping words ( INLINEFORM4 ), SQuAD gives the least portion of overlaps between question-answer pairs although WikiQA comes very close. Fig. FIGREF4 shows the distributions of seven question types grouped deterministically from the lexicons. Although these corpora have been independently developed, a general trend is found, where the what question type dominates, followed by how and who, followed by when and where, and so on. Fig. FIGREF6 shows the distributions of answer categories automatically classified by our Convolutional Neural Network model trained on the data distributed by li:02a. Interestingly, each corpus focuses on different categories, Numeric for WikiQA and SelQA, Entity for SQuAD, and Person for InfoboxQA, which gives enough diversities for statistical learning to build robust models. Answer Retrieval This section describes another selection-based QA task, called answer retrieval, that finds the answer context from a larger dataset, the entire Wikipedia. SQuAD provides no mapping of the answer contexts to Wikipedia, whereas WikiQA and SelQA provide mappings; however, their data do not come from the same version of Wikipedia. We propose an automatic way of mapping the answer contexts from all corpora to the same version of Wikipeda so they can be coherently used for answer retrieval. Each paragraph in Wikipedia is first indexed by Lucene using {1,2,3}-grams, where the paragraphs are separated by WikiExtractor and segmented by NLP4J (28.7M+ paragraphs are indexed). Each answer sentence from the corpora in Table TABREF3 is then queried to Lucene, and the top-5 ranked paragraphs are retrieved. The cosine similarity between each sentence in these paragraphs and the answer sentence is measured for INLINEFORM0 -grams, say INLINEFORM1 . A weight is assigned to each INLINEFORM2 -gram score, say INLINEFORM3 , and the weighted sum is measured: INLINEFORM4 . The fixed weights of INLINEFORM5 are used for our experiments, which can be improved. If there exists a sentence whose INLINEFORM0 , the paragraph consisting of that sentence is considered the silver-standard answer passage. Table TABREF3 shows how robust these silver-standard passages are based on human judgement ( INLINEFORM1 ) and how many passages are collected ( INLINEFORM2 ) for INLINEFORM3 , where the human judgement is performed on 50 random samples for each case. For answer retrieval, a dataset is created by INLINEFORM4 , which gives INLINEFORM5 accuracy and INLINEFORM6 coverage, respectively. Finally, each question is queried to Lucene and the top- INLINEFORM7 paragraphs are retrieved from the entire Wikipedia. If the answer sentence exists within those retrieved paragraphs according to the silver-standard, it is considered correct. Finding a paragraph that includes the answer context out of the entire Wikipedia is an extremely difficult task (128.7M). The last row of Table TABREF3 shows results from answer retrieval. Given INLINEFORM0 , SelQA and SQuAD show about 34% and 35% accuracy, which are reasonable. However, WikiQA shows a significantly lower accuracy of 12.47%; this is because the questions in WikiQA is about twice shorter than the questions in the other corpora such that not enough lexicons can be extracted from these questions for the Lucene search. Answer Selection Answer selection is evaluated by two metrics, mean average precision (MAP) and mean reciprocal rank (MRR). The bigram CNN introduced by yu:14a is used to generate all the results in Table TABREF11 , where models are trained on either single or combined datasets. Clearly, the questions in WikiQA are the most challenging, and adding more training data from the other corpora hurts accuracy due to the uniqueness of query-based questions in this corpus. The best model is achieved by training on W+S+Q for SelQA; adding InfoboxQA hurts accuracy for SelQA although it gives a marginal gain for SQuAD. Just like WikiQA, InfoboxQA performs the best when it is trained on only itself. From our analysis, we suggest that to use models trained on WikiQA and InfoboxQA for short query-like questions, whereas to use ones trained on SelQA and SQuAD for long natural questions. Answer Triggering The results of INLINEFORM0 from the answer retrieval task in Section SECREF13 are used to create the datasets for answer triggering, where about 65% of the questions are not expected to find their answer contexts from the provided paragraphs for SelQA and SQuAD and 87.5% are not expected for WikiQA. Answer triggering is evaluated by the F1 scores as presented in Table TABREF11 , where three corpora are cross validated. The results on WikiQA are pretty low as expected from the poor accuracy on the answer retrieval task. Training on SelQA gives the best models for both WikiQA and SelQA. Training on SQuAD gives the best model for SQuAD although the model trained on SelQA is comparable. Since the answer triggering datasets are about 5 times larger than the answer selection datasets, it is computationally too expensive to combine all data for training. We plan to find a strong machine to perform this experiment in near future. Related work Lately, several deep learning approaches have been proposed for question answering. yu:14a presented a CNN model that recognizes the semantic similarity between two sentences. wang-nyberg:2015:ACL-IJCNLP presented a stacked bidirectional LSTM approach to read words in sequence, then outputs their similarity scores. feng:15a applied a general deep learning framework to non-factoid question answering. santos:16a introduced an attentive pooling mechanism that led to further improvements in selection-based QA. Conclusion We present a comprehensive comparison study of the existing corpora for selection-based question answering. Our intrinsic analysis provides a better understanding of the uniqueness or similarity between these corpora. Our extrinsic analysis shows the strength or weakness of combining these corpora together for statistical learning. Additionally, we create a silver-standard dataset for answer retrieval and triggering, which will be publicly available. In the future, we will explore different ways of improving the quality of our silver-standard datasets by fine-tuning the hyper-parameters.
Do they employ their indexing-based method to create a sample of a QA Wikipedia dataset?
a9337636b52de375c852682a2561af2c1db5ec63
Do Ġthey Ġemploy Ġtheir Ġindex ing - based Ġmethod Ġto Ġcreate Ġa Ġsample Ġof Ġa ĠQ A ĠWikipedia Ġdataset ?
Yes
<s> Do Ġthey Ġemploy Ġtheir Ġindex ing - based Ġmethod Ġto Ġcreate Ġa Ġsample Ġof Ġa ĠQ A ĠWikipedia Ġdataset ? </s> Introduction </s> Question Ġanswering Ġ( Q A ) Ġhas Ġbeen Ġa Ġblo oming Ġresearch Ġfield Ġfor Ġthe Ġlast Ġdecade . ĠSelection - based ĠQ A Ġimplies Ġa Ġfamily Ġof Ġtasks Ġthat Ġfind Ġanswer Ġcontexts Ġfrom Ġlarge Ġdata Ġgiven Ġquestions Ġin Ġnatural Ġlanguage . ĠThree Ġtasks Ġhave Ġbeen Ġproposed Ġfor Ġselection - based ĠQ A . ĠGiven Ġa Ġdocument , Ġanswer Ġextraction ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġfinds Ġanswer Ġphrases Ġwhereas Ġanswer Ġselection ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġand Ġanswer Ġtriggering ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġfind Ġanswer Ġsentences Ġinstead , Ġalthough Ġthe Ġpresence Ġof Ġthe Ġanswer Ġcontext Ġis Ġnot Ġassumed Ġwithin Ġthe Ġprovided Ġdocument Ġfor Ġanswer Ġtriggering Ġbut Ġit Ġis Ġfor Ġthe Ġother Ġtwo Ġtasks . ĠRecently , Ġvarious ĠQ A Ġtasks Ġthat Ġare Ġnot Ġselection - based Ġhave Ġbeen Ġproposed ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ; Ġhowever , Ġselection - based ĠQ A Ġremains Ġstill Ġimportant Ġbecause Ġof Ġits Ġpractical Ġvalue Ġto Ġreal Ġapplications Ġ( e . g ., ĠIBM ĠWatson , ĠMIT ĠStart ). </s> Several Ġdatasets Ġhave Ġbeen Ġreleased Ġfor Ġselection - based ĠQ A . Ġw ang : 07 a Ġcreated Ġthe ĠQ AS ent Ġdataset Ġconsisting Ġof Ġ277 Ġquestions , Ġwhich Ġhas Ġbeen Ġwidely Ġused Ġfor Ġbenchmark ing Ġthe Ġanswer Ġselection Ġtask . Ġf eng : 15 a Ġpresented ĠInsurance Q A Ġcomprising Ġ16 K + Ġquestions Ġon Ġinsurance Ġcontexts . Ġy ang : 15 a Ġintroduced ĠWiki Q A Ġfor Ġanswer Ġselection Ġand Ġtriggering . Ġjur c zyk : 16 Ġcreated ĠSel Q A Ġfor Ġlarge Ġreal - scale Ġanswer Ġtriggering . Ġr aj pur kar 2016 squ ad Ġpresented ĠS Qu AD Ġfor Ġanswer Ġextraction Ġand Ġselection Ġas Ġwell Ġas Ġfor Ġreading Ġcomprehension . ĠFinally , Ġmor ales - E t Al : 2016 : EM N LP 2016 Ġprovided ĠInf ob ox Q A Ġfor Ġanswer Ġselection . </s> These Ġcorpor a Ġmake Ġit Ġpossible Ġto Ġevaluate Ġthe Ġrobust ness Ġof Ġstatistical Ġquestion Ġanswering Ġlearning . ĠAlthough Ġall Ġof Ġthese Ġcorpor a Ġtarget Ġon Ġselection - based ĠQ A , Ġthey Ġare Ġdesigned Ġfor Ġdifferent Ġpurposes Ġsuch Ġthat Ġit Ġis Ġimportant Ġto Ġunderstand Ġthe Ġnature Ġof Ġthese Ġcorpor a Ġso Ġa Ġbetter Ġuse Ġof Ġthem Ġcan Ġbe Ġmade . ĠIn Ġthis Ġpaper , Ġwe Ġmake Ġboth Ġintrinsic Ġand Ġextr ins ic Ġanalyses Ġof Ġfour Ġlatest Ġcorpor a Ġbased Ġon ĠWikipedia , ĠWiki Q A , ĠSel Q A , ĠS Qu AD , Ġand ĠInf ob ox Q A . ĠWe Ġfirst Ġgive Ġa Ġthorough Ġintrinsic Ġanalysis Ġregarding Ġcontextual Ġsimilarities , Ġquestion Ġtypes , Ġand Ġanswer Ġcategories Ġ( Section ĠSEC REF 2 Ġ). ĠWe Ġthen Ġmap Ġquestions Ġin Ġall Ġcorpor a Ġto Ġthe Ġcurrent Ġversion Ġof ĠEnglish ĠWikipedia Ġand Ġbenchmark Ġanother Ġselection - based ĠQ A Ġtask , Ġanswer Ġretrieval Ġ( Section ĠSEC REF 3 Ġ). ĠFinally , Ġwe Ġpresent Ġan Ġextr ins ic Ġanalysis Ġthrough Ġa Ġset Ġof Ġexperiments Ġcross - testing Ġthese Ġcorpor a Ġusing Ġa Ġconv olution al Ġneural Ġnetwork Ġarchitecture Ġ( Section ĠSEC REF 4 Ġ). </s> Int r ins ic ĠAnalysis </s> Four Ġpublicly Ġavailable Ġcorpor a Ġare Ġselected Ġfor Ġour Ġanalysis . ĠThese Ġcorpor a Ġare Ġbased Ġon ĠWikipedia , Ġso Ġmore Ġcomparable Ġthan Ġthe Ġothers , Ġand Ġhave Ġalready Ġbeen Ġused Ġfor Ġthe Ġevaluation Ġof Ġseveral ĠQ A Ġsystems . </s> Wiki Q A ĠB IB REF 6 Ġcomprises Ġquestions Ġselected Ġfrom Ġthe ĠBing Ġsearch Ġqueries , Ġwhere Ġuser Ġclick Ġdata Ġgive Ġthe Ġquestions Ġand Ġtheir Ġcorresponding ĠWikipedia Ġarticles . ĠThe Ġabstract s Ġof Ġthese Ġarticles Ġare Ġthen Ġextracted Ġto Ġcreate Ġanswer Ġcandidates . ĠThe Ġassumption Ġis Ġmade Ġthat Ġif Ġmany Ġqueries Ġlead Ġto Ġthe Ġsame Ġarticle , Ġit Ġmust Ġcontain Ġthe Ġanswer Ġcontext ; Ġhowever , Ġthis Ġassumption Ġfails Ġfor Ġsome Ġoccasions , Ġwhich Ġmakes Ġthis Ġdataset Ġmore Ġchallenging . ĠSince Ġthe Ġexistence Ġof Ġanswer Ġcontexts Ġis Ġnot Ġguaranteed Ġin Ġthis Ġtask , Ġit Ġis Ġcalled Ġanswer Ġtriggering Ġinstead Ġof Ġanswer Ġselection . </s> Sel Q A ĠB IB REF 7 Ġis Ġa Ġproduct Ġof Ġfive Ġannotation Ġtasks Ġthrough Ġcrowds ourcing . ĠIt Ġconsists Ġof Ġabout Ġ8 K Ġquestions Ġwhere Ġa Ġhalf Ġof Ġthe Ġquestions Ġare Ġparaph r ased Ġfrom Ġthe Ġother Ġhalf , Ġaiming Ġto Ġreduce Ġcontextual Ġsimilarities Ġbetween Ġquestions Ġand Ġanswers . ĠEach Ġquestion Ġis Ġassociated Ġwith Ġa Ġsection Ġin ĠWikipedia Ġwhere Ġthe Ġanswer Ġcontext Ġis Ġguaranteed , Ġand Ġalso Ġwith Ġfive Ġsections Ġselected Ġfrom Ġthe Ġentire ĠWikipedia Ġwhere Ġthe Ġselection Ġis Ġmade Ġby Ġthe ĠLuc ene Ġsearch Ġengine . ĠThis Ġsecond Ġdataset Ġdoes Ġnot Ġassume Ġthe Ġexistence Ġof Ġthe Ġanswer Ġcontext , Ġso Ġcan Ġbe Ġused Ġfor Ġthe Ġevaluation Ġof Ġanswer Ġtriggering . </s> S Qu AD ĠB IB REF 12 Ġpresents Ġ107 K + Ġcrowds ourced Ġquestions Ġon Ġ5 36 ĠWikipedia Ġarticles , Ġwhere Ġthe Ġanswer Ġcontexts Ġare Ġguaranteed Ġto Ġexist Ġwithin Ġthe Ġprovided Ġparagraph . ĠIt Ġcontains Ġannotation Ġof Ġanswer Ġphrases Ġas Ġwell Ġas Ġthe Ġpointers Ġto Ġthe Ġsentences Ġincluding Ġthe Ġanswer Ġphrases ; Ġthus , Ġit Ġcan Ġbe Ġused Ġfor Ġboth Ġanswer Ġextraction Ġand Ġselection . ĠThis Ġcorpus Ġalso Ġprovides Ġhuman Ġaccuracy Ġon Ġthose Ġquestions , Ġsetting Ġup Ġa Ġreasonable Ġupper Ġbound Ġfor Ġmachines . ĠTo Ġavoid Ġover fitting , Ġthe Ġevaluation Ġset Ġis Ġnot Ġpublicly Ġavailable Ġalthough Ġsystem Ġoutputs Ġcan Ġbe Ġevaluated Ġby Ġtheir Ġprovided Ġscript . </s> Inf ob ox Q A ĠB IB REF 13 Ġgives Ġ15 K + Ġquestions Ġbased Ġon Ġthe Ġinf ob ox es Ġfrom Ġ150 Ġarticles Ġin ĠWikipedia . ĠEach Ġquestion Ġis Ġcrowds ourced Ġand Ġassociated Ġwith Ġan Ġinf ob ox , Ġwhere Ġeach Ġline Ġof Ġthe Ġinf ob ox Ġis Ġconsidered Ġan Ġanswer Ġcandidate . ĠThis Ġcorpus Ġemphasizes Ġthe Ġgravity Ġof Ġinf ob ox es , Ġwhich Ġsummary Ġarguably Ġthe Ġmost Ġcommonly Ġasked Ġinformation Ġabout Ġthose Ġarticles . ĠAlthough Ġthe Ġnature Ġof Ġthis Ġcorpus Ġis Ġdifferent Ġfrom Ġthe Ġothers , Ġit Ġcan Ġalso Ġbe Ġused Ġto Ġevaluate Ġanswer Ġselection . </s> Analysis </s> All Ġcorpor a Ġprovide Ġdatasets / spl its Ġfor Ġanswer Ġselection , Ġwhereas Ġonly Ġ( Wiki Q A , ĠS Qu AD ) Ġand Ġ( Wiki Q A , ĠSel Q A ) Ġprovide Ġdatasets Ġfor Ġanswer Ġextraction Ġand Ġanswer Ġtriggering , Ġrespectively . ĠS Qu AD Ġis Ġmuch Ġlarger Ġin Ġsize Ġalthough Ġquestions Ġin Ġthis Ġcorpus Ġare Ġoften Ġparaph r ased Ġmultiple Ġtimes . ĠOn Ġthe Ġcontrary , ĠS Qu AD 's Ġaverage Ġcandidates Ġper Ġquestion Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġthe Ġsmallest Ġbecause ĠS Qu AD Ġextracts Ġanswer Ġcandidates Ġfrom Ġparagraphs Ġwhereas Ġthe Ġothers Ġextract Ġthem Ġfrom Ġsections Ġor Ġinf ob ox es Ġthat Ġconsist Ġof Ġbigger Ġcontexts . ĠAlthough ĠInf ob ox Q A Ġis Ġlarger Ġthan ĠWiki Q A Ġor ĠSel Q A , Ġthe Ġnumber Ġof Ġtoken Ġtypes Ġ( ĠIN LINE FORM 1 Ġ) Ġin ĠInf ob ox Q A Ġis Ġsmaller Ġthan Ġthose Ġtwo , Ġdue Ġto Ġthe Ġrepetitive Ġnature Ġof Ġinf ob ox es . </s> All Ġcorpor a Ġshow Ġsimilar Ġaverage Ġanswer Ġcandidate Ġlengths Ġ( ĠIN LINE FORM 0 Ġ), Ġexcept Ġfor ĠInf ob ox Q A Ġwhere Ġeach Ġline Ġin Ġthe Ġinf ob ox Ġis Ġconsidered Ġa Ġcandidate . ĠSel Q A Ġand ĠS Qu AD Ġshow Ġsimilar Ġaverage Ġquestion Ġlengths Ġ( ĠIN LINE FORM 1 Ġ) Ġbecause Ġof Ġthe Ġsimilarity Ġbetween Ġtheir Ġannotation Ġschemes . ĠIt Ġis Ġnot Ġsurprising Ġthat ĠWiki Q A 's Ġaverage Ġquestion Ġlength Ġis Ġthe Ġsmallest , Ġconsidering Ġtheir Ġquestions Ġare Ġtaken Ġfrom Ġsearch Ġqueries . ĠInf ob ox Q A 's Ġaverage Ġquestion Ġlength Ġis Ġrelatively Ġsmall , Ġdue Ġto Ġthe Ġrestricted Ġinformation Ġthat Ġcan Ġbe Ġasked Ġfrom Ġthe Ġinf ob ox es . ĠInf ob ox Q A Ġand ĠWiki Q A Ġshow Ġthe Ġleast Ġquestion - answer Ġword Ġoverl aps Ġover Ġquestions Ġand Ġanswers Ġ( ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġin ĠTable ĠT AB REF 2 Ġ), Ġrespectively . ĠIn Ġterms Ġof Ġthe ĠF 1 - score Ġfor Ġoverlapping Ġwords Ġ( ĠIN LINE FORM 4 Ġ), ĠS Qu AD Ġgives Ġthe Ġleast Ġportion Ġof Ġoverl aps Ġbetween Ġquestion - answer Ġpairs Ġalthough ĠWiki Q A Ġcomes Ġvery Ġclose . </s> Fig . ĠFIG REF 4 Ġshows Ġthe Ġdistributions Ġof Ġseven Ġquestion Ġtypes Ġgrouped Ġdeter minist ically Ġfrom Ġthe Ġlex icons . ĠAlthough Ġthese Ġcorpor a Ġhave Ġbeen Ġindependently Ġdeveloped , Ġa Ġgeneral Ġtrend Ġis Ġfound , Ġwhere Ġthe Ġwhat Ġquestion Ġtype Ġdominates , Ġfollowed Ġby Ġhow Ġand Ġwho , Ġfollowed Ġby Ġwhen Ġand Ġwhere , Ġand Ġso Ġon . </s> Fig . ĠFIG REF 6 Ġshows Ġthe Ġdistributions Ġof Ġanswer Ġcategories Ġautomatically Ġclassified Ġby Ġour ĠConv olution al ĠNeural ĠNetwork Ġmodel Ġtrained Ġon Ġthe Ġdata Ġdistributed Ġby Ġli : 02 a . ĠInterestingly , Ġeach Ġcorpus Ġfocuses Ġon Ġdifferent Ġcategories , ĠN umeric Ġfor ĠWiki Q A Ġand ĠSel Q A , ĠEntity Ġfor ĠS Qu AD , Ġand ĠPerson Ġfor ĠInf ob ox Q A , Ġwhich Ġgives Ġenough Ġdivers ities Ġfor Ġstatistical Ġlearning Ġto Ġbuild Ġrobust Ġmodels . </s> Answer ĠRet ri eval </s> This Ġsection Ġdescribes Ġanother Ġselection - based ĠQ A Ġtask , Ġcalled Ġanswer Ġretrieval , Ġthat Ġfinds Ġthe Ġanswer Ġcontext Ġfrom Ġa Ġlarger Ġdataset , Ġthe Ġentire ĠWikipedia . ĠS Qu AD Ġprovides Ġno Ġmapping Ġof Ġthe Ġanswer Ġcontexts Ġto ĠWikipedia , Ġwhereas ĠWiki Q A Ġand ĠSel Q A Ġprovide Ġm appings ; Ġhowever , Ġtheir Ġdata Ġdo Ġnot Ġcome Ġfrom Ġthe Ġsame Ġversion Ġof ĠWikipedia . ĠWe Ġpropose Ġan Ġautomatic Ġway Ġof Ġmapping Ġthe Ġanswer Ġcontexts Ġfrom Ġall Ġcorpor a Ġto Ġthe Ġsame Ġversion Ġof ĠWik ip eda Ġso Ġthey Ġcan Ġbe Ġcoherent ly Ġused Ġfor Ġanswer Ġretrieval . </s> Each Ġparagraph Ġin ĠWikipedia Ġis Ġfirst Ġindexed Ġby ĠLuc ene Ġusing Ġ{ 1 , 2 , 3 } - gram s , Ġwhere Ġthe Ġparagraphs Ġare Ġseparated Ġby ĠWiki Ext ractor Ġand Ġsegment ed Ġby ĠN LP 4 J Ġ( 28 . 7 M + Ġparagraphs Ġare Ġindexed ). ĠEach Ġanswer Ġsentence Ġfrom Ġthe Ġcorpor a Ġin ĠTable ĠT AB REF 3 Ġis Ġthen Ġquer ied Ġto ĠLuc ene , Ġand Ġthe Ġtop - 5 Ġranked Ġparagraphs Ġare Ġretrieved . ĠThe Ġcos ine Ġsimilarity Ġbetween Ġeach Ġsentence Ġin Ġthese Ġparagraphs Ġand Ġthe Ġanswer Ġsentence Ġis Ġmeasured Ġfor ĠIN LINE FORM 0 Ġ- gram s , Ġsay ĠIN LINE FORM 1 Ġ. ĠA Ġweight Ġis Ġassigned Ġto Ġeach ĠIN LINE FORM 2 Ġ- gram Ġscore , Ġsay ĠIN LINE FORM 3 Ġ, Ġand Ġthe Ġweighted Ġsum Ġis Ġmeasured : ĠIN LINE FORM 4 Ġ. ĠThe Ġfixed Ġweights Ġof ĠIN LINE FORM 5 Ġare Ġused Ġfor Ġour Ġexperiments , Ġwhich Ġcan Ġbe Ġimproved . </s> If Ġthere Ġexists Ġa Ġsentence Ġwhose ĠIN LINE FORM 0 Ġ, Ġthe Ġparagraph Ġconsisting Ġof Ġthat Ġsentence Ġis Ġconsidered Ġthe Ġsilver - standard Ġanswer Ġpassage . ĠTable ĠT AB REF 3 Ġshows Ġhow Ġrobust Ġthese Ġsilver - standard Ġpassages Ġare Ġbased Ġon Ġhuman Ġjudgement Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġhow Ġmany Ġpassages Ġare Ġcollected Ġ( ĠIN LINE FORM 2 Ġ) Ġfor ĠIN LINE FORM 3 Ġ, Ġwhere Ġthe Ġhuman Ġjudgement Ġis Ġperformed Ġon Ġ50 Ġrandom Ġsamples Ġfor Ġeach Ġcase . ĠFor Ġanswer Ġretrieval , Ġa Ġdataset Ġis Ġcreated Ġby ĠIN LINE FORM 4 Ġ, Ġwhich Ġgives ĠIN LINE FORM 5 Ġaccuracy Ġand ĠIN LINE FORM 6 Ġcoverage , Ġrespectively . ĠFinally , Ġeach Ġquestion Ġis Ġquer ied Ġto ĠLuc ene Ġand Ġthe Ġtop - ĠIN LINE FORM 7 Ġparagraphs Ġare Ġretrieved Ġfrom Ġthe Ġentire ĠWikipedia . ĠIf Ġthe Ġanswer Ġsentence Ġexists Ġwithin Ġthose Ġretrieved Ġparagraphs Ġaccording Ġto Ġthe Ġsilver - standard , Ġit Ġis Ġconsidered Ġcorrect . </s> Finding Ġa Ġparagraph Ġthat Ġincludes Ġthe Ġanswer Ġcontext Ġout Ġof Ġthe Ġentire ĠWikipedia Ġis Ġan Ġextremely Ġdifficult Ġtask Ġ( 128 . 7 M ). ĠThe Ġlast Ġrow Ġof ĠTable ĠT AB REF 3 Ġshows Ġresults Ġfrom Ġanswer Ġretrieval . ĠGiven ĠIN LINE FORM 0 Ġ, ĠSel Q A Ġand ĠS Qu AD Ġshow Ġabout Ġ34 % Ġand Ġ35 % Ġaccuracy , Ġwhich Ġare Ġreasonable . ĠHowever , ĠWiki Q A Ġshows Ġa Ġsignificantly Ġlower Ġaccuracy Ġof Ġ12 . 47 %; Ġthis Ġis Ġbecause Ġthe Ġquestions Ġin ĠWiki Q A Ġis Ġabout Ġtwice Ġshorter Ġthan Ġthe Ġquestions Ġin Ġthe Ġother Ġcorpor a Ġsuch Ġthat Ġnot Ġenough Ġlex icons Ġcan Ġbe Ġextracted Ġfrom Ġthese Ġquestions Ġfor Ġthe ĠLuc ene Ġsearch . </s> Answer ĠSelection </s> Answer Ġselection Ġis Ġevaluated Ġby Ġtwo Ġmetrics , Ġmean Ġaverage Ġprecision Ġ( MAP ) Ġand Ġmean Ġreciprocal Ġrank Ġ( MR R ). ĠThe Ġbig ram ĠCNN Ġintroduced Ġby Ġy u : 14 a Ġis Ġused Ġto Ġgenerate Ġall Ġthe Ġresults Ġin ĠTable ĠT AB REF 11 Ġ, Ġwhere Ġmodels Ġare Ġtrained Ġon Ġeither Ġsingle Ġor Ġcombined Ġdatasets . ĠClearly , Ġthe Ġquestions Ġin ĠWiki Q A Ġare Ġthe Ġmost Ġchallenging , Ġand Ġadding Ġmore Ġtraining Ġdata Ġfrom Ġthe Ġother Ġcorpor a Ġhurts Ġaccuracy Ġdue Ġto Ġthe Ġuniqueness Ġof Ġquery - based Ġquestions Ġin Ġthis Ġcorpus . ĠThe Ġbest Ġmodel Ġis Ġachieved Ġby Ġtraining Ġon ĠW + S + Q Ġfor ĠSel Q A ; Ġadding ĠInf ob ox Q A Ġhurts Ġaccuracy Ġfor ĠSel Q A Ġalthough Ġit Ġgives Ġa Ġmarginal Ġgain Ġfor ĠS Qu AD . ĠJust Ġlike ĠWiki Q A , ĠInf ob ox Q A Ġperforms Ġthe Ġbest Ġwhen Ġit Ġis Ġtrained Ġon Ġonly Ġitself . ĠFrom Ġour Ġanalysis , Ġwe Ġsuggest Ġthat Ġto Ġuse Ġmodels Ġtrained Ġon ĠWiki Q A Ġand ĠInf ob ox Q A Ġfor Ġshort Ġquery - like Ġquestions , Ġwhereas Ġto Ġuse Ġones Ġtrained Ġon ĠSel Q A Ġand ĠS Qu AD Ġfor Ġlong Ġnatural Ġquestions . </s> Answer ĠTrigger ing </s> The Ġresults Ġof ĠIN LINE FORM 0 Ġfrom Ġthe Ġanswer Ġretrieval Ġtask Ġin ĠSection ĠSEC REF 13 Ġare Ġused Ġto Ġcreate Ġthe Ġdatasets Ġfor Ġanswer Ġtriggering , Ġwhere Ġabout Ġ65 % Ġof Ġthe Ġquestions Ġare Ġnot Ġexpected Ġto Ġfind Ġtheir Ġanswer Ġcontexts Ġfrom Ġthe Ġprovided Ġparagraphs Ġfor ĠSel Q A Ġand ĠS Qu AD Ġand Ġ87 . 5 % Ġare Ġnot Ġexpected Ġfor ĠWiki Q A . ĠAnswer Ġtriggering Ġis Ġevaluated Ġby Ġthe ĠF 1 Ġscores Ġas Ġpresented Ġin ĠTable ĠT AB REF 11 Ġ, Ġwhere Ġthree Ġcorpor a Ġare Ġcross Ġvalidated . ĠThe Ġresults Ġon ĠWiki Q A Ġare Ġpretty Ġlow Ġas Ġexpected Ġfrom Ġthe Ġpoor Ġaccuracy Ġon Ġthe Ġanswer Ġretrieval Ġtask . ĠTraining Ġon ĠSel Q A Ġgives Ġthe Ġbest Ġmodels Ġfor Ġboth ĠWiki Q A Ġand ĠSel Q A . ĠTraining Ġon ĠS Qu AD Ġgives Ġthe Ġbest Ġmodel Ġfor ĠS Qu AD Ġalthough Ġthe Ġmodel Ġtrained Ġon ĠSel Q A Ġis Ġcomparable . ĠSince Ġthe Ġanswer Ġtriggering Ġdatasets Ġare Ġabout Ġ5 Ġtimes Ġlarger Ġthan Ġthe Ġanswer Ġselection Ġdatasets , Ġit Ġis Ġcomput ationally Ġtoo Ġexpensive Ġto Ġcombine Ġall Ġdata Ġfor Ġtraining . ĠWe Ġplan Ġto Ġfind Ġa Ġstrong Ġmachine Ġto Ġperform Ġthis Ġexperiment Ġin Ġnear Ġfuture . </s> Related Ġwork </s> L ately , Ġseveral Ġdeep Ġlearning Ġapproaches Ġhave Ġbeen Ġproposed Ġfor Ġquestion Ġanswering . Ġy u : 14 a Ġpresented Ġa ĠCNN Ġmodel Ġthat Ġrecognizes Ġthe Ġsemantic Ġsimilarity Ġbetween Ġtwo Ġsentences . Ġw ang - ny berg : 2015 : AC L - IJ CN LP Ġpresented Ġa Ġstacked Ġbid irection al ĠL ST M Ġapproach Ġto Ġread Ġwords Ġin Ġsequence , Ġthen Ġoutputs Ġtheir Ġsimilarity Ġscores . Ġf eng : 15 a Ġapplied Ġa Ġgeneral Ġdeep Ġlearning Ġframework Ġto Ġnon - fact oid Ġquestion Ġanswering . Ġs ant os : 16 a Ġintroduced Ġan Ġattentive Ġpool ing Ġmechanism Ġthat Ġled Ġto Ġfurther Ġimprovements Ġin Ġselection - based ĠQ A . </s> Conclusion </s> We Ġpresent Ġa Ġcomprehensive Ġcomparison Ġstudy Ġof Ġthe Ġexisting Ġcorpor a Ġfor Ġselection - based Ġquestion Ġanswering . ĠOur Ġintrinsic Ġanalysis Ġprovides Ġa Ġbetter Ġunderstanding Ġof Ġthe Ġuniqueness Ġor Ġsimilarity Ġbetween Ġthese Ġcorpor a . ĠOur Ġextr ins ic Ġanalysis Ġshows Ġthe Ġstrength Ġor Ġweakness Ġof Ġcombining Ġthese Ġcorpor a Ġtogether Ġfor Ġstatistical Ġlearning . ĠAdditionally , Ġwe Ġcreate Ġa Ġsilver - standard Ġdataset Ġfor Ġanswer Ġretrieval Ġand Ġtriggering , Ġwhich Ġwill Ġbe Ġpublicly Ġavailable . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġexplore Ġdifferent Ġways Ġof Ġimproving Ġthe Ġquality Ġof Ġour Ġsilver - standard Ġdatasets Ġby Ġfine - tun ing Ġthe Ġhyper - param eters .