INLINEFORM2 
EXTRACTIVE
Answer with content missing: (Table 2) Accuracy of best AS reader results including ensembles are 78.4 and 83.7 when trained on BookTest compared to 71.0 and 68.9 when trained on CBT for Named endity and Common noun respectively.
ABSTRACTIVE
If we take the best psr ensemble trained on CBT as a baseline, improving the model architecture as in BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , continuing to use the original CBT training data, lead to improvements of INLINEFORM0 and INLINEFORM1 absolute on named entities and common nouns respectively. By contrast, inflating the training dataset provided a boost of INLINEFORM2 while using the same model. The ensemble of our models even exceeded the human baseline provided by Facebook BIBREF2 on the Common Noun dataset.
Table TABREF25 shows the accuracy of the psr and other architectures on the CBT validation and test data. The last two rows show the performance of the psr trained on the BookTest dataset; all the other models have been trained on the original CBT training data.
1610.00956
Introduction </s> Since Ġhumans Ġam ass Ġmore Ġand Ġmore Ġgenerally Ġavailable Ġdata Ġin Ġthe Ġform Ġof Ġun struct ured Ġtext Ġit Ġwould Ġbe Ġvery Ġuseful Ġto Ġteach Ġmachines Ġto Ġread Ġand Ġcomprehend Ġsuch Ġdata Ġand Ġthen Ġuse Ġthis Ġunderstanding Ġto Ġanswer Ġour Ġquestions . ĠA Ġsignificant Ġamount Ġof Ġresearch Ġhas Ġrecently Ġfocused Ġon Ġanswering Ġone Ġparticular Ġkind Ġof Ġquestions Ġthe Ġanswer Ġto Ġwhich Ġdepends Ġon Ġunderstanding Ġa Ġcontext Ġdocument . ĠThese Ġare Ġclo ze - style Ġquestions ĠB IB REF 0 Ġwhich Ġrequire Ġthe Ġreader Ġto Ġfill Ġin Ġa Ġmissing Ġword Ġin Ġa Ġsentence . ĠAn Ġimportant Ġadvantage Ġof Ġsuch Ġquestions Ġis Ġthat Ġthey Ġcan Ġbe Ġgenerated Ġautomatically Ġfrom Ġa Ġsuitable Ġtext Ġcorpus Ġwhich Ġallows Ġus Ġto Ġproduce Ġa Ġpractically Ġunlimited Ġamount Ġof Ġthem . ĠThat Ġopens Ġthe Ġtask Ġto Ġnotoriously Ġdata - hung ry Ġdeep - learning Ġtechniques Ġwhich Ġnow Ġseem Ġto Ġoutper form Ġall Ġalternative Ġapproaches . </s> Two Ġsuch Ġlarge - scale Ġdatasets Ġhave Ġrecently Ġbeen Ġproposed Ġby Ġresearchers Ġfrom ĠGoogle ĠDeep Mind Ġand ĠFacebook ĠAI : Ġthe ĠCNN / Daily ĠMail Ġdataset ĠB IB REF 1 Ġand Ġthe ĠChildren 's ĠBook ĠTest Ġ( C BT ) ĠB IB REF 2 Ġrespectively . ĠThese Ġhave Ġattracted Ġa Ġlot Ġof Ġattention Ġfrom Ġthe Ġresearch Ġcommunity ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġwith Ġa Ġnew Ġstate - of - the - art Ġmodel Ġcoming Ġout Ġevery Ġfew Ġweeks . </s> However Ġif Ġour Ġgoal Ġis Ġa Ġproduction - level Ġsystem Ġactually Ġcapable Ġof Ġhelping Ġhumans , Ġwe Ġwant Ġthe Ġmodel Ġto Ġuse Ġall Ġavailable Ġresources Ġas Ġefficiently Ġas Ġpossible . ĠGiven Ġthat </s> we Ġbelieve Ġthat Ġif Ġthe Ġcommunity Ġis Ġstriving Ġto Ġbring Ġthe Ġperformance Ġas Ġfar Ġas Ġpossible , Ġit Ġshould Ġmove Ġits Ġwork Ġto Ġlarger Ġdata . </s> This Ġthinking Ġgoes Ġin Ġline Ġwith Ġrecent Ġdevelopments Ġin Ġthe Ġarea Ġof Ġlanguage Ġmodelling . ĠFor Ġa Ġlong Ġtime Ġmodels Ġwere Ġbeing Ġcompared Ġon Ġseveral Ġ" standard " Ġdatasets Ġwith Ġpublications Ġoften Ġpresenting Ġmin usc ule Ġimprovements Ġin Ġperformance . ĠThen Ġthe Ġlarge - scale ĠOne ĠBillion ĠWord Ġcorpus Ġdataset Ġappeared ĠB IB REF 15 Ġand Ġit Ġallowed ĠJo z ef owicz Ġet Ġal . Ġto Ġtrain Ġmuch Ġlarger ĠL ST M Ġmodels ĠB IB REF 16 Ġthat Ġalmost Ġhal ved Ġthe Ġstate - of - the - art Ġperplex ity Ġon Ġthis Ġdataset . </s> We Ġthink Ġit Ġis Ġtime Ġto Ġmake Ġa Ġsimilar Ġstep Ġin Ġthe Ġarea Ġof Ġtext Ġcomprehension . ĠHence Ġwe Ġare Ġintroducing Ġthe ĠBook Test , Ġa Ġnew Ġdataset Ġvery Ġsimilar Ġto Ġthe ĠChildren 's ĠBook Ġtest Ġbut Ġmore Ġthan Ġ60 Ġtimes Ġlarger Ġto Ġenable Ġtraining Ġlarger Ġmodels Ġeven Ġin Ġthe Ġdomain Ġof Ġtext Ġcomprehension . ĠFurthermore Ġthe Ġmethodology Ġused Ġto Ġcreate Ġour Ġdata Ġcan Ġlater Ġbe Ġused Ġto Ġcreate Ġeven Ġlarger Ġdatasets Ġwhen Ġthe Ġneed Ġarises Ġthanks Ġto Ġfurther Ġtechnological Ġprogress . </s> We Ġshow Ġthat Ġif Ġwe Ġevaluate Ġa Ġmodel Ġtrained Ġon Ġthe Ġnew Ġdataset Ġon Ġthe Ġnow Ġstandard ĠChildren 's ĠBook ĠTest Ġdataset , Ġwe Ġsee Ġan Ġimprovement Ġin Ġaccuracy Ġmuch Ġlarger Ġthan Ġother Ġresearch Ġgroups Ġachieved Ġby Ġenhancing Ġthe Ġmodel Ġarchitecture Ġitself Ġ( while Ġstill Ġusing Ġthe Ġoriginal ĠCB T Ġtraining Ġdata ). ĠBy Ġtraining Ġon Ġthe Ġnew Ġdataset , Ġwe Ġreduce Ġthe Ġprediction Ġerror Ġby Ġalmost Ġone Ġthird . ĠOn Ġthe Ġnamed - entity Ġversion Ġof ĠCB T Ġthis Ġbrings Ġthe Ġensemble Ġof Ġour Ġmodels Ġto Ġthe Ġlevel Ġof Ġhuman Ġbaseline Ġas Ġreported Ġby ĠFacebook ĠB IB REF 2 Ġ. ĠHowever Ġin Ġthe Ġfinal Ġsection Ġwe Ġshow Ġin Ġour Ġown Ġhuman Ġstudy Ġthat Ġthere Ġis Ġstill Ġroom Ġfor Ġimprovement Ġon Ġthe ĠCB T Ġbeyond Ġthe Ġperformance Ġof Ġour Ġmodel . </s> Task ĠDescription </s> A Ġnatural Ġway Ġof Ġtesting Ġa Ġreader 's Ġcomprehension Ġof Ġa Ġtext Ġis Ġto Ġask Ġher Ġa Ġquestion Ġthe Ġanswer Ġto Ġwhich Ġcan Ġbe Ġded uced Ġfrom Ġthe Ġtext . ĠHence Ġthe Ġtask Ġwe Ġare Ġtrying Ġto Ġsolve Ġconsists Ġof Ġanswering Ġa Ġclo ze - style Ġquestion , Ġthe Ġanswer Ġto Ġwhich Ġdepends Ġon Ġthe Ġunderstanding Ġof Ġa Ġcontext Ġdocument Ġprovided Ġwith Ġthe Ġquestion . ĠThe Ġmodel Ġis Ġalso Ġprovided Ġwith Ġa Ġset Ġof Ġpossible Ġanswers Ġfrom Ġwhich Ġthe Ġcorrect Ġone Ġis Ġto Ġbe Ġselected . ĠThis Ġcan Ġbe Ġformal ized Ġas Ġfollows : </s> The Ġtraining Ġdata Ġconsist Ġof Ġtu ples ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġquestion , ĠIN LINE FORM 2 Ġis Ġa Ġdocument Ġthat Ġcontains Ġthe Ġanswer Ġto Ġquestion ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġis Ġa Ġset Ġof Ġpossible Ġanswers Ġand ĠIN LINE FORM 5 Ġis Ġthe Ġground - truth Ġanswer . ĠBoth ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġare Ġsequences Ġof Ġwords Ġfrom Ġvocabulary ĠIN LINE FORM 8 Ġ. ĠWe Ġalso Ġassume Ġthat Ġall Ġpossible Ġanswers Ġare Ġwords Ġfrom Ġthe Ġvocabulary , Ġthat Ġis ĠIN LINE FORM 9 Ġ. ĠIn Ġthe ĠCB T Ġand ĠCNN / Daily ĠMail Ġdatasets Ġit Ġis Ġalso Ġtrue Ġthat Ġthe Ġground - truth Ġanswer ĠIN LINE FORM 10 Ġappears Ġin Ġthe Ġdocument . ĠThis Ġis Ġexploited Ġby Ġmany Ġmachine Ġlearning Ġmodels ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġhowever Ġsome Ġdo Ġnot Ġexplicitly Ġdepend Ġon Ġthis Ġproperty ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 9 </s> Current ĠLand scape </s> We Ġwill Ġnow Ġbriefly Ġreview Ġwhat Ġdatasets Ġfor Ġtext Ġcomprehension Ġhave Ġbeen Ġpublished Ġup Ġto Ġdate Ġand Ġlook Ġat Ġmodels Ġwhich Ġhave Ġbeen Ġrecently Ġapplied Ġto Ġsolving Ġthe Ġtask Ġwe Ġhave Ġjust Ġdescribed . </s> Dat as ets </s> A Ġcrucial Ġcondition Ġfor Ġapplying Ġdeep - learning Ġtechniques Ġis Ġto Ġhave Ġa Ġhuge Ġamount Ġof Ġdata Ġavailable Ġfor Ġtraining . ĠFor Ġquestion Ġanswering Ġthis Ġspecifically Ġmeans Ġhaving Ġa Ġlarge Ġnumber Ġof Ġdocument - question - answer Ġtri ples Ġavailable . ĠWhile Ġthere Ġis Ġan Ġunlimited Ġamount Ġof Ġtext Ġavailable , Ġcoming Ġup Ġwith Ġrelevant Ġquestions Ġand Ġthe Ġcorresponding Ġanswers Ġcan Ġbe Ġextremely Ġlabour - intensive Ġif Ġdone Ġby Ġhuman Ġannot ators . ĠThere Ġwere Ġefforts Ġto Ġprovide Ġsuch Ġhuman - generated Ġdatasets , Ġe . g . ĠMicrosoft 's ĠM CT est ĠB IB REF 17 Ġ, Ġhowever Ġtheir Ġscale Ġis Ġnot Ġsuitable Ġfor Ġdeep Ġlearning Ġwithout Ġpre - training Ġon Ġother Ġdata ĠB IB REF 18 Ġ( such Ġas Ġusing Ġpre - trained Ġword Ġembed ding Ġvectors ). </s> Google ĠDeep Mind Ġmanaged Ġto Ġavoid Ġthis Ġscale Ġissue Ġwith Ġtheir Ġway Ġof Ġgenerating Ġdocument - question - answer Ġtri ples Ġautomatically , Ġclosely Ġfollowed Ġby ĠFacebook Ġwith Ġa Ġsimilar Ġmethod . ĠLet Ġus Ġnow Ġbriefly Ġintroduce Ġthe Ġtwo Ġresulting Ġdatasets Ġwhose Ġproperties Ġare Ġsummarized Ġin ĠTable ĠT AB REF 8 Ġ. </s> These Ġtwo Ġdatasets ĠB IB REF 1 Ġexploit Ġa Ġuseful Ġfeature Ġof Ġonline Ġnews Ġarticles ĠâĢĵ Ġmany Ġarticles Ġinclude Ġa Ġshort Ġsummar izing Ġsentence Ġnear Ġthe Ġtop Ġof Ġthe Ġpage . ĠSince Ġall Ġinformation Ġin Ġthe Ġsummary Ġsentence Ġis Ġalso Ġpresented Ġin Ġthe Ġarticle Ġbody , Ġwe Ġget Ġa Ġnice Ġclo ze - style Ġquestion Ġabout Ġthe Ġarticle Ġcontents Ġby Ġremoving Ġa Ġword Ġfrom Ġthe Ġshort Ġsummary . </s> The Ġdataset 's Ġauthors Ġalso Ġreplaced Ġall Ġnamed Ġentities Ġin Ġthe Ġdataset Ġby Ġanonymous Ġtokens Ġwhich Ġare Ġfurther Ġshuff led Ġfor Ġeach Ġnew Ġbatch . ĠThis Ġforces Ġthe Ġmodel Ġto Ġrely Ġsolely Ġon Ġinformation Ġfrom Ġthe Ġcontext Ġdocument , Ġnot Ġbeing Ġable Ġto Ġtransfer Ġany Ġmeaning Ġof Ġthe Ġnamed Ġentities Ġbetween Ġdocuments . </s> This Ġrestricts Ġthe Ġtask Ġto Ġone Ġspecific Ġaspect Ġof Ġcontext - dependent Ġquestion Ġanswering Ġwhich Ġmay Ġbe Ġuseful Ġhowever Ġit Ġmoves Ġthe Ġtask Ġfurther Ġfrom Ġthe Ġreal Ġapplication Ġscenario , Ġwhere Ġwe Ġwould Ġlike Ġthe Ġmodel Ġto Ġuse Ġall Ġinformation Ġavailable Ġto Ġanswer Ġquestions . ĠFurthermore ĠChen Ġet Ġal . ĠB IB REF 5 Ġhave Ġsuggested Ġthat Ġthis Ġcan Ġmake Ġabout Ġ17 % Ġof Ġthe Ġquestions Ġun answer able Ġeven Ġby Ġhumans . ĠThey Ġalso Ġclaim Ġthat Ġmore Ġthan Ġa Ġhalf Ġof Ġthe Ġquestion Ġsentences Ġare Ġmere Ġparaph r ases Ġor Ġexact Ġmatches Ġof Ġa Ġsingle Ġsentence Ġfrom Ġthe Ġcontext Ġdocument . ĠThis Ġraises Ġa Ġquestion Ġto Ġwhat Ġextent Ġthe Ġdataset Ġcan Ġtest Ġdeeper Ġunderstanding Ġof Ġthe Ġarticles . </s> The ĠChildren 's ĠBook ĠTest ĠB IB REF 2 Ġuses Ġa Ġdifferent Ġsource Ġ- Ġbooks Ġfreely Ġavailable Ġthanks Ġto ĠProject ĠGutenberg . ĠSince Ġno Ġsummary Ġis Ġavailable , Ġeach Ġexample Ġconsists Ġof Ġa Ġcontext Ġdocument Ġformed Ġfrom Ġ20 Ġconsecutive Ġsentences Ġfrom Ġthe Ġstory Ġtogether Ġwith Ġa Ġquestion Ġformed Ġfrom Ġthe Ġsubsequent Ġsentence . </s> The Ġdataset Ġcomes Ġin Ġfour Ġflavours Ġdepending Ġon Ġwhat Ġtype Ġof Ġword Ġis Ġomitted Ġfrom Ġthe Ġquestion Ġsentence . ĠBased Ġon Ġhuman Ġevaluation Ġdone Ġin ĠB IB REF 2 Ġit Ġseems Ġthat ĠNE Ġand ĠCN Ġare Ġmore Ġcontext Ġdependent Ġthan Ġthe Ġother Ġtwo Ġtypes ĠâĢĵ Ġpre pos itions Ġand Ġverbs . ĠTherefore Ġwe Ġ( and Ġall Ġof Ġthe Ġrecent Ġpublications ) Ġfocus Ġonly Ġon Ġthese Ġtwo Ġword Ġtypes . </s> Several Ġnew Ġdatasets Ġrelated Ġto Ġthe Ġ( now Ġalmost Ġstandard ) Ġones Ġabove Ġemerged Ġrecently . ĠWe Ġwill Ġnow Ġbriefly Ġpresent Ġthem Ġand Ġexplain Ġhow Ġthe Ġdataset Ġwe Ġare Ġintroducing Ġin Ġthis Ġarticle Ġdiffers Ġfrom Ġthem . </s> The ĠL AM B ADA Ġdataset ĠB IB REF 19 Ġis Ġdesigned Ġto Ġmeasure Ġprogress Ġin Ġunderstanding Ġcommon - sense Ġquestions Ġabout Ġshort Ġstories Ġthat Ġcan Ġbe Ġeasily Ġanswered Ġby Ġhumans Ġbut Ġcannot Ġbe Ġanswered Ġby Ġcurrent Ġstandard Ġmachine - learning Ġmodels Ġ( e . g . Ġplain ĠL ST M Ġlanguage Ġmodels ). ĠThis Ġdataset Ġis Ġuseful Ġfor Ġmeasuring Ġthe Ġgap Ġbetween Ġhumans Ġand Ġmachine Ġlearning Ġalgorithms . ĠHowever , Ġby Ġcontrast Ġto Ġour ĠBook Test Ġdataset , Ġit Ġwill Ġnot Ġallow Ġus Ġto Ġtrack Ġprogress Ġtowards Ġthe Ġperformance Ġof Ġthe Ġbaseline Ġsystems Ġor Ġon Ġexamples Ġwhere Ġmachine Ġlearning Ġmay Ġshow Ġsuper - human Ġperformance . ĠAlso ĠL AM B ADA Ġis Ġjust Ġa Ġdiagnostic Ġdataset Ġand Ġdoes Ġnot Ġprovide Ġready - to - use Ġquestion - ans w ering Ġtraining Ġdata , Ġjust Ġa Ġplain - text Ġcorpus Ġwhich Ġmay Ġmoreover Ġinclude Ġcopyrighted Ġbooks Ġmaking Ġits Ġuse Ġpotentially Ġproblematic Ġfor Ġsome Ġpurposes . ĠWe Ġare Ġproviding Ġready Ġtraining Ġdata Ġconsisting Ġof Ġcopyright - free Ġbooks Ġonly . </s> The ĠS Qu AD Ġdataset ĠB IB REF 20 Ġbased Ġon ĠWikipedia Ġand Ġthe ĠWho - did - What Ġdataset ĠB IB REF 21 Ġbased Ġon ĠGig aw ord Ġnews Ġarticles Ġare Ġfact oid Ġquestion - ans w ering Ġdatasets Ġwhere Ġa Ġmulti - word Ġanswer Ġshould Ġbe Ġextracted Ġfrom Ġa Ġcontext Ġdocument . ĠThis Ġis Ġin Ġcontrast Ġto Ġthe Ġprevious Ġdatasets , Ġincluding ĠCNN / DM , ĠCB T , ĠL AM B ADA Ġand Ġour Ġnew Ġdataset , Ġwhich Ġrequire Ġonly Ġsingle - word Ġanswers . ĠBoth Ġthese Ġdatasets Ġhowever Ġprovide Ġless Ġthan Ġ130 , 000 Ġtraining Ġquestions , Ġtwo Ġorders Ġof Ġmagnitude Ġless Ġthan Ġour Ġdataset Ġdoes . </s> The ĠStory ĠClo ze ĠTest ĠB IB REF 22 Ġprovides Ġa Ġcrowd - s ourced Ġcorpus Ġof Ġ49 , 255 Ġcommons ense Ġstories Ġfor Ġtraining Ġand Ġ3 , 7 44 Ġtesting Ġstories Ġwith Ġright Ġand Ġwrong Ġendings . ĠHence Ġthe Ġdataset Ġis Ġagain Ġrather Ġsmall . ĠSimilarly Ġto ĠL AM B ADA , Ġthe ĠStory ĠClo ze ĠTest Ġwas Ġdesigned Ġto Ġbe Ġeasily Ġanswer able Ġby Ġhumans . </s> In Ġthe ĠWiki Reading ĠB IB REF 23 Ġdataset Ġthe Ġcontext Ġdocument Ġis Ġformed Ġfrom Ġa ĠWikipedia Ġarticle Ġand Ġthe Ġquestion - answer Ġpair Ġis Ġtaken Ġfrom Ġthe Ġcorresponding ĠWiki Data Ġpage . ĠFor Ġeach Ġentity Ġ( e . g . ĠHillary ĠClinton ), ĠWiki Data Ġcontain Ġa Ġnumber Ġof Ġproperty - value Ġpairs Ġ( e . g . Ġplace Ġof Ġbirth : ĠChicago ) Ġwhich Ġform Ġthe Ġdatasets 's Ġquestion - answer Ġpairs . ĠThe Ġdataset Ġis Ġcertainly Ġrelevant Ġto Ġthe Ġcommunity , Ġhowever Ġthe Ġquestions Ġare Ġof Ġvery Ġlimited Ġvariety Ġwith Ġonly Ġ20 Ġproperties Ġ( and Ġhence Ġunique Ġquestions ) Ġcovering ĠIN LINE FORM 0 Ġof Ġthe Ġdataset . ĠFurthermore Ġmany Ġof Ġthe Ġfrequent Ġproperties Ġare Ġmentioned Ġat Ġa Ġset Ġspot Ġwithin Ġthe Ġarticle Ġ( e . g . Ġthe Ġdate Ġof Ġbirth Ġis Ġalmost Ġalways Ġin Ġbrackets Ġbehind Ġthe Ġname Ġof Ġa Ġperson ) Ġwhich Ġmay Ġmake Ġthe Ġtask Ġeasier Ġfor Ġmachines . ĠWe Ġare Ġtrying Ġto Ġprovide Ġa Ġmore Ġvaried Ġdataset . </s> Although Ġthere Ġare Ġseveral Ġdatasets Ġrelated Ġto Ġtask Ġwe Ġare Ġaiming Ġto Ġsolve , Ġthey Ġdiffer Ġsufficiently Ġfor Ġour Ġdataset Ġto Ġbring Ġnew Ġvalue Ġto Ġthe Ġcommunity . ĠIts Ġbiggest Ġadvantage Ġis Ġits Ġsize Ġwhich Ġcan Ġfurthermore Ġbe Ġeasily Ġup sc aled Ġwithout Ġexpensive Ġhuman Ġannotation . ĠFinally Ġwhile Ġwe Ġare Ġemphasizing Ġthe Ġdifferences , Ġmodels Ġcould Ġcertainly Ġbenefit Ġfrom Ġas Ġdiverse Ġa Ġcollection Ġof Ġdatasets Ġas Ġpossible . </s> Machine ĠLearning ĠModels </s> A Ġfirst Ġmajor Ġwork Ġapplying Ġdeep - learning Ġtechniques Ġto Ġtext Ġcomprehension Ġwas ĠHerman n Ġet Ġal . ĠB IB REF 1 Ġ. ĠThis Ġwork Ġwas Ġfollowed Ġby Ġthe Ġapplication Ġof ĠMemory ĠNetworks Ġto Ġthe Ġsame Ġtask ĠB IB REF 2 Ġ. ĠLater Ġthree Ġmodels Ġemerged Ġaround Ġthe Ġsame Ġtime ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġincluding Ġour Ġps r Ġmodel ĠB IB REF 4 Ġ. ĠThe ĠAS ĠReader Ġinspired Ġseveral Ġsubsequent Ġmodels Ġthat Ġuse Ġit Ġas Ġa Ġsub - component Ġin Ġa Ġdiverse Ġensemble ĠB IB REF 8 Ġ; Ġextend Ġit Ġwith Ġa Ġhierarchical Ġstructure ĠB IB REF 6 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 7 Ġ; Ġcompute Ġattention Ġover Ġthe Ġcontext Ġdocument Ġfor Ġevery Ġword Ġin Ġthe Ġquery ĠB IB REF 10 Ġor Ġuse Ġtwo - way Ġcontext - query Ġattention Ġmechanism Ġfor Ġevery Ġword Ġin Ġthe Ġcontext Ġand Ġthe Ġquery ĠB IB REF 11 Ġthat Ġis Ġsimilar Ġin Ġits Ġspirit Ġto Ġmodels Ġrecently Ġproposed Ġin Ġdifferent Ġdomains , Ġe . g . ĠB IB REF 25 Ġin Ġinformation Ġretrieval . ĠOther Ġneural Ġapproaches Ġto Ġtext Ġcomprehension Ġare Ġexplored Ġin ĠB IB REF 9 Ġ, ĠB IB REF 12 Ġ. </s> P ossible ĠDirections Ġfor ĠImprovements </s> Acc uracy Ġin Ġany Ġmachine Ġlearning Ġtasks Ġcan Ġbe Ġenhanced Ġeither Ġby Ġimproving Ġa Ġmachine Ġlearning Ġmodel Ġor Ġby Ġusing Ġmore Ġin - domain Ġtraining Ġdata . ĠCurrent Ġstate Ġof Ġthe Ġart Ġmodels ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 11 Ġimprove Ġover ĠAS ĠReader 's Ġaccuracy Ġon ĠCB T ĠNE Ġand ĠCN Ġdatasets Ġby Ġ1 - 2 Ġpercent Ġabsolute . ĠThis Ġsuggests Ġthat Ġwith Ġcurrent Ġtechniques Ġthere Ġis Ġonly Ġlimited Ġroom Ġfor Ġimprovement Ġon Ġthe Ġalgorith mic Ġside . </s> The Ġother Ġpossibility Ġto Ġimprove Ġperformance Ġis Ġsimply Ġto Ġuse Ġmore Ġtraining Ġdata . ĠThe Ġimportance Ġof Ġtraining Ġdata Ġwas Ġhighlighted Ġby Ġthe Ġfrequently Ġquoted ĠMercer 's Ġstatement Ġthat ĠâĢ ľ There Ġis Ġno Ġdata Ġlike Ġmore Ġdata . âĢ Ŀ ĠThe Ġobservation Ġthat Ġhaving Ġmore Ġdata Ġis Ġoften Ġmore Ġimportant Ġthan Ġhaving Ġbetter Ġalgorithms Ġhas Ġbeen Ġfrequently Ġstressed Ġsince Ġthen ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> As Ġa Ġstep Ġin Ġthe Ġdirection Ġof Ġexploiting Ġthe Ġpotential Ġof Ġmore Ġdata Ġin Ġthe Ġdomain Ġof Ġtext Ġcomprehension , Ġwe Ġcreated Ġa Ġnew Ġdataset Ġcalled ĠBook Test Ġsimilar Ġto , Ġbut Ġmuch Ġlarger Ġthan Ġthe Ġwidely Ġused ĠCB T Ġand ĠCNN / DM Ġdatasets . </s> Book Test </s> Similarly Ġto Ġthe ĠCB T , Ġour ĠBook Test Ġdataset Ġis Ġderived Ġfrom Ġbooks Ġavailable Ġthrough Ġproject ĠGutenberg . ĠWe Ġused Ġ3 555 Ġcopyright - free Ġbooks Ġto Ġextract ĠCN Ġexamples Ġand Ġ10 507 Ġbooks Ġfor ĠNE Ġexamples , Ġfor Ġcomparison Ġthe ĠCB T Ġdataset Ġwas Ġextracted Ġfrom Ġjust Ġ108 Ġbooks . </s> When Ġcreating Ġour Ġdataset Ġwe Ġfollow Ġthe Ġsame Ġprocedure Ġas Ġwas Ġused Ġto Ġcreate Ġthe ĠCB T Ġdataset ĠB IB REF 2 Ġ. ĠThat Ġis , Ġwe Ġdetect Ġwhether Ġeach Ġsentence Ġcontains Ġeither Ġa Ġnamed Ġentity Ġor Ġa Ġcommon Ġnoun Ġthat Ġalready Ġappeared Ġin Ġone Ġof Ġthe Ġpreceding Ġtwenty Ġsentences . ĠThis Ġword Ġis Ġthen Ġreplaced Ġby Ġa Ġgap Ġtag Ġ( XXXX X ) Ġin Ġthis Ġsentence Ġwhich Ġis Ġhence Ġturned Ġinto Ġa Ġclo ze - style Ġquestion . ĠThe Ġpreceding Ġ20 Ġsentences Ġare Ġused Ġas Ġthe Ġcontext Ġdocument . ĠFor Ġcommon Ġnoun Ġand Ġnamed Ġentity Ġdetection Ġwe Ġuse Ġthe ĠStanford ĠPOS Ġtag ger ĠB IB REF 27 Ġand ĠStanford ĠN ER ĠB IB REF 28 Ġ. </s> The Ġtraining Ġdataset Ġconsists Ġof Ġthe Ġoriginal ĠCB T ĠNE Ġand ĠCN Ġdata Ġextended Ġwith Ġnew ĠNE Ġand ĠCN Ġexamples . ĠThe Ġnew ĠBook Test Ġdataset Ġhence Ġcontains ĠIN LINE FORM 0 Ġtraining Ġexamples Ġand ĠIN LINE FORM 1 Ġtokens . </s> The Ġvalidation Ġdataset Ġconsists Ġof ĠIN LINE FORM 0 ĠNE Ġand ĠIN LINE FORM 1 ĠCN Ġquestions . ĠWe Ġhave Ġone Ġtest Ġset Ġfor ĠNE Ġand Ġone Ġfor ĠCN , Ġeach Ġcontaining ĠIN LINE FORM 2 Ġexamples . ĠThe Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġwere Ġgenerated Ġfrom Ġnon - over l apping Ġsets Ġof Ġbooks . </s> When Ġgenerating Ġthe Ġdataset Ġwe Ġremoved Ġall Ġeditions Ġof Ġbooks Ġused Ġto Ġcreate ĠCB T Ġvalidation Ġand Ġtest Ġsets Ġfrom Ġour Ġtraining Ġdataset . ĠTherefore Ġthe Ġmodels Ġtrained Ġon Ġthe ĠBook Test Ġcorpus Ġcan Ġbe Ġevaluated Ġon Ġthe Ġoriginal ĠCB T Ġdata Ġand Ġthey Ġcan Ġbe Ġcompared Ġwith Ġrecent Ġtext - com pre hens ion Ġmodels Ġutilizing Ġthis Ġdataset ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. </s> Bas elines </s> We Ġwill Ġnow Ġuse Ġour Ġps r Ġmodel Ġto Ġevaluate Ġthe Ġperformance Ġgain Ġfrom Ġincreasing Ġthe Ġdataset Ġsize . </s> AS ĠReader </s> In ĠB IB REF 4 Ġwe Ġintroduced Ġthe Ġps r Ġ, Ġwhich Ġat Ġthe Ġtime Ġof Ġpublication Ġsignificantly Ġoutper formed Ġall Ġother Ġarchitectures Ġon Ġthe ĠCNN , ĠDM Ġand ĠCB T Ġdatasets . ĠThis Ġmodel Ġis Ġbuilt Ġto Ġleverage Ġthe Ġfact Ġthat Ġthe Ġanswer Ġis Ġa Ġsingle Ġword Ġfrom Ġthe Ġcontext Ġdocument . ĠSimilarly Ġto Ġmany Ġother Ġmodels Ġit Ġuses Ġattention Ġover Ġthe Ġdocument ĠâĢĵ Ġintuitive ly Ġa Ġmeasure Ġof Ġhow Ġrelevant Ġeach Ġword Ġis Ġto Ġanswering Ġthe Ġquestion . ĠHowever Ġwhile Ġmost Ġprevious Ġmodels Ġused Ġthis Ġattention Ġas Ġweights Ġto Ġcalculate Ġa Ġblended Ġrepresentation Ġof Ġthe Ġanswer Ġword , Ġwe Ġsimply Ġsum Ġthe Ġattention Ġacross Ġall Ġoccurrences Ġof Ġeach Ġunique Ġwords Ġand Ġthen Ġsimply Ġselect Ġthe Ġword Ġwith Ġthe Ġhighest Ġsum Ġas Ġthe Ġfinal Ġanswer . ĠWhile Ġsimple , Ġthis Ġtrick Ġseems Ġboth Ġto Ġimprove Ġaccuracy Ġand Ġto Ġspeed - up Ġtraining . ĠIt Ġwas Ġadopted Ġby Ġmany Ġsubsequent Ġmodels ĠB IB REF 8 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 24 Ġ. </s> Let Ġus Ġnow Ġdescribe Ġthe Ġmodel Ġin Ġmore Ġdetail . ĠFigure ĠFIG REF 21 Ġmay Ġhelp Ġyou Ġin Ġunderstanding Ġthe Ġfollowing Ġparagraphs . </s> The Ġwords Ġfrom Ġthe Ġdocument Ġand Ġthe Ġquestion Ġare Ġfirst Ġconverted Ġinto Ġvector Ġembed d ings Ġusing Ġa Ġlook - up Ġmatrix ĠIN LINE FORM 0 Ġ. ĠThe Ġdocument Ġis Ġthen Ġread Ġby Ġa Ġbid irection al ĠGR U Ġnetwork ĠB IB REF 29 Ġ. ĠA Ġconc aten ation Ġof Ġthe Ġhidden Ġstates Ġof Ġthe Ġforward Ġand Ġbackward ĠGR Us Ġat Ġeach Ġword Ġis Ġthen Ġused Ġas Ġa Ġcontextual Ġembed ding Ġof Ġthis Ġword , Ġintuitive ly Ġrepresenting Ġthe Ġcontext Ġin Ġwhich Ġthe Ġword Ġis Ġappearing . ĠWe Ġcan Ġalso Ġunderstand Ġit Ġas Ġrepresenting Ġthe Ġset Ġof Ġquestions Ġto Ġwhich Ġthis Ġword Ġmay Ġbe Ġan Ġanswer . </s> Similarly Ġthe Ġquestion Ġis Ġread Ġby Ġa Ġbid irection al ĠGR U Ġbut Ġin Ġthis Ġcase Ġonly Ġthe Ġfinal Ġhidden Ġstates Ġare Ġconc aten ated Ġto Ġform Ġthe Ġquestion Ġembed ding . </s> The Ġattention Ġover Ġeach Ġword Ġin Ġthe Ġcontext Ġis Ġthen Ġcalculated Ġas Ġthe Ġdot Ġproduct Ġof Ġits Ġcontextual Ġembed ding Ġwith Ġthe Ġquestion Ġembed ding . ĠThis Ġattention Ġis Ġthen Ġnormalized Ġby Ġthe Ġsoft max Ġfunction Ġand Ġsummed Ġacross Ġall Ġoccurrences Ġof Ġeach Ġanswer Ġcandidate . ĠThe Ġcandidate Ġwith Ġmost Ġaccumulated Ġattention Ġis Ġselected Ġas Ġthe Ġfinal Ġanswer . </s> For Ġa Ġmore Ġdetailed Ġdescription Ġof Ġthe Ġmodel Ġincluding Ġequations Ġcheck ĠB IB REF 4 Ġ. ĠMore Ġdetails Ġabout Ġthe Ġtraining Ġsetup Ġand Ġmodel Ġhyper param eters Ġcan Ġbe Ġfound Ġin Ġthe ĠAppendix . </s> During Ġour Ġpast Ġexperiments Ġon Ġthe ĠCNN , ĠDM Ġand ĠCB T Ġdatasets ĠB IB REF 4 Ġeach Ġunique Ġword Ġfrom Ġthe Ġtraining , Ġvalidation Ġand Ġtest Ġdatasets Ġhad Ġits Ġrow Ġin Ġthe Ġlook - up Ġmatrix ĠIN LINE FORM 0 Ġ. ĠHowever Ġas Ġwe Ġradically Ġincreased Ġthe Ġdataset Ġsize , Ġthis Ġwould Ġresult Ġin Ġan Ġextremely Ġlarge Ġnumber Ġof Ġmodel Ġparameters Ġso Ġwe Ġdecided Ġto Ġlimit Ġthe Ġvocabulary Ġsize Ġto ĠIN LINE FORM 1 Ġmost Ġfrequent Ġwords . ĠFor Ġeach Ġexample , Ġeach Ġunique Ġout - of - voc abulary Ġword Ġis Ġnow Ġmapped Ġon Ġone Ġof Ġ1000 Ġanonymous Ġtokens Ġwhich Ġare Ġrandomly Ġinitialized Ġand Ġunt rained . ĠFix ing Ġthe Ġembed d ings Ġof Ġthese Ġanonymous Ġtags Ġproved Ġto Ġsignificantly Ġimprove Ġthe Ġperformance . </s> While Ġmostly Ġusing Ġthe Ġoriginal ĠAS ĠReader Ġmodel , Ġwe Ġhave Ġalso Ġtried Ġintroducing Ġa Ġminor Ġtweak Ġin Ġsome Ġinstances Ġof Ġthe Ġmodel . ĠWe Ġtried Ġinitial izing Ġthe Ġcontext Ġenc oder ĠGR U 's Ġhidden Ġstate Ġby Ġletting Ġthe Ġenc oder Ġread Ġthe Ġquestion Ġfirst Ġbefore Ġproceeding Ġto Ġread Ġthe Ġcontext Ġdocument . ĠInt uitive ly Ġthis Ġallows Ġthe Ġenc oder Ġto Ġknow Ġin Ġadvance Ġwhat Ġto Ġlook Ġfor Ġwhen Ġreading Ġover Ġthe Ġcontext Ġdocument . </s> In cluding Ġmodels Ġof Ġthis Ġkind Ġin Ġthe Ġensemble Ġhelped Ġto Ġimprove Ġthe Ġperformance . </s> Results </s> Table ĠT AB REF 25 Ġshows Ġthe Ġaccuracy Ġof Ġthe Ġps r Ġand Ġother Ġarchitectures Ġon Ġthe ĠCB T Ġvalidation Ġand Ġtest Ġdata . ĠThe Ġlast Ġtwo Ġrows Ġshow Ġthe Ġperformance Ġof Ġthe Ġps r Ġtrained Ġon Ġthe ĠBook Test Ġdataset ; Ġall Ġthe Ġother Ġmodels Ġhave Ġbeen Ġtrained Ġon Ġthe Ġoriginal ĠCB T Ġtraining Ġdata . </s> If Ġwe Ġtake Ġthe Ġbest Ġps r Ġensemble Ġtrained Ġon ĠCB T Ġas Ġa Ġbaseline , Ġimproving Ġthe Ġmodel Ġarchitecture Ġas Ġin ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġcontinuing Ġto Ġuse Ġthe Ġoriginal ĠCB T Ġtraining Ġdata , Ġlead Ġto Ġimprovements Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġabsolute Ġon Ġnamed Ġentities Ġand Ġcommon Ġnoun s Ġrespectively . ĠBy Ġcontrast , Ġinfl ating Ġthe Ġtraining Ġdataset Ġprovided Ġa Ġboost Ġof ĠIN LINE FORM 2 Ġwhile Ġusing Ġthe Ġsame Ġmodel . ĠThe Ġensemble Ġof Ġour Ġmodels Ġeven Ġexceeded Ġthe Ġhuman Ġbaseline Ġprovided Ġby ĠFacebook ĠB IB REF 2 Ġon Ġthe ĠCommon ĠN oun Ġdataset . </s> Our Ġmodel Ġtakes Ġapproximately Ġtwo Ġweeks Ġto Ġconverge Ġwhen Ġtrained Ġon Ġthe ĠBook Test Ġdataset Ġon Ġa Ġsingle ĠNvidia ĠTesla ĠK 40 ĠGPU . </s> Discussion </s> Em br acing Ġthe Ġabundance Ġof Ġdata Ġmay Ġmean Ġfocusing Ġon Ġother Ġaspects Ġof Ġsystem Ġdesign Ġthan Ġwith Ġsmaller Ġdata . ĠHere Ġare Ġsome Ġof Ġthe Ġchallenges Ġthat Ġwe Ġneed Ġto Ġface Ġin Ġthis Ġsituation . </s> Firstly , Ġsince Ġthe Ġamount Ġof Ġdata Ġis Ġpractically Ġunlimited ĠâĢĵ Ġwe Ġcould Ġeven Ġgenerate Ġthem Ġon Ġthe Ġfly Ġresulting Ġin Ġcontinuous Ġlearning Ġsimilar Ġto Ġthe ĠNever - End ing ĠLanguage ĠLearning Ġby ĠCarnegie ĠMellon ĠUniversity ĠB IB REF 30 ĠâĢĵ Ġit Ġis Ġnow Ġthe Ġspeed Ġof Ġtraining Ġthat Ġdetermines Ġhow Ġmuch Ġdata Ġthe Ġmodel Ġis Ġable Ġto Ġsee . ĠSince Ġmore Ġtraining Ġdata Ġsignificantly Ġhelp Ġthe Ġmodel Ġperformance , Ġfocusing Ġon Ġspeeding Ġup Ġthe Ġalgorithm Ġmay Ġbe Ġmore Ġimportant Ġthan Ġever Ġbefore . ĠThis Ġmay Ġfor Ġinstance Ġinfluence Ġthe Ġdecision Ġwhether Ġto Ġuse Ġregular ization Ġsuch Ġas Ġdrop out Ġwhich Ġdoes Ġseem Ġto Ġsomewhat Ġimprove Ġthe Ġmodel Ġperformance , Ġhowever Ġusually Ġat Ġa Ġcost Ġof Ġslowing Ġdown Ġtraining . </s> Thanks Ġto Ġits Ġsimplicity , Ġthe Ġps r Ġseems Ġto Ġbe Ġtraining Ġfast Ġ- Ġfor Ġexample Ġaround Ġseven Ġtimes Ġfaster Ġthan Ġthe Ġmodels Ġproposed Ġby ĠChen Ġet Ġal . ĠB IB REF 5 Ġ. ĠHence Ġthe Ġps r Ġmay Ġbe Ġparticularly Ġsuitable Ġfor Ġtraining Ġon Ġlarge Ġdatasets . </s> The Ġsecond Ġchallenge Ġis Ġhow Ġto Ġgeneral ize Ġthe Ġperformance Ġgains Ġfrom Ġlarge Ġdata Ġto Ġa Ġspecific Ġtarget Ġdomain . ĠWhile Ġthere Ġare Ġhuge Ġamounts Ġof Ġnatural Ġlanguage Ġdata Ġin Ġgeneral , Ġit Ġmay Ġnot Ġbe Ġthe Ġcase Ġin Ġthe Ġdomain Ġwhere Ġwe Ġmay Ġwant Ġto Ġultimately Ġapply Ġour Ġmodel . </s> H ence Ġwe Ġare Ġusually Ġnot Ġfacing Ġa Ġscenario Ġof Ġsimply Ġusing Ġa Ġlarger Ġamount Ġof Ġthe Ġsame Ġtraining Ġdata , Ġbut Ġrather Ġextending Ġtraining Ġto Ġa Ġrelated Ġdomain Ġof Ġdata , Ġhoping Ġthat Ġsome Ġof Ġwhat Ġthe Ġmodel Ġlearns Ġon Ġthe Ġnew Ġdata Ġwill Ġstill Ġhelp Ġit Ġon Ġthe Ġoriginal Ġtask . </s> This Ġis Ġhighlighted Ġby Ġour Ġobservations Ġfrom Ġapplying Ġa Ġmodel Ġtrained Ġon Ġthe ĠBook Test Ġto ĠChildren 's ĠBook ĠTest Ġtest Ġdata . ĠIf Ġwe Ġmove Ġmodel Ġtraining Ġfrom Ġjoint ĠCB T ĠNE + CN Ġtraining Ġdata Ġto Ġa Ġsubset Ġof Ġthe ĠBook Test Ġof Ġthe Ġsame Ġsize Ġ( 230 k Ġexamples ), Ġwe Ġsee Ġa Ġdrop Ġin Ġaccuracy Ġof Ġaround Ġ10 % Ġon Ġthe ĠCB T Ġtest Ġdatasets . </s> H ence Ġeven Ġthough Ġthe ĠChildren 's ĠBook ĠTest Ġand ĠBook Test Ġdatasets Ġare Ġalmost Ġas Ġclose Ġas Ġtwo Ġdis j oint Ġdatasets Ġcan Ġget , Ġthe Ġtransfer Ġis Ġstill Ġvery Ġimperfect Ġ. ĠRight ly Ġchoosing Ġdata Ġto Ġaugment Ġthe Ġin - domain Ġtraining Ġdata Ġis Ġcertainly Ġa Ġproblem Ġworth Ġexploring Ġin Ġfuture Ġwork . </s> Our Ġresults Ġshow Ġthat Ġgiven Ġenough Ġdata Ġthe ĠAS ĠReader Ġwas Ġable Ġto Ġexceed Ġthe Ġhuman Ġperformance Ġon ĠCB T ĠCN Ġreported Ġby ĠFacebook . ĠHowever Ġwe Ġhypothesized Ġthat Ġthe Ġsystem Ġis Ġstill Ġnot Ġachieving Ġits Ġfull Ġpotential Ġso Ġwe Ġdecided Ġto Ġexamine Ġthe Ġroom Ġfor Ġimprovement Ġin Ġour Ġown Ġsmall Ġhuman Ġstudy . </s> Human ĠStudy </s> After Ġadding Ġmore Ġdata Ġwe Ġhave Ġthe Ġperformance Ġon Ġthe ĠCB T Ġvalidation Ġand Ġtest Ġdatasets Ġsoaring . ĠHowever Ġis Ġthere Ġstill Ġpotential Ġfor Ġmuch Ġfurther Ġgrowth Ġbeyond Ġthe Ġresults Ġwe Ġhave Ġobserved ? </s> We Ġdecided Ġto Ġexplore Ġthe Ġremaining Ġspace Ġfor Ġimprovement Ġon Ġthe ĠCB T Ġby Ġtesting Ġhumans Ġon Ġa Ġrandom Ġsubset Ġof Ġ50 Ġnamed Ġentity Ġand Ġ50 Ġcommon Ġnoun Ġvalidation Ġquestions Ġthat Ġthe Ġps r Ġensemble Ġcould Ġnot Ġanswer Ġcorrectly . ĠThese Ġquestions Ġwere Ġanswered Ġby Ġ10 Ġnon - native ĠEnglish Ġspeakers Ġfrom Ġour Ġresearch Ġlaboratory , Ġeach Ġon Ġa Ġdis j oint Ġsubset Ġof Ġquestions .. ĠParticipants Ġhad Ġunlimited Ġtime Ġto Ġanswer Ġthe Ġquestions Ġand Ġwere Ġtold Ġthat Ġthese Ġquestions Ġwere Ġnot Ġcorrectly Ġanswered Ġby Ġa Ġmachine , Ġproviding Ġadditional Ġmotivation Ġto Ġprove Ġthey Ġare Ġbetter Ġthan Ġcomputers . ĠThe Ġresults Ġof Ġthe Ġhuman Ġstudy Ġare Ġsummarized Ġin ĠTable ĠT AB REF 28 Ġ. ĠThey Ġshow Ġthat Ġa Ġmajority Ġof Ġquestions Ġthat Ġour Ġsystem Ġcould Ġnot Ġanswer Ġso Ġfar Ġare Ġin Ġfact Ġanswer able . ĠThis Ġsuggests Ġthat Ġ1 ) Ġthe Ġoriginal Ġhuman Ġbas elines Ġmight Ġhave Ġbeen Ġunderestimated , Ġhowever , Ġit Ġmight Ġalso Ġbe Ġthe Ġcase Ġthat Ġthere Ġare Ġsome Ġexamples Ġthat Ġcan Ġbe Ġanswered Ġby Ġmachines Ġand Ġnot Ġby Ġhumans ; Ġ2 ) Ġthere Ġis Ġstill Ġspace Ġfor Ġimprovement . </s> A Ġsystem Ġthat Ġwould Ġanswer Ġcorrectly Ġevery Ġtime Ġwhen Ġeither Ġour Ġensemble Ġor Ġhuman Ġanswered Ġcorrectly Ġwould Ġachieve Ġaccuracy Ġover Ġ92 % Ġpercent Ġon Ġboth Ġvalidation Ġand Ġtest ĠNE Ġdatasets Ġand Ġover Ġ96 % Ġon Ġboth ĠCN Ġdatasets . ĠHence Ġit Ġstill Ġmakes Ġsense Ġto Ġuse ĠCB T Ġdataset Ġto Ġstudy Ġfurther Ġimprovements Ġof Ġtext - com pre hens ion Ġsystems . </s> Conclusion </s> Few Ġways Ġof Ġimproving Ġmodel Ġperformance Ġare Ġas Ġsolidly Ġestablished Ġas Ġusing Ġmore Ġtraining Ġdata . ĠYet Ġwe Ġbelieve Ġthis Ġprinciple Ġhas Ġbeen Ġsomewhat Ġneglected Ġby Ġrecent Ġresearch Ġin Ġtext Ġcomprehension . ĠWhile Ġthere Ġis Ġa Ġpractically Ġunlimited Ġamount Ġof Ġdata Ġavailable Ġin Ġthis Ġfield , Ġmost Ġresearch Ġwas Ġperformed Ġon Ġunnecessarily Ġsmall Ġdatasets . </s> As Ġa Ġgentle Ġreminder Ġto Ġthe Ġcommunity Ġwe Ġhave Ġshown Ġthat Ġsimply Ġinf using Ġa Ġmodel Ġwith Ġmore Ġdata Ġcan Ġyield Ġperformance Ġimprovements Ġof Ġup Ġto ĠIN LINE FORM 0 Ġwhere Ġseveral Ġattempts Ġto Ġimprove Ġthe Ġmodel Ġarchitecture Ġon Ġthe Ġsame Ġtraining Ġdata Ġhave Ġgiven Ġgains Ġof Ġat Ġmost ĠIN LINE FORM 1 Ġcompared Ġto Ġour Ġbest Ġensemble Ġresult . ĠYes , Ġexperiments Ġon Ġsmall Ġdatasets Ġcertainly Ġcan Ġbring Ġuseful Ġinsights . ĠHowever Ġwe Ġbelieve Ġthat Ġthe Ġcommunity Ġshould Ġalso Ġembrace Ġthe Ġreal - world Ġscenario Ġof Ġdata Ġabundance . </s> The ĠBook Test Ġdataset Ġwe Ġare Ġproposing Ġgives Ġthe Ġreading - com pre hens ion Ġcommunity Ġan Ġopportunity Ġto Ġmake Ġa Ġstep Ġin Ġthat Ġdirection . </s> Training ĠDetails </s> The Ġtraining Ġdetails Ġare Ġsimilar Ġto Ġthose Ġin ĠB IB REF 4 Ġhowever Ġwe Ġare Ġincluding Ġthem Ġhere Ġfor Ġcomple teness . </s> To Ġtrain Ġthe Ġmodel Ġwe Ġused Ġsto ch astic Ġgradient Ġdescent Ġwith Ġthe ĠAD AM Ġupdate Ġrule ĠB IB REF 32 Ġand Ġlearning Ġrates Ġof ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġbest Ġlearning Ġrate Ġin Ġour Ġexperiments Ġwas ĠIN LINE FORM 3 Ġ. ĠWe Ġminimized Ġnegative Ġlog - like lihood Ġas Ġthe Ġtraining Ġobjective . </s> The Ġinitial Ġweights Ġin Ġthe Ġword - embed ding Ġmatrix Ġwere Ġdrawn Ġrandomly Ġuniformly Ġfrom Ġthe Ġinterval ĠIN LINE FORM 0 Ġ. ĠWe ights Ġin Ġthe ĠGR U Ġnetworks Ġwere Ġinitialized Ġby Ġrandom Ġorth og onal Ġmat rices ĠB IB REF 34 Ġand Ġbiases Ġwere Ġinitialized Ġto Ġzero . ĠWe Ġalso Ġused Ġa Ġgradient Ġclipping ĠB IB REF 33 Ġthreshold Ġof Ġ10 Ġand Ġbatches Ġof Ġsizes Ġbetween Ġ32 Ġor Ġ256 . ĠIncreasing Ġthe Ġbatch Ġfrom Ġ32 Ġto Ġ128 Ġseems Ġto Ġsignificantly Ġimprove Ġperformance Ġon Ġthe Ġlarge Ġdataset Ġ- Ġsomething Ġwe Ġdid Ġnot Ġobserve Ġon Ġthe Ġoriginal ĠCB T Ġdata . ĠIncreasing Ġthe Ġbatch Ġsize Ġmuch Ġabove Ġ128 Ġis Ġcurrently Ġdifficult Ġdue Ġto Ġmemory Ġconstraints Ġof Ġthe ĠGPU . </s> During Ġtraining Ġwe Ġrandomly Ġshuff led Ġall Ġexamples Ġat Ġthe Ġbeginning Ġof Ġeach Ġepoch . ĠTo Ġspeed Ġup Ġtraining , Ġwe Ġalways Ġpre - fetched Ġ10 Ġbatches Ġworth Ġof Ġexamples Ġand Ġsorted Ġthem Ġaccording Ġto Ġdocument Ġlength . ĠHence Ġeach Ġbatch Ġcontained Ġdocuments Ġof Ġroughly Ġthe Ġsame Ġlength . </s> We Ġalso Ġdid Ġnot Ġuse Ġpre - trained Ġword Ġembed d ings . </s> We Ġdid Ġnot Ġperform Ġany Ġtext Ġpre - processing Ġsince Ġthe Ġdatasets Ġwere Ġalready Ġtoken ized . </s> During Ġtraining Ġwe Ġevaluated Ġthe Ġmodel Ġperformance Ġevery Ġ12 Ġhours Ġand Ġat Ġthe Ġend Ġof Ġeach Ġepoch Ġand Ġstopped Ġtraining Ġwhen Ġthe Ġerror Ġon Ġthe Ġ20 k ĠBook Test Ġvalidation Ġset Ġstarted Ġincreasing . ĠWe Ġexplored Ġthe Ġhyper param eter Ġspace Ġby Ġtraining Ġ67 Ġdifferent Ġmodels ĠThe Ġregion Ġof Ġthe Ġparameter Ġspace Ġthat Ġwe Ġexplored Ġtogether Ġwith Ġthe Ġparameters Ġof Ġthe Ġmodel Ġwith Ġbest Ġvalidation Ġaccuracy Ġare Ġsummarized Ġin ĠTable ĠT AB REF 29 Ġ. </s> Our Ġmodel Ġwas Ġimplemented Ġusing ĠThe ano ĠB IB REF 31 Ġand ĠBlocks ĠB IB REF 35 Ġ. </s> The Ġen semb les Ġwere Ġformed Ġby Ġsimply Ġaveraging Ġthe Ġpredictions Ġfrom Ġthe Ġconstituent Ġsingle Ġmodels . ĠThese Ġsingle Ġmodels Ġwere Ġselected Ġusing Ġthe Ġfollowing Ġalgorithm . </s> We Ġstarted Ġwith Ġthe Ġbest Ġperforming Ġmodel Ġaccording Ġto Ġvalidation Ġperformance . ĠThen Ġin Ġeach Ġstep Ġwe Ġtried Ġadding Ġthe Ġbest Ġperforming Ġmodel Ġthat Ġhad Ġnot Ġbeen Ġpreviously Ġtried . ĠWe Ġkept Ġit Ġin Ġthe Ġensemble Ġif Ġit Ġdid Ġimprove Ġits Ġvalidation Ġperformance Ġand Ġdiscarded Ġit Ġotherwise . ĠThis Ġway Ġwe Ġgradually Ġtried Ġeach Ġmodel Ġonce . ĠWe Ġcall Ġthe Ġresulting Ġmodel Ġa Ġgreedy Ġensemble . ĠWe Ġused Ġthe ĠIN LINE FORM 0 ĠBook Test Ġvalidation Ġdataset Ġfor Ġthis Ġprocedure . </s> The Ġalgorithm Ġwas Ġoffered Ġ10 Ġmodels Ġand Ġselected Ġ5 Ġof Ġthem Ġfor Ġthe Ġfinal Ġensemble .
Introduction Since humans amass more and more generally available data in the form of unstructured text it would be very useful to teach machines to read and comprehend such data and then use this understanding to answer our questions. A significant amount of research has recently focused on answering one particular kind of questions the answer to which depends on understanding a context document. These are cloze-style questions BIBREF0 which require the reader to fill in a missing word in a sentence. An important advantage of such questions is that they can be generated automatically from a suitable text corpus which allows us to produce a practically unlimited amount of them. That opens the task to notoriously data-hungry deep-learning techniques which now seem to outperform all alternative approaches. Two such large-scale datasets have recently been proposed by researchers from Google DeepMind and Facebook AI: the CNN/Daily Mail dataset BIBREF1 and the Children's Book Test (CBT) BIBREF2 respectively. These have attracted a lot of attention from the research community BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 with a new state-of-the-art model coming out every few weeks. However if our goal is a production-level system actually capable of helping humans, we want the model to use all available resources as efficiently as possible. Given that we believe that if the community is striving to bring the performance as far as possible, it should move its work to larger data. This thinking goes in line with recent developments in the area of language modelling. For a long time models were being compared on several "standard" datasets with publications often presenting minuscule improvements in performance. Then the large-scale One Billion Word corpus dataset appeared BIBREF15 and it allowed Jozefowicz et al. to train much larger LSTM models BIBREF16 that almost halved the state-of-the-art perplexity on this dataset. We think it is time to make a similar step in the area of text comprehension. Hence we are introducing the BookTest, a new dataset very similar to the Children's Book test but more than 60 times larger to enable training larger models even in the domain of text comprehension. Furthermore the methodology used to create our data can later be used to create even larger datasets when the need arises thanks to further technological progress. We show that if we evaluate a model trained on the new dataset on the now standard Children's Book Test dataset, we see an improvement in accuracy much larger than other research groups achieved by enhancing the model architecture itself (while still using the original CBT training data). By training on the new dataset, we reduce the prediction error by almost one third. On the named-entity version of CBT this brings the ensemble of our models to the level of human baseline as reported by Facebook BIBREF2 . However in the final section we show in our own human study that there is still room for improvement on the CBT beyond the performance of our model. Task Description A natural way of testing a reader's comprehension of a text is to ask her a question the answer to which can be deduced from the text. Hence the task we are trying to solve consists of answering a cloze-style question, the answer to which depends on the understanding of a context document provided with the question. The model is also provided with a set of possible answers from which the correct one is to be selected. This can be formalized as follows: The training data consist of tuples INLINEFORM0 , where INLINEFORM1 is a question, INLINEFORM2 is a document that contains the answer to question INLINEFORM3 , INLINEFORM4 is a set of possible answers and INLINEFORM5 is the ground-truth answer. Both INLINEFORM6 and INLINEFORM7 are sequences of words from vocabulary INLINEFORM8 . We also assume that all possible answers are words from the vocabulary, that is INLINEFORM9 . In the CBT and CNN/Daily Mail datasets it is also true that the ground-truth answer INLINEFORM10 appears in the document. This is exploited by many machine learning models BIBREF2 , BIBREF4 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF10 , BIBREF11 , BIBREF12 , however some do not explicitly depend on this property BIBREF1 , BIBREF3 , BIBREF5 , BIBREF9 Current Landscape We will now briefly review what datasets for text comprehension have been published up to date and look at models which have been recently applied to solving the task we have just described. Datasets A crucial condition for applying deep-learning techniques is to have a huge amount of data available for training. For question answering this specifically means having a large number of document-question-answer triples available. While there is an unlimited amount of text available, coming up with relevant questions and the corresponding answers can be extremely labour-intensive if done by human annotators. There were efforts to provide such human-generated datasets, e.g. Microsoft's MCTest BIBREF17 , however their scale is not suitable for deep learning without pre-training on other data BIBREF18 (such as using pre-trained word embedding vectors). Google DeepMind managed to avoid this scale issue with their way of generating document-question-answer triples automatically, closely followed by Facebook with a similar method. Let us now briefly introduce the two resulting datasets whose properties are summarized in Table TABREF8 . These two datasets BIBREF1 exploit a useful feature of online news articles – many articles include a short summarizing sentence near the top of the page. Since all information in the summary sentence is also presented in the article body, we get a nice cloze-style question about the article contents by removing a word from the short summary. The dataset's authors also replaced all named entities in the dataset by anonymous tokens which are further shuffled for each new batch. This forces the model to rely solely on information from the context document, not being able to transfer any meaning of the named entities between documents. This restricts the task to one specific aspect of context-dependent question answering which may be useful however it moves the task further from the real application scenario, where we would like the model to use all information available to answer questions. Furthermore Chen et al. BIBREF5 have suggested that this can make about 17% of the questions unanswerable even by humans. They also claim that more than a half of the question sentences are mere paraphrases or exact matches of a single sentence from the context document. This raises a question to what extent the dataset can test deeper understanding of the articles. The Children's Book Test BIBREF2 uses a different source - books freely available thanks to Project Gutenberg. Since no summary is available, each example consists of a context document formed from 20 consecutive sentences from the story together with a question formed from the subsequent sentence. The dataset comes in four flavours depending on what type of word is omitted from the question sentence. Based on human evaluation done in BIBREF2 it seems that NE and CN are more context dependent than the other two types – prepositions and verbs. Therefore we (and all of the recent publications) focus only on these two word types. Several new datasets related to the (now almost standard) ones above emerged recently. We will now briefly present them and explain how the dataset we are introducing in this article differs from them. The LAMBADA dataset BIBREF19 is designed to measure progress in understanding common-sense questions about short stories that can be easily answered by humans but cannot be answered by current standard machine-learning models (e.g. plain LSTM language models). This dataset is useful for measuring the gap between humans and machine learning algorithms. However, by contrast to our BookTest dataset, it will not allow us to track progress towards the performance of the baseline systems or on examples where machine learning may show super-human performance. Also LAMBADA is just a diagnostic dataset and does not provide ready-to-use question-answering training data, just a plain-text corpus which may moreover include copyrighted books making its use potentially problematic for some purposes. We are providing ready training data consisting of copyright-free books only. The SQuAD dataset BIBREF20 based on Wikipedia and the Who-did-What dataset BIBREF21 based on Gigaword news articles are factoid question-answering datasets where a multi-word answer should be extracted from a context document. This is in contrast to the previous datasets, including CNN/DM, CBT, LAMBADA and our new dataset, which require only single-word answers. Both these datasets however provide less than 130,000 training questions, two orders of magnitude less than our dataset does. The Story Cloze Test BIBREF22 provides a crowd-sourced corpus of 49,255 commonsense stories for training and 3,744 testing stories with right and wrong endings. Hence the dataset is again rather small. Similarly to LAMBADA, the Story Cloze Test was designed to be easily answerable by humans. In the WikiReading BIBREF23 dataset the context document is formed from a Wikipedia article and the question-answer pair is taken from the corresponding WikiData page. For each entity (e.g. Hillary Clinton), WikiData contain a number of property-value pairs (e.g. place of birth: Chicago) which form the datasets's question-answer pairs. The dataset is certainly relevant to the community, however the questions are of very limited variety with only 20 properties (and hence unique questions) covering INLINEFORM0 of the dataset. Furthermore many of the frequent properties are mentioned at a set spot within the article (e.g. the date of birth is almost always in brackets behind the name of a person) which may make the task easier for machines. We are trying to provide a more varied dataset. Although there are several datasets related to task we are aiming to solve, they differ sufficiently for our dataset to bring new value to the community. Its biggest advantage is its size which can furthermore be easily upscaled without expensive human annotation. Finally while we are emphasizing the differences, models could certainly benefit from as diverse a collection of datasets as possible. Machine Learning Models A first major work applying deep-learning techniques to text comprehension was Hermann et al. BIBREF1 . This work was followed by the application of Memory Networks to the same task BIBREF2 . Later three models emerged around the same time BIBREF3 , BIBREF4 , BIBREF5 including our psr model BIBREF4 . The AS Reader inspired several subsequent models that use it as a sub-component in a diverse ensemble BIBREF8 ; extend it with a hierarchical structure BIBREF6 , BIBREF24 , BIBREF7 ; compute attention over the context document for every word in the query BIBREF10 or use two-way context-query attention mechanism for every word in the context and the query BIBREF11 that is similar in its spirit to models recently proposed in different domains, e.g. BIBREF25 in information retrieval. Other neural approaches to text comprehension are explored in BIBREF9 , BIBREF12 . Possible Directions for Improvements Accuracy in any machine learning tasks can be enhanced either by improving a machine learning model or by using more in-domain training data. Current state of the art models BIBREF6 , BIBREF7 , BIBREF8 , BIBREF11 improve over AS Reader's accuracy on CBT NE and CN datasets by 1-2 percent absolute. This suggests that with current techniques there is only limited room for improvement on the algorithmic side. The other possibility to improve performance is simply to use more training data. The importance of training data was highlighted by the frequently quoted Mercer's statement that “There is no data like more data.” The observation that having more data is often more important than having better algorithms has been frequently stressed since then BIBREF13 , BIBREF14 . As a step in the direction of exploiting the potential of more data in the domain of text comprehension, we created a new dataset called BookTest similar to, but much larger than the widely used CBT and CNN/DM datasets. BookTest Similarly to the CBT, our BookTest dataset is derived from books available through project Gutenberg. We used 3555 copyright-free books to extract CN examples and 10507 books for NE examples, for comparison the CBT dataset was extracted from just 108 books. When creating our dataset we follow the same procedure as was used to create the CBT dataset BIBREF2 . That is, we detect whether each sentence contains either a named entity or a common noun that already appeared in one of the preceding twenty sentences. This word is then replaced by a gap tag (XXXXX) in this sentence which is hence turned into a cloze-style question. The preceding 20 sentences are used as the context document. For common noun and named entity detection we use the Stanford POS tagger BIBREF27 and Stanford NER BIBREF28 . The training dataset consists of the original CBT NE and CN data extended with new NE and CN examples. The new BookTest dataset hence contains INLINEFORM0 training examples and INLINEFORM1 tokens. The validation dataset consists of INLINEFORM0 NE and INLINEFORM1 CN questions. We have one test set for NE and one for CN, each containing INLINEFORM2 examples. The training, validation and test sets were generated from non-overlapping sets of books. When generating the dataset we removed all editions of books used to create CBT validation and test sets from our training dataset. Therefore the models trained on the BookTest corpus can be evaluated on the original CBT data and they can be compared with recent text-comprehension models utilizing this dataset BIBREF2 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 . Baselines We will now use our psr model to evaluate the performance gain from increasing the dataset size. AS Reader In BIBREF4 we introduced the psr , which at the time of publication significantly outperformed all other architectures on the CNN, DM and CBT datasets. This model is built to leverage the fact that the answer is a single word from the context document. Similarly to many other models it uses attention over the document – intuitively a measure of how relevant each word is to answering the question. However while most previous models used this attention as weights to calculate a blended representation of the answer word, we simply sum the attention across all occurrences of each unique words and then simply select the word with the highest sum as the final answer. While simple, this trick seems both to improve accuracy and to speed-up training. It was adopted by many subsequent models BIBREF8 , BIBREF6 , BIBREF7 , BIBREF10 , BIBREF11 , BIBREF24 . Let us now describe the model in more detail. Figure FIGREF21 may help you in understanding the following paragraphs. The words from the document and the question are first converted into vector embeddings using a look-up matrix INLINEFORM0 . The document is then read by a bidirectional GRU network BIBREF29 . A concatenation of the hidden states of the forward and backward GRUs at each word is then used as a contextual embedding of this word, intuitively representing the context in which the word is appearing. We can also understand it as representing the set of questions to which this word may be an answer. Similarly the question is read by a bidirectional GRU but in this case only the final hidden states are concatenated to form the question embedding. The attention over each word in the context is then calculated as the dot product of its contextual embedding with the question embedding. This attention is then normalized by the softmax function and summed across all occurrences of each answer candidate. The candidate with most accumulated attention is selected as the final answer. For a more detailed description of the model including equations check BIBREF4 . More details about the training setup and model hyperparameters can be found in the Appendix. During our past experiments on the CNN, DM and CBT datasets BIBREF4 each unique word from the training, validation and test datasets had its row in the look-up matrix INLINEFORM0 . However as we radically increased the dataset size, this would result in an extremely large number of model parameters so we decided to limit the vocabulary size to INLINEFORM1 most frequent words. For each example, each unique out-of-vocabulary word is now mapped on one of 1000 anonymous tokens which are randomly initialized and untrained. Fixing the embeddings of these anonymous tags proved to significantly improve the performance. While mostly using the original AS Reader model, we have also tried introducing a minor tweak in some instances of the model. We tried initializing the context encoder GRU's hidden state by letting the encoder read the question first before proceeding to read the context document. Intuitively this allows the encoder to know in advance what to look for when reading over the context document. Including models of this kind in the ensemble helped to improve the performance. Results Table TABREF25 shows the accuracy of the psr and other architectures on the CBT validation and test data. The last two rows show the performance of the psr trained on the BookTest dataset; all the other models have been trained on the original CBT training data. If we take the best psr ensemble trained on CBT as a baseline, improving the model architecture as in BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , continuing to use the original CBT training data, lead to improvements of INLINEFORM0 and INLINEFORM1 absolute on named entities and common nouns respectively. By contrast, inflating the training dataset provided a boost of INLINEFORM2 while using the same model. The ensemble of our models even exceeded the human baseline provided by Facebook BIBREF2 on the Common Noun dataset. Our model takes approximately two weeks to converge when trained on the BookTest dataset on a single Nvidia Tesla K40 GPU. Discussion Embracing the abundance of data may mean focusing on other aspects of system design than with smaller data. Here are some of the challenges that we need to face in this situation. Firstly, since the amount of data is practically unlimited – we could even generate them on the fly resulting in continuous learning similar to the Never-Ending Language Learning by Carnegie Mellon University BIBREF30 – it is now the speed of training that determines how much data the model is able to see. Since more training data significantly help the model performance, focusing on speeding up the algorithm may be more important than ever before. This may for instance influence the decision whether to use regularization such as dropout which does seem to somewhat improve the model performance, however usually at a cost of slowing down training. Thanks to its simplicity, the psr seems to be training fast - for example around seven times faster than the models proposed by Chen et al. BIBREF5 . Hence the psr may be particularly suitable for training on large datasets. The second challenge is how to generalize the performance gains from large data to a specific target domain. While there are huge amounts of natural language data in general, it may not be the case in the domain where we may want to ultimately apply our model. Hence we are usually not facing a scenario of simply using a larger amount of the same training data, but rather extending training to a related domain of data, hoping that some of what the model learns on the new data will still help it on the original task. This is highlighted by our observations from applying a model trained on the BookTest to Children's Book Test test data. If we move model training from joint CBT NE+CN training data to a subset of the BookTest of the same size (230k examples), we see a drop in accuracy of around 10% on the CBT test datasets. Hence even though the Children's Book Test and BookTest datasets are almost as close as two disjoint datasets can get, the transfer is still very imperfect . Rightly choosing data to augment the in-domain training data is certainly a problem worth exploring in future work. Our results show that given enough data the AS Reader was able to exceed the human performance on CBT CN reported by Facebook. However we hypothesized that the system is still not achieving its full potential so we decided to examine the room for improvement in our own small human study. Human Study After adding more data we have the performance on the CBT validation and test datasets soaring. However is there still potential for much further growth beyond the results we have observed? We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly. These questions were answered by 10 non-native English speakers from our research laboratory, each on a disjoint subset of questions.. Participants had unlimited time to answer the questions and were told that these questions were not correctly answered by a machine, providing additional motivation to prove they are better than computers. The results of the human study are summarized in Table TABREF28 . They show that a majority of questions that our system could not answer so far are in fact answerable. This suggests that 1) the original human baselines might have been underestimated, however, it might also be the case that there are some examples that can be answered by machines and not by humans; 2) there is still space for improvement. A system that would answer correctly every time when either our ensemble or human answered correctly would achieve accuracy over 92% percent on both validation and test NE datasets and over 96% on both CN datasets. Hence it still makes sense to use CBT dataset to study further improvements of text-comprehension systems. Conclusion Few ways of improving model performance are as solidly established as using more training data. Yet we believe this principle has been somewhat neglected by recent research in text comprehension. While there is a practically unlimited amount of data available in this field, most research was performed on unnecessarily small datasets. As a gentle reminder to the community we have shown that simply infusing a model with more data can yield performance improvements of up to INLINEFORM0 where several attempts to improve the model architecture on the same training data have given gains of at most INLINEFORM1 compared to our best ensemble result. Yes, experiments on small datasets certainly can bring useful insights. However we believe that the community should also embrace the real-world scenario of data abundance. The BookTest dataset we are proposing gives the reading-comprehension community an opportunity to make a step in that direction. Training Details The training details are similar to those in BIBREF4 however we are including them here for completeness. To train the model we used stochastic gradient descent with the ADAM update rule BIBREF32 and learning rates of INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . The best learning rate in our experiments was INLINEFORM3 . We minimized negative log-likelihood as the training objective. The initial weights in the word-embedding matrix were drawn randomly uniformly from the interval INLINEFORM0 . Weights in the GRU networks were initialized by random orthogonal matrices BIBREF34 and biases were initialized to zero. We also used a gradient clipping BIBREF33 threshold of 10 and batches of sizes between 32 or 256. Increasing the batch from 32 to 128 seems to significantly improve performance on the large dataset - something we did not observe on the original CBT data. Increasing the batch size much above 128 is currently difficult due to memory constraints of the GPU. During training we randomly shuffled all examples at the beginning of each epoch. To speed up training, we always pre-fetched 10 batches worth of examples and sorted them according to document length. Hence each batch contained documents of roughly the same length. We also did not use pre-trained word embeddings. We did not perform any text pre-processing since the datasets were already tokenized. During training we evaluated the model performance every 12 hours and at the end of each epoch and stopped training when the error on the 20k BookTest validation set started increasing. We explored the hyperparameter space by training 67 different models The region of the parameter space that we explored together with the parameters of the model with best validation accuracy are summarized in Table TABREF29 . Our model was implemented using Theano BIBREF31 and Blocks BIBREF35 . The ensembles were formed by simply averaging the predictions from the constituent single models. These single models were selected using the following algorithm. We started with the best performing model according to validation performance. Then in each step we tried adding the best performing model that had not been previously tried. We kept it in the ensemble if it did improve its validation performance and discarded it otherwise. This way we gradually tried each model once. We call the resulting model a greedy ensemble. We used the INLINEFORM0 BookTest validation dataset for this procedure. The algorithm was offered 10 models and selected 5 of them for the final ensemble.
How large are the improvements of the Attention-Sum Reader model when using the BookTest dataset?
d67c01d9b689c052045f3de1b0918bab18c3f174
How Ġlarge Ġare Ġthe Ġimprovements Ġof Ġthe ĠAttention - Sum ĠReader Ġmodel Ġwhen Ġusing Ġthe ĠBook Test Ġdataset ?
IN LINE FORM 2 Ġ
<s> How Ġlarge Ġare Ġthe Ġimprovements Ġof Ġthe ĠAttention - Sum ĠReader Ġmodel Ġwhen Ġusing Ġthe ĠBook Test Ġdataset ? </s> Introduction </s> Since Ġhumans Ġam ass Ġmore Ġand Ġmore Ġgenerally Ġavailable Ġdata Ġin Ġthe Ġform Ġof Ġun struct ured Ġtext Ġit Ġwould Ġbe Ġvery Ġuseful Ġto Ġteach Ġmachines Ġto Ġread Ġand Ġcomprehend Ġsuch Ġdata Ġand Ġthen Ġuse Ġthis Ġunderstanding Ġto Ġanswer Ġour Ġquestions . ĠA Ġsignificant Ġamount Ġof Ġresearch Ġhas Ġrecently Ġfocused Ġon Ġanswering Ġone Ġparticular Ġkind Ġof Ġquestions Ġthe Ġanswer Ġto Ġwhich Ġdepends Ġon Ġunderstanding Ġa Ġcontext Ġdocument . ĠThese Ġare Ġclo ze - style Ġquestions ĠB IB REF 0 Ġwhich Ġrequire Ġthe Ġreader Ġto Ġfill Ġin Ġa Ġmissing Ġword Ġin Ġa Ġsentence . ĠAn Ġimportant Ġadvantage Ġof Ġsuch Ġquestions Ġis Ġthat Ġthey Ġcan Ġbe Ġgenerated Ġautomatically Ġfrom Ġa Ġsuitable Ġtext Ġcorpus Ġwhich Ġallows Ġus Ġto Ġproduce Ġa Ġpractically Ġunlimited Ġamount Ġof Ġthem . ĠThat Ġopens Ġthe Ġtask Ġto Ġnotoriously Ġdata - hung ry Ġdeep - learning Ġtechniques Ġwhich Ġnow Ġseem Ġto Ġoutper form Ġall Ġalternative Ġapproaches . </s> Two Ġsuch Ġlarge - scale Ġdatasets Ġhave Ġrecently Ġbeen Ġproposed Ġby Ġresearchers Ġfrom ĠGoogle ĠDeep Mind Ġand ĠFacebook ĠAI : Ġthe ĠCNN / Daily ĠMail Ġdataset ĠB IB REF 1 Ġand Ġthe ĠChildren 's ĠBook ĠTest Ġ( C BT ) ĠB IB REF 2 Ġrespectively . ĠThese Ġhave Ġattracted Ġa Ġlot Ġof Ġattention Ġfrom Ġthe Ġresearch Ġcommunity ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġwith Ġa Ġnew Ġstate - of - the - art Ġmodel Ġcoming Ġout Ġevery Ġfew Ġweeks . </s> However Ġif Ġour Ġgoal Ġis Ġa Ġproduction - level Ġsystem Ġactually Ġcapable Ġof Ġhelping Ġhumans , Ġwe Ġwant Ġthe Ġmodel Ġto Ġuse Ġall Ġavailable Ġresources Ġas Ġefficiently Ġas Ġpossible . ĠGiven Ġthat </s> we Ġbelieve Ġthat Ġif Ġthe Ġcommunity Ġis Ġstriving Ġto Ġbring Ġthe Ġperformance Ġas Ġfar Ġas Ġpossible , Ġit Ġshould Ġmove Ġits Ġwork Ġto Ġlarger Ġdata . </s> This Ġthinking Ġgoes Ġin Ġline Ġwith Ġrecent Ġdevelopments Ġin Ġthe Ġarea Ġof Ġlanguage Ġmodelling . ĠFor Ġa Ġlong Ġtime Ġmodels Ġwere Ġbeing Ġcompared Ġon Ġseveral Ġ" standard " Ġdatasets Ġwith Ġpublications Ġoften Ġpresenting Ġmin usc ule Ġimprovements Ġin Ġperformance . ĠThen Ġthe Ġlarge - scale ĠOne ĠBillion ĠWord Ġcorpus Ġdataset Ġappeared ĠB IB REF 15 Ġand Ġit Ġallowed ĠJo z ef owicz Ġet Ġal . Ġto Ġtrain Ġmuch Ġlarger ĠL ST M Ġmodels ĠB IB REF 16 Ġthat Ġalmost Ġhal ved Ġthe Ġstate - of - the - art Ġperplex ity Ġon Ġthis Ġdataset . </s> We Ġthink Ġit Ġis Ġtime Ġto Ġmake Ġa Ġsimilar Ġstep Ġin Ġthe Ġarea Ġof Ġtext Ġcomprehension . ĠHence Ġwe Ġare Ġintroducing Ġthe ĠBook Test , Ġa Ġnew Ġdataset Ġvery Ġsimilar Ġto Ġthe ĠChildren 's ĠBook Ġtest Ġbut Ġmore Ġthan Ġ60 Ġtimes Ġlarger Ġto Ġenable Ġtraining Ġlarger Ġmodels Ġeven Ġin Ġthe Ġdomain Ġof Ġtext Ġcomprehension . ĠFurthermore Ġthe Ġmethodology Ġused Ġto Ġcreate Ġour Ġdata Ġcan Ġlater Ġbe Ġused Ġto Ġcreate Ġeven Ġlarger Ġdatasets Ġwhen Ġthe Ġneed Ġarises Ġthanks Ġto Ġfurther Ġtechnological Ġprogress . </s> We Ġshow Ġthat Ġif Ġwe Ġevaluate Ġa Ġmodel Ġtrained Ġon Ġthe Ġnew Ġdataset Ġon Ġthe Ġnow Ġstandard ĠChildren 's ĠBook ĠTest Ġdataset , Ġwe Ġsee Ġan Ġimprovement Ġin Ġaccuracy Ġmuch Ġlarger Ġthan Ġother Ġresearch Ġgroups Ġachieved Ġby Ġenhancing Ġthe Ġmodel Ġarchitecture Ġitself Ġ( while Ġstill Ġusing Ġthe Ġoriginal ĠCB T Ġtraining Ġdata ). ĠBy Ġtraining Ġon Ġthe Ġnew Ġdataset , Ġwe Ġreduce Ġthe Ġprediction Ġerror Ġby Ġalmost Ġone Ġthird . ĠOn Ġthe Ġnamed - entity Ġversion Ġof ĠCB T Ġthis Ġbrings Ġthe Ġensemble Ġof Ġour Ġmodels Ġto Ġthe Ġlevel Ġof Ġhuman Ġbaseline Ġas Ġreported Ġby ĠFacebook ĠB IB REF 2 Ġ. ĠHowever Ġin Ġthe Ġfinal Ġsection Ġwe Ġshow Ġin Ġour Ġown Ġhuman Ġstudy Ġthat Ġthere Ġis Ġstill Ġroom Ġfor Ġimprovement Ġon Ġthe ĠCB T Ġbeyond Ġthe Ġperformance Ġof Ġour Ġmodel . </s> Task ĠDescription </s> A Ġnatural Ġway Ġof Ġtesting Ġa Ġreader 's Ġcomprehension Ġof Ġa Ġtext Ġis Ġto Ġask Ġher Ġa Ġquestion Ġthe Ġanswer Ġto Ġwhich Ġcan Ġbe Ġded uced Ġfrom Ġthe Ġtext . ĠHence Ġthe Ġtask Ġwe Ġare Ġtrying Ġto Ġsolve Ġconsists Ġof Ġanswering Ġa Ġclo ze - style Ġquestion , Ġthe Ġanswer Ġto Ġwhich Ġdepends Ġon Ġthe Ġunderstanding Ġof Ġa Ġcontext Ġdocument Ġprovided Ġwith Ġthe Ġquestion . ĠThe Ġmodel Ġis Ġalso Ġprovided Ġwith Ġa Ġset Ġof Ġpossible Ġanswers Ġfrom Ġwhich Ġthe Ġcorrect Ġone Ġis Ġto Ġbe Ġselected . ĠThis Ġcan Ġbe Ġformal ized Ġas Ġfollows : </s> The Ġtraining Ġdata Ġconsist Ġof Ġtu ples ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġquestion , ĠIN LINE FORM 2 Ġis Ġa Ġdocument Ġthat Ġcontains Ġthe Ġanswer Ġto Ġquestion ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġis Ġa Ġset Ġof Ġpossible Ġanswers Ġand ĠIN LINE FORM 5 Ġis Ġthe Ġground - truth Ġanswer . ĠBoth ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġare Ġsequences Ġof Ġwords Ġfrom Ġvocabulary ĠIN LINE FORM 8 Ġ. ĠWe Ġalso Ġassume Ġthat Ġall Ġpossible Ġanswers Ġare Ġwords Ġfrom Ġthe Ġvocabulary , Ġthat Ġis ĠIN LINE FORM 9 Ġ. ĠIn Ġthe ĠCB T Ġand ĠCNN / Daily ĠMail Ġdatasets Ġit Ġis Ġalso Ġtrue Ġthat Ġthe Ġground - truth Ġanswer ĠIN LINE FORM 10 Ġappears Ġin Ġthe Ġdocument . ĠThis Ġis Ġexploited Ġby Ġmany Ġmachine Ġlearning Ġmodels ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġhowever Ġsome Ġdo Ġnot Ġexplicitly Ġdepend Ġon Ġthis Ġproperty ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 9 </s> Current ĠLand scape </s> We Ġwill Ġnow Ġbriefly Ġreview Ġwhat Ġdatasets Ġfor Ġtext Ġcomprehension Ġhave Ġbeen Ġpublished Ġup Ġto Ġdate Ġand Ġlook Ġat Ġmodels Ġwhich Ġhave Ġbeen Ġrecently Ġapplied Ġto Ġsolving Ġthe Ġtask Ġwe Ġhave Ġjust Ġdescribed . </s> Dat as ets </s> A Ġcrucial Ġcondition Ġfor Ġapplying Ġdeep - learning Ġtechniques Ġis Ġto Ġhave Ġa Ġhuge Ġamount Ġof Ġdata Ġavailable Ġfor Ġtraining . ĠFor Ġquestion Ġanswering Ġthis Ġspecifically Ġmeans Ġhaving Ġa Ġlarge Ġnumber Ġof Ġdocument - question - answer Ġtri ples Ġavailable . ĠWhile Ġthere Ġis Ġan Ġunlimited Ġamount Ġof Ġtext Ġavailable , Ġcoming Ġup Ġwith Ġrelevant Ġquestions Ġand Ġthe Ġcorresponding Ġanswers Ġcan Ġbe Ġextremely Ġlabour - intensive Ġif Ġdone Ġby Ġhuman Ġannot ators . ĠThere Ġwere Ġefforts Ġto Ġprovide Ġsuch Ġhuman - generated Ġdatasets , Ġe . g . ĠMicrosoft 's ĠM CT est ĠB IB REF 17 Ġ, Ġhowever Ġtheir Ġscale Ġis Ġnot Ġsuitable Ġfor Ġdeep Ġlearning Ġwithout Ġpre - training Ġon Ġother Ġdata ĠB IB REF 18 Ġ( such Ġas Ġusing Ġpre - trained Ġword Ġembed ding Ġvectors ). </s> Google ĠDeep Mind Ġmanaged Ġto Ġavoid Ġthis Ġscale Ġissue Ġwith Ġtheir Ġway Ġof Ġgenerating Ġdocument - question - answer Ġtri ples Ġautomatically , Ġclosely Ġfollowed Ġby ĠFacebook Ġwith Ġa Ġsimilar Ġmethod . ĠLet Ġus Ġnow Ġbriefly Ġintroduce Ġthe Ġtwo Ġresulting Ġdatasets Ġwhose Ġproperties Ġare Ġsummarized Ġin ĠTable ĠT AB REF 8 Ġ. </s> These Ġtwo Ġdatasets ĠB IB REF 1 Ġexploit Ġa Ġuseful Ġfeature Ġof Ġonline Ġnews Ġarticles ĠâĢĵ Ġmany Ġarticles Ġinclude Ġa Ġshort Ġsummar izing Ġsentence Ġnear Ġthe Ġtop Ġof Ġthe Ġpage . ĠSince Ġall Ġinformation Ġin Ġthe Ġsummary Ġsentence Ġis Ġalso Ġpresented Ġin Ġthe Ġarticle Ġbody , Ġwe Ġget Ġa Ġnice Ġclo ze - style Ġquestion Ġabout Ġthe Ġarticle Ġcontents Ġby Ġremoving Ġa Ġword Ġfrom Ġthe Ġshort Ġsummary . </s> The Ġdataset 's Ġauthors Ġalso Ġreplaced Ġall Ġnamed Ġentities Ġin Ġthe Ġdataset Ġby Ġanonymous Ġtokens Ġwhich Ġare Ġfurther Ġshuff led Ġfor Ġeach Ġnew Ġbatch . ĠThis Ġforces Ġthe Ġmodel Ġto Ġrely Ġsolely Ġon Ġinformation Ġfrom Ġthe Ġcontext Ġdocument , Ġnot Ġbeing Ġable Ġto Ġtransfer Ġany Ġmeaning Ġof Ġthe Ġnamed Ġentities Ġbetween Ġdocuments . </s> This Ġrestricts Ġthe Ġtask Ġto Ġone Ġspecific Ġaspect Ġof Ġcontext - dependent Ġquestion Ġanswering Ġwhich Ġmay Ġbe Ġuseful Ġhowever Ġit Ġmoves Ġthe Ġtask Ġfurther Ġfrom Ġthe Ġreal Ġapplication Ġscenario , Ġwhere Ġwe Ġwould Ġlike Ġthe Ġmodel Ġto Ġuse Ġall Ġinformation Ġavailable Ġto Ġanswer Ġquestions . ĠFurthermore ĠChen Ġet Ġal . ĠB IB REF 5 Ġhave Ġsuggested Ġthat Ġthis Ġcan Ġmake Ġabout Ġ17 % Ġof Ġthe Ġquestions Ġun answer able Ġeven Ġby Ġhumans . ĠThey Ġalso Ġclaim Ġthat Ġmore Ġthan Ġa Ġhalf Ġof Ġthe Ġquestion Ġsentences Ġare Ġmere Ġparaph r ases Ġor Ġexact Ġmatches Ġof Ġa Ġsingle Ġsentence Ġfrom Ġthe Ġcontext Ġdocument . ĠThis Ġraises Ġa Ġquestion Ġto Ġwhat Ġextent Ġthe Ġdataset Ġcan Ġtest Ġdeeper Ġunderstanding Ġof Ġthe Ġarticles . </s> The ĠChildren 's ĠBook ĠTest ĠB IB REF 2 Ġuses Ġa Ġdifferent Ġsource Ġ- Ġbooks Ġfreely Ġavailable Ġthanks Ġto ĠProject ĠGutenberg . ĠSince Ġno Ġsummary Ġis Ġavailable , Ġeach Ġexample Ġconsists Ġof Ġa Ġcontext Ġdocument Ġformed Ġfrom Ġ20 Ġconsecutive Ġsentences Ġfrom Ġthe Ġstory Ġtogether Ġwith Ġa Ġquestion Ġformed Ġfrom Ġthe Ġsubsequent Ġsentence . </s> The Ġdataset Ġcomes Ġin Ġfour Ġflavours Ġdepending Ġon Ġwhat Ġtype Ġof Ġword Ġis Ġomitted Ġfrom Ġthe Ġquestion Ġsentence . ĠBased Ġon Ġhuman Ġevaluation Ġdone Ġin ĠB IB REF 2 Ġit Ġseems Ġthat ĠNE Ġand ĠCN Ġare Ġmore Ġcontext Ġdependent Ġthan Ġthe Ġother Ġtwo Ġtypes ĠâĢĵ Ġpre pos itions Ġand Ġverbs . ĠTherefore Ġwe Ġ( and Ġall Ġof Ġthe Ġrecent Ġpublications ) Ġfocus Ġonly Ġon Ġthese Ġtwo Ġword Ġtypes . </s> Several Ġnew Ġdatasets Ġrelated Ġto Ġthe Ġ( now Ġalmost Ġstandard ) Ġones Ġabove Ġemerged Ġrecently . ĠWe Ġwill Ġnow Ġbriefly Ġpresent Ġthem Ġand Ġexplain Ġhow Ġthe Ġdataset Ġwe Ġare Ġintroducing Ġin Ġthis Ġarticle Ġdiffers Ġfrom Ġthem . </s> The ĠL AM B ADA Ġdataset ĠB IB REF 19 Ġis Ġdesigned Ġto Ġmeasure Ġprogress Ġin Ġunderstanding Ġcommon - sense Ġquestions Ġabout Ġshort Ġstories Ġthat Ġcan Ġbe Ġeasily Ġanswered Ġby Ġhumans Ġbut Ġcannot Ġbe Ġanswered Ġby Ġcurrent Ġstandard Ġmachine - learning Ġmodels Ġ( e . g . Ġplain ĠL ST M Ġlanguage Ġmodels ). ĠThis Ġdataset Ġis Ġuseful Ġfor Ġmeasuring Ġthe Ġgap Ġbetween Ġhumans Ġand Ġmachine Ġlearning Ġalgorithms . ĠHowever , Ġby Ġcontrast Ġto Ġour ĠBook Test Ġdataset , Ġit Ġwill Ġnot Ġallow Ġus Ġto Ġtrack Ġprogress Ġtowards Ġthe Ġperformance Ġof Ġthe Ġbaseline Ġsystems Ġor Ġon Ġexamples Ġwhere Ġmachine Ġlearning Ġmay Ġshow Ġsuper - human Ġperformance . ĠAlso ĠL AM B ADA Ġis Ġjust Ġa Ġdiagnostic Ġdataset Ġand Ġdoes Ġnot Ġprovide Ġready - to - use Ġquestion - ans w ering Ġtraining Ġdata , Ġjust Ġa Ġplain - text Ġcorpus Ġwhich Ġmay Ġmoreover Ġinclude Ġcopyrighted Ġbooks Ġmaking Ġits Ġuse Ġpotentially Ġproblematic Ġfor Ġsome Ġpurposes . ĠWe Ġare Ġproviding Ġready Ġtraining Ġdata Ġconsisting Ġof Ġcopyright - free Ġbooks Ġonly . </s> The ĠS Qu AD Ġdataset ĠB IB REF 20 Ġbased Ġon ĠWikipedia Ġand Ġthe ĠWho - did - What Ġdataset ĠB IB REF 21 Ġbased Ġon ĠGig aw ord Ġnews Ġarticles Ġare Ġfact oid Ġquestion - ans w ering Ġdatasets Ġwhere Ġa Ġmulti - word Ġanswer Ġshould Ġbe Ġextracted Ġfrom Ġa Ġcontext Ġdocument . ĠThis Ġis Ġin Ġcontrast Ġto Ġthe Ġprevious Ġdatasets , Ġincluding ĠCNN / DM , ĠCB T , ĠL AM B ADA Ġand Ġour Ġnew Ġdataset , Ġwhich Ġrequire Ġonly Ġsingle - word Ġanswers . ĠBoth Ġthese Ġdatasets Ġhowever Ġprovide Ġless Ġthan Ġ130 , 000 Ġtraining Ġquestions , Ġtwo Ġorders Ġof Ġmagnitude Ġless Ġthan Ġour Ġdataset Ġdoes . </s> The ĠStory ĠClo ze ĠTest ĠB IB REF 22 Ġprovides Ġa Ġcrowd - s ourced Ġcorpus Ġof Ġ49 , 255 Ġcommons ense Ġstories Ġfor Ġtraining Ġand Ġ3 , 7 44 Ġtesting Ġstories Ġwith Ġright Ġand Ġwrong Ġendings . ĠHence Ġthe Ġdataset Ġis Ġagain Ġrather Ġsmall . ĠSimilarly Ġto ĠL AM B ADA , Ġthe ĠStory ĠClo ze ĠTest Ġwas Ġdesigned Ġto Ġbe Ġeasily Ġanswer able Ġby Ġhumans . </s> In Ġthe ĠWiki Reading ĠB IB REF 23 Ġdataset Ġthe Ġcontext Ġdocument Ġis Ġformed Ġfrom Ġa ĠWikipedia Ġarticle Ġand Ġthe Ġquestion - answer Ġpair Ġis Ġtaken Ġfrom Ġthe Ġcorresponding ĠWiki Data Ġpage . ĠFor Ġeach Ġentity Ġ( e . g . ĠHillary ĠClinton ), ĠWiki Data Ġcontain Ġa Ġnumber Ġof Ġproperty - value Ġpairs Ġ( e . g . Ġplace Ġof Ġbirth : ĠChicago ) Ġwhich Ġform Ġthe Ġdatasets 's Ġquestion - answer Ġpairs . ĠThe Ġdataset Ġis Ġcertainly Ġrelevant Ġto Ġthe Ġcommunity , Ġhowever Ġthe Ġquestions Ġare Ġof Ġvery Ġlimited Ġvariety Ġwith Ġonly Ġ20 Ġproperties Ġ( and Ġhence Ġunique Ġquestions ) Ġcovering ĠIN LINE FORM 0 Ġof Ġthe Ġdataset . ĠFurthermore Ġmany Ġof Ġthe Ġfrequent Ġproperties Ġare Ġmentioned Ġat Ġa Ġset Ġspot Ġwithin Ġthe Ġarticle Ġ( e . g . Ġthe Ġdate Ġof Ġbirth Ġis Ġalmost Ġalways Ġin Ġbrackets Ġbehind Ġthe Ġname Ġof Ġa Ġperson ) Ġwhich Ġmay Ġmake Ġthe Ġtask Ġeasier Ġfor Ġmachines . ĠWe Ġare Ġtrying Ġto Ġprovide Ġa Ġmore Ġvaried Ġdataset . </s> Although Ġthere Ġare Ġseveral Ġdatasets Ġrelated Ġto Ġtask Ġwe Ġare Ġaiming Ġto Ġsolve , Ġthey Ġdiffer Ġsufficiently Ġfor Ġour Ġdataset Ġto Ġbring Ġnew Ġvalue Ġto Ġthe Ġcommunity . ĠIts Ġbiggest Ġadvantage Ġis Ġits Ġsize Ġwhich Ġcan Ġfurthermore Ġbe Ġeasily Ġup sc aled Ġwithout Ġexpensive Ġhuman Ġannotation . ĠFinally Ġwhile Ġwe Ġare Ġemphasizing Ġthe Ġdifferences , Ġmodels Ġcould Ġcertainly Ġbenefit Ġfrom Ġas Ġdiverse Ġa Ġcollection Ġof Ġdatasets Ġas Ġpossible . </s> Machine ĠLearning ĠModels </s> A Ġfirst Ġmajor Ġwork Ġapplying Ġdeep - learning Ġtechniques Ġto Ġtext Ġcomprehension Ġwas ĠHerman n Ġet Ġal . ĠB IB REF 1 Ġ. ĠThis Ġwork Ġwas Ġfollowed Ġby Ġthe Ġapplication Ġof ĠMemory ĠNetworks Ġto Ġthe Ġsame Ġtask ĠB IB REF 2 Ġ. ĠLater Ġthree Ġmodels Ġemerged Ġaround Ġthe Ġsame Ġtime ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġincluding Ġour Ġps r Ġmodel ĠB IB REF 4 Ġ. ĠThe ĠAS ĠReader Ġinspired Ġseveral Ġsubsequent Ġmodels Ġthat Ġuse Ġit Ġas Ġa Ġsub - component Ġin Ġa Ġdiverse Ġensemble ĠB IB REF 8 Ġ; Ġextend Ġit Ġwith Ġa Ġhierarchical Ġstructure ĠB IB REF 6 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 7 Ġ; Ġcompute Ġattention Ġover Ġthe Ġcontext Ġdocument Ġfor Ġevery Ġword Ġin Ġthe Ġquery ĠB IB REF 10 Ġor Ġuse Ġtwo - way Ġcontext - query Ġattention Ġmechanism Ġfor Ġevery Ġword Ġin Ġthe Ġcontext Ġand Ġthe Ġquery ĠB IB REF 11 Ġthat Ġis Ġsimilar Ġin Ġits Ġspirit Ġto Ġmodels Ġrecently Ġproposed Ġin Ġdifferent Ġdomains , Ġe . g . ĠB IB REF 25 Ġin Ġinformation Ġretrieval . ĠOther Ġneural Ġapproaches Ġto Ġtext Ġcomprehension Ġare Ġexplored Ġin ĠB IB REF 9 Ġ, ĠB IB REF 12 Ġ. </s> P ossible ĠDirections Ġfor ĠImprovements </s> Acc uracy Ġin Ġany Ġmachine Ġlearning Ġtasks Ġcan Ġbe Ġenhanced Ġeither Ġby Ġimproving Ġa Ġmachine Ġlearning Ġmodel Ġor Ġby Ġusing Ġmore Ġin - domain Ġtraining Ġdata . ĠCurrent Ġstate Ġof Ġthe Ġart Ġmodels ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 11 Ġimprove Ġover ĠAS ĠReader 's Ġaccuracy Ġon ĠCB T ĠNE Ġand ĠCN Ġdatasets Ġby Ġ1 - 2 Ġpercent Ġabsolute . ĠThis Ġsuggests Ġthat Ġwith Ġcurrent Ġtechniques Ġthere Ġis Ġonly Ġlimited Ġroom Ġfor Ġimprovement Ġon Ġthe Ġalgorith mic Ġside . </s> The Ġother Ġpossibility Ġto Ġimprove Ġperformance Ġis Ġsimply Ġto Ġuse Ġmore Ġtraining Ġdata . ĠThe Ġimportance Ġof Ġtraining Ġdata Ġwas Ġhighlighted Ġby Ġthe Ġfrequently Ġquoted ĠMercer 's Ġstatement Ġthat ĠâĢ ľ There Ġis Ġno Ġdata Ġlike Ġmore Ġdata . âĢ Ŀ ĠThe Ġobservation Ġthat Ġhaving Ġmore Ġdata Ġis Ġoften Ġmore Ġimportant Ġthan Ġhaving Ġbetter Ġalgorithms Ġhas Ġbeen Ġfrequently Ġstressed Ġsince Ġthen ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> As Ġa Ġstep Ġin Ġthe Ġdirection Ġof Ġexploiting Ġthe Ġpotential Ġof Ġmore Ġdata Ġin Ġthe Ġdomain Ġof Ġtext Ġcomprehension , Ġwe Ġcreated Ġa Ġnew Ġdataset Ġcalled ĠBook Test Ġsimilar Ġto , Ġbut Ġmuch Ġlarger Ġthan Ġthe Ġwidely Ġused ĠCB T Ġand ĠCNN / DM Ġdatasets . </s> Book Test </s> Similarly Ġto Ġthe ĠCB T , Ġour ĠBook Test Ġdataset Ġis Ġderived Ġfrom Ġbooks Ġavailable Ġthrough Ġproject ĠGutenberg . ĠWe Ġused Ġ3 555 Ġcopyright - free Ġbooks Ġto Ġextract ĠCN Ġexamples Ġand Ġ10 507 Ġbooks Ġfor ĠNE Ġexamples , Ġfor Ġcomparison Ġthe ĠCB T Ġdataset Ġwas Ġextracted Ġfrom Ġjust Ġ108 Ġbooks . </s> When Ġcreating Ġour Ġdataset Ġwe Ġfollow Ġthe Ġsame Ġprocedure Ġas Ġwas Ġused Ġto Ġcreate Ġthe ĠCB T Ġdataset ĠB IB REF 2 Ġ. ĠThat Ġis , Ġwe Ġdetect Ġwhether Ġeach Ġsentence Ġcontains Ġeither Ġa Ġnamed Ġentity Ġor Ġa Ġcommon Ġnoun Ġthat Ġalready Ġappeared Ġin Ġone Ġof Ġthe Ġpreceding Ġtwenty Ġsentences . ĠThis Ġword Ġis Ġthen Ġreplaced Ġby Ġa Ġgap Ġtag Ġ( XXXX X ) Ġin Ġthis Ġsentence Ġwhich Ġis Ġhence Ġturned Ġinto Ġa Ġclo ze - style Ġquestion . ĠThe Ġpreceding Ġ20 Ġsentences Ġare Ġused Ġas Ġthe Ġcontext Ġdocument . ĠFor Ġcommon Ġnoun Ġand Ġnamed Ġentity Ġdetection Ġwe Ġuse Ġthe ĠStanford ĠPOS Ġtag ger ĠB IB REF 27 Ġand ĠStanford ĠN ER ĠB IB REF 28 Ġ. </s> The Ġtraining Ġdataset Ġconsists Ġof Ġthe Ġoriginal ĠCB T ĠNE Ġand ĠCN Ġdata Ġextended Ġwith Ġnew ĠNE Ġand ĠCN Ġexamples . ĠThe Ġnew ĠBook Test Ġdataset Ġhence Ġcontains ĠIN LINE FORM 0 Ġtraining Ġexamples Ġand ĠIN LINE FORM 1 Ġtokens . </s> The Ġvalidation Ġdataset Ġconsists Ġof ĠIN LINE FORM 0 ĠNE Ġand ĠIN LINE FORM 1 ĠCN Ġquestions . ĠWe Ġhave Ġone Ġtest Ġset Ġfor ĠNE Ġand Ġone Ġfor ĠCN , Ġeach Ġcontaining ĠIN LINE FORM 2 Ġexamples . ĠThe Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġwere Ġgenerated Ġfrom Ġnon - over l apping Ġsets Ġof Ġbooks . </s> When Ġgenerating Ġthe Ġdataset Ġwe Ġremoved Ġall Ġeditions Ġof Ġbooks Ġused Ġto Ġcreate ĠCB T Ġvalidation Ġand Ġtest Ġsets Ġfrom Ġour Ġtraining Ġdataset . ĠTherefore Ġthe Ġmodels Ġtrained Ġon Ġthe ĠBook Test Ġcorpus Ġcan Ġbe Ġevaluated Ġon Ġthe Ġoriginal ĠCB T Ġdata Ġand Ġthey Ġcan Ġbe Ġcompared Ġwith Ġrecent Ġtext - com pre hens ion Ġmodels Ġutilizing Ġthis Ġdataset ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. </s> Bas elines </s> We Ġwill Ġnow Ġuse Ġour Ġps r Ġmodel Ġto Ġevaluate Ġthe Ġperformance Ġgain Ġfrom Ġincreasing Ġthe Ġdataset Ġsize . </s> AS ĠReader </s> In ĠB IB REF 4 Ġwe Ġintroduced Ġthe Ġps r Ġ, Ġwhich Ġat Ġthe Ġtime Ġof Ġpublication Ġsignificantly Ġoutper formed Ġall Ġother Ġarchitectures Ġon Ġthe ĠCNN , ĠDM Ġand ĠCB T Ġdatasets . ĠThis Ġmodel Ġis Ġbuilt Ġto Ġleverage Ġthe Ġfact Ġthat Ġthe Ġanswer Ġis Ġa Ġsingle Ġword Ġfrom Ġthe Ġcontext Ġdocument . ĠSimilarly Ġto Ġmany Ġother Ġmodels Ġit Ġuses Ġattention Ġover Ġthe Ġdocument ĠâĢĵ Ġintuitive ly Ġa Ġmeasure Ġof Ġhow Ġrelevant Ġeach Ġword Ġis Ġto Ġanswering Ġthe Ġquestion . ĠHowever Ġwhile Ġmost Ġprevious Ġmodels Ġused Ġthis Ġattention Ġas Ġweights Ġto Ġcalculate Ġa Ġblended Ġrepresentation Ġof Ġthe Ġanswer Ġword , Ġwe Ġsimply Ġsum Ġthe Ġattention Ġacross Ġall Ġoccurrences Ġof Ġeach Ġunique Ġwords Ġand Ġthen Ġsimply Ġselect Ġthe Ġword Ġwith Ġthe Ġhighest Ġsum Ġas Ġthe Ġfinal Ġanswer . ĠWhile Ġsimple , Ġthis Ġtrick Ġseems Ġboth Ġto Ġimprove Ġaccuracy Ġand Ġto Ġspeed - up Ġtraining . ĠIt Ġwas Ġadopted Ġby Ġmany Ġsubsequent Ġmodels ĠB IB REF 8 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 24 Ġ. </s> Let Ġus Ġnow Ġdescribe Ġthe Ġmodel Ġin Ġmore Ġdetail . ĠFigure ĠFIG REF 21 Ġmay Ġhelp Ġyou Ġin Ġunderstanding Ġthe Ġfollowing Ġparagraphs . </s> The Ġwords Ġfrom Ġthe Ġdocument Ġand Ġthe Ġquestion Ġare Ġfirst Ġconverted Ġinto Ġvector Ġembed d ings Ġusing Ġa Ġlook - up Ġmatrix ĠIN LINE FORM 0 Ġ. ĠThe Ġdocument Ġis Ġthen Ġread Ġby Ġa Ġbid irection al ĠGR U Ġnetwork ĠB IB REF 29 Ġ. ĠA Ġconc aten ation Ġof Ġthe Ġhidden Ġstates Ġof Ġthe Ġforward Ġand Ġbackward ĠGR Us Ġat Ġeach Ġword Ġis Ġthen Ġused Ġas Ġa Ġcontextual Ġembed ding Ġof Ġthis Ġword , Ġintuitive ly Ġrepresenting Ġthe Ġcontext Ġin Ġwhich Ġthe Ġword Ġis Ġappearing . ĠWe Ġcan Ġalso Ġunderstand Ġit Ġas Ġrepresenting Ġthe Ġset Ġof Ġquestions Ġto Ġwhich Ġthis Ġword Ġmay Ġbe Ġan Ġanswer . </s> Similarly Ġthe Ġquestion Ġis Ġread Ġby Ġa Ġbid irection al ĠGR U Ġbut Ġin Ġthis Ġcase Ġonly Ġthe Ġfinal Ġhidden Ġstates Ġare Ġconc aten ated Ġto Ġform Ġthe Ġquestion Ġembed ding . </s> The Ġattention Ġover Ġeach Ġword Ġin Ġthe Ġcontext Ġis Ġthen Ġcalculated Ġas Ġthe Ġdot Ġproduct Ġof Ġits Ġcontextual Ġembed ding Ġwith Ġthe Ġquestion Ġembed ding . ĠThis Ġattention Ġis Ġthen Ġnormalized Ġby Ġthe Ġsoft max Ġfunction Ġand Ġsummed Ġacross Ġall Ġoccurrences Ġof Ġeach Ġanswer Ġcandidate . ĠThe Ġcandidate Ġwith Ġmost Ġaccumulated Ġattention Ġis Ġselected Ġas Ġthe Ġfinal Ġanswer . </s> For Ġa Ġmore Ġdetailed Ġdescription Ġof Ġthe Ġmodel Ġincluding Ġequations Ġcheck ĠB IB REF 4 Ġ. ĠMore Ġdetails Ġabout Ġthe Ġtraining Ġsetup Ġand Ġmodel Ġhyper param eters Ġcan Ġbe Ġfound Ġin Ġthe ĠAppendix . </s> During Ġour Ġpast Ġexperiments Ġon Ġthe ĠCNN , ĠDM Ġand ĠCB T Ġdatasets ĠB IB REF 4 Ġeach Ġunique Ġword Ġfrom Ġthe Ġtraining , Ġvalidation Ġand Ġtest Ġdatasets Ġhad Ġits Ġrow Ġin Ġthe Ġlook - up Ġmatrix ĠIN LINE FORM 0 Ġ. ĠHowever Ġas Ġwe Ġradically Ġincreased Ġthe Ġdataset Ġsize , Ġthis Ġwould Ġresult Ġin Ġan Ġextremely Ġlarge Ġnumber Ġof Ġmodel Ġparameters Ġso Ġwe Ġdecided Ġto Ġlimit Ġthe Ġvocabulary Ġsize Ġto ĠIN LINE FORM 1 Ġmost Ġfrequent Ġwords . ĠFor Ġeach Ġexample , Ġeach Ġunique Ġout - of - voc abulary Ġword Ġis Ġnow Ġmapped Ġon Ġone Ġof Ġ1000 Ġanonymous Ġtokens Ġwhich Ġare Ġrandomly Ġinitialized Ġand Ġunt rained . ĠFix ing Ġthe Ġembed d ings Ġof Ġthese Ġanonymous Ġtags Ġproved Ġto Ġsignificantly Ġimprove Ġthe Ġperformance . </s> While Ġmostly Ġusing Ġthe Ġoriginal ĠAS ĠReader Ġmodel , Ġwe Ġhave Ġalso Ġtried Ġintroducing Ġa Ġminor Ġtweak Ġin Ġsome Ġinstances Ġof Ġthe Ġmodel . ĠWe Ġtried Ġinitial izing Ġthe Ġcontext Ġenc oder ĠGR U 's Ġhidden Ġstate Ġby Ġletting Ġthe Ġenc oder Ġread Ġthe Ġquestion Ġfirst Ġbefore Ġproceeding Ġto Ġread Ġthe Ġcontext Ġdocument . ĠInt uitive ly Ġthis Ġallows Ġthe Ġenc oder Ġto Ġknow Ġin Ġadvance Ġwhat Ġto Ġlook Ġfor Ġwhen Ġreading Ġover Ġthe Ġcontext Ġdocument . </s> In cluding Ġmodels Ġof Ġthis Ġkind Ġin Ġthe Ġensemble Ġhelped Ġto Ġimprove Ġthe Ġperformance . </s> Results </s> Table ĠT AB REF 25 Ġshows Ġthe Ġaccuracy Ġof Ġthe Ġps r Ġand Ġother Ġarchitectures Ġon Ġthe ĠCB T Ġvalidation Ġand Ġtest Ġdata . ĠThe Ġlast Ġtwo Ġrows Ġshow Ġthe Ġperformance Ġof Ġthe Ġps r Ġtrained Ġon Ġthe ĠBook Test Ġdataset ; Ġall Ġthe Ġother Ġmodels Ġhave Ġbeen Ġtrained Ġon Ġthe Ġoriginal ĠCB T Ġtraining Ġdata . </s> If Ġwe Ġtake Ġthe Ġbest Ġps r Ġensemble Ġtrained Ġon ĠCB T Ġas Ġa Ġbaseline , Ġimproving Ġthe Ġmodel Ġarchitecture Ġas Ġin ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġcontinuing Ġto Ġuse Ġthe Ġoriginal ĠCB T Ġtraining Ġdata , Ġlead Ġto Ġimprovements Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġabsolute Ġon Ġnamed Ġentities Ġand Ġcommon Ġnoun s Ġrespectively . ĠBy Ġcontrast , Ġinfl ating Ġthe Ġtraining Ġdataset Ġprovided Ġa Ġboost Ġof ĠIN LINE FORM 2 Ġwhile Ġusing Ġthe Ġsame Ġmodel . ĠThe Ġensemble Ġof Ġour Ġmodels Ġeven Ġexceeded Ġthe Ġhuman Ġbaseline Ġprovided Ġby ĠFacebook ĠB IB REF 2 Ġon Ġthe ĠCommon ĠN oun Ġdataset . </s> Our Ġmodel Ġtakes Ġapproximately Ġtwo Ġweeks Ġto Ġconverge Ġwhen Ġtrained Ġon Ġthe ĠBook Test Ġdataset Ġon Ġa Ġsingle ĠNvidia ĠTesla ĠK 40 ĠGPU . </s> Discussion </s> Em br acing Ġthe Ġabundance Ġof Ġdata Ġmay Ġmean Ġfocusing Ġon Ġother Ġaspects Ġof Ġsystem Ġdesign Ġthan Ġwith Ġsmaller Ġdata . ĠHere Ġare Ġsome Ġof Ġthe Ġchallenges Ġthat Ġwe Ġneed Ġto Ġface Ġin Ġthis Ġsituation . </s> Firstly , Ġsince Ġthe Ġamount Ġof Ġdata Ġis Ġpractically Ġunlimited ĠâĢĵ Ġwe Ġcould Ġeven Ġgenerate Ġthem Ġon Ġthe Ġfly Ġresulting Ġin Ġcontinuous Ġlearning Ġsimilar Ġto Ġthe ĠNever - End ing ĠLanguage ĠLearning Ġby ĠCarnegie ĠMellon ĠUniversity ĠB IB REF 30 ĠâĢĵ Ġit Ġis Ġnow Ġthe Ġspeed Ġof Ġtraining Ġthat Ġdetermines Ġhow Ġmuch Ġdata Ġthe Ġmodel Ġis Ġable Ġto Ġsee . ĠSince Ġmore Ġtraining Ġdata Ġsignificantly Ġhelp Ġthe Ġmodel Ġperformance , Ġfocusing Ġon Ġspeeding Ġup Ġthe Ġalgorithm Ġmay Ġbe Ġmore Ġimportant Ġthan Ġever Ġbefore . ĠThis Ġmay Ġfor Ġinstance Ġinfluence Ġthe Ġdecision Ġwhether Ġto Ġuse Ġregular ization Ġsuch Ġas Ġdrop out Ġwhich Ġdoes Ġseem Ġto Ġsomewhat Ġimprove Ġthe Ġmodel Ġperformance , Ġhowever Ġusually Ġat Ġa Ġcost Ġof Ġslowing Ġdown Ġtraining . </s> Thanks Ġto Ġits Ġsimplicity , Ġthe Ġps r Ġseems Ġto Ġbe Ġtraining Ġfast Ġ- Ġfor Ġexample Ġaround Ġseven Ġtimes Ġfaster Ġthan Ġthe Ġmodels Ġproposed Ġby ĠChen Ġet Ġal . ĠB IB REF 5 Ġ. ĠHence Ġthe Ġps r Ġmay Ġbe Ġparticularly Ġsuitable Ġfor Ġtraining Ġon Ġlarge Ġdatasets . </s> The Ġsecond Ġchallenge Ġis Ġhow Ġto Ġgeneral ize Ġthe Ġperformance Ġgains Ġfrom Ġlarge Ġdata Ġto Ġa Ġspecific Ġtarget Ġdomain . ĠWhile Ġthere Ġare Ġhuge Ġamounts Ġof Ġnatural Ġlanguage Ġdata Ġin Ġgeneral , Ġit Ġmay Ġnot Ġbe Ġthe Ġcase Ġin Ġthe Ġdomain Ġwhere Ġwe Ġmay Ġwant Ġto Ġultimately Ġapply Ġour Ġmodel . </s> H ence Ġwe Ġare Ġusually Ġnot Ġfacing Ġa Ġscenario Ġof Ġsimply Ġusing Ġa Ġlarger Ġamount Ġof Ġthe Ġsame Ġtraining Ġdata , Ġbut Ġrather Ġextending Ġtraining Ġto Ġa Ġrelated Ġdomain Ġof Ġdata , Ġhoping Ġthat Ġsome Ġof Ġwhat Ġthe Ġmodel Ġlearns Ġon Ġthe Ġnew Ġdata Ġwill Ġstill Ġhelp Ġit Ġon Ġthe Ġoriginal Ġtask . </s> This Ġis Ġhighlighted Ġby Ġour Ġobservations Ġfrom Ġapplying Ġa Ġmodel Ġtrained Ġon Ġthe ĠBook Test Ġto ĠChildren 's ĠBook ĠTest Ġtest Ġdata . ĠIf Ġwe Ġmove Ġmodel Ġtraining Ġfrom Ġjoint ĠCB T ĠNE + CN Ġtraining Ġdata Ġto Ġa Ġsubset Ġof Ġthe ĠBook Test Ġof Ġthe Ġsame Ġsize Ġ( 230 k Ġexamples ), Ġwe Ġsee Ġa Ġdrop Ġin Ġaccuracy Ġof Ġaround Ġ10 % Ġon Ġthe ĠCB T Ġtest Ġdatasets . </s> H ence Ġeven Ġthough Ġthe ĠChildren 's ĠBook ĠTest Ġand ĠBook Test Ġdatasets Ġare Ġalmost Ġas Ġclose Ġas Ġtwo Ġdis j oint Ġdatasets Ġcan Ġget , Ġthe Ġtransfer Ġis Ġstill Ġvery Ġimperfect Ġ. ĠRight ly Ġchoosing Ġdata Ġto Ġaugment Ġthe Ġin - domain Ġtraining Ġdata Ġis Ġcertainly Ġa Ġproblem Ġworth Ġexploring Ġin Ġfuture Ġwork . </s> Our Ġresults Ġshow Ġthat Ġgiven Ġenough Ġdata Ġthe ĠAS ĠReader Ġwas Ġable Ġto Ġexceed Ġthe Ġhuman Ġperformance Ġon ĠCB T ĠCN Ġreported Ġby ĠFacebook . ĠHowever Ġwe Ġhypothesized Ġthat Ġthe Ġsystem Ġis Ġstill Ġnot Ġachieving Ġits Ġfull Ġpotential Ġso Ġwe Ġdecided Ġto Ġexamine Ġthe Ġroom Ġfor Ġimprovement Ġin Ġour Ġown Ġsmall Ġhuman Ġstudy . </s> Human ĠStudy </s> After Ġadding Ġmore Ġdata Ġwe Ġhave Ġthe Ġperformance Ġon Ġthe ĠCB T Ġvalidation Ġand Ġtest Ġdatasets Ġsoaring . ĠHowever Ġis Ġthere Ġstill Ġpotential Ġfor Ġmuch Ġfurther Ġgrowth Ġbeyond Ġthe Ġresults Ġwe Ġhave Ġobserved ? </s> We Ġdecided Ġto Ġexplore Ġthe Ġremaining Ġspace Ġfor Ġimprovement Ġon Ġthe ĠCB T Ġby Ġtesting Ġhumans Ġon Ġa Ġrandom Ġsubset Ġof Ġ50 Ġnamed Ġentity Ġand Ġ50 Ġcommon Ġnoun Ġvalidation Ġquestions Ġthat Ġthe Ġps r Ġensemble Ġcould Ġnot Ġanswer Ġcorrectly . ĠThese Ġquestions Ġwere Ġanswered Ġby Ġ10 Ġnon - native ĠEnglish Ġspeakers Ġfrom Ġour Ġresearch Ġlaboratory , Ġeach Ġon Ġa Ġdis j oint Ġsubset Ġof Ġquestions .. ĠParticipants Ġhad Ġunlimited Ġtime Ġto Ġanswer Ġthe Ġquestions Ġand Ġwere Ġtold Ġthat Ġthese Ġquestions Ġwere Ġnot Ġcorrectly Ġanswered Ġby Ġa Ġmachine , Ġproviding Ġadditional Ġmotivation Ġto Ġprove Ġthey Ġare Ġbetter Ġthan Ġcomputers . ĠThe Ġresults Ġof Ġthe Ġhuman Ġstudy Ġare Ġsummarized Ġin ĠTable ĠT AB REF 28 Ġ. ĠThey Ġshow Ġthat Ġa Ġmajority Ġof Ġquestions Ġthat Ġour Ġsystem Ġcould Ġnot Ġanswer Ġso Ġfar Ġare Ġin Ġfact Ġanswer able . ĠThis Ġsuggests Ġthat Ġ1 ) Ġthe Ġoriginal Ġhuman Ġbas elines Ġmight Ġhave Ġbeen Ġunderestimated , Ġhowever , Ġit Ġmight Ġalso Ġbe Ġthe Ġcase Ġthat Ġthere Ġare Ġsome Ġexamples Ġthat Ġcan Ġbe Ġanswered Ġby Ġmachines Ġand Ġnot Ġby Ġhumans ; Ġ2 ) Ġthere Ġis Ġstill Ġspace Ġfor Ġimprovement . </s> A Ġsystem Ġthat Ġwould Ġanswer Ġcorrectly Ġevery Ġtime Ġwhen Ġeither Ġour Ġensemble Ġor Ġhuman Ġanswered Ġcorrectly Ġwould Ġachieve Ġaccuracy Ġover Ġ92 % Ġpercent Ġon Ġboth Ġvalidation Ġand Ġtest ĠNE Ġdatasets Ġand Ġover Ġ96 % Ġon Ġboth ĠCN Ġdatasets . ĠHence Ġit Ġstill Ġmakes Ġsense Ġto Ġuse ĠCB T Ġdataset Ġto Ġstudy Ġfurther Ġimprovements Ġof Ġtext - com pre hens ion Ġsystems . </s> Conclusion </s> Few Ġways Ġof Ġimproving Ġmodel Ġperformance Ġare Ġas Ġsolidly Ġestablished Ġas Ġusing Ġmore Ġtraining Ġdata . ĠYet Ġwe Ġbelieve Ġthis Ġprinciple Ġhas Ġbeen Ġsomewhat Ġneglected Ġby Ġrecent Ġresearch Ġin Ġtext Ġcomprehension . ĠWhile Ġthere Ġis Ġa Ġpractically Ġunlimited Ġamount Ġof Ġdata Ġavailable Ġin Ġthis Ġfield , Ġmost Ġresearch Ġwas Ġperformed Ġon Ġunnecessarily Ġsmall Ġdatasets . </s> As Ġa Ġgentle Ġreminder Ġto Ġthe Ġcommunity Ġwe Ġhave Ġshown Ġthat Ġsimply Ġinf using Ġa Ġmodel Ġwith Ġmore Ġdata Ġcan Ġyield Ġperformance Ġimprovements Ġof Ġup Ġto ĠIN LINE FORM 0 Ġwhere Ġseveral Ġattempts Ġto Ġimprove Ġthe Ġmodel Ġarchitecture Ġon Ġthe Ġsame Ġtraining Ġdata Ġhave Ġgiven Ġgains Ġof Ġat Ġmost ĠIN LINE FORM 1 Ġcompared Ġto Ġour Ġbest Ġensemble Ġresult . ĠYes , Ġexperiments Ġon Ġsmall Ġdatasets Ġcertainly Ġcan Ġbring Ġuseful Ġinsights . ĠHowever Ġwe Ġbelieve Ġthat Ġthe Ġcommunity Ġshould Ġalso Ġembrace Ġthe Ġreal - world Ġscenario Ġof Ġdata Ġabundance . </s> The ĠBook Test Ġdataset Ġwe Ġare Ġproposing Ġgives Ġthe Ġreading - com pre hens ion Ġcommunity Ġan Ġopportunity Ġto Ġmake Ġa Ġstep Ġin Ġthat Ġdirection . </s> Training ĠDetails </s> The Ġtraining Ġdetails Ġare Ġsimilar Ġto Ġthose Ġin ĠB IB REF 4 Ġhowever Ġwe Ġare Ġincluding Ġthem Ġhere Ġfor Ġcomple teness . </s> To Ġtrain Ġthe Ġmodel Ġwe Ġused Ġsto ch astic Ġgradient Ġdescent Ġwith Ġthe ĠAD AM Ġupdate Ġrule ĠB IB REF 32 Ġand Ġlearning Ġrates Ġof ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġbest Ġlearning Ġrate Ġin Ġour Ġexperiments Ġwas ĠIN LINE FORM 3 Ġ. ĠWe Ġminimized Ġnegative Ġlog - like lihood Ġas Ġthe Ġtraining Ġobjective . </s> The Ġinitial Ġweights Ġin Ġthe Ġword - embed ding Ġmatrix Ġwere Ġdrawn Ġrandomly Ġuniformly Ġfrom Ġthe Ġinterval ĠIN LINE FORM 0 Ġ. ĠWe ights Ġin Ġthe ĠGR U Ġnetworks Ġwere Ġinitialized Ġby Ġrandom Ġorth og onal Ġmat rices ĠB IB REF 34 Ġand Ġbiases Ġwere Ġinitialized Ġto Ġzero . ĠWe Ġalso Ġused Ġa Ġgradient Ġclipping ĠB IB REF 33 Ġthreshold Ġof Ġ10 Ġand Ġbatches Ġof Ġsizes Ġbetween Ġ32 Ġor Ġ256 . ĠIncreasing Ġthe Ġbatch Ġfrom Ġ32 Ġto Ġ128 Ġseems Ġto Ġsignificantly Ġimprove Ġperformance Ġon Ġthe Ġlarge Ġdataset Ġ- Ġsomething Ġwe Ġdid Ġnot Ġobserve Ġon Ġthe Ġoriginal ĠCB T Ġdata . ĠIncreasing Ġthe Ġbatch Ġsize Ġmuch Ġabove Ġ128 Ġis Ġcurrently Ġdifficult Ġdue Ġto Ġmemory Ġconstraints Ġof Ġthe ĠGPU . </s> During Ġtraining Ġwe Ġrandomly Ġshuff led Ġall Ġexamples Ġat Ġthe Ġbeginning Ġof Ġeach Ġepoch . ĠTo Ġspeed Ġup Ġtraining , Ġwe Ġalways Ġpre - fetched Ġ10 Ġbatches Ġworth Ġof Ġexamples Ġand Ġsorted Ġthem Ġaccording Ġto Ġdocument Ġlength . ĠHence Ġeach Ġbatch Ġcontained Ġdocuments Ġof Ġroughly Ġthe Ġsame Ġlength . </s> We Ġalso Ġdid Ġnot Ġuse Ġpre - trained Ġword Ġembed d ings . </s> We Ġdid Ġnot Ġperform Ġany Ġtext Ġpre - processing Ġsince Ġthe Ġdatasets Ġwere Ġalready Ġtoken ized . </s> During Ġtraining Ġwe Ġevaluated Ġthe Ġmodel Ġperformance Ġevery Ġ12 Ġhours Ġand Ġat Ġthe Ġend Ġof Ġeach Ġepoch Ġand Ġstopped Ġtraining Ġwhen Ġthe Ġerror Ġon Ġthe Ġ20 k ĠBook Test Ġvalidation Ġset Ġstarted Ġincreasing . ĠWe Ġexplored Ġthe Ġhyper param eter Ġspace Ġby Ġtraining Ġ67 Ġdifferent Ġmodels ĠThe Ġregion Ġof Ġthe Ġparameter Ġspace Ġthat Ġwe Ġexplored Ġtogether Ġwith Ġthe Ġparameters Ġof Ġthe Ġmodel Ġwith Ġbest Ġvalidation Ġaccuracy Ġare Ġsummarized Ġin ĠTable ĠT AB REF 29 Ġ. </s> Our Ġmodel Ġwas Ġimplemented Ġusing ĠThe ano ĠB IB REF 31 Ġand ĠBlocks ĠB IB REF 35 Ġ. </s> The Ġen semb les Ġwere Ġformed Ġby Ġsimply Ġaveraging Ġthe Ġpredictions Ġfrom Ġthe Ġconstituent Ġsingle Ġmodels . ĠThese Ġsingle Ġmodels Ġwere Ġselected Ġusing Ġthe Ġfollowing Ġalgorithm . </s> We Ġstarted Ġwith Ġthe Ġbest Ġperforming Ġmodel Ġaccording Ġto Ġvalidation Ġperformance . ĠThen Ġin Ġeach Ġstep Ġwe Ġtried Ġadding Ġthe Ġbest Ġperforming Ġmodel Ġthat Ġhad Ġnot Ġbeen Ġpreviously Ġtried . ĠWe Ġkept Ġit Ġin Ġthe Ġensemble Ġif Ġit Ġdid Ġimprove Ġits Ġvalidation Ġperformance Ġand Ġdiscarded Ġit Ġotherwise . ĠThis Ġway Ġwe Ġgradually Ġtried Ġeach Ġmodel Ġonce . ĠWe Ġcall Ġthe Ġresulting Ġmodel Ġa Ġgreedy Ġensemble . ĠWe Ġused Ġthe ĠIN LINE FORM 0 ĠBook Test Ġvalidation Ġdataset Ġfor Ġthis Ġprocedure . </s> The Ġalgorithm Ġwas Ġoffered Ġ10 Ġmodels Ġand Ġselected Ġ5 Ġof Ġthem Ġfor Ġthe Ġfinal Ġensemble .