Unanswerable
NONE
Unanswerable
NONE
1804.00720
Introduction </s> Deep Ġlearning Ġsystems Ġhave Ġshown Ġa Ġlot Ġof Ġpromise Ġfor Ġextract ive ĠQuestion ĠAn sw ering Ġ( Q A ), Ġwith Ġperformance Ġcomparable Ġto Ġhumans Ġwhen Ġlarge Ġscale Ġdata Ġis Ġavailable . ĠHowever , Ġpractitioners Ġlooking Ġto Ġbuild ĠQ A Ġsystems Ġfor Ġspecific Ġapplications Ġmay Ġnot Ġhave Ġthe Ġresources Ġto Ġcollect Ġtens Ġof Ġthousands Ġof Ġquestions Ġon Ġcorpor a Ġof Ġtheir Ġchoice . ĠAt Ġthe Ġsame Ġtime , Ġstate - of - the - art Ġmachine Ġreading Ġsystems Ġdo Ġnot Ġlend Ġwell Ġto Ġlow - resource ĠQ A Ġsettings Ġwhere Ġthe Ġnumber Ġof Ġlabeled Ġquestion - answer Ġpairs Ġare Ġlimited Ġ( c . f . ĠTable Ġ2 Ġ). ĠSemi - super vised ĠQ A Ġmethods Ġlike ĠB IB REF 0 Ġaim Ġto Ġimprove Ġthis Ġperformance Ġby Ġleveraging Ġunl abel ed Ġdata Ġwhich Ġis Ġeasier Ġto Ġcollect . </s> In Ġthis Ġwork , Ġwe Ġpresent Ġa Ġsemi - super vised ĠQ A Ġsystem Ġwhich Ġrequires Ġthe Ġend Ġuser Ġto Ġspecify Ġa Ġset Ġof Ġbase Ġdocuments Ġand Ġonly Ġa Ġsmall Ġset Ġof Ġquestion - answer Ġpairs Ġover Ġa Ġsubset Ġof Ġthese Ġdocuments . ĠOur Ġproposed Ġsystem Ġconsists Ġof Ġthree Ġstages . ĠFirst , Ġwe Ġconstruct Ġclo ze - style Ġquestions Ġ( p redict ing Ġmissing Ġspans Ġof Ġtext ) Ġfrom Ġthe Ġunl abel ed Ġcorpus ; Ġnext , Ġwe Ġuse Ġthe Ġgenerated Ġcl oz es Ġto Ġpre - train Ġa Ġpowerful Ġneural Ġnetwork Ġmodel Ġfor Ġextract ive ĠQ A ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ; Ġand Ġfinally , Ġwe Ġfine - t une Ġthe Ġmodel Ġon Ġthe Ġsmall Ġset Ġof Ġprovided ĠQ A Ġpairs . </s> Our Ġclo ze Ġconstruction Ġprocess Ġbuilds Ġon Ġa Ġtypical Ġwriting Ġphenomenon Ġand Ġdocument Ġstructure : Ġan Ġintroduction Ġpreced es Ġand Ġsummarizes Ġthe Ġmain Ġbody Ġof Ġthe Ġarticle . ĠMany Ġlarge Ġcorpor a Ġfollow Ġsuch Ġa Ġstructure , Ġincluding ĠWikipedia , Ġacademic Ġpapers , Ġand Ġnews Ġarticles . ĠWe Ġhypothes ize Ġthat Ġwe Ġcan Ġbenefit Ġfrom Ġthe Ġun - annot ated Ġcorpor a Ġto Ġbetter Ġanswer Ġvarious Ġquestions ĠâĢĵ Ġat Ġleast Ġones Ġthat Ġare Ġlex ically Ġsimilar Ġto Ġthe Ġcontent Ġin Ġbase Ġdocuments Ġand Ġdirectly Ġrequire Ġfactual Ġinformation . </s> We Ġapply Ġthe Ġproposed Ġsystem Ġon Ġthree Ġdatasets Ġfrom Ġdifferent Ġdomains ĠâĢĵ ĠS Qu AD ĠB IB REF 3 Ġ, ĠTri via Q A - Web ĠB IB REF 4 Ġand Ġthe ĠBio AS Q Ġchallenge ĠB IB REF 5 Ġ. ĠWe Ġobserve Ġsignificant Ġimprovements Ġin Ġa Ġlow - resource Ġsetting Ġacross Ġall Ġthree Ġdatasets . ĠFor ĠS Qu AD Ġand ĠTri via Q A , Ġwe Ġattain Ġan ĠF 1 Ġscore Ġof Ġmore Ġthan Ġ50 % Ġby Ġmerely Ġusing Ġ1 % Ġof Ġthe Ġtraining Ġdata . ĠOur Ġsystem Ġoutper forms Ġthe Ġapproaches Ġfor Ġsemi - super vised ĠQ A Ġpresented Ġin ĠB IB REF 0 Ġ, Ġand Ġa Ġbaseline Ġwhich Ġuses Ġthe Ġsame Ġunl abel ed Ġdata Ġbut Ġwith Ġa Ġlanguage Ġmodeling Ġobjective Ġfor Ġpret raining . ĠIn Ġthe ĠBio AS Q Ġchallenge , Ġwe Ġoutper form Ġthe Ġbest Ġperforming Ġsystem Ġfrom Ġprevious Ġyear 's Ġchallenge , Ġimproving Ġover Ġa Ġbaseline Ġwhich Ġdoes Ġtransfer Ġlearning Ġfrom Ġthe ĠS Qu AD Ġdataset . ĠOur Ġanalysis Ġreveals Ġthat Ġquestions Ġwhich Ġask Ġfor Ġfactual Ġinformation Ġand Ġmatch Ġto Ġspecific Ġparts Ġof Ġthe Ġcontext Ġdocuments Ġbenefit Ġthe Ġmost Ġfrom Ġpret raining Ġon Ġautomatically Ġconstructed Ġcl oz es . </s> Related ĠWork </s> Sem i - super vised Ġlearning Ġaug ments Ġthe Ġlabeled Ġdataset Ġ$ L $ Ġwith Ġa Ġpotentially Ġlarger Ġunl abel ed Ġdataset Ġ$ U $ Ġ. ĠB IB REF 0 Ġpresented Ġa Ġmodel , ĠGD AN , Ġwhich Ġtrained Ġan Ġauxiliary Ġneural Ġnetwork Ġto Ġgenerate Ġquestions Ġfrom Ġpassages Ġby Ġreinforcement Ġlearning , Ġand Ġaugment Ġthe Ġlabeled Ġdataset Ġwith Ġthe Ġgenerated Ġquestions Ġto Ġtrain Ġthe ĠQ A Ġmodel . ĠHere Ġwe Ġuse Ġa Ġmuch Ġsimpler Ġhe uristic Ġto Ġgenerate Ġthe Ġauxiliary Ġquestions , Ġwhich Ġalso Ġturns Ġout Ġto Ġbe Ġmore Ġeffective Ġas Ġwe Ġshow Ġsuperior Ġperformance Ġcompared Ġto ĠGD AN . ĠSeveral Ġapproaches Ġhave Ġbeen Ġsuggested Ġfor Ġgenerating Ġnatural Ġquestions ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġhowever Ġnone Ġof Ġthem Ġshow Ġa Ġsignificant Ġimprovement Ġof Ġusing Ġthe Ġgenerated Ġquestions Ġin Ġa Ġsemi - super vised Ġsetting . ĠRecent Ġpapers Ġalso Ġuse Ġunl abel ed Ġdata Ġfor ĠQ A Ġby Ġtraining Ġlarge Ġlanguage Ġmodels Ġand Ġextracting Ġcontextual Ġword Ġvectors Ġfrom Ġthem Ġto Ġinput Ġto Ġthe ĠQ A Ġmodel ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠThe Ġapplic ability Ġof Ġthis Ġmethod Ġin Ġthe Ġlow - resource Ġsetting Ġis Ġunclear Ġas Ġthe Ġextra Ġinputs Ġincrease Ġthe Ġnumber Ġof Ġparameters Ġin Ġthe ĠQ A Ġmodel , Ġhowever , Ġour Ġpret raining Ġcan Ġbe Ġeasily Ġapplied Ġto Ġthese Ġmodels Ġas Ġwell . </s> Domain Ġadaptation Ġ( and ĠTransfer Ġlearning ) Ġleverage Ġexisting Ġlarge Ġscale Ġdatasets Ġfrom Ġa Ġsource Ġdomain Ġ( or Ġtask ) Ġto Ġimprove Ġperformance Ġon Ġa Ġtarget Ġdomain Ġ( or Ġtask ). ĠFor Ġdeep Ġlearning Ġand ĠQ A , Ġa Ġcommon Ġapproach Ġis Ġto Ġpret rain Ġon Ġthe Ġsource Ġdataset Ġand Ġthen Ġfine - t une Ġon Ġthe Ġtarget Ġdataset ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠB IB REF 14 Ġused ĠS Qu AD Ġas Ġa Ġsource Ġfor Ġthe Ġtarget ĠBio AS Q Ġdataset , Ġand ĠB IB REF 15 Ġused ĠBook ĠTest ĠB IB REF 16 Ġas Ġsource Ġfor Ġthe Ġtarget ĠS Qu AD Ġdataset . ĠB IB REF 17 Ġtransfer Ġlearned Ġmodel Ġlayers Ġfrom Ġthe Ġtasks Ġof Ġsequence Ġlabeling , Ġtext Ġclassification Ġand Ġrelation Ġclassification Ġto Ġshow Ġsmall Ġimprovements Ġon ĠS Qu AD . ĠAll Ġthese Ġworks Ġuse Ġmanually Ġcurated Ġsource Ġdat ats et , Ġwhich Ġin Ġthemselves Ġare Ġexpensive Ġto Ġcollect . ĠInstead , Ġwe Ġshow Ġthat Ġit Ġis Ġpossible Ġto Ġautomatically Ġconstruct Ġthe Ġsource Ġdataset Ġfrom Ġthe Ġsame Ġdomain Ġas Ġthe Ġtarget , Ġwhich Ġturns Ġout Ġto Ġbe Ġmore Ġbeneficial Ġin Ġterms Ġof Ġperformance Ġas Ġwell Ġ( c . f . ĠSection Ġ" Exper iments Ġ& ĠResults " Ġ). ĠSeveral Ġclo ze Ġdatasets Ġhave Ġbeen Ġproposed Ġin Ġthe Ġliterature Ġwhich Ġuse Ġhe ur istics Ġfor Ġconstruction ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠWe Ġfurther Ġsee Ġthe Ġusability Ġof Ġsuch Ġa Ġdataset Ġin Ġa Ġsemi - super vised Ġsetting . </s> Method ology </s> Our Ġsystem Ġcomprises Ġof Ġfollowing Ġthree Ġsteps : </s> Cl o ze Ġgeneration : ĠMost Ġof Ġthe Ġdocuments Ġtypically Ġfollow Ġa Ġtemplate , Ġthey Ġbegin Ġwith Ġan Ġintroduction Ġthat Ġprovides Ġan Ġoverview Ġand Ġa Ġbrief Ġsummary Ġfor Ġwhat Ġis Ġto Ġfollow . ĠWe Ġassume Ġsuch Ġa Ġstructure Ġwhile Ġconstructing Ġour Ġclo ze Ġstyle Ġquestions . ĠWhen Ġthere Ġis Ġno Ġclear Ġdem arc ation , Ġwe Ġtreat Ġthe Ġfirst Ġ$ K \ % $ Ġ( hyper param eter , Ġin Ġour Ġcase Ġ20 %) Ġof Ġthe Ġdocument Ġas Ġthe Ġintroduction . ĠWhile Ġnoisy , Ġthis Ġhe uristic Ġgenerates Ġa Ġlarge Ġnumber Ġof Ġcl oz es Ġgiven Ġany Ġcorpus , Ġwhich Ġwe Ġfound Ġto Ġbe Ġbeneficial Ġfor Ġsemi - super vised Ġlearning Ġdespite Ġthe Ġnoise . </s> We Ġuse Ġa Ġstandard ĠN LP Ġpipeline Ġbased Ġon ĠStanford ĠCore N LP Ġ( for ĠS Qu AD , ĠTri va Q A Ġand ĠPubMed ) Ġand Ġthe ĠB AN NER ĠNamed ĠEntity ĠRecogn izer Ġ( only Ġfor ĠPubMed Ġarticles ) Ġto Ġidentify Ġentities Ġand Ġphrases . ĠAss ume Ġthat Ġa Ġdocument Ġcomprises Ġof Ġintroduction Ġsentences Ġ$\ l brace Ġq _ 1 , Ġq _ 2 , Ġ... Ġq _ n \ r brace Ġ$ Ġ, Ġand Ġthe Ġremaining Ġpassages Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ Ġ. ĠAdditionally , Ġlet 's Ġsay Ġthat Ġeach Ġsentence Ġ$ q _ i $ Ġin Ġintroduction Ġis Ġcomposed Ġof Ġwords Ġ$\ l brace Ġw _ 1 , Ġw _ 2 , Ġ... Ġw _{ l _{ q _ i }} \ r brace Ġ$ Ġ, Ġwhere Ġ$ l _{ q _ i } $ Ġis Ġthe Ġlength Ġof Ġ$ q _ i $ Ġ. ĠWe Ġconsider Ġa Ġ$\ text { match } Ġ( q _ i , Ġp _ j ) $ Ġ, Ġif Ġthere Ġis Ġan Ġexact Ġstring Ġmatch Ġof Ġa Ġsequence Ġof Ġwords Ġ$\ l brace Ġw _ k , Ġw _{ k + 1 }, Ġ.. Ġw _{ l _{ q _ i }} \ r brace Ġ$ Ġbetween Ġthe Ġsentence Ġ$ q _ i $ Ġand Ġpassage Ġ$ p _ j $ Ġ. ĠIf Ġthis Ġsequence Ġis Ġeither Ġa Ġnoun Ġphrase , Ġverb Ġphrase , Ġadjective Ġphrase Ġor Ġa Ġnamed Ġentity Ġin Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 0 Ġ, Ġas Ġrecognized Ġby ĠCore N LP Ġor ĠB AN NER , Ġwe Ġselect Ġit Ġas Ġan Ġanswer Ġspan Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 1 Ġ. ĠAdditionally , Ġwe Ġuse Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 2 Ġas Ġthe Ġpassage Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 3 Ġand Ġform Ġa Ġclo ze Ġquestion Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 4 Ġfrom Ġthe Ġanswer Ġbearing Ġsentence Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 5 Ġby Ġreplacing Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 6 Ġwith Ġa Ġplaceholder . ĠAs Ġa Ġresult , Ġwe Ġobtain Ġpassage - question - answer Ġ( Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 7 Ġ) Ġtri ples Ġ( Table Ġ1 Ġshows Ġan Ġexample ). ĠAs Ġa Ġpost - processing Ġstep , Ġwe Ġpr une Ġout Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 8 Ġtri ples Ġwhere Ġthe Ġword Ġoverlap Ġbetween Ġthe Ġquestion Ġ( Q ) Ġand Ġpassage Ġ( P ) Ġis Ġless Ġthan Ġ2 Ġwords Ġ( after Ġexcluding Ġthe Ġstop Ġwords ). </s> The Ġprocess Ġrelies Ġon Ġthe Ġfact Ġthat Ġanswer Ġcandidates Ġfrom Ġthe Ġintroduction Ġare Ġlikely Ġto Ġbe Ġdiscussed Ġin Ġdetail Ġin Ġthe Ġremainder Ġof Ġthe Ġarticle . ĠIn Ġeffect , Ġthe Ġclo ze Ġquestion Ġfrom Ġthe Ġintroduction Ġand Ġthe Ġmatching Ġparagraph Ġin Ġthe Ġbody Ġforms Ġa Ġquestion Ġand Ġcontext Ġpassage Ġpair . ĠWe Ġcreate Ġtwo Ġclo ze Ġdatasets , Ġone Ġeach Ġfrom ĠWikipedia Ġcorpus Ġ( for ĠS Qu AD Ġand ĠTri via Q A ) Ġand ĠP UB Med Ġacademic Ġpapers Ġ( for Ġthe ĠBio AS Q Ġchallenge ), Ġconsisting Ġof Ġ2 . 2 M Ġand Ġ1 M Ġcl oz es Ġrespectively . ĠFrom Ġanalyzing Ġthe Ġclo ze Ġdata Ġmanually , Ġwe Ġwere Ġable Ġto Ġanswer Ġ76 % Ġtimes Ġfor Ġthe ĠWikipedia Ġset Ġand Ġ80 % Ġtimes Ġfor Ġthe ĠP UB Med Ġset Ġusing Ġthe Ġinformation Ġin Ġthe Ġpassage . ĠIn Ġmost Ġcases Ġthe Ġclo ze Ġparaph r ased Ġthe Ġinformation Ġin Ġthe Ġpassage , Ġwhich Ġwe Ġhypothesized Ġto Ġbe Ġa Ġuseful Ġsignal Ġfor Ġthe Ġdownstream ĠQ A Ġtask . </s> We Ġalso Ġinvestigate Ġthe Ġutility Ġof Ġforming Ġsubs ets Ġof Ġthe Ġlarge Ġclo ze Ġcorpus , Ġwhere Ġwe Ġselect Ġthe Ġtop Ġpassage - question - answer Ġtri ples , Ġbased Ġon Ġthe Ġdifferent Ġcriteria , Ġlike Ġi ) Ġj acc ard Ġsimilarity Ġof Ġanswer Ġbearing Ġsentence Ġin Ġintroduction Ġand Ġthe Ġpassage Ġii ) Ġthe Ġtf - id f Ġscores Ġof Ġanswer Ġcandidates Ġand Ġiii ) Ġthe Ġlength Ġof Ġanswer Ġcandidates . ĠHowever , Ġwe Ġempir ically Ġfind Ġthat Ġwe Ġwere Ġbetter Ġoff Ġusing Ġthe Ġentire Ġset Ġrather Ġthan Ġthese Ġsubs ets . </s> Pre - training : ĠWe Ġmake Ġuse Ġof Ġthe Ġgenerated Ġclo ze Ġdataset Ġto Ġpre - train Ġan Ġexpressive Ġneural Ġnetwork Ġdesigned Ġfor Ġthe Ġtask Ġof Ġreading Ġcomprehension . ĠWe Ġwork Ġwith Ġtwo Ġpublicly Ġavailable Ġneural Ġnetwork Ġmodels ĠâĢĵ Ġthe ĠGA ĠReader ĠB IB REF 2 Ġ( to Ġenable Ġcomparison Ġwith Ġprior Ġwork ) Ġand ĠBi DA F Ġ+ ĠSelf - Att ention Ġ( SA ) Ġmodel Ġfrom ĠB IB REF 1 Ġ( which Ġis Ġamong Ġthe Ġbest Ġperforming Ġmodels Ġon ĠS Qu AD Ġand ĠTri via Q A ). ĠAfter Ġpret raining , Ġthe Ġperformance Ġof ĠBi DA F + SA Ġon Ġa Ġdev Ġset Ġof Ġthe Ġ( Wikipedia ) Ġclo ze Ġquestions Ġis Ġ0 . 58 ĠF 1 Ġscore Ġand Ġ0 . 55 ĠEx act ĠMatch Ġ( EM ) Ġscore . ĠThis Ġimplies Ġthat Ġthe Ġclo ze Ġcorpus Ġis Ġneither Ġtoo Ġeasy , Ġnor Ġtoo Ġdifficult Ġto Ġanswer . </s> Fine ĠTun ing : ĠWe Ġfine Ġtune Ġthe Ġpre - trained Ġmodel , Ġfrom Ġthe Ġprevious Ġstep , Ġover Ġa Ġsmall Ġset Ġof Ġlabelled Ġquestion - answer Ġpairs . ĠAs Ġwe Ġshall Ġlater Ġsee , Ġthis Ġstep Ġis Ġcrucial , Ġand Ġit Ġonly Ġrequires Ġa Ġhandful Ġof Ġlabelled Ġquestions Ġto Ġachieve Ġa Ġsignificant Ġproportion Ġof Ġthe Ġperformance Ġtypically Ġattained Ġby Ġtraining Ġon Ġtens Ġof Ġthousands Ġof Ġquestions . </s> Dat as ets </s> We Ġapply Ġour Ġsystem Ġto Ġthree Ġdatasets Ġfrom Ġdifferent Ġdomains . ĠS Qu AD ĠB IB REF 3 Ġconsists Ġof Ġquestions Ġwhose Ġanswers Ġare Ġfree Ġform Ġspans Ġof Ġtext Ġfrom Ġpassages Ġin ĠWikipedia Ġarticles . ĠWe Ġfollow Ġthe Ġsame Ġsetting Ġas Ġin ĠB IB REF 0 Ġ, Ġand Ġsplit Ġ$ 10 \ % $ Ġof Ġtraining Ġquestions Ġas Ġthe Ġtest Ġset , Ġand Ġreport Ġperformance Ġwhen Ġtraining Ġon Ġsubs ets Ġof Ġthe Ġremaining Ġdata Ġranging Ġfrom Ġ$ 1 \ % $ Ġto Ġ$ 90 \ % $ Ġof Ġthe Ġfull Ġset . ĠWe Ġalso Ġreport Ġthe Ġperformance Ġon Ġthe Ġdev Ġset Ġwhen Ġtrained Ġon Ġthe Ġfull Ġtraining Ġset Ġ( Ġ$ 1 ^ \ ast Ġ$ Ġin ĠTable Ġ2 Ġ). ĠWe Ġuse Ġthe Ġsame Ġhyper param eter Ġsettings Ġas Ġin Ġprior Ġwork . ĠWe Ġcompare Ġand Ġstudy Ġfour Ġdifferent Ġsettings : Ġ1 ) Ġthe ĠSuper vised ĠLearning Ġ( SL ) Ġsetting , Ġwhich Ġis Ġonly Ġtrained Ġon Ġthe Ġsupervised Ġdata , Ġ2 ) Ġthe Ġbest Ġperforming ĠGD AN Ġmodel Ġfrom ĠB IB REF 0 Ġ, Ġ3 ) Ġpret raining Ġon Ġa ĠLanguage ĠModel ing Ġ( LM ) Ġobjective Ġand Ġfine - tun ing Ġon Ġthe Ġsupervised Ġdata , Ġand Ġ4 ) Ġpret raining Ġon Ġthe ĠClo ze Ġdataset Ġand Ġfine - tun ing Ġon Ġthe Ġsupervised Ġdata . ĠThe ĠLM Ġand ĠClo ze Ġmethods Ġuse Ġexactly Ġthe Ġsame Ġdata Ġfor Ġpret raining , Ġbut Ġdiffer Ġin Ġthe Ġloss Ġfunctions Ġused . ĠWe Ġreport ĠF 1 Ġand ĠEM Ġscores Ġon Ġour Ġtest Ġset Ġusing Ġthe Ġofficial Ġevaluation Ġscripts Ġprovided Ġby Ġthe Ġauthors Ġof Ġthe Ġdataset . </s> Trivia Q A ĠB IB REF 4 Ġcomprises Ġof Ġover Ġ95 K Ġweb Ġquestion - answer - evidence Ġtri ples . ĠLike ĠS Qu AD , Ġthe Ġanswers Ġare Ġspans Ġof Ġtext . ĠSimilar Ġto Ġthe Ġsetting Ġin ĠS Qu AD , Ġwe Ġcreate Ġmultiple Ġsmaller Ġsubs ets Ġof Ġthe Ġentire Ġset . ĠFor Ġour Ġsemi - super vised ĠQ A Ġsystem , Ġwe Ġuse Ġthe ĠBi DA F + SA Ġmodel ĠB IB REF 1 ĠâĢĵ Ġthe Ġhighest Ġperforming Ġpublicly Ġavailable Ġsystem Ġfor ĠTri va Q A . ĠHere Ġagain , Ġwe Ġcompare Ġthe Ġsupervised Ġlearning Ġ( SL ) Ġsettings Ġagainst Ġthe Ġpret raining Ġon ĠClo ze Ġset Ġand Ġfine Ġtuning Ġon Ġthe Ġsupervised Ġset . ĠWe Ġreport ĠF 1 Ġand ĠEM Ġscores Ġon Ġthe Ġdev Ġset . </s> We Ġalso Ġtest Ġon Ġthe ĠBio AS Q Ġ5 b Ġdataset , Ġwhich Ġconsists Ġof Ġquestion - answer Ġpairs Ġfrom ĠPubMed Ġabstract s . ĠWe Ġuse Ġthe Ġpublicly Ġavailable Ġsystem Ġfrom ĠB IB REF 14 Ġ, Ġand Ġfollow Ġthe Ġexact Ġsame Ġsetup Ġas Ġtheirs , Ġfocusing Ġonly Ġon Ġfact oid Ġand Ġlist Ġquestions . ĠFor Ġthis Ġsetting , Ġthere Ġare Ġonly Ġ8 99 Ġquestions Ġfor Ġtraining . ĠSince Ġthis Ġis Ġalready Ġa Ġlow - resource Ġproblem Ġwe Ġonly Ġreport Ġresults Ġusing Ġ5 - fold Ġcross - valid ation Ġon Ġall Ġthe Ġavailable Ġdata . ĠWe Ġreport ĠMean ĠRec ip rocal ĠRank Ġ( MR R ) Ġon Ġthe Ġfact oid Ġquestions , Ġand ĠF 1 Ġscore Ġfor Ġthe Ġlist Ġquestions . </s> Main ĠResults </s> Table Ġ2 Ġshows Ġa Ġcomparison Ġof Ġthe Ġdiscussed Ġsettings Ġon Ġboth ĠS Qu AD Ġand ĠTri via Q A . ĠWithout Ġany Ġfine - tun ing Ġ( column Ġ0 ) Ġthe Ġperformance Ġis Ġlow , Ġprobably Ġbecause Ġthe Ġmodel Ġnever Ġsaw Ġa Ġreal Ġquestion , Ġbut Ġwe Ġsee Ġsignificant Ġgains Ġwith ĠClo ze Ġpret raining Ġeven Ġwith Ġvery Ġlittle Ġlabeled Ġdata . ĠThe ĠBi DA F + SA Ġmodel , Ġexceeds Ġan ĠF 1 Ġscore Ġof Ġ$ 50 \ % $ Ġwith Ġonly Ġ$ 1 \ % $ Ġof Ġthe Ġtraining Ġdata Ġ( 454 Ġquestions Ġfor ĠS Qu AD , Ġand Ġ7 46 Ġquestions Ġfor ĠTri via Q A ), Ġand Ġapproaches Ġ$ 90 \ % $ Ġof Ġthe Ġbest Ġperformance Ġwith Ġonly Ġ$ 10 \ % $ Ġlabeled Ġdata . ĠThe Ġgains Ġover Ġthe ĠSL Ġsetting , Ġhowever , Ġdiminish Ġas Ġthe Ġsize Ġof Ġthe Ġlabeled Ġset Ġincreases Ġand Ġare Ġsmall Ġwhen Ġthe Ġfull Ġdataset Ġis Ġavailable . </s> Cl o ze Ġpret raining Ġoutper forms Ġthe ĠGD AN Ġbaseline Ġfrom ĠB IB REF 0 Ġusing Ġthe Ġsame ĠS Qu AD Ġdataset Ġsplits . ĠAdditionally , Ġwe Ġshow Ġimprovements Ġin Ġthe Ġ$ 90 \ % $ Ġdata Ġcase Ġunlike ĠGD AN . ĠOur Ġapproach Ġis Ġalso Ġapplicable Ġin Ġthe Ġextremely Ġlow - resource Ġsetting Ġof Ġ$ 1 \ % $ Ġdata , Ġwhich Ġwe Ġsuspect ĠGD AN Ġmight Ġhave Ġtrouble Ġwith Ġsince Ġit Ġuses Ġthe Ġlabeled Ġdata Ġto Ġdo Ġreinforcement Ġlearning . ĠFurthermore , Ġwe Ġare Ġable Ġto Ġuse Ġthe Ġsame Ġclo ze Ġdataset Ġto Ġimprove Ġperformance Ġon Ġboth ĠS Qu AD Ġand ĠTri via Q A Ġdatasets . ĠWhen Ġwe Ġuse Ġthe Ġsame Ġunl abel ed Ġdata Ġto Ġpre - train Ġwith Ġa Ġlanguage Ġmodeling Ġobjective , Ġthe Ġperformance Ġis Ġworse , Ġshowing Ġthe Ġbias Ġwe Ġintroduce Ġby Ġconstructing Ġcl oz es Ġis Ġimportant . </s> On Ġthe ĠBio AS Q Ġdataset Ġ( Table Ġ3 Ġ) Ġwe Ġagain Ġsee Ġa Ġsignificant Ġimprovement Ġwhen Ġpret raining Ġwith Ġthe Ġclo ze Ġquestions Ġover Ġthe Ġsupervised Ġbaseline . ĠThe Ġimprovement Ġis Ġsmaller Ġthan Ġwhat Ġwe Ġobserve Ġwith ĠS Qu AD Ġand ĠTri via Q A Ġdatasets ĠâĢĵ Ġwe Ġbelieve Ġthis Ġis Ġbecause Ġquestions Ġare Ġgenerally Ġmore Ġdifficult Ġin ĠBio AS Q . ĠB IB REF 14 Ġshowed Ġthat Ġpret raining Ġon ĠS Qu AD Ġdataset Ġimproves Ġthe Ġdownstream Ġperformance Ġon ĠBio AS Q . ĠHere , Ġwe Ġshow Ġa Ġmuch Ġlarger Ġimprovement Ġby Ġpret raining Ġon Ġclo ze Ġquestions Ġconstructed Ġin Ġan Ġun super vised Ġmanner Ġfrom Ġthe Ġsame Ġdomain . </s> Analysis </s> Reg ression ĠAnalysis : ĠTo Ġunderstand Ġwhich Ġtypes Ġof Ġquestions Ġbenefit Ġfrom Ġpre - training , Ġwe Ġpre - specified Ġcertain Ġfeatures Ġ( see ĠFigure Ġ1 Ġright ) Ġfor Ġeach Ġof Ġthe Ġdev Ġset Ġquestions Ġin ĠS Qu AD , Ġand Ġthen Ġperformed Ġlinear Ġregression Ġto Ġpredict Ġthe ĠF 1 Ġscore Ġfor Ġthat Ġquestion Ġfrom Ġthese Ġfeatures . ĠWe Ġpredict Ġthe ĠF 1 Ġscores Ġfrom Ġthe Ġclo ze Ġpret rained Ġmodel Ġ( Ġ$ y ^ {\ text { cl o ze }} $ Ġ), Ġthe Ġsupervised Ġmodel Ġ( Ġ$ y ^ {\ text { sl }} $ Ġ), Ġand Ġthe Ġdifference Ġof Ġthe Ġtwo Ġ( Ġ$ y ^ {\ text { cl o ze }} - y ^ {\ text { sl }} $ Ġ), Ġwhen Ġusing Ġ$ 10 \ % $ Ġof Ġlabeled Ġdata . ĠThe Ġcoefficients Ġof Ġthe Ġfitted Ġmodel Ġare Ġshown Ġin ĠFigure Ġ1 Ġ( left ) Ġalong Ġwith Ġtheir Ġstd Ġerrors . ĠPositive Ġcoefficients Ġindicate Ġthat Ġa Ġhigh Ġvalue Ġof Ġthat Ġfeature Ġis Ġpredictive Ġof Ġa Ġhigh ĠF 1 Ġscore , Ġand Ġa Ġnegative Ġcoefficient Ġindicates Ġthat Ġa Ġsmall Ġvalue Ġof Ġthat Ġfeature Ġis Ġpredictive Ġof Ġa Ġhigh ĠF 1 Ġscore Ġ( or Ġa Ġhigh Ġdifference Ġof ĠF 1 Ġscores Ġfrom Ġthe Ġtwo Ġmodels Ġin Ġthe Ġcase Ġof Ġ$ y ^ {\ text { cl o ze }} - y ^ {\ text { sl }} $ Ġ). </s> The Ġtwo Ġstrongest Ġeffects Ġwe Ġobserve Ġare Ġthat Ġa Ġhigh Ġlex ical Ġoverlap Ġbetween Ġthe Ġquestion Ġand Ġthe Ġsentence Ġcontaining Ġthe Ġanswer Ġis Ġindicative Ġof Ġhigh Ġboost Ġwith Ġpret raining , Ġand Ġthat Ġa Ġhigh Ġlex ical Ġoverlap Ġbetween Ġthe Ġquestion Ġand Ġthe Ġwhole Ġpassage Ġis Ġindicative Ġof Ġthe Ġopposite . ĠThis Ġis Ġhardly Ġsurprising , Ġsince Ġour Ġclo ze Ġconstruction Ġprocess Ġis Ġbiased Ġtowards Ġquestions Ġwhich Ġhave Ġa Ġsimilar Ġphr asing Ġto Ġthe Ġanswer Ġsentences Ġin Ġcontext . ĠHence , Ġtest Ġquestions Ġwith Ġa Ġsimilar Ġproperty Ġare Ġanswered Ġcorrectly Ġafter Ġpret raining , Ġwhereas Ġthose Ġwith Ġa Ġhigh Ġoverlap Ġwith Ġthe Ġwhole Ġpassage Ġtend Ġto Ġhave Ġlower Ġperformance . ĠThe Ġpret raining Ġalso Ġfavors Ġquestions Ġwith Ġshort Ġanswers Ġbecause Ġthe Ġclo ze Ġconstruction Ġprocess Ġproduces Ġshort Ġanswer Ġspans . ĠAlso Ġpassages Ġand Ġquestions Ġwhich Ġconsist Ġof Ġtokens Ġinf requent Ġin Ġthe ĠS Qu AD Ġtraining Ġcorpus Ġreceive Ġa Ġlarge Ġboost Ġafter Ġpret raining , Ġsince Ġthe Ġunl abel ed Ġdata Ġcovers Ġa Ġlarger Ġdomain . </s> Performance Ġon Ġquestion Ġtypes : ĠFigure Ġ2 Ġshows Ġthe Ġaverage Ġgain Ġin ĠF 1 Ġscore Ġfor Ġdifferent Ġtypes Ġof Ġquestions , Ġwhen Ġwe Ġpret rain Ġon Ġthe Ġcl oz es Ġcompared Ġto Ġthe Ġsupervised Ġcase . ĠThis Ġanalysis Ġis Ġdone Ġon Ġthe Ġ$ 10 \ % $ Ġsplit Ġof Ġthe ĠS Qu AD Ġtraining Ġset . ĠWe Ġconsider Ġtwo Ġclass ifications Ġof Ġeach Ġquestion ĠâĢĵ Ġone Ġdetermined Ġon Ġthe Ġfirst Ġword Ġ( usually Ġa Ġwh - word ) Ġof Ġthe Ġquestion Ġ( Figure Ġ2 Ġ( bottom )) Ġand Ġone Ġbased Ġon Ġthe Ġoutput Ġof Ġa Ġseparate Ġquestion Ġtype Ġclass ifier Ġadapted Ġfrom ĠB IB REF 21 Ġ. ĠWe Ġuse Ġthe Ġcoarse Ġgrain Ġlabels Ġnamely ĠAb bre viation Ġ( AB BR ), ĠEntity Ġ( ENT Y ), ĠDescription Ġ( DES C ), ĠHuman Ġ( H UM ), ĠLocation Ġ( LOC ), ĠN umeric Ġ( NUM ) Ġtrained Ġon Ġa ĠLog istic ĠReg ression Ġclassification Ġsystem Ġ. ĠWhile Ġthere Ġis Ġan Ġimprovement Ġacross Ġthe Ġboard , Ġwe Ġfind Ġthat Ġabbre viation Ġquestions Ġin Ġparticular Ġreceive Ġa Ġlarge Ġboost . ĠAlso , Ġ" why " Ġquestions Ġshow Ġthe Ġleast Ġimprovement , Ġwhich Ġis Ġin Ġline Ġwith Ġour Ġexpectation , Ġsince Ġthese Ġusually Ġrequire Ġreasoning Ġor Ġworld Ġknowledge Ġwhich Ġclo ze Ġquestions Ġrarely Ġrequire . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġshow Ġthat Ġpre - training ĠQ A Ġmodels Ġwith Ġautomatically Ġconstructed Ġclo ze Ġquestions Ġimproves Ġthe Ġperformance Ġof Ġthe Ġmodels Ġsignificantly , Ġespecially Ġwhen Ġthere Ġare Ġfew Ġlabeled Ġexamples . ĠThe Ġperformance Ġof Ġthe Ġmodel Ġtrained Ġonly Ġon Ġthe Ġclo ze Ġquestions Ġis Ġpoor , Ġvalid ating Ġthe Ġneed Ġfor Ġfine - tun ing . ĠThrough Ġregression Ġanalysis , Ġwe Ġfind Ġthat Ġpret raining Ġhelps Ġwith Ġquestions Ġwhich Ġask Ġfor Ġfactual Ġinformation Ġlocated Ġin Ġa Ġspecific Ġpart Ġof Ġthe Ġcontext . ĠFor Ġfuture Ġwork , Ġwe Ġplan Ġto Ġexplore Ġthe Ġactive Ġlearning Ġsetup Ġfor Ġthis Ġtask ĠâĢĵ Ġspecifically , Ġwhich Ġpassages Ġand Ġ/ Ġor Ġtypes Ġof Ġquestions Ġcan Ġwe Ġselect Ġto Ġannot ate , Ġsuch Ġthat Ġthere Ġis Ġa Ġmaximum Ġperformance Ġgain Ġfrom Ġfine - tun ing . ĠWe Ġalso Ġwant Ġto Ġexplore Ġhow Ġto Ġadapt Ġclo ze Ġstyle Ġpre - training Ġto ĠN LP Ġtasks Ġother Ġthan ĠQ A . </s> Acknowled gments </s> B hu wan ĠD hing ra Ġis Ġsupported Ġby ĠNS F Ġunder Ġgrants ĠCC F - 14 140 30 Ġand ĠI IS - 12 50 956 Ġand Ġby Ġgrants Ġfrom ĠGoogle . ĠDanish ĠPr uth i Ġand ĠD he er aj ĠRaj ag opal Ġare Ġsupported Ġby Ġthe ĠDAR PA ĠBig ĠMechan ism Ġprogram Ġunder ĠAR O Ġcontract ĠW 911 NF - 14 - 1 - 04 36 .
Introduction Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available. However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice. At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c.f. Table 2 ). Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect. In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents. Our proposed system consists of three stages. First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs. Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article. Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles. We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information. We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5 . We observe significant improvements in a low-resource setting across all three datasets. For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data. Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining. In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset. Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes. Related Work Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$ . BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model. Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN. Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting. Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11 . The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well. Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task). For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13 . BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset. BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD. All these works use manually curated source datatset, which in themselves are expensive to collect. Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c.f. Section "Experiments & Results" ). Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20 . We further see the usability of such a dataset in a semi-supervised setting. Methodology Our system comprises of following three steps: Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow. We assume such a structure while constructing our cloze style questions. When there is no clear demarcation, we treat the first $K\%$ (hyperparameter, in our case 20%) of the document as the introduction. While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise. We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases. Assume that a document comprises of introduction sentences $\lbrace q_1, q_2, ... q_n\rbrace $ , and the remaining passages $\lbrace p_1, p_2, .. p_m\rbrace $ . Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\lbrace w_1, w_2, ... w_{l_{q_i}}\rbrace $ , where $l_{q_i}$ is the length of $q_i$ . We consider a $\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\lbrace w_k, w_{k+1}, .. w_{l_{q_i}}\rbrace $ between the sentence $q_i$ and passage $p_j$ . If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\lbrace p_1, p_2, .. p_m\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\lbrace p_1, p_2, .. p_m\rbrace $1 . Additionally, we use $\lbrace p_1, p_2, .. p_m\rbrace $2 as the passage $\lbrace p_1, p_2, .. p_m\rbrace $3 and form a cloze question $\lbrace p_1, p_2, .. p_m\rbrace $4 from the answer bearing sentence $\lbrace p_1, p_2, .. p_m\rbrace $5 by replacing $\lbrace p_1, p_2, .. p_m\rbrace $6 with a placeholder. As a result, we obtain passage-question-answer ( $\lbrace p_1, p_2, .. p_m\rbrace $7 ) triples (Table 1 shows an example). As a post-processing step, we prune out $\lbrace p_1, p_2, .. p_m\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words). The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article. In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair. We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2.2M and 1M clozes respectively. From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage. In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task. We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates. However, we empirically find that we were better off using the entire set rather than these subsets. Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension. We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA). After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0.58 F1 score and 0.55 Exact Match (EM) score. This implies that the cloze corpus is neither too easy, nor too difficult to answer. Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs. As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions. Datasets We apply our system to three datasets from different domains. SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles. We follow the same setting as in BIBREF0 , and split $10\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\%$ to $90\%$ of the full set. We also report the performance on the dev set when trained on the full training set ( $1^\ast $ in Table 2 ). We use the same hyperparameter settings as in prior work. We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data. The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used. We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset. TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples. Like SQuAD, the answers are spans of text. Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set. For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA. Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set. We report F1 and EM scores on the dev set. We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts. We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions. For this setting, there are only 899 questions for training. Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data. We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions. Main Results Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA. Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data. The BiDAF+SA model, exceeds an F1 score of $50\%$ with only $1\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\%$ of the best performance with only $10\%$ labeled data. The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available. Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits. Additionally, we show improvements in the $90\%$ data case unlike GDAN. Our approach is also applicable in the extremely low-resource setting of $1\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning. Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets. When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important. On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline. The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ. BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ. Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain. Analysis Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features. We predict the F1 scores from the cloze pretrained model ( $y^{\text{cloze}}$ ), the supervised model ( $y^{\text{sl}}$ ), and the difference of the two ( $y^{\text{cloze}}-y^{\text{sl}}$ ), when using $10\%$ of labeled data. The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors. Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\text{cloze}}-y^{\text{sl}}$ ). The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite. This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context. Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance. The pretraining also favors questions with short answers because the cloze construction process produces short answer spans. Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain. Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case. This analysis is done on the $10\%$ split of the SQuAD training set. We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21 . We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system . While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost. Also, "why" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require. Conclusion In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples. The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning. Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context. For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning. We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA. Acknowledgments Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google. Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436.
Is it possible to convert a cloze-style questions to a naturally-looking questions?
63c3550c6fb42f41a0c93133e9fca12ac00df9b3
Is Ġit Ġpossible Ġto Ġconvert Ġa Ġclo ze - style Ġquestions Ġto Ġa Ġnaturally - looking Ġquestions ?
Un answer able
<s> Is Ġit Ġpossible Ġto Ġconvert Ġa Ġclo ze - style Ġquestions Ġto Ġa Ġnaturally - looking Ġquestions ? </s> Introduction </s> Deep Ġlearning Ġsystems Ġhave Ġshown Ġa Ġlot Ġof Ġpromise Ġfor Ġextract ive ĠQuestion ĠAn sw ering Ġ( Q A ), Ġwith Ġperformance Ġcomparable Ġto Ġhumans Ġwhen Ġlarge Ġscale Ġdata Ġis Ġavailable . ĠHowever , Ġpractitioners Ġlooking Ġto Ġbuild ĠQ A Ġsystems Ġfor Ġspecific Ġapplications Ġmay Ġnot Ġhave Ġthe Ġresources Ġto Ġcollect Ġtens Ġof Ġthousands Ġof Ġquestions Ġon Ġcorpor a Ġof Ġtheir Ġchoice . ĠAt Ġthe Ġsame Ġtime , Ġstate - of - the - art Ġmachine Ġreading Ġsystems Ġdo Ġnot Ġlend Ġwell Ġto Ġlow - resource ĠQ A Ġsettings Ġwhere Ġthe Ġnumber Ġof Ġlabeled Ġquestion - answer Ġpairs Ġare Ġlimited Ġ( c . f . ĠTable Ġ2 Ġ). ĠSemi - super vised ĠQ A Ġmethods Ġlike ĠB IB REF 0 Ġaim Ġto Ġimprove Ġthis Ġperformance Ġby Ġleveraging Ġunl abel ed Ġdata Ġwhich Ġis Ġeasier Ġto Ġcollect . </s> In Ġthis Ġwork , Ġwe Ġpresent Ġa Ġsemi - super vised ĠQ A Ġsystem Ġwhich Ġrequires Ġthe Ġend Ġuser Ġto Ġspecify Ġa Ġset Ġof Ġbase Ġdocuments Ġand Ġonly Ġa Ġsmall Ġset Ġof Ġquestion - answer Ġpairs Ġover Ġa Ġsubset Ġof Ġthese Ġdocuments . ĠOur Ġproposed Ġsystem Ġconsists Ġof Ġthree Ġstages . ĠFirst , Ġwe Ġconstruct Ġclo ze - style Ġquestions Ġ( p redict ing Ġmissing Ġspans Ġof Ġtext ) Ġfrom Ġthe Ġunl abel ed Ġcorpus ; Ġnext , Ġwe Ġuse Ġthe Ġgenerated Ġcl oz es Ġto Ġpre - train Ġa Ġpowerful Ġneural Ġnetwork Ġmodel Ġfor Ġextract ive ĠQ A ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ; Ġand Ġfinally , Ġwe Ġfine - t une Ġthe Ġmodel Ġon Ġthe Ġsmall Ġset Ġof Ġprovided ĠQ A Ġpairs . </s> Our Ġclo ze Ġconstruction Ġprocess Ġbuilds Ġon Ġa Ġtypical Ġwriting Ġphenomenon Ġand Ġdocument Ġstructure : Ġan Ġintroduction Ġpreced es Ġand Ġsummarizes Ġthe Ġmain Ġbody Ġof Ġthe Ġarticle . ĠMany Ġlarge Ġcorpor a Ġfollow Ġsuch Ġa Ġstructure , Ġincluding ĠWikipedia , Ġacademic Ġpapers , Ġand Ġnews Ġarticles . ĠWe Ġhypothes ize Ġthat Ġwe Ġcan Ġbenefit Ġfrom Ġthe Ġun - annot ated Ġcorpor a Ġto Ġbetter Ġanswer Ġvarious Ġquestions ĠâĢĵ Ġat Ġleast Ġones Ġthat Ġare Ġlex ically Ġsimilar Ġto Ġthe Ġcontent Ġin Ġbase Ġdocuments Ġand Ġdirectly Ġrequire Ġfactual Ġinformation . </s> We Ġapply Ġthe Ġproposed Ġsystem Ġon Ġthree Ġdatasets Ġfrom Ġdifferent Ġdomains ĠâĢĵ ĠS Qu AD ĠB IB REF 3 Ġ, ĠTri via Q A - Web ĠB IB REF 4 Ġand Ġthe ĠBio AS Q Ġchallenge ĠB IB REF 5 Ġ. ĠWe Ġobserve Ġsignificant Ġimprovements Ġin Ġa Ġlow - resource Ġsetting Ġacross Ġall Ġthree Ġdatasets . ĠFor ĠS Qu AD Ġand ĠTri via Q A , Ġwe Ġattain Ġan ĠF 1 Ġscore Ġof Ġmore Ġthan Ġ50 % Ġby Ġmerely Ġusing Ġ1 % Ġof Ġthe Ġtraining Ġdata . ĠOur Ġsystem Ġoutper forms Ġthe Ġapproaches Ġfor Ġsemi - super vised ĠQ A Ġpresented Ġin ĠB IB REF 0 Ġ, Ġand Ġa Ġbaseline Ġwhich Ġuses Ġthe Ġsame Ġunl abel ed Ġdata Ġbut Ġwith Ġa Ġlanguage Ġmodeling Ġobjective Ġfor Ġpret raining . ĠIn Ġthe ĠBio AS Q Ġchallenge , Ġwe Ġoutper form Ġthe Ġbest Ġperforming Ġsystem Ġfrom Ġprevious Ġyear 's Ġchallenge , Ġimproving Ġover Ġa Ġbaseline Ġwhich Ġdoes Ġtransfer Ġlearning Ġfrom Ġthe ĠS Qu AD Ġdataset . ĠOur Ġanalysis Ġreveals Ġthat Ġquestions Ġwhich Ġask Ġfor Ġfactual Ġinformation Ġand Ġmatch Ġto Ġspecific Ġparts Ġof Ġthe Ġcontext Ġdocuments Ġbenefit Ġthe Ġmost Ġfrom Ġpret raining Ġon Ġautomatically Ġconstructed Ġcl oz es . </s> Related ĠWork </s> Sem i - super vised Ġlearning Ġaug ments Ġthe Ġlabeled Ġdataset Ġ$ L $ Ġwith Ġa Ġpotentially Ġlarger Ġunl abel ed Ġdataset Ġ$ U $ Ġ. ĠB IB REF 0 Ġpresented Ġa Ġmodel , ĠGD AN , Ġwhich Ġtrained Ġan Ġauxiliary Ġneural Ġnetwork Ġto Ġgenerate Ġquestions Ġfrom Ġpassages Ġby Ġreinforcement Ġlearning , Ġand Ġaugment Ġthe Ġlabeled Ġdataset Ġwith Ġthe Ġgenerated Ġquestions Ġto Ġtrain Ġthe ĠQ A Ġmodel . ĠHere Ġwe Ġuse Ġa Ġmuch Ġsimpler Ġhe uristic Ġto Ġgenerate Ġthe Ġauxiliary Ġquestions , Ġwhich Ġalso Ġturns Ġout Ġto Ġbe Ġmore Ġeffective Ġas Ġwe Ġshow Ġsuperior Ġperformance Ġcompared Ġto ĠGD AN . ĠSeveral Ġapproaches Ġhave Ġbeen Ġsuggested Ġfor Ġgenerating Ġnatural Ġquestions ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġhowever Ġnone Ġof Ġthem Ġshow Ġa Ġsignificant Ġimprovement Ġof Ġusing Ġthe Ġgenerated Ġquestions Ġin Ġa Ġsemi - super vised Ġsetting . ĠRecent Ġpapers Ġalso Ġuse Ġunl abel ed Ġdata Ġfor ĠQ A Ġby Ġtraining Ġlarge Ġlanguage Ġmodels Ġand Ġextracting Ġcontextual Ġword Ġvectors Ġfrom Ġthem Ġto Ġinput Ġto Ġthe ĠQ A Ġmodel ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠThe Ġapplic ability Ġof Ġthis Ġmethod Ġin Ġthe Ġlow - resource Ġsetting Ġis Ġunclear Ġas Ġthe Ġextra Ġinputs Ġincrease Ġthe Ġnumber Ġof Ġparameters Ġin Ġthe ĠQ A Ġmodel , Ġhowever , Ġour Ġpret raining Ġcan Ġbe Ġeasily Ġapplied Ġto Ġthese Ġmodels Ġas Ġwell . </s> Domain Ġadaptation Ġ( and ĠTransfer Ġlearning ) Ġleverage Ġexisting Ġlarge Ġscale Ġdatasets Ġfrom Ġa Ġsource Ġdomain Ġ( or Ġtask ) Ġto Ġimprove Ġperformance Ġon Ġa Ġtarget Ġdomain Ġ( or Ġtask ). ĠFor Ġdeep Ġlearning Ġand ĠQ A , Ġa Ġcommon Ġapproach Ġis Ġto Ġpret rain Ġon Ġthe Ġsource Ġdataset Ġand Ġthen Ġfine - t une Ġon Ġthe Ġtarget Ġdataset ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠB IB REF 14 Ġused ĠS Qu AD Ġas Ġa Ġsource Ġfor Ġthe Ġtarget ĠBio AS Q Ġdataset , Ġand ĠB IB REF 15 Ġused ĠBook ĠTest ĠB IB REF 16 Ġas Ġsource Ġfor Ġthe Ġtarget ĠS Qu AD Ġdataset . ĠB IB REF 17 Ġtransfer Ġlearned Ġmodel Ġlayers Ġfrom Ġthe Ġtasks Ġof Ġsequence Ġlabeling , Ġtext Ġclassification Ġand Ġrelation Ġclassification Ġto Ġshow Ġsmall Ġimprovements Ġon ĠS Qu AD . ĠAll Ġthese Ġworks Ġuse Ġmanually Ġcurated Ġsource Ġdat ats et , Ġwhich Ġin Ġthemselves Ġare Ġexpensive Ġto Ġcollect . ĠInstead , Ġwe Ġshow Ġthat Ġit Ġis Ġpossible Ġto Ġautomatically Ġconstruct Ġthe Ġsource Ġdataset Ġfrom Ġthe Ġsame Ġdomain Ġas Ġthe Ġtarget , Ġwhich Ġturns Ġout Ġto Ġbe Ġmore Ġbeneficial Ġin Ġterms Ġof Ġperformance Ġas Ġwell Ġ( c . f . ĠSection Ġ" Exper iments Ġ& ĠResults " Ġ). ĠSeveral Ġclo ze Ġdatasets Ġhave Ġbeen Ġproposed Ġin Ġthe Ġliterature Ġwhich Ġuse Ġhe ur istics Ġfor Ġconstruction ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠWe Ġfurther Ġsee Ġthe Ġusability Ġof Ġsuch Ġa Ġdataset Ġin Ġa Ġsemi - super vised Ġsetting . </s> Method ology </s> Our Ġsystem Ġcomprises Ġof Ġfollowing Ġthree Ġsteps : </s> Cl o ze Ġgeneration : ĠMost Ġof Ġthe Ġdocuments Ġtypically Ġfollow Ġa Ġtemplate , Ġthey Ġbegin Ġwith Ġan Ġintroduction Ġthat Ġprovides Ġan Ġoverview Ġand Ġa Ġbrief Ġsummary Ġfor Ġwhat Ġis Ġto Ġfollow . ĠWe Ġassume Ġsuch Ġa Ġstructure Ġwhile Ġconstructing Ġour Ġclo ze Ġstyle Ġquestions . ĠWhen Ġthere Ġis Ġno Ġclear Ġdem arc ation , Ġwe Ġtreat Ġthe Ġfirst Ġ$ K \ % $ Ġ( hyper param eter , Ġin Ġour Ġcase Ġ20 %) Ġof Ġthe Ġdocument Ġas Ġthe Ġintroduction . ĠWhile Ġnoisy , Ġthis Ġhe uristic Ġgenerates Ġa Ġlarge Ġnumber Ġof Ġcl oz es Ġgiven Ġany Ġcorpus , Ġwhich Ġwe Ġfound Ġto Ġbe Ġbeneficial Ġfor Ġsemi - super vised Ġlearning Ġdespite Ġthe Ġnoise . </s> We Ġuse Ġa Ġstandard ĠN LP Ġpipeline Ġbased Ġon ĠStanford ĠCore N LP Ġ( for ĠS Qu AD , ĠTri va Q A Ġand ĠPubMed ) Ġand Ġthe ĠB AN NER ĠNamed ĠEntity ĠRecogn izer Ġ( only Ġfor ĠPubMed Ġarticles ) Ġto Ġidentify Ġentities Ġand Ġphrases . ĠAss ume Ġthat Ġa Ġdocument Ġcomprises Ġof Ġintroduction Ġsentences Ġ$\ l brace Ġq _ 1 , Ġq _ 2 , Ġ... Ġq _ n \ r brace Ġ$ Ġ, Ġand Ġthe Ġremaining Ġpassages Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ Ġ. ĠAdditionally , Ġlet 's Ġsay Ġthat Ġeach Ġsentence Ġ$ q _ i $ Ġin Ġintroduction Ġis Ġcomposed Ġof Ġwords Ġ$\ l brace Ġw _ 1 , Ġw _ 2 , Ġ... Ġw _{ l _{ q _ i }} \ r brace Ġ$ Ġ, Ġwhere Ġ$ l _{ q _ i } $ Ġis Ġthe Ġlength Ġof Ġ$ q _ i $ Ġ. ĠWe Ġconsider Ġa Ġ$\ text { match } Ġ( q _ i , Ġp _ j ) $ Ġ, Ġif Ġthere Ġis Ġan Ġexact Ġstring Ġmatch Ġof Ġa Ġsequence Ġof Ġwords Ġ$\ l brace Ġw _ k , Ġw _{ k + 1 }, Ġ.. Ġw _{ l _{ q _ i }} \ r brace Ġ$ Ġbetween Ġthe Ġsentence Ġ$ q _ i $ Ġand Ġpassage Ġ$ p _ j $ Ġ. ĠIf Ġthis Ġsequence Ġis Ġeither Ġa Ġnoun Ġphrase , Ġverb Ġphrase , Ġadjective Ġphrase Ġor Ġa Ġnamed Ġentity Ġin Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 0 Ġ, Ġas Ġrecognized Ġby ĠCore N LP Ġor ĠB AN NER , Ġwe Ġselect Ġit Ġas Ġan Ġanswer Ġspan Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 1 Ġ. ĠAdditionally , Ġwe Ġuse Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 2 Ġas Ġthe Ġpassage Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 3 Ġand Ġform Ġa Ġclo ze Ġquestion Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 4 Ġfrom Ġthe Ġanswer Ġbearing Ġsentence Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 5 Ġby Ġreplacing Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 6 Ġwith Ġa Ġplaceholder . ĠAs Ġa Ġresult , Ġwe Ġobtain Ġpassage - question - answer Ġ( Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 7 Ġ) Ġtri ples Ġ( Table Ġ1 Ġshows Ġan Ġexample ). ĠAs Ġa Ġpost - processing Ġstep , Ġwe Ġpr une Ġout Ġ$\ l brace Ġp _ 1 , Ġp _ 2 , Ġ.. Ġp _ m \ r brace Ġ$ 8 Ġtri ples Ġwhere Ġthe Ġword Ġoverlap Ġbetween Ġthe Ġquestion Ġ( Q ) Ġand Ġpassage Ġ( P ) Ġis Ġless Ġthan Ġ2 Ġwords Ġ( after Ġexcluding Ġthe Ġstop Ġwords ). </s> The Ġprocess Ġrelies Ġon Ġthe Ġfact Ġthat Ġanswer Ġcandidates Ġfrom Ġthe Ġintroduction Ġare Ġlikely Ġto Ġbe Ġdiscussed Ġin Ġdetail Ġin Ġthe Ġremainder Ġof Ġthe Ġarticle . ĠIn Ġeffect , Ġthe Ġclo ze Ġquestion Ġfrom Ġthe Ġintroduction Ġand Ġthe Ġmatching Ġparagraph Ġin Ġthe Ġbody Ġforms Ġa Ġquestion Ġand Ġcontext Ġpassage Ġpair . ĠWe Ġcreate Ġtwo Ġclo ze Ġdatasets , Ġone Ġeach Ġfrom ĠWikipedia Ġcorpus Ġ( for ĠS Qu AD Ġand ĠTri via Q A ) Ġand ĠP UB Med Ġacademic Ġpapers Ġ( for Ġthe ĠBio AS Q Ġchallenge ), Ġconsisting Ġof Ġ2 . 2 M Ġand Ġ1 M Ġcl oz es Ġrespectively . ĠFrom Ġanalyzing Ġthe Ġclo ze Ġdata Ġmanually , Ġwe Ġwere Ġable Ġto Ġanswer Ġ76 % Ġtimes Ġfor Ġthe ĠWikipedia Ġset Ġand Ġ80 % Ġtimes Ġfor Ġthe ĠP UB Med Ġset Ġusing Ġthe Ġinformation Ġin Ġthe Ġpassage . ĠIn Ġmost Ġcases Ġthe Ġclo ze Ġparaph r ased Ġthe Ġinformation Ġin Ġthe Ġpassage , Ġwhich Ġwe Ġhypothesized Ġto Ġbe Ġa Ġuseful Ġsignal Ġfor Ġthe Ġdownstream ĠQ A Ġtask . </s> We Ġalso Ġinvestigate Ġthe Ġutility Ġof Ġforming Ġsubs ets Ġof Ġthe Ġlarge Ġclo ze Ġcorpus , Ġwhere Ġwe Ġselect Ġthe Ġtop Ġpassage - question - answer Ġtri ples , Ġbased Ġon Ġthe Ġdifferent Ġcriteria , Ġlike Ġi ) Ġj acc ard Ġsimilarity Ġof Ġanswer Ġbearing Ġsentence Ġin Ġintroduction Ġand Ġthe Ġpassage Ġii ) Ġthe Ġtf - id f Ġscores Ġof Ġanswer Ġcandidates Ġand Ġiii ) Ġthe Ġlength Ġof Ġanswer Ġcandidates . ĠHowever , Ġwe Ġempir ically Ġfind Ġthat Ġwe Ġwere Ġbetter Ġoff Ġusing Ġthe Ġentire Ġset Ġrather Ġthan Ġthese Ġsubs ets . </s> Pre - training : ĠWe Ġmake Ġuse Ġof Ġthe Ġgenerated Ġclo ze Ġdataset Ġto Ġpre - train Ġan Ġexpressive Ġneural Ġnetwork Ġdesigned Ġfor Ġthe Ġtask Ġof Ġreading Ġcomprehension . ĠWe Ġwork Ġwith Ġtwo Ġpublicly Ġavailable Ġneural Ġnetwork Ġmodels ĠâĢĵ Ġthe ĠGA ĠReader ĠB IB REF 2 Ġ( to Ġenable Ġcomparison Ġwith Ġprior Ġwork ) Ġand ĠBi DA F Ġ+ ĠSelf - Att ention Ġ( SA ) Ġmodel Ġfrom ĠB IB REF 1 Ġ( which Ġis Ġamong Ġthe Ġbest Ġperforming Ġmodels Ġon ĠS Qu AD Ġand ĠTri via Q A ). ĠAfter Ġpret raining , Ġthe Ġperformance Ġof ĠBi DA F + SA Ġon Ġa Ġdev Ġset Ġof Ġthe Ġ( Wikipedia ) Ġclo ze Ġquestions Ġis Ġ0 . 58 ĠF 1 Ġscore Ġand Ġ0 . 55 ĠEx act ĠMatch Ġ( EM ) Ġscore . ĠThis Ġimplies Ġthat Ġthe Ġclo ze Ġcorpus Ġis Ġneither Ġtoo Ġeasy , Ġnor Ġtoo Ġdifficult Ġto Ġanswer . </s> Fine ĠTun ing : ĠWe Ġfine Ġtune Ġthe Ġpre - trained Ġmodel , Ġfrom Ġthe Ġprevious Ġstep , Ġover Ġa Ġsmall Ġset Ġof Ġlabelled Ġquestion - answer Ġpairs . ĠAs Ġwe Ġshall Ġlater Ġsee , Ġthis Ġstep Ġis Ġcrucial , Ġand Ġit Ġonly Ġrequires Ġa Ġhandful Ġof Ġlabelled Ġquestions Ġto Ġachieve Ġa Ġsignificant Ġproportion Ġof Ġthe Ġperformance Ġtypically Ġattained Ġby Ġtraining Ġon Ġtens Ġof Ġthousands Ġof Ġquestions . </s> Dat as ets </s> We Ġapply Ġour Ġsystem Ġto Ġthree Ġdatasets Ġfrom Ġdifferent Ġdomains . ĠS Qu AD ĠB IB REF 3 Ġconsists Ġof Ġquestions Ġwhose Ġanswers Ġare Ġfree Ġform Ġspans Ġof Ġtext Ġfrom Ġpassages Ġin ĠWikipedia Ġarticles . ĠWe Ġfollow Ġthe Ġsame Ġsetting Ġas Ġin ĠB IB REF 0 Ġ, Ġand Ġsplit Ġ$ 10 \ % $ Ġof Ġtraining Ġquestions Ġas Ġthe Ġtest Ġset , Ġand Ġreport Ġperformance Ġwhen Ġtraining Ġon Ġsubs ets Ġof Ġthe Ġremaining Ġdata Ġranging Ġfrom Ġ$ 1 \ % $ Ġto Ġ$ 90 \ % $ Ġof Ġthe Ġfull Ġset . ĠWe Ġalso Ġreport Ġthe Ġperformance Ġon Ġthe Ġdev Ġset Ġwhen Ġtrained Ġon Ġthe Ġfull Ġtraining Ġset Ġ( Ġ$ 1 ^ \ ast Ġ$ Ġin ĠTable Ġ2 Ġ). ĠWe Ġuse Ġthe Ġsame Ġhyper param eter Ġsettings Ġas Ġin Ġprior Ġwork . ĠWe Ġcompare Ġand Ġstudy Ġfour Ġdifferent Ġsettings : Ġ1 ) Ġthe ĠSuper vised ĠLearning Ġ( SL ) Ġsetting , Ġwhich Ġis Ġonly Ġtrained Ġon Ġthe Ġsupervised Ġdata , Ġ2 ) Ġthe Ġbest Ġperforming ĠGD AN Ġmodel Ġfrom ĠB IB REF 0 Ġ, Ġ3 ) Ġpret raining Ġon Ġa ĠLanguage ĠModel ing Ġ( LM ) Ġobjective Ġand Ġfine - tun ing Ġon Ġthe Ġsupervised Ġdata , Ġand Ġ4 ) Ġpret raining Ġon Ġthe ĠClo ze Ġdataset Ġand Ġfine - tun ing Ġon Ġthe Ġsupervised Ġdata . ĠThe ĠLM Ġand ĠClo ze Ġmethods Ġuse Ġexactly Ġthe Ġsame Ġdata Ġfor Ġpret raining , Ġbut Ġdiffer Ġin Ġthe Ġloss Ġfunctions Ġused . ĠWe Ġreport ĠF 1 Ġand ĠEM Ġscores Ġon Ġour Ġtest Ġset Ġusing Ġthe Ġofficial Ġevaluation Ġscripts Ġprovided Ġby Ġthe Ġauthors Ġof Ġthe Ġdataset . </s> Trivia Q A ĠB IB REF 4 Ġcomprises Ġof Ġover Ġ95 K Ġweb Ġquestion - answer - evidence Ġtri ples . ĠLike ĠS Qu AD , Ġthe Ġanswers Ġare Ġspans Ġof Ġtext . ĠSimilar Ġto Ġthe Ġsetting Ġin ĠS Qu AD , Ġwe Ġcreate Ġmultiple Ġsmaller Ġsubs ets Ġof Ġthe Ġentire Ġset . ĠFor Ġour Ġsemi - super vised ĠQ A Ġsystem , Ġwe Ġuse Ġthe ĠBi DA F + SA Ġmodel ĠB IB REF 1 ĠâĢĵ Ġthe Ġhighest Ġperforming Ġpublicly Ġavailable Ġsystem Ġfor ĠTri va Q A . ĠHere Ġagain , Ġwe Ġcompare Ġthe Ġsupervised Ġlearning Ġ( SL ) Ġsettings Ġagainst Ġthe Ġpret raining Ġon ĠClo ze Ġset Ġand Ġfine Ġtuning Ġon Ġthe Ġsupervised Ġset . ĠWe Ġreport ĠF 1 Ġand ĠEM Ġscores Ġon Ġthe Ġdev Ġset . </s> We Ġalso Ġtest Ġon Ġthe ĠBio AS Q Ġ5 b Ġdataset , Ġwhich Ġconsists Ġof Ġquestion - answer Ġpairs Ġfrom ĠPubMed Ġabstract s . ĠWe Ġuse Ġthe Ġpublicly Ġavailable Ġsystem Ġfrom ĠB IB REF 14 Ġ, Ġand Ġfollow Ġthe Ġexact Ġsame Ġsetup Ġas Ġtheirs , Ġfocusing Ġonly Ġon Ġfact oid Ġand Ġlist Ġquestions . ĠFor Ġthis Ġsetting , Ġthere Ġare Ġonly Ġ8 99 Ġquestions Ġfor Ġtraining . ĠSince Ġthis Ġis Ġalready Ġa Ġlow - resource Ġproblem Ġwe Ġonly Ġreport Ġresults Ġusing Ġ5 - fold Ġcross - valid ation Ġon Ġall Ġthe Ġavailable Ġdata . ĠWe Ġreport ĠMean ĠRec ip rocal ĠRank Ġ( MR R ) Ġon Ġthe Ġfact oid Ġquestions , Ġand ĠF 1 Ġscore Ġfor Ġthe Ġlist Ġquestions . </s> Main ĠResults </s> Table Ġ2 Ġshows Ġa Ġcomparison Ġof Ġthe Ġdiscussed Ġsettings Ġon Ġboth ĠS Qu AD Ġand ĠTri via Q A . ĠWithout Ġany Ġfine - tun ing Ġ( column Ġ0 ) Ġthe Ġperformance Ġis Ġlow , Ġprobably Ġbecause Ġthe Ġmodel Ġnever Ġsaw Ġa Ġreal Ġquestion , Ġbut Ġwe Ġsee Ġsignificant Ġgains Ġwith ĠClo ze Ġpret raining Ġeven Ġwith Ġvery Ġlittle Ġlabeled Ġdata . ĠThe ĠBi DA F + SA Ġmodel , Ġexceeds Ġan ĠF 1 Ġscore Ġof Ġ$ 50 \ % $ Ġwith Ġonly Ġ$ 1 \ % $ Ġof Ġthe Ġtraining Ġdata Ġ( 454 Ġquestions Ġfor ĠS Qu AD , Ġand Ġ7 46 Ġquestions Ġfor ĠTri via Q A ), Ġand Ġapproaches Ġ$ 90 \ % $ Ġof Ġthe Ġbest Ġperformance Ġwith Ġonly Ġ$ 10 \ % $ Ġlabeled Ġdata . ĠThe Ġgains Ġover Ġthe ĠSL Ġsetting , Ġhowever , Ġdiminish Ġas Ġthe Ġsize Ġof Ġthe Ġlabeled Ġset Ġincreases Ġand Ġare Ġsmall Ġwhen Ġthe Ġfull Ġdataset Ġis Ġavailable . </s> Cl o ze Ġpret raining Ġoutper forms Ġthe ĠGD AN Ġbaseline Ġfrom ĠB IB REF 0 Ġusing Ġthe Ġsame ĠS Qu AD Ġdataset Ġsplits . ĠAdditionally , Ġwe Ġshow Ġimprovements Ġin Ġthe Ġ$ 90 \ % $ Ġdata Ġcase Ġunlike ĠGD AN . ĠOur Ġapproach Ġis Ġalso Ġapplicable Ġin Ġthe Ġextremely Ġlow - resource Ġsetting Ġof Ġ$ 1 \ % $ Ġdata , Ġwhich Ġwe Ġsuspect ĠGD AN Ġmight Ġhave Ġtrouble Ġwith Ġsince Ġit Ġuses Ġthe Ġlabeled Ġdata Ġto Ġdo Ġreinforcement Ġlearning . ĠFurthermore , Ġwe Ġare Ġable Ġto Ġuse Ġthe Ġsame Ġclo ze Ġdataset Ġto Ġimprove Ġperformance Ġon Ġboth ĠS Qu AD Ġand ĠTri via Q A Ġdatasets . ĠWhen Ġwe Ġuse Ġthe Ġsame Ġunl abel ed Ġdata Ġto Ġpre - train Ġwith Ġa Ġlanguage Ġmodeling Ġobjective , Ġthe Ġperformance Ġis Ġworse , Ġshowing Ġthe Ġbias Ġwe Ġintroduce Ġby Ġconstructing Ġcl oz es Ġis Ġimportant . </s> On Ġthe ĠBio AS Q Ġdataset Ġ( Table Ġ3 Ġ) Ġwe Ġagain Ġsee Ġa Ġsignificant Ġimprovement Ġwhen Ġpret raining Ġwith Ġthe Ġclo ze Ġquestions Ġover Ġthe Ġsupervised Ġbaseline . ĠThe Ġimprovement Ġis Ġsmaller Ġthan Ġwhat Ġwe Ġobserve Ġwith ĠS Qu AD Ġand ĠTri via Q A Ġdatasets ĠâĢĵ Ġwe Ġbelieve Ġthis Ġis Ġbecause Ġquestions Ġare Ġgenerally Ġmore Ġdifficult Ġin ĠBio AS Q . ĠB IB REF 14 Ġshowed Ġthat Ġpret raining Ġon ĠS Qu AD Ġdataset Ġimproves Ġthe Ġdownstream Ġperformance Ġon ĠBio AS Q . ĠHere , Ġwe Ġshow Ġa Ġmuch Ġlarger Ġimprovement Ġby Ġpret raining Ġon Ġclo ze Ġquestions Ġconstructed Ġin Ġan Ġun super vised Ġmanner Ġfrom Ġthe Ġsame Ġdomain . </s> Analysis </s> Reg ression ĠAnalysis : ĠTo Ġunderstand Ġwhich Ġtypes Ġof Ġquestions Ġbenefit Ġfrom Ġpre - training , Ġwe Ġpre - specified Ġcertain Ġfeatures Ġ( see ĠFigure Ġ1 Ġright ) Ġfor Ġeach Ġof Ġthe Ġdev Ġset Ġquestions Ġin ĠS Qu AD , Ġand Ġthen Ġperformed Ġlinear Ġregression Ġto Ġpredict Ġthe ĠF 1 Ġscore Ġfor Ġthat Ġquestion Ġfrom Ġthese Ġfeatures . ĠWe Ġpredict Ġthe ĠF 1 Ġscores Ġfrom Ġthe Ġclo ze Ġpret rained Ġmodel Ġ( Ġ$ y ^ {\ text { cl o ze }} $ Ġ), Ġthe Ġsupervised Ġmodel Ġ( Ġ$ y ^ {\ text { sl }} $ Ġ), Ġand Ġthe Ġdifference Ġof Ġthe Ġtwo Ġ( Ġ$ y ^ {\ text { cl o ze }} - y ^ {\ text { sl }} $ Ġ), Ġwhen Ġusing Ġ$ 10 \ % $ Ġof Ġlabeled Ġdata . ĠThe Ġcoefficients Ġof Ġthe Ġfitted Ġmodel Ġare Ġshown Ġin ĠFigure Ġ1 Ġ( left ) Ġalong Ġwith Ġtheir Ġstd Ġerrors . ĠPositive Ġcoefficients Ġindicate Ġthat Ġa Ġhigh Ġvalue Ġof Ġthat Ġfeature Ġis Ġpredictive Ġof Ġa Ġhigh ĠF 1 Ġscore , Ġand Ġa Ġnegative Ġcoefficient Ġindicates Ġthat Ġa Ġsmall Ġvalue Ġof Ġthat Ġfeature Ġis Ġpredictive Ġof Ġa Ġhigh ĠF 1 Ġscore Ġ( or Ġa Ġhigh Ġdifference Ġof ĠF 1 Ġscores Ġfrom Ġthe Ġtwo Ġmodels Ġin Ġthe Ġcase Ġof Ġ$ y ^ {\ text { cl o ze }} - y ^ {\ text { sl }} $ Ġ). </s> The Ġtwo Ġstrongest Ġeffects Ġwe Ġobserve Ġare Ġthat Ġa Ġhigh Ġlex ical Ġoverlap Ġbetween Ġthe Ġquestion Ġand Ġthe Ġsentence Ġcontaining Ġthe Ġanswer Ġis Ġindicative Ġof Ġhigh Ġboost Ġwith Ġpret raining , Ġand Ġthat Ġa Ġhigh Ġlex ical Ġoverlap Ġbetween Ġthe Ġquestion Ġand Ġthe Ġwhole Ġpassage Ġis Ġindicative Ġof Ġthe Ġopposite . ĠThis Ġis Ġhardly Ġsurprising , Ġsince Ġour Ġclo ze Ġconstruction Ġprocess Ġis Ġbiased Ġtowards Ġquestions Ġwhich Ġhave Ġa Ġsimilar Ġphr asing Ġto Ġthe Ġanswer Ġsentences Ġin Ġcontext . ĠHence , Ġtest Ġquestions Ġwith Ġa Ġsimilar Ġproperty Ġare Ġanswered Ġcorrectly Ġafter Ġpret raining , Ġwhereas Ġthose Ġwith Ġa Ġhigh Ġoverlap Ġwith Ġthe Ġwhole Ġpassage Ġtend Ġto Ġhave Ġlower Ġperformance . ĠThe Ġpret raining Ġalso Ġfavors Ġquestions Ġwith Ġshort Ġanswers Ġbecause Ġthe Ġclo ze Ġconstruction Ġprocess Ġproduces Ġshort Ġanswer Ġspans . ĠAlso Ġpassages Ġand Ġquestions Ġwhich Ġconsist Ġof Ġtokens Ġinf requent Ġin Ġthe ĠS Qu AD Ġtraining Ġcorpus Ġreceive Ġa Ġlarge Ġboost Ġafter Ġpret raining , Ġsince Ġthe Ġunl abel ed Ġdata Ġcovers Ġa Ġlarger Ġdomain . </s> Performance Ġon Ġquestion Ġtypes : ĠFigure Ġ2 Ġshows Ġthe Ġaverage Ġgain Ġin ĠF 1 Ġscore Ġfor Ġdifferent Ġtypes Ġof Ġquestions , Ġwhen Ġwe Ġpret rain Ġon Ġthe Ġcl oz es Ġcompared Ġto Ġthe Ġsupervised Ġcase . ĠThis Ġanalysis Ġis Ġdone Ġon Ġthe Ġ$ 10 \ % $ Ġsplit Ġof Ġthe ĠS Qu AD Ġtraining Ġset . ĠWe Ġconsider Ġtwo Ġclass ifications Ġof Ġeach Ġquestion ĠâĢĵ Ġone Ġdetermined Ġon Ġthe Ġfirst Ġword Ġ( usually Ġa Ġwh - word ) Ġof Ġthe Ġquestion Ġ( Figure Ġ2 Ġ( bottom )) Ġand Ġone Ġbased Ġon Ġthe Ġoutput Ġof Ġa Ġseparate Ġquestion Ġtype Ġclass ifier Ġadapted Ġfrom ĠB IB REF 21 Ġ. ĠWe Ġuse Ġthe Ġcoarse Ġgrain Ġlabels Ġnamely ĠAb bre viation Ġ( AB BR ), ĠEntity Ġ( ENT Y ), ĠDescription Ġ( DES C ), ĠHuman Ġ( H UM ), ĠLocation Ġ( LOC ), ĠN umeric Ġ( NUM ) Ġtrained Ġon Ġa ĠLog istic ĠReg ression Ġclassification Ġsystem Ġ. ĠWhile Ġthere Ġis Ġan Ġimprovement Ġacross Ġthe Ġboard , Ġwe Ġfind Ġthat Ġabbre viation Ġquestions Ġin Ġparticular Ġreceive Ġa Ġlarge Ġboost . ĠAlso , Ġ" why " Ġquestions Ġshow Ġthe Ġleast Ġimprovement , Ġwhich Ġis Ġin Ġline Ġwith Ġour Ġexpectation , Ġsince Ġthese Ġusually Ġrequire Ġreasoning Ġor Ġworld Ġknowledge Ġwhich Ġclo ze Ġquestions Ġrarely Ġrequire . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġshow Ġthat Ġpre - training ĠQ A Ġmodels Ġwith Ġautomatically Ġconstructed Ġclo ze Ġquestions Ġimproves Ġthe Ġperformance Ġof Ġthe Ġmodels Ġsignificantly , Ġespecially Ġwhen Ġthere Ġare Ġfew Ġlabeled Ġexamples . ĠThe Ġperformance Ġof Ġthe Ġmodel Ġtrained Ġonly Ġon Ġthe Ġclo ze Ġquestions Ġis Ġpoor , Ġvalid ating Ġthe Ġneed Ġfor Ġfine - tun ing . ĠThrough Ġregression Ġanalysis , Ġwe Ġfind Ġthat Ġpret raining Ġhelps Ġwith Ġquestions Ġwhich Ġask Ġfor Ġfactual Ġinformation Ġlocated Ġin Ġa Ġspecific Ġpart Ġof Ġthe Ġcontext . ĠFor Ġfuture Ġwork , Ġwe Ġplan Ġto Ġexplore Ġthe Ġactive Ġlearning Ġsetup Ġfor Ġthis Ġtask ĠâĢĵ Ġspecifically , Ġwhich Ġpassages Ġand Ġ/ Ġor Ġtypes Ġof Ġquestions Ġcan Ġwe Ġselect Ġto Ġannot ate , Ġsuch Ġthat Ġthere Ġis Ġa Ġmaximum Ġperformance Ġgain Ġfrom Ġfine - tun ing . ĠWe Ġalso Ġwant Ġto Ġexplore Ġhow Ġto Ġadapt Ġclo ze Ġstyle Ġpre - training Ġto ĠN LP Ġtasks Ġother Ġthan ĠQ A . </s> Acknowled gments </s> B hu wan ĠD hing ra Ġis Ġsupported Ġby ĠNS F Ġunder Ġgrants ĠCC F - 14 140 30 Ġand ĠI IS - 12 50 956 Ġand Ġby Ġgrants Ġfrom ĠGoogle . ĠDanish ĠPr uth i Ġand ĠD he er aj ĠRaj ag opal Ġare Ġsupported Ġby Ġthe ĠDAR PA ĠBig ĠMechan ism Ġprogram Ġunder ĠAR O Ġcontract ĠW 911 NF - 14 - 1 - 04 36 .