SAN Baseline, BNA, DocQA, R.M-Reader, R.M-Reader+Verifier and DocQA+ELMo
ABSTRACTIVE
BNA, DocQA, R.M-Reader, R.M-Reader + Verifier, DocQA + ELMo, R.M-Reader+Verifier+ELMo
ABSTRACTIVE
Table TABREF21 reports comparison results in literature published . Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo). Comparing with the much complicated model R.M.-Reader + Verifier, which includes several components, our model still outperforms by 0.7 in terms of F1 score. Furthermore, we observe that ELMo gives a great boosting on the performance, e.g., 2.8 points in terms of F1 for DocQA. This encourages us to incorporate ELMo into our model in future. The results in terms of EM and F1 is summarized in Table TABREF20 . We observe that Joint SAN outperforms the SAN baseline with a large margin, e.g., 67.89 vs 69.27 (+1.38) and 70.68 vs 72.20 (+1.52) in terms of EM and F1 scores respectively, so it demonstrates the effectiveness of the joint optimization. By incorporating the output information of classifier into Joint SAN, it obtains a slight improvement, e.g., 72.2 vs 72.66 (+0.46) in terms of F1 score. By analyzing the results, we found that in most cases when our model extract an NULL string answer, the classifier also predicts it as an unanswerable question with a high probability. FLOAT SELECTED: Table 2: Comparison with published results in literature. 1: results are extracted from (Rajpurkar et al., 2018); 2: results are extracted from (Hu et al., 2018). ∗: it is unclear which model is used. #: we only evaluate the Joint SAN in the submission.
FLOAT SELECTED: Table 2: Comparison with published results in literature. 1: results are extracted from (Rajpurkar et al., 2018); 2: results are extracted from (Hu et al., 2018). ∗: it is unclear which model is used. #: we only evaluate the Joint SAN in the submission. Table TABREF21 reports comparison results in literature published . Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo). Comparing with the much complicated model R.M.-Reader + Verifier, which includes several components, our model still outperforms by 0.7 in terms of F1 score. Furthermore, we observe that ELMo gives a great boosting on the performance, e.g., 2.8 points in terms of F1 for DocQA. This encourages us to incorporate ELMo into our model in future.
1809.09194
Background </s> Te aching Ġmachine Ġto Ġread Ġand Ġcomprehend Ġa Ġgiven Ġpassage / paragraph Ġand Ġanswer Ġits Ġcorresponding Ġquestions Ġis Ġa Ġchallenging Ġtask . ĠIt Ġis Ġalso Ġone Ġof Ġthe Ġlong - term Ġgoals Ġof Ġnatural Ġlanguage Ġunderstanding , Ġand Ġhas Ġimportant Ġapplications Ġin Ġe . g ., Ġbuilding Ġintelligent Ġagents Ġfor Ġconversation Ġand Ġcustomer Ġservice Ġsupport . ĠIn Ġa Ġreal Ġworld Ġsetting , Ġit Ġis Ġnecessary Ġto Ġjudge Ġwhether Ġthe Ġgiven Ġquestions Ġare Ġanswer able Ġgiven Ġthe Ġavailable Ġknowledge , Ġand Ġthen Ġgenerate Ġcorrect Ġanswers Ġfor Ġthe Ġones Ġwhich Ġare Ġable Ġto Ġinfer Ġan Ġanswer Ġin Ġthe Ġpassage Ġor Ġan Ġempty Ġanswer Ġ( as Ġan Ġun answer able Ġquestion ) Ġotherwise . </s> In Ġcomparison Ġwith Ġmany Ġexisting ĠM RC Ġsystems ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġwhich Ġextract Ġanswers Ġby Ġfinding Ġa Ġsub - string Ġin Ġthe Ġpassages / paragraph s , Ġwe Ġpropose Ġa Ġmodel Ġthat Ġnot Ġonly Ġextracts Ġanswers Ġbut Ġalso Ġpredicts Ġwhether Ġsuch Ġan Ġanswer Ġshould Ġexist . ĠUsing Ġa Ġmulti - task Ġlearning Ġapproach Ġ( c . f . ĠB IB REF 5 Ġ), Ġwe Ġextend Ġthe ĠSt och astic ĠAnswer ĠNetwork Ġ( SAN ) ĠB IB REF 1 Ġfor ĠM RC Ġanswer Ġspan Ġdetector Ġto Ġinclude Ġa Ġclass ifier Ġthat Ġwhether Ġthe Ġquestion Ġis Ġun answer able . ĠThe Ġun answer able Ġclass ifier Ġis Ġa Ġpair - wise Ġclassification Ġmodel ĠB IB REF 6 Ġwhich Ġpredicts Ġa Ġlabel Ġindicating Ġwhether Ġthe Ġgiven Ġpair Ġof Ġa Ġpassage Ġand Ġa Ġquestion Ġis Ġun answer able . ĠThe Ġtwo Ġmodels Ġshare Ġthe Ġsame Ġlower Ġlayer Ġto Ġsave Ġthe Ġnumber Ġof Ġparameters , Ġand Ġseparate Ġthe Ġtop Ġlayers Ġfor Ġdifferent Ġtasks Ġ( the Ġspan Ġdetector Ġand Ġbinary Ġclass ifier ). </s> Our Ġmodel Ġis Ġpretty Ġsimple Ġand Ġintuitive , Ġyet Ġefficient . ĠWithout Ġrelying Ġon Ġthe Ġlarge Ġpre - trained Ġlanguage Ġmodels Ġ( EL Mo ) ĠB IB REF 7 Ġ, Ġthe Ġproposed Ġmodel Ġachieves Ġcompetitive Ġresults Ġto Ġthe Ġstate - of - the - art Ġon ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD ) Ġ2 . 0 . </s> The Ġcontribution Ġof Ġthis Ġwork Ġis Ġsummarized Ġas Ġfollows . ĠFirst , Ġwe Ġpropose Ġa Ġsimple Ġyet Ġefficient Ġmodel Ġfor ĠM RC Ġthat Ġhandles Ġun answer able Ġquestions Ġand Ġis Ġoptimized Ġjointly . ĠSecond , Ġour Ġmodel Ġachieves Ġcompetitive Ġresults Ġon ĠS Qu AD Ġv 2 . 0 . </s> Model </s> The ĠMachine ĠReading ĠComp rehens ion Ġis Ġa Ġtask Ġwhich Ġtakes Ġa Ġquestion ĠIN LINE FORM 0 Ġand Ġa Ġpassage / paragraph ĠIN LINE FORM 1 Ġas Ġinputs , Ġand Ġaims Ġto Ġfind Ġan Ġanswer Ġspan ĠIN LINE FORM 2 Ġin ĠIN LINE FORM 3 Ġ. ĠWe Ġassume Ġthat Ġif Ġthe Ġquestion Ġis Ġanswer able , Ġthe Ġanswer ĠIN LINE FORM 4 Ġexists Ġin ĠIN LINE FORM 5 Ġas Ġa Ġcontiguous Ġtext Ġstring ; Ġotherwise , ĠIN LINE FORM 6 Ġis Ġan Ġempty Ġstring Ġindicating Ġan Ġun answer able Ġquestion . ĠNote Ġthat Ġto Ġhandle Ġthe Ġun answer able Ġquestions , Ġwe Ġmanually Ġappend Ġa Ġdump y Ġtext Ġstring ĠNULL Ġat Ġthe Ġend Ġof Ġeach Ġcorresponding Ġpassage / paragraph . ĠForm ally , Ġthe Ġanswer Ġis Ġformulated Ġas ĠIN LINE FORM 7 Ġ. ĠIn Ġcase Ġof Ġun answer able Ġquestions , ĠIN LINE FORM 8 Ġpoints Ġto Ġthe Ġlast Ġtoken Ġof Ġthe Ġpassage . </s> Our Ġmodel Ġis Ġa Ġvariation Ġof ĠSAN ĠB IB REF 1 Ġ, Ġas Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ. ĠThe Ġmain Ġdifference Ġis Ġthe Ġadditional Ġbinary Ġclass ifier Ġadded Ġin Ġthe Ġmodel Ġjustifying Ġwhether Ġthe Ġquestion Ġis Ġun answer able . ĠRough ly , Ġthe Ġmodel Ġincludes Ġtwo Ġdifferent Ġlayers : Ġthe Ġshared Ġlayer Ġand Ġtask Ġspecific Ġlayer . ĠThe Ġshared Ġlayer Ġis Ġalmost Ġidentical Ġto Ġthe Ġlower Ġlayers Ġof ĠSAN , Ġwhich Ġhas Ġa Ġlex icon Ġencoding Ġlayer , Ġa Ġcontextual Ġlayer Ġand Ġa Ġmemory Ġgeneration Ġlayer . ĠOn Ġtop Ġof Ġit , Ġthere Ġare Ġdifferent Ġanswer Ġmodules Ġfor Ġdifferent Ġtasks . ĠWe Ġemploy Ġthe ĠSAN Ġanswer Ġmodule Ġfor Ġthe Ġspan Ġdetector Ġand Ġa Ġone - layer Ġfeed Ġforward Ġneural Ġnetwork Ġfor Ġthe Ġbinary Ġclassification Ġtask . ĠIt Ġcan Ġalso Ġbe Ġviewed Ġas Ġa Ġmulti - task Ġlearning ĠB IB REF 8 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 9 Ġ. ĠWe Ġwill Ġbriefly Ġdescribe Ġthe Ġmodel Ġfrom Ġground Ġup Ġas Ġfollows . ĠDet ailed Ġdescriptions Ġcan Ġbe Ġfound Ġin ĠB IB REF 1 Ġ. </s> Lex icon ĠEnc oding ĠLayer . ĠWe Ġmap Ġthe Ġsymbolic / surface Ġfeature Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġinto Ġneural Ġspace Ġvia Ġword Ġembed d ings Ġ, Ġ16 - dim Ġpart - of - speech Ġ( POS ) Ġtagging Ġembed d ings , Ġ8 - dim Ġnamed - entity Ġembed d ings Ġand Ġ4 - dim Ġhard - rule Ġfeatures . ĠNote Ġthat Ġwe Ġuse Ġsmall Ġembed ding Ġsize Ġof ĠPOS Ġand ĠN ER Ġto Ġreduce Ġmodel Ġsize Ġand Ġthey Ġmainly Ġserve Ġthe Ġrole Ġof Ġcoarse - gr ained Ġword Ġclusters . ĠAdditionally , Ġwe Ġuse Ġquestion Ġenhanced Ġpassages Ġword Ġembed d ings Ġwhich Ġcan Ġview wed Ġas Ġsoft Ġmatching Ġbetween Ġquestions Ġand Ġpassages . ĠAt Ġlast , Ġwe Ġuse Ġtwo Ġseparate Ġtwo - layer Ġposition - wise ĠFeed - Forward ĠNetworks Ġ( FF N ) ĠB IB REF 11 Ġ, ĠB IB REF 1 Ġto Ġmap Ġboth Ġquestion Ġand Ġpassage Ġenc od ings Ġinto Ġthe Ġsame Ġdimension . ĠAs Ġresults , Ġwe Ġobtain Ġthe Ġfinal Ġlex icon Ġembed d ings Ġfor Ġthe Ġtokens Ġfor ĠIN LINE FORM 2 Ġas Ġa Ġmatrix ĠIN LINE FORM 3 Ġ, Ġand Ġtokens Ġin ĠIN LINE FORM 4 Ġas ĠIN LINE FORM 5 Ġ. </s> Context ual ĠEnc oding ĠLayer . ĠA Ġshared Ġtwo - l ayers ĠBi L ST M Ġis Ġused Ġon Ġthe Ġtop Ġto Ġencode Ġthe Ġcontextual Ġinformation Ġof Ġboth Ġpassages Ġand Ġquestions . ĠTo Ġavoid Ġover fitting , Ġwe Ġconc aten ate Ġa Ġpre - trained Ġ600 - dimensional ĠCo Ve Ġvectors ĠB IB REF 12 Ġtrained Ġon ĠGerman - English Ġmachine Ġtranslation Ġdataset , Ġwith Ġthe Ġaforementioned Ġlex icon Ġembed d ings Ġas Ġthe Ġfinal Ġinput Ġof Ġthe Ġcontextual Ġencoding Ġlayer , Ġand Ġalso Ġwith Ġthe Ġoutput Ġof Ġthe Ġfirst Ġcontextual Ġencoding Ġlayer Ġas Ġthe Ġinput Ġof Ġits Ġsecond Ġencoding Ġlayer . ĠThus , Ġwe Ġobtain Ġthe Ġfinal Ġrepresentation Ġof Ġthe Ġcontextual Ġencoding Ġlayer Ġby Ġa Ġconc aten ation Ġof Ġthe Ġoutputs Ġof Ġtwo ĠBi L ST M : ĠIN LINE FORM 0 Ġfor Ġquestions Ġand ĠIN LINE FORM 1 Ġfor Ġpassages . </s> Memory ĠGeneration ĠLayer . ĠIn Ġthis Ġlayer , Ġwe Ġgenerate Ġa Ġworking Ġmemory Ġby Ġf using Ġinformation Ġfrom Ġboth Ġpassages ĠIN LINE FORM 0 Ġand Ġquestions ĠIN LINE FORM 1 Ġ. ĠThe Ġattention Ġfunction ĠB IB REF 11 Ġis Ġused Ġto Ġcompute Ġthe Ġsimilarity Ġscore Ġbetween Ġpassages Ġand Ġquestions Ġas : ĠIN LINE FORM 2 </s> Note Ġthat ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġis Ġtransformed Ġfrom ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġby Ġone Ġlayer Ġneural Ġnetwork ĠIN LINE FORM 4 Ġ, Ġrespectively . ĠA Ġquestion - aware Ġpassage Ġrepresentation Ġis Ġcomputed Ġas ĠIN LINE FORM 5 Ġ. ĠAfter Ġthat , Ġwe Ġuse Ġthe Ġmethod Ġof ĠB IB REF 13 Ġto Ġapply Ġself Ġattention Ġto Ġthe Ġpassage : ĠIN LINE FORM 6 </s> where ĠIN LINE FORM 0 Ġmeans Ġthat Ġwe Ġonly Ġdrop Ġdiagonal Ġelements Ġon Ġthe Ġsimilarity Ġmatrix Ġ( i . e ., Ġattention Ġwith Ġitself ). ĠAt Ġlast , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġconc aten ated Ġand Ġare Ġpassed Ġthrough Ġa ĠBi L ST M Ġto Ġform Ġthe Ġfinal Ġmemory : ĠIN LINE FORM 3 Ġ. </s> Sp an Ġdetector . ĠWe Ġadopt Ġa Ġmulti - turn Ġanswer Ġmodule Ġfor Ġthe Ġspan Ġdetector ĠB IB REF 1 Ġ. ĠForm ally , Ġat Ġtime Ġstep ĠIN LINE FORM 0 Ġin Ġthe Ġrange Ġof ĠIN LINE FORM 1 Ġ, Ġthe Ġstate Ġis Ġdefined Ġby ĠIN LINE FORM 2 Ġ. ĠThe Ġinitial Ġstate ĠIN LINE FORM 3 Ġis Ġthe Ġsummary Ġof Ġthe ĠIN LINE FORM 4 Ġ: ĠIN LINE FORM 5 Ġ, Ġwhere ĠIN LINE FORM 6 Ġ. ĠHere , ĠIN LINE FORM 7 Ġis Ġcomputed Ġfrom Ġthe Ġprevious Ġstate ĠIN LINE FORM 8 Ġand Ġmemory ĠIN LINE FORM 9 Ġ: ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. ĠFinally , Ġa Ġbil in ear Ġfunction Ġis Ġused Ġto Ġfind Ġthe Ġbegin Ġand Ġend Ġpoint Ġof Ġanswer Ġspans Ġat Ġeach Ġreasoning Ġstep ĠIN LINE FORM 12 Ġ: ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> The Ġfinal Ġprediction Ġis Ġthe Ġaverage Ġof Ġeach Ġtime Ġstep : ĠIN LINE FORM 0 Ġ. ĠWe Ġrandomly Ġapply Ġdrop out Ġon Ġthe Ġstep Ġlevel Ġin Ġeach Ġtime Ġstep Ġduring Ġtraining , Ġas Ġdone Ġin ĠB IB REF 1 Ġ. </s> Un answer able Ġclass ifier . ĠWe Ġadopt Ġa Ġone - layer Ġneural Ġnetwork Ġas Ġour Ġun answer able Ġbinary Ġclass ifier : ĠDIS PLAY FORM 0 </s> , Ġwhere ĠIN LINE FORM 0 Ġis Ġthe Ġsummary Ġof Ġthe Ġmemory : ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġ. ĠIN LINE FORM 3 Ġdenotes Ġthe Ġprobability Ġof Ġthe Ġquestion Ġwhich Ġis Ġun answer able . </s> Object ive ĠThe Ġobjective Ġfunction Ġof Ġthe Ġjoint Ġmodel Ġhas Ġtwo Ġparts : ĠDIS PLAY FORM 0 </s> Following ĠB IB REF 0 Ġ, Ġthe Ġspan Ġloss Ġfunction Ġis Ġdefined : ĠDIS PLAY FORM 0 </s> The Ġobjective Ġfunction Ġof Ġthe Ġbinary Ġclass ifier Ġis Ġdefined : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġbinary Ġvariable : ĠIN LINE FORM 1 Ġindicates Ġthe Ġquestion Ġis Ġun answer able Ġand ĠIN LINE FORM 2 Ġdenotes Ġthe Ġquestion Ġis Ġanswer able . </s> Setup </s> We Ġevaluate Ġour Ġsystem Ġon ĠS Qu AD Ġ2 . 0 Ġdataset ĠB IB REF 14 Ġ, Ġa Ġnew ĠM RC Ġdataset Ġwhich Ġis Ġa Ġcombination Ġof ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD ) Ġ1 . 0 ĠB IB REF 15 Ġand Ġadditional Ġun answer able Ġquestion - answer Ġpairs . ĠThe Ġanswer able Ġpairs Ġare Ġaround Ġ100 K ; Ġwhile Ġthe Ġun answer able Ġquestions Ġare Ġaround Ġ53 K . ĠThis Ġdataset Ġcontains Ġabout Ġ23 K Ġpassages Ġand Ġthey Ġcome Ġfrom Ġapproximately Ġ500 ĠWikipedia Ġarticles . ĠAll Ġthe Ġquestions Ġand Ġanswers Ġare Ġobtained Ġby Ġcrowd - s ourcing . ĠTwo Ġevaluation Ġmetrics Ġare Ġused : ĠEx act ĠMatch Ġ( EM ) Ġand ĠMacro - aver aged ĠF 1 Ġscore Ġ( F 1 ) ĠB IB REF 14 Ġ. </s> Im plementation Ġdetails </s> We Ġutilize Ġspa Cy Ġtool Ġto Ġtoken ize Ġthe Ġboth Ġpassages Ġand Ġquestions , Ġand Ġgenerate Ġle mma , Ġpart - of - speech Ġand Ġnamed Ġentity Ġtags . ĠThe Ġword Ġembed d ings Ġare Ġinitialized Ġwith Ġpre - trained Ġ300 - dimensional ĠGl o Ve ĠB IB REF 10 Ġ. ĠA Ġ2 - layer ĠBi L ST M Ġis Ġused Ġencoding Ġthe Ġcontextual Ġinformation Ġof Ġboth Ġquestions Ġand Ġpassages . ĠRegarding Ġthe Ġhidden Ġsize Ġof Ġour Ġmodel , Ġwe Ġsearch Ġgreed ily Ġamong ĠIN LINE FORM 0 Ġ. ĠDuring Ġtraining , ĠAdam ax ĠB IB REF 16 Ġis Ġused Ġas Ġour Ġoptim izer . ĠThe Ġmin - batch Ġsize Ġis Ġset Ġto Ġ32 . ĠThe Ġlearning Ġrate Ġis Ġinitialized Ġto Ġ0 . 002 Ġand Ġit Ġis Ġhal ved Ġafter Ġevery Ġ10 Ġepoch s . ĠThe Ġdrop out Ġrate Ġis Ġset Ġto Ġ0 . 1 . ĠTo Ġprevent Ġover fitting , Ġwe Ġalso Ġrandomly Ġset Ġ0 . 5 % Ġwords Ġin Ġboth Ġpassages Ġand Ġquestions Ġas Ġunknown Ġwords Ġduring Ġthe Ġtraining . ĠHere , Ġwe Ġuse Ġa Ġspecial Ġtoken Ġun k Ġto Ġindicate Ġa Ġword Ġwhich Ġdoesn 't Ġappear Ġin ĠGl o Ve . ĠIN LINE FORM 1 Ġin ĠE q ĠEQ REF 9 Ġis Ġset Ġto Ġ1 . </s> Results </s> We Ġwould Ġlike Ġto Ġinvestigate Ġeffectiveness Ġthe Ġproposed Ġjoint Ġmodel . ĠTo Ġdo Ġso , Ġthe Ġsame Ġshared Ġlayer / arch itect ure Ġis Ġemployed Ġin Ġthe Ġfollowing Ġvariants Ġof Ġthe Ġproposed Ġmodel : </s> The Ġresults Ġin Ġterms Ġof ĠEM Ġand ĠF 1 Ġis Ġsummarized Ġin ĠTable ĠT AB REF 20 Ġ. ĠWe Ġobserve Ġthat ĠJoint ĠSAN Ġoutper forms Ġthe ĠSAN Ġbaseline Ġwith Ġa Ġlarge Ġmargin , Ġe . g ., Ġ67 . 89 Ġvs Ġ69 . 27 Ġ(+ 1 . 38 ) Ġand Ġ70 . 68 Ġvs Ġ72 . 20 Ġ(+ 1 . 52 ) Ġin Ġterms Ġof ĠEM Ġand ĠF 1 Ġscores Ġrespectively , Ġso Ġit Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġthe Ġjoint Ġoptimization . ĠBy Ġincorporating Ġthe Ġoutput Ġinformation Ġof Ġclass ifier Ġinto ĠJoint ĠSAN , Ġit Ġob tains Ġa Ġslight Ġimprovement , Ġe . g ., Ġ72 . 2 Ġvs Ġ72 . 66 Ġ(+ 0 . 46 ) Ġin Ġterms Ġof ĠF 1 Ġscore . ĠBy Ġanalyzing Ġthe Ġresults , Ġwe Ġfound Ġthat Ġin Ġmost Ġcases Ġwhen Ġour Ġmodel Ġextract Ġan ĠNULL Ġstring Ġanswer , Ġthe Ġclass ifier Ġalso Ġpredicts Ġit Ġas Ġan Ġun answer able Ġquestion Ġwith Ġa Ġhigh Ġprobability . </s> Table ĠT AB REF 21 Ġreports Ġcomparison Ġresults Ġin Ġliterature Ġpublished Ġ. ĠOur Ġmodel Ġachieves Ġstate - of - the - art Ġon Ġdevelopment Ġdataset Ġin Ġsetting Ġwithout Ġpre - trained Ġlarge Ġlanguage Ġmodel Ġ( EL Mo ). ĠComp aring Ġwith Ġthe Ġmuch Ġcomplicated Ġmodel ĠR . M .- Reader Ġ+ ĠVer ifier , Ġwhich Ġincludes Ġseveral Ġcomponents , Ġour Ġmodel Ġstill Ġoutper forms Ġby Ġ0 . 7 Ġin Ġterms Ġof ĠF 1 Ġscore . ĠFurthermore , Ġwe Ġobserve Ġthat ĠEL Mo Ġgives Ġa Ġgreat Ġboosting Ġon Ġthe Ġperformance , Ġe . g ., Ġ2 . 8 Ġpoints Ġin Ġterms Ġof ĠF 1 Ġfor ĠDoc Q A . ĠThis Ġencourages Ġus Ġto Ġincorporate ĠEL Mo Ġinto Ġour Ġmodel Ġin Ġfuture . </s> Analysis . ĠTo Ġbetter Ġunderstand Ġour Ġmodel , Ġwe Ġanalyze Ġthe Ġaccuracy Ġof Ġthe Ġclass ifier Ġin Ġour Ġjoint Ġmodel . ĠWe Ġobtain Ġ75 . 3 Ġclassification Ġaccuracy Ġon Ġthe Ġdevelopment Ġwith Ġthe Ġthreshold Ġ0 . 5 . ĠBy Ġincreasing Ġvalue Ġof ĠIN LINE FORM 0 Ġin ĠE q ĠEQ REF 9 Ġ, Ġthe Ġclassification Ġaccuracy Ġreached Ġto Ġ76 . 8 Ġ( ĠIN LINE FORM 1 Ġ), Ġhowever Ġthe Ġfinal Ġresults Ġof Ġour Ġmodel Ġonly Ġhave Ġa Ġsmall Ġimprovement Ġ(+ 0 . 2 Ġin Ġterms Ġof ĠF 1 Ġscore ). ĠIt Ġshows Ġthat Ġit Ġis Ġimportant Ġto Ġmake Ġbalance Ġbetween Ġthese Ġtwo Ġcomponents : Ġthe Ġspan Ġdetector Ġand Ġun answer able Ġclass ifier . </s> Conclusion </s> To Ġsum Ġup , Ġwe Ġproposed Ġa Ġsimple Ġyet Ġefficient Ġmodel Ġbased Ġon ĠSAN . ĠIt Ġshowed Ġthat Ġthe Ġjoint Ġlearning Ġalgorithm Ġboosted Ġthe Ġperformance Ġon ĠS Qu AD Ġ2 . 0 . ĠWe Ġalso Ġwould Ġlike Ġto Ġincorporate ĠEL Mo Ġinto Ġour Ġmodel Ġin Ġfuture . </s> Acknowled gments </s> We Ġthank ĠY ich ong ĠXu , ĠShu oh ang ĠWang Ġand ĠShe ng ĠZhang Ġfor Ġvaluable Ġdiscussions Ġand Ġcomments . ĠWe Ġalso Ġthank ĠRobin ĠJ ia Ġfor Ġthe Ġhelp Ġon ĠS Qu AD Ġevaluations .
Background Teaching machine to read and comprehend a given passage/paragraph and answer its corresponding questions is a challenging task. It is also one of the long-term goals of natural language understanding, and has important applications in e.g., building intelligent agents for conversation and customer service support. In a real world setting, it is necessary to judge whether the given questions are answerable given the available knowledge, and then generate correct answers for the ones which are able to infer an answer in the passage or an empty answer (as an unanswerable question) otherwise. In comparison with many existing MRC systems BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , which extract answers by finding a sub-string in the passages/paragraphs, we propose a model that not only extracts answers but also predicts whether such an answer should exist. Using a multi-task learning approach (c.f. BIBREF5 ), we extend the Stochastic Answer Network (SAN) BIBREF1 for MRC answer span detector to include a classifier that whether the question is unanswerable. The unanswerable classifier is a pair-wise classification model BIBREF6 which predicts a label indicating whether the given pair of a passage and a question is unanswerable. The two models share the same lower layer to save the number of parameters, and separate the top layers for different tasks (the span detector and binary classifier). Our model is pretty simple and intuitive, yet efficient. Without relying on the large pre-trained language models (ELMo) BIBREF7 , the proposed model achieves competitive results to the state-of-the-art on Stanford Question Answering Dataset (SQuAD) 2.0. The contribution of this work is summarized as follows. First, we propose a simple yet efficient model for MRC that handles unanswerable questions and is optimized jointly. Second, our model achieves competitive results on SQuAD v2.0. Model The Machine Reading Comprehension is a task which takes a question INLINEFORM0 and a passage/paragraph INLINEFORM1 as inputs, and aims to find an answer span INLINEFORM2 in INLINEFORM3 . We assume that if the question is answerable, the answer INLINEFORM4 exists in INLINEFORM5 as a contiguous text string; otherwise, INLINEFORM6 is an empty string indicating an unanswerable question. Note that to handle the unanswerable questions, we manually append a dumpy text string NULL at the end of each corresponding passage/paragraph. Formally, the answer is formulated as INLINEFORM7 . In case of unanswerable questions, INLINEFORM8 points to the last token of the passage. Our model is a variation of SAN BIBREF1 , as shown in Figure FIGREF2 . The main difference is the additional binary classifier added in the model justifying whether the question is unanswerable. Roughly, the model includes two different layers: the shared layer and task specific layer. The shared layer is almost identical to the lower layers of SAN, which has a lexicon encoding layer, a contextual layer and a memory generation layer. On top of it, there are different answer modules for different tasks. We employ the SAN answer module for the span detector and a one-layer feed forward neural network for the binary classification task. It can also be viewed as a multi-task learning BIBREF8 , BIBREF5 , BIBREF9 . We will briefly describe the model from ground up as follows. Detailed descriptions can be found in BIBREF1 . Lexicon Encoding Layer. We map the symbolic/surface feature of INLINEFORM0 and INLINEFORM1 into neural space via word embeddings , 16-dim part-of-speech (POS) tagging embeddings, 8-dim named-entity embeddings and 4-dim hard-rule features. Note that we use small embedding size of POS and NER to reduce model size and they mainly serve the role of coarse-grained word clusters. Additionally, we use question enhanced passages word embeddings which can viewwed as soft matching between questions and passages. At last, we use two separate two-layer position-wise Feed-Forward Networks (FFN) BIBREF11 , BIBREF1 to map both question and passage encodings into the same dimension. As results, we obtain the final lexicon embeddings for the tokens for INLINEFORM2 as a matrix INLINEFORM3 , and tokens in INLINEFORM4 as INLINEFORM5 . Contextual Encoding Layer. A shared two-layers BiLSTM is used on the top to encode the contextual information of both passages and questions. To avoid overfitting, we concatenate a pre-trained 600-dimensional CoVe vectors BIBREF12 trained on German-English machine translation dataset, with the aforementioned lexicon embeddings as the final input of the contextual encoding layer, and also with the output of the first contextual encoding layer as the input of its second encoding layer. Thus, we obtain the final representation of the contextual encoding layer by a concatenation of the outputs of two BiLSTM: INLINEFORM0 for questions and INLINEFORM1 for passages. Memory Generation Layer. In this layer, we generate a working memory by fusing information from both passages INLINEFORM0 and questions INLINEFORM1 . The attention function BIBREF11 is used to compute the similarity score between passages and questions as: INLINEFORM2 Note that INLINEFORM0 and INLINEFORM1 is transformed from INLINEFORM2 and INLINEFORM3 by one layer neural network INLINEFORM4 , respectively. A question-aware passage representation is computed as INLINEFORM5 . After that, we use the method of BIBREF13 to apply self attention to the passage: INLINEFORM6 where INLINEFORM0 means that we only drop diagonal elements on the similarity matrix (i.e., attention with itself). At last, INLINEFORM1 and INLINEFORM2 are concatenated and are passed through a BiLSTM to form the final memory: INLINEFORM3 . Span detector. We adopt a multi-turn answer module for the span detector BIBREF1 . Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2 . The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6 . Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11 . Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1 The final prediction is the average of each time step: INLINEFORM0 . We randomly apply dropout on the step level in each time step during training, as done in BIBREF1 . Unanswerable classifier. We adopt a one-layer neural network as our unanswerable binary classifier: DISPLAYFORM0 , where INLINEFORM0 is the summary of the memory: INLINEFORM1 , where INLINEFORM2 . INLINEFORM3 denotes the probability of the question which is unanswerable. Objective The objective function of the joint model has two parts: DISPLAYFORM0 Following BIBREF0 , the span loss function is defined: DISPLAYFORM0 The objective function of the binary classifier is defined: DISPLAYFORM0 where INLINEFORM0 is a binary variable: INLINEFORM1 indicates the question is unanswerable and INLINEFORM2 denotes the question is answerable. Setup We evaluate our system on SQuAD 2.0 dataset BIBREF14 , a new MRC dataset which is a combination of Stanford Question Answering Dataset (SQuAD) 1.0 BIBREF15 and additional unanswerable question-answer pairs. The answerable pairs are around 100K; while the unanswerable questions are around 53K. This dataset contains about 23K passages and they come from approximately 500 Wikipedia articles. All the questions and answers are obtained by crowd-sourcing. Two evaluation metrics are used: Exact Match (EM) and Macro-averaged F1 score (F1) BIBREF14 . Implementation details We utilize spaCy tool to tokenize the both passages and questions, and generate lemma, part-of-speech and named entity tags. The word embeddings are initialized with pre-trained 300-dimensional GloVe BIBREF10 . A 2-layer BiLSTM is used encoding the contextual information of both questions and passages. Regarding the hidden size of our model, we search greedily among INLINEFORM0 . During training, Adamax BIBREF16 is used as our optimizer. The min-batch size is set to 32. The learning rate is initialized to 0.002 and it is halved after every 10 epochs. The dropout rate is set to 0.1. To prevent overfitting, we also randomly set 0.5% words in both passages and questions as unknown words during the training. Here, we use a special token unk to indicate a word which doesn't appear in GloVe. INLINEFORM1 in Eq EQREF9 is set to 1. Results We would like to investigate effectiveness the proposed joint model. To do so, the same shared layer/architecture is employed in the following variants of the proposed model: The results in terms of EM and F1 is summarized in Table TABREF20 . We observe that Joint SAN outperforms the SAN baseline with a large margin, e.g., 67.89 vs 69.27 (+1.38) and 70.68 vs 72.20 (+1.52) in terms of EM and F1 scores respectively, so it demonstrates the effectiveness of the joint optimization. By incorporating the output information of classifier into Joint SAN, it obtains a slight improvement, e.g., 72.2 vs 72.66 (+0.46) in terms of F1 score. By analyzing the results, we found that in most cases when our model extract an NULL string answer, the classifier also predicts it as an unanswerable question with a high probability. Table TABREF21 reports comparison results in literature published . Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo). Comparing with the much complicated model R.M.-Reader + Verifier, which includes several components, our model still outperforms by 0.7 in terms of F1 score. Furthermore, we observe that ELMo gives a great boosting on the performance, e.g., 2.8 points in terms of F1 for DocQA. This encourages us to incorporate ELMo into our model in future. Analysis. To better understand our model, we analyze the accuracy of the classifier in our joint model. We obtain 75.3 classification accuracy on the development with the threshold 0.5. By increasing value of INLINEFORM0 in Eq EQREF9 , the classification accuracy reached to 76.8 ( INLINEFORM1 ), however the final results of our model only have a small improvement (+0.2 in terms of F1 score). It shows that it is important to make balance between these two components: the span detector and unanswerable classifier. Conclusion To sum up, we proposed a simple yet efficient model based on SAN. It showed that the joint learning algorithm boosted the performance on SQuAD 2.0. We also would like to incorporate ELMo into our model in future. Acknowledgments We thank Yichong Xu, Shuohang Wang and Sheng Zhang for valuable discussions and comments. We also thank Robin Jia for the help on SQuAD evaluations.
What other models do they compare to?
d3dbb5c22ef204d85707d2d24284cc77fa816b6c
What Ġother Ġmodels Ġdo Ġthey Ġcompare Ġto ?
SAN ĠBas eline , ĠB NA , ĠDoc Q A , ĠR . M - Reader , ĠR . M - Reader + Ver ifier Ġand ĠDoc Q A + EL Mo
<s> What Ġother Ġmodels Ġdo Ġthey Ġcompare Ġto ? </s> Background </s> Te aching Ġmachine Ġto Ġread Ġand Ġcomprehend Ġa Ġgiven Ġpassage / paragraph Ġand Ġanswer Ġits Ġcorresponding Ġquestions Ġis Ġa Ġchallenging Ġtask . ĠIt Ġis Ġalso Ġone Ġof Ġthe Ġlong - term Ġgoals Ġof Ġnatural Ġlanguage Ġunderstanding , Ġand Ġhas Ġimportant Ġapplications Ġin Ġe . g ., Ġbuilding Ġintelligent Ġagents Ġfor Ġconversation Ġand Ġcustomer Ġservice Ġsupport . ĠIn Ġa Ġreal Ġworld Ġsetting , Ġit Ġis Ġnecessary Ġto Ġjudge Ġwhether Ġthe Ġgiven Ġquestions Ġare Ġanswer able Ġgiven Ġthe Ġavailable Ġknowledge , Ġand Ġthen Ġgenerate Ġcorrect Ġanswers Ġfor Ġthe Ġones Ġwhich Ġare Ġable Ġto Ġinfer Ġan Ġanswer Ġin Ġthe Ġpassage Ġor Ġan Ġempty Ġanswer Ġ( as Ġan Ġun answer able Ġquestion ) Ġotherwise . </s> In Ġcomparison Ġwith Ġmany Ġexisting ĠM RC Ġsystems ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġwhich Ġextract Ġanswers Ġby Ġfinding Ġa Ġsub - string Ġin Ġthe Ġpassages / paragraph s , Ġwe Ġpropose Ġa Ġmodel Ġthat Ġnot Ġonly Ġextracts Ġanswers Ġbut Ġalso Ġpredicts Ġwhether Ġsuch Ġan Ġanswer Ġshould Ġexist . ĠUsing Ġa Ġmulti - task Ġlearning Ġapproach Ġ( c . f . ĠB IB REF 5 Ġ), Ġwe Ġextend Ġthe ĠSt och astic ĠAnswer ĠNetwork Ġ( SAN ) ĠB IB REF 1 Ġfor ĠM RC Ġanswer Ġspan Ġdetector Ġto Ġinclude Ġa Ġclass ifier Ġthat Ġwhether Ġthe Ġquestion Ġis Ġun answer able . ĠThe Ġun answer able Ġclass ifier Ġis Ġa Ġpair - wise Ġclassification Ġmodel ĠB IB REF 6 Ġwhich Ġpredicts Ġa Ġlabel Ġindicating Ġwhether Ġthe Ġgiven Ġpair Ġof Ġa Ġpassage Ġand Ġa Ġquestion Ġis Ġun answer able . ĠThe Ġtwo Ġmodels Ġshare Ġthe Ġsame Ġlower Ġlayer Ġto Ġsave Ġthe Ġnumber Ġof Ġparameters , Ġand Ġseparate Ġthe Ġtop Ġlayers Ġfor Ġdifferent Ġtasks Ġ( the Ġspan Ġdetector Ġand Ġbinary Ġclass ifier ). </s> Our Ġmodel Ġis Ġpretty Ġsimple Ġand Ġintuitive , Ġyet Ġefficient . ĠWithout Ġrelying Ġon Ġthe Ġlarge Ġpre - trained Ġlanguage Ġmodels Ġ( EL Mo ) ĠB IB REF 7 Ġ, Ġthe Ġproposed Ġmodel Ġachieves Ġcompetitive Ġresults Ġto Ġthe Ġstate - of - the - art Ġon ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD ) Ġ2 . 0 . </s> The Ġcontribution Ġof Ġthis Ġwork Ġis Ġsummarized Ġas Ġfollows . ĠFirst , Ġwe Ġpropose Ġa Ġsimple Ġyet Ġefficient Ġmodel Ġfor ĠM RC Ġthat Ġhandles Ġun answer able Ġquestions Ġand Ġis Ġoptimized Ġjointly . ĠSecond , Ġour Ġmodel Ġachieves Ġcompetitive Ġresults Ġon ĠS Qu AD Ġv 2 . 0 . </s> Model </s> The ĠMachine ĠReading ĠComp rehens ion Ġis Ġa Ġtask Ġwhich Ġtakes Ġa Ġquestion ĠIN LINE FORM 0 Ġand Ġa Ġpassage / paragraph ĠIN LINE FORM 1 Ġas Ġinputs , Ġand Ġaims Ġto Ġfind Ġan Ġanswer Ġspan ĠIN LINE FORM 2 Ġin ĠIN LINE FORM 3 Ġ. ĠWe Ġassume Ġthat Ġif Ġthe Ġquestion Ġis Ġanswer able , Ġthe Ġanswer ĠIN LINE FORM 4 Ġexists Ġin ĠIN LINE FORM 5 Ġas Ġa Ġcontiguous Ġtext Ġstring ; Ġotherwise , ĠIN LINE FORM 6 Ġis Ġan Ġempty Ġstring Ġindicating Ġan Ġun answer able Ġquestion . ĠNote Ġthat Ġto Ġhandle Ġthe Ġun answer able Ġquestions , Ġwe Ġmanually Ġappend Ġa Ġdump y Ġtext Ġstring ĠNULL Ġat Ġthe Ġend Ġof Ġeach Ġcorresponding Ġpassage / paragraph . ĠForm ally , Ġthe Ġanswer Ġis Ġformulated Ġas ĠIN LINE FORM 7 Ġ. ĠIn Ġcase Ġof Ġun answer able Ġquestions , ĠIN LINE FORM 8 Ġpoints Ġto Ġthe Ġlast Ġtoken Ġof Ġthe Ġpassage . </s> Our Ġmodel Ġis Ġa Ġvariation Ġof ĠSAN ĠB IB REF 1 Ġ, Ġas Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ. ĠThe Ġmain Ġdifference Ġis Ġthe Ġadditional Ġbinary Ġclass ifier Ġadded Ġin Ġthe Ġmodel Ġjustifying Ġwhether Ġthe Ġquestion Ġis Ġun answer able . ĠRough ly , Ġthe Ġmodel Ġincludes Ġtwo Ġdifferent Ġlayers : Ġthe Ġshared Ġlayer Ġand Ġtask Ġspecific Ġlayer . ĠThe Ġshared Ġlayer Ġis Ġalmost Ġidentical Ġto Ġthe Ġlower Ġlayers Ġof ĠSAN , Ġwhich Ġhas Ġa Ġlex icon Ġencoding Ġlayer , Ġa Ġcontextual Ġlayer Ġand Ġa Ġmemory Ġgeneration Ġlayer . ĠOn Ġtop Ġof Ġit , Ġthere Ġare Ġdifferent Ġanswer Ġmodules Ġfor Ġdifferent Ġtasks . ĠWe Ġemploy Ġthe ĠSAN Ġanswer Ġmodule Ġfor Ġthe Ġspan Ġdetector Ġand Ġa Ġone - layer Ġfeed Ġforward Ġneural Ġnetwork Ġfor Ġthe Ġbinary Ġclassification Ġtask . ĠIt Ġcan Ġalso Ġbe Ġviewed Ġas Ġa Ġmulti - task Ġlearning ĠB IB REF 8 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 9 Ġ. ĠWe Ġwill Ġbriefly Ġdescribe Ġthe Ġmodel Ġfrom Ġground Ġup Ġas Ġfollows . ĠDet ailed Ġdescriptions Ġcan Ġbe Ġfound Ġin ĠB IB REF 1 Ġ. </s> Lex icon ĠEnc oding ĠLayer . ĠWe Ġmap Ġthe Ġsymbolic / surface Ġfeature Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġinto Ġneural Ġspace Ġvia Ġword Ġembed d ings Ġ, Ġ16 - dim Ġpart - of - speech Ġ( POS ) Ġtagging Ġembed d ings , Ġ8 - dim Ġnamed - entity Ġembed d ings Ġand Ġ4 - dim Ġhard - rule Ġfeatures . ĠNote Ġthat Ġwe Ġuse Ġsmall Ġembed ding Ġsize Ġof ĠPOS Ġand ĠN ER Ġto Ġreduce Ġmodel Ġsize Ġand Ġthey Ġmainly Ġserve Ġthe Ġrole Ġof Ġcoarse - gr ained Ġword Ġclusters . ĠAdditionally , Ġwe Ġuse Ġquestion Ġenhanced Ġpassages Ġword Ġembed d ings Ġwhich Ġcan Ġview wed Ġas Ġsoft Ġmatching Ġbetween Ġquestions Ġand Ġpassages . ĠAt Ġlast , Ġwe Ġuse Ġtwo Ġseparate Ġtwo - layer Ġposition - wise ĠFeed - Forward ĠNetworks Ġ( FF N ) ĠB IB REF 11 Ġ, ĠB IB REF 1 Ġto Ġmap Ġboth Ġquestion Ġand Ġpassage Ġenc od ings Ġinto Ġthe Ġsame Ġdimension . ĠAs Ġresults , Ġwe Ġobtain Ġthe Ġfinal Ġlex icon Ġembed d ings Ġfor Ġthe Ġtokens Ġfor ĠIN LINE FORM 2 Ġas Ġa Ġmatrix ĠIN LINE FORM 3 Ġ, Ġand Ġtokens Ġin ĠIN LINE FORM 4 Ġas ĠIN LINE FORM 5 Ġ. </s> Context ual ĠEnc oding ĠLayer . ĠA Ġshared Ġtwo - l ayers ĠBi L ST M Ġis Ġused Ġon Ġthe Ġtop Ġto Ġencode Ġthe Ġcontextual Ġinformation Ġof Ġboth Ġpassages Ġand Ġquestions . ĠTo Ġavoid Ġover fitting , Ġwe Ġconc aten ate Ġa Ġpre - trained Ġ600 - dimensional ĠCo Ve Ġvectors ĠB IB REF 12 Ġtrained Ġon ĠGerman - English Ġmachine Ġtranslation Ġdataset , Ġwith Ġthe Ġaforementioned Ġlex icon Ġembed d ings Ġas Ġthe Ġfinal Ġinput Ġof Ġthe Ġcontextual Ġencoding Ġlayer , Ġand Ġalso Ġwith Ġthe Ġoutput Ġof Ġthe Ġfirst Ġcontextual Ġencoding Ġlayer Ġas Ġthe Ġinput Ġof Ġits Ġsecond Ġencoding Ġlayer . ĠThus , Ġwe Ġobtain Ġthe Ġfinal Ġrepresentation Ġof Ġthe Ġcontextual Ġencoding Ġlayer Ġby Ġa Ġconc aten ation Ġof Ġthe Ġoutputs Ġof Ġtwo ĠBi L ST M : ĠIN LINE FORM 0 Ġfor Ġquestions Ġand ĠIN LINE FORM 1 Ġfor Ġpassages . </s> Memory ĠGeneration ĠLayer . ĠIn Ġthis Ġlayer , Ġwe Ġgenerate Ġa Ġworking Ġmemory Ġby Ġf using Ġinformation Ġfrom Ġboth Ġpassages ĠIN LINE FORM 0 Ġand Ġquestions ĠIN LINE FORM 1 Ġ. ĠThe Ġattention Ġfunction ĠB IB REF 11 Ġis Ġused Ġto Ġcompute Ġthe Ġsimilarity Ġscore Ġbetween Ġpassages Ġand Ġquestions Ġas : ĠIN LINE FORM 2 </s> Note Ġthat ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġis Ġtransformed Ġfrom ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġby Ġone Ġlayer Ġneural Ġnetwork ĠIN LINE FORM 4 Ġ, Ġrespectively . ĠA Ġquestion - aware Ġpassage Ġrepresentation Ġis Ġcomputed Ġas ĠIN LINE FORM 5 Ġ. ĠAfter Ġthat , Ġwe Ġuse Ġthe Ġmethod Ġof ĠB IB REF 13 Ġto Ġapply Ġself Ġattention Ġto Ġthe Ġpassage : ĠIN LINE FORM 6 </s> where ĠIN LINE FORM 0 Ġmeans Ġthat Ġwe Ġonly Ġdrop Ġdiagonal Ġelements Ġon Ġthe Ġsimilarity Ġmatrix Ġ( i . e ., Ġattention Ġwith Ġitself ). ĠAt Ġlast , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġconc aten ated Ġand Ġare Ġpassed Ġthrough Ġa ĠBi L ST M Ġto Ġform Ġthe Ġfinal Ġmemory : ĠIN LINE FORM 3 Ġ. </s> Sp an Ġdetector . ĠWe Ġadopt Ġa Ġmulti - turn Ġanswer Ġmodule Ġfor Ġthe Ġspan Ġdetector ĠB IB REF 1 Ġ. ĠForm ally , Ġat Ġtime Ġstep ĠIN LINE FORM 0 Ġin Ġthe Ġrange Ġof ĠIN LINE FORM 1 Ġ, Ġthe Ġstate Ġis Ġdefined Ġby ĠIN LINE FORM 2 Ġ. ĠThe Ġinitial Ġstate ĠIN LINE FORM 3 Ġis Ġthe Ġsummary Ġof Ġthe ĠIN LINE FORM 4 Ġ: ĠIN LINE FORM 5 Ġ, Ġwhere ĠIN LINE FORM 6 Ġ. ĠHere , ĠIN LINE FORM 7 Ġis Ġcomputed Ġfrom Ġthe Ġprevious Ġstate ĠIN LINE FORM 8 Ġand Ġmemory ĠIN LINE FORM 9 Ġ: ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. ĠFinally , Ġa Ġbil in ear Ġfunction Ġis Ġused Ġto Ġfind Ġthe Ġbegin Ġand Ġend Ġpoint Ġof Ġanswer Ġspans Ġat Ġeach Ġreasoning Ġstep ĠIN LINE FORM 12 Ġ: ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> The Ġfinal Ġprediction Ġis Ġthe Ġaverage Ġof Ġeach Ġtime Ġstep : ĠIN LINE FORM 0 Ġ. ĠWe Ġrandomly Ġapply Ġdrop out Ġon Ġthe Ġstep Ġlevel Ġin Ġeach Ġtime Ġstep Ġduring Ġtraining , Ġas Ġdone Ġin ĠB IB REF 1 Ġ. </s> Un answer able Ġclass ifier . ĠWe Ġadopt Ġa Ġone - layer Ġneural Ġnetwork Ġas Ġour Ġun answer able Ġbinary Ġclass ifier : ĠDIS PLAY FORM 0 </s> , Ġwhere ĠIN LINE FORM 0 Ġis Ġthe Ġsummary Ġof Ġthe Ġmemory : ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġ. ĠIN LINE FORM 3 Ġdenotes Ġthe Ġprobability Ġof Ġthe Ġquestion Ġwhich Ġis Ġun answer able . </s> Object ive ĠThe Ġobjective Ġfunction Ġof Ġthe Ġjoint Ġmodel Ġhas Ġtwo Ġparts : ĠDIS PLAY FORM 0 </s> Following ĠB IB REF 0 Ġ, Ġthe Ġspan Ġloss Ġfunction Ġis Ġdefined : ĠDIS PLAY FORM 0 </s> The Ġobjective Ġfunction Ġof Ġthe Ġbinary Ġclass ifier Ġis Ġdefined : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġbinary Ġvariable : ĠIN LINE FORM 1 Ġindicates Ġthe Ġquestion Ġis Ġun answer able Ġand ĠIN LINE FORM 2 Ġdenotes Ġthe Ġquestion Ġis Ġanswer able . </s> Setup </s> We Ġevaluate Ġour Ġsystem Ġon ĠS Qu AD Ġ2 . 0 Ġdataset ĠB IB REF 14 Ġ, Ġa Ġnew ĠM RC Ġdataset Ġwhich Ġis Ġa Ġcombination Ġof ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD ) Ġ1 . 0 ĠB IB REF 15 Ġand Ġadditional Ġun answer able Ġquestion - answer Ġpairs . ĠThe Ġanswer able Ġpairs Ġare Ġaround Ġ100 K ; Ġwhile Ġthe Ġun answer able Ġquestions Ġare Ġaround Ġ53 K . ĠThis Ġdataset Ġcontains Ġabout Ġ23 K Ġpassages Ġand Ġthey Ġcome Ġfrom Ġapproximately Ġ500 ĠWikipedia Ġarticles . ĠAll Ġthe Ġquestions Ġand Ġanswers Ġare Ġobtained Ġby Ġcrowd - s ourcing . ĠTwo Ġevaluation Ġmetrics Ġare Ġused : ĠEx act ĠMatch Ġ( EM ) Ġand ĠMacro - aver aged ĠF 1 Ġscore Ġ( F 1 ) ĠB IB REF 14 Ġ. </s> Im plementation Ġdetails </s> We Ġutilize Ġspa Cy Ġtool Ġto Ġtoken ize Ġthe Ġboth Ġpassages Ġand Ġquestions , Ġand Ġgenerate Ġle mma , Ġpart - of - speech Ġand Ġnamed Ġentity Ġtags . ĠThe Ġword Ġembed d ings Ġare Ġinitialized Ġwith Ġpre - trained Ġ300 - dimensional ĠGl o Ve ĠB IB REF 10 Ġ. ĠA Ġ2 - layer ĠBi L ST M Ġis Ġused Ġencoding Ġthe Ġcontextual Ġinformation Ġof Ġboth Ġquestions Ġand Ġpassages . ĠRegarding Ġthe Ġhidden Ġsize Ġof Ġour Ġmodel , Ġwe Ġsearch Ġgreed ily Ġamong ĠIN LINE FORM 0 Ġ. ĠDuring Ġtraining , ĠAdam ax ĠB IB REF 16 Ġis Ġused Ġas Ġour Ġoptim izer . ĠThe Ġmin - batch Ġsize Ġis Ġset Ġto Ġ32 . ĠThe Ġlearning Ġrate Ġis Ġinitialized Ġto Ġ0 . 002 Ġand Ġit Ġis Ġhal ved Ġafter Ġevery Ġ10 Ġepoch s . ĠThe Ġdrop out Ġrate Ġis Ġset Ġto Ġ0 . 1 . ĠTo Ġprevent Ġover fitting , Ġwe Ġalso Ġrandomly Ġset Ġ0 . 5 % Ġwords Ġin Ġboth Ġpassages Ġand Ġquestions Ġas Ġunknown Ġwords Ġduring Ġthe Ġtraining . ĠHere , Ġwe Ġuse Ġa Ġspecial Ġtoken Ġun k Ġto Ġindicate Ġa Ġword Ġwhich Ġdoesn 't Ġappear Ġin ĠGl o Ve . ĠIN LINE FORM 1 Ġin ĠE q ĠEQ REF 9 Ġis Ġset Ġto Ġ1 . </s> Results </s> We Ġwould Ġlike Ġto Ġinvestigate Ġeffectiveness Ġthe Ġproposed Ġjoint Ġmodel . ĠTo Ġdo Ġso , Ġthe Ġsame Ġshared Ġlayer / arch itect ure Ġis Ġemployed Ġin Ġthe Ġfollowing Ġvariants Ġof Ġthe Ġproposed Ġmodel : </s> The Ġresults Ġin Ġterms Ġof ĠEM Ġand ĠF 1 Ġis Ġsummarized Ġin ĠTable ĠT AB REF 20 Ġ. ĠWe Ġobserve Ġthat ĠJoint ĠSAN Ġoutper forms Ġthe ĠSAN Ġbaseline Ġwith Ġa Ġlarge Ġmargin , Ġe . g ., Ġ67 . 89 Ġvs Ġ69 . 27 Ġ(+ 1 . 38 ) Ġand Ġ70 . 68 Ġvs Ġ72 . 20 Ġ(+ 1 . 52 ) Ġin Ġterms Ġof ĠEM Ġand ĠF 1 Ġscores Ġrespectively , Ġso Ġit Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġthe Ġjoint Ġoptimization . ĠBy Ġincorporating Ġthe Ġoutput Ġinformation Ġof Ġclass ifier Ġinto ĠJoint ĠSAN , Ġit Ġob tains Ġa Ġslight Ġimprovement , Ġe . g ., Ġ72 . 2 Ġvs Ġ72 . 66 Ġ(+ 0 . 46 ) Ġin Ġterms Ġof ĠF 1 Ġscore . ĠBy Ġanalyzing Ġthe Ġresults , Ġwe Ġfound Ġthat Ġin Ġmost Ġcases Ġwhen Ġour Ġmodel Ġextract Ġan ĠNULL Ġstring Ġanswer , Ġthe Ġclass ifier Ġalso Ġpredicts Ġit Ġas Ġan Ġun answer able Ġquestion Ġwith Ġa Ġhigh Ġprobability . </s> Table ĠT AB REF 21 Ġreports Ġcomparison Ġresults Ġin Ġliterature Ġpublished Ġ. ĠOur Ġmodel Ġachieves Ġstate - of - the - art Ġon Ġdevelopment Ġdataset Ġin Ġsetting Ġwithout Ġpre - trained Ġlarge Ġlanguage Ġmodel Ġ( EL Mo ). ĠComp aring Ġwith Ġthe Ġmuch Ġcomplicated Ġmodel ĠR . M .- Reader Ġ+ ĠVer ifier , Ġwhich Ġincludes Ġseveral Ġcomponents , Ġour Ġmodel Ġstill Ġoutper forms Ġby Ġ0 . 7 Ġin Ġterms Ġof ĠF 1 Ġscore . ĠFurthermore , Ġwe Ġobserve Ġthat ĠEL Mo Ġgives Ġa Ġgreat Ġboosting Ġon Ġthe Ġperformance , Ġe . g ., Ġ2 . 8 Ġpoints Ġin Ġterms Ġof ĠF 1 Ġfor ĠDoc Q A . ĠThis Ġencourages Ġus Ġto Ġincorporate ĠEL Mo Ġinto Ġour Ġmodel Ġin Ġfuture . </s> Analysis . ĠTo Ġbetter Ġunderstand Ġour Ġmodel , Ġwe Ġanalyze Ġthe Ġaccuracy Ġof Ġthe Ġclass ifier Ġin Ġour Ġjoint Ġmodel . ĠWe Ġobtain Ġ75 . 3 Ġclassification Ġaccuracy Ġon Ġthe Ġdevelopment Ġwith Ġthe Ġthreshold Ġ0 . 5 . ĠBy Ġincreasing Ġvalue Ġof ĠIN LINE FORM 0 Ġin ĠE q ĠEQ REF 9 Ġ, Ġthe Ġclassification Ġaccuracy Ġreached Ġto Ġ76 . 8 Ġ( ĠIN LINE FORM 1 Ġ), Ġhowever Ġthe Ġfinal Ġresults Ġof Ġour Ġmodel Ġonly Ġhave Ġa Ġsmall Ġimprovement Ġ(+ 0 . 2 Ġin Ġterms Ġof ĠF 1 Ġscore ). ĠIt Ġshows Ġthat Ġit Ġis Ġimportant Ġto Ġmake Ġbalance Ġbetween Ġthese Ġtwo Ġcomponents : Ġthe Ġspan Ġdetector Ġand Ġun answer able Ġclass ifier . </s> Conclusion </s> To Ġsum Ġup , Ġwe Ġproposed Ġa Ġsimple Ġyet Ġefficient Ġmodel Ġbased Ġon ĠSAN . ĠIt Ġshowed Ġthat Ġthe Ġjoint Ġlearning Ġalgorithm Ġboosted Ġthe Ġperformance Ġon ĠS Qu AD Ġ2 . 0 . ĠWe Ġalso Ġwould Ġlike Ġto Ġincorporate ĠEL Mo Ġinto Ġour Ġmodel Ġin Ġfuture . </s> Acknowled gments </s> We Ġthank ĠY ich ong ĠXu , ĠShu oh ang ĠWang Ġand ĠShe ng ĠZhang Ġfor Ġvaluable Ġdiscussions Ġand Ġcomments . ĠWe Ġalso Ġthank ĠRobin ĠJ ia Ġfor Ġthe Ġhelp Ġon ĠS Qu AD Ġevaluations .