Unanswerable
NONE
1709.05036
Introduction </s> Many Ġmachine Ġlearning Ġmodels Ġin Ġquestion Ġanswering Ġtasks Ġoften Ġinvolve Ġmatching Ġmechanism . ĠFor Ġexample , Ġin Ġfact oid Ġquestion Ġanswering Ġsuch Ġas ĠS Qu AD ĠB IB REF 1 Ġ, Ġone Ġneeds Ġto Ġmatch Ġbetween Ġquery Ġand Ġcorpus Ġin Ġorder Ġto Ġfind Ġout Ġthe Ġmost Ġpossible Ġfragment Ġas Ġanswer . ĠIn Ġmultiple Ġchoice Ġquestion Ġanswering , Ġsuch Ġas ĠMC ĠTest ĠB IB REF 2 Ġ, Ġmatching Ġmechanism Ġcan Ġalso Ġhelp Ġmake Ġthe Ġcorrect Ġdecision . </s> The Ġeasiest Ġway Ġof Ġmatching Ġis Ġto Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġbetween Ġtwo Ġvectors . ĠIt Ġis Ġgenerally Ġdone Ġby Ġtwo Ġstep : ĠFirst , Ġencode Ġtext Ġinto Ġword Ġvectors , Ġsentence Ġvectors Ġor Ġparagraph Ġvectors . ĠSecond , Ġsimply Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġbetween Ġtarget Ġvectors . ĠThis Ġmethod Ġperforms Ġwell Ġwhen Ġapplied Ġto Ġword - level Ġmatching . ĠHowever , Ġas Ġfor Ġmatching Ġbetween Ġsentences Ġor Ġparagraphs , Ġa Ġsingle Ġvector Ġis Ġnot Ġsufficient Ġto Ġencode Ġall Ġthe Ġimportant Ġinformation . ĠIn Ġorder Ġto Ġsolve Ġthis Ġproblem , ĠWang Ġand ĠJiang Ġproposed Ġa ĠâĢ ľ comp are - agg regate âĢ Ŀ ĠB IB REF 3 Ġframework Ġthat Ġperforms Ġword - level Ġmatching Ġusing Ġmultiple Ġtechniques Ġfollowed Ġby Ġaggregation Ġwith Ġconv olution al Ġneural Ġnetwork . ĠIn Ġtheir Ġwork , Ġthey Ġshow Ġthat Ġcompare - agg regate Ġframework Ġcan Ġeffectively Ġmatch Ġtwo Ġsequences Ġthrough Ġa Ġwide Ġrange . </s> Although Ġ" comp are - agg regate " Ġmatching Ġmechanism Ġperforms Ġwell Ġon Ġmultiple Ġquestion Ġanswering Ġtasks , Ġit Ġhas Ġtwo Ġdeficiencies . ĠFirst , Ġit Ġtends Ġto Ġaggregate Ġpassively Ġthrough Ġthe Ġsequence Ġrather Ġthan Ġtake Ġthe Ġimportance Ġof Ġeach Ġelement Ġinto Ġaccount . ĠThat Ġis , Ġ" comp are Ġaggregate " Ġmodel Ġconsiders Ġall Ġthe Ġsequential Ġcontents Ġequally . ĠSecond , Ġ" comp are Ġaggregate " Ġcan Ġonly Ġtake Ġfew Ġneighboring Ġelements Ġinto Ġaccount Ġat Ġthe Ġsame Ġtime Ġbecause Ġof Ġthe Ġlimitation Ġof ĠCNN Ġkernel Ġsize . </s> In Ġthis Ġpaper , Ġwe Ġpropose ĠQuery - based ĠAttention ĠCNN Ġ( Q AC NN ) Ġto Ġdeal Ġwith Ġthe Ġdeficiencies Ġabove . ĠFirst , Ġwe Ġadd Ġquery - based Ġattention Ġmechanism Ġinto Ġoriginal Ġ" comp are Ġaggregate " Ġmodel . ĠMoreover , ĠWe Ġre - design Ġthe Ġaggregation Ġmechanism Ġin Ġ" comp are Ġaggregate " Ġto Ġa Ġtwo - st aged ĠCNN Ġarchitecture Ġwhich Ġcomprises Ġword - level Ġaggregation Ġand Ġsentence - level Ġaggregation . ĠIn Ġthis Ġway , ĠQ AC NN Ġcan Ġefficiently Ġextract Ġfeatures Ġcross Ġsentences . </s> Our Ġmodel Ġconsists Ġof Ġthree Ġcomponents : Ġ1 ) ĠThe Ġsimilarity Ġmapping Ġlayer Ġwhich Ġconverts Ġthe Ġinput Ġpassage , Ġquery Ġand Ġchoice Ġinto Ġfeature Ġrepresentation Ġand Ġperform Ġa Ġsimilarity Ġoperation Ġto Ġeach Ġother . Ġ2 ) ĠThe Ġattention - based ĠCNN Ġmatching Ġnetwork Ġcomposed Ġof Ġa Ġtwo - st aged ĠCNN Ġfocusing Ġon Ġword - level Ġand Ġsentence - level Ġmatching Ġrespectively . Ġ3 ) ĠThe Ġprediction Ġlayer Ġwhich Ġmakes Ġthe Ġfinal Ġdecision . </s> The Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare Ġthree - fold . ĠFirst , Ġwe Ġintroduce Ġa Ġtwo - st aged ĠCNN Ġarchitecture Ġwhich Ġintegrates Ġinformation Ġfrom Ġword - level Ġto Ġsentence - level , Ġand Ġthen Ġfrom Ġsentence - level Ġto Ġpassage - level . ĠSecond , Ġwe Ġintroduce Ġattention Ġmechanism Ġinto Ġthis Ġnet . ĠWe Ġuse Ġspecially Ġdesigned ĠCNN Ġstructure Ġand Ġattention Ġmechanism Ġto Ġrecognize Ġthe Ġpattern Ġof Ġsimilarity Ġmap Ġand Ġeventually Ġidentify Ġspecific Ġsynt actic Ġstructure Ġof Ġqueries . ĠBy Ġtransforming Ġpassage - query Ġfeature Ġinto Ġattention Ġmaps Ġand Ġapplying Ġit Ġto Ġpassage - choice Ġmatching Ġresult , Ġwe Ġreasonably Ġgive Ġweight Ġto Ġevery Ġword Ġin Ġthe Ġpassage . ĠLastly , Ġour Ġmodel Ġreaches Ġ79 . 99 % Ġaccuracy Ġon Ġthe ĠMovie Q A Ġdataset Ġwhich Ġyields Ġtop Ġ1 Ġresult Ġon Ġthis Ġdataset . </s> Q AC NN </s> In Ġthis Ġquestion Ġanswering Ġtask , Ġa Ġreading Ġpassage Ġ, Ġa Ġquery Ġand Ġseveral Ġanswer Ġchoices Ġare Ġgiven . ĠP Ġdenotes Ġthe Ġpassage , ĠQ Ġdenotes Ġquery Ġand ĠC Ġdenotes Ġone Ġof Ġthe Ġmultiple Ġchoices . ĠThe Ġtarget Ġof Ġthe Ġmodel Ġis Ġto Ġchoose Ġa Ġcorrect Ġanswer ĠA Ġfrom Ġmultiple Ġchoices Ġbased Ġon Ġinform ations Ġof ĠP Ġand ĠQ . </s> Fig . ĠFIG REF 1 Ġis Ġthe Ġpipeline Ġoverview Ġof ĠQ AC NN . ĠFirst , Ġwe Ġuse Ġembed ding Ġlayer Ġto Ġtransform ĠP , ĠQ , Ġand ĠC Ġinto Ġword Ġembed ding . ĠThen Ġthe Ġcompare Ġlayer Ġgenerates Ġpassage - query Ġsimilarity Ġmap ĠIN LINE FORM 0 Ġand Ġpassage - choice Ġsimilarity Ġmap ĠIN LINE FORM 1 Ġ. ĠThe Ġfollowing Ġpart Ġis Ġthe Ġmain Ġcomponent Ġof ĠQ AC NN . ĠIt Ġconsists Ġof Ġtwo - st aged ĠCNN Ġarchitecture . ĠThe Ġfirst Ġstage Ġprojects Ġword - level Ġfeature Ġinto Ġsentence - level , Ġand Ġthe Ġsecond Ġstage Ġprojects Ġsentence - level Ġfeature Ġinto Ġpassage - level . ĠMoreover , Ġwe Ġapply Ġquery - based Ġattention Ġmechanism Ġto Ġeach Ġstage Ġon Ġthe Ġbasis Ġof ĠIN LINE FORM 2 Ġfeature Ġat Ġword Ġlevel Ġand Ġsentence Ġlevel Ġrespectively . ĠAfter ĠQ AC NN ĠLayer , Ġwe Ġobtain Ġeach Ġchoice Ġanswer Ġfeature . ĠFinally , Ġa Ġprediction Ġlayer Ġcollects Ġoutput Ġinformation Ġfrom Ġevery Ġchoice Ġfeature Ġand Ġreturns Ġthe Ġmost Ġpossible Ġanswer . </s> Similar ity ĠM apping ĠLayer </s> Similar ity ĠM apping ĠLayer Ġis Ġcomposed Ġof Ġtwo Ġpart : Ġembed ding Ġlayer Ġand Ġcompare Ġlayer . ĠGiven Ġa Ġpassage ĠP Ġwith ĠIN LINE FORM 0 Ġsentences , Ġa Ġquery ĠQ , Ġand Ġa Ġchoice ĠC , Ġthe Ġembed ding Ġlayer Ġtransforms Ġevery Ġwords Ġin ĠP , ĠQ Ġand ĠC Ġinto Ġword Ġembed ding : ĠDIS PLAY FORM 0 </s> IN LINE FORM 0 Ġis Ġthe Ġlength Ġof Ġa Ġsentence Ġin Ġpassage , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġthe Ġlength Ġof Ġquery Ġand Ġlength Ġof Ġone Ġsingle Ġchoice Ġrespectively . ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġare Ġword Ġembed d ings . ĠWord Ġembed ding Ġcan Ġbe Ġobtained Ġby Ġany Ġtype Ġof Ġembed ding Ġtechnique , Ġsuch Ġas Ġrecurrent Ġneural Ġnetwork ĠB IB REF 4 Ġ, ĠSequence - to - sequence Ġmodel ĠB IB REF 5 Ġ, ĠWord 2 vec ĠB IB REF 6 Ġ, Ġetc . ĠIn Ġour Ġwork , Ġwe Ġsimply Ġuse Ġpre - trained ĠGl o Ve Ġword Ġvectors ĠB IB REF 7 Ġas Ġthe Ġembed ding Ġwithout Ġany Ġfurther Ġmodification Ġor Ġtraining . </s> After Ġword Ġembed ding Ġstep , ĠWe Ġwant Ġto Ġacquire Ġsimilarity Ġmap Ġwhich Ġtells Ġus Ġlocation Ġrelationship Ġbetween Ġpassage Ġand Ġquery , Ġpassage Ġand Ġchoices . ĠWe Ġuse Ġcompare Ġlayer Ġto Ġcompare Ġeach Ġpassage Ġsentence ĠIN LINE FORM 0 Ġto ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġat Ġword Ġlevel Ġseparately Ġas ĠFig . ĠFIG REF 7 Ġand ĠFig . ĠFIG REF 8 Ġshow . ĠDIS PLAY FORM 0 </s> That Ġis , Ġwe Ġcompare Ġeach Ġword Ġin Ġsentences Ġof Ġpassage Ġto Ġeach Ġword Ġin Ġquery Ġand Ġchoice . ĠWe Ġuse Ġcos ine Ġsimilarity Ġas Ġthe Ġcomparing Ġmethod Ġhere . ĠThis Ġstep Ġcreates Ġtwo Ġsimilarity Ġmap , Ġpassage - query Ġsimilarity Ġmap ĠIN LINE FORM 0 Ġand Ġpassage - choice Ġsimilarity Ġmap ĠIN LINE FORM 1 Ġ. </s> Q AC NN ĠLayer </s> We Ġpropose Ġan Ġattention Ġconv olution al Ġmatching Ġlayer Ġto Ġintegrate Ġtwo Ġsimilarity Ġmaps Ġgiven Ġabove . ĠThat Ġis , ĠQ AC NN ĠLayer Ġis Ġused Ġto Ġlearn Ġthe Ġlocation Ġrelationship Ġpattern . ĠIt Ġcontains Ġa Ġtwo - st aged ĠCNN Ġcombined Ġwith Ġquery - based Ġattention Ġmechanism . ĠEach Ġstage Ġcomprises Ġtwo Ġmajor Ġpart : Ġattention Ġmap Ġand Ġoutput Ġrepresentation . </s> Fig . ĠFIG REF 10 Ġshows Ġthe Ġarchitecture Ġof Ġthe Ġattention Ġmap Ġin Ġfirst Ġstage ĠCNN . ĠWe Ġchoose ĠIN LINE FORM 0 Ġsentence Ġslice ĠIN LINE FORM 1 Ġin ĠIN LINE FORM 2 Ġ, Ġand Ġapply ĠCNN Ġto Ġit Ġusing Ġthe Ġconv olution Ġkernel ĠIN LINE FORM 3 Ġ, Ġwhere Ġsupers cript ĠIN LINE FORM 4 Ġdenotes Ġattention Ġmap , Ġsubscript ĠIN LINE FORM 5 Ġdenotes Ġthe Ġfirst Ġstage ĠCNN . ĠSymbol ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġrepresent Ġwidth Ġof Ġkernel Ġand Ġnumber Ġof Ġkernel Ġrespectively . ĠThe Ġgenerated Ġfeature ĠIN LINE FORM 8 Ġis Ġas Ġfollow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġbias . ĠWith ĠIN LINE FORM 1 Ġcovering Ġwhole Ġquery Ġand Ġseveral Ġwords Ġin Ġthe Ġpassage , Ġconv olution Ġkernels Ġwould Ġlearn Ġthe Ġquery Ġsynt actic Ġstructure Ġand Ġgive Ġweight Ġto Ġeach Ġpassage 's Ġlocation . ĠThat 's Ġwhy Ġwe Ġuse Ġs igm oid Ġfunction Ġas Ġactivation Ġfunction Ġin Ġthis Ġstage . ĠFurthermore , Ġwe Ġperform Ġmax pool ing Ġto ĠIN LINE FORM 2 Ġperpend icularly Ġin Ġorder Ġto Ġfind Ġthe Ġlargest Ġweight Ġbetween Ġdifferent Ġkernels Ġin Ġthe Ġsame Ġlocation , Ġusing Ġmax pool Ġkernel Ġshaped ĠIN LINE FORM 3 Ġ, Ġand Ġthen Ġgenerate Ġword - level Ġattention Ġmap ĠIN LINE FORM 4 Ġfor Ġeach Ġsentence . </s> In Ġthis Ġstage , Ġwe Ġwant Ġto Ġacquire Ġpassage 's Ġsentence Ġfeatures Ġbased Ġon Ġthe Ġquery Ġand Ġthe Ġchoice Ġrespectively . ĠWe Ġapply ĠCNN Ġto ĠIN LINE FORM 0 Ġto Ġaggregate Ġpattern Ġof Ġlocation Ġrelationship Ġand Ġacquire Ġchoice - based Ġsentence Ġfeatures . ĠAlso , Ġwe Ġapply ĠCNN Ġto ĠIN LINE FORM 1 Ġto Ġacquire Ġquery - based Ġsentence Ġfeatures . ĠCNN Ġarchitecture Ġof Ġoutput Ġrepresentation Ġpart ĠFig . ĠFIG REF 11 Ġis Ġsimilar Ġto Ġwhich Ġof Ġattention Ġmap Ġpart , Ġbut Ġwe Ġuse Ġdifferent Ġkernels ĠIN LINE FORM 2 Ġand Ġdifferent Ġbias ĠIN LINE FORM 3 Ġ: ĠDIS PLAY FORM 0 </s> where Ġthe Ġsupers cript ĠIN LINE FORM 0 Ġdenotes Ġoutput Ġrepresentation . ĠWe Ġapply ĠIN LINE FORM 3 Ġon ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġthen Ġfinally Ġgenerate ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġusing Ġeq . 4 . ĠWe Ġthen Ġmultiply ĠIN LINE FORM 8 Ġby Ġthe Ġword - level Ġattention Ġmap ĠIN LINE FORM 9 Ġwhich Ġcomes Ġfrom Ġstage Ġ2 . 2 . 1 Ġelement - wise Ġthrough Ġthe Ġfirst Ġdimension . ĠAt Ġlast , Ġwe Ġmax pool ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġhorizontally Ġwith Ġkernel Ġshape ĠIN LINE FORM 12 Ġto Ġget Ġthe Ġquery - based Ġsentence Ġfeatures ĠIN LINE FORM 13 Ġand Ġchoice - based Ġsentence Ġfeatures ĠIN LINE FORM 14 Ġ. </s> Fig . ĠFIG REF 28 Ġis Ġthe Ġarchitecture Ġof Ġattention Ġmap Ġin Ġthe Ġsecond Ġstage ĠCNN . ĠBased Ġon Ġfirst Ġstage Ġquery - based Ġsentence Ġfeatures Ġfrom Ġsection ĠUID 15 Ġ, Ġwe Ġwant Ġto Ġacquire Ġsentence - level Ġattention Ġmap . ĠThe Ġinput Ġof Ġthis Ġstage Ġis ĠIN LINE FORM 0 Ġ, Ġwhich Ġwill Ġbe Ġfurther Ġrefined Ġby ĠCNN Ġwith Ġkernel ĠIN LINE FORM 1 Ġand Ġgenerates Ġintermediate Ġfeatures ĠIN LINE FORM 2 Ġ. ĠDIS PLAY FORM 0 </s> Then , Ġsame Ġas Ġattention Ġmap Ġof Ġfirst Ġstage , Ġwe Ġmax pool ĠIN LINE FORM 0 Ġwith Ġkernel Ġshaped ĠIN LINE FORM 1 Ġ, Ġand Ġobtain Ġsentence - level Ġattention Ġmap ĠIN LINE FORM 2 Ġ. </s> Output Ġrepresentation Ġpart Ġof Ġthe Ġsecond Ġstage Ġin ĠFig . ĠFIG REF 29 Ġhas Ġtwo Ġinput , Ġsentence - level Ġattention Ġmap ĠIN LINE FORM 0 Ġand Ġsentence - level Ġfeatures ĠIN LINE FORM 1 Ġ. ĠThe Ġequations Ġhere Ġare Ġsimilar Ġto Ġthose Ġpreviously Ġmentioned . ĠAs Ġfollow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ. ĠOutput Ġrepresentation Ġof Ġcertain Ġchoice ĠIN LINE FORM 3 Ġis Ġthe Ġfinal Ġoutput Ġof ĠQ AC NN ĠLayer . </s> Pred iction ĠLayer </s> Pred iction ĠLayer Ġis Ġthe Ġfinal Ġpart Ġof ĠQ AC NN . ĠWe Ġuse ĠIN LINE FORM 0 Ġto Ġrepresent Ġthe Ġfinal Ġoutput Ġrepresentation Ġof Ġthe ĠIN LINE FORM 1 Ġchoice Ġ. ĠIn Ġorder Ġto Ġfind Ġout Ġthe Ġmost Ġcorrect Ġchoice , Ġwe Ġsimply Ġpass ĠIN LINE FORM 2 Ġto Ġtwo Ġfully - connected Ġlayer Ġand Ġcompute Ġprobability Ġfor Ġeach Ġchoice Ġusing Ġsoft max Ġas Ġfollows : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġ. </s> Im plementation Ġdetails </s> In Ġthe Ġpre processing Ġstep , Ġwe Ġused Ġpre - trained ĠGl o Ve Ġvectors Ġfor Ġword Ġembed d ings , Ġand Ġthey Ġwould Ġnot Ġbe Ġupdated Ġduring Ġtraining ; ĠWe Ġpadded Ġsentence Ġnumber Ġin Ġeach Ġpassage Ġto Ġ101 , Ġall Ġword Ġnumber Ġin Ġeach Ġsentence Ġto Ġ100 . ĠWord Ġnumber Ġof Ġqueries Ġand Ġchoices Ġwere Ġpadded Ġto Ġ50 . ĠFor Ġall Ġkernels Ġof ĠCNN , ĠIN LINE FORM 0 Ġ, Ġeach Ġof Ġwhich Ġhas Ġthree Ġdifferent Ġkernel Ġwidth ĠIN LINE FORM 1 Ġ; Ġeach Ġof Ġthem Ġhas Ġsame Ġkernel Ġnumber ĠIN LINE FORM 2 Ġ. ĠWe Ġutilized Ġdrop out Ġin Ġeach ĠCNN Ġlayer Ġwith Ġdrop out Ġrate Ġ0 . 8 . ĠWe Ġused ĠAdam ĠB IB REF 8 Ġoptim izer Ġto Ġoptimize Ġour Ġmodel Ġwith Ġinitial Ġlearning Ġrate Ġ0 . 001 . </s> Exper imental ĠResult </s> We Ġmainly Ġfocus Ġon Ġthe ĠMovie Q A Ġdataset Ġto Ġtrain Ġand Ġevaluate Ġour Ġmodel . ĠMovie Q A Ġdataset Ġaims Ġto Ġevaluate Ġautomatic Ġstory Ġcomprehension Ġfrom Ġboth Ġvideo Ġand Ġtext . ĠThe Ġdata Ġset Ġconsists Ġof Ġalmost Ġ15 , 000 Ġmultiple Ġchoice Ġquestion Ġanswers . ĠD iverse Ġinformation Ġin Ġthis Ġdataset Ġlike Ġplots , Ġscripts , Ġsub - title Ġand Ġvideo Ġcapt ions Ġcan Ġbe Ġused Ġto Ġinfer Ġanswers . ĠIn Ġour Ġtask , Ġonly Ġplot Ġinform ations Ġare Ġused . ĠThis Ġchallenging Ġdataset Ġis Ġsuitable Ġto Ġevaluate ĠQ AC NN Ġbecause Ġmovie Ġplots Ġare Ġlonger Ġthan Ġnormal Ġreading Ġcomprehension Ġtask . ĠEach Ġquestion Ġcomes Ġwith Ġa Ġset Ġof Ġfive Ġhighly Ġplausible Ġchoices , Ġonly Ġone Ġof Ġwhich Ġis Ġcorrect ; ĠIn Ġthe ĠMovie Q A Ġbenchmark , Ġthere Ġare Ġ1958 ĠQ A Ġpairs Ġin Ġthe Ġval Ġset Ġand Ġ3 138 ĠQ A Ġpairs Ġin Ġthe Ġtest Ġset . </s> We Ġused Ġensemble Ġmodel Ġin Ġthis Ġdataset . ĠThe Ġensemble Ġmodel Ġconsists Ġof Ġeight Ġtraining Ġruns Ġmodels Ġwith Ġidentical Ġstructure Ġand Ġhyper - param eter . ĠIn Ġthe Ġval Ġset , Ġwe Ġachieve Ġ77 . 6 % Ġaccuracy Ġwith Ġsingle Ġmodel Ġand Ġ79 . 0 % Ġaccuracy Ġwith Ġensemble Ġmodel . ĠIn Ġthe Ġtest Ġset , Ġas Ġthe ĠTable Ġ1 . Ġshows , Ġour Ġmodel Ġachieves Ġ79 . 99 Ġ% Ġaccuracy Ġwith Ġensemble Ġmodel Ġand Ġis Ġthe Ġstate Ġof Ġthe Ġart . </s> We Ġalso Ġapplied Ġour Ġmodel Ġto ĠM CT est Ġdataset Ġwhich Ġrequires Ġmachines Ġto Ġanswer Ġmultiple - choice Ġreading Ġcomprehension Ġquestions Ġabout Ġfictional Ġstories . ĠThe Ġoriginal Ġpaper Ġdescribes Ġthat Ġa Ġbaseline Ġmethod Ġuses Ġa Ġcombination Ġof Ġa Ġsliding Ġwindow Ġscore Ġand Ġa Ġdistance Ġbased Ġ. ĠThey Ġachieve Ġ66 . 7 % Ġand Ġ56 . 7 % Ġon ĠMC 500 Ġand ĠMC 160 Ġseparately . ĠBecause Ġof Ġthe Ġrestricted Ġtraining Ġset Ġand Ġdevelopment Ġset , Ġwe Ġtrained Ġour Ġmodel Ġon ĠMovie Q A Ġtraining Ġset Ġand Ġapplied Ġthe Ġresult Ġto Ġtest ĠM CT est Ġdataset . ĠOn ĠM CT est Ġdataset , Ġwe Ġstill Ġoutper form Ġbaseline Ġand Ġachieve Ġ68 . 1 % Ġaccuracy Ġon ĠMC 160 Ġand Ġ61 . 5 % Ġaccuracy Ġon ĠMC 500 . </s> Discussion </s> Q AC NN Ġis Ġa Ġpowerful Ġnetwork Ġfocusing Ġon Ġmultiple Ġchoice ĠQ A Ġtask . ĠIt Ġmatches Ġbetween Ġpassage Ġand Ġchoices Ġbased Ġon Ġquery Ġinformation . ĠOne Ġof Ġthe Ġmost Ġimportant Ġidea Ġin ĠQ AC NN Ġis Ġtwo - st aged Ġattention Ġmap . ĠThe Ġfirst Ġattention Ġmap Ġis Ġat Ġword Ġlevel , Ġrepresenting Ġthe Ġimportance Ġof Ġeach Ġword Ġin Ġparagraph Ġto Ġa Ġcertain Ġquestion ; Ġthe Ġsecond Ġattention Ġmap , Ġhowever Ġis Ġat Ġsentence Ġlevel , Ġrepresenting Ġthe Ġimportance Ġof Ġeach Ġsentence Ġin Ġparagraph Ġto Ġa Ġcertain Ġquestion . </s> In Ġthis Ġsection , Ġwe Ġdesigned Ġseveral Ġexperiments Ġto Ġtest Ġhow Ġtwo - stage Ġmechanism Ġand Ġattention Ġmaps Ġimpact Ġon Ġour Ġmodel . </s> In Ġthis Ġexperiment , we Ġfocused Ġon Ġthe Ġdifference Ġbetween Ġone - stage ĠQ AC NN Ġand Ġtwo - stage ĠQ AC NN . ĠFor Ġone - stage ĠQ AC NN , Ġwe Ġdidn 't Ġsplit Ġan Ġentire Ġpassage Ġinto Ġsentences . ĠThat Ġis , Ġthe Ġshape Ġof Ġpassage - query Ġsimilarity Ġmap ĠIN LINE FORM 0 Ġand Ġpassage - choice Ġsimilarity Ġmap ĠIN LINE FORM 1 Ġare Ġ2 D Ġrather Ġthan Ġ3 D . ĠWe Ġconv olved Ġthem Ġdirectly Ġon Ġword - level Ġand Ġoutput Ġpassage Ġfeature Ġwithout Ġsecond - stage Ġinvolved . ĠThe Ġresult Ġis Ġshown Ġon Ġtable ĠT AB REF 36 Ġ. ĠThe Ġresult Ġshows Ġthat Ġthe Ġmodified Ġone Ġstaged ĠQ AC NN Ġreaches Ġ66 . 8 % Ġaccuracy Ġon Ġvalidation Ġset , Ġwhich Ġis Ġten Ġpercent Ġlower Ġthan Ġ78 . 1 %, Ġthe Ġoriginal ĠQ AC NN Ġaccuracy Ġon Ġvalidation Ġset . </s> In Ġthis Ġexperiment , Ġour Ġtarget Ġis Ġto Ġvalidate Ġthe Ġeffect Ġof Ġquery - based Ġattention Ġin ĠQ AC NN . ĠWe Ġmodified Ġthree Ġdifferent Ġstructures Ġfrom Ġoriginal ĠQ AC NN ĠLayer Ġbelow : </s> 1 ) ĠFor Ġthe Ġfirst Ġone , Ġwe Ġmodified ĠQ AC NN ĠLayer Ġin Ġsection ĠSEC REF 12 Ġand Ġremoved Ġboth Ġsentence - level Ġattention Ġmap Ġand Ġword - level Ġattention Ġmap Ġpart Ġfrom Ġit . ĠHowever , Ġthis Ġmodified Ġmodel Ġwould Ġhave Ġa Ġdeficiency Ġof Ġquery Ġinformation . ĠTherefore , Ġwe Ġconc aten ated Ġthe Ġfinal Ġoutput Ġrepresentation Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġtogether Ġbefore Ġprediction Ġlayer . ĠThe Ġexperiment Ġresult Ġis Ġshown Ġon Ġthe ĠTable ĠT AB REF 36 Ġ. ĠThe Ġresult Ġis Ġalmost Ġten Ġpercent Ġless Ġthan Ġthe Ġoriginal Ġone . Ġ2 ) ĠFor Ġthe Ġsecond Ġone , Ġwe Ġonly Ġremoved Ġsentence - level Ġattention Ġin Ġsection ĠUID 18 Ġfrom ĠQ AC NN ĠLayer Ġand Ġkept Ġword - level Ġattention Ġin Ġthe Ġmodel . Ġ3 ) ĠFor Ġthe Ġlast Ġone , Ġinstead Ġof Ġremoving Ġsentence - level Ġattention , Ġwe Ġremoved Ġword - level Ġattention Ġfrom ĠQ AC NN ĠLayer . </s> The Ġresult Ġis Ġshown Ġin ĠTable ĠT AB REF 36 Ġ. ĠWe Ġcan Ġsee Ġthat ĠQ AC NN ( with Ġonly Ġword - level Ġattention ) Ġperforms Ġbetter Ġthan ĠQ AC NN ( without Ġattention ); ĠQ AC NN ( with Ġonly Ġsentence - level Ġattention ) Ġperforms Ġbetter Ġthan ĠQ AC NN ( with Ġonly Ġword - level Ġattention ); ĠAnd Ġoriginal ĠQ AC NN Ġwhich Ġcontains Ġboth Ġword - level Ġand Ġsentence - level Ġattention Ġdoes Ġthe Ġbest Ġjob Ġamong Ġall . ĠThus , Ġnot Ġonly Ġword - level Ġattention Ġbut Ġalso Ġsentence - level Ġattention Ġcan Ġcontribute Ġto Ġthe Ġperformance Ġof ĠQ AC NN . ĠHowever , Ġsentence - level Ġattention Ġseems Ġto Ġplay Ġa Ġmore Ġimportant Ġrole . </s> Figure . ĠFIG REF 31 Ġis Ġthe Ġvisualization Ġof Ġtwo Ġattention Ġmaps Ġand Ġtheir Ġcorresponding Ġquestion . ĠThe Ġupper Ġhalf Ġof ĠFigure . ĠFIG REF 31 Ġis Ġthe Ġattention Ġmap Ġat Ġsentence Ġlevel . ĠWe Ġpicked Ġthe Ġsentence Ġwith Ġthe Ġlargest Ġattention Ġvalue Ġas Ġtarget Ġsentence Ġand Ġexamine Ġit . ĠThus , Ġwe Ġcould Ġget Ġthe Ġlower Ġhalf Ġof ĠFigure . ĠFIG REF 31 Ġ, Ġwhich Ġshows Ġthe Ġattention Ġmap Ġat Ġword Ġlevel Ġon Ġthe Ġtarget Ġsentence . ĠWe Ġused Ġa Ġquestion Ġfrom Ġmovie , ĠHarry ĠPotter , Ġas Ġan Ġexample . ĠThe Ġresult Ġshows Ġthat Ġthe Ġsentence Ġwith Ġthe Ġlargest Ġattention Ġvalue Ġis Ġexactly Ġwhere Ġthe Ġcorrect Ġanswer Ġcomes Ġfrom . ĠIt Ġturns Ġout Ġthat Ġsentence - level Ġattention Ġmap Ġcan Ġsuccessfully Ġfind Ġout Ġwhich Ġsentence Ġcontains Ġthe Ġinformation Ġof Ġcorrect Ġanswer . ĠAs Ġfor Ġword - level Ġattention Ġmap , Ġwe Ġcan Ġeasily Ġsee Ġthat Ġthe Ġattention Ġmap Ġfocus Ġmainly Ġon Ġthe Ġend Ġof Ġtarget Ġsentence , Ġwhich Ġis Ġobviously Ġmore Ġimportant Ġfor Ġanswering Ġthis Ġquestion . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġan Ġefficient Ġmatching Ġmechanism Ġon Ġmultiple Ġchoice Ġquestion Ġanswering Ġtask . ĠWe Ġintroduce Ġtwo - st aged ĠCNN Ġto Ġmatch Ġpassage Ġand Ġchoice Ġon Ġword Ġlevel Ġand Ġsentence Ġlevel . ĠIn Ġaddition , Ġwe Ġuse Ġquery - based ĠCNN Ġattention Ġto Ġenhance Ġmatching Ġeffect . </s> The Ġpower Ġof Ġthe Ġmodel Ġis Ġverified Ġon ĠMovie Q A Ġdataset , Ġwhich Ġyielded Ġthe Ġstate Ġof Ġthe Ġart Ġresult Ġon Ġthe Ġdataset . ĠIn Ġthe Ġfuture , Ġwe Ġare Ġnow Ġworking Ġon Ġtraining Ġour Ġmodel Ġbased Ġon Ġour Ġown Ġtrained Ġembed ding Ġwith ĠTF - ID F ĠB IB REF 9 Ġweight ing . ĠFurthermore , Ġwe Ġwould Ġlike Ġto Ġtest Ġour Ġmodel Ġon Ġopen - answer Ġtask Ġlike ĠS Qu a D Ġby Ġseeing Ġthe Ġwhole Ġcorpus Ġas Ġan ĠâĢ ľ answer Ġpool " Ġand Ġsolve Ġit Ġlike Ġmultiple Ġchoice Ġquestion .
Introduction Many machine learning models in question answering tasks often involve matching mechanism. For example, in factoid question answering such as SQuAD BIBREF1 , one needs to match between query and corpus in order to find out the most possible fragment as answer. In multiple choice question answering, such as MC Test BIBREF2 , matching mechanism can also help make the correct decision. The easiest way of matching is to calculate the cosine similarity between two vectors. It is generally done by two step: First, encode text into word vectors, sentence vectors or paragraph vectors. Second, simply calculate the cosine similarity between target vectors. This method performs well when applied to word-level matching. However, as for matching between sentences or paragraphs, a single vector is not sufficient to encode all the important information. In order to solve this problem, Wang and Jiang proposed a “compare-aggregate” BIBREF3 framework that performs word-level matching using multiple techniques followed by aggregation with convolutional neural network. In their work, they show that compare-aggregate framework can effectively match two sequences through a wide range. Although "compare-aggregate" matching mechanism performs well on multiple question answering tasks, it has two deficiencies. First, it tends to aggregate passively through the sequence rather than take the importance of each element into account. That is, "compare aggregate" model considers all the sequential contents equally. Second, "compare aggregate" can only take few neighboring elements into account at the same time because of the limitation of CNN kernel size. In this paper, we propose Query-based Attention CNN (QACNN) to deal with the deficiencies above. First, we add query-based attention mechanism into original "compare aggregate" model. Moreover, We re-design the aggregation mechanism in "compare aggregate" to a two-staged CNN architecture which comprises word-level aggregation and sentence-level aggregation. In this way, QACNN can efficiently extract features cross sentences. Our model consists of three components: 1) The similarity mapping layer which converts the input passage, query and choice into feature representation and perform a similarity operation to each other. 2) The attention-based CNN matching network composed of a two-staged CNN focusing on word-level and sentence-level matching respectively. 3) The prediction layer which makes the final decision. The main contributions of this work are three-fold. First, we introduce a two-staged CNN architecture which integrates information from word-level to sentence-level, and then from sentence-level to passage-level. Second, we introduce attention mechanism into this net. We use specially designed CNN structure and attention mechanism to recognize the pattern of similarity map and eventually identify specific syntactic structure of queries. By transforming passage-query feature into attention maps and applying it to passage-choice matching result, we reasonably give weight to every word in the passage. Lastly, our model reaches 79.99% accuracy on the MovieQA dataset which yields top 1 result on this dataset. QACNN In this question answering task, a reading passage , a query and several answer choices are given. P denotes the passage, Q denotes query and C denotes one of the multiple choices. The target of the model is to choose a correct answer A from multiple choices based on informations of P and Q. Fig. FIGREF1 is the pipeline overview of QACNN. First, we use embedding layer to transform P, Q, and C into word embedding. Then the compare layer generates passage-query similarity map INLINEFORM0 and passage-choice similarity map INLINEFORM1 . The following part is the main component of QACNN. It consists of two-staged CNN architecture. The first stage projects word-level feature into sentence-level, and the second stage projects sentence-level feature into passage-level. Moreover, we apply query-based attention mechanism to each stage on the basis of INLINEFORM2 feature at word level and sentence level respectively. After QACNN Layer, we obtain each choice answer feature. Finally, a prediction layer collects output information from every choice feature and returns the most possible answer. Similarity Mapping Layer Similarity Mapping Layer is composed of two part: embedding layer and compare layer. Given a passage P with INLINEFORM0 sentences, a query Q, and a choice C, the embedding layer transforms every words in P, Q and C into word embedding: DISPLAYFORM0 INLINEFORM0 is the length of a sentence in passage, and INLINEFORM1 and INLINEFORM2 are the length of query and length of one single choice respectively. INLINEFORM5 , INLINEFORM6 and INLINEFORM7 are word embeddings. Word embedding can be obtained by any type of embedding technique, such as recurrent neural network BIBREF4 , Sequence-to-sequence model BIBREF5 , Word2vec BIBREF6 , etc. In our work, we simply use pre-trained GloVe word vectors BIBREF7 as the embedding without any further modification or training. After word embedding step, We want to acquire similarity map which tells us location relationship between passage and query, passage and choices. We use compare layer to compare each passage sentence INLINEFORM0 to INLINEFORM1 and INLINEFORM2 at word level separately as Fig. FIGREF7 and Fig. FIGREF8 show. DISPLAYFORM0 That is, we compare each word in sentences of passage to each word in query and choice. We use cosine similarity as the comparing method here. This step creates two similarity map, passage-query similarity map INLINEFORM0 and passage-choice similarity map INLINEFORM1 . QACNN Layer We propose an attention convolutional matching layer to integrate two similarity maps given above. That is, QACNN Layer is used to learn the location relationship pattern. It contains a two-staged CNN combined with query-based attention mechanism. Each stage comprises two major part: attention map and output representation. Fig. FIGREF10 shows the architecture of the attention map in first stage CNN. We choose INLINEFORM0 sentence slice INLINEFORM1 in INLINEFORM2 , and apply CNN to it using the convolution kernel INLINEFORM3 , where superscript INLINEFORM4 denotes attention map, subscript INLINEFORM5 denotes the first stage CNN. Symbol INLINEFORM6 and INLINEFORM7 represent width of kernel and number of kernel respectively. The generated feature INLINEFORM8 is as follow: DISPLAYFORM0 where INLINEFORM0 is the bias. With INLINEFORM1 covering whole query and several words in the passage, convolution kernels would learn the query syntactic structure and give weight to each passage's location. That's why we use sigmoid function as activation function in this stage. Furthermore, we perform maxpooling to INLINEFORM2 perpendicularly in order to find the largest weight between different kernels in the same location, using maxpool kernel shaped INLINEFORM3 , and then generate word-level attention map INLINEFORM4 for each sentence. In this stage, we want to acquire passage's sentence features based on the query and the choice respectively. We apply CNN to INLINEFORM0 to aggregate pattern of location relationship and acquire choice-based sentence features. Also, we apply CNN to INLINEFORM1 to acquire query-based sentence features. CNN architecture of output representation part Fig. FIGREF11 is similar to which of attention map part, but we use different kernels INLINEFORM2 and different bias INLINEFORM3 : DISPLAYFORM0 where the superscript INLINEFORM0 denotes output representation. We apply INLINEFORM3 on INLINEFORM4 and INLINEFORM5 then finally generate INLINEFORM6 and INLINEFORM7 using eq.4. We then multiply INLINEFORM8 by the word-level attention map INLINEFORM9 which comes from stage 2.2.1 element-wise through the first dimension. At last, we maxpool INLINEFORM10 and INLINEFORM11 horizontally with kernel shape INLINEFORM12 to get the query-based sentence features INLINEFORM13 and choice-based sentence features INLINEFORM14 . Fig. FIGREF28 is the architecture of attention map in the second stage CNN. Based on first stage query-based sentence features from section UID15 , we want to acquire sentence-level attention map. The input of this stage is INLINEFORM0 , which will be further refined by CNN with kernel INLINEFORM1 and generates intermediate features INLINEFORM2 . DISPLAYFORM0 Then, same as attention map of first stage, we maxpool INLINEFORM0 with kernel shaped INLINEFORM1 , and obtain sentence-level attention map INLINEFORM2 . Output representation part of the second stage in Fig. FIGREF29 has two input, sentence-level attention map INLINEFORM0 and sentence-level features INLINEFORM1 . The equations here are similar to those previously mentioned. As follow: DISPLAYFORM0 where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 . Output representation of certain choice INLINEFORM3 is the final output of QACNN Layer. Prediction Layer Prediction Layer is the final part of QACNN. We use INLINEFORM0 to represent the final output representation of the INLINEFORM1 choice . In order to find out the most correct choice, we simply pass INLINEFORM2 to two fully-connected layer and compute probability for each choice using softmax as follows: DISPLAYFORM0 DISPLAYFORM1 where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 . Implementation details In the preprocessing step, we used pre-trained GloVe vectors for word embeddings, and they would not be updated during training; We padded sentence number in each passage to 101, all word number in each sentence to 100. Word number of queries and choices were padded to 50. For all kernels of CNN, INLINEFORM0 , each of which has three different kernel width INLINEFORM1 ; each of them has same kernel number INLINEFORM2 . We utilized dropout in each CNN layer with dropout rate 0.8. We used Adam BIBREF8 optimizer to optimize our model with initial learning rate 0.001. Experimental Result We mainly focus on the MovieQA dataset to train and evaluate our model. MovieQA dataset aims to evaluate automatic story comprehension from both video and text. The data set consists of almost 15,000 multiple choice question answers. Diverse information in this dataset like plots, scripts, sub-title and video captions can be used to infer answers. In our task, only plot informations are used. This challenging dataset is suitable to evaluate QACNN because movie plots are longer than normal reading comprehension task. Each question comes with a set of five highly plausible choices, only one of which is correct; In the MovieQA benchmark, there are 1958 QA pairs in the val set and 3138 QA pairs in the test set. We used ensemble model in this dataset. The ensemble model consists of eight training runs models with identical structure and hyper-parameter. In the val set, we achieve 77.6% accuracy with single model and 79.0% accuracy with ensemble model. In the test set, as the Table 1. shows, our model achieves 79.99 % accuracy with ensemble model and is the state of the art. We also applied our model to MCTest dataset which requires machines to answer multiple-choice reading comprehension questions about fictional stories. The original paper describes that a baseline method uses a combination of a sliding window score and a distance based . They achieve 66.7% and 56.7% on MC500 and MC160 separately. Because of the restricted training set and development set, we trained our model on MovieQA training set and applied the result to test MCTest dataset. On MCTest dataset, we still outperform baseline and achieve 68.1% accuracy on MC160 and 61.5% accuracy on MC500. Discussion QACNN is a powerful network focusing on multiple choice QA task. It matches between passage and choices based on query information. One of the most important idea in QACNN is two-staged attention map. The first attention map is at word level, representing the importance of each word in paragraph to a certain question; the second attention map, however is at sentence level, representing the importance of each sentence in paragraph to a certain question. In this section, we designed several experiments to test how two-stage mechanism and attention maps impact on our model. In this experiment,we focused on the difference between one-stage QACNN and two-stage QACNN. For one-stage QACNN, we didn't split an entire passage into sentences. That is, the shape of passage-query similarity map INLINEFORM0 and passage-choice similarity map INLINEFORM1 are 2D rather than 3D. We convolved them directly on word-level and output passage feature without second-stage involved. The result is shown on table TABREF36 . The result shows that the modified one staged QACNN reaches 66.8% accuracy on validation set, which is ten percent lower than 78.1%, the original QACNN accuracy on validation set. In this experiment, our target is to validate the effect of query-based attention in QACNN. We modified three different structures from original QACNN Layer below: 1) For the first one, we modified QACNN Layer in section SECREF12 and removed both sentence-level attention map and word-level attention map part from it. However, this modified model would have a deficiency of query information. Therefore, we concatenated the final output representation of INLINEFORM0 and INLINEFORM1 together before prediction layer. The experiment result is shown on the Table TABREF36 . The result is almost ten percent less than the original one. 2) For the second one, we only removed sentence-level attention in section UID18 from QACNN Layer and kept word-level attention in the model. 3) For the last one, instead of removing sentence-level attention, we removed word-level attention from QACNN Layer. The result is shown in Table TABREF36 . We can see that QACNN(with only word-level attention) performs better than QACNN(without attention); QACNN(with only sentence-level attention) performs better than QACNN(with only word-level attention); And original QACNN which contains both word-level and sentence-level attention does the best job among all. Thus, not only word-level attention but also sentence-level attention can contribute to the performance of QACNN. However, sentence-level attention seems to play a more important role. Figure. FIGREF31 is the visualization of two attention maps and their corresponding question. The upper half of Figure. FIGREF31 is the attention map at sentence level. We picked the sentence with the largest attention value as target sentence and examine it. Thus, we could get the lower half of Figure. FIGREF31 , which shows the attention map at word level on the target sentence. We used a question from movie, Harry Potter, as an example. The result shows that the sentence with the largest attention value is exactly where the correct answer comes from. It turns out that sentence-level attention map can successfully find out which sentence contains the information of correct answer. As for word-level attention map, we can easily see that the attention map focus mainly on the end of target sentence, which is obviously more important for answering this question. Conclusion In this paper, we present an efficient matching mechanism on multiple choice question answering task. We introduce two-staged CNN to match passage and choice on word level and sentence level. In addition, we use query-based CNN attention to enhance matching effect. The power of the model is verified on MovieQA dataset, which yielded the state of the art result on the dataset. In the future, we are now working on training our model based on our own trained embedding with TF-IDF BIBREF9 weighting. Furthermore, we would like to test our model on open-answer task like SQuaD by seeing the whole corpus as an “answer pool" and solve it like multiple choice question.
How do they split text to obtain sentence levels?
0101ebfbaba75fd47868ad0c796ac44ebc19c566
How Ġdo Ġthey Ġsplit Ġtext Ġto Ġobtain Ġsentence Ġlevels ?
Un answer able
<s> How Ġdo Ġthey Ġsplit Ġtext Ġto Ġobtain Ġsentence Ġlevels ? </s> Introduction </s> Many Ġmachine Ġlearning Ġmodels Ġin Ġquestion Ġanswering Ġtasks Ġoften Ġinvolve Ġmatching Ġmechanism . ĠFor Ġexample , Ġin Ġfact oid Ġquestion Ġanswering Ġsuch Ġas ĠS Qu AD ĠB IB REF 1 Ġ, Ġone Ġneeds Ġto Ġmatch Ġbetween Ġquery Ġand Ġcorpus Ġin Ġorder Ġto Ġfind Ġout Ġthe Ġmost Ġpossible Ġfragment Ġas Ġanswer . ĠIn Ġmultiple Ġchoice Ġquestion Ġanswering , Ġsuch Ġas ĠMC ĠTest ĠB IB REF 2 Ġ, Ġmatching Ġmechanism Ġcan Ġalso Ġhelp Ġmake Ġthe Ġcorrect Ġdecision . </s> The Ġeasiest Ġway Ġof Ġmatching Ġis Ġto Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġbetween Ġtwo Ġvectors . ĠIt Ġis Ġgenerally Ġdone Ġby Ġtwo Ġstep : ĠFirst , Ġencode Ġtext Ġinto Ġword Ġvectors , Ġsentence Ġvectors Ġor Ġparagraph Ġvectors . ĠSecond , Ġsimply Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġbetween Ġtarget Ġvectors . ĠThis Ġmethod Ġperforms Ġwell Ġwhen Ġapplied Ġto Ġword - level Ġmatching . ĠHowever , Ġas Ġfor Ġmatching Ġbetween Ġsentences Ġor Ġparagraphs , Ġa Ġsingle Ġvector Ġis Ġnot Ġsufficient Ġto Ġencode Ġall Ġthe Ġimportant Ġinformation . ĠIn Ġorder Ġto Ġsolve Ġthis Ġproblem , ĠWang Ġand ĠJiang Ġproposed Ġa ĠâĢ ľ comp are - agg regate âĢ Ŀ ĠB IB REF 3 Ġframework Ġthat Ġperforms Ġword - level Ġmatching Ġusing Ġmultiple Ġtechniques Ġfollowed Ġby Ġaggregation Ġwith Ġconv olution al Ġneural Ġnetwork . ĠIn Ġtheir Ġwork , Ġthey Ġshow Ġthat Ġcompare - agg regate Ġframework Ġcan Ġeffectively Ġmatch Ġtwo Ġsequences Ġthrough Ġa Ġwide Ġrange . </s> Although Ġ" comp are - agg regate " Ġmatching Ġmechanism Ġperforms Ġwell Ġon Ġmultiple Ġquestion Ġanswering Ġtasks , Ġit Ġhas Ġtwo Ġdeficiencies . ĠFirst , Ġit Ġtends Ġto Ġaggregate Ġpassively Ġthrough Ġthe Ġsequence Ġrather Ġthan Ġtake Ġthe Ġimportance Ġof Ġeach Ġelement Ġinto Ġaccount . ĠThat Ġis , Ġ" comp are Ġaggregate " Ġmodel Ġconsiders Ġall Ġthe Ġsequential Ġcontents Ġequally . ĠSecond , Ġ" comp are Ġaggregate " Ġcan Ġonly Ġtake Ġfew Ġneighboring Ġelements Ġinto Ġaccount Ġat Ġthe Ġsame Ġtime Ġbecause Ġof Ġthe Ġlimitation Ġof ĠCNN Ġkernel Ġsize . </s> In Ġthis Ġpaper , Ġwe Ġpropose ĠQuery - based ĠAttention ĠCNN Ġ( Q AC NN ) Ġto Ġdeal Ġwith Ġthe Ġdeficiencies Ġabove . ĠFirst , Ġwe Ġadd Ġquery - based Ġattention Ġmechanism Ġinto Ġoriginal Ġ" comp are Ġaggregate " Ġmodel . ĠMoreover , ĠWe Ġre - design Ġthe Ġaggregation Ġmechanism Ġin Ġ" comp are Ġaggregate " Ġto Ġa Ġtwo - st aged ĠCNN Ġarchitecture Ġwhich Ġcomprises Ġword - level Ġaggregation Ġand Ġsentence - level Ġaggregation . ĠIn Ġthis Ġway , ĠQ AC NN Ġcan Ġefficiently Ġextract Ġfeatures Ġcross Ġsentences . </s> Our Ġmodel Ġconsists Ġof Ġthree Ġcomponents : Ġ1 ) ĠThe Ġsimilarity Ġmapping Ġlayer Ġwhich Ġconverts Ġthe Ġinput Ġpassage , Ġquery Ġand Ġchoice Ġinto Ġfeature Ġrepresentation Ġand Ġperform Ġa Ġsimilarity Ġoperation Ġto Ġeach Ġother . Ġ2 ) ĠThe Ġattention - based ĠCNN Ġmatching Ġnetwork Ġcomposed Ġof Ġa Ġtwo - st aged ĠCNN Ġfocusing Ġon Ġword - level Ġand Ġsentence - level Ġmatching Ġrespectively . Ġ3 ) ĠThe Ġprediction Ġlayer Ġwhich Ġmakes Ġthe Ġfinal Ġdecision . </s> The Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare Ġthree - fold . ĠFirst , Ġwe Ġintroduce Ġa Ġtwo - st aged ĠCNN Ġarchitecture Ġwhich Ġintegrates Ġinformation Ġfrom Ġword - level Ġto Ġsentence - level , Ġand Ġthen Ġfrom Ġsentence - level Ġto Ġpassage - level . ĠSecond , Ġwe Ġintroduce Ġattention Ġmechanism Ġinto Ġthis Ġnet . ĠWe Ġuse Ġspecially Ġdesigned ĠCNN Ġstructure Ġand Ġattention Ġmechanism Ġto Ġrecognize Ġthe Ġpattern Ġof Ġsimilarity Ġmap Ġand Ġeventually Ġidentify Ġspecific Ġsynt actic Ġstructure Ġof Ġqueries . ĠBy Ġtransforming Ġpassage - query Ġfeature Ġinto Ġattention Ġmaps Ġand Ġapplying Ġit Ġto Ġpassage - choice Ġmatching Ġresult , Ġwe Ġreasonably Ġgive Ġweight Ġto Ġevery Ġword Ġin Ġthe Ġpassage . ĠLastly , Ġour Ġmodel Ġreaches Ġ79 . 99 % Ġaccuracy Ġon Ġthe ĠMovie Q A Ġdataset Ġwhich Ġyields Ġtop Ġ1 Ġresult Ġon Ġthis Ġdataset . </s> Q AC NN </s> In Ġthis Ġquestion Ġanswering Ġtask , Ġa Ġreading Ġpassage Ġ, Ġa Ġquery Ġand Ġseveral Ġanswer Ġchoices Ġare Ġgiven . ĠP Ġdenotes Ġthe Ġpassage , ĠQ Ġdenotes Ġquery Ġand ĠC Ġdenotes Ġone Ġof Ġthe Ġmultiple Ġchoices . ĠThe Ġtarget Ġof Ġthe Ġmodel Ġis Ġto Ġchoose Ġa Ġcorrect Ġanswer ĠA Ġfrom Ġmultiple Ġchoices Ġbased Ġon Ġinform ations Ġof ĠP Ġand ĠQ . </s> Fig . ĠFIG REF 1 Ġis Ġthe Ġpipeline Ġoverview Ġof ĠQ AC NN . ĠFirst , Ġwe Ġuse Ġembed ding Ġlayer Ġto Ġtransform ĠP , ĠQ , Ġand ĠC Ġinto Ġword Ġembed ding . ĠThen Ġthe Ġcompare Ġlayer Ġgenerates Ġpassage - query Ġsimilarity Ġmap ĠIN LINE FORM 0 Ġand Ġpassage - choice Ġsimilarity Ġmap ĠIN LINE FORM 1 Ġ. ĠThe Ġfollowing Ġpart Ġis Ġthe Ġmain Ġcomponent Ġof ĠQ AC NN . ĠIt Ġconsists Ġof Ġtwo - st aged ĠCNN Ġarchitecture . ĠThe Ġfirst Ġstage Ġprojects Ġword - level Ġfeature Ġinto Ġsentence - level , Ġand Ġthe Ġsecond Ġstage Ġprojects Ġsentence - level Ġfeature Ġinto Ġpassage - level . ĠMoreover , Ġwe Ġapply Ġquery - based Ġattention Ġmechanism Ġto Ġeach Ġstage Ġon Ġthe Ġbasis Ġof ĠIN LINE FORM 2 Ġfeature Ġat Ġword Ġlevel Ġand Ġsentence Ġlevel Ġrespectively . ĠAfter ĠQ AC NN ĠLayer , Ġwe Ġobtain Ġeach Ġchoice Ġanswer Ġfeature . ĠFinally , Ġa Ġprediction Ġlayer Ġcollects Ġoutput Ġinformation Ġfrom Ġevery Ġchoice Ġfeature Ġand Ġreturns Ġthe Ġmost Ġpossible Ġanswer . </s> Similar ity ĠM apping ĠLayer </s> Similar ity ĠM apping ĠLayer Ġis Ġcomposed Ġof Ġtwo Ġpart : Ġembed ding Ġlayer Ġand Ġcompare Ġlayer . ĠGiven Ġa Ġpassage ĠP Ġwith ĠIN LINE FORM 0 Ġsentences , Ġa Ġquery ĠQ , Ġand Ġa Ġchoice ĠC , Ġthe Ġembed ding Ġlayer Ġtransforms Ġevery Ġwords Ġin ĠP , ĠQ Ġand ĠC Ġinto Ġword Ġembed ding : ĠDIS PLAY FORM 0 </s> IN LINE FORM 0 Ġis Ġthe Ġlength Ġof Ġa Ġsentence Ġin Ġpassage , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġthe Ġlength Ġof Ġquery Ġand Ġlength Ġof Ġone Ġsingle Ġchoice Ġrespectively . ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġare Ġword Ġembed d ings . ĠWord Ġembed ding Ġcan Ġbe Ġobtained Ġby Ġany Ġtype Ġof Ġembed ding Ġtechnique , Ġsuch Ġas Ġrecurrent Ġneural Ġnetwork ĠB IB REF 4 Ġ, ĠSequence - to - sequence Ġmodel ĠB IB REF 5 Ġ, ĠWord 2 vec ĠB IB REF 6 Ġ, Ġetc . ĠIn Ġour Ġwork , Ġwe Ġsimply Ġuse Ġpre - trained ĠGl o Ve Ġword Ġvectors ĠB IB REF 7 Ġas Ġthe Ġembed ding Ġwithout Ġany Ġfurther Ġmodification Ġor Ġtraining . </s> After Ġword Ġembed ding Ġstep , ĠWe Ġwant Ġto Ġacquire Ġsimilarity Ġmap Ġwhich Ġtells Ġus Ġlocation Ġrelationship Ġbetween Ġpassage Ġand Ġquery , Ġpassage Ġand Ġchoices . ĠWe Ġuse Ġcompare Ġlayer Ġto Ġcompare Ġeach Ġpassage Ġsentence ĠIN LINE FORM 0 Ġto ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġat Ġword Ġlevel Ġseparately Ġas ĠFig . ĠFIG REF 7 Ġand ĠFig . ĠFIG REF 8 Ġshow . ĠDIS PLAY FORM 0 </s> That Ġis , Ġwe Ġcompare Ġeach Ġword Ġin Ġsentences Ġof Ġpassage Ġto Ġeach Ġword Ġin Ġquery Ġand Ġchoice . ĠWe Ġuse Ġcos ine Ġsimilarity Ġas Ġthe Ġcomparing Ġmethod Ġhere . ĠThis Ġstep Ġcreates Ġtwo Ġsimilarity Ġmap , Ġpassage - query Ġsimilarity Ġmap ĠIN LINE FORM 0 Ġand Ġpassage - choice Ġsimilarity Ġmap ĠIN LINE FORM 1 Ġ. </s> Q AC NN ĠLayer </s> We Ġpropose Ġan Ġattention Ġconv olution al Ġmatching Ġlayer Ġto Ġintegrate Ġtwo Ġsimilarity Ġmaps Ġgiven Ġabove . ĠThat Ġis , ĠQ AC NN ĠLayer Ġis Ġused Ġto Ġlearn Ġthe Ġlocation Ġrelationship Ġpattern . ĠIt Ġcontains Ġa Ġtwo - st aged ĠCNN Ġcombined Ġwith Ġquery - based Ġattention Ġmechanism . ĠEach Ġstage Ġcomprises Ġtwo Ġmajor Ġpart : Ġattention Ġmap Ġand Ġoutput Ġrepresentation . </s> Fig . ĠFIG REF 10 Ġshows Ġthe Ġarchitecture Ġof Ġthe Ġattention Ġmap Ġin Ġfirst Ġstage ĠCNN . ĠWe Ġchoose ĠIN LINE FORM 0 Ġsentence Ġslice ĠIN LINE FORM 1 Ġin ĠIN LINE FORM 2 Ġ, Ġand Ġapply ĠCNN Ġto Ġit Ġusing Ġthe Ġconv olution Ġkernel ĠIN LINE FORM 3 Ġ, Ġwhere Ġsupers cript ĠIN LINE FORM 4 Ġdenotes Ġattention Ġmap , Ġsubscript ĠIN LINE FORM 5 Ġdenotes Ġthe Ġfirst Ġstage ĠCNN . ĠSymbol ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġrepresent Ġwidth Ġof Ġkernel Ġand Ġnumber Ġof Ġkernel Ġrespectively . ĠThe Ġgenerated Ġfeature ĠIN LINE FORM 8 Ġis Ġas Ġfollow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġbias . ĠWith ĠIN LINE FORM 1 Ġcovering Ġwhole Ġquery Ġand Ġseveral Ġwords Ġin Ġthe Ġpassage , Ġconv olution Ġkernels Ġwould Ġlearn Ġthe Ġquery Ġsynt actic Ġstructure Ġand Ġgive Ġweight Ġto Ġeach Ġpassage 's Ġlocation . ĠThat 's Ġwhy Ġwe Ġuse Ġs igm oid Ġfunction Ġas Ġactivation Ġfunction Ġin Ġthis Ġstage . ĠFurthermore , Ġwe Ġperform Ġmax pool ing Ġto ĠIN LINE FORM 2 Ġperpend icularly Ġin Ġorder Ġto Ġfind Ġthe Ġlargest Ġweight Ġbetween Ġdifferent Ġkernels Ġin Ġthe Ġsame Ġlocation , Ġusing Ġmax pool Ġkernel Ġshaped ĠIN LINE FORM 3 Ġ, Ġand Ġthen Ġgenerate Ġword - level Ġattention Ġmap ĠIN LINE FORM 4 Ġfor Ġeach Ġsentence . </s> In Ġthis Ġstage , Ġwe Ġwant Ġto Ġacquire Ġpassage 's Ġsentence Ġfeatures Ġbased Ġon Ġthe Ġquery Ġand Ġthe Ġchoice Ġrespectively . ĠWe Ġapply ĠCNN Ġto ĠIN LINE FORM 0 Ġto Ġaggregate Ġpattern Ġof Ġlocation Ġrelationship Ġand Ġacquire Ġchoice - based Ġsentence Ġfeatures . ĠAlso , Ġwe Ġapply ĠCNN Ġto ĠIN LINE FORM 1 Ġto Ġacquire Ġquery - based Ġsentence Ġfeatures . ĠCNN Ġarchitecture Ġof Ġoutput Ġrepresentation Ġpart ĠFig . ĠFIG REF 11 Ġis Ġsimilar Ġto Ġwhich Ġof Ġattention Ġmap Ġpart , Ġbut Ġwe Ġuse Ġdifferent Ġkernels ĠIN LINE FORM 2 Ġand Ġdifferent Ġbias ĠIN LINE FORM 3 Ġ: ĠDIS PLAY FORM 0 </s> where Ġthe Ġsupers cript ĠIN LINE FORM 0 Ġdenotes Ġoutput Ġrepresentation . ĠWe Ġapply ĠIN LINE FORM 3 Ġon ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġthen Ġfinally Ġgenerate ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġusing Ġeq . 4 . ĠWe Ġthen Ġmultiply ĠIN LINE FORM 8 Ġby Ġthe Ġword - level Ġattention Ġmap ĠIN LINE FORM 9 Ġwhich Ġcomes Ġfrom Ġstage Ġ2 . 2 . 1 Ġelement - wise Ġthrough Ġthe Ġfirst Ġdimension . ĠAt Ġlast , Ġwe Ġmax pool ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġhorizontally Ġwith Ġkernel Ġshape ĠIN LINE FORM 12 Ġto Ġget Ġthe Ġquery - based Ġsentence Ġfeatures ĠIN LINE FORM 13 Ġand Ġchoice - based Ġsentence Ġfeatures ĠIN LINE FORM 14 Ġ. </s> Fig . ĠFIG REF 28 Ġis Ġthe Ġarchitecture Ġof Ġattention Ġmap Ġin Ġthe Ġsecond Ġstage ĠCNN . ĠBased Ġon Ġfirst Ġstage Ġquery - based Ġsentence Ġfeatures Ġfrom Ġsection ĠUID 15 Ġ, Ġwe Ġwant Ġto Ġacquire Ġsentence - level Ġattention Ġmap . ĠThe Ġinput Ġof Ġthis Ġstage Ġis ĠIN LINE FORM 0 Ġ, Ġwhich Ġwill Ġbe Ġfurther Ġrefined Ġby ĠCNN Ġwith Ġkernel ĠIN LINE FORM 1 Ġand Ġgenerates Ġintermediate Ġfeatures ĠIN LINE FORM 2 Ġ. ĠDIS PLAY FORM 0 </s> Then , Ġsame Ġas Ġattention Ġmap Ġof Ġfirst Ġstage , Ġwe Ġmax pool ĠIN LINE FORM 0 Ġwith Ġkernel Ġshaped ĠIN LINE FORM 1 Ġ, Ġand Ġobtain Ġsentence - level Ġattention Ġmap ĠIN LINE FORM 2 Ġ. </s> Output Ġrepresentation Ġpart Ġof Ġthe Ġsecond Ġstage Ġin ĠFig . ĠFIG REF 29 Ġhas Ġtwo Ġinput , Ġsentence - level Ġattention Ġmap ĠIN LINE FORM 0 Ġand Ġsentence - level Ġfeatures ĠIN LINE FORM 1 Ġ. ĠThe Ġequations Ġhere Ġare Ġsimilar Ġto Ġthose Ġpreviously Ġmentioned . ĠAs Ġfollow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ. ĠOutput Ġrepresentation Ġof Ġcertain Ġchoice ĠIN LINE FORM 3 Ġis Ġthe Ġfinal Ġoutput Ġof ĠQ AC NN ĠLayer . </s> Pred iction ĠLayer </s> Pred iction ĠLayer Ġis Ġthe Ġfinal Ġpart Ġof ĠQ AC NN . ĠWe Ġuse ĠIN LINE FORM 0 Ġto Ġrepresent Ġthe Ġfinal Ġoutput Ġrepresentation Ġof Ġthe ĠIN LINE FORM 1 Ġchoice Ġ. ĠIn Ġorder Ġto Ġfind Ġout Ġthe Ġmost Ġcorrect Ġchoice , Ġwe Ġsimply Ġpass ĠIN LINE FORM 2 Ġto Ġtwo Ġfully - connected Ġlayer Ġand Ġcompute Ġprobability Ġfor Ġeach Ġchoice Ġusing Ġsoft max Ġas Ġfollows : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġ. </s> Im plementation Ġdetails </s> In Ġthe Ġpre processing Ġstep , Ġwe Ġused Ġpre - trained ĠGl o Ve Ġvectors Ġfor Ġword Ġembed d ings , Ġand Ġthey Ġwould Ġnot Ġbe Ġupdated Ġduring Ġtraining ; ĠWe Ġpadded Ġsentence Ġnumber Ġin Ġeach Ġpassage Ġto Ġ101 , Ġall Ġword Ġnumber Ġin Ġeach Ġsentence Ġto Ġ100 . ĠWord Ġnumber Ġof Ġqueries Ġand Ġchoices Ġwere Ġpadded Ġto Ġ50 . ĠFor Ġall Ġkernels Ġof ĠCNN , ĠIN LINE FORM 0 Ġ, Ġeach Ġof Ġwhich Ġhas Ġthree Ġdifferent Ġkernel Ġwidth ĠIN LINE FORM 1 Ġ; Ġeach Ġof Ġthem Ġhas Ġsame Ġkernel Ġnumber ĠIN LINE FORM 2 Ġ. ĠWe Ġutilized Ġdrop out Ġin Ġeach ĠCNN Ġlayer Ġwith Ġdrop out Ġrate Ġ0 . 8 . ĠWe Ġused ĠAdam ĠB IB REF 8 Ġoptim izer Ġto Ġoptimize Ġour Ġmodel Ġwith Ġinitial Ġlearning Ġrate Ġ0 . 001 . </s> Exper imental ĠResult </s> We Ġmainly Ġfocus Ġon Ġthe ĠMovie Q A Ġdataset Ġto Ġtrain Ġand Ġevaluate Ġour Ġmodel . ĠMovie Q A Ġdataset Ġaims Ġto Ġevaluate Ġautomatic Ġstory Ġcomprehension Ġfrom Ġboth Ġvideo Ġand Ġtext . ĠThe Ġdata Ġset Ġconsists Ġof Ġalmost Ġ15 , 000 Ġmultiple Ġchoice Ġquestion Ġanswers . ĠD iverse Ġinformation Ġin Ġthis Ġdataset Ġlike Ġplots , Ġscripts , Ġsub - title Ġand Ġvideo Ġcapt ions Ġcan Ġbe Ġused Ġto Ġinfer Ġanswers . ĠIn Ġour Ġtask , Ġonly Ġplot Ġinform ations Ġare Ġused . ĠThis Ġchallenging Ġdataset Ġis Ġsuitable Ġto Ġevaluate ĠQ AC NN Ġbecause Ġmovie Ġplots Ġare Ġlonger Ġthan Ġnormal Ġreading Ġcomprehension Ġtask . ĠEach Ġquestion Ġcomes Ġwith Ġa Ġset Ġof Ġfive Ġhighly Ġplausible Ġchoices , Ġonly Ġone Ġof Ġwhich Ġis Ġcorrect ; ĠIn Ġthe ĠMovie Q A Ġbenchmark , Ġthere Ġare Ġ1958 ĠQ A Ġpairs Ġin Ġthe Ġval Ġset Ġand Ġ3 138 ĠQ A Ġpairs Ġin Ġthe Ġtest Ġset . </s> We Ġused Ġensemble Ġmodel Ġin Ġthis Ġdataset . ĠThe Ġensemble Ġmodel Ġconsists Ġof Ġeight Ġtraining Ġruns Ġmodels Ġwith Ġidentical Ġstructure Ġand Ġhyper - param eter . ĠIn Ġthe Ġval Ġset , Ġwe Ġachieve Ġ77 . 6 % Ġaccuracy Ġwith Ġsingle Ġmodel Ġand Ġ79 . 0 % Ġaccuracy Ġwith Ġensemble Ġmodel . ĠIn Ġthe Ġtest Ġset , Ġas Ġthe ĠTable Ġ1 . Ġshows , Ġour Ġmodel Ġachieves Ġ79 . 99 Ġ% Ġaccuracy Ġwith Ġensemble Ġmodel Ġand Ġis Ġthe Ġstate Ġof Ġthe Ġart . </s> We Ġalso Ġapplied Ġour Ġmodel Ġto ĠM CT est Ġdataset Ġwhich Ġrequires Ġmachines Ġto Ġanswer Ġmultiple - choice Ġreading Ġcomprehension Ġquestions Ġabout Ġfictional Ġstories . ĠThe Ġoriginal Ġpaper Ġdescribes Ġthat Ġa Ġbaseline Ġmethod Ġuses Ġa Ġcombination Ġof Ġa Ġsliding Ġwindow Ġscore Ġand Ġa Ġdistance Ġbased Ġ. ĠThey Ġachieve Ġ66 . 7 % Ġand Ġ56 . 7 % Ġon ĠMC 500 Ġand ĠMC 160 Ġseparately . ĠBecause Ġof Ġthe Ġrestricted Ġtraining Ġset Ġand Ġdevelopment Ġset , Ġwe Ġtrained Ġour Ġmodel Ġon ĠMovie Q A Ġtraining Ġset Ġand Ġapplied Ġthe Ġresult Ġto Ġtest ĠM CT est Ġdataset . ĠOn ĠM CT est Ġdataset , Ġwe Ġstill Ġoutper form Ġbaseline Ġand Ġachieve Ġ68 . 1 % Ġaccuracy Ġon ĠMC 160 Ġand Ġ61 . 5 % Ġaccuracy Ġon ĠMC 500 . </s> Discussion </s> Q AC NN Ġis Ġa Ġpowerful Ġnetwork Ġfocusing Ġon Ġmultiple Ġchoice ĠQ A Ġtask . ĠIt Ġmatches Ġbetween Ġpassage Ġand Ġchoices Ġbased Ġon Ġquery Ġinformation . ĠOne Ġof Ġthe Ġmost Ġimportant Ġidea Ġin ĠQ AC NN Ġis Ġtwo - st aged Ġattention Ġmap . ĠThe Ġfirst Ġattention Ġmap Ġis Ġat Ġword Ġlevel , Ġrepresenting Ġthe Ġimportance Ġof Ġeach Ġword Ġin Ġparagraph Ġto Ġa Ġcertain Ġquestion ; Ġthe Ġsecond Ġattention Ġmap , Ġhowever Ġis Ġat Ġsentence Ġlevel , Ġrepresenting Ġthe Ġimportance Ġof Ġeach Ġsentence Ġin Ġparagraph Ġto Ġa Ġcertain Ġquestion . </s> In Ġthis Ġsection , Ġwe Ġdesigned Ġseveral Ġexperiments Ġto Ġtest Ġhow Ġtwo - stage Ġmechanism Ġand Ġattention Ġmaps Ġimpact Ġon Ġour Ġmodel . </s> In Ġthis Ġexperiment , we Ġfocused Ġon Ġthe Ġdifference Ġbetween Ġone - stage ĠQ AC NN Ġand Ġtwo - stage ĠQ AC NN . ĠFor Ġone - stage ĠQ AC NN , Ġwe Ġdidn 't Ġsplit Ġan Ġentire Ġpassage Ġinto Ġsentences . ĠThat Ġis , Ġthe Ġshape Ġof Ġpassage - query Ġsimilarity Ġmap ĠIN LINE FORM 0 Ġand Ġpassage - choice Ġsimilarity Ġmap ĠIN LINE FORM 1 Ġare Ġ2 D Ġrather Ġthan Ġ3 D . ĠWe Ġconv olved Ġthem Ġdirectly Ġon Ġword - level Ġand Ġoutput Ġpassage Ġfeature Ġwithout Ġsecond - stage Ġinvolved . ĠThe Ġresult Ġis Ġshown Ġon Ġtable ĠT AB REF 36 Ġ. ĠThe Ġresult Ġshows Ġthat Ġthe Ġmodified Ġone Ġstaged ĠQ AC NN Ġreaches Ġ66 . 8 % Ġaccuracy Ġon Ġvalidation Ġset , Ġwhich Ġis Ġten Ġpercent Ġlower Ġthan Ġ78 . 1 %, Ġthe Ġoriginal ĠQ AC NN Ġaccuracy Ġon Ġvalidation Ġset . </s> In Ġthis Ġexperiment , Ġour Ġtarget Ġis Ġto Ġvalidate Ġthe Ġeffect Ġof Ġquery - based Ġattention Ġin ĠQ AC NN . ĠWe Ġmodified Ġthree Ġdifferent Ġstructures Ġfrom Ġoriginal ĠQ AC NN ĠLayer Ġbelow : </s> 1 ) ĠFor Ġthe Ġfirst Ġone , Ġwe Ġmodified ĠQ AC NN ĠLayer Ġin Ġsection ĠSEC REF 12 Ġand Ġremoved Ġboth Ġsentence - level Ġattention Ġmap Ġand Ġword - level Ġattention Ġmap Ġpart Ġfrom Ġit . ĠHowever , Ġthis Ġmodified Ġmodel Ġwould Ġhave Ġa Ġdeficiency Ġof Ġquery Ġinformation . ĠTherefore , Ġwe Ġconc aten ated Ġthe Ġfinal Ġoutput Ġrepresentation Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġtogether Ġbefore Ġprediction Ġlayer . ĠThe Ġexperiment Ġresult Ġis Ġshown Ġon Ġthe ĠTable ĠT AB REF 36 Ġ. ĠThe Ġresult Ġis Ġalmost Ġten Ġpercent Ġless Ġthan Ġthe Ġoriginal Ġone . Ġ2 ) ĠFor Ġthe Ġsecond Ġone , Ġwe Ġonly Ġremoved Ġsentence - level Ġattention Ġin Ġsection ĠUID 18 Ġfrom ĠQ AC NN ĠLayer Ġand Ġkept Ġword - level Ġattention Ġin Ġthe Ġmodel . Ġ3 ) ĠFor Ġthe Ġlast Ġone , Ġinstead Ġof Ġremoving Ġsentence - level Ġattention , Ġwe Ġremoved Ġword - level Ġattention Ġfrom ĠQ AC NN ĠLayer . </s> The Ġresult Ġis Ġshown Ġin ĠTable ĠT AB REF 36 Ġ. ĠWe Ġcan Ġsee Ġthat ĠQ AC NN ( with Ġonly Ġword - level Ġattention ) Ġperforms Ġbetter Ġthan ĠQ AC NN ( without Ġattention ); ĠQ AC NN ( with Ġonly Ġsentence - level Ġattention ) Ġperforms Ġbetter Ġthan ĠQ AC NN ( with Ġonly Ġword - level Ġattention ); ĠAnd Ġoriginal ĠQ AC NN Ġwhich Ġcontains Ġboth Ġword - level Ġand Ġsentence - level Ġattention Ġdoes Ġthe Ġbest Ġjob Ġamong Ġall . ĠThus , Ġnot Ġonly Ġword - level Ġattention Ġbut Ġalso Ġsentence - level Ġattention Ġcan Ġcontribute Ġto Ġthe Ġperformance Ġof ĠQ AC NN . ĠHowever , Ġsentence - level Ġattention Ġseems Ġto Ġplay Ġa Ġmore Ġimportant Ġrole . </s> Figure . ĠFIG REF 31 Ġis Ġthe Ġvisualization Ġof Ġtwo Ġattention Ġmaps Ġand Ġtheir Ġcorresponding Ġquestion . ĠThe Ġupper Ġhalf Ġof ĠFigure . ĠFIG REF 31 Ġis Ġthe Ġattention Ġmap Ġat Ġsentence Ġlevel . ĠWe Ġpicked Ġthe Ġsentence Ġwith Ġthe Ġlargest Ġattention Ġvalue Ġas Ġtarget Ġsentence Ġand Ġexamine Ġit . ĠThus , Ġwe Ġcould Ġget Ġthe Ġlower Ġhalf Ġof ĠFigure . ĠFIG REF 31 Ġ, Ġwhich Ġshows Ġthe Ġattention Ġmap Ġat Ġword Ġlevel Ġon Ġthe Ġtarget Ġsentence . ĠWe Ġused Ġa Ġquestion Ġfrom Ġmovie , ĠHarry ĠPotter , Ġas Ġan Ġexample . ĠThe Ġresult Ġshows Ġthat Ġthe Ġsentence Ġwith Ġthe Ġlargest Ġattention Ġvalue Ġis Ġexactly Ġwhere Ġthe Ġcorrect Ġanswer Ġcomes Ġfrom . ĠIt Ġturns Ġout Ġthat Ġsentence - level Ġattention Ġmap Ġcan Ġsuccessfully Ġfind Ġout Ġwhich Ġsentence Ġcontains Ġthe Ġinformation Ġof Ġcorrect Ġanswer . ĠAs Ġfor Ġword - level Ġattention Ġmap , Ġwe Ġcan Ġeasily Ġsee Ġthat Ġthe Ġattention Ġmap Ġfocus Ġmainly Ġon Ġthe Ġend Ġof Ġtarget Ġsentence , Ġwhich Ġis Ġobviously Ġmore Ġimportant Ġfor Ġanswering Ġthis Ġquestion . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġan Ġefficient Ġmatching Ġmechanism Ġon Ġmultiple Ġchoice Ġquestion Ġanswering Ġtask . ĠWe Ġintroduce Ġtwo - st aged ĠCNN Ġto Ġmatch Ġpassage Ġand Ġchoice Ġon Ġword Ġlevel Ġand Ġsentence Ġlevel . ĠIn Ġaddition , Ġwe Ġuse Ġquery - based ĠCNN Ġattention Ġto Ġenhance Ġmatching Ġeffect . </s> The Ġpower Ġof Ġthe Ġmodel Ġis Ġverified Ġon ĠMovie Q A Ġdataset , Ġwhich Ġyielded Ġthe Ġstate Ġof Ġthe Ġart Ġresult Ġon Ġthe Ġdataset . ĠIn Ġthe Ġfuture , Ġwe Ġare Ġnow Ġworking Ġon Ġtraining Ġour Ġmodel Ġbased Ġon Ġour Ġown Ġtrained Ġembed ding Ġwith ĠTF - ID F ĠB IB REF 9 Ġweight ing . ĠFurthermore , Ġwe Ġwould Ġlike Ġto Ġtest Ġour Ġmodel Ġon Ġopen - answer Ġtask Ġlike ĠS Qu a D Ġby Ġseeing Ġthe Ġwhole Ġcorpus Ġas Ġan ĠâĢ ľ answer Ġpool " Ġand Ġsolve Ġit Ġlike Ġmultiple Ġchoice Ġquestion .