By evaluating their model on adversarial sets containing misleading sentences
ABSTRACTIVE
we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise
EXTRACTIVE
MRC Dataset. The MRC dataset used in this paper is SQuAD 1.1, which contains over INLINEFORM0 passage-question pairs and has been randomly partitioned into three parts: a training set ( INLINEFORM1 ), a development set ( INLINEFORM2 ), and a test set ( INLINEFORM3 ). Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models. The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models. Specifically, each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage. We compare KAR with other MRC models in both performance and the robustness to noise. Specifically, we not only evaluate the performance of KAR on the development set and the test set, but also do this on the adversarial sets. As for the comparative objects, we only consider the single MRC models that rank in the top 20 on the SQuAD 1.1 leader board and have reported their performance on the adversarial sets. There are totally five such comparative objects, which can be considered as representatives of the state-of-the-art MRC models. As shown in Table TABREF12 , on the development set and the test set, the performance of KAR is on par with that of the state-of-the-art MRC models; on the adversarial sets, KAR outperforms the state-of-the-art MRC models by a large margin. That is to say, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them.
MRC Dataset. The MRC dataset used in this paper is SQuAD 1.1, which contains over INLINEFORM0 passage-question pairs and has been randomly partitioned into three parts: a training set ( INLINEFORM1 ), a development set ( INLINEFORM2 ), and a test set ( INLINEFORM3 ). Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models. The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models. Specifically, each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage.
1809.03449
Introduction </s> Machine ĠReading ĠComp rehens ion Ġ( M RC ), Ġas Ġthe Ġname Ġsuggests , Ġrequires Ġa Ġmachine Ġto Ġread Ġa Ġpassage Ġand Ġanswer Ġits Ġrelevant Ġquestions . ĠSince Ġthe Ġanswer Ġto Ġeach Ġquestion Ġis Ġsupposed Ġto Ġstem Ġfrom Ġthe Ġcorresponding Ġpassage , Ġa Ġcommon ĠM RC Ġsolution Ġis Ġto Ġdevelop Ġa Ġneural - network - based ĠM RC Ġmodel Ġthat Ġpredicts Ġan Ġanswer Ġspan Ġ( i . e . Ġthe Ġanswer Ġstart Ġposition Ġand Ġthe Ġanswer Ġend Ġposition ) Ġfrom Ġthe Ġpassage Ġof Ġeach Ġgiven Ġpassage - question Ġpair . ĠTo Ġfacilitate Ġthe Ġexplor ations Ġand Ġinnovations Ġin Ġthis Ġarea , Ġmany ĠM RC Ġdatasets Ġhave Ġbeen Ġestablished , Ġsuch Ġas ĠS Qu AD ĠB IB REF 0 Ġ, ĠMS ĠMAR CO ĠB IB REF 1 Ġ, Ġand ĠTri via Q A ĠB IB REF 2 Ġ. ĠConsequently , Ġmany Ġpioneering ĠM RC Ġmodels Ġhave Ġbeen Ġproposed , Ġsuch Ġas ĠBi DA F ĠB IB REF 3 Ġ, ĠR - NET ĠB IB REF 4 Ġ, Ġand ĠQ AN et ĠB IB REF 5 Ġ. ĠAccording Ġto Ġthe Ġleader Ġboard Ġof ĠS Qu AD , Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġhave Ġachieved Ġthe Ġsame Ġperformance Ġas Ġhuman Ġbeings . ĠHowever , Ġdoes Ġthis Ġimply Ġthat Ġthey Ġhave Ġpossessed Ġthe Ġsame Ġreading Ġcomprehension Ġability Ġas Ġhuman Ġbeings ? </s> OF ĠCOUR SE ĠNOT . ĠThere Ġis Ġa Ġhuge Ġgap Ġbetween ĠM RC Ġmodels Ġand Ġhuman Ġbeings , Ġwhich Ġis Ġmainly Ġreflected Ġin Ġthe Ġhunger Ġfor Ġdata Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠOn Ġthe Ġone Ġhand , Ġdeveloping ĠM RC Ġmodels Ġrequires Ġa Ġlarge Ġamount Ġof Ġtraining Ġexamples Ġ( i . e . Ġthe Ġpassage - question Ġpairs Ġlabeled Ġwith Ġanswer Ġspans ), Ġwhile Ġhuman Ġbeings Ġcan Ġachieve Ġgood Ġperformance Ġon Ġevaluation Ġexamples Ġ( i . e . Ġthe Ġpassage - question Ġpairs Ġto Ġaddress ) Ġwithout Ġtraining Ġexamples . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 6 Ġrevealed Ġthat Ġintentionally Ġinjected Ġnoise Ġ( e . g . Ġmisleading Ġsentences ) Ġin Ġevaluation Ġexamples Ġcauses Ġthe Ġperformance Ġof ĠM RC Ġmodels Ġto Ġdrop Ġsignificantly , Ġwhile Ġhuman Ġbeings Ġare Ġfar Ġless Ġlikely Ġto Ġsuffer Ġfrom Ġthis . ĠThe Ġreason Ġfor Ġthese Ġphenomena , Ġwe Ġbelieve , Ġis Ġthat ĠM RC Ġmodels Ġcan Ġonly Ġutilize Ġthe Ġknowledge Ġcontained Ġin Ġeach Ġgiven Ġpassage - question Ġpair , Ġbut Ġin Ġaddition Ġto Ġthis , Ġhuman Ġbeings Ġcan Ġalso Ġutilize Ġgeneral Ġknowledge . ĠA Ġtypical Ġcategory Ġof Ġgeneral Ġknowledge Ġis Ġinter - word Ġsemantic Ġconnections . ĠAs Ġshown Ġin ĠTable ĠT AB REF 1 Ġ, Ġsuch Ġgeneral Ġknowledge Ġis Ġessential Ġto Ġthe Ġreading Ġcomprehension Ġability Ġof Ġhuman Ġbeings . </s> A Ġpromising Ġstrategy Ġto Ġbridge Ġthe Ġgap Ġmentioned Ġabove Ġis Ġto Ġintegrate Ġthe Ġneural Ġnetworks Ġof ĠM RC Ġmodels Ġwith Ġthe Ġgeneral Ġknowledge Ġof Ġhuman Ġbeings . ĠTo Ġthis Ġend , Ġit Ġis Ġnecessary Ġto Ġsolve Ġtwo Ġproblems : Ġextracting Ġgeneral Ġknowledge Ġfrom Ġpassage - question Ġpairs Ġand Ġutilizing Ġthe Ġextracted Ġgeneral Ġknowledge Ġin Ġthe Ġprediction Ġof Ġanswer Ġspans . ĠThe Ġfirst Ġproblem Ġcan Ġbe Ġsolved Ġwith Ġknowledge Ġbases , Ġwhich Ġstore Ġgeneral Ġknowledge Ġin Ġstructured Ġforms . ĠA Ġbroad Ġvariety Ġof Ġknowledge Ġbases Ġare Ġavailable , Ġsuch Ġas ĠWord Net ĠB IB REF 7 Ġstoring Ġsemantic Ġknowledge , ĠConcept Net ĠB IB REF 8 Ġstoring Ġcommons ense Ġknowledge , Ġand ĠFree base ĠB IB REF 9 Ġstoring Ġfact oid Ġknowledge . ĠIn Ġthis Ġpaper , Ġwe Ġlimit Ġthe Ġscope Ġof Ġgeneral Ġknowledge Ġto Ġinter - word Ġsemantic Ġconnections , Ġand Ġthus Ġuse ĠWord Net Ġas Ġour Ġknowledge Ġbase . ĠThe Ġexisting Ġway Ġto Ġsolve Ġthe Ġsecond Ġproblem Ġis Ġto Ġencode Ġgeneral Ġknowledge Ġin Ġvector Ġspace Ġso Ġthat Ġthe Ġencoding Ġresults Ġcan Ġbe Ġused Ġto Ġenhance Ġthe Ġlex ical Ġor Ġcontextual Ġrepresentations Ġof Ġwords ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠHowever , Ġthis Ġis Ġan Ġimplicit Ġway Ġto Ġutilize Ġgeneral Ġknowledge , Ġsince Ġin Ġthis Ġway Ġwe Ġcan Ġneither Ġunderstand Ġnor Ġcontrol Ġthe Ġfunctioning Ġof Ġgeneral Ġknowledge . ĠIn Ġthis Ġpaper , Ġwe Ġdiscard Ġthe Ġexisting Ġimplicit Ġway Ġand Ġinstead Ġexplore Ġan Ġexplicit Ġ( i . e . Ġunderstandable Ġand Ġcontro ll able ) Ġway Ġto Ġutilize Ġgeneral Ġknowledge . </s> The Ġcontribution Ġof Ġthis Ġpaper Ġis Ġtwo - fold . ĠOn Ġthe Ġone Ġhand , Ġwe Ġpropose Ġa Ġdata Ġenrichment Ġmethod , Ġwhich Ġuses ĠWord Net Ġto Ġextract Ġinter - word Ġsemantic Ġconnections Ġas Ġgeneral Ġknowledge Ġfrom Ġeach Ġgiven Ġpassage - question Ġpair . ĠOn Ġthe Ġother Ġhand , Ġwe Ġpropose Ġan Ġend - to - end ĠM RC Ġmodel Ġnamed Ġas ĠKnowledge ĠA ided ĠReader Ġ( K AR ), Ġwhich Ġexplicitly Ġuses Ġthe Ġabove Ġextracted Ġgeneral Ġknowledge Ġto Ġassist Ġits Ġattention Ġmechanisms . ĠBased Ġon Ġthe Ġdata Ġenrichment Ġmethod , ĠK AR Ġis Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġand Ġsignificantly Ġmore Ġrobust Ġto Ġnoise Ġthan Ġthem . ĠWhen Ġonly Ġa Ġsubset Ġ( ĠIN LINE FORM 0 ĠâĢĵ ĠIN LINE FORM 1 Ġ) Ġof Ġthe Ġtraining Ġexamples Ġare Ġavailable , ĠK AR Ġoutper forms Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġby Ġa Ġlarge Ġmargin , Ġand Ġis Ġstill Ġreasonably Ġrobust Ġto Ġnoise . </s> Data ĠEn rich ment ĠMethod </s> In Ġthis Ġsection , Ġwe Ġelaborate Ġa ĠWord Net - based Ġdata Ġenrichment Ġmethod , Ġwhich Ġis Ġaimed Ġat Ġextracting Ġinter - word Ġsemantic Ġconnections Ġfrom Ġeach Ġpassage - question Ġpair Ġin Ġour ĠM RC Ġdataset . ĠThe Ġextraction Ġis Ġperformed Ġin Ġa Ġcontro ll able Ġmanner , Ġand Ġthe Ġextracted Ġresults Ġare Ġprovided Ġas Ġgeneral Ġknowledge Ġto Ġour ĠM RC Ġmodel . </s> Sem antic ĠRel ation ĠChain </s> Word Net Ġis Ġa Ġlex ical Ġdatabase Ġof ĠEnglish , Ġwhere Ġwords Ġare Ġorganized Ġinto Ġsy ns ets Ġaccording Ġto Ġtheir Ġsenses . ĠA Ġsyn set Ġis Ġa Ġset Ġof Ġwords Ġexpressing Ġthe Ġsame Ġsense Ġso Ġthat Ġa Ġword Ġhaving Ġmultiple Ġsenses Ġbelongs Ġto Ġmultiple Ġsy ns ets , Ġwith Ġeach Ġsyn set Ġcorresponding Ġto Ġa Ġsense . ĠSy ns ets Ġare Ġfurther Ġrelated Ġto Ġeach Ġother Ġthrough Ġsemantic Ġrelations . ĠAccording Ġto Ġthe ĠWord Net Ġinterface Ġprovided Ġby ĠNL T K ĠB IB REF 12 Ġ, Ġthere Ġare Ġtotally Ġsixteen Ġtypes Ġof Ġsemantic Ġrelations Ġ( e . g . Ġhyper ny ms , Ġhyp onyms , Ġhol onyms , Ġmer onyms , Ġattributes , Ġetc .). ĠBased Ġon Ġsyn set Ġand Ġsemantic Ġrelation , Ġwe Ġdefine Ġa Ġnew Ġconcept : Ġsemantic Ġrelation Ġchain . ĠA Ġsemantic Ġrelation Ġchain Ġis Ġa Ġconc aten ated Ġsequence Ġof Ġsemantic Ġrelations , Ġwhich Ġlinks Ġa Ġsyn set Ġto Ġanother Ġsyn set . ĠFor Ġexample , Ġthe Ġsyn set ĠâĢ ľ ker atin . n . 01 âĢ Ŀ Ġis Ġrelated Ġto Ġthe Ġsyn set ĠâĢ ľ fe ather . n . 01 âĢ Ŀ Ġthrough Ġthe Ġsemantic Ġrelation ĠâĢ ľ sub st ance Ġhol onym âĢ Ŀ , Ġthe Ġsyn set ĠâĢ ľ fe ather . n . 01 âĢ Ŀ Ġis Ġrelated Ġto Ġthe Ġsyn set ĠâĢ ľ bird . n . 01 âĢ Ŀ Ġthrough Ġthe Ġsemantic Ġrelation ĠâĢ ľ part Ġhol onym âĢ Ŀ , Ġand Ġthe Ġsyn set ĠâĢ ľ bird . n . 01 âĢ Ŀ Ġis Ġrelated Ġto Ġthe Ġsyn set ĠâĢ ľ par rot . n . 01 âĢ Ŀ Ġthrough Ġthe Ġsemantic Ġrelation ĠâĢ ľ hyp onym âĢ Ŀ , Ġthus ĠâĢ ľ sub st ance Ġhol onym ĠIN LINE FORM 0 Ġpart Ġhol onym ĠIN LINE FORM 1 Ġhyp onym âĢ Ŀ Ġis Ġa Ġsemantic Ġrelation Ġchain , Ġwhich Ġlinks Ġthe Ġsyn set ĠâĢ ľ ker atin . n . 01 âĢ Ŀ Ġto Ġthe Ġsyn set ĠâĢ ľ par rot . n . 01 âĢ Ŀ . ĠWe Ġname Ġeach Ġsemantic Ġrelation Ġin Ġa Ġsemantic Ġrelation Ġchain Ġas Ġa Ġhop , Ġtherefore Ġthe Ġabove Ġsemantic Ġrelation Ġchain Ġis Ġa Ġ3 - hop Ġchain . ĠBy Ġthe Ġway , Ġeach Ġsingle Ġsemantic Ġrelation Ġis Ġequivalent Ġto Ġa Ġ1 - hop Ġchain . </s> Inter - word ĠSem antic ĠConnection </s> The Ġkey Ġproblem Ġin Ġthe Ġdata Ġenrichment Ġmethod Ġis Ġdetermining Ġwhether Ġa Ġword Ġis Ġsem antically Ġconnected Ġto Ġanother Ġword . ĠIf Ġso , Ġwe Ġsay Ġthat Ġthere Ġexists Ġan Ġinter - word Ġsemantic Ġconnection Ġbetween Ġthem . ĠTo Ġsolve Ġthis Ġproblem , Ġwe Ġdefine Ġanother Ġnew Ġconcept : Ġthe Ġextended Ġsy ns ets Ġof Ġa Ġword . ĠGiven Ġa Ġword ĠIN LINE FORM 0 Ġ, Ġwhose Ġsy ns ets Ġare Ġrepresented Ġas Ġa Ġset ĠIN LINE FORM 1 Ġ, Ġwe Ġuse Ġanother Ġset ĠIN LINE FORM 2 Ġto Ġrepresent Ġits Ġextended Ġsy ns ets , Ġwhich Ġincludes Ġall Ġthe Ġsy ns ets Ġthat Ġare Ġin ĠIN LINE FORM 3 Ġor Ġthat Ġcan Ġbe Ġlinked Ġto Ġfrom ĠIN LINE FORM 4 Ġthrough Ġsemantic Ġrelation Ġchains . ĠThe oret ically , Ġif Ġthere Ġis Ġno Ġlimitation Ġon Ġsemantic Ġrelation Ġchains , ĠIN LINE FORM 5 Ġwill Ġinclude Ġall Ġthe Ġsy ns ets Ġin ĠWord Net , Ġwhich Ġis Ġmeaningless Ġin Ġmost Ġsituations . ĠTherefore , Ġwe Ġuse Ġa Ġhyper - param eter ĠIN LINE FORM 6 Ġto Ġrepresent Ġthe Ġpermitted Ġmaximum Ġhop Ġcount Ġof Ġsemantic Ġrelation Ġchains . ĠThat Ġis Ġto Ġsay , Ġonly Ġthe Ġchains Ġhaving Ġno Ġmore Ġthan ĠIN LINE FORM 7 Ġhops Ġcan Ġbe Ġused Ġto Ġconstruct ĠIN LINE FORM 8 Ġso Ġthat ĠIN LINE FORM 9 Ġbecomes Ġa Ġfunction Ġof ĠIN LINE FORM 10 Ġ: ĠIN LINE FORM 11 Ġ( if ĠIN LINE FORM 12 Ġ, Ġwe Ġwill Ġhave ĠIN LINE FORM 13 Ġ). ĠBased Ġon Ġthe Ġabove Ġstatements , Ġwe Ġformulate Ġa Ġhe uristic Ġrule Ġfor Ġdetermining Ġinter - word Ġsemantic Ġconnections : Ġa Ġword ĠIN LINE FORM 14 Ġis Ġsem antically Ġconnected Ġto Ġanother Ġword ĠIN LINE FORM 15 Ġif Ġand Ġonly Ġif ĠIN LINE FORM 16 Ġ. </s> General ĠKnowledge ĠExt raction </s> Given Ġa Ġpassage - question Ġpair , Ġthe Ġinter - word Ġsemantic Ġconnections Ġthat Ġconnect Ġany Ġword Ġto Ġany Ġpassage Ġword Ġare Ġregarded Ġas Ġthe Ġgeneral Ġknowledge Ġwe Ġneed Ġto Ġextract . ĠConsidering Ġthe Ġrequirements Ġof Ġour ĠM RC Ġmodel , Ġwe Ġonly Ġextract Ġthe Ġpositional Ġinformation Ġof Ġsuch Ġinter - word Ġsemantic Ġconnections . ĠSpecifically , Ġfor Ġeach Ġword ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġa Ġset ĠIN LINE FORM 1 Ġ, Ġwhich Ġincludes Ġthe Ġpositions Ġof Ġthe Ġpassage Ġwords Ġthat ĠIN LINE FORM 2 Ġis Ġsem antically Ġconnected Ġto Ġ( if ĠIN LINE FORM 3 Ġitself Ġis Ġa Ġpassage Ġword , Ġwe Ġwill Ġexclude Ġits Ġown Ġposition Ġfrom ĠIN LINE FORM 4 Ġ). ĠWe Ġcan Ġcontrol Ġthe Ġamount Ġof Ġthe Ġextracted Ġresults Ġby Ġsetting Ġthe Ġhyper - param eter ĠIN LINE FORM 5 Ġ: Ġif Ġwe Ġset ĠIN LINE FORM 6 Ġto Ġ0 , Ġinter - word Ġsemantic Ġconnections Ġwill Ġonly Ġexist Ġbetween Ġsyn onyms ; Ġif Ġwe Ġincrease ĠIN LINE FORM 7 Ġ, Ġinter - word Ġsemantic Ġconnections Ġwill Ġexist Ġbetween Ġmore Ġwords . ĠThat Ġis Ġto Ġsay , Ġby Ġincreasing ĠIN LINE FORM 8 Ġwithin Ġa Ġcertain Ġrange , Ġwe Ġcan Ġusually Ġextract Ġmore Ġinter - word Ġsemantic Ġconnections Ġfrom Ġa Ġpassage - question Ġpair , Ġand Ġthus Ġcan Ġprovide Ġthe ĠM RC Ġmodel Ġwith Ġmore Ġgeneral Ġknowledge . ĠHowever , Ġdue Ġto Ġthe Ġcomplexity Ġand Ġdiversity Ġof Ġnatural Ġlanguages , Ġonly Ġa Ġpart Ġof Ġthe Ġextracted Ġresults Ġcan Ġserve Ġas Ġuseful Ġgeneral Ġknowledge , Ġwhile Ġthe Ġrest Ġof Ġthem Ġare Ġuseless Ġfor Ġthe Ġprediction Ġof Ġanswer Ġspans , Ġand Ġthe Ġproportion Ġof Ġthe Ġuseless Ġpart Ġalways Ġrises Ġwhen ĠIN LINE FORM 9 Ġis Ġset Ġlarger . ĠTherefore Ġwe Ġset ĠIN LINE FORM 10 Ġthrough Ġcross Ġvalidation Ġ( i . e . Ġaccording Ġto Ġthe Ġperformance Ġof Ġthe ĠM RC Ġmodel Ġon Ġthe Ġdevelopment Ġexamples ). </s> Know ledge ĠA ided ĠReader </s> In Ġthis Ġsection , Ġwe Ġelaborate Ġour ĠM RC Ġmodel : ĠKnowledge ĠA ided ĠReader Ġ( K AR ). ĠThe Ġkey Ġcomponents Ġof Ġmost Ġexisting ĠM RC Ġmodels Ġare Ġtheir Ġattention Ġmechanisms ĠB IB REF 13 Ġ, Ġwhich Ġare Ġaimed Ġat Ġf using Ġthe Ġassociated Ġrepresentations Ġof Ġeach Ġgiven Ġpassage - question Ġpair . ĠThese Ġattention Ġmechanisms Ġgenerally Ġfall Ġinto Ġtwo Ġcategories : Ġthe Ġfirst Ġone , Ġwhich Ġwe Ġname Ġas Ġmutual Ġattention , Ġis Ġaimed Ġat Ġf using Ġthe Ġquestion Ġrepresentations Ġinto Ġthe Ġpassage Ġrepresentations Ġso Ġas Ġto Ġobtain Ġthe Ġquestion - aware Ġpassage Ġrepresentations ; Ġthe Ġsecond Ġone , Ġwhich Ġwe Ġname Ġas Ġself Ġattention , Ġis Ġaimed Ġat Ġf using Ġthe Ġquestion - aware Ġpassage Ġrepresentations Ġinto Ġthemselves Ġso Ġas Ġto Ġobtain Ġthe Ġfinal Ġpassage Ġrepresentations . ĠAlthough ĠK AR Ġis Ġequipped Ġwith Ġboth Ġcategories , Ġits Ġmost Ġremarkable Ġfeature Ġis Ġthat Ġit Ġexplicitly Ġuses Ġthe Ġgeneral Ġknowledge Ġextracted Ġby Ġthe Ġdata Ġenrichment Ġmethod Ġto Ġassist Ġits Ġattention Ġmechanisms . ĠTherefore Ġwe Ġseparately Ġname Ġthe Ġattention Ġmechanisms Ġof ĠK AR Ġas Ġknowledge Ġaided Ġmutual Ġattention Ġand Ġknowledge Ġaided Ġself Ġattention . </s> Task ĠDefinition </s> Given Ġa Ġpassage ĠIN LINE FORM 0 Ġand Ġa Ġrelevant Ġquestion ĠIN LINE FORM 1 Ġ, Ġthe Ġtask Ġis Ġto Ġpredict Ġan Ġanswer Ġspan ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġ, Ġso Ġthat Ġthe Ġresulting Ġsubsequ ence ĠIN LINE FORM 4 Ġfrom ĠIN LINE FORM 5 Ġis Ġan Ġanswer Ġto ĠIN LINE FORM 6 Ġ. </s> Overall ĠArchitecture </s> As Ġshown Ġin ĠFigure ĠFIG REF 7 Ġ, ĠK AR Ġis Ġan Ġend - to - end ĠM RC Ġmodel Ġconsisting Ġof Ġfive Ġlayers : </s> Lex icon ĠEmb edd ing ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġwords Ġto Ġthe Ġlex icon Ġembed d ings . ĠThe Ġlex icon Ġembed ding Ġof Ġeach Ġword Ġis Ġcomposed Ġof Ġits Ġword Ġembed ding Ġand Ġcharacter Ġembed ding . ĠFor Ġeach Ġword , Ġwe Ġuse Ġthe Ġpre - trained ĠGl o Ve ĠB IB REF 14 Ġword Ġvector Ġas Ġits Ġword Ġembed ding , Ġand Ġobtain Ġits Ġcharacter Ġembed ding Ġwith Ġa ĠConv olution al ĠNeural ĠNetwork Ġ( CNN ) ĠB IB REF 15 Ġ. ĠFor Ġboth Ġthe Ġpassage Ġand Ġthe Ġquestion , Ġwe Ġpass Ġthe Ġconc aten ation Ġof Ġthe Ġword Ġembed d ings Ġand Ġthe Ġcharacter Ġembed d ings Ġthrough Ġa Ġshared Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 0 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġpassage Ġlex icon Ġembed d ings ĠIN LINE FORM 1 Ġand Ġthe Ġquestion Ġlex icon Ġembed d ings ĠIN LINE FORM 2 Ġ. </s> Context ĠEmb edd ing ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġlex icon Ġembed d ings Ġto Ġthe Ġcontext Ġembed d ings . ĠFor Ġboth Ġthe Ġpassage Ġand Ġthe Ġquestion , Ġwe Ġprocess Ġthe Ġlex icon Ġembed d ings Ġ( i . e . ĠIN LINE FORM 0 Ġfor Ġthe Ġpassage Ġand ĠIN LINE FORM 1 Ġfor Ġthe Ġquestion ) Ġwith Ġa Ġshared Ġbid irection al ĠL ST M Ġ( Bi L ST M ) ĠB IB REF 16 Ġ, Ġwhose Ġhidden Ġstate Ġdimension ality Ġis ĠIN LINE FORM 2 Ġ. ĠBy Ġconc aten ating Ġthe Ġforward ĠL ST M Ġoutputs Ġand Ġthe Ġbackward ĠL ST M Ġoutputs , Ġwe Ġobtain Ġthe Ġpassage Ġcontext Ġembed d ings ĠIN LINE FORM 3 Ġand Ġthe Ġquestion Ġcontext Ġembed d ings ĠIN LINE FORM 4 Ġ. </s> Co arse ĠMemory ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġcontext Ġembed d ings Ġto Ġthe Ġcoarse Ġmemories . ĠFirst Ġwe Ġuse Ġknowledge Ġaided Ġmutual Ġattention Ġ( introdu ced Ġlater ) Ġto Ġfuse ĠIN LINE FORM 0 Ġinto ĠIN LINE FORM 1 Ġ, Ġthe Ġoutputs Ġof Ġwhich Ġare Ġrepresented Ġas ĠIN LINE FORM 2 Ġ. ĠThen Ġwe Ġprocess ĠIN LINE FORM 3 Ġwith Ġa ĠBi L ST M , Ġwhose Ġhidden Ġstate Ġdimension ality Ġis ĠIN LINE FORM 4 Ġ. ĠBy Ġconc aten ating Ġthe Ġforward ĠL ST M Ġoutputs Ġand Ġthe Ġbackward ĠL ST M Ġoutputs , Ġwe Ġobtain Ġthe Ġcoarse Ġmemories ĠIN LINE FORM 5 Ġ, Ġwhich Ġare Ġthe Ġquestion - aware Ġpassage Ġrepresentations . </s> Ref ined ĠMemory ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġcoarse Ġmemories Ġto Ġthe Ġrefined Ġmemories . ĠFirst Ġwe Ġuse Ġknowledge Ġaided Ġself Ġattention Ġ( introdu ced Ġlater ) Ġto Ġfuse ĠIN LINE FORM 0 Ġinto Ġthemselves , Ġthe Ġoutputs Ġof Ġwhich Ġare Ġrepresented Ġas ĠIN LINE FORM 1 Ġ. ĠThen Ġwe Ġprocess ĠIN LINE FORM 2 Ġwith Ġa ĠBi L ST M , Ġwhose Ġhidden Ġstate Ġdimension ality Ġis ĠIN LINE FORM 3 Ġ. ĠBy Ġconc aten ating Ġthe Ġforward ĠL ST M Ġoutputs Ġand Ġthe Ġbackward ĠL ST M Ġoutputs , Ġwe Ġobtain Ġthe Ġrefined Ġmemories ĠIN LINE FORM 4 Ġ, Ġwhich Ġare Ġthe Ġfinal Ġpassage Ġrepresentations . </s> Answer ĠSpan ĠPrediction ĠLayer . ĠThis Ġlayer Ġpredicts Ġthe Ġanswer Ġstart Ġposition Ġand Ġthe Ġanswer Ġend Ġposition Ġbased Ġon Ġthe Ġabove Ġlayers . ĠFirst Ġwe Ġobtain Ġthe Ġanswer Ġstart Ġposition Ġdistribution ĠIN LINE FORM 0 Ġ: ĠIN LINE FORM 1 ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters ; ĠIN LINE FORM 3 Ġrepresents Ġthe Ġrefined Ġmemory Ġof Ġeach Ġpassage Ġword ĠIN LINE FORM 4 Ġ( i . e . Ġthe ĠIN LINE FORM 5 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 6 Ġ); ĠIN LINE FORM 7 Ġrepresents Ġthe Ġquestion Ġsummary Ġobtained Ġby Ġperforming Ġan Ġattention Ġpool ing Ġover ĠIN LINE FORM 8 Ġ. ĠThen Ġwe Ġobtain Ġthe Ġanswer Ġend Ġposition Ġdistribution ĠIN LINE FORM 9 Ġ: ĠIN LINE FORM 10 ĠIN LINE FORM 11 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters ; ĠIN LINE FORM 3 Ġrepresents Ġvector Ġconc aten ation . ĠFinally Ġwe Ġconstruct Ġan Ġanswer Ġspan Ġprediction Ġmatrix ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġrepresents Ġthe Ġupper Ġtriangular Ġmatrix Ġof Ġa Ġmatrix ĠIN LINE FORM 6 Ġ. ĠTherefore , Ġfor Ġthe Ġtraining , Ġwe Ġminimize ĠIN LINE FORM 7 Ġon Ġeach Ġtraining Ġexample Ġwhose Ġlabeled Ġanswer Ġspan Ġis ĠIN LINE FORM 8 Ġ; Ġfor Ġthe Ġinference , Ġwe Ġseparately Ġtake Ġthe Ġrow Ġindex Ġand Ġcolumn Ġindex Ġof Ġthe Ġmaximum Ġelement Ġin ĠIN LINE FORM 9 Ġas ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. </s> Know ledge ĠA ided ĠMutual ĠAttention </s> As Ġa Ġpart Ġof Ġthe Ġcoarse Ġmemory Ġlayer , Ġknowledge Ġaided Ġmutual Ġattention Ġis Ġaimed Ġat Ġf using Ġthe Ġquestion Ġcontext Ġembed d ings ĠIN LINE FORM 0 Ġinto Ġthe Ġpassage Ġcontext Ġembed d ings ĠIN LINE FORM 1 Ġ, Ġwhere Ġthe Ġkey Ġproblem Ġis Ġto Ġcalculate Ġthe Ġsimilarity Ġbetween Ġeach Ġpassage Ġcontext Ġembed ding ĠIN LINE FORM 2 Ġ( i . e . Ġthe ĠIN LINE FORM 3 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 4 Ġ) Ġand Ġeach Ġquestion Ġcontext Ġembed ding ĠIN LINE FORM 5 Ġ( i . e . Ġthe ĠIN LINE FORM 6 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 7 Ġ). ĠTo Ġsolve Ġthis Ġproblem , ĠB IB REF 3 Ġproposed Ġa Ġsimilarity Ġfunction : ĠIN LINE FORM 8 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġtrain able Ġparameter ; ĠIN LINE FORM 1 Ġrepresents Ġelement - wise Ġmultiplication . ĠThis Ġsimilarity Ġfunction Ġhas Ġalso Ġbeen Ġadopted Ġby Ġseveral Ġother Ġworks ĠB IB REF 17 Ġ, ĠB IB REF 5 Ġ. ĠHowever , Ġsince Ġcontext Ġembed d ings Ġcontain Ġhigh - level Ġinformation , Ġwe Ġbelieve Ġthat Ġintroducing Ġthe Ġpre - ext racted Ġgeneral Ġknowledge Ġinto Ġthe Ġcalculation Ġof Ġsuch Ġsimilarities Ġwill Ġmake Ġthe Ġresults Ġmore Ġreasonable . ĠTherefore Ġwe Ġmodify Ġthe Ġabove Ġsimilarity Ġfunction Ġto Ġthe Ġfollowing Ġform : ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġenhanced Ġcontext Ġembed ding Ġof Ġa Ġword ĠIN LINE FORM 1 Ġ. ĠWe Ġuse Ġthe Ġpre - ext racted Ġgeneral Ġknowledge Ġto Ġconstruct Ġthe Ġenhanced Ġcontext Ġembed d ings . ĠSpecifically , Ġfor Ġeach Ġword ĠIN LINE FORM 2 Ġ, Ġwhose Ġcontext Ġembed ding Ġis ĠIN LINE FORM 3 Ġ, Ġto Ġconstruct Ġits Ġenhanced Ġcontext Ġembed ding ĠIN LINE FORM 4 Ġ, Ġfirst Ġrecall Ġthat Ġwe Ġhave Ġextracted Ġa Ġset ĠIN LINE FORM 5 Ġ, Ġwhich Ġincludes Ġthe Ġpositions Ġof Ġthe Ġpassage Ġwords Ġthat ĠIN LINE FORM 6 Ġis Ġsem antically Ġconnected Ġto , Ġthus Ġby Ġgathering Ġthe Ġcolumns Ġin ĠIN LINE FORM 7 Ġwhose Ġindexes Ġare Ġgiven Ġby ĠIN LINE FORM 8 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġcontext Ġembed d ings ĠIN LINE FORM 9 Ġ. ĠThen Ġby Ġconstructing Ġa ĠIN LINE FORM 10 Ġ- att ended Ġsummary Ġof ĠIN LINE FORM 11 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġvector ĠIN LINE FORM 12 Ġ( if ĠIN LINE FORM 13 Ġ, Ġwhich Ġmakes ĠIN LINE FORM 14 Ġ, Ġwe Ġwill Ġset ĠIN LINE FORM 15 Ġ): ĠIN LINE FORM 16 ĠIN LINE FORM 17 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters ; ĠIN LINE FORM 3 Ġrepresents Ġthe ĠIN LINE FORM 4 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 5 Ġ. ĠFinally Ġwe Ġpass Ġthe Ġconc aten ation Ġof ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġthrough Ġa Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 8 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġenhanced Ġcontext Ġembed ding ĠIN LINE FORM 9 Ġ. </s> Based Ġon Ġthe Ġmodified Ġsimilarity Ġfunction Ġand Ġthe Ġenhanced Ġcontext Ġembed d ings , Ġto Ġperform Ġknowledge Ġaided Ġmutual Ġattention , Ġfirst Ġwe Ġconstruct Ġa Ġknowledge Ġaided Ġsimilarity Ġmatrix ĠIN LINE FORM 0 Ġ, Ġwhere Ġeach Ġelement ĠIN LINE FORM 1 Ġ. ĠThen Ġfollowing ĠB IB REF 5 Ġ, Ġwe Ġconstruct Ġthe Ġpassage - att ended Ġquestion Ġsumm aries ĠIN LINE FORM 2 Ġand Ġthe Ġquestion - att ended Ġpassage Ġsumm aries ĠIN LINE FORM 3 Ġ: ĠIN LINE FORM 4 ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġrepresents Ġsoft max Ġalong Ġthe Ġrow Ġdimension Ġand ĠIN LINE FORM 1 Ġalong Ġthe Ġcolumn Ġdimension . ĠFinally Ġfollowing ĠB IB REF 17 Ġ, Ġwe Ġpass Ġthe Ġconc aten ation Ġof ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġthrough Ġa Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 6 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġoutputs ĠIN LINE FORM 7 Ġ. </s> Know ledge ĠA ided ĠSelf ĠAttention </s> As Ġa Ġpart Ġof Ġthe Ġrefined Ġmemory Ġlayer , Ġknowledge Ġaided Ġself Ġattention Ġis Ġaimed Ġat Ġf using Ġthe Ġcoarse Ġmemories ĠIN LINE FORM 0 Ġinto Ġthemselves . ĠIf Ġwe Ġsimply Ġfollow Ġthe Ġself Ġattent ions Ġof Ġother Ġworks ĠB IB REF 4 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 17 Ġ, Ġthen Ġfor Ġeach Ġpassage Ġword ĠIN LINE FORM 1 Ġ, Ġwe Ġshould Ġfuse Ġits Ġcoarse Ġmemory ĠIN LINE FORM 2 Ġ( i . e . Ġthe ĠIN LINE FORM 3 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 4 Ġ) Ġwith Ġthe Ġcoarse Ġmemories Ġof Ġall Ġthe Ġother Ġpassage Ġwords . ĠHowever , Ġwe Ġbelieve Ġthat Ġthis Ġis Ġboth Ġunnecessary Ġand Ġdistracting , Ġsince Ġeach Ġpassage Ġword Ġhas Ġnothing Ġto Ġdo Ġwith Ġmany Ġof Ġthe Ġother Ġpassage Ġwords . ĠThus Ġwe Ġuse Ġthe Ġpre - ext racted Ġgeneral Ġknowledge Ġto Ġguarantee Ġthat Ġthe Ġfusion Ġof Ġcoarse Ġmemories Ġfor Ġeach Ġpassage Ġword Ġwill Ġonly Ġinvolve Ġa Ġprecise Ġsubset Ġof Ġthe Ġother Ġpassage Ġwords . ĠSpecifically , Ġfor Ġeach Ġpassage Ġword ĠIN LINE FORM 5 Ġ, Ġwhose Ġcoarse Ġmemory Ġis ĠIN LINE FORM 6 Ġ, Ġto Ġperform Ġthe Ġfusion Ġof Ġcoarse Ġmemories , Ġfirst Ġrecall Ġthat Ġwe Ġhave Ġextracted Ġa Ġset ĠIN LINE FORM 7 Ġ, Ġwhich Ġincludes Ġthe Ġpositions Ġof Ġthe Ġother Ġpassage Ġwords Ġthat ĠIN LINE FORM 8 Ġis Ġsem antically Ġconnected Ġto , Ġthus Ġby Ġgathering Ġthe Ġcolumns Ġin ĠIN LINE FORM 9 Ġwhose Ġindexes Ġare Ġgiven Ġby ĠIN LINE FORM 10 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġcoarse Ġmemories ĠIN LINE FORM 11 Ġ. ĠThen Ġby Ġconstructing Ġa ĠIN LINE FORM 12 Ġ- att ended Ġsummary Ġof ĠIN LINE FORM 13 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġvector ĠIN LINE FORM 14 Ġ( if ĠIN LINE FORM 15 Ġ, Ġwhich Ġmakes ĠIN LINE FORM 16 Ġ, Ġwe Ġwill Ġset ĠIN LINE FORM 17 Ġ): ĠIN LINE FORM 18 ĠIN LINE FORM 19 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters . ĠFinally Ġwe Ġpass Ġthe Ġconc aten ation Ġof ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġthrough Ġa Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 5 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġfusion Ġresult ĠIN LINE FORM 6 Ġ, Ġand Ġfurther Ġthe Ġoutputs ĠIN LINE FORM 7 Ġ. </s> Related ĠWorks </s> Att ention ĠMechan isms . ĠBesides Ġthose Ġmentioned Ġabove , Ġother Ġinteresting Ġattention Ġmechanisms Ġinclude Ġperforming Ġmulti - round Ġalignment Ġto Ġavoid Ġthe Ġproblems Ġof Ġattention Ġredundancy Ġand Ġattention Ġdeficiency ĠB IB REF 20 Ġ, Ġand Ġusing Ġmutual Ġattention Ġas Ġa Ġskip - connect or Ġto Ġdensely Ġconnect Ġpair wise Ġlayers ĠB IB REF 21 Ġ. </s> Data ĠAug mentation . ĠIt Ġis Ġproved Ġthat Ġproperly Ġaugment ing Ġtraining Ġexamples Ġcan Ġimprove Ġthe Ġperformance Ġof ĠM RC Ġmodels . ĠFor Ġexample , ĠB IB REF 22 Ġtrained Ġa Ġgener ative Ġmodel Ġto Ġgenerate Ġquestions Ġbased Ġon Ġunl abel ed Ġtext , Ġwhich Ġsubstantially Ġboosted Ġtheir Ġperformance ; ĠB IB REF 5 Ġtrained Ġa Ġback - and - forth Ġtranslation Ġmodel Ġto Ġparaph rase Ġtraining Ġexamples , Ġwhich Ġbrought Ġthem Ġa Ġsignificant Ġperformance Ġgain . </s> Multi - step ĠReason ing . ĠInspired Ġby Ġthe Ġfact Ġthat Ġhuman Ġbeings Ġare Ġcapable Ġof Ġunderstanding Ġcomplex Ġdocuments Ġby Ġreading Ġthem Ġover Ġand Ġover Ġagain , Ġmulti - step Ġreasoning Ġwas Ġproposed Ġto Ġbetter Ġdeal Ġwith Ġdifficult ĠM RC Ġtasks . ĠFor Ġexample , ĠB IB REF 23 Ġused Ġreinforcement Ġlearning Ġto Ġdynamically Ġdetermine Ġthe Ġnumber Ġof Ġreasoning Ġsteps ; ĠB IB REF 19 Ġfixed Ġthe Ġnumber Ġof Ġreasoning Ġsteps , Ġbut Ġused Ġsto ch astic Ġdrop out Ġin Ġthe Ġoutput Ġlayer Ġto Ġavoid Ġstep Ġbias . </s> L ingu istic ĠEmb edd ings . ĠIt Ġis Ġboth Ġeasy Ġand Ġeffective Ġto Ġincorporate Ġlinguistic Ġembed d ings Ġinto Ġthe Ġinput Ġlayer Ġof ĠM RC Ġmodels . ĠFor Ġexample , ĠB IB REF 24 Ġand ĠB IB REF 19 Ġused ĠPOS Ġembed d ings Ġand ĠN ER Ġembed d ings Ġto Ġconstruct Ġtheir Ġinput Ġembed d ings ; ĠB IB REF 25 Ġused Ġstructural Ġembed d ings Ġbased Ġon Ġparsing Ġtrees Ġto Ġconstructed Ġtheir Ġinput Ġembed d ings . </s> Transfer ĠLearning . ĠSeveral Ġrecent Ġbreakthrough s Ġin ĠM RC Ġbenefit Ġfrom Ġfeature - based Ġtransfer Ġlearning ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġand Ġfine - tun ing - based Ġtransfer Ġlearning ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, Ġwhich Ġare Ġbased Ġon Ġcertain Ġword - level Ġor Ġsentence - level Ġmodels Ġpre - trained Ġon Ġlarge Ġexternal Ġcorpor a Ġin Ġcertain Ġsupervised Ġor Ġun super vised Ġmanners . </s> Exper imental ĠSettings </s> M RC ĠDat as et . ĠThe ĠM RC Ġdataset Ġused Ġin Ġthis Ġpaper Ġis ĠS Qu AD Ġ1 . 1 , Ġwhich Ġcontains Ġover ĠIN LINE FORM 0 Ġpassage - question Ġpairs Ġand Ġhas Ġbeen Ġrandomly Ġpartition ed Ġinto Ġthree Ġparts : Ġa Ġtraining Ġset Ġ( ĠIN LINE FORM 1 Ġ), Ġa Ġdevelopment Ġset Ġ( ĠIN LINE FORM 2 Ġ), Ġand Ġa Ġtest Ġset Ġ( ĠIN LINE FORM 3 Ġ). ĠBesides , Ġwe Ġalso Ġuse Ġtwo Ġof Ġits Ġadvers arial Ġsets , Ġnamely ĠAdd Sent Ġand ĠAdd One Sent ĠB IB REF 6 Ġ, Ġto Ġevaluate Ġthe Ġrobust ness Ġto Ġnoise Ġof ĠM RC Ġmodels . ĠThe Ġpassages Ġin Ġthe Ġadvers arial Ġsets Ġcontain Ġmisleading Ġsentences , Ġwhich Ġare Ġaimed Ġat Ġdistracting ĠM RC Ġmodels . ĠSpecifically , Ġeach Ġpassage Ġin ĠAdd Sent Ġcontains Ġseveral Ġsentences Ġthat Ġare Ġsimilar Ġto Ġthe Ġquestion Ġbut Ġnot Ġcontradictory Ġto Ġthe Ġanswer , Ġwhile Ġeach Ġpassage Ġin ĠAdd One Sent Ġcontains Ġa Ġhuman - approved Ġrandom Ġsentence Ġthat Ġmay Ġbe Ġunrelated Ġto Ġthe Ġpassage . </s> Im plementation ĠDetails . ĠWe Ġtoken ize Ġthe ĠM RC Ġdataset Ġwith Ġspa Cy Ġ2 . 0 . 13 ĠB IB REF 30 Ġ, Ġmanipulate ĠWord Net Ġ3 . 0 Ġwith ĠNL T K Ġ3 . 3 , Ġand Ġimplement ĠK AR Ġwith ĠT ensor Flow Ġ1 . 11 . 0 ĠB IB REF 31 Ġ. ĠFor Ġthe Ġdata Ġenrichment Ġmethod , Ġwe Ġset Ġthe Ġhyper - param eter ĠIN LINE FORM 0 Ġto Ġ3 . ĠFor Ġthe Ġdense Ġlayers Ġand Ġthe ĠBi L ST Ms , Ġwe Ġset Ġthe Ġdimension ality Ġunit ĠIN LINE FORM 1 Ġto Ġ600 . ĠFor Ġmodel Ġoptimization , Ġwe Ġapply Ġthe ĠAdam ĠB IB REF 32 Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof ĠIN LINE FORM 2 Ġand Ġa Ġmini - batch Ġsize Ġof Ġ32 . ĠFor Ġmodel Ġevaluation , Ġwe Ġuse ĠEx act ĠMatch Ġ( EM ) Ġand ĠF 1 Ġscore Ġas Ġevaluation Ġmetrics . ĠTo Ġavoid Ġover fitting , Ġwe Ġapply Ġdrop out ĠB IB REF 33 Ġto Ġthe Ġdense Ġlayers Ġand Ġthe ĠBi L ST Ms Ġwith Ġa Ġdrop out Ġrate Ġof ĠIN LINE FORM 3 Ġ. ĠTo Ġboost Ġthe Ġperformance , Ġwe Ġapply Ġexponential Ġmoving Ġaverage Ġwith Ġa Ġdecay Ġrate Ġof ĠIN LINE FORM 4 Ġ. </s> Model ĠComparison Ġin Ġboth ĠPerformance Ġand Ġthe ĠRob ust ness Ġto ĠNoise </s> We Ġcompare ĠK AR Ġwith Ġother ĠM RC Ġmodels Ġin Ġboth Ġperformance Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠSpecifically , Ġwe Ġnot Ġonly Ġevaluate Ġthe Ġperformance Ġof ĠK AR Ġon Ġthe Ġdevelopment Ġset Ġand Ġthe Ġtest Ġset , Ġbut Ġalso Ġdo Ġthis Ġon Ġthe Ġadvers arial Ġsets . ĠAs Ġfor Ġthe Ġcomparative Ġobjects , Ġwe Ġonly Ġconsider Ġthe Ġsingle ĠM RC Ġmodels Ġthat Ġrank Ġin Ġthe Ġtop Ġ20 Ġon Ġthe ĠS Qu AD Ġ1 . 1 Ġleader Ġboard Ġand Ġhave Ġreported Ġtheir Ġperformance Ġon Ġthe Ġadvers arial Ġsets . ĠThere Ġare Ġtotally Ġfive Ġsuch Ġcomparative Ġobjects , Ġwhich Ġcan Ġbe Ġconsidered Ġas Ġrepresentatives Ġof Ġthe Ġstate - of - the - art ĠM RC Ġmodels . ĠAs Ġshown Ġin ĠTable ĠT AB REF 12 Ġ, Ġon Ġthe Ġdevelopment Ġset Ġand Ġthe Ġtest Ġset , Ġthe Ġperformance Ġof ĠK AR Ġis Ġon Ġpar Ġwith Ġthat Ġof Ġthe Ġstate - of - the - art ĠM RC Ġmodels ; Ġon Ġthe Ġadvers arial Ġsets , ĠK AR Ġoutper forms Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġby Ġa Ġlarge Ġmargin . ĠThat Ġis Ġto Ġsay , ĠK AR Ġis Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġand Ġsignificantly Ġmore Ġrobust Ġto Ġnoise Ġthan Ġthem . </s> To Ġverify Ġthe Ġeffectiveness Ġof Ġgeneral Ġknowledge , Ġwe Ġfirst Ġstudy Ġthe Ġrelationship Ġbetween Ġthe Ġamount Ġof Ġgeneral Ġknowledge Ġand Ġthe Ġperformance Ġof ĠK AR . ĠAs Ġshown Ġin ĠTable ĠT AB REF 13 Ġ, Ġby Ġincreasing ĠIN LINE FORM 0 Ġfrom Ġ0 Ġto Ġ5 Ġin Ġthe Ġdata Ġenrichment Ġmethod , Ġthe Ġamount Ġof Ġgeneral Ġknowledge Ġrises Ġmon oton ically , Ġbut Ġthe Ġperformance Ġof ĠK AR Ġfirst Ġrises Ġuntil ĠIN LINE FORM 1 Ġreaches Ġ3 Ġand Ġthen Ġdrops Ġdown . ĠThen Ġwe Ġconduct Ġan Ġab lation Ġstudy Ġby Ġreplacing Ġthe Ġknowledge Ġaided Ġattention Ġmechanisms Ġwith Ġthe Ġmutual Ġattention Ġproposed Ġby ĠB IB REF 3 Ġand Ġthe Ġself Ġattention Ġproposed Ġby ĠB IB REF 4 Ġseparately , Ġand Ġfind Ġthat Ġthe ĠF 1 Ġscore Ġof ĠK AR Ġdrops Ġby ĠIN LINE FORM 2 Ġon Ġthe Ġdevelopment Ġset , ĠIN LINE FORM 3 Ġon ĠAdd Sent , Ġand ĠIN LINE FORM 4 Ġon ĠAdd One Sent . ĠFinally Ġwe Ġfind Ġthat Ġafter Ġonly Ġone Ġepoch Ġof Ġtraining , ĠK AR Ġalready Ġachieves Ġan ĠEM Ġof ĠIN LINE FORM 5 Ġand Ġan ĠF 1 Ġscore Ġof ĠIN LINE FORM 6 Ġon Ġthe Ġdevelopment Ġset , Ġwhich Ġis Ġeven Ġbetter Ġthan Ġthe Ġfinal Ġperformance Ġof Ġseveral Ġstrong Ġbas elines , Ġsuch Ġas ĠDC N Ġ( EM Ġ/ ĠF 1 : ĠIN LINE FORM 7 Ġ/ ĠIN LINE FORM 8 Ġ) ĠB IB REF 36 Ġand ĠBi DA F Ġ( EM Ġ/ ĠF 1 : ĠIN LINE FORM 9 Ġ/ ĠIN LINE FORM 10 Ġ) ĠB IB REF 3 Ġ. ĠThe Ġabove Ġempirical Ġfindings Ġimply Ġthat Ġgeneral Ġknowledge Ġindeed Ġplays Ġan Ġeffective Ġrole Ġin ĠK AR . </s> To Ġdemonstrate Ġthe Ġadvantage Ġof Ġour Ġexplicit Ġway Ġto Ġutilize Ġgeneral Ġknowledge Ġover Ġthe Ġexisting Ġimplicit Ġway , Ġwe Ġcompare Ġthe Ġperformance Ġof ĠK AR Ġwith Ġthat Ġreported Ġby ĠB IB REF 10 Ġ, Ġwhich Ġused Ġan Ġencoding - based Ġmethod Ġto Ġutilize Ġthe Ġgeneral Ġknowledge Ġdynamically Ġretrieved Ġfrom ĠWikipedia Ġand ĠConcept Net . ĠSince Ġtheir Ġbest Ġmodel Ġonly Ġachieved Ġan ĠEM Ġof ĠIN LINE FORM 0 Ġand Ġan ĠF 1 Ġscore Ġof ĠIN LINE FORM 1 Ġon Ġthe Ġdevelopment Ġset , Ġwhich Ġis Ġmuch Ġlower Ġthan Ġthe Ġperformance Ġof ĠK AR , Ġwe Ġhave Ġgood Ġreason Ġto Ġbelieve Ġthat Ġour Ġexplicit Ġway Ġworks Ġbetter Ġthan Ġthe Ġexisting Ġimplicit Ġway . </s> Model ĠComparison Ġin Ġthe ĠHunger Ġfor ĠData </s> We Ġcompare ĠK AR Ġwith Ġother ĠM RC Ġmodels Ġin Ġthe Ġhunger Ġfor Ġdata . ĠSpecifically , Ġinstead Ġof Ġusing Ġall Ġthe Ġtraining Ġexamples , Ġwe Ġproduce Ġseveral Ġtraining Ġsubs ets Ġ( i . e . Ġsubs ets Ġof Ġthe Ġtraining Ġexamples ) Ġso Ġas Ġto Ġstudy Ġthe Ġrelationship Ġbetween Ġthe Ġproportion Ġof Ġthe Ġavailable Ġtraining Ġexamples Ġand Ġthe Ġperformance . ĠWe Ġproduce Ġeach Ġtraining Ġsubset Ġby Ġsampling Ġa Ġspecific Ġnumber Ġof Ġquestions Ġfrom Ġall Ġthe Ġquestions Ġrelevant Ġto Ġeach Ġpassage . ĠBy Ġseparately Ġsampling Ġ1 , Ġ2 , Ġ3 , Ġand Ġ4 Ġquestions Ġon Ġeach Ġpassage , Ġwe Ġobtain Ġfour Ġtraining Ġsubs ets , Ġwhich Ġseparately Ġcontain ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġof Ġthe Ġtraining Ġexamples . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 15 Ġ, Ġwith ĠK AR , ĠSAN Ġ( re - im ple mented ), Ġand ĠQ AN et Ġ( re - im ple mented Ġwithout Ġdata Ġaug mentation ) Ġtrained Ġon Ġthese Ġtraining Ġsubs ets , Ġwe Ġevaluate Ġtheir Ġperformance Ġon Ġthe Ġdevelopment Ġset , Ġand Ġfind Ġthat ĠK AR Ġperforms Ġmuch Ġbetter Ġthan ĠSAN Ġand ĠQ AN et . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 16 Ġand ĠFigure ĠFIG REF 17 Ġ, Ġwith Ġthe Ġabove ĠK AR , ĠSAN , Ġand ĠQ AN et Ġtrained Ġon Ġthe Ġsame Ġtraining Ġsubs ets , Ġwe Ġalso Ġevaluate Ġtheir Ġperformance Ġon Ġthe Ġadvers arial Ġsets , Ġand Ġstill Ġfind Ġthat ĠK AR Ġperforms Ġmuch Ġbetter Ġthan ĠSAN Ġand ĠQ AN et . ĠThat Ġis Ġto Ġsay , Ġwhen Ġonly Ġa Ġsubset Ġof Ġthe Ġtraining Ġexamples Ġare Ġavailable , ĠK AR Ġoutper forms Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġby Ġa Ġlarge Ġmargin , Ġand Ġis Ġstill Ġreasonably Ġrobust Ġto Ġnoise . </s> Analysis </s> According Ġto Ġthe Ġexperimental Ġresults , ĠK AR Ġis Ġnot Ġonly Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġbut Ġalso Ġsuperior Ġto Ġthem Ġin Ġterms Ġof Ġboth Ġthe Ġhunger Ġfor Ġdata Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠThe Ġreasons Ġfor Ġthese Ġachievements , Ġwe Ġbelieve , Ġare Ġas Ġfollows : </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġinnov atively Ġintegrate Ġthe Ġneural Ġnetworks Ġof ĠM RC Ġmodels Ġwith Ġthe Ġgeneral Ġknowledge Ġof Ġhuman Ġbeings . ĠSpecifically , Ġinter - word Ġsemantic Ġconnections Ġare Ġfirst Ġextracted Ġfrom Ġeach Ġgiven Ġpassage - question Ġpair Ġby Ġa ĠWord Net - based Ġdata Ġenrichment Ġmethod , Ġand Ġthen Ġprovided Ġas Ġgeneral Ġknowledge Ġto Ġan Ġend - to - end ĠM RC Ġmodel Ġnamed Ġas ĠKnowledge ĠA ided ĠReader Ġ( K AR ), Ġwhich Ġexplicitly Ġuses Ġthe Ġgeneral Ġknowledge Ġto Ġassist Ġits Ġattention Ġmechanisms . ĠExperimental Ġresults Ġshow Ġthat ĠK AR Ġis Ġnot Ġonly Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġbut Ġalso Ġsuperior Ġto Ġthem Ġin Ġterms Ġof Ġboth Ġthe Ġhunger Ġfor Ġdata Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠIn Ġthe Ġfuture , Ġwe Ġplan Ġto Ġuse Ġsome Ġlarger Ġknowledge Ġbases , Ġsuch Ġas ĠConcept Net Ġand ĠFree base , Ġto Ġimprove Ġthe Ġquality Ġand Ġscope Ġof Ġthe Ġgeneral Ġknowledge . </s> Acknowled gments </s> This Ġwork Ġis Ġpartially Ġsupported Ġby Ġa Ġresearch Ġdonation Ġfrom Ġi FL Y TE K ĠCo ., ĠLtd ., ĠHe fe i , ĠChina , Ġand Ġa Ġdiscovery Ġgrant Ġfrom ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġ( NS ERC ) Ġof ĠCanada .
Introduction Machine Reading Comprehension (MRC), as the name suggests, requires a machine to read a passage and answer its relevant questions. Since the answer to each question is supposed to stem from the corresponding passage, a common MRC solution is to develop a neural-network-based MRC model that predicts an answer span (i.e. the answer start position and the answer end position) from the passage of each given passage-question pair. To facilitate the explorations and innovations in this area, many MRC datasets have been established, such as SQuAD BIBREF0 , MS MARCO BIBREF1 , and TriviaQA BIBREF2 . Consequently, many pioneering MRC models have been proposed, such as BiDAF BIBREF3 , R-NET BIBREF4 , and QANet BIBREF5 . According to the leader board of SQuAD, the state-of-the-art MRC models have achieved the same performance as human beings. However, does this imply that they have possessed the same reading comprehension ability as human beings? OF COURSE NOT. There is a huge gap between MRC models and human beings, which is mainly reflected in the hunger for data and the robustness to noise. On the one hand, developing MRC models requires a large amount of training examples (i.e. the passage-question pairs labeled with answer spans), while human beings can achieve good performance on evaluation examples (i.e. the passage-question pairs to address) without training examples. On the other hand, BIBREF6 revealed that intentionally injected noise (e.g. misleading sentences) in evaluation examples causes the performance of MRC models to drop significantly, while human beings are far less likely to suffer from this. The reason for these phenomena, we believe, is that MRC models can only utilize the knowledge contained in each given passage-question pair, but in addition to this, human beings can also utilize general knowledge. A typical category of general knowledge is inter-word semantic connections. As shown in Table TABREF1 , such general knowledge is essential to the reading comprehension ability of human beings. A promising strategy to bridge the gap mentioned above is to integrate the neural networks of MRC models with the general knowledge of human beings. To this end, it is necessary to solve two problems: extracting general knowledge from passage-question pairs and utilizing the extracted general knowledge in the prediction of answer spans. The first problem can be solved with knowledge bases, which store general knowledge in structured forms. A broad variety of knowledge bases are available, such as WordNet BIBREF7 storing semantic knowledge, ConceptNet BIBREF8 storing commonsense knowledge, and Freebase BIBREF9 storing factoid knowledge. In this paper, we limit the scope of general knowledge to inter-word semantic connections, and thus use WordNet as our knowledge base. The existing way to solve the second problem is to encode general knowledge in vector space so that the encoding results can be used to enhance the lexical or contextual representations of words BIBREF10 , BIBREF11 . However, this is an implicit way to utilize general knowledge, since in this way we can neither understand nor control the functioning of general knowledge. In this paper, we discard the existing implicit way and instead explore an explicit (i.e. understandable and controllable) way to utilize general knowledge. The contribution of this paper is two-fold. On the one hand, we propose a data enrichment method, which uses WordNet to extract inter-word semantic connections as general knowledge from each given passage-question pair. On the other hand, we propose an end-to-end MRC model named as Knowledge Aided Reader (KAR), which explicitly uses the above extracted general knowledge to assist its attention mechanisms. Based on the data enrichment method, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them. When only a subset ( INLINEFORM0 – INLINEFORM1 ) of the training examples are available, KAR outperforms the state-of-the-art MRC models by a large margin, and is still reasonably robust to noise. Data Enrichment Method In this section, we elaborate a WordNet-based data enrichment method, which is aimed at extracting inter-word semantic connections from each passage-question pair in our MRC dataset. The extraction is performed in a controllable manner, and the extracted results are provided as general knowledge to our MRC model. Semantic Relation Chain WordNet is a lexical database of English, where words are organized into synsets according to their senses. A synset is a set of words expressing the same sense so that a word having multiple senses belongs to multiple synsets, with each synset corresponding to a sense. Synsets are further related to each other through semantic relations. According to the WordNet interface provided by NLTK BIBREF12 , there are totally sixteen types of semantic relations (e.g. hypernyms, hyponyms, holonyms, meronyms, attributes, etc.). Based on synset and semantic relation, we define a new concept: semantic relation chain. A semantic relation chain is a concatenated sequence of semantic relations, which links a synset to another synset. For example, the synset “keratin.n.01” is related to the synset “feather.n.01” through the semantic relation “substance holonym”, the synset “feather.n.01” is related to the synset “bird.n.01” through the semantic relation “part holonym”, and the synset “bird.n.01” is related to the synset “parrot.n.01” through the semantic relation “hyponym”, thus “substance holonym INLINEFORM0 part holonym INLINEFORM1 hyponym” is a semantic relation chain, which links the synset “keratin.n.01” to the synset “parrot.n.01”. We name each semantic relation in a semantic relation chain as a hop, therefore the above semantic relation chain is a 3-hop chain. By the way, each single semantic relation is equivalent to a 1-hop chain. Inter-word Semantic Connection The key problem in the data enrichment method is determining whether a word is semantically connected to another word. If so, we say that there exists an inter-word semantic connection between them. To solve this problem, we define another new concept: the extended synsets of a word. Given a word INLINEFORM0 , whose synsets are represented as a set INLINEFORM1 , we use another set INLINEFORM2 to represent its extended synsets, which includes all the synsets that are in INLINEFORM3 or that can be linked to from INLINEFORM4 through semantic relation chains. Theoretically, if there is no limitation on semantic relation chains, INLINEFORM5 will include all the synsets in WordNet, which is meaningless in most situations. Therefore, we use a hyper-parameter INLINEFORM6 to represent the permitted maximum hop count of semantic relation chains. That is to say, only the chains having no more than INLINEFORM7 hops can be used to construct INLINEFORM8 so that INLINEFORM9 becomes a function of INLINEFORM10 : INLINEFORM11 (if INLINEFORM12 , we will have INLINEFORM13 ). Based on the above statements, we formulate a heuristic rule for determining inter-word semantic connections: a word INLINEFORM14 is semantically connected to another word INLINEFORM15 if and only if INLINEFORM16 . General Knowledge Extraction Given a passage-question pair, the inter-word semantic connections that connect any word to any passage word are regarded as the general knowledge we need to extract. Considering the requirements of our MRC model, we only extract the positional information of such inter-word semantic connections. Specifically, for each word INLINEFORM0 , we extract a set INLINEFORM1 , which includes the positions of the passage words that INLINEFORM2 is semantically connected to (if INLINEFORM3 itself is a passage word, we will exclude its own position from INLINEFORM4 ). We can control the amount of the extracted results by setting the hyper-parameter INLINEFORM5 : if we set INLINEFORM6 to 0, inter-word semantic connections will only exist between synonyms; if we increase INLINEFORM7 , inter-word semantic connections will exist between more words. That is to say, by increasing INLINEFORM8 within a certain range, we can usually extract more inter-word semantic connections from a passage-question pair, and thus can provide the MRC model with more general knowledge. However, due to the complexity and diversity of natural languages, only a part of the extracted results can serve as useful general knowledge, while the rest of them are useless for the prediction of answer spans, and the proportion of the useless part always rises when INLINEFORM9 is set larger. Therefore we set INLINEFORM10 through cross validation (i.e. according to the performance of the MRC model on the development examples). Knowledge Aided Reader In this section, we elaborate our MRC model: Knowledge Aided Reader (KAR). The key components of most existing MRC models are their attention mechanisms BIBREF13 , which are aimed at fusing the associated representations of each given passage-question pair. These attention mechanisms generally fall into two categories: the first one, which we name as mutual attention, is aimed at fusing the question representations into the passage representations so as to obtain the question-aware passage representations; the second one, which we name as self attention, is aimed at fusing the question-aware passage representations into themselves so as to obtain the final passage representations. Although KAR is equipped with both categories, its most remarkable feature is that it explicitly uses the general knowledge extracted by the data enrichment method to assist its attention mechanisms. Therefore we separately name the attention mechanisms of KAR as knowledge aided mutual attention and knowledge aided self attention. Task Definition Given a passage INLINEFORM0 and a relevant question INLINEFORM1 , the task is to predict an answer span INLINEFORM2 , where INLINEFORM3 , so that the resulting subsequence INLINEFORM4 from INLINEFORM5 is an answer to INLINEFORM6 . Overall Architecture As shown in Figure FIGREF7 , KAR is an end-to-end MRC model consisting of five layers: Lexicon Embedding Layer. This layer maps the words to the lexicon embeddings. The lexicon embedding of each word is composed of its word embedding and character embedding. For each word, we use the pre-trained GloVe BIBREF14 word vector as its word embedding, and obtain its character embedding with a Convolutional Neural Network (CNN) BIBREF15 . For both the passage and the question, we pass the concatenation of the word embeddings and the character embeddings through a shared dense layer with ReLU activation, whose output dimensionality is INLINEFORM0 . Therefore we obtain the passage lexicon embeddings INLINEFORM1 and the question lexicon embeddings INLINEFORM2 . Context Embedding Layer. This layer maps the lexicon embeddings to the context embeddings. For both the passage and the question, we process the lexicon embeddings (i.e. INLINEFORM0 for the passage and INLINEFORM1 for the question) with a shared bidirectional LSTM (BiLSTM) BIBREF16 , whose hidden state dimensionality is INLINEFORM2 . By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the passage context embeddings INLINEFORM3 and the question context embeddings INLINEFORM4 . Coarse Memory Layer. This layer maps the context embeddings to the coarse memories. First we use knowledge aided mutual attention (introduced later) to fuse INLINEFORM0 into INLINEFORM1 , the outputs of which are represented as INLINEFORM2 . Then we process INLINEFORM3 with a BiLSTM, whose hidden state dimensionality is INLINEFORM4 . By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the coarse memories INLINEFORM5 , which are the question-aware passage representations. Refined Memory Layer. This layer maps the coarse memories to the refined memories. First we use knowledge aided self attention (introduced later) to fuse INLINEFORM0 into themselves, the outputs of which are represented as INLINEFORM1 . Then we process INLINEFORM2 with a BiLSTM, whose hidden state dimensionality is INLINEFORM3 . By concatenating the forward LSTM outputs and the backward LSTM outputs, we obtain the refined memories INLINEFORM4 , which are the final passage representations. Answer Span Prediction Layer. This layer predicts the answer start position and the answer end position based on the above layers. First we obtain the answer start position distribution INLINEFORM0 : INLINEFORM1 INLINEFORM2 where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents the refined memory of each passage word INLINEFORM4 (i.e. the INLINEFORM5 -th column in INLINEFORM6 ); INLINEFORM7 represents the question summary obtained by performing an attention pooling over INLINEFORM8 . Then we obtain the answer end position distribution INLINEFORM9 : INLINEFORM10 INLINEFORM11 where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents vector concatenation. Finally we construct an answer span prediction matrix INLINEFORM4 , where INLINEFORM5 represents the upper triangular matrix of a matrix INLINEFORM6 . Therefore, for the training, we minimize INLINEFORM7 on each training example whose labeled answer span is INLINEFORM8 ; for the inference, we separately take the row index and column index of the maximum element in INLINEFORM9 as INLINEFORM10 and INLINEFORM11 . Knowledge Aided Mutual Attention As a part of the coarse memory layer, knowledge aided mutual attention is aimed at fusing the question context embeddings INLINEFORM0 into the passage context embeddings INLINEFORM1 , where the key problem is to calculate the similarity between each passage context embedding INLINEFORM2 (i.e. the INLINEFORM3 -th column in INLINEFORM4 ) and each question context embedding INLINEFORM5 (i.e. the INLINEFORM6 -th column in INLINEFORM7 ). To solve this problem, BIBREF3 proposed a similarity function: INLINEFORM8 where INLINEFORM0 is a trainable parameter; INLINEFORM1 represents element-wise multiplication. This similarity function has also been adopted by several other works BIBREF17 , BIBREF5 . However, since context embeddings contain high-level information, we believe that introducing the pre-extracted general knowledge into the calculation of such similarities will make the results more reasonable. Therefore we modify the above similarity function to the following form: INLINEFORM2 where INLINEFORM0 represents the enhanced context embedding of a word INLINEFORM1 . We use the pre-extracted general knowledge to construct the enhanced context embeddings. Specifically, for each word INLINEFORM2 , whose context embedding is INLINEFORM3 , to construct its enhanced context embedding INLINEFORM4 , first recall that we have extracted a set INLINEFORM5 , which includes the positions of the passage words that INLINEFORM6 is semantically connected to, thus by gathering the columns in INLINEFORM7 whose indexes are given by INLINEFORM8 , we obtain the matching context embeddings INLINEFORM9 . Then by constructing a INLINEFORM10 -attended summary of INLINEFORM11 , we obtain the matching vector INLINEFORM12 (if INLINEFORM13 , which makes INLINEFORM14 , we will set INLINEFORM15 ): INLINEFORM16 INLINEFORM17 where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters; INLINEFORM3 represents the INLINEFORM4 -th column in INLINEFORM5 . Finally we pass the concatenation of INLINEFORM6 and INLINEFORM7 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM8 . Therefore we obtain the enhanced context embedding INLINEFORM9 . Based on the modified similarity function and the enhanced context embeddings, to perform knowledge aided mutual attention, first we construct a knowledge aided similarity matrix INLINEFORM0 , where each element INLINEFORM1 . Then following BIBREF5 , we construct the passage-attended question summaries INLINEFORM2 and the question-attended passage summaries INLINEFORM3 : INLINEFORM4 INLINEFORM5 where INLINEFORM0 represents softmax along the row dimension and INLINEFORM1 along the column dimension. Finally following BIBREF17 , we pass the concatenation of INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM6 . Therefore we obtain the outputs INLINEFORM7 . Knowledge Aided Self Attention As a part of the refined memory layer, knowledge aided self attention is aimed at fusing the coarse memories INLINEFORM0 into themselves. If we simply follow the self attentions of other works BIBREF4 , BIBREF18 , BIBREF19 , BIBREF17 , then for each passage word INLINEFORM1 , we should fuse its coarse memory INLINEFORM2 (i.e. the INLINEFORM3 -th column in INLINEFORM4 ) with the coarse memories of all the other passage words. However, we believe that this is both unnecessary and distracting, since each passage word has nothing to do with many of the other passage words. Thus we use the pre-extracted general knowledge to guarantee that the fusion of coarse memories for each passage word will only involve a precise subset of the other passage words. Specifically, for each passage word INLINEFORM5 , whose coarse memory is INLINEFORM6 , to perform the fusion of coarse memories, first recall that we have extracted a set INLINEFORM7 , which includes the positions of the other passage words that INLINEFORM8 is semantically connected to, thus by gathering the columns in INLINEFORM9 whose indexes are given by INLINEFORM10 , we obtain the matching coarse memories INLINEFORM11 . Then by constructing a INLINEFORM12 -attended summary of INLINEFORM13 , we obtain the matching vector INLINEFORM14 (if INLINEFORM15 , which makes INLINEFORM16 , we will set INLINEFORM17 ): INLINEFORM18 INLINEFORM19 where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are trainable parameters. Finally we pass the concatenation of INLINEFORM3 and INLINEFORM4 through a dense layer with ReLU activation, whose output dimensionality is INLINEFORM5 . Therefore we obtain the fusion result INLINEFORM6 , and further the outputs INLINEFORM7 . Related Works Attention Mechanisms. Besides those mentioned above, other interesting attention mechanisms include performing multi-round alignment to avoid the problems of attention redundancy and attention deficiency BIBREF20 , and using mutual attention as a skip-connector to densely connect pairwise layers BIBREF21 . Data Augmentation. It is proved that properly augmenting training examples can improve the performance of MRC models. For example, BIBREF22 trained a generative model to generate questions based on unlabeled text, which substantially boosted their performance; BIBREF5 trained a back-and-forth translation model to paraphrase training examples, which brought them a significant performance gain. Multi-step Reasoning. Inspired by the fact that human beings are capable of understanding complex documents by reading them over and over again, multi-step reasoning was proposed to better deal with difficult MRC tasks. For example, BIBREF23 used reinforcement learning to dynamically determine the number of reasoning steps; BIBREF19 fixed the number of reasoning steps, but used stochastic dropout in the output layer to avoid step bias. Linguistic Embeddings. It is both easy and effective to incorporate linguistic embeddings into the input layer of MRC models. For example, BIBREF24 and BIBREF19 used POS embeddings and NER embeddings to construct their input embeddings; BIBREF25 used structural embeddings based on parsing trees to constructed their input embeddings. Transfer Learning. Several recent breakthroughs in MRC benefit from feature-based transfer learning BIBREF26 , BIBREF27 and fine-tuning-based transfer learning BIBREF28 , BIBREF29 , which are based on certain word-level or sentence-level models pre-trained on large external corpora in certain supervised or unsupervised manners. Experimental Settings MRC Dataset. The MRC dataset used in this paper is SQuAD 1.1, which contains over INLINEFORM0 passage-question pairs and has been randomly partitioned into three parts: a training set ( INLINEFORM1 ), a development set ( INLINEFORM2 ), and a test set ( INLINEFORM3 ). Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models. The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models. Specifically, each passage in AddSent contains several sentences that are similar to the question but not contradictory to the answer, while each passage in AddOneSent contains a human-approved random sentence that may be unrelated to the passage. Implementation Details. We tokenize the MRC dataset with spaCy 2.0.13 BIBREF30 , manipulate WordNet 3.0 with NLTK 3.3, and implement KAR with TensorFlow 1.11.0 BIBREF31 . For the data enrichment method, we set the hyper-parameter INLINEFORM0 to 3. For the dense layers and the BiLSTMs, we set the dimensionality unit INLINEFORM1 to 600. For model optimization, we apply the Adam BIBREF32 optimizer with a learning rate of INLINEFORM2 and a mini-batch size of 32. For model evaluation, we use Exact Match (EM) and F1 score as evaluation metrics. To avoid overfitting, we apply dropout BIBREF33 to the dense layers and the BiLSTMs with a dropout rate of INLINEFORM3 . To boost the performance, we apply exponential moving average with a decay rate of INLINEFORM4 . Model Comparison in both Performance and the Robustness to Noise We compare KAR with other MRC models in both performance and the robustness to noise. Specifically, we not only evaluate the performance of KAR on the development set and the test set, but also do this on the adversarial sets. As for the comparative objects, we only consider the single MRC models that rank in the top 20 on the SQuAD 1.1 leader board and have reported their performance on the adversarial sets. There are totally five such comparative objects, which can be considered as representatives of the state-of-the-art MRC models. As shown in Table TABREF12 , on the development set and the test set, the performance of KAR is on par with that of the state-of-the-art MRC models; on the adversarial sets, KAR outperforms the state-of-the-art MRC models by a large margin. That is to say, KAR is comparable in performance with the state-of-the-art MRC models, and significantly more robust to noise than them. To verify the effectiveness of general knowledge, we first study the relationship between the amount of general knowledge and the performance of KAR. As shown in Table TABREF13 , by increasing INLINEFORM0 from 0 to 5 in the data enrichment method, the amount of general knowledge rises monotonically, but the performance of KAR first rises until INLINEFORM1 reaches 3 and then drops down. Then we conduct an ablation study by replacing the knowledge aided attention mechanisms with the mutual attention proposed by BIBREF3 and the self attention proposed by BIBREF4 separately, and find that the F1 score of KAR drops by INLINEFORM2 on the development set, INLINEFORM3 on AddSent, and INLINEFORM4 on AddOneSent. Finally we find that after only one epoch of training, KAR already achieves an EM of INLINEFORM5 and an F1 score of INLINEFORM6 on the development set, which is even better than the final performance of several strong baselines, such as DCN (EM / F1: INLINEFORM7 / INLINEFORM8 ) BIBREF36 and BiDAF (EM / F1: INLINEFORM9 / INLINEFORM10 ) BIBREF3 . The above empirical findings imply that general knowledge indeed plays an effective role in KAR. To demonstrate the advantage of our explicit way to utilize general knowledge over the existing implicit way, we compare the performance of KAR with that reported by BIBREF10 , which used an encoding-based method to utilize the general knowledge dynamically retrieved from Wikipedia and ConceptNet. Since their best model only achieved an EM of INLINEFORM0 and an F1 score of INLINEFORM1 on the development set, which is much lower than the performance of KAR, we have good reason to believe that our explicit way works better than the existing implicit way. Model Comparison in the Hunger for Data We compare KAR with other MRC models in the hunger for data. Specifically, instead of using all the training examples, we produce several training subsets (i.e. subsets of the training examples) so as to study the relationship between the proportion of the available training examples and the performance. We produce each training subset by sampling a specific number of questions from all the questions relevant to each passage. By separately sampling 1, 2, 3, and 4 questions on each passage, we obtain four training subsets, which separately contain INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 of the training examples. As shown in Figure FIGREF15 , with KAR, SAN (re-implemented), and QANet (re-implemented without data augmentation) trained on these training subsets, we evaluate their performance on the development set, and find that KAR performs much better than SAN and QANet. As shown in Figure FIGREF16 and Figure FIGREF17 , with the above KAR, SAN, and QANet trained on the same training subsets, we also evaluate their performance on the adversarial sets, and still find that KAR performs much better than SAN and QANet. That is to say, when only a subset of the training examples are available, KAR outperforms the state-of-the-art MRC models by a large margin, and is still reasonably robust to noise. Analysis According to the experimental results, KAR is not only comparable in performance with the state-of-the-art MRC models, but also superior to them in terms of both the hunger for data and the robustness to noise. The reasons for these achievements, we believe, are as follows: Conclusion In this paper, we innovatively integrate the neural networks of MRC models with the general knowledge of human beings. Specifically, inter-word semantic connections are first extracted from each given passage-question pair by a WordNet-based data enrichment method, and then provided as general knowledge to an end-to-end MRC model named as Knowledge Aided Reader (KAR), which explicitly uses the general knowledge to assist its attention mechanisms. Experimental results show that KAR is not only comparable in performance with the state-of-the-art MRC models, but also superior to them in terms of both the hunger for data and the robustness to noise. In the future, we plan to use some larger knowledge bases, such as ConceptNet and Freebase, to improve the quality and scope of the general knowledge. Acknowledgments This work is partially supported by a research donation from iFLYTEK Co., Ltd., Hefei, China, and a discovery grant from Natural Sciences and Engineering Research Council (NSERC) of Canada.
How do the authors examine whether a model is robust to noise or not?
b0dbe75047310fec4d4ce787be5c32935fc4e37b
How Ġdo Ġthe Ġauthors Ġexamine Ġwhether Ġa Ġmodel Ġis Ġrobust Ġto Ġnoise Ġor Ġnot ?
By Ġevaluating Ġtheir Ġmodel Ġon Ġadvers arial Ġsets Ġcontaining Ġmisleading Ġsentences
<s> How Ġdo Ġthe Ġauthors Ġexamine Ġwhether Ġa Ġmodel Ġis Ġrobust Ġto Ġnoise Ġor Ġnot ? </s> Introduction </s> Machine ĠReading ĠComp rehens ion Ġ( M RC ), Ġas Ġthe Ġname Ġsuggests , Ġrequires Ġa Ġmachine Ġto Ġread Ġa Ġpassage Ġand Ġanswer Ġits Ġrelevant Ġquestions . ĠSince Ġthe Ġanswer Ġto Ġeach Ġquestion Ġis Ġsupposed Ġto Ġstem Ġfrom Ġthe Ġcorresponding Ġpassage , Ġa Ġcommon ĠM RC Ġsolution Ġis Ġto Ġdevelop Ġa Ġneural - network - based ĠM RC Ġmodel Ġthat Ġpredicts Ġan Ġanswer Ġspan Ġ( i . e . Ġthe Ġanswer Ġstart Ġposition Ġand Ġthe Ġanswer Ġend Ġposition ) Ġfrom Ġthe Ġpassage Ġof Ġeach Ġgiven Ġpassage - question Ġpair . ĠTo Ġfacilitate Ġthe Ġexplor ations Ġand Ġinnovations Ġin Ġthis Ġarea , Ġmany ĠM RC Ġdatasets Ġhave Ġbeen Ġestablished , Ġsuch Ġas ĠS Qu AD ĠB IB REF 0 Ġ, ĠMS ĠMAR CO ĠB IB REF 1 Ġ, Ġand ĠTri via Q A ĠB IB REF 2 Ġ. ĠConsequently , Ġmany Ġpioneering ĠM RC Ġmodels Ġhave Ġbeen Ġproposed , Ġsuch Ġas ĠBi DA F ĠB IB REF 3 Ġ, ĠR - NET ĠB IB REF 4 Ġ, Ġand ĠQ AN et ĠB IB REF 5 Ġ. ĠAccording Ġto Ġthe Ġleader Ġboard Ġof ĠS Qu AD , Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġhave Ġachieved Ġthe Ġsame Ġperformance Ġas Ġhuman Ġbeings . ĠHowever , Ġdoes Ġthis Ġimply Ġthat Ġthey Ġhave Ġpossessed Ġthe Ġsame Ġreading Ġcomprehension Ġability Ġas Ġhuman Ġbeings ? </s> OF ĠCOUR SE ĠNOT . ĠThere Ġis Ġa Ġhuge Ġgap Ġbetween ĠM RC Ġmodels Ġand Ġhuman Ġbeings , Ġwhich Ġis Ġmainly Ġreflected Ġin Ġthe Ġhunger Ġfor Ġdata Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠOn Ġthe Ġone Ġhand , Ġdeveloping ĠM RC Ġmodels Ġrequires Ġa Ġlarge Ġamount Ġof Ġtraining Ġexamples Ġ( i . e . Ġthe Ġpassage - question Ġpairs Ġlabeled Ġwith Ġanswer Ġspans ), Ġwhile Ġhuman Ġbeings Ġcan Ġachieve Ġgood Ġperformance Ġon Ġevaluation Ġexamples Ġ( i . e . Ġthe Ġpassage - question Ġpairs Ġto Ġaddress ) Ġwithout Ġtraining Ġexamples . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 6 Ġrevealed Ġthat Ġintentionally Ġinjected Ġnoise Ġ( e . g . Ġmisleading Ġsentences ) Ġin Ġevaluation Ġexamples Ġcauses Ġthe Ġperformance Ġof ĠM RC Ġmodels Ġto Ġdrop Ġsignificantly , Ġwhile Ġhuman Ġbeings Ġare Ġfar Ġless Ġlikely Ġto Ġsuffer Ġfrom Ġthis . ĠThe Ġreason Ġfor Ġthese Ġphenomena , Ġwe Ġbelieve , Ġis Ġthat ĠM RC Ġmodels Ġcan Ġonly Ġutilize Ġthe Ġknowledge Ġcontained Ġin Ġeach Ġgiven Ġpassage - question Ġpair , Ġbut Ġin Ġaddition Ġto Ġthis , Ġhuman Ġbeings Ġcan Ġalso Ġutilize Ġgeneral Ġknowledge . ĠA Ġtypical Ġcategory Ġof Ġgeneral Ġknowledge Ġis Ġinter - word Ġsemantic Ġconnections . ĠAs Ġshown Ġin ĠTable ĠT AB REF 1 Ġ, Ġsuch Ġgeneral Ġknowledge Ġis Ġessential Ġto Ġthe Ġreading Ġcomprehension Ġability Ġof Ġhuman Ġbeings . </s> A Ġpromising Ġstrategy Ġto Ġbridge Ġthe Ġgap Ġmentioned Ġabove Ġis Ġto Ġintegrate Ġthe Ġneural Ġnetworks Ġof ĠM RC Ġmodels Ġwith Ġthe Ġgeneral Ġknowledge Ġof Ġhuman Ġbeings . ĠTo Ġthis Ġend , Ġit Ġis Ġnecessary Ġto Ġsolve Ġtwo Ġproblems : Ġextracting Ġgeneral Ġknowledge Ġfrom Ġpassage - question Ġpairs Ġand Ġutilizing Ġthe Ġextracted Ġgeneral Ġknowledge Ġin Ġthe Ġprediction Ġof Ġanswer Ġspans . ĠThe Ġfirst Ġproblem Ġcan Ġbe Ġsolved Ġwith Ġknowledge Ġbases , Ġwhich Ġstore Ġgeneral Ġknowledge Ġin Ġstructured Ġforms . ĠA Ġbroad Ġvariety Ġof Ġknowledge Ġbases Ġare Ġavailable , Ġsuch Ġas ĠWord Net ĠB IB REF 7 Ġstoring Ġsemantic Ġknowledge , ĠConcept Net ĠB IB REF 8 Ġstoring Ġcommons ense Ġknowledge , Ġand ĠFree base ĠB IB REF 9 Ġstoring Ġfact oid Ġknowledge . ĠIn Ġthis Ġpaper , Ġwe Ġlimit Ġthe Ġscope Ġof Ġgeneral Ġknowledge Ġto Ġinter - word Ġsemantic Ġconnections , Ġand Ġthus Ġuse ĠWord Net Ġas Ġour Ġknowledge Ġbase . ĠThe Ġexisting Ġway Ġto Ġsolve Ġthe Ġsecond Ġproblem Ġis Ġto Ġencode Ġgeneral Ġknowledge Ġin Ġvector Ġspace Ġso Ġthat Ġthe Ġencoding Ġresults Ġcan Ġbe Ġused Ġto Ġenhance Ġthe Ġlex ical Ġor Ġcontextual Ġrepresentations Ġof Ġwords ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠHowever , Ġthis Ġis Ġan Ġimplicit Ġway Ġto Ġutilize Ġgeneral Ġknowledge , Ġsince Ġin Ġthis Ġway Ġwe Ġcan Ġneither Ġunderstand Ġnor Ġcontrol Ġthe Ġfunctioning Ġof Ġgeneral Ġknowledge . ĠIn Ġthis Ġpaper , Ġwe Ġdiscard Ġthe Ġexisting Ġimplicit Ġway Ġand Ġinstead Ġexplore Ġan Ġexplicit Ġ( i . e . Ġunderstandable Ġand Ġcontro ll able ) Ġway Ġto Ġutilize Ġgeneral Ġknowledge . </s> The Ġcontribution Ġof Ġthis Ġpaper Ġis Ġtwo - fold . ĠOn Ġthe Ġone Ġhand , Ġwe Ġpropose Ġa Ġdata Ġenrichment Ġmethod , Ġwhich Ġuses ĠWord Net Ġto Ġextract Ġinter - word Ġsemantic Ġconnections Ġas Ġgeneral Ġknowledge Ġfrom Ġeach Ġgiven Ġpassage - question Ġpair . ĠOn Ġthe Ġother Ġhand , Ġwe Ġpropose Ġan Ġend - to - end ĠM RC Ġmodel Ġnamed Ġas ĠKnowledge ĠA ided ĠReader Ġ( K AR ), Ġwhich Ġexplicitly Ġuses Ġthe Ġabove Ġextracted Ġgeneral Ġknowledge Ġto Ġassist Ġits Ġattention Ġmechanisms . ĠBased Ġon Ġthe Ġdata Ġenrichment Ġmethod , ĠK AR Ġis Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġand Ġsignificantly Ġmore Ġrobust Ġto Ġnoise Ġthan Ġthem . ĠWhen Ġonly Ġa Ġsubset Ġ( ĠIN LINE FORM 0 ĠâĢĵ ĠIN LINE FORM 1 Ġ) Ġof Ġthe Ġtraining Ġexamples Ġare Ġavailable , ĠK AR Ġoutper forms Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġby Ġa Ġlarge Ġmargin , Ġand Ġis Ġstill Ġreasonably Ġrobust Ġto Ġnoise . </s> Data ĠEn rich ment ĠMethod </s> In Ġthis Ġsection , Ġwe Ġelaborate Ġa ĠWord Net - based Ġdata Ġenrichment Ġmethod , Ġwhich Ġis Ġaimed Ġat Ġextracting Ġinter - word Ġsemantic Ġconnections Ġfrom Ġeach Ġpassage - question Ġpair Ġin Ġour ĠM RC Ġdataset . ĠThe Ġextraction Ġis Ġperformed Ġin Ġa Ġcontro ll able Ġmanner , Ġand Ġthe Ġextracted Ġresults Ġare Ġprovided Ġas Ġgeneral Ġknowledge Ġto Ġour ĠM RC Ġmodel . </s> Sem antic ĠRel ation ĠChain </s> Word Net Ġis Ġa Ġlex ical Ġdatabase Ġof ĠEnglish , Ġwhere Ġwords Ġare Ġorganized Ġinto Ġsy ns ets Ġaccording Ġto Ġtheir Ġsenses . ĠA Ġsyn set Ġis Ġa Ġset Ġof Ġwords Ġexpressing Ġthe Ġsame Ġsense Ġso Ġthat Ġa Ġword Ġhaving Ġmultiple Ġsenses Ġbelongs Ġto Ġmultiple Ġsy ns ets , Ġwith Ġeach Ġsyn set Ġcorresponding Ġto Ġa Ġsense . ĠSy ns ets Ġare Ġfurther Ġrelated Ġto Ġeach Ġother Ġthrough Ġsemantic Ġrelations . ĠAccording Ġto Ġthe ĠWord Net Ġinterface Ġprovided Ġby ĠNL T K ĠB IB REF 12 Ġ, Ġthere Ġare Ġtotally Ġsixteen Ġtypes Ġof Ġsemantic Ġrelations Ġ( e . g . Ġhyper ny ms , Ġhyp onyms , Ġhol onyms , Ġmer onyms , Ġattributes , Ġetc .). ĠBased Ġon Ġsyn set Ġand Ġsemantic Ġrelation , Ġwe Ġdefine Ġa Ġnew Ġconcept : Ġsemantic Ġrelation Ġchain . ĠA Ġsemantic Ġrelation Ġchain Ġis Ġa Ġconc aten ated Ġsequence Ġof Ġsemantic Ġrelations , Ġwhich Ġlinks Ġa Ġsyn set Ġto Ġanother Ġsyn set . ĠFor Ġexample , Ġthe Ġsyn set ĠâĢ ľ ker atin . n . 01 âĢ Ŀ Ġis Ġrelated Ġto Ġthe Ġsyn set ĠâĢ ľ fe ather . n . 01 âĢ Ŀ Ġthrough Ġthe Ġsemantic Ġrelation ĠâĢ ľ sub st ance Ġhol onym âĢ Ŀ , Ġthe Ġsyn set ĠâĢ ľ fe ather . n . 01 âĢ Ŀ Ġis Ġrelated Ġto Ġthe Ġsyn set ĠâĢ ľ bird . n . 01 âĢ Ŀ Ġthrough Ġthe Ġsemantic Ġrelation ĠâĢ ľ part Ġhol onym âĢ Ŀ , Ġand Ġthe Ġsyn set ĠâĢ ľ bird . n . 01 âĢ Ŀ Ġis Ġrelated Ġto Ġthe Ġsyn set ĠâĢ ľ par rot . n . 01 âĢ Ŀ Ġthrough Ġthe Ġsemantic Ġrelation ĠâĢ ľ hyp onym âĢ Ŀ , Ġthus ĠâĢ ľ sub st ance Ġhol onym ĠIN LINE FORM 0 Ġpart Ġhol onym ĠIN LINE FORM 1 Ġhyp onym âĢ Ŀ Ġis Ġa Ġsemantic Ġrelation Ġchain , Ġwhich Ġlinks Ġthe Ġsyn set ĠâĢ ľ ker atin . n . 01 âĢ Ŀ Ġto Ġthe Ġsyn set ĠâĢ ľ par rot . n . 01 âĢ Ŀ . ĠWe Ġname Ġeach Ġsemantic Ġrelation Ġin Ġa Ġsemantic Ġrelation Ġchain Ġas Ġa Ġhop , Ġtherefore Ġthe Ġabove Ġsemantic Ġrelation Ġchain Ġis Ġa Ġ3 - hop Ġchain . ĠBy Ġthe Ġway , Ġeach Ġsingle Ġsemantic Ġrelation Ġis Ġequivalent Ġto Ġa Ġ1 - hop Ġchain . </s> Inter - word ĠSem antic ĠConnection </s> The Ġkey Ġproblem Ġin Ġthe Ġdata Ġenrichment Ġmethod Ġis Ġdetermining Ġwhether Ġa Ġword Ġis Ġsem antically Ġconnected Ġto Ġanother Ġword . ĠIf Ġso , Ġwe Ġsay Ġthat Ġthere Ġexists Ġan Ġinter - word Ġsemantic Ġconnection Ġbetween Ġthem . ĠTo Ġsolve Ġthis Ġproblem , Ġwe Ġdefine Ġanother Ġnew Ġconcept : Ġthe Ġextended Ġsy ns ets Ġof Ġa Ġword . ĠGiven Ġa Ġword ĠIN LINE FORM 0 Ġ, Ġwhose Ġsy ns ets Ġare Ġrepresented Ġas Ġa Ġset ĠIN LINE FORM 1 Ġ, Ġwe Ġuse Ġanother Ġset ĠIN LINE FORM 2 Ġto Ġrepresent Ġits Ġextended Ġsy ns ets , Ġwhich Ġincludes Ġall Ġthe Ġsy ns ets Ġthat Ġare Ġin ĠIN LINE FORM 3 Ġor Ġthat Ġcan Ġbe Ġlinked Ġto Ġfrom ĠIN LINE FORM 4 Ġthrough Ġsemantic Ġrelation Ġchains . ĠThe oret ically , Ġif Ġthere Ġis Ġno Ġlimitation Ġon Ġsemantic Ġrelation Ġchains , ĠIN LINE FORM 5 Ġwill Ġinclude Ġall Ġthe Ġsy ns ets Ġin ĠWord Net , Ġwhich Ġis Ġmeaningless Ġin Ġmost Ġsituations . ĠTherefore , Ġwe Ġuse Ġa Ġhyper - param eter ĠIN LINE FORM 6 Ġto Ġrepresent Ġthe Ġpermitted Ġmaximum Ġhop Ġcount Ġof Ġsemantic Ġrelation Ġchains . ĠThat Ġis Ġto Ġsay , Ġonly Ġthe Ġchains Ġhaving Ġno Ġmore Ġthan ĠIN LINE FORM 7 Ġhops Ġcan Ġbe Ġused Ġto Ġconstruct ĠIN LINE FORM 8 Ġso Ġthat ĠIN LINE FORM 9 Ġbecomes Ġa Ġfunction Ġof ĠIN LINE FORM 10 Ġ: ĠIN LINE FORM 11 Ġ( if ĠIN LINE FORM 12 Ġ, Ġwe Ġwill Ġhave ĠIN LINE FORM 13 Ġ). ĠBased Ġon Ġthe Ġabove Ġstatements , Ġwe Ġformulate Ġa Ġhe uristic Ġrule Ġfor Ġdetermining Ġinter - word Ġsemantic Ġconnections : Ġa Ġword ĠIN LINE FORM 14 Ġis Ġsem antically Ġconnected Ġto Ġanother Ġword ĠIN LINE FORM 15 Ġif Ġand Ġonly Ġif ĠIN LINE FORM 16 Ġ. </s> General ĠKnowledge ĠExt raction </s> Given Ġa Ġpassage - question Ġpair , Ġthe Ġinter - word Ġsemantic Ġconnections Ġthat Ġconnect Ġany Ġword Ġto Ġany Ġpassage Ġword Ġare Ġregarded Ġas Ġthe Ġgeneral Ġknowledge Ġwe Ġneed Ġto Ġextract . ĠConsidering Ġthe Ġrequirements Ġof Ġour ĠM RC Ġmodel , Ġwe Ġonly Ġextract Ġthe Ġpositional Ġinformation Ġof Ġsuch Ġinter - word Ġsemantic Ġconnections . ĠSpecifically , Ġfor Ġeach Ġword ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġa Ġset ĠIN LINE FORM 1 Ġ, Ġwhich Ġincludes Ġthe Ġpositions Ġof Ġthe Ġpassage Ġwords Ġthat ĠIN LINE FORM 2 Ġis Ġsem antically Ġconnected Ġto Ġ( if ĠIN LINE FORM 3 Ġitself Ġis Ġa Ġpassage Ġword , Ġwe Ġwill Ġexclude Ġits Ġown Ġposition Ġfrom ĠIN LINE FORM 4 Ġ). ĠWe Ġcan Ġcontrol Ġthe Ġamount Ġof Ġthe Ġextracted Ġresults Ġby Ġsetting Ġthe Ġhyper - param eter ĠIN LINE FORM 5 Ġ: Ġif Ġwe Ġset ĠIN LINE FORM 6 Ġto Ġ0 , Ġinter - word Ġsemantic Ġconnections Ġwill Ġonly Ġexist Ġbetween Ġsyn onyms ; Ġif Ġwe Ġincrease ĠIN LINE FORM 7 Ġ, Ġinter - word Ġsemantic Ġconnections Ġwill Ġexist Ġbetween Ġmore Ġwords . ĠThat Ġis Ġto Ġsay , Ġby Ġincreasing ĠIN LINE FORM 8 Ġwithin Ġa Ġcertain Ġrange , Ġwe Ġcan Ġusually Ġextract Ġmore Ġinter - word Ġsemantic Ġconnections Ġfrom Ġa Ġpassage - question Ġpair , Ġand Ġthus Ġcan Ġprovide Ġthe ĠM RC Ġmodel Ġwith Ġmore Ġgeneral Ġknowledge . ĠHowever , Ġdue Ġto Ġthe Ġcomplexity Ġand Ġdiversity Ġof Ġnatural Ġlanguages , Ġonly Ġa Ġpart Ġof Ġthe Ġextracted Ġresults Ġcan Ġserve Ġas Ġuseful Ġgeneral Ġknowledge , Ġwhile Ġthe Ġrest Ġof Ġthem Ġare Ġuseless Ġfor Ġthe Ġprediction Ġof Ġanswer Ġspans , Ġand Ġthe Ġproportion Ġof Ġthe Ġuseless Ġpart Ġalways Ġrises Ġwhen ĠIN LINE FORM 9 Ġis Ġset Ġlarger . ĠTherefore Ġwe Ġset ĠIN LINE FORM 10 Ġthrough Ġcross Ġvalidation Ġ( i . e . Ġaccording Ġto Ġthe Ġperformance Ġof Ġthe ĠM RC Ġmodel Ġon Ġthe Ġdevelopment Ġexamples ). </s> Know ledge ĠA ided ĠReader </s> In Ġthis Ġsection , Ġwe Ġelaborate Ġour ĠM RC Ġmodel : ĠKnowledge ĠA ided ĠReader Ġ( K AR ). ĠThe Ġkey Ġcomponents Ġof Ġmost Ġexisting ĠM RC Ġmodels Ġare Ġtheir Ġattention Ġmechanisms ĠB IB REF 13 Ġ, Ġwhich Ġare Ġaimed Ġat Ġf using Ġthe Ġassociated Ġrepresentations Ġof Ġeach Ġgiven Ġpassage - question Ġpair . ĠThese Ġattention Ġmechanisms Ġgenerally Ġfall Ġinto Ġtwo Ġcategories : Ġthe Ġfirst Ġone , Ġwhich Ġwe Ġname Ġas Ġmutual Ġattention , Ġis Ġaimed Ġat Ġf using Ġthe Ġquestion Ġrepresentations Ġinto Ġthe Ġpassage Ġrepresentations Ġso Ġas Ġto Ġobtain Ġthe Ġquestion - aware Ġpassage Ġrepresentations ; Ġthe Ġsecond Ġone , Ġwhich Ġwe Ġname Ġas Ġself Ġattention , Ġis Ġaimed Ġat Ġf using Ġthe Ġquestion - aware Ġpassage Ġrepresentations Ġinto Ġthemselves Ġso Ġas Ġto Ġobtain Ġthe Ġfinal Ġpassage Ġrepresentations . ĠAlthough ĠK AR Ġis Ġequipped Ġwith Ġboth Ġcategories , Ġits Ġmost Ġremarkable Ġfeature Ġis Ġthat Ġit Ġexplicitly Ġuses Ġthe Ġgeneral Ġknowledge Ġextracted Ġby Ġthe Ġdata Ġenrichment Ġmethod Ġto Ġassist Ġits Ġattention Ġmechanisms . ĠTherefore Ġwe Ġseparately Ġname Ġthe Ġattention Ġmechanisms Ġof ĠK AR Ġas Ġknowledge Ġaided Ġmutual Ġattention Ġand Ġknowledge Ġaided Ġself Ġattention . </s> Task ĠDefinition </s> Given Ġa Ġpassage ĠIN LINE FORM 0 Ġand Ġa Ġrelevant Ġquestion ĠIN LINE FORM 1 Ġ, Ġthe Ġtask Ġis Ġto Ġpredict Ġan Ġanswer Ġspan ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġ, Ġso Ġthat Ġthe Ġresulting Ġsubsequ ence ĠIN LINE FORM 4 Ġfrom ĠIN LINE FORM 5 Ġis Ġan Ġanswer Ġto ĠIN LINE FORM 6 Ġ. </s> Overall ĠArchitecture </s> As Ġshown Ġin ĠFigure ĠFIG REF 7 Ġ, ĠK AR Ġis Ġan Ġend - to - end ĠM RC Ġmodel Ġconsisting Ġof Ġfive Ġlayers : </s> Lex icon ĠEmb edd ing ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġwords Ġto Ġthe Ġlex icon Ġembed d ings . ĠThe Ġlex icon Ġembed ding Ġof Ġeach Ġword Ġis Ġcomposed Ġof Ġits Ġword Ġembed ding Ġand Ġcharacter Ġembed ding . ĠFor Ġeach Ġword , Ġwe Ġuse Ġthe Ġpre - trained ĠGl o Ve ĠB IB REF 14 Ġword Ġvector Ġas Ġits Ġword Ġembed ding , Ġand Ġobtain Ġits Ġcharacter Ġembed ding Ġwith Ġa ĠConv olution al ĠNeural ĠNetwork Ġ( CNN ) ĠB IB REF 15 Ġ. ĠFor Ġboth Ġthe Ġpassage Ġand Ġthe Ġquestion , Ġwe Ġpass Ġthe Ġconc aten ation Ġof Ġthe Ġword Ġembed d ings Ġand Ġthe Ġcharacter Ġembed d ings Ġthrough Ġa Ġshared Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 0 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġpassage Ġlex icon Ġembed d ings ĠIN LINE FORM 1 Ġand Ġthe Ġquestion Ġlex icon Ġembed d ings ĠIN LINE FORM 2 Ġ. </s> Context ĠEmb edd ing ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġlex icon Ġembed d ings Ġto Ġthe Ġcontext Ġembed d ings . ĠFor Ġboth Ġthe Ġpassage Ġand Ġthe Ġquestion , Ġwe Ġprocess Ġthe Ġlex icon Ġembed d ings Ġ( i . e . ĠIN LINE FORM 0 Ġfor Ġthe Ġpassage Ġand ĠIN LINE FORM 1 Ġfor Ġthe Ġquestion ) Ġwith Ġa Ġshared Ġbid irection al ĠL ST M Ġ( Bi L ST M ) ĠB IB REF 16 Ġ, Ġwhose Ġhidden Ġstate Ġdimension ality Ġis ĠIN LINE FORM 2 Ġ. ĠBy Ġconc aten ating Ġthe Ġforward ĠL ST M Ġoutputs Ġand Ġthe Ġbackward ĠL ST M Ġoutputs , Ġwe Ġobtain Ġthe Ġpassage Ġcontext Ġembed d ings ĠIN LINE FORM 3 Ġand Ġthe Ġquestion Ġcontext Ġembed d ings ĠIN LINE FORM 4 Ġ. </s> Co arse ĠMemory ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġcontext Ġembed d ings Ġto Ġthe Ġcoarse Ġmemories . ĠFirst Ġwe Ġuse Ġknowledge Ġaided Ġmutual Ġattention Ġ( introdu ced Ġlater ) Ġto Ġfuse ĠIN LINE FORM 0 Ġinto ĠIN LINE FORM 1 Ġ, Ġthe Ġoutputs Ġof Ġwhich Ġare Ġrepresented Ġas ĠIN LINE FORM 2 Ġ. ĠThen Ġwe Ġprocess ĠIN LINE FORM 3 Ġwith Ġa ĠBi L ST M , Ġwhose Ġhidden Ġstate Ġdimension ality Ġis ĠIN LINE FORM 4 Ġ. ĠBy Ġconc aten ating Ġthe Ġforward ĠL ST M Ġoutputs Ġand Ġthe Ġbackward ĠL ST M Ġoutputs , Ġwe Ġobtain Ġthe Ġcoarse Ġmemories ĠIN LINE FORM 5 Ġ, Ġwhich Ġare Ġthe Ġquestion - aware Ġpassage Ġrepresentations . </s> Ref ined ĠMemory ĠLayer . ĠThis Ġlayer Ġmaps Ġthe Ġcoarse Ġmemories Ġto Ġthe Ġrefined Ġmemories . ĠFirst Ġwe Ġuse Ġknowledge Ġaided Ġself Ġattention Ġ( introdu ced Ġlater ) Ġto Ġfuse ĠIN LINE FORM 0 Ġinto Ġthemselves , Ġthe Ġoutputs Ġof Ġwhich Ġare Ġrepresented Ġas ĠIN LINE FORM 1 Ġ. ĠThen Ġwe Ġprocess ĠIN LINE FORM 2 Ġwith Ġa ĠBi L ST M , Ġwhose Ġhidden Ġstate Ġdimension ality Ġis ĠIN LINE FORM 3 Ġ. ĠBy Ġconc aten ating Ġthe Ġforward ĠL ST M Ġoutputs Ġand Ġthe Ġbackward ĠL ST M Ġoutputs , Ġwe Ġobtain Ġthe Ġrefined Ġmemories ĠIN LINE FORM 4 Ġ, Ġwhich Ġare Ġthe Ġfinal Ġpassage Ġrepresentations . </s> Answer ĠSpan ĠPrediction ĠLayer . ĠThis Ġlayer Ġpredicts Ġthe Ġanswer Ġstart Ġposition Ġand Ġthe Ġanswer Ġend Ġposition Ġbased Ġon Ġthe Ġabove Ġlayers . ĠFirst Ġwe Ġobtain Ġthe Ġanswer Ġstart Ġposition Ġdistribution ĠIN LINE FORM 0 Ġ: ĠIN LINE FORM 1 ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters ; ĠIN LINE FORM 3 Ġrepresents Ġthe Ġrefined Ġmemory Ġof Ġeach Ġpassage Ġword ĠIN LINE FORM 4 Ġ( i . e . Ġthe ĠIN LINE FORM 5 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 6 Ġ); ĠIN LINE FORM 7 Ġrepresents Ġthe Ġquestion Ġsummary Ġobtained Ġby Ġperforming Ġan Ġattention Ġpool ing Ġover ĠIN LINE FORM 8 Ġ. ĠThen Ġwe Ġobtain Ġthe Ġanswer Ġend Ġposition Ġdistribution ĠIN LINE FORM 9 Ġ: ĠIN LINE FORM 10 ĠIN LINE FORM 11 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters ; ĠIN LINE FORM 3 Ġrepresents Ġvector Ġconc aten ation . ĠFinally Ġwe Ġconstruct Ġan Ġanswer Ġspan Ġprediction Ġmatrix ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġrepresents Ġthe Ġupper Ġtriangular Ġmatrix Ġof Ġa Ġmatrix ĠIN LINE FORM 6 Ġ. ĠTherefore , Ġfor Ġthe Ġtraining , Ġwe Ġminimize ĠIN LINE FORM 7 Ġon Ġeach Ġtraining Ġexample Ġwhose Ġlabeled Ġanswer Ġspan Ġis ĠIN LINE FORM 8 Ġ; Ġfor Ġthe Ġinference , Ġwe Ġseparately Ġtake Ġthe Ġrow Ġindex Ġand Ġcolumn Ġindex Ġof Ġthe Ġmaximum Ġelement Ġin ĠIN LINE FORM 9 Ġas ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. </s> Know ledge ĠA ided ĠMutual ĠAttention </s> As Ġa Ġpart Ġof Ġthe Ġcoarse Ġmemory Ġlayer , Ġknowledge Ġaided Ġmutual Ġattention Ġis Ġaimed Ġat Ġf using Ġthe Ġquestion Ġcontext Ġembed d ings ĠIN LINE FORM 0 Ġinto Ġthe Ġpassage Ġcontext Ġembed d ings ĠIN LINE FORM 1 Ġ, Ġwhere Ġthe Ġkey Ġproblem Ġis Ġto Ġcalculate Ġthe Ġsimilarity Ġbetween Ġeach Ġpassage Ġcontext Ġembed ding ĠIN LINE FORM 2 Ġ( i . e . Ġthe ĠIN LINE FORM 3 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 4 Ġ) Ġand Ġeach Ġquestion Ġcontext Ġembed ding ĠIN LINE FORM 5 Ġ( i . e . Ġthe ĠIN LINE FORM 6 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 7 Ġ). ĠTo Ġsolve Ġthis Ġproblem , ĠB IB REF 3 Ġproposed Ġa Ġsimilarity Ġfunction : ĠIN LINE FORM 8 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġtrain able Ġparameter ; ĠIN LINE FORM 1 Ġrepresents Ġelement - wise Ġmultiplication . ĠThis Ġsimilarity Ġfunction Ġhas Ġalso Ġbeen Ġadopted Ġby Ġseveral Ġother Ġworks ĠB IB REF 17 Ġ, ĠB IB REF 5 Ġ. ĠHowever , Ġsince Ġcontext Ġembed d ings Ġcontain Ġhigh - level Ġinformation , Ġwe Ġbelieve Ġthat Ġintroducing Ġthe Ġpre - ext racted Ġgeneral Ġknowledge Ġinto Ġthe Ġcalculation Ġof Ġsuch Ġsimilarities Ġwill Ġmake Ġthe Ġresults Ġmore Ġreasonable . ĠTherefore Ġwe Ġmodify Ġthe Ġabove Ġsimilarity Ġfunction Ġto Ġthe Ġfollowing Ġform : ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġenhanced Ġcontext Ġembed ding Ġof Ġa Ġword ĠIN LINE FORM 1 Ġ. ĠWe Ġuse Ġthe Ġpre - ext racted Ġgeneral Ġknowledge Ġto Ġconstruct Ġthe Ġenhanced Ġcontext Ġembed d ings . ĠSpecifically , Ġfor Ġeach Ġword ĠIN LINE FORM 2 Ġ, Ġwhose Ġcontext Ġembed ding Ġis ĠIN LINE FORM 3 Ġ, Ġto Ġconstruct Ġits Ġenhanced Ġcontext Ġembed ding ĠIN LINE FORM 4 Ġ, Ġfirst Ġrecall Ġthat Ġwe Ġhave Ġextracted Ġa Ġset ĠIN LINE FORM 5 Ġ, Ġwhich Ġincludes Ġthe Ġpositions Ġof Ġthe Ġpassage Ġwords Ġthat ĠIN LINE FORM 6 Ġis Ġsem antically Ġconnected Ġto , Ġthus Ġby Ġgathering Ġthe Ġcolumns Ġin ĠIN LINE FORM 7 Ġwhose Ġindexes Ġare Ġgiven Ġby ĠIN LINE FORM 8 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġcontext Ġembed d ings ĠIN LINE FORM 9 Ġ. ĠThen Ġby Ġconstructing Ġa ĠIN LINE FORM 10 Ġ- att ended Ġsummary Ġof ĠIN LINE FORM 11 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġvector ĠIN LINE FORM 12 Ġ( if ĠIN LINE FORM 13 Ġ, Ġwhich Ġmakes ĠIN LINE FORM 14 Ġ, Ġwe Ġwill Ġset ĠIN LINE FORM 15 Ġ): ĠIN LINE FORM 16 ĠIN LINE FORM 17 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters ; ĠIN LINE FORM 3 Ġrepresents Ġthe ĠIN LINE FORM 4 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 5 Ġ. ĠFinally Ġwe Ġpass Ġthe Ġconc aten ation Ġof ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġthrough Ġa Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 8 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġenhanced Ġcontext Ġembed ding ĠIN LINE FORM 9 Ġ. </s> Based Ġon Ġthe Ġmodified Ġsimilarity Ġfunction Ġand Ġthe Ġenhanced Ġcontext Ġembed d ings , Ġto Ġperform Ġknowledge Ġaided Ġmutual Ġattention , Ġfirst Ġwe Ġconstruct Ġa Ġknowledge Ġaided Ġsimilarity Ġmatrix ĠIN LINE FORM 0 Ġ, Ġwhere Ġeach Ġelement ĠIN LINE FORM 1 Ġ. ĠThen Ġfollowing ĠB IB REF 5 Ġ, Ġwe Ġconstruct Ġthe Ġpassage - att ended Ġquestion Ġsumm aries ĠIN LINE FORM 2 Ġand Ġthe Ġquestion - att ended Ġpassage Ġsumm aries ĠIN LINE FORM 3 Ġ: ĠIN LINE FORM 4 ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġrepresents Ġsoft max Ġalong Ġthe Ġrow Ġdimension Ġand ĠIN LINE FORM 1 Ġalong Ġthe Ġcolumn Ġdimension . ĠFinally Ġfollowing ĠB IB REF 17 Ġ, Ġwe Ġpass Ġthe Ġconc aten ation Ġof ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġthrough Ġa Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 6 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġoutputs ĠIN LINE FORM 7 Ġ. </s> Know ledge ĠA ided ĠSelf ĠAttention </s> As Ġa Ġpart Ġof Ġthe Ġrefined Ġmemory Ġlayer , Ġknowledge Ġaided Ġself Ġattention Ġis Ġaimed Ġat Ġf using Ġthe Ġcoarse Ġmemories ĠIN LINE FORM 0 Ġinto Ġthemselves . ĠIf Ġwe Ġsimply Ġfollow Ġthe Ġself Ġattent ions Ġof Ġother Ġworks ĠB IB REF 4 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 17 Ġ, Ġthen Ġfor Ġeach Ġpassage Ġword ĠIN LINE FORM 1 Ġ, Ġwe Ġshould Ġfuse Ġits Ġcoarse Ġmemory ĠIN LINE FORM 2 Ġ( i . e . Ġthe ĠIN LINE FORM 3 Ġ- th Ġcolumn Ġin ĠIN LINE FORM 4 Ġ) Ġwith Ġthe Ġcoarse Ġmemories Ġof Ġall Ġthe Ġother Ġpassage Ġwords . ĠHowever , Ġwe Ġbelieve Ġthat Ġthis Ġis Ġboth Ġunnecessary Ġand Ġdistracting , Ġsince Ġeach Ġpassage Ġword Ġhas Ġnothing Ġto Ġdo Ġwith Ġmany Ġof Ġthe Ġother Ġpassage Ġwords . ĠThus Ġwe Ġuse Ġthe Ġpre - ext racted Ġgeneral Ġknowledge Ġto Ġguarantee Ġthat Ġthe Ġfusion Ġof Ġcoarse Ġmemories Ġfor Ġeach Ġpassage Ġword Ġwill Ġonly Ġinvolve Ġa Ġprecise Ġsubset Ġof Ġthe Ġother Ġpassage Ġwords . ĠSpecifically , Ġfor Ġeach Ġpassage Ġword ĠIN LINE FORM 5 Ġ, Ġwhose Ġcoarse Ġmemory Ġis ĠIN LINE FORM 6 Ġ, Ġto Ġperform Ġthe Ġfusion Ġof Ġcoarse Ġmemories , Ġfirst Ġrecall Ġthat Ġwe Ġhave Ġextracted Ġa Ġset ĠIN LINE FORM 7 Ġ, Ġwhich Ġincludes Ġthe Ġpositions Ġof Ġthe Ġother Ġpassage Ġwords Ġthat ĠIN LINE FORM 8 Ġis Ġsem antically Ġconnected Ġto , Ġthus Ġby Ġgathering Ġthe Ġcolumns Ġin ĠIN LINE FORM 9 Ġwhose Ġindexes Ġare Ġgiven Ġby ĠIN LINE FORM 10 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġcoarse Ġmemories ĠIN LINE FORM 11 Ġ. ĠThen Ġby Ġconstructing Ġa ĠIN LINE FORM 12 Ġ- att ended Ġsummary Ġof ĠIN LINE FORM 13 Ġ, Ġwe Ġobtain Ġthe Ġmatching Ġvector ĠIN LINE FORM 14 Ġ( if ĠIN LINE FORM 15 Ġ, Ġwhich Ġmakes ĠIN LINE FORM 16 Ġ, Ġwe Ġwill Ġset ĠIN LINE FORM 17 Ġ): ĠIN LINE FORM 18 ĠIN LINE FORM 19 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters . ĠFinally Ġwe Ġpass Ġthe Ġconc aten ation Ġof ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġthrough Ġa Ġdense Ġlayer Ġwith ĠRe LU Ġactivation , Ġwhose Ġoutput Ġdimension ality Ġis ĠIN LINE FORM 5 Ġ. ĠTherefore Ġwe Ġobtain Ġthe Ġfusion Ġresult ĠIN LINE FORM 6 Ġ, Ġand Ġfurther Ġthe Ġoutputs ĠIN LINE FORM 7 Ġ. </s> Related ĠWorks </s> Att ention ĠMechan isms . ĠBesides Ġthose Ġmentioned Ġabove , Ġother Ġinteresting Ġattention Ġmechanisms Ġinclude Ġperforming Ġmulti - round Ġalignment Ġto Ġavoid Ġthe Ġproblems Ġof Ġattention Ġredundancy Ġand Ġattention Ġdeficiency ĠB IB REF 20 Ġ, Ġand Ġusing Ġmutual Ġattention Ġas Ġa Ġskip - connect or Ġto Ġdensely Ġconnect Ġpair wise Ġlayers ĠB IB REF 21 Ġ. </s> Data ĠAug mentation . ĠIt Ġis Ġproved Ġthat Ġproperly Ġaugment ing Ġtraining Ġexamples Ġcan Ġimprove Ġthe Ġperformance Ġof ĠM RC Ġmodels . ĠFor Ġexample , ĠB IB REF 22 Ġtrained Ġa Ġgener ative Ġmodel Ġto Ġgenerate Ġquestions Ġbased Ġon Ġunl abel ed Ġtext , Ġwhich Ġsubstantially Ġboosted Ġtheir Ġperformance ; ĠB IB REF 5 Ġtrained Ġa Ġback - and - forth Ġtranslation Ġmodel Ġto Ġparaph rase Ġtraining Ġexamples , Ġwhich Ġbrought Ġthem Ġa Ġsignificant Ġperformance Ġgain . </s> Multi - step ĠReason ing . ĠInspired Ġby Ġthe Ġfact Ġthat Ġhuman Ġbeings Ġare Ġcapable Ġof Ġunderstanding Ġcomplex Ġdocuments Ġby Ġreading Ġthem Ġover Ġand Ġover Ġagain , Ġmulti - step Ġreasoning Ġwas Ġproposed Ġto Ġbetter Ġdeal Ġwith Ġdifficult ĠM RC Ġtasks . ĠFor Ġexample , ĠB IB REF 23 Ġused Ġreinforcement Ġlearning Ġto Ġdynamically Ġdetermine Ġthe Ġnumber Ġof Ġreasoning Ġsteps ; ĠB IB REF 19 Ġfixed Ġthe Ġnumber Ġof Ġreasoning Ġsteps , Ġbut Ġused Ġsto ch astic Ġdrop out Ġin Ġthe Ġoutput Ġlayer Ġto Ġavoid Ġstep Ġbias . </s> L ingu istic ĠEmb edd ings . ĠIt Ġis Ġboth Ġeasy Ġand Ġeffective Ġto Ġincorporate Ġlinguistic Ġembed d ings Ġinto Ġthe Ġinput Ġlayer Ġof ĠM RC Ġmodels . ĠFor Ġexample , ĠB IB REF 24 Ġand ĠB IB REF 19 Ġused ĠPOS Ġembed d ings Ġand ĠN ER Ġembed d ings Ġto Ġconstruct Ġtheir Ġinput Ġembed d ings ; ĠB IB REF 25 Ġused Ġstructural Ġembed d ings Ġbased Ġon Ġparsing Ġtrees Ġto Ġconstructed Ġtheir Ġinput Ġembed d ings . </s> Transfer ĠLearning . ĠSeveral Ġrecent Ġbreakthrough s Ġin ĠM RC Ġbenefit Ġfrom Ġfeature - based Ġtransfer Ġlearning ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġand Ġfine - tun ing - based Ġtransfer Ġlearning ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, Ġwhich Ġare Ġbased Ġon Ġcertain Ġword - level Ġor Ġsentence - level Ġmodels Ġpre - trained Ġon Ġlarge Ġexternal Ġcorpor a Ġin Ġcertain Ġsupervised Ġor Ġun super vised Ġmanners . </s> Exper imental ĠSettings </s> M RC ĠDat as et . ĠThe ĠM RC Ġdataset Ġused Ġin Ġthis Ġpaper Ġis ĠS Qu AD Ġ1 . 1 , Ġwhich Ġcontains Ġover ĠIN LINE FORM 0 Ġpassage - question Ġpairs Ġand Ġhas Ġbeen Ġrandomly Ġpartition ed Ġinto Ġthree Ġparts : Ġa Ġtraining Ġset Ġ( ĠIN LINE FORM 1 Ġ), Ġa Ġdevelopment Ġset Ġ( ĠIN LINE FORM 2 Ġ), Ġand Ġa Ġtest Ġset Ġ( ĠIN LINE FORM 3 Ġ). ĠBesides , Ġwe Ġalso Ġuse Ġtwo Ġof Ġits Ġadvers arial Ġsets , Ġnamely ĠAdd Sent Ġand ĠAdd One Sent ĠB IB REF 6 Ġ, Ġto Ġevaluate Ġthe Ġrobust ness Ġto Ġnoise Ġof ĠM RC Ġmodels . ĠThe Ġpassages Ġin Ġthe Ġadvers arial Ġsets Ġcontain Ġmisleading Ġsentences , Ġwhich Ġare Ġaimed Ġat Ġdistracting ĠM RC Ġmodels . ĠSpecifically , Ġeach Ġpassage Ġin ĠAdd Sent Ġcontains Ġseveral Ġsentences Ġthat Ġare Ġsimilar Ġto Ġthe Ġquestion Ġbut Ġnot Ġcontradictory Ġto Ġthe Ġanswer , Ġwhile Ġeach Ġpassage Ġin ĠAdd One Sent Ġcontains Ġa Ġhuman - approved Ġrandom Ġsentence Ġthat Ġmay Ġbe Ġunrelated Ġto Ġthe Ġpassage . </s> Im plementation ĠDetails . ĠWe Ġtoken ize Ġthe ĠM RC Ġdataset Ġwith Ġspa Cy Ġ2 . 0 . 13 ĠB IB REF 30 Ġ, Ġmanipulate ĠWord Net Ġ3 . 0 Ġwith ĠNL T K Ġ3 . 3 , Ġand Ġimplement ĠK AR Ġwith ĠT ensor Flow Ġ1 . 11 . 0 ĠB IB REF 31 Ġ. ĠFor Ġthe Ġdata Ġenrichment Ġmethod , Ġwe Ġset Ġthe Ġhyper - param eter ĠIN LINE FORM 0 Ġto Ġ3 . ĠFor Ġthe Ġdense Ġlayers Ġand Ġthe ĠBi L ST Ms , Ġwe Ġset Ġthe Ġdimension ality Ġunit ĠIN LINE FORM 1 Ġto Ġ600 . ĠFor Ġmodel Ġoptimization , Ġwe Ġapply Ġthe ĠAdam ĠB IB REF 32 Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof ĠIN LINE FORM 2 Ġand Ġa Ġmini - batch Ġsize Ġof Ġ32 . ĠFor Ġmodel Ġevaluation , Ġwe Ġuse ĠEx act ĠMatch Ġ( EM ) Ġand ĠF 1 Ġscore Ġas Ġevaluation Ġmetrics . ĠTo Ġavoid Ġover fitting , Ġwe Ġapply Ġdrop out ĠB IB REF 33 Ġto Ġthe Ġdense Ġlayers Ġand Ġthe ĠBi L ST Ms Ġwith Ġa Ġdrop out Ġrate Ġof ĠIN LINE FORM 3 Ġ. ĠTo Ġboost Ġthe Ġperformance , Ġwe Ġapply Ġexponential Ġmoving Ġaverage Ġwith Ġa Ġdecay Ġrate Ġof ĠIN LINE FORM 4 Ġ. </s> Model ĠComparison Ġin Ġboth ĠPerformance Ġand Ġthe ĠRob ust ness Ġto ĠNoise </s> We Ġcompare ĠK AR Ġwith Ġother ĠM RC Ġmodels Ġin Ġboth Ġperformance Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠSpecifically , Ġwe Ġnot Ġonly Ġevaluate Ġthe Ġperformance Ġof ĠK AR Ġon Ġthe Ġdevelopment Ġset Ġand Ġthe Ġtest Ġset , Ġbut Ġalso Ġdo Ġthis Ġon Ġthe Ġadvers arial Ġsets . ĠAs Ġfor Ġthe Ġcomparative Ġobjects , Ġwe Ġonly Ġconsider Ġthe Ġsingle ĠM RC Ġmodels Ġthat Ġrank Ġin Ġthe Ġtop Ġ20 Ġon Ġthe ĠS Qu AD Ġ1 . 1 Ġleader Ġboard Ġand Ġhave Ġreported Ġtheir Ġperformance Ġon Ġthe Ġadvers arial Ġsets . ĠThere Ġare Ġtotally Ġfive Ġsuch Ġcomparative Ġobjects , Ġwhich Ġcan Ġbe Ġconsidered Ġas Ġrepresentatives Ġof Ġthe Ġstate - of - the - art ĠM RC Ġmodels . ĠAs Ġshown Ġin ĠTable ĠT AB REF 12 Ġ, Ġon Ġthe Ġdevelopment Ġset Ġand Ġthe Ġtest Ġset , Ġthe Ġperformance Ġof ĠK AR Ġis Ġon Ġpar Ġwith Ġthat Ġof Ġthe Ġstate - of - the - art ĠM RC Ġmodels ; Ġon Ġthe Ġadvers arial Ġsets , ĠK AR Ġoutper forms Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġby Ġa Ġlarge Ġmargin . ĠThat Ġis Ġto Ġsay , ĠK AR Ġis Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġand Ġsignificantly Ġmore Ġrobust Ġto Ġnoise Ġthan Ġthem . </s> To Ġverify Ġthe Ġeffectiveness Ġof Ġgeneral Ġknowledge , Ġwe Ġfirst Ġstudy Ġthe Ġrelationship Ġbetween Ġthe Ġamount Ġof Ġgeneral Ġknowledge Ġand Ġthe Ġperformance Ġof ĠK AR . ĠAs Ġshown Ġin ĠTable ĠT AB REF 13 Ġ, Ġby Ġincreasing ĠIN LINE FORM 0 Ġfrom Ġ0 Ġto Ġ5 Ġin Ġthe Ġdata Ġenrichment Ġmethod , Ġthe Ġamount Ġof Ġgeneral Ġknowledge Ġrises Ġmon oton ically , Ġbut Ġthe Ġperformance Ġof ĠK AR Ġfirst Ġrises Ġuntil ĠIN LINE FORM 1 Ġreaches Ġ3 Ġand Ġthen Ġdrops Ġdown . ĠThen Ġwe Ġconduct Ġan Ġab lation Ġstudy Ġby Ġreplacing Ġthe Ġknowledge Ġaided Ġattention Ġmechanisms Ġwith Ġthe Ġmutual Ġattention Ġproposed Ġby ĠB IB REF 3 Ġand Ġthe Ġself Ġattention Ġproposed Ġby ĠB IB REF 4 Ġseparately , Ġand Ġfind Ġthat Ġthe ĠF 1 Ġscore Ġof ĠK AR Ġdrops Ġby ĠIN LINE FORM 2 Ġon Ġthe Ġdevelopment Ġset , ĠIN LINE FORM 3 Ġon ĠAdd Sent , Ġand ĠIN LINE FORM 4 Ġon ĠAdd One Sent . ĠFinally Ġwe Ġfind Ġthat Ġafter Ġonly Ġone Ġepoch Ġof Ġtraining , ĠK AR Ġalready Ġachieves Ġan ĠEM Ġof ĠIN LINE FORM 5 Ġand Ġan ĠF 1 Ġscore Ġof ĠIN LINE FORM 6 Ġon Ġthe Ġdevelopment Ġset , Ġwhich Ġis Ġeven Ġbetter Ġthan Ġthe Ġfinal Ġperformance Ġof Ġseveral Ġstrong Ġbas elines , Ġsuch Ġas ĠDC N Ġ( EM Ġ/ ĠF 1 : ĠIN LINE FORM 7 Ġ/ ĠIN LINE FORM 8 Ġ) ĠB IB REF 36 Ġand ĠBi DA F Ġ( EM Ġ/ ĠF 1 : ĠIN LINE FORM 9 Ġ/ ĠIN LINE FORM 10 Ġ) ĠB IB REF 3 Ġ. ĠThe Ġabove Ġempirical Ġfindings Ġimply Ġthat Ġgeneral Ġknowledge Ġindeed Ġplays Ġan Ġeffective Ġrole Ġin ĠK AR . </s> To Ġdemonstrate Ġthe Ġadvantage Ġof Ġour Ġexplicit Ġway Ġto Ġutilize Ġgeneral Ġknowledge Ġover Ġthe Ġexisting Ġimplicit Ġway , Ġwe Ġcompare Ġthe Ġperformance Ġof ĠK AR Ġwith Ġthat Ġreported Ġby ĠB IB REF 10 Ġ, Ġwhich Ġused Ġan Ġencoding - based Ġmethod Ġto Ġutilize Ġthe Ġgeneral Ġknowledge Ġdynamically Ġretrieved Ġfrom ĠWikipedia Ġand ĠConcept Net . ĠSince Ġtheir Ġbest Ġmodel Ġonly Ġachieved Ġan ĠEM Ġof ĠIN LINE FORM 0 Ġand Ġan ĠF 1 Ġscore Ġof ĠIN LINE FORM 1 Ġon Ġthe Ġdevelopment Ġset , Ġwhich Ġis Ġmuch Ġlower Ġthan Ġthe Ġperformance Ġof ĠK AR , Ġwe Ġhave Ġgood Ġreason Ġto Ġbelieve Ġthat Ġour Ġexplicit Ġway Ġworks Ġbetter Ġthan Ġthe Ġexisting Ġimplicit Ġway . </s> Model ĠComparison Ġin Ġthe ĠHunger Ġfor ĠData </s> We Ġcompare ĠK AR Ġwith Ġother ĠM RC Ġmodels Ġin Ġthe Ġhunger Ġfor Ġdata . ĠSpecifically , Ġinstead Ġof Ġusing Ġall Ġthe Ġtraining Ġexamples , Ġwe Ġproduce Ġseveral Ġtraining Ġsubs ets Ġ( i . e . Ġsubs ets Ġof Ġthe Ġtraining Ġexamples ) Ġso Ġas Ġto Ġstudy Ġthe Ġrelationship Ġbetween Ġthe Ġproportion Ġof Ġthe Ġavailable Ġtraining Ġexamples Ġand Ġthe Ġperformance . ĠWe Ġproduce Ġeach Ġtraining Ġsubset Ġby Ġsampling Ġa Ġspecific Ġnumber Ġof Ġquestions Ġfrom Ġall Ġthe Ġquestions Ġrelevant Ġto Ġeach Ġpassage . ĠBy Ġseparately Ġsampling Ġ1 , Ġ2 , Ġ3 , Ġand Ġ4 Ġquestions Ġon Ġeach Ġpassage , Ġwe Ġobtain Ġfour Ġtraining Ġsubs ets , Ġwhich Ġseparately Ġcontain ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġof Ġthe Ġtraining Ġexamples . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 15 Ġ, Ġwith ĠK AR , ĠSAN Ġ( re - im ple mented ), Ġand ĠQ AN et Ġ( re - im ple mented Ġwithout Ġdata Ġaug mentation ) Ġtrained Ġon Ġthese Ġtraining Ġsubs ets , Ġwe Ġevaluate Ġtheir Ġperformance Ġon Ġthe Ġdevelopment Ġset , Ġand Ġfind Ġthat ĠK AR Ġperforms Ġmuch Ġbetter Ġthan ĠSAN Ġand ĠQ AN et . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 16 Ġand ĠFigure ĠFIG REF 17 Ġ, Ġwith Ġthe Ġabove ĠK AR , ĠSAN , Ġand ĠQ AN et Ġtrained Ġon Ġthe Ġsame Ġtraining Ġsubs ets , Ġwe Ġalso Ġevaluate Ġtheir Ġperformance Ġon Ġthe Ġadvers arial Ġsets , Ġand Ġstill Ġfind Ġthat ĠK AR Ġperforms Ġmuch Ġbetter Ġthan ĠSAN Ġand ĠQ AN et . ĠThat Ġis Ġto Ġsay , Ġwhen Ġonly Ġa Ġsubset Ġof Ġthe Ġtraining Ġexamples Ġare Ġavailable , ĠK AR Ġoutper forms Ġthe Ġstate - of - the - art ĠM RC Ġmodels Ġby Ġa Ġlarge Ġmargin , Ġand Ġis Ġstill Ġreasonably Ġrobust Ġto Ġnoise . </s> Analysis </s> According Ġto Ġthe Ġexperimental Ġresults , ĠK AR Ġis Ġnot Ġonly Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġbut Ġalso Ġsuperior Ġto Ġthem Ġin Ġterms Ġof Ġboth Ġthe Ġhunger Ġfor Ġdata Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠThe Ġreasons Ġfor Ġthese Ġachievements , Ġwe Ġbelieve , Ġare Ġas Ġfollows : </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġinnov atively Ġintegrate Ġthe Ġneural Ġnetworks Ġof ĠM RC Ġmodels Ġwith Ġthe Ġgeneral Ġknowledge Ġof Ġhuman Ġbeings . ĠSpecifically , Ġinter - word Ġsemantic Ġconnections Ġare Ġfirst Ġextracted Ġfrom Ġeach Ġgiven Ġpassage - question Ġpair Ġby Ġa ĠWord Net - based Ġdata Ġenrichment Ġmethod , Ġand Ġthen Ġprovided Ġas Ġgeneral Ġknowledge Ġto Ġan Ġend - to - end ĠM RC Ġmodel Ġnamed Ġas ĠKnowledge ĠA ided ĠReader Ġ( K AR ), Ġwhich Ġexplicitly Ġuses Ġthe Ġgeneral Ġknowledge Ġto Ġassist Ġits Ġattention Ġmechanisms . ĠExperimental Ġresults Ġshow Ġthat ĠK AR Ġis Ġnot Ġonly Ġcomparable Ġin Ġperformance Ġwith Ġthe Ġstate - of - the - art ĠM RC Ġmodels , Ġbut Ġalso Ġsuperior Ġto Ġthem Ġin Ġterms Ġof Ġboth Ġthe Ġhunger Ġfor Ġdata Ġand Ġthe Ġrobust ness Ġto Ġnoise . ĠIn Ġthe Ġfuture , Ġwe Ġplan Ġto Ġuse Ġsome Ġlarger Ġknowledge Ġbases , Ġsuch Ġas ĠConcept Net Ġand ĠFree base , Ġto Ġimprove Ġthe Ġquality Ġand Ġscope Ġof Ġthe Ġgeneral Ġknowledge . </s> Acknowled gments </s> This Ġwork Ġis Ġpartially Ġsupported Ġby Ġa Ġresearch Ġdonation Ġfrom Ġi FL Y TE K ĠCo ., ĠLtd ., ĠHe fe i , ĠChina , Ġand Ġa Ġdiscovery Ġgrant Ġfrom ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġ( NS ERC ) Ġof ĠCanada .