(c) previous best model trained on S-SQuAD BIBREF5 by using Dr.QA BIBREF20 
EXTRACTIVE
Unanswerable
NONE
To better demonstrate the effectiveness of the proposed model, we compare with baselines and show the results in Table TABREF12 . The baselines are: (a) trained on S-SQuAD, (b) trained on T-SQuAD and then fine-tuned on S-SQuAD, and (c) previous best model trained on S-SQuAD BIBREF5 by using Dr.QA BIBREF20 . We also compare to the approach proposed by Lan et al. BIBREF16 in the row (d). This approach is originally proposed for spoken language understanding, and we adopt the same approach on the setting here. The approach models domain-specific features from the source and target domains separately by two different embedding encoders with a shared embedding encoder for modeling domain-general features. The domain-general parameters are adversarially trained by domain discriminator. Row (e) is the model that the weights of all layers are tied between the source domain and the target domain. Row (f) uses the same architecture as row (e) with an additional domain discriminator applied to the embedding encoder. It can be found that row (f) outperforms row (e), indicating that the proposed domain adversarial learning is helpful. Therefore, our following experiments contain domain adversarial learning. The proposed approach (row (f)) outperforms previous best model (row (c)) by 2% EM score and over 1.5% F1 score. We also show the results of applying the domain discriminator to the top of context query attention layer in row (g), which obtains poor performance. To sum it up, incorporating adversarial learning by applying the domain discriminator on top of the embedding encoder layer is effective.
1904.07904
Introduction </s> Question Ġanswering Ġ( Q A ) Ġhas Ġdrawn Ġa Ġlot Ġof Ġattention Ġin Ġthe Ġpast Ġfew Ġyears . ĠQ A Ġtasks Ġon Ġimages ĠB IB REF 0 Ġhave Ġbeen Ġwidely Ġstudied , Ġbut Ġmost Ġfocused Ġon Ġunderstanding Ġtext Ġdocuments ĠB IB REF 1 Ġ. ĠA Ġrepresentative Ġdataset Ġin Ġtext ĠQ A Ġis ĠS Qu AD ĠB IB REF 1 Ġ, Ġin Ġwhich Ġseveral Ġend - to - end Ġneural Ġmodels Ġhave Ġaccomplished Ġpromising Ġperformance ĠB IB REF 2 Ġ. ĠAlthough Ġthere Ġis Ġa Ġsignificant Ġprogress Ġin Ġmachine Ġcomprehension Ġ( MC ) Ġon Ġtext Ġdocuments , ĠMC Ġon Ġspoken Ġcontent Ġis Ġa Ġmuch Ġless Ġinvestigated Ġfield . ĠIn Ġspoken Ġquestion Ġanswering Ġ( S Q A ), Ġafter Ġtransc ribing Ġspoken Ġcontent Ġinto Ġtext Ġby Ġautomatic Ġspeech Ġrecognition Ġ( AS R ), Ġtypical Ġapproaches Ġuse Ġinformation Ġretrieval Ġ( IR ) Ġtechniques ĠB IB REF 3 Ġto Ġfind Ġthe Ġproper Ġanswer Ġfrom Ġthe ĠAS R Ġhypotheses . ĠOne Ġattempt Ġtowards ĠQ A Ġof Ġspoken Ġcontent Ġis ĠTO E FL Ġlistening Ġcomprehension Ġby Ġmachine ĠB IB REF 4 Ġ. ĠTO E FL Ġis Ġan ĠEnglish Ġexamination Ġthat Ġtests Ġthe Ġknowledge Ġand Ġskills Ġof Ġacademic ĠEnglish Ġfor ĠEnglish Ġlearners Ġwhose Ġnative Ġlanguages Ġare Ġnot ĠEnglish . ĠAnother ĠSQ A Ġcorpus Ġis ĠSp oken - S Qu AD ĠB IB REF 5 Ġ, Ġwhich Ġis Ġautomatically Ġgenerated Ġfrom ĠS Qu AD Ġdataset Ġthrough ĠGoogle ĠText - to - Spe ech Ġ( T TS ) Ġsystem . ĠRecently ĠO DS Q A , Ġa ĠSQ A Ġcorpus Ġrecorded Ġby Ġreal Ġspeakers , Ġis Ġreleased ĠB IB REF 6 Ġ. </s> To Ġmitigate Ġthe Ġimpact Ġof Ġspeech Ġrecognition Ġerrors , Ġusing Ġsub - word Ġunits Ġis Ġa Ġpopular Ġapproach Ġfor Ġspeech - related Ġdownstream Ġtasks . ĠIt Ġhas Ġbeen Ġapplied Ġto Ġspoken Ġdocument Ġretrieval ĠB IB REF 7 Ġand Ġspoken Ġterm Ġdetection ĠB IB REF 8 ĠThe Ġprior Ġwork Ġshowed Ġthat , Ġusing Ġphon ect ic Ġsub - word Ġunits Ġbrought Ġimprovements Ġfor Ġboth ĠSp oken - S Qu AD Ġand ĠO DS Q A ĠB IB REF 5 Ġ. </s> Instead Ġof Ġconsidering Ġsub - word Ġfeatures , Ġthis Ġpaper Ġproposes Ġa Ġnovel Ġapproach Ġto Ġmitigate Ġthe Ġimpact Ġof ĠAS R Ġerrors . ĠWe Ġconsider Ġreference Ġtranscript ions Ġand ĠAS R Ġhypotheses Ġas Ġtwo Ġdomains , Ġand Ġadapt Ġthe Ġsource Ġdomain Ġdata Ġ( reference Ġtranscript ions ) Ġto Ġthe Ġtarget Ġdomain Ġdata Ġ( AS R Ġhypotheses ) Ġby Ġprojecting Ġthese Ġtwo Ġdomains Ġin Ġthe Ġshared Ġcommon Ġspace . ĠTherefore , Ġit Ġcan Ġeffectively Ġbenefit Ġthe ĠSQ A Ġmodel Ġby Ġimproving Ġthe Ġrobust ness Ġto ĠAS R Ġerrors Ġin Ġthe ĠSQ A Ġmodel . </s> Domain Ġadaptation Ġhas Ġbeen Ġsuccessfully Ġapplied Ġon Ġcomputer Ġvision ĠB IB REF 9 Ġand Ġspeech Ġrecognition ĠB IB REF 10 Ġ. ĠIt Ġis Ġalso Ġwidely Ġstudied Ġon ĠN LP Ġtasks Ġsuch Ġas Ġsequence Ġtagging Ġand Ġparsing ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠRecently , Ġadvers arial Ġdomain Ġadaptation Ġhas Ġalready Ġbeen Ġexplored Ġon Ġspoken Ġlanguage Ġunderstanding Ġ( SL U ). ĠLiu Ġand ĠLane Ġlearned Ġdomain - general Ġfeatures Ġto Ġbenefit Ġfrom Ġmultiple Ġdialogue Ġdatasets ĠB IB REF 14 Ġ; ĠZhu Ġet Ġal . Ġlearned Ġto Ġtransfer Ġthe Ġmodel Ġfrom Ġthe Ġtranscripts Ġside Ġto Ġthe ĠAS R Ġhypotheses Ġside ĠB IB REF 15 Ġ; ĠLan Ġet Ġal . Ġconstructed Ġa Ġshared Ġspace Ġfor Ġslot Ġtagging Ġand Ġlanguage Ġmodel ĠB IB REF 16 Ġ. ĠThis Ġpaper Ġextends Ġthe Ġcapability Ġof Ġadvers arial Ġdomain Ġadaptation Ġfor ĠSQ A , Ġwhich Ġhas Ġnot Ġbeen Ġexplored Ġyet . </s> Sp oken ĠQuestion ĠAn sw ering </s> In ĠSQ A , Ġeach Ġsample Ġis Ġa Ġtriple , ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġquestion Ġin Ġeither Ġspoken Ġor Ġtext Ġform , ĠIN LINE FORM 2 Ġis Ġa Ġmulti - sent ence Ġspoken - form Ġdocument , Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġanswer Ġin Ġtext Ġfrom . ĠThe Ġtask Ġof Ġthis Ġwork Ġis Ġextract ive ĠSQ A ; Ġthat Ġmeans ĠIN LINE FORM 4 Ġis Ġa Ġword Ġspan Ġfrom Ġthe Ġreference Ġtranscription Ġof ĠIN LINE FORM 5 Ġ. ĠAn Ġoverview Ġframework Ġof ĠSQ A Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġframe Ġthe Ġsource Ġdomain Ġas Ġreference Ġtranscript ions Ġand Ġthe Ġtarget Ġdomain Ġas ĠAS R Ġhypotheses . ĠHence , Ġwe Ġcan Ġcollect Ġsource Ġdomain Ġdata Ġmore Ġeasily , Ġand Ġadapt Ġthe Ġmodel Ġto Ġthe Ġtarget Ġdomain . </s> In Ġthis Ġtask , Ġwhen Ġthe Ġmachine Ġis Ġgiven Ġa Ġspoken Ġdocument , Ġit Ġneeds Ġto Ġfind Ġthe Ġanswer Ġof Ġa Ġquestion Ġfrom Ġthe Ġspoken Ġdocument . ĠSQ A Ġcan Ġbe Ġsolved Ġby Ġthe Ġconc aten ation Ġof Ġan ĠAS R Ġmodule Ġand Ġa Ġquestion Ġanswering Ġmodule . ĠGiven Ġthe ĠAS R Ġhypotheses Ġof Ġa Ġspoken Ġdocument Ġand Ġa Ġquestion , Ġthe Ġquestion Ġanswering Ġmodule Ġcan Ġoutput Ġa Ġtext Ġanswer . </s> The Ġmost Ġintuitive Ġway Ġto Ġevaluate Ġthe Ġtext Ġanswer Ġis Ġto Ġdirectly Ġcompute Ġthe ĠEx act ĠMatch Ġ( EM ) Ġand ĠMacro - aver aged ĠF 1 Ġscores Ġ( F 1 ) Ġbetween Ġthe Ġpredicted Ġtext Ġanswer Ġand Ġthe Ġground - truth Ġtext Ġanswer . ĠWe Ġused Ġthe Ġstandard Ġevaluation Ġscript Ġfrom ĠS Qu AD ĠB IB REF 1 Ġto Ġevaluate Ġthe Ġperformance . </s> Question ĠAn sw ering ĠModel </s> The Ġused Ġarchitecture Ġof Ġthe ĠQ A Ġmodel Ġis Ġbriefly Ġsummarized Ġbelow . ĠHere Ġwe Ġchoose ĠQ AN et ĠB IB REF 2 Ġas Ġthe Ġbase Ġmodel Ġdue Ġto Ġthe Ġfollowing Ġreasons : Ġ1 ) Ġit Ġachieves Ġthe Ġsecond Ġbest Ġperformance Ġon ĠS Qu AD , Ġand Ġ2 ) Ġsince Ġthere Ġare Ġcompletely Ġno Ġrecurrent Ġnetworks Ġin ĠQ AN et , Ġits Ġtraining Ġspeed Ġis Ġ5 x Ġfaster Ġthan ĠBi DA F ĠB IB REF 17 Ġwhen Ġreaching Ġthe Ġsame Ġperformance Ġon ĠS Qu AD . </s> The Ġnetwork Ġarchitecture Ġis Ġillustrated Ġin ĠFigure ĠFIG REF 2 Ġ. ĠThe Ġleft Ġblocks Ġand Ġthe Ġright Ġblocks Ġform Ġtwo ĠQ AN ets , Ġeach Ġof Ġwhich Ġtakes Ġa Ġdocument Ġand Ġa Ġquestion Ġas Ġthe Ġinput Ġand Ġoutputs Ġan Ġanswer . ĠIn ĠQ AN et , Ġfirst ly , Ġan Ġembed ding Ġenc oder Ġob tains Ġword Ġand Ġcharacter Ġembed d ings Ġfor Ġeach Ġword Ġin ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġand Ġthen Ġmodels Ġthe Ġtemporal Ġinteractions Ġbetween Ġwords Ġand Ġref ines Ġword Ġvectors Ġto Ġcontextual ized Ġword Ġrepresentations . ĠAll Ġenc oder Ġblocks Ġused Ġin ĠQ AN et Ġare Ġcomposed Ġexclusively Ġof Ġdepth - wise Ġsepar able Ġconv olutions Ġand Ġself - att ention . ĠThe Ġintuition Ġhere Ġis Ġthat Ġconv olution Ġcomponents Ġcan Ġmodel Ġlocal Ġinteractions Ġand Ġself - att ention Ġcomponents Ġfocus Ġon Ġmodeling Ġglobal Ġinteractions . ĠThe Ġcontext - query Ġattention Ġlayer Ġgenerates Ġthe Ġquestion - document Ġsimilarity Ġmatrix Ġand Ġcomp utes Ġthe Ġquestion - aware Ġvector Ġrepresentations Ġof Ġthe Ġcontext Ġwords . ĠAfter Ġthat , Ġa Ġmodel Ġenc oder Ġlayer Ġcontaining Ġseven Ġenc oder Ġblocks Ġcaptures Ġthe Ġinteractions Ġamong Ġthe Ġcontext Ġwords Ġconditioned Ġon Ġthe Ġquestion . ĠFinally , Ġthe Ġoutput Ġlayer Ġpredicts Ġa Ġstart Ġposition Ġand Ġan Ġend Ġposition Ġin Ġthe Ġdocument Ġto Ġextract Ġthe Ġanswer Ġspan Ġfrom Ġthe Ġdocument . </s> Domain ĠAdapt ation ĠApproach </s> The Ġmain Ġfocus Ġof Ġthis Ġpaper Ġis Ġto Ġapply Ġdomain Ġadaptation Ġfor ĠSQ A . ĠIn Ġthis Ġapproach , Ġwe Ġhave Ġtwo ĠSQ A Ġmodels Ġ( Q AN ets ), Ġone Ġtrained Ġfrom Ġtarget Ġdomain Ġdata Ġ( AS R Ġhypotheses ) Ġand Ġanother Ġtrained Ġfrom Ġsource Ġdomain Ġdata Ġ( reference Ġtranscript ions ). ĠBecause Ġthe Ġtwo Ġdomains Ġshare Ġcommon Ġinformation , Ġsome Ġlayers Ġin Ġthese Ġtwo Ġmodels Ġcan Ġbe Ġtied Ġin Ġorder Ġto Ġmodel Ġthe Ġshared Ġfeatures . ĠHence , Ġwe Ġcan Ġchoose Ġwhether Ġeach Ġlayer Ġin Ġthe ĠQ A Ġmodel Ġshould Ġbe Ġshared . ĠT ying Ġthe Ġweights Ġbetween Ġthe Ġsource Ġlayer Ġand Ġthe Ġtarget Ġlayer Ġin Ġorder Ġto Ġlearn Ġa Ġsymm etric Ġmapping Ġis Ġto Ġproject Ġboth Ġsource Ġand Ġtarget Ġdomain Ġdata Ġto Ġa Ġshared Ġcommon Ġspace . ĠDifferent Ġcombinations Ġwill Ġbe Ġinvestigated Ġin Ġour Ġexperiments . </s> More Ġspecifically , Ġwe Ġincorporate Ġa Ġdomain Ġdiscrim inator Ġinto Ġthe ĠSQ A Ġmodel Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ, Ġwhich Ġcan Ġenforce Ġthe Ġembed ding Ġenc oder Ġto Ġproject Ġthe Ġsentences Ġfrom Ġboth Ġsource Ġand Ġtarget Ġdomains Ġinto Ġa Ġshared Ġcommon Ġspace Ġand Ġconsequ entially Ġto Ġbe ĠAS R - error Ġrobust . ĠAlthough Ġthe Ġembed ding Ġenc oder Ġfor Ġboth Ġdomains Ġmay Ġimplicitly Ġlearn Ġsome Ġcommon Ġlatent Ġrepresentations , Ġadvers arial Ġlearning Ġcan Ġprovide Ġa Ġmore Ġdirect Ġtraining Ġsignal Ġfor Ġalign ing Ġthe Ġoutput Ġdistribution Ġof Ġthe Ġembed ding Ġenc oder Ġfrom Ġboth Ġdomains . ĠThe Ġembed ding Ġenc oder Ġtakes Ġin Ġa Ġsequence Ġof Ġword Ġvectors Ġand Ġgenerates Ġa Ġsequence Ġof Ġhidden Ġvectors Ġwith Ġthe Ġsame Ġlength . ĠWe Ġuse ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ( ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ) Ġto Ġrepresent Ġthe Ġhidden Ġvector Ġsequence Ġgiven Ġthe Ġquestion ĠIN LINE FORM 4 Ġand Ġthe Ġdocument ĠIN LINE FORM 5 Ġin Ġthe Ġtarget Ġ( source ) Ġdomain Ġrespectively . </s> The Ġdomain Ġdiscrim inator ĠIN LINE FORM 0 Ġfocuses Ġon Ġidentifying Ġthe Ġdomain Ġof Ġthe Ġvector Ġsequence Ġis Ġfrom Ġgiven ĠIN LINE FORM 1 Ġor ĠIN LINE FORM 2 Ġ, Ġwhere Ġthe Ġobjective Ġis Ġto Ġminimize ĠIN LINE FORM 3 Ġ. ĠDIS PLAY FORM 0 </s> Given Ġa Ġtraining Ġexample Ġfrom Ġthe Ġtarget Ġdomain Ġ( ĠIN LINE FORM 0 Ġ), ĠIN LINE FORM 1 Ġlearns Ġto Ġassign Ġa Ġlower Ġscore Ġto ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġin Ġthat Ġexample , Ġthat Ġis , Ġto Ġminimize ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ. ĠOn Ġthe Ġother Ġhand , Ġgiven Ġa Ġtraining Ġexample Ġfrom Ġthe Ġsource Ġdomain Ġ( ĠIN LINE FORM 6 Ġ), ĠIN LINE FORM 7 Ġlearns Ġto Ġassign Ġa Ġlarger Ġvalue Ġto ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ. </s> Furthermore , Ġwe Ġupdate Ġthe Ġparameters Ġof Ġthe Ġembed ding Ġenc od ers Ġto Ġmaximize Ġthe Ġdomain Ġclassification Ġloss ĠIN LINE FORM 0 Ġ, Ġwhich Ġworks Ġadvers ar ially Ġtowards Ġthe Ġdomain Ġdiscrim inator . ĠWe Ġthus Ġexpect Ġthe Ġmodel Ġto Ġlearn Ġfeatures Ġand Ġstructures Ġthat Ġcan Ġgeneral ize Ġacross Ġdomains Ġwhen Ġthe Ġoutputs Ġof ĠIN LINE FORM 1 Ġare Ġindistinguishable Ġfrom Ġthe Ġoutputs Ġof ĠIN LINE FORM 2 Ġ. ĠThe Ġloss Ġfunction Ġfor Ġembed ding Ġenc oder , ĠIN LINE FORM 3 Ġ, Ġis Ġformulated Ġas ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġhyper param eter . ĠThe Ġtwo Ġembed ding Ġenc od ers Ġin Ġthe ĠQ A Ġmodel Ġare Ġlearned Ġto Ġmaximize ĠIN LINE FORM 1 Ġwhile Ġminimizing Ġthe Ġloss Ġfor ĠQ A , ĠIN LINE FORM 2 Ġ. ĠBecause Ġthe Ġparameters Ġof Ġother Ġlayers Ġin ĠQ A Ġmodel Ġare Ġindependent Ġto Ġthe Ġloss Ġof Ġthe Ġdomain Ġdiscrim inator , Ġthe Ġloss Ġfunction Ġof Ġother Ġlayers , ĠIN LINE FORM 3 Ġ, Ġis Ġequivalent Ġto ĠIN LINE FORM 4 Ġ, Ġthat Ġis , ĠIN LINE FORM 5 Ġ. </s> Although Ġthe Ġdiscrim inator Ġis Ġapplied Ġto Ġthe Ġoutput Ġof Ġembed ding Ġenc oder Ġin ĠFigure ĠFIG REF 2 Ġ, Ġit Ġcan Ġbe Ġalso Ġapplied Ġto Ġother Ġlayers . ĠConsidering Ġthat Ġalmost Ġall ĠQ A Ġmodel Ġcontains Ġsuch Ġembed ding Ġenc od ers , Ġthe Ġproposed Ġapproach Ġis Ġexpected Ġto Ġgeneral ize Ġto Ġother ĠQ A Ġmodels Ġin Ġaddition Ġto ĠQ AN et . </s> Corp us </s> Sp oken - S Qu AD Ġis Ġchosen Ġas Ġthe Ġtarget Ġdomain Ġdata Ġfor Ġtraining Ġand Ġtesting . ĠSp oken - S Qu AD ĠB IB REF 5 Ġis Ġan Ġautomatically Ġgenerated Ġcorpus Ġin Ġwhich Ġthe Ġdocument Ġis Ġin Ġspoken Ġform Ġand Ġthe Ġquestion Ġis Ġin Ġtext Ġform . ĠThe Ġreference Ġtranscript ions Ġare Ġfrom ĠS Qu AD ĠB IB REF 1 Ġ. ĠThere Ġare Ġ37 , 111 Ġand Ġ5 , 351 Ġquestion Ġanswer Ġpairs Ġin Ġthe Ġtraining Ġand Ġtesting Ġsets Ġrespectively , Ġand Ġthe Ġword Ġerror Ġrate Ġ( WER ) Ġof Ġboth Ġsets Ġis Ġaround Ġ22 . 7 %. </s> The Ġoriginal ĠS Qu AD , ĠText - S Qu AD , Ġis Ġchosen Ġas Ġthe Ġsource Ġdomain Ġdata , Ġwhere Ġonly Ġquestion Ġanswering Ġpairs Ġappearing Ġin ĠSp oken - S Qu AD Ġare Ġutilized . ĠIn Ġour Ġtask Ġsetting , Ġduring Ġtraining Ġwe Ġtrain Ġthe Ġproposed ĠQ A Ġmodel Ġon Ġboth ĠText - S Qu AD Ġand ĠSp oken - S Qu AD Ġtraining Ġsets . ĠWhile Ġin Ġthe Ġtesting Ġstage , Ġwe Ġevaluate Ġthe Ġperformance Ġon ĠSp oken - S Qu AD Ġtesting Ġset . </s> Exper iment ĠSetup </s> We Ġutilize Ġfast text ĠB IB REF 18 Ġto Ġgenerate Ġthe Ġembed d ings Ġof Ġall Ġwords Ġfrom Ġboth ĠText - S Qu AD Ġand ĠSp oken - S Qu AD . ĠWe Ġadopt Ġthe Ġphon eme Ġsequence Ġembed d ings Ġto Ġreplace Ġthe Ġoriginal Ġcharacter Ġsequence Ġembed d ings Ġusing Ġthe Ġmethod Ġproposed Ġby ĠLi Ġet Ġal . ĠB IB REF 5 Ġ. ĠThe Ġsource Ġdomain Ġmodel Ġand Ġthe Ġtarget Ġdomain Ġmodel Ġshare Ġthe Ġsame Ġset Ġof Ġword Ġembed ding Ġmatrix Ġto Ġimprove Ġthe Ġalignment Ġbetween Ġthese Ġtwo Ġdomains . </s> W - GAN Ġis Ġadopted Ġfor Ġour Ġdomain Ġdiscrim inator ĠB IB REF 19 Ġ, Ġwhich Ġstacks Ġ5 Ġresidual Ġblocks Ġof Ġ1 D Ġconv olution al Ġlayers Ġwith Ġ96 Ġfilters Ġand Ġfilter Ġsize Ġ5 Ġfollowed Ġby Ġone Ġlinear Ġlayer Ġto Ġconvert Ġeach Ġinput Ġvector Ġsequence Ġinto Ġone Ġscal ar Ġvalue . </s> All Ġmodels Ġused Ġin Ġthe Ġexperiments Ġare Ġtrained Ġwith Ġbatch Ġsize Ġ20 , Ġusing Ġadam Ġwith Ġlearning Ġrate ĠIN LINE FORM 0 Ġand Ġthe Ġearly Ġstop Ġstrategy . ĠThe Ġdimension Ġof Ġthe Ġhidden Ġstate Ġis Ġset Ġto Ġ96 Ġfor Ġall Ġlayers , Ġand Ġthe Ġnumber Ġof Ġself - att ention Ġheads Ġis Ġset Ġto Ġ2 . ĠThe Ġsetup Ġis Ġslightly Ġdifferent Ġbut Ġbetter Ġthan Ġthe Ġsetting Ġsuggested Ġby Ġthe Ġoriginal ĠQ An et . </s> Results </s> First , Ġwe Ġhighlight Ġthe Ġdomain Ġmismatch Ġphenomenon Ġin Ġour Ġexperiments Ġshown Ġin ĠTable ĠT AB REF 9 Ġ. ĠRow Ġ( a ) Ġis Ġwhen ĠQ AN et Ġis Ġtrained Ġon ĠText - S Qu AD , Ġrow Ġ( b ) Ġis Ġwhen ĠQ AN et Ġis Ġtrained Ġon ĠSp oken - S Qu AD , Ġand Ġrow Ġ( c ) Ġis Ġwhen ĠQ AN et Ġis Ġtrained Ġon ĠText - S Qu AD Ġand Ġthen Ġfin et uned Ġon ĠSp oken - S Qu AD . ĠThe Ġcolumns Ġshow Ġthe Ġevaluation Ġon Ġthe Ġtesting Ġsets Ġof ĠText - S Qu AD Ġand ĠSp oken - S Qu AD . </s> It Ġis Ġclear Ġthat Ġthe Ġperformance Ġdrops Ġa Ġlot Ġwhen Ġthe Ġtraining Ġand Ġtesting Ġdata Ġmismatch , Ġindicating Ġthat Ġmodel Ġtraining Ġon ĠAS R Ġhypotheses Ġcan Ġnot Ġgeneral ize Ġwell Ġon Ġreference Ġtranscript ions . ĠThe Ġperformance Ġgap Ġis Ġnearly Ġ20 % ĠF 1 Ġscore Ġ( 72 % Ġto Ġ55 %). ĠThe Ġrow Ġ( c ) Ġshows Ġthe Ġimproved Ġperformance Ġwhen Ġtesting Ġon ĠS - S Qu AD Ġdue Ġto Ġthe Ġtransfer Ġlearning Ġvia Ġfine - tun ing . </s> To Ġbetter Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġmodel , Ġwe Ġcompare Ġwith Ġbas elines Ġand Ġshow Ġthe Ġresults Ġin ĠTable ĠT AB REF 12 Ġ. ĠThe Ġbas elines Ġare : Ġ( a ) Ġtrained Ġon ĠS - S Qu AD , Ġ( b ) Ġtrained Ġon ĠT - S Qu AD Ġand Ġthen Ġfine - tun ed Ġon ĠS - S Qu AD , Ġand Ġ( c ) Ġprevious Ġbest Ġmodel Ġtrained Ġon ĠS - S Qu AD ĠB IB REF 5 Ġby Ġusing ĠDr . Q A ĠB IB REF 20 Ġ. ĠWe Ġalso Ġcompare Ġto Ġthe Ġapproach Ġproposed Ġby ĠLan Ġet Ġal . ĠB IB REF 16 Ġin Ġthe Ġrow Ġ( d ). ĠThis Ġapproach Ġis Ġoriginally Ġproposed Ġfor Ġspoken Ġlanguage Ġunderstanding , Ġand Ġwe Ġadopt Ġthe Ġsame Ġapproach Ġon Ġthe Ġsetting Ġhere . ĠThe Ġapproach Ġmodels Ġdomain - specific Ġfeatures Ġfrom Ġthe Ġsource Ġand Ġtarget Ġdomains Ġseparately Ġby Ġtwo Ġdifferent Ġembed ding Ġenc od ers Ġwith Ġa Ġshared Ġembed ding Ġenc oder Ġfor Ġmodeling Ġdomain - general Ġfeatures . ĠThe Ġdomain - general Ġparameters Ġare Ġadvers ar ially Ġtrained Ġby Ġdomain Ġdiscrim inator . </s> Row Ġ( e ) Ġis Ġthe Ġmodel Ġthat Ġthe Ġweights Ġof Ġall Ġlayers Ġare Ġtied Ġbetween Ġthe Ġsource Ġdomain Ġand Ġthe Ġtarget Ġdomain . ĠRow Ġ( f ) Ġuses Ġthe Ġsame Ġarchitecture Ġas Ġrow Ġ( e ) Ġwith Ġan Ġadditional Ġdomain Ġdiscrim inator Ġapplied Ġto Ġthe Ġembed ding Ġenc oder . ĠIt Ġcan Ġbe Ġfound Ġthat Ġrow Ġ( f ) Ġoutper forms Ġrow Ġ( e ), Ġindicating Ġthat Ġthe Ġproposed Ġdomain Ġadvers arial Ġlearning Ġis Ġhelpful . ĠTherefore , Ġour Ġfollowing Ġexperiments Ġcontain Ġdomain Ġadvers arial Ġlearning . ĠThe Ġproposed Ġapproach Ġ( row Ġ( f )) Ġoutper forms Ġprevious Ġbest Ġmodel Ġ( row Ġ( c )) Ġby Ġ2 % ĠEM Ġscore Ġand Ġover Ġ1 . 5 % ĠF 1 Ġscore . ĠWe Ġalso Ġshow Ġthe Ġresults Ġof Ġapplying Ġthe Ġdomain Ġdiscrim inator Ġto Ġthe Ġtop Ġof Ġcontext Ġquery Ġattention Ġlayer Ġin Ġrow Ġ( g ), Ġwhich Ġob tains Ġpoor Ġperformance . ĠTo Ġsum Ġit Ġup , Ġincorporating Ġadvers arial Ġlearning Ġby Ġapplying Ġthe Ġdomain Ġdiscrim inator Ġon Ġtop Ġof Ġthe Ġembed ding Ġenc oder Ġlayer Ġis Ġeffective . </s> Layer Ġweight Ġtying Ġor Ġunt ying Ġwithin Ġthe Ġmodel Ġindicates Ġdifferent Ġlevels Ġof Ġsymm etric Ġmapping Ġbetween Ġthe Ġsource Ġand Ġtarget Ġdomains . ĠDifferent Ġcombinations Ġare Ġinvestigated Ġand Ġshown Ġin ĠTable ĠT AB REF 14 Ġ. ĠThe Ġrow Ġ( a ) Ġin Ġwhich Ġall Ġlayers Ġare Ġtied Ġis Ġthe Ġrow Ġ( e ) Ġof ĠTable ĠT AB REF 12 Ġ. ĠThe Ġresults Ġshow Ġthat Ġunt ying Ġcontext - query Ġattention Ġlayer ĠL 2 Ġ( rows Ġ( c , Ġf , Ġg )) Ġor Ġmodel Ġenc oder Ġlayer ĠL 3 Ġ( rows Ġ( d , Ġf , Ġh )) Ġlead Ġto Ġdegener ated Ġsolutions Ġin Ġcomparison Ġto Ġrow Ġ( a ) Ġwhere Ġall Ġlayers Ġare Ġtied . ĠUnt ying Ġboth Ġof Ġthem Ġsimultaneously Ġleads Ġto Ġthe Ġworst Ġperformance Ġwhich Ġis Ġeven Ġworse Ġthan Ġthe Ġfin et uning Ġ( row Ġ( g ) Ġv . s . Ġ( c ) Ġfrom ĠTable ĠT AB REF 12 Ġ). ĠThese Ġresults Ġimply Ġthat Ġsharing Ġthe Ġcontext - query Ġattention Ġlayer Ġand Ġthe Ġmodel Ġenc oder Ġlayer Ġare Ġimportant Ġfor Ġdomain Ġadaptation Ġon ĠSQ A . ĠWe Ġconjecture Ġthat Ġthese Ġtwo Ġlayers Ġbenefit Ġfrom Ġtraining Ġon Ġsource Ġdomain Ġdata Ġwhere Ġthere Ġare Ġno ĠAS R Ġerrors , Ġso Ġthe ĠQ A Ġmodel Ġlearns Ġto Ġconduct Ġattention Ġor Ġfurther Ġreason Ġwell Ġon Ġtarget Ġdomain Ġdata Ġwith ĠAS R Ġerrors . </s> Overall , Ġit Ġis Ġnot Ġbeneficial Ġto Ġunt ie Ġany Ġlayer , Ġbecause Ġno Ġinformation Ġcan Ġbe Ġshared Ġacross Ġdifferent Ġdomains . ĠUnt ying Ġthe Ġembed ding Ġenc oder ĠL 1 Ġand Ġthe Ġoutput Ġlayer ĠL 4 Ġleads Ġto Ġthe Ġleast Ġdegradation Ġin Ġcomparison Ġto Ġrow Ġ( a ). </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġincorporate Ġa Ġdomain Ġdiscrim inator Ġto Ġalign Ġthe Ġmism atched Ġdomains Ġbetween ĠAS R Ġhypotheses Ġand Ġreference Ġtranscript ions . ĠThe Ġadvers arial Ġlearning Ġallows Ġthe Ġend - to - end ĠQ A Ġmodel Ġto Ġlearn Ġdomain - inv ari ant Ġfeatures Ġand Ġimprove Ġthe Ġrobust ness Ġto ĠAS R Ġerrors . ĠThe Ġexperiments Ġdemonstrate Ġthat Ġthe Ġproposed Ġmodel Ġsuccessfully Ġachieves Ġsuperior Ġperformance Ġand Ġoutper forms Ġthe Ġprevious Ġbest Ġmodel Ġby Ġ2 % ĠEM Ġscore Ġand Ġover Ġ1 . 5 % ĠF 1 Ġscore .
Introduction Question answering (QA) has drawn a lot of attention in the past few years. QA tasks on images BIBREF0 have been widely studied, but most focused on understanding text documents BIBREF1 . A representative dataset in text QA is SQuAD BIBREF1 , in which several end-to-end neural models have accomplished promising performance BIBREF2 . Although there is a significant progress in machine comprehension (MC) on text documents, MC on spoken content is a much less investigated field. In spoken question answering (SQA), after transcribing spoken content into text by automatic speech recognition (ASR), typical approaches use information retrieval (IR) techniques BIBREF3 to find the proper answer from the ASR hypotheses. One attempt towards QA of spoken content is TOEFL listening comprehension by machine BIBREF4 . TOEFL is an English examination that tests the knowledge and skills of academic English for English learners whose native languages are not English. Another SQA corpus is Spoken-SQuAD BIBREF5 , which is automatically generated from SQuAD dataset through Google Text-to-Speech (TTS) system. Recently ODSQA, a SQA corpus recorded by real speakers, is released BIBREF6 . To mitigate the impact of speech recognition errors, using sub-word units is a popular approach for speech-related downstream tasks. It has been applied to spoken document retrieval BIBREF7 and spoken term detection BIBREF8 The prior work showed that, using phonectic sub-word units brought improvements for both Spoken-SQuAD and ODSQA BIBREF5 . Instead of considering sub-word features, this paper proposes a novel approach to mitigate the impact of ASR errors. We consider reference transcriptions and ASR hypotheses as two domains, and adapt the source domain data (reference transcriptions) to the target domain data (ASR hypotheses) by projecting these two domains in the shared common space. Therefore, it can effectively benefit the SQA model by improving the robustness to ASR errors in the SQA model. Domain adaptation has been successfully applied on computer vision BIBREF9 and speech recognition BIBREF10 . It is also widely studied on NLP tasks such as sequence tagging and parsing BIBREF11 , BIBREF12 , BIBREF13 . Recently, adversarial domain adaptation has already been explored on spoken language understanding (SLU). Liu and Lane learned domain-general features to benefit from multiple dialogue datasets BIBREF14 ; Zhu et al. learned to transfer the model from the transcripts side to the ASR hypotheses side BIBREF15 ; Lan et al. constructed a shared space for slot tagging and language model BIBREF16 . This paper extends the capability of adversarial domain adaptation for SQA, which has not been explored yet. Spoken Question Answering In SQA, each sample is a triple, INLINEFORM0 , where INLINEFORM1 is a question in either spoken or text form, INLINEFORM2 is a multi-sentence spoken-form document, and INLINEFORM3 is the answer in text from. The task of this work is extractive SQA; that means INLINEFORM4 is a word span from the reference transcription of INLINEFORM5 . An overview framework of SQA is shown in Figure FIGREF1 . In this paper, we frame the source domain as reference transcriptions and the target domain as ASR hypotheses. Hence, we can collect source domain data more easily, and adapt the model to the target domain. In this task, when the machine is given a spoken document, it needs to find the answer of a question from the spoken document. SQA can be solved by the concatenation of an ASR module and a question answering module. Given the ASR hypotheses of a spoken document and a question, the question answering module can output a text answer. The most intuitive way to evaluate the text answer is to directly compute the Exact Match (EM) and Macro-averaged F1 scores (F1) between the predicted text answer and the ground-truth text answer. We used the standard evaluation script from SQuAD BIBREF1 to evaluate the performance. Question Answering Model The used architecture of the QA model is briefly summarized below. Here we choose QANet BIBREF2 as the base model due to the following reasons: 1) it achieves the second best performance on SQuAD, and 2) since there are completely no recurrent networks in QANet, its training speed is 5x faster than BiDAF BIBREF17 when reaching the same performance on SQuAD. The network architecture is illustrated in Figure FIGREF2 . The left blocks and the right blocks form two QANets, each of which takes a document and a question as the input and outputs an answer. In QANet, firstly, an embedding encoder obtains word and character embeddings for each word in INLINEFORM0 or INLINEFORM1 and then models the temporal interactions between words and refines word vectors to contextualized word representations. All encoder blocks used in QANet are composed exclusively of depth-wise separable convolutions and self-attention. The intuition here is that convolution components can model local interactions and self-attention components focus on modeling global interactions. The context-query attention layer generates the question-document similarity matrix and computes the question-aware vector representations of the context words. After that, a model encoder layer containing seven encoder blocks captures the interactions among the context words conditioned on the question. Finally, the output layer predicts a start position and an end position in the document to extract the answer span from the document. Domain Adaptation Approach The main focus of this paper is to apply domain adaptation for SQA. In this approach, we have two SQA models (QANets), one trained from target domain data (ASR hypotheses) and another trained from source domain data (reference transcriptions). Because the two domains share common information, some layers in these two models can be tied in order to model the shared features. Hence, we can choose whether each layer in the QA model should be shared. Tying the weights between the source layer and the target layer in order to learn a symmetric mapping is to project both source and target domain data to a shared common space. Different combinations will be investigated in our experiments. More specifically, we incorporate a domain discriminator into the SQA model shown in Figure FIGREF2 , which can enforce the embedding encoder to project the sentences from both source and target domains into a shared common space and consequentially to be ASR-error robust. Although the embedding encoder for both domains may implicitly learn some common latent representations, adversarial learning can provide a more direct training signal for aligning the output distribution of the embedding encoder from both domains. The embedding encoder takes in a sequence of word vectors and generates a sequence of hidden vectors with the same length. We use INLINEFORM0 and INLINEFORM1 ( INLINEFORM2 and INLINEFORM3 ) to represent the hidden vector sequence given the question INLINEFORM4 and the document INLINEFORM5 in the target (source) domain respectively. The domain discriminator INLINEFORM0 focuses on identifying the domain of the vector sequence is from given INLINEFORM1 or INLINEFORM2 , where the objective is to minimize INLINEFORM3 . DISPLAYFORM0 Given a training example from the target domain ( INLINEFORM0 ), INLINEFORM1 learns to assign a lower score to INLINEFORM2 and INLINEFORM3 in that example, that is, to minimize INLINEFORM4 and INLINEFORM5 . On the other hand, given a training example from the source domain ( INLINEFORM6 ), INLINEFORM7 learns to assign a larger value to INLINEFORM8 and INLINEFORM9 . Furthermore, we update the parameters of the embedding encoders to maximize the domain classification loss INLINEFORM0 , which works adversarially towards the domain discriminator. We thus expect the model to learn features and structures that can generalize across domains when the outputs of INLINEFORM1 are indistinguishable from the outputs of INLINEFORM2 . The loss function for embedding encoder, INLINEFORM3 , is formulated as DISPLAYFORM0 where INLINEFORM0 is a hyperparameter. The two embedding encoders in the QA model are learned to maximize INLINEFORM1 while minimizing the loss for QA, INLINEFORM2 . Because the parameters of other layers in QA model are independent to the loss of the domain discriminator, the loss function of other layers, INLINEFORM3 , is equivalent to INLINEFORM4 , that is, INLINEFORM5 . Although the discriminator is applied to the output of embedding encoder in Figure FIGREF2 , it can be also applied to other layers. Considering that almost all QA model contains such embedding encoders, the proposed approach is expected to generalize to other QA models in addition to QANet. Corpus Spoken-SQuAD is chosen as the target domain data for training and testing. Spoken-SQuAD BIBREF5 is an automatically generated corpus in which the document is in spoken form and the question is in text form. The reference transcriptions are from SQuAD BIBREF1 . There are 37,111 and 5,351 question answer pairs in the training and testing sets respectively, and the word error rate (WER) of both sets is around 22.7%. The original SQuAD, Text-SQuAD, is chosen as the source domain data, where only question answering pairs appearing in Spoken-SQuAD are utilized. In our task setting, during training we train the proposed QA model on both Text-SQuAD and Spoken-SQuAD training sets. While in the testing stage, we evaluate the performance on Spoken-SQuAD testing set. Experiment Setup We utilize fasttext BIBREF18 to generate the embeddings of all words from both Text-SQuAD and Spoken-SQuAD. We adopt the phoneme sequence embeddings to replace the original character sequence embeddings using the method proposed by Li et al. BIBREF5 . The source domain model and the target domain model share the same set of word embedding matrix to improve the alignment between these two domains. W-GAN is adopted for our domain discriminator BIBREF19 , which stacks 5 residual blocks of 1D convolutional layers with 96 filters and filter size 5 followed by one linear layer to convert each input vector sequence into one scalar value. All models used in the experiments are trained with batch size 20, using adam with learning rate INLINEFORM0 and the early stop strategy. The dimension of the hidden state is set to 96 for all layers, and the number of self-attention heads is set to 2. The setup is slightly different but better than the setting suggested by the original QAnet. Results First, we highlight the domain mismatch phenomenon in our experiments shown in Table TABREF9 . Row (a) is when QANet is trained on Text-SQuAD, row (b) is when QANet is trained on Spoken-SQuAD, and row (c) is when QANet is trained on Text-SQuAD and then finetuned on Spoken-SQuAD. The columns show the evaluation on the testing sets of Text-SQuAD and Spoken-SQuAD. It is clear that the performance drops a lot when the training and testing data mismatch, indicating that model training on ASR hypotheses can not generalize well on reference transcriptions. The performance gap is nearly 20% F1 score (72% to 55%). The row (c) shows the improved performance when testing on S-SQuAD due to the transfer learning via fine-tuning. To better demonstrate the effectiveness of the proposed model, we compare with baselines and show the results in Table TABREF12 . The baselines are: (a) trained on S-SQuAD, (b) trained on T-SQuAD and then fine-tuned on S-SQuAD, and (c) previous best model trained on S-SQuAD BIBREF5 by using Dr.QA BIBREF20 . We also compare to the approach proposed by Lan et al. BIBREF16 in the row (d). This approach is originally proposed for spoken language understanding, and we adopt the same approach on the setting here. The approach models domain-specific features from the source and target domains separately by two different embedding encoders with a shared embedding encoder for modeling domain-general features. The domain-general parameters are adversarially trained by domain discriminator. Row (e) is the model that the weights of all layers are tied between the source domain and the target domain. Row (f) uses the same architecture as row (e) with an additional domain discriminator applied to the embedding encoder. It can be found that row (f) outperforms row (e), indicating that the proposed domain adversarial learning is helpful. Therefore, our following experiments contain domain adversarial learning. The proposed approach (row (f)) outperforms previous best model (row (c)) by 2% EM score and over 1.5% F1 score. We also show the results of applying the domain discriminator to the top of context query attention layer in row (g), which obtains poor performance. To sum it up, incorporating adversarial learning by applying the domain discriminator on top of the embedding encoder layer is effective. Layer weight tying or untying within the model indicates different levels of symmetric mapping between the source and target domains. Different combinations are investigated and shown in Table TABREF14 . The row (a) in which all layers are tied is the row (e) of Table TABREF12 . The results show that untying context-query attention layer L2 (rows (c, f, g)) or model encoder layer L3 (rows (d, f, h)) lead to degenerated solutions in comparison to row (a) where all layers are tied. Untying both of them simultaneously leads to the worst performance which is even worse than the finetuning (row (g) v.s. (c) from Table TABREF12 ). These results imply that sharing the context-query attention layer and the model encoder layer are important for domain adaptation on SQA. We conjecture that these two layers benefit from training on source domain data where there are no ASR errors, so the QA model learns to conduct attention or further reason well on target domain data with ASR errors. Overall, it is not beneficial to untie any layer, because no information can be shared across different domains. Untying the embedding encoder L1 and the output layer L4 leads to the least degradation in comparison to row (a). Conclusion In this work, we incorporate a domain discriminator to align the mismatched domains between ASR hypotheses and reference transcriptions. The adversarial learning allows the end-to-end QA model to learn domain-invariant features and improve the robustness to ASR errors. The experiments demonstrate that the proposed model successfully achieves superior performance and outperforms the previous best model by 2% EM score and over 1.5% F1 score.
What was the previous best model?
7ee5c45b127fb284a4a9e72bb9b980a602f7445a
What Ġwas Ġthe Ġprevious Ġbest Ġmodel ?
( c ) Ġprevious Ġbest Ġmodel Ġtrained Ġon ĠS - S Qu AD ĠB IB REF 5 Ġby Ġusing ĠDr . Q A ĠB IB REF 20 Ġ
<s> What Ġwas Ġthe Ġprevious Ġbest Ġmodel ? </s> Introduction </s> Question Ġanswering Ġ( Q A ) Ġhas Ġdrawn Ġa Ġlot Ġof Ġattention Ġin Ġthe Ġpast Ġfew Ġyears . ĠQ A Ġtasks Ġon Ġimages ĠB IB REF 0 Ġhave Ġbeen Ġwidely Ġstudied , Ġbut Ġmost Ġfocused Ġon Ġunderstanding Ġtext Ġdocuments ĠB IB REF 1 Ġ. ĠA Ġrepresentative Ġdataset Ġin Ġtext ĠQ A Ġis ĠS Qu AD ĠB IB REF 1 Ġ, Ġin Ġwhich Ġseveral Ġend - to - end Ġneural Ġmodels Ġhave Ġaccomplished Ġpromising Ġperformance ĠB IB REF 2 Ġ. ĠAlthough Ġthere Ġis Ġa Ġsignificant Ġprogress Ġin Ġmachine Ġcomprehension Ġ( MC ) Ġon Ġtext Ġdocuments , ĠMC Ġon Ġspoken Ġcontent Ġis Ġa Ġmuch Ġless Ġinvestigated Ġfield . ĠIn Ġspoken Ġquestion Ġanswering Ġ( S Q A ), Ġafter Ġtransc ribing Ġspoken Ġcontent Ġinto Ġtext Ġby Ġautomatic Ġspeech Ġrecognition Ġ( AS R ), Ġtypical Ġapproaches Ġuse Ġinformation Ġretrieval Ġ( IR ) Ġtechniques ĠB IB REF 3 Ġto Ġfind Ġthe Ġproper Ġanswer Ġfrom Ġthe ĠAS R Ġhypotheses . ĠOne Ġattempt Ġtowards ĠQ A Ġof Ġspoken Ġcontent Ġis ĠTO E FL Ġlistening Ġcomprehension Ġby Ġmachine ĠB IB REF 4 Ġ. ĠTO E FL Ġis Ġan ĠEnglish Ġexamination Ġthat Ġtests Ġthe Ġknowledge Ġand Ġskills Ġof Ġacademic ĠEnglish Ġfor ĠEnglish Ġlearners Ġwhose Ġnative Ġlanguages Ġare Ġnot ĠEnglish . ĠAnother ĠSQ A Ġcorpus Ġis ĠSp oken - S Qu AD ĠB IB REF 5 Ġ, Ġwhich Ġis Ġautomatically Ġgenerated Ġfrom ĠS Qu AD Ġdataset Ġthrough ĠGoogle ĠText - to - Spe ech Ġ( T TS ) Ġsystem . ĠRecently ĠO DS Q A , Ġa ĠSQ A Ġcorpus Ġrecorded Ġby Ġreal Ġspeakers , Ġis Ġreleased ĠB IB REF 6 Ġ. </s> To Ġmitigate Ġthe Ġimpact Ġof Ġspeech Ġrecognition Ġerrors , Ġusing Ġsub - word Ġunits Ġis Ġa Ġpopular Ġapproach Ġfor Ġspeech - related Ġdownstream Ġtasks . ĠIt Ġhas Ġbeen Ġapplied Ġto Ġspoken Ġdocument Ġretrieval ĠB IB REF 7 Ġand Ġspoken Ġterm Ġdetection ĠB IB REF 8 ĠThe Ġprior Ġwork Ġshowed Ġthat , Ġusing Ġphon ect ic Ġsub - word Ġunits Ġbrought Ġimprovements Ġfor Ġboth ĠSp oken - S Qu AD Ġand ĠO DS Q A ĠB IB REF 5 Ġ. </s> Instead Ġof Ġconsidering Ġsub - word Ġfeatures , Ġthis Ġpaper Ġproposes Ġa Ġnovel Ġapproach Ġto Ġmitigate Ġthe Ġimpact Ġof ĠAS R Ġerrors . ĠWe Ġconsider Ġreference Ġtranscript ions Ġand ĠAS R Ġhypotheses Ġas Ġtwo Ġdomains , Ġand Ġadapt Ġthe Ġsource Ġdomain Ġdata Ġ( reference Ġtranscript ions ) Ġto Ġthe Ġtarget Ġdomain Ġdata Ġ( AS R Ġhypotheses ) Ġby Ġprojecting Ġthese Ġtwo Ġdomains Ġin Ġthe Ġshared Ġcommon Ġspace . ĠTherefore , Ġit Ġcan Ġeffectively Ġbenefit Ġthe ĠSQ A Ġmodel Ġby Ġimproving Ġthe Ġrobust ness Ġto ĠAS R Ġerrors Ġin Ġthe ĠSQ A Ġmodel . </s> Domain Ġadaptation Ġhas Ġbeen Ġsuccessfully Ġapplied Ġon Ġcomputer Ġvision ĠB IB REF 9 Ġand Ġspeech Ġrecognition ĠB IB REF 10 Ġ. ĠIt Ġis Ġalso Ġwidely Ġstudied Ġon ĠN LP Ġtasks Ġsuch Ġas Ġsequence Ġtagging Ġand Ġparsing ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠRecently , Ġadvers arial Ġdomain Ġadaptation Ġhas Ġalready Ġbeen Ġexplored Ġon Ġspoken Ġlanguage Ġunderstanding Ġ( SL U ). ĠLiu Ġand ĠLane Ġlearned Ġdomain - general Ġfeatures Ġto Ġbenefit Ġfrom Ġmultiple Ġdialogue Ġdatasets ĠB IB REF 14 Ġ; ĠZhu Ġet Ġal . Ġlearned Ġto Ġtransfer Ġthe Ġmodel Ġfrom Ġthe Ġtranscripts Ġside Ġto Ġthe ĠAS R Ġhypotheses Ġside ĠB IB REF 15 Ġ; ĠLan Ġet Ġal . Ġconstructed Ġa Ġshared Ġspace Ġfor Ġslot Ġtagging Ġand Ġlanguage Ġmodel ĠB IB REF 16 Ġ. ĠThis Ġpaper Ġextends Ġthe Ġcapability Ġof Ġadvers arial Ġdomain Ġadaptation Ġfor ĠSQ A , Ġwhich Ġhas Ġnot Ġbeen Ġexplored Ġyet . </s> Sp oken ĠQuestion ĠAn sw ering </s> In ĠSQ A , Ġeach Ġsample Ġis Ġa Ġtriple , ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġquestion Ġin Ġeither Ġspoken Ġor Ġtext Ġform , ĠIN LINE FORM 2 Ġis Ġa Ġmulti - sent ence Ġspoken - form Ġdocument , Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġanswer Ġin Ġtext Ġfrom . ĠThe Ġtask Ġof Ġthis Ġwork Ġis Ġextract ive ĠSQ A ; Ġthat Ġmeans ĠIN LINE FORM 4 Ġis Ġa Ġword Ġspan Ġfrom Ġthe Ġreference Ġtranscription Ġof ĠIN LINE FORM 5 Ġ. ĠAn Ġoverview Ġframework Ġof ĠSQ A Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġframe Ġthe Ġsource Ġdomain Ġas Ġreference Ġtranscript ions Ġand Ġthe Ġtarget Ġdomain Ġas ĠAS R Ġhypotheses . ĠHence , Ġwe Ġcan Ġcollect Ġsource Ġdomain Ġdata Ġmore Ġeasily , Ġand Ġadapt Ġthe Ġmodel Ġto Ġthe Ġtarget Ġdomain . </s> In Ġthis Ġtask , Ġwhen Ġthe Ġmachine Ġis Ġgiven Ġa Ġspoken Ġdocument , Ġit Ġneeds Ġto Ġfind Ġthe Ġanswer Ġof Ġa Ġquestion Ġfrom Ġthe Ġspoken Ġdocument . ĠSQ A Ġcan Ġbe Ġsolved Ġby Ġthe Ġconc aten ation Ġof Ġan ĠAS R Ġmodule Ġand Ġa Ġquestion Ġanswering Ġmodule . ĠGiven Ġthe ĠAS R Ġhypotheses Ġof Ġa Ġspoken Ġdocument Ġand Ġa Ġquestion , Ġthe Ġquestion Ġanswering Ġmodule Ġcan Ġoutput Ġa Ġtext Ġanswer . </s> The Ġmost Ġintuitive Ġway Ġto Ġevaluate Ġthe Ġtext Ġanswer Ġis Ġto Ġdirectly Ġcompute Ġthe ĠEx act ĠMatch Ġ( EM ) Ġand ĠMacro - aver aged ĠF 1 Ġscores Ġ( F 1 ) Ġbetween Ġthe Ġpredicted Ġtext Ġanswer Ġand Ġthe Ġground - truth Ġtext Ġanswer . ĠWe Ġused Ġthe Ġstandard Ġevaluation Ġscript Ġfrom ĠS Qu AD ĠB IB REF 1 Ġto Ġevaluate Ġthe Ġperformance . </s> Question ĠAn sw ering ĠModel </s> The Ġused Ġarchitecture Ġof Ġthe ĠQ A Ġmodel Ġis Ġbriefly Ġsummarized Ġbelow . ĠHere Ġwe Ġchoose ĠQ AN et ĠB IB REF 2 Ġas Ġthe Ġbase Ġmodel Ġdue Ġto Ġthe Ġfollowing Ġreasons : Ġ1 ) Ġit Ġachieves Ġthe Ġsecond Ġbest Ġperformance Ġon ĠS Qu AD , Ġand Ġ2 ) Ġsince Ġthere Ġare Ġcompletely Ġno Ġrecurrent Ġnetworks Ġin ĠQ AN et , Ġits Ġtraining Ġspeed Ġis Ġ5 x Ġfaster Ġthan ĠBi DA F ĠB IB REF 17 Ġwhen Ġreaching Ġthe Ġsame Ġperformance Ġon ĠS Qu AD . </s> The Ġnetwork Ġarchitecture Ġis Ġillustrated Ġin ĠFigure ĠFIG REF 2 Ġ. ĠThe Ġleft Ġblocks Ġand Ġthe Ġright Ġblocks Ġform Ġtwo ĠQ AN ets , Ġeach Ġof Ġwhich Ġtakes Ġa Ġdocument Ġand Ġa Ġquestion Ġas Ġthe Ġinput Ġand Ġoutputs Ġan Ġanswer . ĠIn ĠQ AN et , Ġfirst ly , Ġan Ġembed ding Ġenc oder Ġob tains Ġword Ġand Ġcharacter Ġembed d ings Ġfor Ġeach Ġword Ġin ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġand Ġthen Ġmodels Ġthe Ġtemporal Ġinteractions Ġbetween Ġwords Ġand Ġref ines Ġword Ġvectors Ġto Ġcontextual ized Ġword Ġrepresentations . ĠAll Ġenc oder Ġblocks Ġused Ġin ĠQ AN et Ġare Ġcomposed Ġexclusively Ġof Ġdepth - wise Ġsepar able Ġconv olutions Ġand Ġself - att ention . ĠThe Ġintuition Ġhere Ġis Ġthat Ġconv olution Ġcomponents Ġcan Ġmodel Ġlocal Ġinteractions Ġand Ġself - att ention Ġcomponents Ġfocus Ġon Ġmodeling Ġglobal Ġinteractions . ĠThe Ġcontext - query Ġattention Ġlayer Ġgenerates Ġthe Ġquestion - document Ġsimilarity Ġmatrix Ġand Ġcomp utes Ġthe Ġquestion - aware Ġvector Ġrepresentations Ġof Ġthe Ġcontext Ġwords . ĠAfter Ġthat , Ġa Ġmodel Ġenc oder Ġlayer Ġcontaining Ġseven Ġenc oder Ġblocks Ġcaptures Ġthe Ġinteractions Ġamong Ġthe Ġcontext Ġwords Ġconditioned Ġon Ġthe Ġquestion . ĠFinally , Ġthe Ġoutput Ġlayer Ġpredicts Ġa Ġstart Ġposition Ġand Ġan Ġend Ġposition Ġin Ġthe Ġdocument Ġto Ġextract Ġthe Ġanswer Ġspan Ġfrom Ġthe Ġdocument . </s> Domain ĠAdapt ation ĠApproach </s> The Ġmain Ġfocus Ġof Ġthis Ġpaper Ġis Ġto Ġapply Ġdomain Ġadaptation Ġfor ĠSQ A . ĠIn Ġthis Ġapproach , Ġwe Ġhave Ġtwo ĠSQ A Ġmodels Ġ( Q AN ets ), Ġone Ġtrained Ġfrom Ġtarget Ġdomain Ġdata Ġ( AS R Ġhypotheses ) Ġand Ġanother Ġtrained Ġfrom Ġsource Ġdomain Ġdata Ġ( reference Ġtranscript ions ). ĠBecause Ġthe Ġtwo Ġdomains Ġshare Ġcommon Ġinformation , Ġsome Ġlayers Ġin Ġthese Ġtwo Ġmodels Ġcan Ġbe Ġtied Ġin Ġorder Ġto Ġmodel Ġthe Ġshared Ġfeatures . ĠHence , Ġwe Ġcan Ġchoose Ġwhether Ġeach Ġlayer Ġin Ġthe ĠQ A Ġmodel Ġshould Ġbe Ġshared . ĠT ying Ġthe Ġweights Ġbetween Ġthe Ġsource Ġlayer Ġand Ġthe Ġtarget Ġlayer Ġin Ġorder Ġto Ġlearn Ġa Ġsymm etric Ġmapping Ġis Ġto Ġproject Ġboth Ġsource Ġand Ġtarget Ġdomain Ġdata Ġto Ġa Ġshared Ġcommon Ġspace . ĠDifferent Ġcombinations Ġwill Ġbe Ġinvestigated Ġin Ġour Ġexperiments . </s> More Ġspecifically , Ġwe Ġincorporate Ġa Ġdomain Ġdiscrim inator Ġinto Ġthe ĠSQ A Ġmodel Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ, Ġwhich Ġcan Ġenforce Ġthe Ġembed ding Ġenc oder Ġto Ġproject Ġthe Ġsentences Ġfrom Ġboth Ġsource Ġand Ġtarget Ġdomains Ġinto Ġa Ġshared Ġcommon Ġspace Ġand Ġconsequ entially Ġto Ġbe ĠAS R - error Ġrobust . ĠAlthough Ġthe Ġembed ding Ġenc oder Ġfor Ġboth Ġdomains Ġmay Ġimplicitly Ġlearn Ġsome Ġcommon Ġlatent Ġrepresentations , Ġadvers arial Ġlearning Ġcan Ġprovide Ġa Ġmore Ġdirect Ġtraining Ġsignal Ġfor Ġalign ing Ġthe Ġoutput Ġdistribution Ġof Ġthe Ġembed ding Ġenc oder Ġfrom Ġboth Ġdomains . ĠThe Ġembed ding Ġenc oder Ġtakes Ġin Ġa Ġsequence Ġof Ġword Ġvectors Ġand Ġgenerates Ġa Ġsequence Ġof Ġhidden Ġvectors Ġwith Ġthe Ġsame Ġlength . ĠWe Ġuse ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ( ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ) Ġto Ġrepresent Ġthe Ġhidden Ġvector Ġsequence Ġgiven Ġthe Ġquestion ĠIN LINE FORM 4 Ġand Ġthe Ġdocument ĠIN LINE FORM 5 Ġin Ġthe Ġtarget Ġ( source ) Ġdomain Ġrespectively . </s> The Ġdomain Ġdiscrim inator ĠIN LINE FORM 0 Ġfocuses Ġon Ġidentifying Ġthe Ġdomain Ġof Ġthe Ġvector Ġsequence Ġis Ġfrom Ġgiven ĠIN LINE FORM 1 Ġor ĠIN LINE FORM 2 Ġ, Ġwhere Ġthe Ġobjective Ġis Ġto Ġminimize ĠIN LINE FORM 3 Ġ. ĠDIS PLAY FORM 0 </s> Given Ġa Ġtraining Ġexample Ġfrom Ġthe Ġtarget Ġdomain Ġ( ĠIN LINE FORM 0 Ġ), ĠIN LINE FORM 1 Ġlearns Ġto Ġassign Ġa Ġlower Ġscore Ġto ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġin Ġthat Ġexample , Ġthat Ġis , Ġto Ġminimize ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ. ĠOn Ġthe Ġother Ġhand , Ġgiven Ġa Ġtraining Ġexample Ġfrom Ġthe Ġsource Ġdomain Ġ( ĠIN LINE FORM 6 Ġ), ĠIN LINE FORM 7 Ġlearns Ġto Ġassign Ġa Ġlarger Ġvalue Ġto ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ. </s> Furthermore , Ġwe Ġupdate Ġthe Ġparameters Ġof Ġthe Ġembed ding Ġenc od ers Ġto Ġmaximize Ġthe Ġdomain Ġclassification Ġloss ĠIN LINE FORM 0 Ġ, Ġwhich Ġworks Ġadvers ar ially Ġtowards Ġthe Ġdomain Ġdiscrim inator . ĠWe Ġthus Ġexpect Ġthe Ġmodel Ġto Ġlearn Ġfeatures Ġand Ġstructures Ġthat Ġcan Ġgeneral ize Ġacross Ġdomains Ġwhen Ġthe Ġoutputs Ġof ĠIN LINE FORM 1 Ġare Ġindistinguishable Ġfrom Ġthe Ġoutputs Ġof ĠIN LINE FORM 2 Ġ. ĠThe Ġloss Ġfunction Ġfor Ġembed ding Ġenc oder , ĠIN LINE FORM 3 Ġ, Ġis Ġformulated Ġas ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġhyper param eter . ĠThe Ġtwo Ġembed ding Ġenc od ers Ġin Ġthe ĠQ A Ġmodel Ġare Ġlearned Ġto Ġmaximize ĠIN LINE FORM 1 Ġwhile Ġminimizing Ġthe Ġloss Ġfor ĠQ A , ĠIN LINE FORM 2 Ġ. ĠBecause Ġthe Ġparameters Ġof Ġother Ġlayers Ġin ĠQ A Ġmodel Ġare Ġindependent Ġto Ġthe Ġloss Ġof Ġthe Ġdomain Ġdiscrim inator , Ġthe Ġloss Ġfunction Ġof Ġother Ġlayers , ĠIN LINE FORM 3 Ġ, Ġis Ġequivalent Ġto ĠIN LINE FORM 4 Ġ, Ġthat Ġis , ĠIN LINE FORM 5 Ġ. </s> Although Ġthe Ġdiscrim inator Ġis Ġapplied Ġto Ġthe Ġoutput Ġof Ġembed ding Ġenc oder Ġin ĠFigure ĠFIG REF 2 Ġ, Ġit Ġcan Ġbe Ġalso Ġapplied Ġto Ġother Ġlayers . ĠConsidering Ġthat Ġalmost Ġall ĠQ A Ġmodel Ġcontains Ġsuch Ġembed ding Ġenc od ers , Ġthe Ġproposed Ġapproach Ġis Ġexpected Ġto Ġgeneral ize Ġto Ġother ĠQ A Ġmodels Ġin Ġaddition Ġto ĠQ AN et . </s> Corp us </s> Sp oken - S Qu AD Ġis Ġchosen Ġas Ġthe Ġtarget Ġdomain Ġdata Ġfor Ġtraining Ġand Ġtesting . ĠSp oken - S Qu AD ĠB IB REF 5 Ġis Ġan Ġautomatically Ġgenerated Ġcorpus Ġin Ġwhich Ġthe Ġdocument Ġis Ġin Ġspoken Ġform Ġand Ġthe Ġquestion Ġis Ġin Ġtext Ġform . ĠThe Ġreference Ġtranscript ions Ġare Ġfrom ĠS Qu AD ĠB IB REF 1 Ġ. ĠThere Ġare Ġ37 , 111 Ġand Ġ5 , 351 Ġquestion Ġanswer Ġpairs Ġin Ġthe Ġtraining Ġand Ġtesting Ġsets Ġrespectively , Ġand Ġthe Ġword Ġerror Ġrate Ġ( WER ) Ġof Ġboth Ġsets Ġis Ġaround Ġ22 . 7 %. </s> The Ġoriginal ĠS Qu AD , ĠText - S Qu AD , Ġis Ġchosen Ġas Ġthe Ġsource Ġdomain Ġdata , Ġwhere Ġonly Ġquestion Ġanswering Ġpairs Ġappearing Ġin ĠSp oken - S Qu AD Ġare Ġutilized . ĠIn Ġour Ġtask Ġsetting , Ġduring Ġtraining Ġwe Ġtrain Ġthe Ġproposed ĠQ A Ġmodel Ġon Ġboth ĠText - S Qu AD Ġand ĠSp oken - S Qu AD Ġtraining Ġsets . ĠWhile Ġin Ġthe Ġtesting Ġstage , Ġwe Ġevaluate Ġthe Ġperformance Ġon ĠSp oken - S Qu AD Ġtesting Ġset . </s> Exper iment ĠSetup </s> We Ġutilize Ġfast text ĠB IB REF 18 Ġto Ġgenerate Ġthe Ġembed d ings Ġof Ġall Ġwords Ġfrom Ġboth ĠText - S Qu AD Ġand ĠSp oken - S Qu AD . ĠWe Ġadopt Ġthe Ġphon eme Ġsequence Ġembed d ings Ġto Ġreplace Ġthe Ġoriginal Ġcharacter Ġsequence Ġembed d ings Ġusing Ġthe Ġmethod Ġproposed Ġby ĠLi Ġet Ġal . ĠB IB REF 5 Ġ. ĠThe Ġsource Ġdomain Ġmodel Ġand Ġthe Ġtarget Ġdomain Ġmodel Ġshare Ġthe Ġsame Ġset Ġof Ġword Ġembed ding Ġmatrix Ġto Ġimprove Ġthe Ġalignment Ġbetween Ġthese Ġtwo Ġdomains . </s> W - GAN Ġis Ġadopted Ġfor Ġour Ġdomain Ġdiscrim inator ĠB IB REF 19 Ġ, Ġwhich Ġstacks Ġ5 Ġresidual Ġblocks Ġof Ġ1 D Ġconv olution al Ġlayers Ġwith Ġ96 Ġfilters Ġand Ġfilter Ġsize Ġ5 Ġfollowed Ġby Ġone Ġlinear Ġlayer Ġto Ġconvert Ġeach Ġinput Ġvector Ġsequence Ġinto Ġone Ġscal ar Ġvalue . </s> All Ġmodels Ġused Ġin Ġthe Ġexperiments Ġare Ġtrained Ġwith Ġbatch Ġsize Ġ20 , Ġusing Ġadam Ġwith Ġlearning Ġrate ĠIN LINE FORM 0 Ġand Ġthe Ġearly Ġstop Ġstrategy . ĠThe Ġdimension Ġof Ġthe Ġhidden Ġstate Ġis Ġset Ġto Ġ96 Ġfor Ġall Ġlayers , Ġand Ġthe Ġnumber Ġof Ġself - att ention Ġheads Ġis Ġset Ġto Ġ2 . ĠThe Ġsetup Ġis Ġslightly Ġdifferent Ġbut Ġbetter Ġthan Ġthe Ġsetting Ġsuggested Ġby Ġthe Ġoriginal ĠQ An et . </s> Results </s> First , Ġwe Ġhighlight Ġthe Ġdomain Ġmismatch Ġphenomenon Ġin Ġour Ġexperiments Ġshown Ġin ĠTable ĠT AB REF 9 Ġ. ĠRow Ġ( a ) Ġis Ġwhen ĠQ AN et Ġis Ġtrained Ġon ĠText - S Qu AD , Ġrow Ġ( b ) Ġis Ġwhen ĠQ AN et Ġis Ġtrained Ġon ĠSp oken - S Qu AD , Ġand Ġrow Ġ( c ) Ġis Ġwhen ĠQ AN et Ġis Ġtrained Ġon ĠText - S Qu AD Ġand Ġthen Ġfin et uned Ġon ĠSp oken - S Qu AD . ĠThe Ġcolumns Ġshow Ġthe Ġevaluation Ġon Ġthe Ġtesting Ġsets Ġof ĠText - S Qu AD Ġand ĠSp oken - S Qu AD . </s> It Ġis Ġclear Ġthat Ġthe Ġperformance Ġdrops Ġa Ġlot Ġwhen Ġthe Ġtraining Ġand Ġtesting Ġdata Ġmismatch , Ġindicating Ġthat Ġmodel Ġtraining Ġon ĠAS R Ġhypotheses Ġcan Ġnot Ġgeneral ize Ġwell Ġon Ġreference Ġtranscript ions . ĠThe Ġperformance Ġgap Ġis Ġnearly Ġ20 % ĠF 1 Ġscore Ġ( 72 % Ġto Ġ55 %). ĠThe Ġrow Ġ( c ) Ġshows Ġthe Ġimproved Ġperformance Ġwhen Ġtesting Ġon ĠS - S Qu AD Ġdue Ġto Ġthe Ġtransfer Ġlearning Ġvia Ġfine - tun ing . </s> To Ġbetter Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġmodel , Ġwe Ġcompare Ġwith Ġbas elines Ġand Ġshow Ġthe Ġresults Ġin ĠTable ĠT AB REF 12 Ġ. ĠThe Ġbas elines Ġare : Ġ( a ) Ġtrained Ġon ĠS - S Qu AD , Ġ( b ) Ġtrained Ġon ĠT - S Qu AD Ġand Ġthen Ġfine - tun ed Ġon ĠS - S Qu AD , Ġand Ġ( c ) Ġprevious Ġbest Ġmodel Ġtrained Ġon ĠS - S Qu AD ĠB IB REF 5 Ġby Ġusing ĠDr . Q A ĠB IB REF 20 Ġ. ĠWe Ġalso Ġcompare Ġto Ġthe Ġapproach Ġproposed Ġby ĠLan Ġet Ġal . ĠB IB REF 16 Ġin Ġthe Ġrow Ġ( d ). ĠThis Ġapproach Ġis Ġoriginally Ġproposed Ġfor Ġspoken Ġlanguage Ġunderstanding , Ġand Ġwe Ġadopt Ġthe Ġsame Ġapproach Ġon Ġthe Ġsetting Ġhere . ĠThe Ġapproach Ġmodels Ġdomain - specific Ġfeatures Ġfrom Ġthe Ġsource Ġand Ġtarget Ġdomains Ġseparately Ġby Ġtwo Ġdifferent Ġembed ding Ġenc od ers Ġwith Ġa Ġshared Ġembed ding Ġenc oder Ġfor Ġmodeling Ġdomain - general Ġfeatures . ĠThe Ġdomain - general Ġparameters Ġare Ġadvers ar ially Ġtrained Ġby Ġdomain Ġdiscrim inator . </s> Row Ġ( e ) Ġis Ġthe Ġmodel Ġthat Ġthe Ġweights Ġof Ġall Ġlayers Ġare Ġtied Ġbetween Ġthe Ġsource Ġdomain Ġand Ġthe Ġtarget Ġdomain . ĠRow Ġ( f ) Ġuses Ġthe Ġsame Ġarchitecture Ġas Ġrow Ġ( e ) Ġwith Ġan Ġadditional Ġdomain Ġdiscrim inator Ġapplied Ġto Ġthe Ġembed ding Ġenc oder . ĠIt Ġcan Ġbe Ġfound Ġthat Ġrow Ġ( f ) Ġoutper forms Ġrow Ġ( e ), Ġindicating Ġthat Ġthe Ġproposed Ġdomain Ġadvers arial Ġlearning Ġis Ġhelpful . ĠTherefore , Ġour Ġfollowing Ġexperiments Ġcontain Ġdomain Ġadvers arial Ġlearning . ĠThe Ġproposed Ġapproach Ġ( row Ġ( f )) Ġoutper forms Ġprevious Ġbest Ġmodel Ġ( row Ġ( c )) Ġby Ġ2 % ĠEM Ġscore Ġand Ġover Ġ1 . 5 % ĠF 1 Ġscore . ĠWe Ġalso Ġshow Ġthe Ġresults Ġof Ġapplying Ġthe Ġdomain Ġdiscrim inator Ġto Ġthe Ġtop Ġof Ġcontext Ġquery Ġattention Ġlayer Ġin Ġrow Ġ( g ), Ġwhich Ġob tains Ġpoor Ġperformance . ĠTo Ġsum Ġit Ġup , Ġincorporating Ġadvers arial Ġlearning Ġby Ġapplying Ġthe Ġdomain Ġdiscrim inator Ġon Ġtop Ġof Ġthe Ġembed ding Ġenc oder Ġlayer Ġis Ġeffective . </s> Layer Ġweight Ġtying Ġor Ġunt ying Ġwithin Ġthe Ġmodel Ġindicates Ġdifferent Ġlevels Ġof Ġsymm etric Ġmapping Ġbetween Ġthe Ġsource Ġand Ġtarget Ġdomains . ĠDifferent Ġcombinations Ġare Ġinvestigated Ġand Ġshown Ġin ĠTable ĠT AB REF 14 Ġ. ĠThe Ġrow Ġ( a ) Ġin Ġwhich Ġall Ġlayers Ġare Ġtied Ġis Ġthe Ġrow Ġ( e ) Ġof ĠTable ĠT AB REF 12 Ġ. ĠThe Ġresults Ġshow Ġthat Ġunt ying Ġcontext - query Ġattention Ġlayer ĠL 2 Ġ( rows Ġ( c , Ġf , Ġg )) Ġor Ġmodel Ġenc oder Ġlayer ĠL 3 Ġ( rows Ġ( d , Ġf , Ġh )) Ġlead Ġto Ġdegener ated Ġsolutions Ġin Ġcomparison Ġto Ġrow Ġ( a ) Ġwhere Ġall Ġlayers Ġare Ġtied . ĠUnt ying Ġboth Ġof Ġthem Ġsimultaneously Ġleads Ġto Ġthe Ġworst Ġperformance Ġwhich Ġis Ġeven Ġworse Ġthan Ġthe Ġfin et uning Ġ( row Ġ( g ) Ġv . s . Ġ( c ) Ġfrom ĠTable ĠT AB REF 12 Ġ). ĠThese Ġresults Ġimply Ġthat Ġsharing Ġthe Ġcontext - query Ġattention Ġlayer Ġand Ġthe Ġmodel Ġenc oder Ġlayer Ġare Ġimportant Ġfor Ġdomain Ġadaptation Ġon ĠSQ A . ĠWe Ġconjecture Ġthat Ġthese Ġtwo Ġlayers Ġbenefit Ġfrom Ġtraining Ġon Ġsource Ġdomain Ġdata Ġwhere Ġthere Ġare Ġno ĠAS R Ġerrors , Ġso Ġthe ĠQ A Ġmodel Ġlearns Ġto Ġconduct Ġattention Ġor Ġfurther Ġreason Ġwell Ġon Ġtarget Ġdomain Ġdata Ġwith ĠAS R Ġerrors . </s> Overall , Ġit Ġis Ġnot Ġbeneficial Ġto Ġunt ie Ġany Ġlayer , Ġbecause Ġno Ġinformation Ġcan Ġbe Ġshared Ġacross Ġdifferent Ġdomains . ĠUnt ying Ġthe Ġembed ding Ġenc oder ĠL 1 Ġand Ġthe Ġoutput Ġlayer ĠL 4 Ġleads Ġto Ġthe Ġleast Ġdegradation Ġin Ġcomparison Ġto Ġrow Ġ( a ). </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġincorporate Ġa Ġdomain Ġdiscrim inator Ġto Ġalign Ġthe Ġmism atched Ġdomains Ġbetween ĠAS R Ġhypotheses Ġand Ġreference Ġtranscript ions . ĠThe Ġadvers arial Ġlearning Ġallows Ġthe Ġend - to - end ĠQ A Ġmodel Ġto Ġlearn Ġdomain - inv ari ant Ġfeatures Ġand Ġimprove Ġthe Ġrobust ness Ġto ĠAS R Ġerrors . ĠThe Ġexperiments Ġdemonstrate Ġthat Ġthe Ġproposed Ġmodel Ġsuccessfully Ġachieves Ġsuperior Ġperformance Ġand Ġoutper forms Ġthe Ġprevious Ġbest Ġmodel Ġby Ġ2 % ĠEM Ġscore Ġand Ġover Ġ1 . 5 % ĠF 1 Ġscore .