Vanilla ST baseline, encoder pre-training, in which the ST encoder is initialized from an ASR model, decoder pre-training, in which the ST decoder is initialized from an MT model, encoder-decoder pre-training, where both the encoder and decoder are pre-trained, many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models, Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation
EXTRACTIVE
Vanilla ST baseline, Pre-training baselines, Multi-task baselines, Many-to-many+pre-training, Triangle+pre-train
EXTRACTIVE
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus.

Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data.

Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data.

Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. 
EXTRACTIVE
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus. Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data. Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data. Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation. Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model. Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder. For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model. The Triangle model is fine-tuned under their multi-task manner.
We compare our method with following baselines. Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus. Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data. Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data. Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation. Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model. Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder. For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model. The Triangle model is fine-tuned under their multi-task manner.
We compare our method with following baselines. Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus. Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data. Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data. Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation. Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model. Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder. For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model. The Triangle model is fine-tuned under their multi-task manner.
1909.07575
Introduction </s> Spe ech - to - Text Ġtranslation Ġ( ST ) Ġis Ġessential Ġfor Ġa Ġwide Ġrange Ġof Ġscenarios : Ġfor Ġexample Ġin Ġemergency Ġcalls , Ġwhere Ġagents Ġhave Ġto Ġrespond Ġemer gent Ġrequests Ġin Ġa Ġforeign Ġlanguage ĠB IB REF 0 ; Ġor Ġin Ġonline Ġcourses , Ġwhere Ġaudiences Ġand Ġspeakers Ġuse Ġdifferent Ġlanguages ĠB IB REF 1 . ĠTo Ġtackle Ġthis Ġproblem , Ġexisting Ġapproaches Ġcan Ġbe Ġcategorized Ġinto Ġcasc aded Ġmethod ĠB IB REF 2 , ĠB IB REF 3 , Ġwhere Ġa Ġmachine Ġtranslation Ġ( MT ) Ġmodel Ġtranslates Ġoutputs Ġof Ġan Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystem Ġinto Ġtarget Ġlanguage , Ġand Ġend - to - end Ġmethod ĠB IB REF 4 , ĠB IB REF 5 , Ġwhere Ġa Ġsingle Ġmodel Ġlearns Ġacoustic Ġframes Ġto Ġtarget Ġword Ġsequence Ġm appings Ġin Ġone Ġstep Ġtowards Ġthe Ġfinal Ġobjective Ġof Ġinterest . ĠAlthough Ġthe Ġcasc aded Ġmodel Ġremains Ġthe Ġdominant Ġapproach Ġdue Ġto Ġits Ġbetter Ġperformance , Ġthe Ġend - to - end Ġmethod Ġbecomes Ġmore Ġand Ġmore Ġpopular Ġbecause Ġit Ġhas Ġlower Ġlatency Ġby Ġavoiding Ġinf erences Ġwith Ġtwo Ġmodels Ġand Ġrect ifies Ġthe Ġerror Ġpropagation Ġin Ġtheory . </s> Since Ġit Ġis Ġhard Ġto Ġobtain Ġa Ġlarge - scale ĠST Ġdataset , Ġmulti - task Ġlearning ĠB IB REF 5 , ĠB IB REF 6 Ġand Ġpre - training Ġtechniques ĠB IB REF 7 Ġhave Ġbeen Ġapplied Ġto Ġend - to - end ĠST Ġmodel Ġto Ġleverage Ġlarge - scale Ġdatasets Ġof ĠAS R Ġand ĠMT . ĠA Ġcommon Ġpractice Ġis Ġto Ġpre - train Ġtwo Ġenc oder - dec oder Ġmodels Ġfor ĠAS R Ġand ĠMT Ġrespectively , Ġand Ġthen Ġinitialize Ġthe ĠST Ġmodel Ġwith Ġthe Ġenc oder Ġof Ġthe ĠAS R Ġmodel Ġand Ġthe Ġdec oder Ġof Ġthe ĠMT Ġmodel . ĠSub sequently , Ġthe ĠST Ġmodel Ġis Ġoptimized Ġwith Ġthe Ġmulti - task Ġlearning Ġby Ġweighing Ġthe Ġlosses Ġof ĠAS R , ĠMT , Ġand ĠST . ĠThis Ġapproach , Ġhowever , Ġcauses Ġa Ġhuge Ġgap Ġbetween Ġpre - training Ġand Ġfine - tun ing , Ġwhich Ġare Ġsummarized Ġinto Ġthree Ġfolds : </s> Sub net ĠWaste : ĠThe ĠST Ġsystem Ġjust Ġre uses Ġthe ĠAS R Ġenc oder Ġand Ġthe ĠMT Ġdec oder , Ġwhile Ġdisc ards Ġother Ġpre - trained Ġsub nets , Ġsuch Ġas Ġthe ĠMT Ġenc oder . ĠConsequently , Ġvaluable Ġsemantic Ġinformation Ġcaptured Ġby Ġthe ĠMT Ġenc oder Ġcannot Ġbe Ġinherited Ġby Ġthe Ġfinal ĠST Ġsystem . </s> Role ĠM ism atch : ĠThe Ġspeech Ġenc oder Ġplays Ġdifferent Ġroles Ġin Ġpre - training Ġand Ġfine - tun ing . ĠThe Ġenc oder Ġis Ġa Ġpure Ġacoustic Ġmodel Ġin Ġpre - training , Ġwhile Ġit Ġhas Ġto Ġextract Ġsemantic Ġand Ġlinguistic Ġfeatures Ġadditionally Ġin Ġfine - tun ing , Ġwhich Ġsignificantly Ġincreases Ġthe Ġlearning Ġdifficulty . </s> Non - pre - trained ĠAttention ĠModule : ĠPrevious Ġwork ĠB IB REF 6 Ġtrains Ġattention Ġmodules Ġfor ĠAS R , ĠMT Ġand ĠST Ġrespectively , Ġhence , Ġthe Ġattention Ġmodule Ġof ĠST Ġdoes Ġnot Ġbenefit Ġfrom Ġthe Ġpre - training . </s> To Ġaddress Ġthese Ġissues , Ġwe Ġpropose Ġa ĠT andem ĠConnection ist ĠEnc oding ĠNetwork Ġ( TC EN ), Ġwhich Ġis Ġable Ġto Ġreuse Ġall Ġsub nets Ġin Ġpre - training , Ġkeep Ġthe Ġroles Ġof Ġsub nets Ġconsistent , Ġand Ġpre - train Ġthe Ġattention Ġmodule . ĠCon crete ly , Ġthe ĠTC EN Ġconsists Ġof Ġthree Ġcomponents , Ġa Ġspeech Ġenc oder , Ġa Ġtext Ġenc oder , Ġand Ġa Ġtarget Ġtext Ġdec oder . ĠDifferent Ġfrom Ġthe Ġprevious Ġwork Ġthat Ġpre - tr ains Ġan Ġenc oder - dec oder Ġbased ĠAS R Ġmodel , Ġwe Ġonly Ġpre - train Ġan ĠAS R Ġenc oder Ġby Ġoptimizing Ġthe ĠConnection ist ĠTem poral ĠClassification Ġ( CT C ) ĠB IB REF 8 Ġobjective Ġfunction . ĠIn Ġthis Ġway , Ġthe Ġadditional Ġdec oder Ġof ĠAS R Ġis Ġnot Ġrequired Ġwhile Ġkeeping Ġthe Ġability Ġto Ġread Ġacoustic Ġfeatures Ġinto Ġthe Ġsource Ġlanguage Ġspace Ġby Ġthe Ġspeech Ġenc oder . ĠBesides , Ġthe Ġtext Ġenc oder Ġand Ġdec oder Ġcan Ġbe Ġpre - trained Ġon Ġa Ġlarge ĠMT Ġdataset . ĠAfter Ġthat , Ġwe Ġemploy Ġcommon Ġused Ġmulti - task Ġlearning Ġmethod Ġto Ġjointly Ġlearn ĠAS R , ĠMT Ġand ĠST Ġtasks . </s> Compared Ġto Ġprior Ġworks , Ġthe Ġenc oder Ġof ĠTC EN Ġis Ġa Ġconc aten ation Ġof Ġan ĠAS R Ġenc oder Ġand Ġan ĠMT Ġenc oder Ġand Ġour Ġmodel Ġdoes Ġnot Ġhave Ġan ĠAS R Ġdec oder , Ġso Ġthe Ġsub net Ġwaste Ġissue Ġis Ġsolved . ĠFurthermore , Ġthe Ġtwo Ġenc od ers Ġwork Ġat Ġtandem , Ġdis ent angling Ġacoustic Ġfeature Ġextraction Ġand Ġlinguistic Ġfeature Ġextraction , Ġensuring Ġthe Ġrole Ġconsistency Ġbetween Ġpre - training Ġand Ġfine - tun ing . ĠMoreover , Ġwe Ġreuse Ġthe Ġpre - trained ĠMT Ġattention Ġmodule Ġin ĠST , Ġso Ġwe Ġcan Ġleverage Ġthe Ġalignment Ġinformation Ġlearned Ġin Ġpre - training . </s> Since Ġthe Ġtext Ġenc oder Ġconsumes Ġword Ġembed d ings Ġof Ġplausible Ġtexts Ġin ĠMT Ġtask Ġbut Ġuses Ġspeech Ġenc oder Ġoutputs Ġin ĠST Ġtask , Ġanother Ġquestion Ġis Ġhow Ġone Ġguarantees Ġthe Ġspeech Ġenc oder Ġoutputs Ġare Ġconsistent Ġwith Ġthe Ġword Ġembed d ings . ĠWe Ġfurther Ġmodify Ġour Ġmodel Ġto Ġachieve Ġsemantic Ġconsistency Ġand Ġlength Ġconsistency . ĠSpecifically , Ġ( 1 ) Ġthe Ġprojection Ġmatrix Ġat Ġthe ĠC TC Ġclassification Ġlayer Ġfor ĠAS R Ġis Ġshared Ġwith Ġthe Ġword Ġembed ding Ġmatrix , Ġensuring Ġthat Ġthey Ġare Ġmapped Ġto Ġthe Ġsame Ġlatent Ġspace , Ġand Ġ( 2 ) Ġthe Ġlength Ġof Ġthe Ġspeech Ġenc oder Ġoutput Ġis Ġproportional Ġto Ġthe Ġlength Ġof Ġthe Ġinput Ġframe , Ġso Ġit Ġis Ġmuch Ġlonger Ġthan Ġa Ġnatural Ġsentence . ĠTo Ġbridge Ġthe Ġlength Ġgap , Ġsource Ġsentences Ġin ĠMT Ġare Ġleng then ed Ġby Ġadding Ġword Ġrepet itions Ġand Ġblank Ġtokens Ġto Ġmimic Ġthe ĠC TC Ġoutput Ġsequences . </s> We Ġconduct Ġcomprehensive Ġexperiments Ġon Ġthe ĠI W SL T 18 Ġspeech Ġtranslation Ġbenchmark ĠB IB REF 1 , Ġdemonstrating Ġthe Ġeffectiveness Ġof Ġeach Ġcomponent . ĠOur Ġmodel Ġis Ġsignificantly Ġbetter Ġthan Ġprevious Ġmethods Ġby Ġ3 . 6 Ġand Ġ2 . 2 ĠB LE U Ġscores Ġfor Ġthe Ġsub word - level Ġdecoding Ġand Ġcharacter - level Ġdecoding Ġstrategies , Ġrespectively . </s> Our Ġcontributions Ġare Ġthree - f olds : Ġ1 ) Ġwe Ġshed Ġlight Ġon Ġwhy Ġprevious ĠST Ġmodels Ġcannot Ġsufficiently Ġutilize Ġthe Ġknowledge Ġlearned Ġfrom Ġthe Ġpre - training Ġprocess ; Ġ2 ) Ġwe Ġpropose Ġa Ġnew ĠST Ġmodel , Ġwhich Ġallev iates Ġshortcomings Ġin Ġexisting Ġmethods ; Ġand Ġ3 ) Ġwe Ġempir ically Ġevaluate Ġthe Ġproposed Ġmodel Ġon Ġa Ġlarge - scale Ġpublic Ġdataset . </s> Background Ġ: :: ĠProblem ĠForm ulation </s> End - to - end Ġspeech Ġtranslation Ġaims Ġto Ġtranslate Ġa Ġpiece Ġof Ġaudio Ġinto Ġa Ġtarget - language Ġtranslation Ġin Ġone Ġstep . ĠThe Ġraw Ġspeech Ġsignals Ġare Ġusually Ġconverted Ġto Ġsequences Ġof Ġacoustic Ġfeatures , Ġe . g . ĠMel Ġfilter bank Ġfeatures . ĠHere , Ġwe Ġdefine Ġthe Ġspeech Ġfeature Ġsequence Ġas Ġ$\ math bf Ġ{ x } Ġ= Ġ( x _ 1 , Ġ\ cd ots Ġ, Ġx _{ T _ x }) $. The Ġtranscription Ġand Ġtranslation Ġsequences Ġare Ġden oted Ġas Ġ$\ math bf Ġ{ y ^{ s }} Ġ= Ġ( y _ 1 ^{ s }, Ġ\ cd ots Ġ, Ġy _{ T _ s } ^{ s }) $, Ġand Ġ$\ math bf Ġ{ y ^{ t }} Ġ= Ġ( y _ 1 ^{ t }, Ġ\ cd ots Ġ, Ġy _{ T _ t } ^{ t }) $ Ġre pect ively . ĠEach Ġsymbol Ġin Ġ$\ math bf Ġ{ y ^{ s }} $ Ġor Ġ$\ math bf Ġ{ y ^{ t }} $ Ġis Ġan Ġinteger Ġindex Ġof Ġthe Ġsymbol Ġin Ġa Ġvocabulary Ġ$ V _{ src } $ Ġor Ġ$ V _{ tr g } $ Ġrespectively Ġ( e . g . Ġ$ y ^ s _ i = k , Ġk \ in Ġ[ 0 , Ġ| V _{ src } | - 1 ] $ ). ĠIn Ġthis Ġwork , Ġwe Ġsuppose Ġthat Ġan ĠAS R Ġdataset , Ġan ĠMT Ġdataset , Ġand Ġa ĠST Ġdataset Ġare Ġavailable , Ġden oted Ġas Ġ$\ math cal Ġ{ A } Ġ= Ġ\ l brace Ġ( \ math bf Ġ{ x _ i }, Ġ\ math bf Ġ{ y ^{ s } _ i } )\ r brace Ġ_ { i = 0 } ^ I $, Ġ$\ math cal Ġ{ M } Ġ= \ l brace Ġ( \ math bf Ġ{ y ^{ s } _ j }, Ġ\ math bf Ġ{ y ^{ t } _ j } )\ r brace Ġ_ { j = 0 } ^ J $ Ġand Ġ$ Ġ\ math cal Ġ{ S } Ġ= \ l brace Ġ( \ math bf Ġ{ x _ l }, Ġ\ math bf Ġ{ y ^{ t } _ l } )\ r brace Ġ_ { l = 0 } ^ L $ Ġrespectively . ĠGiven Ġa Ġnew Ġpiece Ġof Ġaudio Ġ$\ math bf Ġ{ x } $, Ġour Ġgoal Ġis Ġto Ġlearn Ġan Ġend Ġto Ġend Ġmodel Ġto Ġgenerate Ġa Ġtranslation Ġsentence Ġ$\ math bf Ġ{ y ^{ t }} $ Ġwithout Ġgenerating Ġan Ġintermediate Ġresult Ġ$\ math bf Ġ{ y ^{ s }} $. </s> Background Ġ: :: ĠMulti - Task ĠLearning Ġand ĠPre - training Ġfor ĠST </s> To Ġleverage Ġlarge Ġscale ĠAS R Ġand ĠMT Ġdata , Ġmulti - task Ġlearning Ġand Ġpre - training Ġtechniques Ġare Ġwidely Ġemployed Ġto Ġimprove Ġthe ĠST Ġsystem . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 4 , Ġthere Ġare Ġthree Ġpopular Ġmulti - task Ġstrategies Ġfor ĠST , Ġincluding Ġ1 ) Ġone - to - many Ġsetting , Ġin Ġwhich Ġa Ġspeech Ġenc oder Ġis Ġshared Ġbetween ĠAS R Ġand ĠST Ġtasks ; Ġ2 ) Ġmany - to - one Ġsetting Ġin Ġwhich Ġa Ġdec oder Ġis Ġshared Ġbetween ĠMT Ġand ĠST Ġtasks ; Ġand Ġ3 ) Ġmany - to - many Ġsetting Ġwhere Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġshared . </s> A Ġmany - to - many Ġmulti - task Ġmodel Ġcontains Ġtwo Ġenc od ers Ġas Ġwell Ġas Ġtwo Ġdec od ers . ĠIt Ġcan Ġbe Ġjointly Ġtrained Ġon ĠAS R , ĠMT , Ġand ĠST Ġtasks . ĠAs Ġthe Ġattention Ġmodule Ġis Ġtask - specific , Ġthree Ġattent ions Ġare Ġdefined . </s> Usually , Ġthe Ġsize Ġof Ġ$\ math cal Ġ{ A } $ Ġand Ġ$\ math cal Ġ{ M } $ Ġis Ġmuch Ġlarger Ġthan Ġ$\ math cal Ġ{ S } $. ĠTherefore , Ġthe Ġcommon Ġtraining Ġpractice Ġis Ġto Ġpre - train Ġthe Ġmodel Ġon ĠAS R Ġand ĠMT Ġtasks Ġand Ġthen Ġfine - t une Ġit Ġwith Ġa Ġmulti - task Ġlearning Ġmanner . ĠHowever , Ġas Ġaforementioned , Ġthis Ġmethod Ġsuffers Ġfrom Ġsub net Ġwaste , Ġrole Ġmismatch Ġand Ġnon - pre - trained Ġattention Ġissues , Ġwhich Ġseverely Ġlimits Ġthe Ġend - to - end ĠST Ġperformance . </s> Our Ġmethod </s> In Ġthis Ġsection , Ġwe Ġfirst Ġintroduce Ġthe Ġarchitecture Ġof ĠTC EN , Ġwhich Ġconsists Ġof Ġtwo Ġenc od ers Ġconnected Ġin Ġtandem , Ġand Ġone Ġdec oder Ġwith Ġan Ġattention Ġmodule . ĠThen Ġwe Ġgive Ġthe Ġpre - training Ġand Ġfine - tun ing Ġstrategy Ġfor ĠTC EN . ĠFinally , Ġwe Ġpropose Ġour Ġsolutions Ġfor Ġsemantic Ġand Ġlength Ġinconsistency Ġproblems , Ġwhich Ġare Ġcaused Ġby Ġmulti - task Ġlearning . </s> Our Ġmethod Ġ: :: ĠTC EN ĠArchitecture </s> Figure ĠFIG REF 5 Ġsketches Ġthe Ġoverall Ġarchitecture Ġof ĠTC EN , Ġincluding Ġa Ġspeech Ġenc oder Ġ$ enc _ s $, Ġa Ġtext Ġenc oder Ġ$ enc _ t $ Ġand Ġa Ġdec oder Ġ$ dec $ Ġwith Ġan Ġattention Ġmodule Ġ$ att $. ĠDuring Ġtraining , Ġthe Ġ$ enc _ s $ Ġacts Ġlike Ġan Ġacoustic Ġmodel Ġwhich Ġreads Ġthe Ġinput Ġ$\ math bf Ġ{ x } $ Ġto Ġword Ġor Ġsub word Ġrepresentations Ġ$\ math bf Ġ{ h ^ s } $, Ġthen Ġ$ enc _ t $ Ġlearns Ġhigh - level Ġlinguistic Ġknowledge Ġinto Ġhidden Ġrepresentations Ġ$\ math bf Ġ{ h ^ t } $. ĠFinally , Ġthe Ġ$ dec $ Ġdefines Ġa Ġdistribution Ġprobability Ġover Ġtarget Ġwords . ĠThe Ġadvantage Ġof Ġour Ġarchitecture Ġis Ġthat Ġtwo Ġenc od ers Ġdis ent angle Ġacoustic Ġfeature Ġextraction Ġand Ġlinguistic Ġfeature Ġextraction , Ġmaking Ġsure Ġthat Ġvaluable Ġknowledge Ġlearned Ġfrom ĠAS R Ġand ĠMT Ġtasks Ġcan Ġbe Ġeffectively Ġlever aged Ġfor ĠST Ġtraining . ĠBesides , Ġevery Ġmodule Ġin Ġpre - training Ġcan Ġbe Ġutilized Ġin Ġfine - tun ing , Ġallev iating Ġthe Ġsub net Ġwaste Ġproblem . </s> Follow ĠB IB REF 9 Ġin ag uma 2018 speech , Ġwe Ġuse ĠCNN - Bi L ST M Ġarchitecture Ġto Ġbuild Ġour Ġmodel . ĠSpecifically , Ġthe Ġinput Ġfeatures Ġ$\ math bf Ġ{ x } $ Ġare Ġorganized Ġas Ġa Ġsequence Ġof Ġfeature Ġvectors Ġin Ġlength Ġ$ T _ x $. ĠThen , Ġ$\ math bf Ġ{ x } $ Ġis Ġpassed Ġinto Ġa Ġstack Ġof Ġtwo Ġconv olution al Ġlayers Ġfollowed Ġby Ġmax - pool ing : </s> where Ġ$\ math bf Ġ{ v } ^{ ( l - 1 )} $ Ġis Ġfeature Ġmaps Ġin Ġlast Ġlayer Ġand Ġ$\ math bf Ġ{ W } ^{ ( l )} $ Ġis Ġthe Ġfilter . ĠThe Ġmax - pool ing Ġlayers Ġdowns ample Ġthe Ġsequence Ġin Ġlength Ġby Ġa Ġtotal Ġfactor Ġof Ġfour . ĠThe Ġdown - sam pled Ġfeature Ġsequence Ġis Ġfurther Ġfed Ġinto Ġa Ġstack Ġof Ġfive Ġbid irection al Ġ$ d $ - dimensional ĠL ST M Ġlayers : </s> where Ġ$ [ ; ] $ Ġdenotes Ġthe Ġvector Ġconc aten ation . ĠThe Ġfinal Ġoutput Ġrepresentation Ġfrom Ġthe Ġspeech Ġenc oder Ġis Ġden oted Ġas Ġ$\ math bf Ġ{ h ^ s } =( h ^ s _ 1 , Ġ\ cd ots Ġ, Ġh ^ s _{ \ frac { T _ x }{ 4 }} ) $, Ġwhere Ġ$ h _ i ^ s Ġ\ in Ġ\ math bb Ġ{ R } ^ d $. </s> The Ġtext Ġenc oder Ġ$ enc _ t $ Ġconsists Ġof Ġtwo Ġbid irection al ĠL ST M Ġlayers . ĠIn ĠST Ġtask , Ġ$ enc _ t $ Ġaccepts Ġspeech Ġenc oder Ġoutput Ġ$\ math bf Ġ{ h } ^ s $ Ġas Ġinput . ĠWhile Ġin ĠMT , Ġ$ enc _ t $ Ġconsumes Ġthe Ġword Ġembed ding Ġrepresentation Ġ$\ math bf Ġ{ e ^ s } $ Ġderived Ġfrom Ġ$\ math bf Ġ{ y ^ s } $, Ġwhere Ġeach Ġelement Ġ$ e ^ s _ i $ Ġis Ġcomputed Ġby Ġchoosing Ġthe Ġ$ y _ i ^ s $ - th Ġvector Ġfrom Ġthe Ġsource Ġembed ding Ġmatrix Ġ$ W _{ E ^ s } $. ĠThe Ġgoal Ġof Ġ$ enc _ t $ Ġis Ġto Ġextract Ġhigh - level Ġlinguistic Ġfeatures Ġlike Ġsynt actic Ġfeatures Ġor Ġsemantic Ġfeatures Ġfrom Ġlower Ġlevel Ġsub word Ġrepresentations Ġ$\ math bf Ġ{ h } ^ s $ Ġor Ġ$\ math bf Ġ{ e } ^ s $. ĠSince Ġ$\ math bf Ġ{ h } ^ s $ Ġand Ġ$\ math bf Ġ{ e } ^ s $ Ġbelong Ġto Ġdifferent Ġlatent Ġspace Ġand Ġhave Ġdifferent Ġlengths , Ġthere Ġremain Ġsemantic Ġand Ġlength Ġinconsistency Ġproblems . ĠWe Ġwill Ġprovide Ġour Ġsolutions Ġin ĠSection ĠSEC REF 21 . ĠThe Ġoutput Ġsequence Ġof Ġ$ enc _ t $ Ġis Ġden oted Ġas Ġ$\ math bf Ġ{ h } ^ t $. </s> The Ġdec oder Ġis Ġdefined Ġas Ġtwo Ġun id irection al ĠL ST M Ġlayers Ġwith Ġan Ġadditive Ġattention Ġ$ att $. ĠIt Ġpredicts Ġtarget Ġsequence Ġ$\ math bf Ġ{ y ^{ t }} $ Ġby Ġestimating Ġconditional Ġprobability Ġ$ P (\ math bf Ġ{ y ^{ t }} | \ math bf Ġ{ x }) $ : </s> Here , Ġ$ z _ k $ Ġis Ġthe Ġthe Ġhidden Ġstate Ġof Ġthe Ġde oc der ĠR NN Ġat Ġ$ k $ Ġstep Ġand Ġ$ c _ k $ Ġis Ġa Ġtime - dependent Ġcontext Ġvector Ġcomputed Ġby Ġthe Ġattention Ġ$ att $. </s> Our Ġmethod Ġ: :: ĠTraining ĠProcedure </s> Following Ġprevious Ġwork , Ġwe Ġsplit Ġthe Ġtraining Ġprocedure Ġto Ġpre - training Ġand Ġfine - tun ing Ġstages . ĠIn Ġpre - training Ġstage , Ġthe Ġspeech Ġenc oder Ġ$ enc _ s $ Ġis Ġtrained Ġtowards ĠC TC Ġobjective Ġusing Ġdataset Ġ$\ math cal Ġ{ A } $, Ġwhile Ġthe Ġtext Ġenc oder Ġ$ enc _ t $ Ġand Ġthe Ġdec oder Ġ$ dec $ Ġare Ġtrained Ġon ĠMT Ġdataset Ġ$\ math cal Ġ{ M } $. ĠIn Ġfine - tun ing Ġstage , Ġwe Ġjointly Ġtrain Ġthe Ġmodel Ġon ĠAS R , ĠMT , Ġand ĠST Ġtasks . </s> Our Ġmethod Ġ: :: ĠTraining ĠProcedure Ġ: :: ĠPre - training </s> To Ġsufficiently Ġutilize Ġthe Ġlarge Ġdataset Ġ$\ math cal Ġ{ A } $ Ġand Ġ$\ math cal Ġ{ M } $, Ġthe Ġmodel Ġis Ġpre - trained Ġon ĠC TC - based ĠAS R Ġtask Ġand ĠMT Ġtask Ġin Ġthe Ġpre - training Ġstage . </s> For ĠAS R Ġtask , Ġin Ġorder Ġto Ġget Ġrid Ġof Ġthe Ġrequirement Ġfor Ġdec oder Ġand Ġenable Ġthe Ġ$ enc _ s $ Ġto Ġgenerate Ġsub word Ġrepresentation , Ġwe Ġleverage Ġconnection ist Ġtemporal Ġclassification Ġ( CT C ) ĠB IB REF 8 Ġloss Ġto Ġtrain Ġthe Ġspeech Ġenc oder . </s> Given Ġan Ġinput Ġ$\ math bf Ġ{ x } $, Ġ$ enc _ s $ Ġemits Ġa Ġsequence Ġof Ġhidden Ġvectors Ġ$\ math bf Ġ{ h ^ s } $, Ġthen Ġa Ġsoft max Ġclassification Ġlayer Ġpredicts Ġa ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $, Ġwhere Ġ$\ pi Ġ_ t Ġ\ in ĠV _{ src } Ġ\ cup Ġ$ Ġ{ ` -' } Ġis Ġthe Ġobserving Ġlabel Ġat Ġparticular ĠR NN Ġstep Ġ$ t $, Ġand Ġ` -' Ġis Ġthe Ġblank Ġtoken Ġrepresenting Ġno Ġobserved Ġlabels : </s> where Ġ$ W _{ ct c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġ( | V _{ src } | + 1 )} $ Ġis Ġthe Ġweight Ġmatrix Ġin Ġthe Ġclassification Ġlayer Ġand Ġ$ T $ Ġis Ġthe Ġtotal Ġlength Ġof Ġenc oder ĠR NN . </s> A Ġlegal ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $ Ġis Ġa Ġvariation Ġof Ġthe Ġsource Ġtranscription Ġ$\ math bf Ġ{ y } ^ s $ Ġby Ġallowing Ġoccurrences Ġof Ġblank Ġtokens Ġand Ġrepet itions , Ġas Ġshown Ġin ĠTable ĠT AB REF 14 . ĠFor Ġeach Ġtranscription Ġ$\ math bf Ġ{ y } $, Ġthere Ġexist Ġmany Ġlegal ĠC TC Ġpaths Ġin Ġlength Ġ$ T $. ĠThe ĠC TC Ġobjective Ġtrains Ġthe Ġmodel Ġto Ġmaximize Ġthe Ġprobability Ġof Ġobserving Ġthe Ġgolden Ġsequence Ġ$\ math bf Ġ{ y } ^ s $, Ġwhich Ġis Ġcalculated Ġby Ġsum ming Ġthe Ġprobabilities Ġof Ġall Ġpossible Ġlegal Ġpaths : </s> where Ġ$\ Ph i Ġ_ T ( y ) $ Ġis Ġthe Ġset Ġof Ġall Ġlegal ĠC TC Ġpaths Ġfor Ġsequence Ġ$\ math bf Ġ{ y } $ Ġwith Ġlength Ġ$ T $. ĠThe Ġloss Ġcan Ġbe Ġeasily Ġcomputed Ġusing Ġforward - back ward Ġalgorithm . ĠMore Ġdetails Ġabout ĠC TC Ġare Ġprovided Ġin Ġsupplementary Ġmaterial . </s> For ĠMT Ġtask , Ġwe Ġuse Ġthe Ġcross - ent ropy Ġloss Ġas Ġthe Ġtraining Ġobjective . ĠDuring Ġtraining , Ġ$\ math bf Ġ{ y ^ s } $ Ġis Ġconverted Ġto Ġembed ding Ġvectors Ġ$\ math bf Ġ{ e ^ s } $ Ġthrough Ġembed ding Ġlayer Ġ$ W _{ E ^ s } $, Ġthen Ġ$ enc _ t $ Ġconsumes Ġ$\ math bf Ġ{ e ^ s } $ Ġand Ġpass Ġthe Ġoutput Ġ$\ math bf Ġ{ h ^ t } $ Ġto Ġdec oder . ĠThe Ġobjective Ġfunction Ġis Ġdefined Ġas : </s> Our Ġmethod Ġ: :: ĠTraining ĠProcedure Ġ: :: ĠFine - t une </s> In Ġfine - t une Ġstage , Ġwe Ġjointly Ġupdate Ġthe Ġmodel Ġon ĠAS R , ĠMT , Ġand ĠST Ġtasks . ĠThe Ġtraining Ġfor ĠAS R Ġand ĠMT Ġfollows Ġthe Ġsame Ġprocess Ġas Ġit Ġwas Ġin Ġpre - training Ġstage . </s> For ĠST Ġtask , Ġthe Ġ$ enc _ s $ Ġreads Ġthe Ġinput Ġ$\ math bf Ġ{ x } $ Ġand Ġgenerates Ġ$\ math bf Ġ{ h ^ s } $, Ġthen Ġ$ enc _ t $ Ġlearns Ġhigh - level Ġlinguistic Ġknowledge Ġinto Ġ$\ math bf Ġ{ h ^ t } $. ĠFinally , Ġthe Ġ$ dec $ Ġpredicts Ġthe Ġtarget Ġsentence . ĠThe ĠST Ġloss Ġfunction Ġis Ġdefined Ġas : </s> Following Ġthe Ġupdate Ġstrategy Ġproposed Ġby ĠB IB REF 11 Ġl u ong 2015 multi , Ġwe Ġallocate Ġa Ġdifferent Ġtraining Ġratio Ġ$\ alpha Ġ_ i $ Ġfor Ġeach Ġtask . ĠWhen Ġswitching Ġbetween Ġtasks , Ġwe Ġselect Ġrandomly Ġa Ġnew Ġtask Ġ$ i $ Ġwith Ġprobability Ġ$\ frac {\ alpha Ġ_ i }{ \ sum Ġ_ { j }\ alpha Ġ_ { j }} $. </s> Our Ġmethod Ġ: :: ĠSub net - Cons ist ency </s> Our Ġmodel Ġkeeps Ġrole Ġconsistency Ġbetween Ġpre - training Ġand Ġfine - tun ing Ġby Ġconnecting Ġtwo Ġenc od ers Ġfor ĠST Ġtask . ĠHowever , Ġthis Ġleads Ġto Ġsome Ġnew Ġproblems : Ġ1 ) ĠThe Ġtext Ġenc oder Ġconsumes Ġ$\ math bf Ġ{ e ^ s } $ Ġduring ĠMT Ġtraining , Ġwhile Ġit Ġaccepts Ġ$\ math bf Ġ{ h ^ s } $ Ġduring ĠST Ġtraining . ĠHowever , Ġ$\ math bf Ġ{ e ^ s } $ Ġand Ġ$\ math bf Ġ{ h ^ s } $ Ġmay Ġnot Ġfollow Ġthe Ġsame Ġdistribution , Ġresulting Ġin Ġthe Ġsemantic Ġinconsistency . Ġ2 ) ĠBesides , Ġthe Ġlength Ġof Ġ$\ math bf Ġ{ h ^ s } $ Ġis Ġnot Ġthe Ġsame Ġorder Ġof Ġmagnitude Ġwith Ġthe Ġlength Ġof Ġ$\ math bf Ġ{ e ^ s } $, Ġresulting Ġin Ġthe Ġlength Ġinconsistency . </s> In Ġresponse Ġto Ġthe Ġabove Ġtwo Ġchallenges , Ġwe Ġpropose Ġtwo Ġcounter measures : Ġ1 ) ĠWe Ġshare Ġweights Ġbetween ĠC TC Ġclassification Ġlayer Ġand Ġsource - end Ġword Ġembed ding Ġlayer Ġduring Ġtraining Ġof ĠAS R Ġand ĠMT , Ġencouraging Ġ$\ math bf Ġ{ e ^ s } $ Ġand Ġ$\ math bf Ġ{ h ^ s } $ Ġin Ġthe Ġsame Ġspace . Ġ2 ) We Ġfeed Ġthe Ġtext Ġenc oder Ġsource Ġsentences Ġin Ġthe Ġformat Ġof ĠC TC Ġpath , Ġwhich Ġare Ġgenerated Ġfrom Ġa Ġseq 2 seq Ġmodel , Ġmaking Ġit Ġmore Ġrobust Ġtoward Ġlong Ġinputs . </s> Our Ġmethod Ġ: :: ĠSub net - Cons ist ency Ġ: :: ĠSem antic ĠCons ist ency </s> As Ġshown Ġin ĠFigure ĠFIG REF 5 , Ġduring Ġmulti - task Ġtraining , Ġtwo Ġdifferent Ġhidden Ġfeatures Ġwill Ġbe Ġfed Ġinto Ġthe Ġtext Ġenc oder Ġ$ enc _ t $ : Ġthe Ġembed ding Ġrepresentation Ġ$\ math bf Ġ{ e } ^ s $ Ġin ĠMT Ġtask , Ġand Ġthe Ġ$ enc _ s $ Ġoutput Ġ$\ math bf Ġ{ h ^ s } $ Ġin ĠST Ġtask . ĠWithout Ġany Ġregular ization , Ġthey Ġmay Ġbelong Ġto Ġdifferent Ġlatent Ġspaces . ĠDue Ġto Ġthe Ġspace Ġgap , Ġthe Ġ$ enc _ t $ Ġhas Ġto Ġcompromise Ġbetween Ġtwo Ġtasks , Ġlimiting Ġits Ġperformance Ġon Ġindividual Ġtasks . </s> To Ġbridge Ġthe Ġspace Ġgap , Ġour Ġidea Ġis Ġto Ġpull Ġ$\ math bf Ġ{ h ^ s } $ Ġinto Ġthe Ġlatent Ġspace Ġwhere Ġ$\ math bf Ġ{ e } ^ s $ Ġbelong . ĠSpecifically , Ġwe Ġshare Ġthe Ġweight Ġ$ W _{ ct c } $ Ġin ĠC TC Ġclassification Ġlayer Ġwith Ġthe Ġsource Ġembed ding Ġweights Ġ$ W _{ E ^ s } $, Ġwhich Ġmeans Ġ$ W _{ ct c } Ġ= ĠW _{ E ^ s } $. ĠIn Ġthis Ġway , Ġwhen Ġpredicting Ġthe ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $, Ġthe Ġprobability Ġof Ġobserving Ġthe Ġparticular Ġlabel Ġ$ w _ i Ġ\ in ĠV _{ src }\ cup Ġ${ ` -' } Ġat Ġtime Ġstep Ġ$ t $, Ġ$ p (\ pi Ġ_ t = w _ i | \ math bf Ġ{ x }) $, Ġis Ġcomputed Ġby Ġnormal izing Ġthe Ġproduct Ġof Ġhidden Ġvector Ġ$ h _ t ^ s $ Ġand Ġthe Ġ$ i $ - th Ġvector Ġin Ġ$ W _{ E ^ s } $ : </s> The Ġloss Ġfunction Ġcloses Ġthe Ġdistance Ġbetween Ġ$ h ^ s _ t $ Ġand Ġgolden Ġembed ding Ġvector , Ġencouraging Ġ$\ math bf Ġ{ h } ^ s $ Ġhave Ġthe Ġsame Ġdistribution Ġwith Ġ$\ math bf Ġ{ e } ^ s $. </s> Our Ġmethod Ġ: :: ĠSub net - Cons ist ency Ġ: :: ĠLength ĠCons ist ency </s> Another Ġexisting Ġproblem Ġis Ġlength Ġinconsistency . ĠThe Ġlength Ġof Ġthe Ġsequence Ġ$\ math bf Ġ{ h ^ s } $ Ġis Ġproportional Ġto Ġthe Ġlength Ġof Ġthe Ġinput Ġframe Ġ$\ math bf Ġ{ x } $, Ġwhich Ġis Ġmuch Ġlonger Ġthan Ġthe Ġlength Ġof Ġ$\ math bf Ġ{ e ^ s } $. ĠTo Ġsolve Ġthis Ġproblem , Ġwe Ġtrain Ġan ĠR NN - based Ġseq 2 seq Ġmodel Ġto Ġtransform Ġnormal Ġsource Ġsentences Ġto Ġnoisy Ġsentences Ġin ĠC TC Ġpath Ġformat , Ġand Ġreplace Ġstandard ĠMT Ġwith Ġden o ising ĠMT Ġfor Ġmulti - t asking . </s> Specifically , Ġwe Ġfirst Ġtrain Ġa ĠC TC ĠAS R Ġmodel Ġbased Ġon Ġdataset Ġ$\ math cal Ġ{ A } Ġ= Ġ\ l brace Ġ( \ math bf Ġ{ x } _ i , Ġ\ math bf Ġ{ y } ^ s _ i )\ r brace Ġ_ { i = 0 } ^{ I } $, Ġand Ġgenerate Ġa ĠC TC - path Ġ$\ math bf Ġ{\ pi Ġ} _ i $ Ġfor Ġeach Ġaudio Ġ$\ math bf Ġ{ x } _ i $ Ġby Ġgreedy Ġdecoding . ĠThen Ġwe Ġdefine Ġan Ġoperation Ġ$ S (\ cd ot Ġ) $, Ġwhich Ġconverts Ġa ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $ Ġto Ġa Ġsequence Ġof Ġthe Ġunique Ġtokens Ġ$\ math bf Ġ{ u } $ Ġand Ġa Ġsequence Ġof Ġrepetition Ġtimes Ġfor Ġeach Ġtoken Ġ$\ math bf Ġ{ l } $, Ġden oted Ġas Ġ$ S (\ math bf Ġ{\ pi Ġ}) Ġ= Ġ( \ math bf Ġ{ u }, Ġ\ math bf Ġ{ l }) $. ĠNotably , Ġthe Ġoperation Ġis Ġreversible , Ġmeaning Ġthat Ġ$ S ^{ - 1 } Ġ( \ math bf Ġ{ u }, Ġ\ math bf Ġ{ l }) = \ math bf Ġ{\ pi Ġ} $. ĠWe Ġuse Ġthe Ġexample Ġ$\ math bf Ġ{\ pi Ġ_ 1 } $ Ġin ĠTable ĠT AB REF 14 Ġand Ġshow Ġthe Ġcorresponding Ġ$\ math bf Ġ{ u } $ Ġand Ġ$\ math bf Ġ{ l } $ Ġin ĠTable ĠT AB REF 24 . </s> Then Ġwe Ġbuild Ġa Ġdataset Ġ$\ math cal Ġ{ P } Ġ= Ġ\ l brace Ġ( \ math bf Ġ{ y ^ s } _ i , Ġ\ math bf Ġ{ u } _ i , Ġ\ math bf Ġ{ l } _ i )\ r brace Ġ_ { i = 0 } ^{ I } $ Ġby Ġdecoding Ġall Ġthe Ġaudio Ġpieces Ġin Ġ$\ math cal Ġ{ A } $ Ġand Ġtransform Ġthe Ġresulting Ġpath Ġby Ġthe Ġoperation Ġ$ S (\ cd ot Ġ) $. ĠAfter Ġthat , Ġwe Ġtrain Ġa Ġseq 2 seq Ġmodel , Ġas Ġshown Ġin ĠFigure ĠFIG REF 25 , Ġwhich Ġtakes Ġ$ Ġ\ math bf Ġ{ y ^ s } _ i $ Ġas Ġinput Ġand Ġdec odes Ġ$\ math bf Ġ{ u } _ i , Ġ\ math bf Ġ{ l } _ i $ Ġas Ġoutputs . ĠWith Ġthe Ġseq 2 seq Ġmodel , Ġa Ġnoisy ĠMT Ġdataset Ġ$\ math cal Ġ{ M } ^ {\ prime Ġ} = \ l brace Ġ( \ math bf Ġ{\ pi Ġ} _ l , Ġ\ math bf Ġ{ y ^ t } _ l )\ r brace Ġ_ { l = 0 } ^{ L } $ Ġis Ġobtained Ġby Ġconverting Ġevery Ġsource Ġsentence Ġ$\ math bf Ġ{ y ^ s } _ i Ġ\ in Ġ\ math cal Ġ{ M } $ Ġto Ġ$\ math bf Ġ{\ pi Ġ_ i } $, Ġwhere Ġ$\ math bf Ġ{\ pi Ġ} _ i Ġ= ĠS ^{ - 1 } (\ math bf Ġ{ u } _ i , Ġ\ math bf Ġ{ l } _ i ) $. ĠWe Ġdid Ġnot Ġuse Ġthe Ġstandard Ġseq 2 seq Ġmodel Ġwhich Ġtakes Ġ$\ math bf Ġ{ y ^ s } $ Ġas Ġinput Ġand Ġgenerates Ġ$\ math bf Ġ{\ pi Ġ} $ Ġdirectly , Ġsince Ġthere Ġare Ġtoo Ġmany Ġblank Ġtokens Ġ` -' Ġin Ġ$\ math bf Ġ{\ pi Ġ} $ Ġand Ġthe Ġmodel Ġtends Ġto Ġgenerate Ġa Ġlong Ġsequence Ġwith Ġonly Ġblank Ġtokens . ĠDuring ĠMT Ġtraining , Ġwe Ġrandomly Ġsample Ġtext Ġpairs Ġfrom Ġ$\ math cal Ġ{ M } ^ {\ prime Ġ} $ Ġand Ġ$\ math cal Ġ{ M } $ Ġaccording Ġto Ġa Ġhyper - param eter Ġ$ k $. ĠAfter Ġtuning Ġon Ġthe Ġvalidation Ġset , Ġabout Ġ$ 30 \ % $ Ġpairs Ġare Ġsampled Ġfrom Ġ$\ math cal Ġ{ M } ^ {\ prime Ġ} $. ĠIn Ġthis Ġway , Ġthe Ġ$ enc _ t $ Ġis Ġmore Ġrobust Ġtoward Ġthe Ġlonger Ġinputs Ġgiven Ġby Ġthe Ġ$ enc _ s $. </s> Exper iments </s> We Ġconduct Ġexperiments Ġon Ġthe ĠI W SL T 18 Ġspeech Ġtranslation Ġtask ĠB IB REF 1 . ĠSince ĠI W SL T Ġparticip ators Ġuse Ġdifferent Ġdata Ġpre - processing Ġmethods , Ġwe Ġreproduce Ġseveral Ġcompetitive Ġbas elines Ġbased Ġon Ġthe ĠESP net ĠB IB REF 12 Ġfor Ġa Ġfair Ġcomparison . </s> Exper iments Ġ: :: ĠDat as et Ġ: :: ĠSpeech Ġtranslation Ġdata : </s> The Ġorganizer Ġprovides Ġa Ġspeech Ġtranslation Ġcorpus Ġextracting Ġfrom Ġthe ĠTED Ġtalk Ġ( ST - TED ), Ġwhich Ġconsists Ġof Ġraw ĠEnglish Ġwave Ġfiles , ĠEnglish Ġtranscript ions , Ġand Ġaligned ĠGerman Ġtranslations . ĠThe Ġcorpus Ġcontains Ġ272 Ġhours Ġof ĠEnglish Ġspeech Ġwith Ġ171 k Ġsegments . ĠWe Ġsplit Ġ2 k Ġsegments Ġfrom Ġthe Ġcorpus Ġas Ġdev Ġset Ġand Ġt st 2010 , Ġt st 2013 , Ġt st 2014 , Ġt st 2015 Ġare Ġused Ġas Ġtest Ġsets . </s> Spe ech Ġrecognition Ġdata : ĠAside Ġfrom ĠST - TED , ĠTED - LI UM 2 Ġcorpus ĠB IB REF 13 Ġis Ġprovided Ġas Ġspeech Ġrecognition Ġdata , Ġwhich Ġcontains Ġ207 Ġhours Ġof ĠEnglish Ġspeech Ġand Ġ93 k Ġtranscript Ġsentences . </s> Text Ġtranslation Ġdata : ĠWe Ġuse Ġtranscription Ġand Ġtranslation Ġpairs Ġin Ġthe ĠST - TED Ġcorpus Ġand ĠW IT 3 Ġas Ġin - domain ĠMT Ġdata , Ġwhich Ġcontains Ġ130 k Ġand Ġ200 k Ġsentence Ġpairs Ġrespectively . ĠW MT 2018 Ġis Ġused Ġas Ġout - of - domain Ġtraining Ġdata Ġwhich Ġconsists Ġof Ġ41 M Ġsentence Ġpairs . </s> Data Ġpre processing : ĠFor Ġspeech Ġdata , Ġthe Ġutter ances Ġare Ġsegment ed Ġinto Ġmultiple Ġframes Ġwith Ġa Ġ25 Ġms Ġwindow Ġsize Ġand Ġa Ġ10 Ġms Ġstep Ġsize . ĠThen Ġwe Ġextract Ġ80 - channel Ġlog - Mel Ġfilter Ġbank Ġand Ġ3 - dimensional Ġpitch Ġfeatures Ġusing ĠK aldi ĠB IB REF 14 , Ġresulting Ġin Ġ83 - dimensional Ġinput Ġfeatures . ĠWe Ġnormal ize Ġthem Ġby Ġthe Ġmean Ġand Ġthe Ġstandard Ġdeviation Ġon Ġthe Ġwhole Ġtraining Ġset . ĠThe Ġutter ances Ġwith Ġmore Ġthan Ġ3000 Ġframes Ġare Ġdiscarded . ĠThe Ġtranscripts Ġin ĠST - TED Ġare Ġin Ġtrue - case Ġwith Ġpunct uation Ġwhile Ġin ĠTED - LI UM 2 , Ġtranscripts Ġare Ġin Ġlower - case Ġand Ġunp unct uated . ĠThus , Ġwe Ġlower case Ġall Ġthe Ġsentences Ġand Ġremove Ġthe Ġpunct uation Ġto Ġkeep Ġconsistent . ĠTo Ġincrease Ġthe Ġamount Ġof Ġtraining Ġdata , Ġwe Ġperform Ġspeed Ġpert urb ation Ġon Ġthe Ġraw Ġsignals Ġwith Ġspeed Ġfactors Ġ0 . 9 Ġand Ġ1 . 1 . ĠFor Ġthe Ġtext Ġtranslation Ġdata , Ġsentences Ġlonger Ġthan Ġ80 Ġwords Ġor Ġshorter Ġthan Ġ10 Ġwords Ġare Ġremoved . ĠBesides , Ġwe Ġdiscard Ġpairs Ġwhose Ġlength Ġratio Ġbetween Ġsource Ġand Ġtarget Ġsentence Ġis Ġsmaller Ġthan Ġ0 . 5 Ġor Ġlarger Ġthan Ġ2 . 0 . ĠWord Ġtoken ization Ġis Ġperformed Ġusing Ġthe ĠMoses Ġscripts Ġand Ġboth ĠEnglish Ġand ĠGerman Ġwords Ġare Ġin Ġlower - case . </s> We Ġuse Ġtwo Ġdifferent Ġsets Ġof Ġvocabulary Ġfor Ġour Ġexperiments . ĠFor Ġthe Ġsub word Ġexperiments , Ġboth ĠEnglish Ġand ĠGerman Ġvoc ab ular ies Ġare Ġgenerated Ġusing Ġsentence piece ĠB IB REF 15 Ġwith Ġa Ġfixed Ġsize Ġof Ġ5 k Ġtokens . ĠB IB REF 9 Ġin ag uma 2018 speech Ġshow Ġthat Ġincreasing Ġthe Ġvocabulary Ġsize Ġis Ġnot Ġhelpful Ġfor ĠST Ġtask . ĠFor Ġthe Ġcharacter Ġexperiments , Ġboth ĠEnglish Ġand ĠGerman Ġsentences Ġare Ġrepresented Ġin Ġthe Ġcharacter Ġlevel . </s> For Ġevaluation , Ġwe Ġsegment Ġeach Ġaudio Ġwith Ġthe ĠLI UM ĠSp k D iar ization Ġtool ĠB IB REF 16 Ġand Ġthen Ġperform ĠMW ER Ġsegment ation Ġwith ĠRW TH Ġtool kit ĠB IB REF 17 . ĠWe Ġuse Ġlower case ĠB LE U Ġas Ġevaluation Ġmetric . </s> Exper iments Ġ: :: ĠBas eline ĠModels Ġand ĠImplementation </s> We Ġcompare Ġour Ġmethod Ġwith Ġfollowing Ġbas elines . </s> Van illa ĠST Ġbaseline : ĠThe Ġvanilla ĠST ĠB IB REF 9 Ġhas Ġonly Ġa Ġspeech Ġenc oder Ġand Ġa Ġdec oder . ĠIt Ġis Ġtrained Ġfrom Ġscratch Ġon Ġthe ĠST - TED Ġcorpus . </s> Pre - training Ġbas elines : ĠWe Ġconduct Ġthree Ġpre - training Ġbaseline Ġexperiments : Ġ1 ) Ġenc oder Ġpre - training , Ġin Ġwhich Ġthe ĠST Ġenc oder Ġis Ġinitialized Ġfrom Ġan ĠAS R Ġmodel ; Ġ2 ) Ġdec oder Ġpre - training , Ġin Ġwhich Ġthe ĠST Ġdec oder Ġis Ġinitialized Ġfrom Ġan ĠMT Ġmodel ; Ġand Ġ3 ) Ġenc oder - dec oder Ġpre - training , Ġwhere Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġpre - trained . ĠThe ĠAS R Ġmodel Ġhas Ġthe Ġsame Ġarchitecture Ġwith Ġvanilla ĠST Ġmodel , Ġtrained Ġon Ġthe Ġmixture Ġof ĠST - TED Ġand ĠTED - LI UM 2 Ġcorpus . ĠThe ĠMT Ġmodel Ġhas Ġa Ġtext Ġenc oder Ġand Ġdec oder Ġwith Ġthe Ġsame Ġarchitecture Ġof Ġwhich Ġin ĠTC EN . ĠIt Ġis Ġfirst Ġtrained Ġon ĠW MT Ġdata Ġ( out - of - domain ) Ġand Ġthen Ġfine - tun ed Ġon Ġin - domain Ġdata . </s> Multi - task Ġbas elines : ĠWe Ġalso Ġconduct Ġthree Ġmulti - task Ġbaseline Ġexperiments Ġincluding Ġone - to - many Ġsetting , Ġmany - to - one Ġsetting , Ġand Ġmany - to - many Ġsetting . ĠIn Ġthe Ġfirst Ġtwo Ġsettings , Ġwe Ġtrain Ġthe Ġmodel Ġwith Ġ$\ alpha Ġ_ { st } = 0 . 75 $ Ġwhile Ġ$\ alpha Ġ_ { as r } = 0 . 25 $ Ġor Ġ$\ alpha Ġ_ { mt } = 0 . 25 $. ĠFor Ġmany - to - many Ġsetting , Ġwe Ġuse Ġ$\ alpha Ġ_ { st } = 0 . 6 , Ġ\ alpha Ġ_ { as r } = 0 . 2 $ Ġand Ġ$\ alpha Ġ_ { mt } = 0 . 2 $ .. ĠFor ĠMT Ġtask , Ġwe Ġuse Ġonly Ġin - domain Ġdata . </s> Many - to - many + pre - training : ĠWe Ġtrain Ġa Ġmany - to - many Ġmulti - task Ġmodel Ġwhere Ġthe Ġenc od ers Ġand Ġdec od ers Ġare Ġderived Ġfrom Ġpre - trained ĠAS R Ġand ĠMT Ġmodels . ĠTriangle + pre - train : ĠB IB REF 18 ĠD BL P : conf / na acl / An ast as opoulos C 18 Ġproposed Ġa Ġtriangle Ġmulti - task Ġstrategy Ġfor Ġspeech Ġtranslation . ĠTheir Ġmodel Ġsolves Ġthe Ġsub net Ġwaste Ġissue Ġby Ġconc aten ating Ġan ĠST Ġdec oder Ġto Ġan ĠAS R Ġenc oder - dec oder Ġmodel . ĠNotably , Ġtheir ĠST Ġdec oder Ġcan Ġconsume Ġrepresentations Ġfrom Ġthe Ġspeech Ġenc oder Ġas Ġwell Ġas Ġthe ĠAS R Ġdec oder . ĠFor Ġa Ġfair Ġcomparison , Ġthe Ġspeech Ġenc oder Ġand Ġthe ĠAS R Ġdec oder Ġare Ġinitialized Ġfrom Ġthe Ġpre - trained ĠAS R Ġmodel . ĠThe ĠTriangle Ġmodel Ġis Ġfine - tun ed Ġunder Ġtheir Ġmulti - task Ġmanner . </s> All Ġour Ġbas elines Ġas Ġwell Ġas ĠTC EN Ġare Ġimplemented Ġbased Ġon ĠESP net ĠB IB REF 12 , Ġthe ĠR NN Ġsize Ġis Ġset Ġas Ġ$ d = 1024 $ Ġfor Ġall Ġmodels . ĠWe Ġuse Ġa Ġdrop out Ġof Ġ0 . 3 Ġfor Ġembed d ings Ġand Ġenc od ers , Ġand Ġtrain Ġusing ĠAd ad elta Ġwith Ġinitial Ġlearning Ġrate Ġof Ġ1 . 0 Ġfor Ġa Ġmaximum Ġof Ġ10 Ġepoch s . </s> For Ġtraining Ġof ĠTC EN , Ġwe Ġset Ġ$\ alpha Ġ_ { as r } = 0 . 2 $ Ġand Ġ$\ alpha Ġ_ { mt } = 0 . 8 $ Ġin Ġthe Ġpre - training Ġstage , Ġsince Ġthe ĠMT Ġdataset Ġis Ġmuch Ġlarger Ġthan ĠAS R Ġdataset . ĠFor Ġfine - t une , Ġwe Ġuse Ġ$\ alpha Ġ_ { st } = 0 . 6 , Ġ\ alpha Ġ_ { as r } = 0 . 2 $ Ġand Ġ$\ alpha Ġ_ { mt } = 0 . 2 $, Ġsame Ġas Ġthe Ġ` many - to - many ' Ġbaseline . </s> For Ġtesting , Ġwe Ġselect Ġthe Ġmodel Ġwith Ġthe Ġbest Ġaccuracy Ġon Ġspeech Ġtranslation Ġtask Ġon Ġdev Ġset . ĠAt Ġinference Ġtime , Ġwe Ġuse Ġa Ġbeam Ġsize Ġof Ġ10 , Ġand Ġthe Ġbeam Ġscores Ġinclude Ġlength Ġnormal ization Ġwith Ġa Ġweight Ġof Ġ0 . 2 . </s> Exper iments Ġ: :: ĠExperimental ĠResults </s> Table ĠT AB REF 29 Ġshows Ġthe Ġresults Ġon Ġfour Ġtest Ġsets Ġas Ġwell Ġas Ġthe Ġaverage Ġperformance . ĠOur Ġmethod Ġsignificantly Ġoutper forms Ġthe Ġstrong Ġ` many - to - many + pret rain ' Ġbaseline Ġby Ġ3 . 6 Ġand Ġ2 . 2 ĠB LE U Ġscores Ġrespectively , Ġindicating Ġthe Ġproposed Ġmethod Ġis Ġvery Ġeffective Ġthat Ġsubstantially Ġimproves Ġthe Ġtranslation Ġquality . ĠBesides , Ġboth Ġpre - training Ġand Ġmulti - task Ġlearning Ġcan Ġimprove Ġtranslation Ġquality , Ġand Ġthe Ġpre - training Ġsettings Ġ( 2 nd - 4 th Ġrows ) Ġare Ġmore Ġeffective Ġcompared Ġto Ġmulti - task Ġsettings Ġ( 5 th - 8 th Ġrows ). ĠWe Ġobserve Ġa Ġperformance Ġdegradation Ġin Ġthe Ġ` tri angle + pret rain ' Ġbaseline . ĠCompared Ġto Ġour Ġmethod , Ġwhere Ġthe Ġdec oder Ġreceives Ġhigher - level Ġsynt actic Ġand Ġsemantic Ġlinguistic Ġknowledge Ġextracted Ġfrom Ġtext Ġenc oder , Ġtheir ĠAS R Ġdec oder Ġcan Ġonly Ġprovide Ġlower Ġword - level Ġlinguistic Ġinformation . ĠBesides , Ġsince Ġtheir Ġmodel Ġlacks Ġtext Ġenc oder Ġand Ġthe Ġarchitecture Ġof ĠST Ġdec oder Ġis Ġdifferent Ġfrom ĠMT Ġdec oder , Ġtheir Ġmodel Ġcannot Ġutilize Ġthe Ġlarge - scale ĠMT Ġdata Ġin Ġall Ġthe Ġtraining Ġstages . ĠInterestingly , Ġwe Ġfind Ġthat Ġthe Ġchar - level Ġmodels Ġoutper form Ġthe Ġsub word - level Ġmodels Ġin Ġall Ġsettings , Ġespecially Ġin Ġvanilla Ġbaseline . ĠA Ġsimilar Ġphenomenon Ġis Ġobserved Ġby ĠB IB REF 6 Ġber ard 2018 end . ĠA Ġpossible Ġexplanation Ġis Ġthat Ġlearning Ġthe Ġalign ments Ġbetween Ġspeech Ġframes Ġand Ġsub word Ġunits Ġin Ġanother Ġlanguage Ġis Ġnotoriously Ġdifficult . ĠOur Ġmethod Ġcan Ġbring Ġmore Ġgains Ġin Ġthe Ġsub word Ġsetting Ġsince Ġour Ġmodel Ġis Ġgood Ġat Ġlearning Ġthe Ġtext - to - text Ġalignment Ġand Ġthe Ġsub word - level Ġalignment Ġis Ġmore Ġhelpful Ġto Ġthe Ġtranslation Ġquality . </s> Exper iments Ġ: :: ĠDiscussion Ġ: :: ĠAb lation ĠStudy </s> To Ġbetter Ġunderstand Ġthe Ġcontribution Ġof Ġeach Ġcomponent , Ġwe Ġperform Ġan Ġab lation Ġstudy Ġon Ġsub word - level Ġexperiments . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 37 . ĠIn Ġ` - MT Ġnoise ' Ġsetting , Ġwe Ġdo Ġnot Ġadd Ġnoise Ġto Ġsource Ġsentences Ġfor ĠMT . ĠIn Ġ` - weight Ġsharing ' Ġsetting , Ġwe Ġuse Ġdifferent Ġparameters Ġin ĠC TC Ġclassification Ġlayer Ġand Ġsource Ġembed ding Ġlayer . ĠThese Ġtwo Ġexperiments Ġprove Ġthat Ġboth Ġweight Ġsharing Ġand Ġusing Ġnoisy ĠMT Ġinput Ġbenefit Ġto Ġthe Ġfinal Ġtranslation Ġquality . ĠPerformance Ġde grades Ġmore Ġin Ġ` - weight Ġsharing ', Ġindicating Ġthe Ġsemantic Ġconsistency Ġcontributes Ġmore Ġto Ġour Ġmodel . ĠIn Ġthe Ġ` - pret rain ' Ġexperiment , Ġwe Ġremove Ġthe Ġpre - training Ġstage Ġand Ġdirectly Ġupdate Ġthe Ġmodel Ġon Ġthree Ġtasks , Ġleading Ġto Ġa Ġdramatic Ġdecrease Ġon ĠB LE U Ġscore , Ġindicating Ġthe Ġpre - training Ġis Ġan Ġindispensable Ġstep Ġfor Ġend - to - end ĠST . </s> Exper iments Ġ: :: ĠDiscussion Ġ: :: ĠLearning ĠCurve </s> It Ġis Ġinteresting Ġto Ġinvestigate Ġwhy Ġour Ġmethod Ġis Ġsuperior Ġto Ġbas elines . ĠWe Ġfind Ġthat ĠTC EN Ġachieves Ġa Ġhigher Ġfinal Ġresult Ġowing Ġto Ġa Ġbetter Ġstart - point Ġin Ġfine - tun ing . ĠFigure ĠFIG REF 39 Ġprovides Ġlearning Ġcurves Ġof Ġsub word Ġaccuracy Ġon Ġvalidation Ġset . ĠThe Ġx - axis Ġdenotes Ġthe Ġfine - tun ing Ġtraining Ġsteps . ĠThe Ġvanilla Ġmodel Ġstarts Ġat Ġa Ġlow Ġaccuracy , Ġbecause Ġits Ġnetworks Ġare Ġnot Ġpre - trained Ġon Ġthe ĠAS R Ġand ĠMT Ġdata . ĠThe Ġtrends Ġof Ġour Ġmodel Ġand Ġ` many - to - many + pret rain ' Ġare Ġsimilar , Ġbut Ġour Ġmodel Ġoutper forms Ġit Ġabout Ġfive Ġpoints Ġin Ġthe Ġwhole Ġfine - tun ing Ġprocess . ĠIt Ġindicates Ġthat Ġthe Ġgain Ġcomes Ġfrom Ġbrid ging Ġthe Ġgap Ġbetween Ġpre - training Ġand Ġfine - tun ing Ġrather Ġthan Ġa Ġbetter Ġfine - tun ing Ġprocess . </s> Exper iments Ġ: :: ĠDiscussion Ġ: :: ĠCompared Ġwith Ġa ĠC asc aded ĠSystem </s> Table ĠT AB REF 29 Ġcompares Ġour Ġmodel Ġwith Ġend - to - end Ġbas elines . ĠHere , Ġwe Ġcompare Ġour Ġmodel Ġwith Ġcasc aded Ġsystems . ĠWe Ġbuild Ġa Ġcasc aded Ġsystem Ġby Ġcombining Ġthe ĠAS R Ġmodel Ġand ĠMT Ġmodel Ġused Ġin Ġpre - training Ġbaseline . ĠWord Ġerror Ġrate Ġ( WER ) Ġof Ġthe ĠAS R Ġsystem Ġand ĠB LE U Ġscore Ġof Ġthe ĠMT Ġsystem Ġare Ġreported Ġin Ġthe Ġsupplementary Ġmaterial . ĠIn Ġaddition Ġto Ġa Ġsimple Ġcombination Ġof Ġthe ĠAS R Ġand ĠMT Ġsystems , Ġwe Ġalso Ġre - se gment Ġthe ĠAS R Ġoutputs Ġbefore Ġfeeding Ġto Ġthe ĠMT Ġsystem , Ġden oted Ġas Ġcasc aded + re - se g . ĠSpecifically , Ġwe Ġtrain Ġa Ġseq 2 seq Ġmodel ĠB IB REF 19 Ġon Ġthe ĠMT Ġdataset , Ġwhere Ġthe Ġsource Ġside Ġis Ġa Ġno Ġpunct uation Ġsentence Ġand Ġthe Ġtarget Ġside Ġis Ġa Ġnatural Ġsentence . ĠAfter Ġthat , Ġwe Ġuse Ġthe Ġseq 2 seq Ġmodel Ġto Ġadd Ġsentence Ġboundaries Ġand Ġpunct uation Ġon ĠAS R Ġoutputs . ĠExperimental Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 41 . ĠOur Ġend - to - end Ġmodel Ġoutper forms Ġthe Ġsimple Ġcasc aded Ġmodel Ġover Ġ2 ĠB LE U Ġscores , Ġand Ġit Ġachieves Ġa Ġcomparable Ġperformance Ġwith Ġthe Ġcasc aded Ġmodel Ġcombining Ġwith Ġa Ġsentence Ġre - se gment Ġmodel . </s> Related ĠWork </s> Early Ġworks Ġconduct Ġspeech Ġtranslation Ġin Ġa Ġpipeline Ġmanner ĠB IB REF 2 , ĠB IB REF 20 , Ġwhere Ġthe ĠAS R Ġoutput Ġlatt ices Ġare Ġfed Ġinto Ġan ĠMT Ġsystem Ġto Ġgenerate Ġtarget Ġsentences . ĠH MM ĠB IB REF 21 , ĠD ense Net ĠB IB REF 22 , ĠTD NN ĠB IB REF 23 Ġare Ġcommonly Ġused ĠAS R Ġsystems , Ġwhile ĠR NN Ġwith Ġattention ĠB IB REF 19 Ġand ĠTrans former ĠB IB REF 10 Ġare Ġtop Ġchoices Ġfor ĠMT . ĠTo Ġenhance Ġthe Ġrobust ness Ġof Ġthe ĠN MT Ġmodel Ġtowards ĠAS R Ġerrors , ĠB IB REF 24 ĠD BL P : conf / e acl / Ts vet kov MD 14 Ġand ĠB IB REF 25 ĠD BL P : conf / as ru / C hen H HL 17 Ġpropose Ġto Ġsimulate Ġthe Ġnoise Ġin Ġtraining Ġand Ġinference . </s> To Ġavoid Ġerror Ġpropagation Ġand Ġhigh Ġlatency Ġissues , Ġrecent Ġworks Ġpropose Ġtranslating Ġthe Ġacoustic Ġspeech Ġinto Ġtext Ġin Ġtarget Ġlanguage Ġwithout Ġyielding Ġthe Ġsource Ġtranscription ĠB IB REF 4 . ĠSince ĠST Ġdata Ġis Ġscarce , Ġpre - training ĠB IB REF 7 , Ġmulti - task Ġlearning ĠB IB REF 4 , ĠB IB REF 6 , Ġcurriculum Ġlearning ĠB IB REF 26 , Ġattention - pass ing ĠB IB REF 27 , Ġand Ġknowledge Ġdist illation ĠB IB REF 28 , ĠB IB REF 29 Ġstrategies Ġhave Ġbeen Ġexplored Ġto Ġutilize ĠAS R Ġdata Ġand ĠMT Ġdata . ĠSpecifically , ĠB IB REF 5 ĠD BL P : conf / inters pe ech / Weiss C J WC 17 Ġshow Ġimprovements Ġof Ġperformance Ġby Ġtraining Ġthe ĠST Ġmodel Ġjointly Ġwith Ġthe ĠAS R Ġand Ġthe ĠMT Ġmodel . ĠB IB REF 6 Ġber ard 2018 end Ġobserve Ġfaster Ġconvergence Ġand Ġbetter Ġresults Ġdue Ġto Ġpre - training Ġand Ġmulti - task Ġlearning Ġon Ġa Ġlarger Ġdataset . ĠB IB REF 7 ĠD BL P : conf / na acl / B ans al K LL G 19 Ġshow Ġthat Ġpre - training Ġa Ġspeech Ġenc oder Ġon Ġone Ġlanguage Ġcan Ġimprove ĠST Ġquality Ġon Ġa Ġdifferent Ġsource Ġlanguage . ĠAll Ġof Ġthem Ġfollow Ġthe Ġtraditional Ġmulti - task Ġtraining Ġstrategies . ĠB IB REF 26 ĠD BL P : j ournals / cor r / abs - 18 02 - 06 003 Ġpropose Ġto Ġuse Ġcurriculum Ġlearning Ġto Ġimprove ĠST Ġperformance Ġon Ġsynt act ically Ġdistant Ġlanguage Ġpairs . ĠTo Ġeffectively Ġleverage Ġtranscript ions Ġin ĠST Ġdata , ĠB IB REF 18 ĠD BL P : conf / na acl / An ast as opoulos C 18 Ġaugment Ġthe Ġmulti - task Ġmodel Ġwhere Ġthe Ġtarget Ġdec oder Ġreceives Ġinformation Ġfrom Ġthe Ġsource Ġdec oder Ġand Ġthey Ġshow Ġimprovements Ġon Ġlow - resource Ġspeech Ġtranslation . ĠTheir Ġmodel Ġjust Ġconsumes ĠAS R Ġand ĠST Ġdata , Ġin Ġcontrast , Ġour Ġwork Ġsufficiently Ġutilizes Ġthe Ġlarge - scale ĠMT Ġdata Ġto Ġcapture Ġthe Ġrich Ġsemantic Ġknowledge . ĠB IB REF 30 ĠD BL P : conf / ic ass p / J ia J MW CC AL W 19 Ġuse Ġpre - trained ĠMT Ġand Ġtext - to - speech Ġ( T TS ) Ġsynthesis Ġmodels Ġto Ġconvert Ġweak ly Ġsupervised Ġdata Ġinto ĠST Ġpairs Ġand Ġdemonstrate Ġthat Ġan Ġend - to - end ĠMT Ġmodel Ġcan Ġbe Ġtrained Ġusing Ġonly Ġsynthes ised Ġdata . </s> Conclusion </s> This Ġpaper Ġhas Ġinvestigated Ġthe Ġend - to - end Ġmethod Ġfor ĠST . ĠIt Ġhas Ġdiscussed Ġwhy Ġthere Ġis Ġa Ġhuge Ġgap Ġbetween Ġpre - training Ġand Ġfine - tun ing Ġin Ġprevious Ġmethods . ĠTo Ġalleviate Ġthese Ġissues , Ġwe Ġhave Ġproposed Ġa Ġmethod , Ġwhich Ġis Ġcapable Ġof Ġre using Ġevery Ġsub - net Ġand Ġkeeping Ġthe Ġrole Ġof Ġsub - net Ġconsistent Ġbetween Ġpre - training Ġand Ġfine - tun ing . ĠEm pir ical Ġstudies Ġhave Ġdemonstrated Ġthat Ġour Ġmodel Ġsignificantly Ġoutper forms Ġbas elines .
Introduction Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1. To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest. Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory. Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT. A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model. Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST. This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds: Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder. Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system. Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning. The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty. Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training. To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module. Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder. Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function. In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder. Besides, the text encoder and decoder can be pre-trained on a large MT dataset. After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks. Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved. Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning. Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training. Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings. We further modify our model to achieve semantic consistency and length consistency. Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence. To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences. We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component. Our model is significantly better than previous methods by 3.6 and 2.2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively. Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset. Background ::: Problem Formulation End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step. The raw speech signals are usually converted to sequences of acoustic features, e.g. Mel filterbank features. Here, we define the speech feature sequence as $\mathbf {x} = (x_1, \cdots , x_{T_x})$.The transcription and translation sequences are denoted as $\mathbf {y^{s}} = (y_1^{s}, \cdots , y_{T_s}^{s})$, and $\mathbf {y^{t}} = (y_1^{t}, \cdots , y_{T_t}^{t})$ repectively. Each symbol in $\mathbf {y^{s}}$ or $\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e.g. $y^s_i=k, k\in [0, |V_{src}|-1]$). In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\mathcal {A} = \lbrace (\mathbf {x_i}, \mathbf {y^{s}_i})\rbrace _{i=0}^I$, $\mathcal {M} =\lbrace (\mathbf {y^{s}_j}, \mathbf {y^{t}_j})\rbrace _{j=0}^J$ and $ \mathcal {S} =\lbrace (\mathbf {x_l}, \mathbf {y^{t}_l})\rbrace _{l=0}^L$ respectively. Given a new piece of audio $\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\mathbf {y^{t}}$ without generating an intermediate result $\mathbf {y^{s}}$. Background ::: Multi-Task Learning and Pre-training for ST To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system. As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared. A many-to-many multi-task model contains two encoders as well as two decoders. It can be jointly trained on ASR, MT, and ST tasks. As the attention module is task-specific, three attentions are defined. Usually, the size of $\mathcal {A}$ and $\mathcal {M}$ is much larger than $\mathcal {S}$. Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner. However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance. Our method In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module. Then we give the pre-training and fine-tuning strategy for TCEN. Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning. Our method ::: TCEN Architecture Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$. During training, the $enc_s$ acts like an acoustic model which reads the input $\mathbf {x}$ to word or subword representations $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\mathbf {h^t}$. Finally, the $dec$ defines a distribution probability over target words. The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training. Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem. Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model. Specifically, the input features $\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$. Then, $\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling: where $\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\mathbf {W}^{(l)}$ is the filter. The max-pooling layers downsample the sequence in length by a total factor of four. The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers: where $[;]$ denotes the vector concatenation. The final output representation from the speech encoder is denoted as $\mathbf {h^s}=(h^s_1, \cdots , h^s_{\frac{T_x}{4}})$, where $h_i^s \in \mathbb {R}^d$. The text encoder $enc_t$ consists of two bidirectional LSTM layers. In ST task, $enc_t$ accepts speech encoder output $\mathbf {h}^s$ as input. While in MT, $enc_t$ consumes the word embedding representation $\mathbf {e^s}$ derived from $\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$. The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\mathbf {h}^s$ or $\mathbf {e}^s$. Since $\mathbf {h}^s$ and $\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems. We will provide our solutions in Section SECREF21. The output sequence of $enc_t$ is denoted as $\mathbf {h}^t$. The decoder is defined as two unidirectional LSTM layers with an additive attention $att$. It predicts target sequence $\mathbf {y^{t}}$ by estimating conditional probability $P(\mathbf {y^{t}}|\mathbf {x})$: Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$. Our method ::: Training Procedure Following previous work, we split the training procedure to pre-training and fine-tuning stages. In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\mathcal {M}$. In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks. Our method ::: Training Procedure ::: Pre-training To sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage. For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder. Given an input $\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\mathbf {\pi }$, where $\pi _t \in V_{src} \cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels: where $W_{ctc} \in \mathbb {R}^{d \times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN. A legal CTC path $\mathbf {\pi }$ is a variation of the source transcription $\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14. For each transcription $\mathbf {y}$, there exist many legal CTC paths in length $T$. The CTC objective trains the model to maximize the probability of observing the golden sequence $\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths: where $\Phi _T(y)$ is the set of all legal CTC paths for sequence $\mathbf {y}$ with length $T$. The loss can be easily computed using forward-backward algorithm. More details about CTC are provided in supplementary material. For MT task, we use the cross-entropy loss as the training objective. During training, $\mathbf {y^s}$ is converted to embedding vectors $\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\mathbf {e^s}$ and pass the output $\mathbf {h^t}$ to decoder. The objective function is defined as: Our method ::: Training Procedure ::: Fine-tune In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks. The training for ASR and MT follows the same process as it was in pre-training stage. For ST task, the $enc_s$ reads the input $\mathbf {x}$ and generates $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\mathbf {h^t}$. Finally, the $dec$ predicts the target sentence. The ST loss function is defined as: Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\alpha _i$ for each task. When switching between tasks, we select randomly a new task $i$ with probability $\frac{\alpha _i}{\sum _{j}\alpha _{j}}$. Our method ::: Subnet-Consistency Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task. However, this leads to some new problems: 1) The text encoder consumes $\mathbf {e^s}$ during MT training, while it accepts $\mathbf {h^s}$ during ST training. However, $\mathbf {e^s}$ and $\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency. 2) Besides, the length of $\mathbf {h^s}$ is not the same order of magnitude with the length of $\mathbf {e^s}$, resulting in the length inconsistency. In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\mathbf {e^s}$ and $\mathbf {h^s}$ in the same space. 2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs. Our method ::: Subnet-Consistency ::: Semantic Consistency As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\mathbf {e}^s$ in MT task, and the $enc_s$ output $\mathbf {h^s}$ in ST task. Without any regularization, they may belong to different latent spaces. Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks. To bridge the space gap, our idea is to pull $\mathbf {h^s}$ into the latent space where $\mathbf {e}^s$ belong. Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$. In this way, when predicting the CTC path $\mathbf {\pi }$, the probability of observing the particular label $w_i \in V_{src}\cup ${`-'} at time step $t$, $p(\pi _t=w_i|\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$: The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\mathbf {h}^s$ have the same distribution with $\mathbf {e}^s$. Our method ::: Subnet-Consistency ::: Length Consistency Another existing problem is length inconsistency. The length of the sequence $\mathbf {h^s}$ is proportional to the length of the input frame $\mathbf {x}$, which is much longer than the length of $\mathbf {e^s}$. To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking. Specifically, we first train a CTC ASR model based on dataset $\mathcal {A} = \lbrace (\mathbf {x}_i, \mathbf {y}^s_i)\rbrace _{i=0}^{I}$, and generate a CTC-path $\mathbf {\pi }_i$ for each audio $\mathbf {x}_i$ by greedy decoding. Then we define an operation $S(\cdot )$, which converts a CTC path $\mathbf {\pi }$ to a sequence of the unique tokens $\mathbf {u}$ and a sequence of repetition times for each token $\mathbf {l}$, denoted as $S(\mathbf {\pi }) = (\mathbf {u}, \mathbf {l})$. Notably, the operation is reversible, meaning that $S^{-1} (\mathbf {u}, \mathbf {l})=\mathbf {\pi }$. We use the example $\mathbf {\pi _1}$ in Table TABREF14 and show the corresponding $\mathbf {u}$ and $\mathbf {l}$ in Table TABREF24. Then we build a dataset $\mathcal {P} = \lbrace (\mathbf {y^s}_i, \mathbf {u}_i, \mathbf {l}_i)\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\mathcal {A}$ and transform the resulting path by the operation $S(\cdot )$. After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \mathbf {y^s}_i$ as input and decodes $\mathbf {u}_i, \mathbf {l}_i$ as outputs. With the seq2seq model, a noisy MT dataset $\mathcal {M}^{\prime }=\lbrace (\mathbf {\pi }_l, \mathbf {y^t}_l)\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\mathbf {y^s}_i \in \mathcal {M}$ to $\mathbf {\pi _i}$, where $\mathbf {\pi }_i = S^{-1}(\mathbf {u}_i, \mathbf {l}_i)$. We did not use the standard seq2seq model which takes $\mathbf {y^s}$ as input and generates $\mathbf {\pi }$ directly, since there are too many blank tokens `-' in $\mathbf {\pi }$ and the model tends to generate a long sequence with only blank tokens. During MT training, we randomly sample text pairs from $\mathcal {M}^{\prime }$ and $\mathcal {M}$ according to a hyper-parameter $k$. After tuning on the validation set, about $30\%$ pairs are sampled from $\mathcal {M}^{\prime }$. In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$. Experiments We conduct experiments on the IWSLT18 speech translation task BIBREF1. Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison. Experiments ::: Dataset ::: Speech translation data: The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations. The corpus contains 272 hours of English speech with 171k segments. We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets. Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences. Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively. WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs. Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size. Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features. We normalize them by the mean and the standard deviation on the whole training set. The utterances with more than 3000 frames are discarded. The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated. Thus, we lowercase all the sentences and remove the punctuation to keep consistent. To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0.9 and 1.1. For the text translation data, sentences longer than 80 words or shorter than 10 words are removed. Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0.5 or larger than 2.0. Word tokenization is performed using the Moses scripts and both English and German words are in lower-case. We use two different sets of vocabulary for our experiments. For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens. BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task. For the character experiments, both English and German sentences are represented in the character level. For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17. We use lowercase BLEU as evaluation metric. Experiments ::: Baseline Models and Implementation We compare our method with following baselines. Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus. Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data. Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data. Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation. Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model. Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder. For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model. The Triangle model is fine-tuned under their multi-task manner. All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models. We use a dropout of 0.3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1.0 for a maximum of 10 epochs. For training of TCEN, we set $\alpha _{asr}=0.2$ and $\alpha _{mt}=0.8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset. For fine-tune, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$, same as the `many-to-many' baseline. For testing, we select the model with the best accuracy on speech translation task on dev set. At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0.2. Experiments ::: Experimental Results Table TABREF29 shows the results on four test sets as well as the average performance. Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3.6 and 2.2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality. Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows). We observe a performance degradation in the `triangle+pretrain' baseline. Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information. Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages. Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline. A similar phenomenon is observed by BIBREF6 berard2018end. A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult. Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality. Experiments ::: Discussion ::: Ablation Study To better understand the contribution of each component, we perform an ablation study on subword-level experiments. The results are shown in Table TABREF37. In `-MT noise' setting, we do not add noise to source sentences for MT. In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer. These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality. Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model. In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST. Experiments ::: Discussion ::: Learning Curve It is interesting to investigate why our method is superior to baselines. We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning. Figure FIGREF39 provides learning curves of subword accuracy on validation set. The x-axis denotes the fine-tuning training steps. The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data. The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process. It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process. Experiments ::: Discussion ::: Compared with a Cascaded System Table TABREF29 compares our model with end-to-end baselines. Here, we compare our model with cascaded systems. We build a cascaded system by combining the ASR model and MT model used in pre-training baseline. Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material. In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg. Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence. After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs. Experimental results are shown in Table TABREF41. Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model. Related Work Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences. HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT. To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference. To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4. Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data. Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model. BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset. BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language. All of them follow the traditional multi-task training strategies. BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs. To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation. Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge. BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data. Conclusion This paper has investigated the end-to-end method for ST. It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods. To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning. Empirical studies have demonstrated that our model significantly outperforms baselines.
What are the baselines?
af34051bf3e628c1e2a00b110bb84e5f018b419f
What Ġare Ġthe Ġbas elines ?
Van illa ĠST Ġbaseline , Ġenc oder Ġpre - training , Ġin Ġwhich Ġthe ĠST Ġenc oder Ġis Ġinitialized Ġfrom Ġan ĠAS R Ġmodel , Ġdec oder Ġpre - training , Ġin Ġwhich Ġthe ĠST Ġdec oder Ġis Ġinitialized Ġfrom Ġan ĠMT Ġmodel , Ġenc oder - dec oder Ġpre - training , Ġwhere Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġpre - trained , Ġmany - to - many Ġmulti - task Ġmodel Ġwhere Ġthe Ġenc od ers Ġand Ġdec od ers Ġare Ġderived Ġfrom Ġpre - trained ĠAS R Ġand ĠMT Ġmodels , ĠTriangle + pre - train : ĠB IB REF 18 ĠD BL P : conf / na acl / An ast as opoulos C 18 Ġproposed Ġa Ġtriangle Ġmulti - task Ġstrategy Ġfor Ġspeech Ġtranslation
<s> What Ġare Ġthe Ġbas elines ? </s> Introduction </s> Spe ech - to - Text Ġtranslation Ġ( ST ) Ġis Ġessential Ġfor Ġa Ġwide Ġrange Ġof Ġscenarios : Ġfor Ġexample Ġin Ġemergency Ġcalls , Ġwhere Ġagents Ġhave Ġto Ġrespond Ġemer gent Ġrequests Ġin Ġa Ġforeign Ġlanguage ĠB IB REF 0 ; Ġor Ġin Ġonline Ġcourses , Ġwhere Ġaudiences Ġand Ġspeakers Ġuse Ġdifferent Ġlanguages ĠB IB REF 1 . ĠTo Ġtackle Ġthis Ġproblem , Ġexisting Ġapproaches Ġcan Ġbe Ġcategorized Ġinto Ġcasc aded Ġmethod ĠB IB REF 2 , ĠB IB REF 3 , Ġwhere Ġa Ġmachine Ġtranslation Ġ( MT ) Ġmodel Ġtranslates Ġoutputs Ġof Ġan Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystem Ġinto Ġtarget Ġlanguage , Ġand Ġend - to - end Ġmethod ĠB IB REF 4 , ĠB IB REF 5 , Ġwhere Ġa Ġsingle Ġmodel Ġlearns Ġacoustic Ġframes Ġto Ġtarget Ġword Ġsequence Ġm appings Ġin Ġone Ġstep Ġtowards Ġthe Ġfinal Ġobjective Ġof Ġinterest . ĠAlthough Ġthe Ġcasc aded Ġmodel Ġremains Ġthe Ġdominant Ġapproach Ġdue Ġto Ġits Ġbetter Ġperformance , Ġthe Ġend - to - end Ġmethod Ġbecomes Ġmore Ġand Ġmore Ġpopular Ġbecause Ġit Ġhas Ġlower Ġlatency Ġby Ġavoiding Ġinf erences Ġwith Ġtwo Ġmodels Ġand Ġrect ifies Ġthe Ġerror Ġpropagation Ġin Ġtheory . </s> Since Ġit Ġis Ġhard Ġto Ġobtain Ġa Ġlarge - scale ĠST Ġdataset , Ġmulti - task Ġlearning ĠB IB REF 5 , ĠB IB REF 6 Ġand Ġpre - training Ġtechniques ĠB IB REF 7 Ġhave Ġbeen Ġapplied Ġto Ġend - to - end ĠST Ġmodel Ġto Ġleverage Ġlarge - scale Ġdatasets Ġof ĠAS R Ġand ĠMT . ĠA Ġcommon Ġpractice Ġis Ġto Ġpre - train Ġtwo Ġenc oder - dec oder Ġmodels Ġfor ĠAS R Ġand ĠMT Ġrespectively , Ġand Ġthen Ġinitialize Ġthe ĠST Ġmodel Ġwith Ġthe Ġenc oder Ġof Ġthe ĠAS R Ġmodel Ġand Ġthe Ġdec oder Ġof Ġthe ĠMT Ġmodel . ĠSub sequently , Ġthe ĠST Ġmodel Ġis Ġoptimized Ġwith Ġthe Ġmulti - task Ġlearning Ġby Ġweighing Ġthe Ġlosses Ġof ĠAS R , ĠMT , Ġand ĠST . ĠThis Ġapproach , Ġhowever , Ġcauses Ġa Ġhuge Ġgap Ġbetween Ġpre - training Ġand Ġfine - tun ing , Ġwhich Ġare Ġsummarized Ġinto Ġthree Ġfolds : </s> Sub net ĠWaste : ĠThe ĠST Ġsystem Ġjust Ġre uses Ġthe ĠAS R Ġenc oder Ġand Ġthe ĠMT Ġdec oder , Ġwhile Ġdisc ards Ġother Ġpre - trained Ġsub nets , Ġsuch Ġas Ġthe ĠMT Ġenc oder . ĠConsequently , Ġvaluable Ġsemantic Ġinformation Ġcaptured Ġby Ġthe ĠMT Ġenc oder Ġcannot Ġbe Ġinherited Ġby Ġthe Ġfinal ĠST Ġsystem . </s> Role ĠM ism atch : ĠThe Ġspeech Ġenc oder Ġplays Ġdifferent Ġroles Ġin Ġpre - training Ġand Ġfine - tun ing . ĠThe Ġenc oder Ġis Ġa Ġpure Ġacoustic Ġmodel Ġin Ġpre - training , Ġwhile Ġit Ġhas Ġto Ġextract Ġsemantic Ġand Ġlinguistic Ġfeatures Ġadditionally Ġin Ġfine - tun ing , Ġwhich Ġsignificantly Ġincreases Ġthe Ġlearning Ġdifficulty . </s> Non - pre - trained ĠAttention ĠModule : ĠPrevious Ġwork ĠB IB REF 6 Ġtrains Ġattention Ġmodules Ġfor ĠAS R , ĠMT Ġand ĠST Ġrespectively , Ġhence , Ġthe Ġattention Ġmodule Ġof ĠST Ġdoes Ġnot Ġbenefit Ġfrom Ġthe Ġpre - training . </s> To Ġaddress Ġthese Ġissues , Ġwe Ġpropose Ġa ĠT andem ĠConnection ist ĠEnc oding ĠNetwork Ġ( TC EN ), Ġwhich Ġis Ġable Ġto Ġreuse Ġall Ġsub nets Ġin Ġpre - training , Ġkeep Ġthe Ġroles Ġof Ġsub nets Ġconsistent , Ġand Ġpre - train Ġthe Ġattention Ġmodule . ĠCon crete ly , Ġthe ĠTC EN Ġconsists Ġof Ġthree Ġcomponents , Ġa Ġspeech Ġenc oder , Ġa Ġtext Ġenc oder , Ġand Ġa Ġtarget Ġtext Ġdec oder . ĠDifferent Ġfrom Ġthe Ġprevious Ġwork Ġthat Ġpre - tr ains Ġan Ġenc oder - dec oder Ġbased ĠAS R Ġmodel , Ġwe Ġonly Ġpre - train Ġan ĠAS R Ġenc oder Ġby Ġoptimizing Ġthe ĠConnection ist ĠTem poral ĠClassification Ġ( CT C ) ĠB IB REF 8 Ġobjective Ġfunction . ĠIn Ġthis Ġway , Ġthe Ġadditional Ġdec oder Ġof ĠAS R Ġis Ġnot Ġrequired Ġwhile Ġkeeping Ġthe Ġability Ġto Ġread Ġacoustic Ġfeatures Ġinto Ġthe Ġsource Ġlanguage Ġspace Ġby Ġthe Ġspeech Ġenc oder . ĠBesides , Ġthe Ġtext Ġenc oder Ġand Ġdec oder Ġcan Ġbe Ġpre - trained Ġon Ġa Ġlarge ĠMT Ġdataset . ĠAfter Ġthat , Ġwe Ġemploy Ġcommon Ġused Ġmulti - task Ġlearning Ġmethod Ġto Ġjointly Ġlearn ĠAS R , ĠMT Ġand ĠST Ġtasks . </s> Compared Ġto Ġprior Ġworks , Ġthe Ġenc oder Ġof ĠTC EN Ġis Ġa Ġconc aten ation Ġof Ġan ĠAS R Ġenc oder Ġand Ġan ĠMT Ġenc oder Ġand Ġour Ġmodel Ġdoes Ġnot Ġhave Ġan ĠAS R Ġdec oder , Ġso Ġthe Ġsub net Ġwaste Ġissue Ġis Ġsolved . ĠFurthermore , Ġthe Ġtwo Ġenc od ers Ġwork Ġat Ġtandem , Ġdis ent angling Ġacoustic Ġfeature Ġextraction Ġand Ġlinguistic Ġfeature Ġextraction , Ġensuring Ġthe Ġrole Ġconsistency Ġbetween Ġpre - training Ġand Ġfine - tun ing . ĠMoreover , Ġwe Ġreuse Ġthe Ġpre - trained ĠMT Ġattention Ġmodule Ġin ĠST , Ġso Ġwe Ġcan Ġleverage Ġthe Ġalignment Ġinformation Ġlearned Ġin Ġpre - training . </s> Since Ġthe Ġtext Ġenc oder Ġconsumes Ġword Ġembed d ings Ġof Ġplausible Ġtexts Ġin ĠMT Ġtask Ġbut Ġuses Ġspeech Ġenc oder Ġoutputs Ġin ĠST Ġtask , Ġanother Ġquestion Ġis Ġhow Ġone Ġguarantees Ġthe Ġspeech Ġenc oder Ġoutputs Ġare Ġconsistent Ġwith Ġthe Ġword Ġembed d ings . ĠWe Ġfurther Ġmodify Ġour Ġmodel Ġto Ġachieve Ġsemantic Ġconsistency Ġand Ġlength Ġconsistency . ĠSpecifically , Ġ( 1 ) Ġthe Ġprojection Ġmatrix Ġat Ġthe ĠC TC Ġclassification Ġlayer Ġfor ĠAS R Ġis Ġshared Ġwith Ġthe Ġword Ġembed ding Ġmatrix , Ġensuring Ġthat Ġthey Ġare Ġmapped Ġto Ġthe Ġsame Ġlatent Ġspace , Ġand Ġ( 2 ) Ġthe Ġlength Ġof Ġthe Ġspeech Ġenc oder Ġoutput Ġis Ġproportional Ġto Ġthe Ġlength Ġof Ġthe Ġinput Ġframe , Ġso Ġit Ġis Ġmuch Ġlonger Ġthan Ġa Ġnatural Ġsentence . ĠTo Ġbridge Ġthe Ġlength Ġgap , Ġsource Ġsentences Ġin ĠMT Ġare Ġleng then ed Ġby Ġadding Ġword Ġrepet itions Ġand Ġblank Ġtokens Ġto Ġmimic Ġthe ĠC TC Ġoutput Ġsequences . </s> We Ġconduct Ġcomprehensive Ġexperiments Ġon Ġthe ĠI W SL T 18 Ġspeech Ġtranslation Ġbenchmark ĠB IB REF 1 , Ġdemonstrating Ġthe Ġeffectiveness Ġof Ġeach Ġcomponent . ĠOur Ġmodel Ġis Ġsignificantly Ġbetter Ġthan Ġprevious Ġmethods Ġby Ġ3 . 6 Ġand Ġ2 . 2 ĠB LE U Ġscores Ġfor Ġthe Ġsub word - level Ġdecoding Ġand Ġcharacter - level Ġdecoding Ġstrategies , Ġrespectively . </s> Our Ġcontributions Ġare Ġthree - f olds : Ġ1 ) Ġwe Ġshed Ġlight Ġon Ġwhy Ġprevious ĠST Ġmodels Ġcannot Ġsufficiently Ġutilize Ġthe Ġknowledge Ġlearned Ġfrom Ġthe Ġpre - training Ġprocess ; Ġ2 ) Ġwe Ġpropose Ġa Ġnew ĠST Ġmodel , Ġwhich Ġallev iates Ġshortcomings Ġin Ġexisting Ġmethods ; Ġand Ġ3 ) Ġwe Ġempir ically Ġevaluate Ġthe Ġproposed Ġmodel Ġon Ġa Ġlarge - scale Ġpublic Ġdataset . </s> Background Ġ: :: ĠProblem ĠForm ulation </s> End - to - end Ġspeech Ġtranslation Ġaims Ġto Ġtranslate Ġa Ġpiece Ġof Ġaudio Ġinto Ġa Ġtarget - language Ġtranslation Ġin Ġone Ġstep . ĠThe Ġraw Ġspeech Ġsignals Ġare Ġusually Ġconverted Ġto Ġsequences Ġof Ġacoustic Ġfeatures , Ġe . g . ĠMel Ġfilter bank Ġfeatures . ĠHere , Ġwe Ġdefine Ġthe Ġspeech Ġfeature Ġsequence Ġas Ġ$\ math bf Ġ{ x } Ġ= Ġ( x _ 1 , Ġ\ cd ots Ġ, Ġx _{ T _ x }) $. The Ġtranscription Ġand Ġtranslation Ġsequences Ġare Ġden oted Ġas Ġ$\ math bf Ġ{ y ^{ s }} Ġ= Ġ( y _ 1 ^{ s }, Ġ\ cd ots Ġ, Ġy _{ T _ s } ^{ s }) $, Ġand Ġ$\ math bf Ġ{ y ^{ t }} Ġ= Ġ( y _ 1 ^{ t }, Ġ\ cd ots Ġ, Ġy _{ T _ t } ^{ t }) $ Ġre pect ively . ĠEach Ġsymbol Ġin Ġ$\ math bf Ġ{ y ^{ s }} $ Ġor Ġ$\ math bf Ġ{ y ^{ t }} $ Ġis Ġan Ġinteger Ġindex Ġof Ġthe Ġsymbol Ġin Ġa Ġvocabulary Ġ$ V _{ src } $ Ġor Ġ$ V _{ tr g } $ Ġrespectively Ġ( e . g . Ġ$ y ^ s _ i = k , Ġk \ in Ġ[ 0 , Ġ| V _{ src } | - 1 ] $ ). ĠIn Ġthis Ġwork , Ġwe Ġsuppose Ġthat Ġan ĠAS R Ġdataset , Ġan ĠMT Ġdataset , Ġand Ġa ĠST Ġdataset Ġare Ġavailable , Ġden oted Ġas Ġ$\ math cal Ġ{ A } Ġ= Ġ\ l brace Ġ( \ math bf Ġ{ x _ i }, Ġ\ math bf Ġ{ y ^{ s } _ i } )\ r brace Ġ_ { i = 0 } ^ I $, Ġ$\ math cal Ġ{ M } Ġ= \ l brace Ġ( \ math bf Ġ{ y ^{ s } _ j }, Ġ\ math bf Ġ{ y ^{ t } _ j } )\ r brace Ġ_ { j = 0 } ^ J $ Ġand Ġ$ Ġ\ math cal Ġ{ S } Ġ= \ l brace Ġ( \ math bf Ġ{ x _ l }, Ġ\ math bf Ġ{ y ^{ t } _ l } )\ r brace Ġ_ { l = 0 } ^ L $ Ġrespectively . ĠGiven Ġa Ġnew Ġpiece Ġof Ġaudio Ġ$\ math bf Ġ{ x } $, Ġour Ġgoal Ġis Ġto Ġlearn Ġan Ġend Ġto Ġend Ġmodel Ġto Ġgenerate Ġa Ġtranslation Ġsentence Ġ$\ math bf Ġ{ y ^{ t }} $ Ġwithout Ġgenerating Ġan Ġintermediate Ġresult Ġ$\ math bf Ġ{ y ^{ s }} $. </s> Background Ġ: :: ĠMulti - Task ĠLearning Ġand ĠPre - training Ġfor ĠST </s> To Ġleverage Ġlarge Ġscale ĠAS R Ġand ĠMT Ġdata , Ġmulti - task Ġlearning Ġand Ġpre - training Ġtechniques Ġare Ġwidely Ġemployed Ġto Ġimprove Ġthe ĠST Ġsystem . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 4 , Ġthere Ġare Ġthree Ġpopular Ġmulti - task Ġstrategies Ġfor ĠST , Ġincluding Ġ1 ) Ġone - to - many Ġsetting , Ġin Ġwhich Ġa Ġspeech Ġenc oder Ġis Ġshared Ġbetween ĠAS R Ġand ĠST Ġtasks ; Ġ2 ) Ġmany - to - one Ġsetting Ġin Ġwhich Ġa Ġdec oder Ġis Ġshared Ġbetween ĠMT Ġand ĠST Ġtasks ; Ġand Ġ3 ) Ġmany - to - many Ġsetting Ġwhere Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġshared . </s> A Ġmany - to - many Ġmulti - task Ġmodel Ġcontains Ġtwo Ġenc od ers Ġas Ġwell Ġas Ġtwo Ġdec od ers . ĠIt Ġcan Ġbe Ġjointly Ġtrained Ġon ĠAS R , ĠMT , Ġand ĠST Ġtasks . ĠAs Ġthe Ġattention Ġmodule Ġis Ġtask - specific , Ġthree Ġattent ions Ġare Ġdefined . </s> Usually , Ġthe Ġsize Ġof Ġ$\ math cal Ġ{ A } $ Ġand Ġ$\ math cal Ġ{ M } $ Ġis Ġmuch Ġlarger Ġthan Ġ$\ math cal Ġ{ S } $. ĠTherefore , Ġthe Ġcommon Ġtraining Ġpractice Ġis Ġto Ġpre - train Ġthe Ġmodel Ġon ĠAS R Ġand ĠMT Ġtasks Ġand Ġthen Ġfine - t une Ġit Ġwith Ġa Ġmulti - task Ġlearning Ġmanner . ĠHowever , Ġas Ġaforementioned , Ġthis Ġmethod Ġsuffers Ġfrom Ġsub net Ġwaste , Ġrole Ġmismatch Ġand Ġnon - pre - trained Ġattention Ġissues , Ġwhich Ġseverely Ġlimits Ġthe Ġend - to - end ĠST Ġperformance . </s> Our Ġmethod </s> In Ġthis Ġsection , Ġwe Ġfirst Ġintroduce Ġthe Ġarchitecture Ġof ĠTC EN , Ġwhich Ġconsists Ġof Ġtwo Ġenc od ers Ġconnected Ġin Ġtandem , Ġand Ġone Ġdec oder Ġwith Ġan Ġattention Ġmodule . ĠThen Ġwe Ġgive Ġthe Ġpre - training Ġand Ġfine - tun ing Ġstrategy Ġfor ĠTC EN . ĠFinally , Ġwe Ġpropose Ġour Ġsolutions Ġfor Ġsemantic Ġand Ġlength Ġinconsistency Ġproblems , Ġwhich Ġare Ġcaused Ġby Ġmulti - task Ġlearning . </s> Our Ġmethod Ġ: :: ĠTC EN ĠArchitecture </s> Figure ĠFIG REF 5 Ġsketches Ġthe Ġoverall Ġarchitecture Ġof ĠTC EN , Ġincluding Ġa Ġspeech Ġenc oder Ġ$ enc _ s $, Ġa Ġtext Ġenc oder Ġ$ enc _ t $ Ġand Ġa Ġdec oder Ġ$ dec $ Ġwith Ġan Ġattention Ġmodule Ġ$ att $. ĠDuring Ġtraining , Ġthe Ġ$ enc _ s $ Ġacts Ġlike Ġan Ġacoustic Ġmodel Ġwhich Ġreads Ġthe Ġinput Ġ$\ math bf Ġ{ x } $ Ġto Ġword Ġor Ġsub word Ġrepresentations Ġ$\ math bf Ġ{ h ^ s } $, Ġthen Ġ$ enc _ t $ Ġlearns Ġhigh - level Ġlinguistic Ġknowledge Ġinto Ġhidden Ġrepresentations Ġ$\ math bf Ġ{ h ^ t } $. ĠFinally , Ġthe Ġ$ dec $ Ġdefines Ġa Ġdistribution Ġprobability Ġover Ġtarget Ġwords . ĠThe Ġadvantage Ġof Ġour Ġarchitecture Ġis Ġthat Ġtwo Ġenc od ers Ġdis ent angle Ġacoustic Ġfeature Ġextraction Ġand Ġlinguistic Ġfeature Ġextraction , Ġmaking Ġsure Ġthat Ġvaluable Ġknowledge Ġlearned Ġfrom ĠAS R Ġand ĠMT Ġtasks Ġcan Ġbe Ġeffectively Ġlever aged Ġfor ĠST Ġtraining . ĠBesides , Ġevery Ġmodule Ġin Ġpre - training Ġcan Ġbe Ġutilized Ġin Ġfine - tun ing , Ġallev iating Ġthe Ġsub net Ġwaste Ġproblem . </s> Follow ĠB IB REF 9 Ġin ag uma 2018 speech , Ġwe Ġuse ĠCNN - Bi L ST M Ġarchitecture Ġto Ġbuild Ġour Ġmodel . ĠSpecifically , Ġthe Ġinput Ġfeatures Ġ$\ math bf Ġ{ x } $ Ġare Ġorganized Ġas Ġa Ġsequence Ġof Ġfeature Ġvectors Ġin Ġlength Ġ$ T _ x $. ĠThen , Ġ$\ math bf Ġ{ x } $ Ġis Ġpassed Ġinto Ġa Ġstack Ġof Ġtwo Ġconv olution al Ġlayers Ġfollowed Ġby Ġmax - pool ing : </s> where Ġ$\ math bf Ġ{ v } ^{ ( l - 1 )} $ Ġis Ġfeature Ġmaps Ġin Ġlast Ġlayer Ġand Ġ$\ math bf Ġ{ W } ^{ ( l )} $ Ġis Ġthe Ġfilter . ĠThe Ġmax - pool ing Ġlayers Ġdowns ample Ġthe Ġsequence Ġin Ġlength Ġby Ġa Ġtotal Ġfactor Ġof Ġfour . ĠThe Ġdown - sam pled Ġfeature Ġsequence Ġis Ġfurther Ġfed Ġinto Ġa Ġstack Ġof Ġfive Ġbid irection al Ġ$ d $ - dimensional ĠL ST M Ġlayers : </s> where Ġ$ [ ; ] $ Ġdenotes Ġthe Ġvector Ġconc aten ation . ĠThe Ġfinal Ġoutput Ġrepresentation Ġfrom Ġthe Ġspeech Ġenc oder Ġis Ġden oted Ġas Ġ$\ math bf Ġ{ h ^ s } =( h ^ s _ 1 , Ġ\ cd ots Ġ, Ġh ^ s _{ \ frac { T _ x }{ 4 }} ) $, Ġwhere Ġ$ h _ i ^ s Ġ\ in Ġ\ math bb Ġ{ R } ^ d $. </s> The Ġtext Ġenc oder Ġ$ enc _ t $ Ġconsists Ġof Ġtwo Ġbid irection al ĠL ST M Ġlayers . ĠIn ĠST Ġtask , Ġ$ enc _ t $ Ġaccepts Ġspeech Ġenc oder Ġoutput Ġ$\ math bf Ġ{ h } ^ s $ Ġas Ġinput . ĠWhile Ġin ĠMT , Ġ$ enc _ t $ Ġconsumes Ġthe Ġword Ġembed ding Ġrepresentation Ġ$\ math bf Ġ{ e ^ s } $ Ġderived Ġfrom Ġ$\ math bf Ġ{ y ^ s } $, Ġwhere Ġeach Ġelement Ġ$ e ^ s _ i $ Ġis Ġcomputed Ġby Ġchoosing Ġthe Ġ$ y _ i ^ s $ - th Ġvector Ġfrom Ġthe Ġsource Ġembed ding Ġmatrix Ġ$ W _{ E ^ s } $. ĠThe Ġgoal Ġof Ġ$ enc _ t $ Ġis Ġto Ġextract Ġhigh - level Ġlinguistic Ġfeatures Ġlike Ġsynt actic Ġfeatures Ġor Ġsemantic Ġfeatures Ġfrom Ġlower Ġlevel Ġsub word Ġrepresentations Ġ$\ math bf Ġ{ h } ^ s $ Ġor Ġ$\ math bf Ġ{ e } ^ s $. ĠSince Ġ$\ math bf Ġ{ h } ^ s $ Ġand Ġ$\ math bf Ġ{ e } ^ s $ Ġbelong Ġto Ġdifferent Ġlatent Ġspace Ġand Ġhave Ġdifferent Ġlengths , Ġthere Ġremain Ġsemantic Ġand Ġlength Ġinconsistency Ġproblems . ĠWe Ġwill Ġprovide Ġour Ġsolutions Ġin ĠSection ĠSEC REF 21 . ĠThe Ġoutput Ġsequence Ġof Ġ$ enc _ t $ Ġis Ġden oted Ġas Ġ$\ math bf Ġ{ h } ^ t $. </s> The Ġdec oder Ġis Ġdefined Ġas Ġtwo Ġun id irection al ĠL ST M Ġlayers Ġwith Ġan Ġadditive Ġattention Ġ$ att $. ĠIt Ġpredicts Ġtarget Ġsequence Ġ$\ math bf Ġ{ y ^{ t }} $ Ġby Ġestimating Ġconditional Ġprobability Ġ$ P (\ math bf Ġ{ y ^{ t }} | \ math bf Ġ{ x }) $ : </s> Here , Ġ$ z _ k $ Ġis Ġthe Ġthe Ġhidden Ġstate Ġof Ġthe Ġde oc der ĠR NN Ġat Ġ$ k $ Ġstep Ġand Ġ$ c _ k $ Ġis Ġa Ġtime - dependent Ġcontext Ġvector Ġcomputed Ġby Ġthe Ġattention Ġ$ att $. </s> Our Ġmethod Ġ: :: ĠTraining ĠProcedure </s> Following Ġprevious Ġwork , Ġwe Ġsplit Ġthe Ġtraining Ġprocedure Ġto Ġpre - training Ġand Ġfine - tun ing Ġstages . ĠIn Ġpre - training Ġstage , Ġthe Ġspeech Ġenc oder Ġ$ enc _ s $ Ġis Ġtrained Ġtowards ĠC TC Ġobjective Ġusing Ġdataset Ġ$\ math cal Ġ{ A } $, Ġwhile Ġthe Ġtext Ġenc oder Ġ$ enc _ t $ Ġand Ġthe Ġdec oder Ġ$ dec $ Ġare Ġtrained Ġon ĠMT Ġdataset Ġ$\ math cal Ġ{ M } $. ĠIn Ġfine - tun ing Ġstage , Ġwe Ġjointly Ġtrain Ġthe Ġmodel Ġon ĠAS R , ĠMT , Ġand ĠST Ġtasks . </s> Our Ġmethod Ġ: :: ĠTraining ĠProcedure Ġ: :: ĠPre - training </s> To Ġsufficiently Ġutilize Ġthe Ġlarge Ġdataset Ġ$\ math cal Ġ{ A } $ Ġand Ġ$\ math cal Ġ{ M } $, Ġthe Ġmodel Ġis Ġpre - trained Ġon ĠC TC - based ĠAS R Ġtask Ġand ĠMT Ġtask Ġin Ġthe Ġpre - training Ġstage . </s> For ĠAS R Ġtask , Ġin Ġorder Ġto Ġget Ġrid Ġof Ġthe Ġrequirement Ġfor Ġdec oder Ġand Ġenable Ġthe Ġ$ enc _ s $ Ġto Ġgenerate Ġsub word Ġrepresentation , Ġwe Ġleverage Ġconnection ist Ġtemporal Ġclassification Ġ( CT C ) ĠB IB REF 8 Ġloss Ġto Ġtrain Ġthe Ġspeech Ġenc oder . </s> Given Ġan Ġinput Ġ$\ math bf Ġ{ x } $, Ġ$ enc _ s $ Ġemits Ġa Ġsequence Ġof Ġhidden Ġvectors Ġ$\ math bf Ġ{ h ^ s } $, Ġthen Ġa Ġsoft max Ġclassification Ġlayer Ġpredicts Ġa ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $, Ġwhere Ġ$\ pi Ġ_ t Ġ\ in ĠV _{ src } Ġ\ cup Ġ$ Ġ{ ` -' } Ġis Ġthe Ġobserving Ġlabel Ġat Ġparticular ĠR NN Ġstep Ġ$ t $, Ġand Ġ` -' Ġis Ġthe Ġblank Ġtoken Ġrepresenting Ġno Ġobserved Ġlabels : </s> where Ġ$ W _{ ct c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġ( | V _{ src } | + 1 )} $ Ġis Ġthe Ġweight Ġmatrix Ġin Ġthe Ġclassification Ġlayer Ġand Ġ$ T $ Ġis Ġthe Ġtotal Ġlength Ġof Ġenc oder ĠR NN . </s> A Ġlegal ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $ Ġis Ġa Ġvariation Ġof Ġthe Ġsource Ġtranscription Ġ$\ math bf Ġ{ y } ^ s $ Ġby Ġallowing Ġoccurrences Ġof Ġblank Ġtokens Ġand Ġrepet itions , Ġas Ġshown Ġin ĠTable ĠT AB REF 14 . ĠFor Ġeach Ġtranscription Ġ$\ math bf Ġ{ y } $, Ġthere Ġexist Ġmany Ġlegal ĠC TC Ġpaths Ġin Ġlength Ġ$ T $. ĠThe ĠC TC Ġobjective Ġtrains Ġthe Ġmodel Ġto Ġmaximize Ġthe Ġprobability Ġof Ġobserving Ġthe Ġgolden Ġsequence Ġ$\ math bf Ġ{ y } ^ s $, Ġwhich Ġis Ġcalculated Ġby Ġsum ming Ġthe Ġprobabilities Ġof Ġall Ġpossible Ġlegal Ġpaths : </s> where Ġ$\ Ph i Ġ_ T ( y ) $ Ġis Ġthe Ġset Ġof Ġall Ġlegal ĠC TC Ġpaths Ġfor Ġsequence Ġ$\ math bf Ġ{ y } $ Ġwith Ġlength Ġ$ T $. ĠThe Ġloss Ġcan Ġbe Ġeasily Ġcomputed Ġusing Ġforward - back ward Ġalgorithm . ĠMore Ġdetails Ġabout ĠC TC Ġare Ġprovided Ġin Ġsupplementary Ġmaterial . </s> For ĠMT Ġtask , Ġwe Ġuse Ġthe Ġcross - ent ropy Ġloss Ġas Ġthe Ġtraining Ġobjective . ĠDuring Ġtraining , Ġ$\ math bf Ġ{ y ^ s } $ Ġis Ġconverted Ġto Ġembed ding Ġvectors Ġ$\ math bf Ġ{ e ^ s } $ Ġthrough Ġembed ding Ġlayer Ġ$ W _{ E ^ s } $, Ġthen Ġ$ enc _ t $ Ġconsumes Ġ$\ math bf Ġ{ e ^ s } $ Ġand Ġpass Ġthe Ġoutput Ġ$\ math bf Ġ{ h ^ t } $ Ġto Ġdec oder . ĠThe Ġobjective Ġfunction Ġis Ġdefined Ġas : </s> Our Ġmethod Ġ: :: ĠTraining ĠProcedure Ġ: :: ĠFine - t une </s> In Ġfine - t une Ġstage , Ġwe Ġjointly Ġupdate Ġthe Ġmodel Ġon ĠAS R , ĠMT , Ġand ĠST Ġtasks . ĠThe Ġtraining Ġfor ĠAS R Ġand ĠMT Ġfollows Ġthe Ġsame Ġprocess Ġas Ġit Ġwas Ġin Ġpre - training Ġstage . </s> For ĠST Ġtask , Ġthe Ġ$ enc _ s $ Ġreads Ġthe Ġinput Ġ$\ math bf Ġ{ x } $ Ġand Ġgenerates Ġ$\ math bf Ġ{ h ^ s } $, Ġthen Ġ$ enc _ t $ Ġlearns Ġhigh - level Ġlinguistic Ġknowledge Ġinto Ġ$\ math bf Ġ{ h ^ t } $. ĠFinally , Ġthe Ġ$ dec $ Ġpredicts Ġthe Ġtarget Ġsentence . ĠThe ĠST Ġloss Ġfunction Ġis Ġdefined Ġas : </s> Following Ġthe Ġupdate Ġstrategy Ġproposed Ġby ĠB IB REF 11 Ġl u ong 2015 multi , Ġwe Ġallocate Ġa Ġdifferent Ġtraining Ġratio Ġ$\ alpha Ġ_ i $ Ġfor Ġeach Ġtask . ĠWhen Ġswitching Ġbetween Ġtasks , Ġwe Ġselect Ġrandomly Ġa Ġnew Ġtask Ġ$ i $ Ġwith Ġprobability Ġ$\ frac {\ alpha Ġ_ i }{ \ sum Ġ_ { j }\ alpha Ġ_ { j }} $. </s> Our Ġmethod Ġ: :: ĠSub net - Cons ist ency </s> Our Ġmodel Ġkeeps Ġrole Ġconsistency Ġbetween Ġpre - training Ġand Ġfine - tun ing Ġby Ġconnecting Ġtwo Ġenc od ers Ġfor ĠST Ġtask . ĠHowever , Ġthis Ġleads Ġto Ġsome Ġnew Ġproblems : Ġ1 ) ĠThe Ġtext Ġenc oder Ġconsumes Ġ$\ math bf Ġ{ e ^ s } $ Ġduring ĠMT Ġtraining , Ġwhile Ġit Ġaccepts Ġ$\ math bf Ġ{ h ^ s } $ Ġduring ĠST Ġtraining . ĠHowever , Ġ$\ math bf Ġ{ e ^ s } $ Ġand Ġ$\ math bf Ġ{ h ^ s } $ Ġmay Ġnot Ġfollow Ġthe Ġsame Ġdistribution , Ġresulting Ġin Ġthe Ġsemantic Ġinconsistency . Ġ2 ) ĠBesides , Ġthe Ġlength Ġof Ġ$\ math bf Ġ{ h ^ s } $ Ġis Ġnot Ġthe Ġsame Ġorder Ġof Ġmagnitude Ġwith Ġthe Ġlength Ġof Ġ$\ math bf Ġ{ e ^ s } $, Ġresulting Ġin Ġthe Ġlength Ġinconsistency . </s> In Ġresponse Ġto Ġthe Ġabove Ġtwo Ġchallenges , Ġwe Ġpropose Ġtwo Ġcounter measures : Ġ1 ) ĠWe Ġshare Ġweights Ġbetween ĠC TC Ġclassification Ġlayer Ġand Ġsource - end Ġword Ġembed ding Ġlayer Ġduring Ġtraining Ġof ĠAS R Ġand ĠMT , Ġencouraging Ġ$\ math bf Ġ{ e ^ s } $ Ġand Ġ$\ math bf Ġ{ h ^ s } $ Ġin Ġthe Ġsame Ġspace . Ġ2 ) We Ġfeed Ġthe Ġtext Ġenc oder Ġsource Ġsentences Ġin Ġthe Ġformat Ġof ĠC TC Ġpath , Ġwhich Ġare Ġgenerated Ġfrom Ġa Ġseq 2 seq Ġmodel , Ġmaking Ġit Ġmore Ġrobust Ġtoward Ġlong Ġinputs . </s> Our Ġmethod Ġ: :: ĠSub net - Cons ist ency Ġ: :: ĠSem antic ĠCons ist ency </s> As Ġshown Ġin ĠFigure ĠFIG REF 5 , Ġduring Ġmulti - task Ġtraining , Ġtwo Ġdifferent Ġhidden Ġfeatures Ġwill Ġbe Ġfed Ġinto Ġthe Ġtext Ġenc oder Ġ$ enc _ t $ : Ġthe Ġembed ding Ġrepresentation Ġ$\ math bf Ġ{ e } ^ s $ Ġin ĠMT Ġtask , Ġand Ġthe Ġ$ enc _ s $ Ġoutput Ġ$\ math bf Ġ{ h ^ s } $ Ġin ĠST Ġtask . ĠWithout Ġany Ġregular ization , Ġthey Ġmay Ġbelong Ġto Ġdifferent Ġlatent Ġspaces . ĠDue Ġto Ġthe Ġspace Ġgap , Ġthe Ġ$ enc _ t $ Ġhas Ġto Ġcompromise Ġbetween Ġtwo Ġtasks , Ġlimiting Ġits Ġperformance Ġon Ġindividual Ġtasks . </s> To Ġbridge Ġthe Ġspace Ġgap , Ġour Ġidea Ġis Ġto Ġpull Ġ$\ math bf Ġ{ h ^ s } $ Ġinto Ġthe Ġlatent Ġspace Ġwhere Ġ$\ math bf Ġ{ e } ^ s $ Ġbelong . ĠSpecifically , Ġwe Ġshare Ġthe Ġweight Ġ$ W _{ ct c } $ Ġin ĠC TC Ġclassification Ġlayer Ġwith Ġthe Ġsource Ġembed ding Ġweights Ġ$ W _{ E ^ s } $, Ġwhich Ġmeans Ġ$ W _{ ct c } Ġ= ĠW _{ E ^ s } $. ĠIn Ġthis Ġway , Ġwhen Ġpredicting Ġthe ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $, Ġthe Ġprobability Ġof Ġobserving Ġthe Ġparticular Ġlabel Ġ$ w _ i Ġ\ in ĠV _{ src }\ cup Ġ${ ` -' } Ġat Ġtime Ġstep Ġ$ t $, Ġ$ p (\ pi Ġ_ t = w _ i | \ math bf Ġ{ x }) $, Ġis Ġcomputed Ġby Ġnormal izing Ġthe Ġproduct Ġof Ġhidden Ġvector Ġ$ h _ t ^ s $ Ġand Ġthe Ġ$ i $ - th Ġvector Ġin Ġ$ W _{ E ^ s } $ : </s> The Ġloss Ġfunction Ġcloses Ġthe Ġdistance Ġbetween Ġ$ h ^ s _ t $ Ġand Ġgolden Ġembed ding Ġvector , Ġencouraging Ġ$\ math bf Ġ{ h } ^ s $ Ġhave Ġthe Ġsame Ġdistribution Ġwith Ġ$\ math bf Ġ{ e } ^ s $. </s> Our Ġmethod Ġ: :: ĠSub net - Cons ist ency Ġ: :: ĠLength ĠCons ist ency </s> Another Ġexisting Ġproblem Ġis Ġlength Ġinconsistency . ĠThe Ġlength Ġof Ġthe Ġsequence Ġ$\ math bf Ġ{ h ^ s } $ Ġis Ġproportional Ġto Ġthe Ġlength Ġof Ġthe Ġinput Ġframe Ġ$\ math bf Ġ{ x } $, Ġwhich Ġis Ġmuch Ġlonger Ġthan Ġthe Ġlength Ġof Ġ$\ math bf Ġ{ e ^ s } $. ĠTo Ġsolve Ġthis Ġproblem , Ġwe Ġtrain Ġan ĠR NN - based Ġseq 2 seq Ġmodel Ġto Ġtransform Ġnormal Ġsource Ġsentences Ġto Ġnoisy Ġsentences Ġin ĠC TC Ġpath Ġformat , Ġand Ġreplace Ġstandard ĠMT Ġwith Ġden o ising ĠMT Ġfor Ġmulti - t asking . </s> Specifically , Ġwe Ġfirst Ġtrain Ġa ĠC TC ĠAS R Ġmodel Ġbased Ġon Ġdataset Ġ$\ math cal Ġ{ A } Ġ= Ġ\ l brace Ġ( \ math bf Ġ{ x } _ i , Ġ\ math bf Ġ{ y } ^ s _ i )\ r brace Ġ_ { i = 0 } ^{ I } $, Ġand Ġgenerate Ġa ĠC TC - path Ġ$\ math bf Ġ{\ pi Ġ} _ i $ Ġfor Ġeach Ġaudio Ġ$\ math bf Ġ{ x } _ i $ Ġby Ġgreedy Ġdecoding . ĠThen Ġwe Ġdefine Ġan Ġoperation Ġ$ S (\ cd ot Ġ) $, Ġwhich Ġconverts Ġa ĠC TC Ġpath Ġ$\ math bf Ġ{\ pi Ġ} $ Ġto Ġa Ġsequence Ġof Ġthe Ġunique Ġtokens Ġ$\ math bf Ġ{ u } $ Ġand Ġa Ġsequence Ġof Ġrepetition Ġtimes Ġfor Ġeach Ġtoken Ġ$\ math bf Ġ{ l } $, Ġden oted Ġas Ġ$ S (\ math bf Ġ{\ pi Ġ}) Ġ= Ġ( \ math bf Ġ{ u }, Ġ\ math bf Ġ{ l }) $. ĠNotably , Ġthe Ġoperation Ġis Ġreversible , Ġmeaning Ġthat Ġ$ S ^{ - 1 } Ġ( \ math bf Ġ{ u }, Ġ\ math bf Ġ{ l }) = \ math bf Ġ{\ pi Ġ} $. ĠWe Ġuse Ġthe Ġexample Ġ$\ math bf Ġ{\ pi Ġ_ 1 } $ Ġin ĠTable ĠT AB REF 14 Ġand Ġshow Ġthe Ġcorresponding Ġ$\ math bf Ġ{ u } $ Ġand Ġ$\ math bf Ġ{ l } $ Ġin ĠTable ĠT AB REF 24 . </s> Then Ġwe Ġbuild Ġa Ġdataset Ġ$\ math cal Ġ{ P } Ġ= Ġ\ l brace Ġ( \ math bf Ġ{ y ^ s } _ i , Ġ\ math bf Ġ{ u } _ i , Ġ\ math bf Ġ{ l } _ i )\ r brace Ġ_ { i = 0 } ^{ I } $ Ġby Ġdecoding Ġall Ġthe Ġaudio Ġpieces Ġin Ġ$\ math cal Ġ{ A } $ Ġand Ġtransform Ġthe Ġresulting Ġpath Ġby Ġthe Ġoperation Ġ$ S (\ cd ot Ġ) $. ĠAfter Ġthat , Ġwe Ġtrain Ġa Ġseq 2 seq Ġmodel , Ġas Ġshown Ġin ĠFigure ĠFIG REF 25 , Ġwhich Ġtakes Ġ$ Ġ\ math bf Ġ{ y ^ s } _ i $ Ġas Ġinput Ġand Ġdec odes Ġ$\ math bf Ġ{ u } _ i , Ġ\ math bf Ġ{ l } _ i $ Ġas Ġoutputs . ĠWith Ġthe Ġseq 2 seq Ġmodel , Ġa Ġnoisy ĠMT Ġdataset Ġ$\ math cal Ġ{ M } ^ {\ prime Ġ} = \ l brace Ġ( \ math bf Ġ{\ pi Ġ} _ l , Ġ\ math bf Ġ{ y ^ t } _ l )\ r brace Ġ_ { l = 0 } ^{ L } $ Ġis Ġobtained Ġby Ġconverting Ġevery Ġsource Ġsentence Ġ$\ math bf Ġ{ y ^ s } _ i Ġ\ in Ġ\ math cal Ġ{ M } $ Ġto Ġ$\ math bf Ġ{\ pi Ġ_ i } $, Ġwhere Ġ$\ math bf Ġ{\ pi Ġ} _ i Ġ= ĠS ^{ - 1 } (\ math bf Ġ{ u } _ i , Ġ\ math bf Ġ{ l } _ i ) $. ĠWe Ġdid Ġnot Ġuse Ġthe Ġstandard Ġseq 2 seq Ġmodel Ġwhich Ġtakes Ġ$\ math bf Ġ{ y ^ s } $ Ġas Ġinput Ġand Ġgenerates Ġ$\ math bf Ġ{\ pi Ġ} $ Ġdirectly , Ġsince Ġthere Ġare Ġtoo Ġmany Ġblank Ġtokens Ġ` -' Ġin Ġ$\ math bf Ġ{\ pi Ġ} $ Ġand Ġthe Ġmodel Ġtends Ġto Ġgenerate Ġa Ġlong Ġsequence Ġwith Ġonly Ġblank Ġtokens . ĠDuring ĠMT Ġtraining , Ġwe Ġrandomly Ġsample Ġtext Ġpairs Ġfrom Ġ$\ math cal Ġ{ M } ^ {\ prime Ġ} $ Ġand Ġ$\ math cal Ġ{ M } $ Ġaccording Ġto Ġa Ġhyper - param eter Ġ$ k $. ĠAfter Ġtuning Ġon Ġthe Ġvalidation Ġset , Ġabout Ġ$ 30 \ % $ Ġpairs Ġare Ġsampled Ġfrom Ġ$\ math cal Ġ{ M } ^ {\ prime Ġ} $. ĠIn Ġthis Ġway , Ġthe Ġ$ enc _ t $ Ġis Ġmore Ġrobust Ġtoward Ġthe Ġlonger Ġinputs Ġgiven Ġby Ġthe Ġ$ enc _ s $. </s> Exper iments </s> We Ġconduct Ġexperiments Ġon Ġthe ĠI W SL T 18 Ġspeech Ġtranslation Ġtask ĠB IB REF 1 . ĠSince ĠI W SL T Ġparticip ators Ġuse Ġdifferent Ġdata Ġpre - processing Ġmethods , Ġwe Ġreproduce Ġseveral Ġcompetitive Ġbas elines Ġbased Ġon Ġthe ĠESP net ĠB IB REF 12 Ġfor Ġa Ġfair Ġcomparison . </s> Exper iments Ġ: :: ĠDat as et Ġ: :: ĠSpeech Ġtranslation Ġdata : </s> The Ġorganizer Ġprovides Ġa Ġspeech Ġtranslation Ġcorpus Ġextracting Ġfrom Ġthe ĠTED Ġtalk Ġ( ST - TED ), Ġwhich Ġconsists Ġof Ġraw ĠEnglish Ġwave Ġfiles , ĠEnglish Ġtranscript ions , Ġand Ġaligned ĠGerman Ġtranslations . ĠThe Ġcorpus Ġcontains Ġ272 Ġhours Ġof ĠEnglish Ġspeech Ġwith Ġ171 k Ġsegments . ĠWe Ġsplit Ġ2 k Ġsegments Ġfrom Ġthe Ġcorpus Ġas Ġdev Ġset Ġand Ġt st 2010 , Ġt st 2013 , Ġt st 2014 , Ġt st 2015 Ġare Ġused Ġas Ġtest Ġsets . </s> Spe ech Ġrecognition Ġdata : ĠAside Ġfrom ĠST - TED , ĠTED - LI UM 2 Ġcorpus ĠB IB REF 13 Ġis Ġprovided Ġas Ġspeech Ġrecognition Ġdata , Ġwhich Ġcontains Ġ207 Ġhours Ġof ĠEnglish Ġspeech Ġand Ġ93 k Ġtranscript Ġsentences . </s> Text Ġtranslation Ġdata : ĠWe Ġuse Ġtranscription Ġand Ġtranslation Ġpairs Ġin Ġthe ĠST - TED Ġcorpus Ġand ĠW IT 3 Ġas Ġin - domain ĠMT Ġdata , Ġwhich Ġcontains Ġ130 k Ġand Ġ200 k Ġsentence Ġpairs Ġrespectively . ĠW MT 2018 Ġis Ġused Ġas Ġout - of - domain Ġtraining Ġdata Ġwhich Ġconsists Ġof Ġ41 M Ġsentence Ġpairs . </s> Data Ġpre processing : ĠFor Ġspeech Ġdata , Ġthe Ġutter ances Ġare Ġsegment ed Ġinto Ġmultiple Ġframes Ġwith Ġa Ġ25 Ġms Ġwindow Ġsize Ġand Ġa Ġ10 Ġms Ġstep Ġsize . ĠThen Ġwe Ġextract Ġ80 - channel Ġlog - Mel Ġfilter Ġbank Ġand Ġ3 - dimensional Ġpitch Ġfeatures Ġusing ĠK aldi ĠB IB REF 14 , Ġresulting Ġin Ġ83 - dimensional Ġinput Ġfeatures . ĠWe Ġnormal ize Ġthem Ġby Ġthe Ġmean Ġand Ġthe Ġstandard Ġdeviation Ġon Ġthe Ġwhole Ġtraining Ġset . ĠThe Ġutter ances Ġwith Ġmore Ġthan Ġ3000 Ġframes Ġare Ġdiscarded . ĠThe Ġtranscripts Ġin ĠST - TED Ġare Ġin Ġtrue - case Ġwith Ġpunct uation Ġwhile Ġin ĠTED - LI UM 2 , Ġtranscripts Ġare Ġin Ġlower - case Ġand Ġunp unct uated . ĠThus , Ġwe Ġlower case Ġall Ġthe Ġsentences Ġand Ġremove Ġthe Ġpunct uation Ġto Ġkeep Ġconsistent . ĠTo Ġincrease Ġthe Ġamount Ġof Ġtraining Ġdata , Ġwe Ġperform Ġspeed Ġpert urb ation Ġon Ġthe Ġraw Ġsignals Ġwith Ġspeed Ġfactors Ġ0 . 9 Ġand Ġ1 . 1 . ĠFor Ġthe Ġtext Ġtranslation Ġdata , Ġsentences Ġlonger Ġthan Ġ80 Ġwords Ġor Ġshorter Ġthan Ġ10 Ġwords Ġare Ġremoved . ĠBesides , Ġwe Ġdiscard Ġpairs Ġwhose Ġlength Ġratio Ġbetween Ġsource Ġand Ġtarget Ġsentence Ġis Ġsmaller Ġthan Ġ0 . 5 Ġor Ġlarger Ġthan Ġ2 . 0 . ĠWord Ġtoken ization Ġis Ġperformed Ġusing Ġthe ĠMoses Ġscripts Ġand Ġboth ĠEnglish Ġand ĠGerman Ġwords Ġare Ġin Ġlower - case . </s> We Ġuse Ġtwo Ġdifferent Ġsets Ġof Ġvocabulary Ġfor Ġour Ġexperiments . ĠFor Ġthe Ġsub word Ġexperiments , Ġboth ĠEnglish Ġand ĠGerman Ġvoc ab ular ies Ġare Ġgenerated Ġusing Ġsentence piece ĠB IB REF 15 Ġwith Ġa Ġfixed Ġsize Ġof Ġ5 k Ġtokens . ĠB IB REF 9 Ġin ag uma 2018 speech Ġshow Ġthat Ġincreasing Ġthe Ġvocabulary Ġsize Ġis Ġnot Ġhelpful Ġfor ĠST Ġtask . ĠFor Ġthe Ġcharacter Ġexperiments , Ġboth ĠEnglish Ġand ĠGerman Ġsentences Ġare Ġrepresented Ġin Ġthe Ġcharacter Ġlevel . </s> For Ġevaluation , Ġwe Ġsegment Ġeach Ġaudio Ġwith Ġthe ĠLI UM ĠSp k D iar ization Ġtool ĠB IB REF 16 Ġand Ġthen Ġperform ĠMW ER Ġsegment ation Ġwith ĠRW TH Ġtool kit ĠB IB REF 17 . ĠWe Ġuse Ġlower case ĠB LE U Ġas Ġevaluation Ġmetric . </s> Exper iments Ġ: :: ĠBas eline ĠModels Ġand ĠImplementation </s> We Ġcompare Ġour Ġmethod Ġwith Ġfollowing Ġbas elines . </s> Van illa ĠST Ġbaseline : ĠThe Ġvanilla ĠST ĠB IB REF 9 Ġhas Ġonly Ġa Ġspeech Ġenc oder Ġand Ġa Ġdec oder . ĠIt Ġis Ġtrained Ġfrom Ġscratch Ġon Ġthe ĠST - TED Ġcorpus . </s> Pre - training Ġbas elines : ĠWe Ġconduct Ġthree Ġpre - training Ġbaseline Ġexperiments : Ġ1 ) Ġenc oder Ġpre - training , Ġin Ġwhich Ġthe ĠST Ġenc oder Ġis Ġinitialized Ġfrom Ġan ĠAS R Ġmodel ; Ġ2 ) Ġdec oder Ġpre - training , Ġin Ġwhich Ġthe ĠST Ġdec oder Ġis Ġinitialized Ġfrom Ġan ĠMT Ġmodel ; Ġand Ġ3 ) Ġenc oder - dec oder Ġpre - training , Ġwhere Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġpre - trained . ĠThe ĠAS R Ġmodel Ġhas Ġthe Ġsame Ġarchitecture Ġwith Ġvanilla ĠST Ġmodel , Ġtrained Ġon Ġthe Ġmixture Ġof ĠST - TED Ġand ĠTED - LI UM 2 Ġcorpus . ĠThe ĠMT Ġmodel Ġhas Ġa Ġtext Ġenc oder Ġand Ġdec oder Ġwith Ġthe Ġsame Ġarchitecture Ġof Ġwhich Ġin ĠTC EN . ĠIt Ġis Ġfirst Ġtrained Ġon ĠW MT Ġdata Ġ( out - of - domain ) Ġand Ġthen Ġfine - tun ed Ġon Ġin - domain Ġdata . </s> Multi - task Ġbas elines : ĠWe Ġalso Ġconduct Ġthree Ġmulti - task Ġbaseline Ġexperiments Ġincluding Ġone - to - many Ġsetting , Ġmany - to - one Ġsetting , Ġand Ġmany - to - many Ġsetting . ĠIn Ġthe Ġfirst Ġtwo Ġsettings , Ġwe Ġtrain Ġthe Ġmodel Ġwith Ġ$\ alpha Ġ_ { st } = 0 . 75 $ Ġwhile Ġ$\ alpha Ġ_ { as r } = 0 . 25 $ Ġor Ġ$\ alpha Ġ_ { mt } = 0 . 25 $. ĠFor Ġmany - to - many Ġsetting , Ġwe Ġuse Ġ$\ alpha Ġ_ { st } = 0 . 6 , Ġ\ alpha Ġ_ { as r } = 0 . 2 $ Ġand Ġ$\ alpha Ġ_ { mt } = 0 . 2 $ .. ĠFor ĠMT Ġtask , Ġwe Ġuse Ġonly Ġin - domain Ġdata . </s> Many - to - many + pre - training : ĠWe Ġtrain Ġa Ġmany - to - many Ġmulti - task Ġmodel Ġwhere Ġthe Ġenc od ers Ġand Ġdec od ers Ġare Ġderived Ġfrom Ġpre - trained ĠAS R Ġand ĠMT Ġmodels . ĠTriangle + pre - train : ĠB IB REF 18 ĠD BL P : conf / na acl / An ast as opoulos C 18 Ġproposed Ġa Ġtriangle Ġmulti - task Ġstrategy Ġfor Ġspeech Ġtranslation . ĠTheir Ġmodel Ġsolves Ġthe Ġsub net Ġwaste Ġissue Ġby Ġconc aten ating Ġan ĠST Ġdec oder Ġto Ġan ĠAS R Ġenc oder - dec oder Ġmodel . ĠNotably , Ġtheir ĠST Ġdec oder Ġcan Ġconsume Ġrepresentations Ġfrom Ġthe Ġspeech Ġenc oder Ġas Ġwell Ġas Ġthe ĠAS R Ġdec oder . ĠFor Ġa Ġfair Ġcomparison , Ġthe Ġspeech Ġenc oder Ġand Ġthe ĠAS R Ġdec oder Ġare Ġinitialized Ġfrom Ġthe Ġpre - trained ĠAS R Ġmodel . ĠThe ĠTriangle Ġmodel Ġis Ġfine - tun ed Ġunder Ġtheir Ġmulti - task Ġmanner . </s> All Ġour Ġbas elines Ġas Ġwell Ġas ĠTC EN Ġare Ġimplemented Ġbased Ġon ĠESP net ĠB IB REF 12 , Ġthe ĠR NN Ġsize Ġis Ġset Ġas Ġ$ d = 1024 $ Ġfor Ġall Ġmodels . ĠWe Ġuse Ġa Ġdrop out Ġof Ġ0 . 3 Ġfor Ġembed d ings Ġand Ġenc od ers , Ġand Ġtrain Ġusing ĠAd ad elta Ġwith Ġinitial Ġlearning Ġrate Ġof Ġ1 . 0 Ġfor Ġa Ġmaximum Ġof Ġ10 Ġepoch s . </s> For Ġtraining Ġof ĠTC EN , Ġwe Ġset Ġ$\ alpha Ġ_ { as r } = 0 . 2 $ Ġand Ġ$\ alpha Ġ_ { mt } = 0 . 8 $ Ġin Ġthe Ġpre - training Ġstage , Ġsince Ġthe ĠMT Ġdataset Ġis Ġmuch Ġlarger Ġthan ĠAS R Ġdataset . ĠFor Ġfine - t une , Ġwe Ġuse Ġ$\ alpha Ġ_ { st } = 0 . 6 , Ġ\ alpha Ġ_ { as r } = 0 . 2 $ Ġand Ġ$\ alpha Ġ_ { mt } = 0 . 2 $, Ġsame Ġas Ġthe Ġ` many - to - many ' Ġbaseline . </s> For Ġtesting , Ġwe Ġselect Ġthe Ġmodel Ġwith Ġthe Ġbest Ġaccuracy Ġon Ġspeech Ġtranslation Ġtask Ġon Ġdev Ġset . ĠAt Ġinference Ġtime , Ġwe Ġuse Ġa Ġbeam Ġsize Ġof Ġ10 , Ġand Ġthe Ġbeam Ġscores Ġinclude Ġlength Ġnormal ization Ġwith Ġa Ġweight Ġof Ġ0 . 2 . </s> Exper iments Ġ: :: ĠExperimental ĠResults </s> Table ĠT AB REF 29 Ġshows Ġthe Ġresults Ġon Ġfour Ġtest Ġsets Ġas Ġwell Ġas Ġthe Ġaverage Ġperformance . ĠOur Ġmethod Ġsignificantly Ġoutper forms Ġthe Ġstrong Ġ` many - to - many + pret rain ' Ġbaseline Ġby Ġ3 . 6 Ġand Ġ2 . 2 ĠB LE U Ġscores Ġrespectively , Ġindicating Ġthe Ġproposed Ġmethod Ġis Ġvery Ġeffective Ġthat Ġsubstantially Ġimproves Ġthe Ġtranslation Ġquality . ĠBesides , Ġboth Ġpre - training Ġand Ġmulti - task Ġlearning Ġcan Ġimprove Ġtranslation Ġquality , Ġand Ġthe Ġpre - training Ġsettings Ġ( 2 nd - 4 th Ġrows ) Ġare Ġmore Ġeffective Ġcompared Ġto Ġmulti - task Ġsettings Ġ( 5 th - 8 th Ġrows ). ĠWe Ġobserve Ġa Ġperformance Ġdegradation Ġin Ġthe Ġ` tri angle + pret rain ' Ġbaseline . ĠCompared Ġto Ġour Ġmethod , Ġwhere Ġthe Ġdec oder Ġreceives Ġhigher - level Ġsynt actic Ġand Ġsemantic Ġlinguistic Ġknowledge Ġextracted Ġfrom Ġtext Ġenc oder , Ġtheir ĠAS R Ġdec oder Ġcan Ġonly Ġprovide Ġlower Ġword - level Ġlinguistic Ġinformation . ĠBesides , Ġsince Ġtheir Ġmodel Ġlacks Ġtext Ġenc oder Ġand Ġthe Ġarchitecture Ġof ĠST Ġdec oder Ġis Ġdifferent Ġfrom ĠMT Ġdec oder , Ġtheir Ġmodel Ġcannot Ġutilize Ġthe Ġlarge - scale ĠMT Ġdata Ġin Ġall Ġthe Ġtraining Ġstages . ĠInterestingly , Ġwe Ġfind Ġthat Ġthe Ġchar - level Ġmodels Ġoutper form Ġthe Ġsub word - level Ġmodels Ġin Ġall Ġsettings , Ġespecially Ġin Ġvanilla Ġbaseline . ĠA Ġsimilar Ġphenomenon Ġis Ġobserved Ġby ĠB IB REF 6 Ġber ard 2018 end . ĠA Ġpossible Ġexplanation Ġis Ġthat Ġlearning Ġthe Ġalign ments Ġbetween Ġspeech Ġframes Ġand Ġsub word Ġunits Ġin Ġanother Ġlanguage Ġis Ġnotoriously Ġdifficult . ĠOur Ġmethod Ġcan Ġbring Ġmore Ġgains Ġin Ġthe Ġsub word Ġsetting Ġsince Ġour Ġmodel Ġis Ġgood Ġat Ġlearning Ġthe Ġtext - to - text Ġalignment Ġand Ġthe Ġsub word - level Ġalignment Ġis Ġmore Ġhelpful Ġto Ġthe Ġtranslation Ġquality . </s> Exper iments Ġ: :: ĠDiscussion Ġ: :: ĠAb lation ĠStudy </s> To Ġbetter Ġunderstand Ġthe Ġcontribution Ġof Ġeach Ġcomponent , Ġwe Ġperform Ġan Ġab lation Ġstudy Ġon Ġsub word - level Ġexperiments . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 37 . ĠIn Ġ` - MT Ġnoise ' Ġsetting , Ġwe Ġdo Ġnot Ġadd Ġnoise Ġto Ġsource Ġsentences Ġfor ĠMT . ĠIn Ġ` - weight Ġsharing ' Ġsetting , Ġwe Ġuse Ġdifferent Ġparameters Ġin ĠC TC Ġclassification Ġlayer Ġand Ġsource Ġembed ding Ġlayer . ĠThese Ġtwo Ġexperiments Ġprove Ġthat Ġboth Ġweight Ġsharing Ġand Ġusing Ġnoisy ĠMT Ġinput Ġbenefit Ġto Ġthe Ġfinal Ġtranslation Ġquality . ĠPerformance Ġde grades Ġmore Ġin Ġ` - weight Ġsharing ', Ġindicating Ġthe Ġsemantic Ġconsistency Ġcontributes Ġmore Ġto Ġour Ġmodel . ĠIn Ġthe Ġ` - pret rain ' Ġexperiment , Ġwe Ġremove Ġthe Ġpre - training Ġstage Ġand Ġdirectly Ġupdate Ġthe Ġmodel Ġon Ġthree Ġtasks , Ġleading Ġto Ġa Ġdramatic Ġdecrease Ġon ĠB LE U Ġscore , Ġindicating Ġthe Ġpre - training Ġis Ġan Ġindispensable Ġstep Ġfor Ġend - to - end ĠST . </s> Exper iments Ġ: :: ĠDiscussion Ġ: :: ĠLearning ĠCurve </s> It Ġis Ġinteresting Ġto Ġinvestigate Ġwhy Ġour Ġmethod Ġis Ġsuperior Ġto Ġbas elines . ĠWe Ġfind Ġthat ĠTC EN Ġachieves Ġa Ġhigher Ġfinal Ġresult Ġowing Ġto Ġa Ġbetter Ġstart - point Ġin Ġfine - tun ing . ĠFigure ĠFIG REF 39 Ġprovides Ġlearning Ġcurves Ġof Ġsub word Ġaccuracy Ġon Ġvalidation Ġset . ĠThe Ġx - axis Ġdenotes Ġthe Ġfine - tun ing Ġtraining Ġsteps . ĠThe Ġvanilla Ġmodel Ġstarts Ġat Ġa Ġlow Ġaccuracy , Ġbecause Ġits Ġnetworks Ġare Ġnot Ġpre - trained Ġon Ġthe ĠAS R Ġand ĠMT Ġdata . ĠThe Ġtrends Ġof Ġour Ġmodel Ġand Ġ` many - to - many + pret rain ' Ġare Ġsimilar , Ġbut Ġour Ġmodel Ġoutper forms Ġit Ġabout Ġfive Ġpoints Ġin Ġthe Ġwhole Ġfine - tun ing Ġprocess . ĠIt Ġindicates Ġthat Ġthe Ġgain Ġcomes Ġfrom Ġbrid ging Ġthe Ġgap Ġbetween Ġpre - training Ġand Ġfine - tun ing Ġrather Ġthan Ġa Ġbetter Ġfine - tun ing Ġprocess . </s> Exper iments Ġ: :: ĠDiscussion Ġ: :: ĠCompared Ġwith Ġa ĠC asc aded ĠSystem </s> Table ĠT AB REF 29 Ġcompares Ġour Ġmodel Ġwith Ġend - to - end Ġbas elines . ĠHere , Ġwe Ġcompare Ġour Ġmodel Ġwith Ġcasc aded Ġsystems . ĠWe Ġbuild Ġa Ġcasc aded Ġsystem Ġby Ġcombining Ġthe ĠAS R Ġmodel Ġand ĠMT Ġmodel Ġused Ġin Ġpre - training Ġbaseline . ĠWord Ġerror Ġrate Ġ( WER ) Ġof Ġthe ĠAS R Ġsystem Ġand ĠB LE U Ġscore Ġof Ġthe ĠMT Ġsystem Ġare Ġreported Ġin Ġthe Ġsupplementary Ġmaterial . ĠIn Ġaddition Ġto Ġa Ġsimple Ġcombination Ġof Ġthe ĠAS R Ġand ĠMT Ġsystems , Ġwe Ġalso Ġre - se gment Ġthe ĠAS R Ġoutputs Ġbefore Ġfeeding Ġto Ġthe ĠMT Ġsystem , Ġden oted Ġas Ġcasc aded + re - se g . ĠSpecifically , Ġwe Ġtrain Ġa Ġseq 2 seq Ġmodel ĠB IB REF 19 Ġon Ġthe ĠMT Ġdataset , Ġwhere Ġthe Ġsource Ġside Ġis Ġa Ġno Ġpunct uation Ġsentence Ġand Ġthe Ġtarget Ġside Ġis Ġa Ġnatural Ġsentence . ĠAfter Ġthat , Ġwe Ġuse Ġthe Ġseq 2 seq Ġmodel Ġto Ġadd Ġsentence Ġboundaries Ġand Ġpunct uation Ġon ĠAS R Ġoutputs . ĠExperimental Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 41 . ĠOur Ġend - to - end Ġmodel Ġoutper forms Ġthe Ġsimple Ġcasc aded Ġmodel Ġover Ġ2 ĠB LE U Ġscores , Ġand Ġit Ġachieves Ġa Ġcomparable Ġperformance Ġwith Ġthe Ġcasc aded Ġmodel Ġcombining Ġwith Ġa Ġsentence Ġre - se gment Ġmodel . </s> Related ĠWork </s> Early Ġworks Ġconduct Ġspeech Ġtranslation Ġin Ġa Ġpipeline Ġmanner ĠB IB REF 2 , ĠB IB REF 20 , Ġwhere Ġthe ĠAS R Ġoutput Ġlatt ices Ġare Ġfed Ġinto Ġan ĠMT Ġsystem Ġto Ġgenerate Ġtarget Ġsentences . ĠH MM ĠB IB REF 21 , ĠD ense Net ĠB IB REF 22 , ĠTD NN ĠB IB REF 23 Ġare Ġcommonly Ġused ĠAS R Ġsystems , Ġwhile ĠR NN Ġwith Ġattention ĠB IB REF 19 Ġand ĠTrans former ĠB IB REF 10 Ġare Ġtop Ġchoices Ġfor ĠMT . ĠTo Ġenhance Ġthe Ġrobust ness Ġof Ġthe ĠN MT Ġmodel Ġtowards ĠAS R Ġerrors , ĠB IB REF 24 ĠD BL P : conf / e acl / Ts vet kov MD 14 Ġand ĠB IB REF 25 ĠD BL P : conf / as ru / C hen H HL 17 Ġpropose Ġto Ġsimulate Ġthe Ġnoise Ġin Ġtraining Ġand Ġinference . </s> To Ġavoid Ġerror Ġpropagation Ġand Ġhigh Ġlatency Ġissues , Ġrecent Ġworks Ġpropose Ġtranslating Ġthe Ġacoustic Ġspeech Ġinto Ġtext Ġin Ġtarget Ġlanguage Ġwithout Ġyielding Ġthe Ġsource Ġtranscription ĠB IB REF 4 . ĠSince ĠST Ġdata Ġis Ġscarce , Ġpre - training ĠB IB REF 7 , Ġmulti - task Ġlearning ĠB IB REF 4 , ĠB IB REF 6 , Ġcurriculum Ġlearning ĠB IB REF 26 , Ġattention - pass ing ĠB IB REF 27 , Ġand Ġknowledge Ġdist illation ĠB IB REF 28 , ĠB IB REF 29 Ġstrategies Ġhave Ġbeen Ġexplored Ġto Ġutilize ĠAS R Ġdata Ġand ĠMT Ġdata . ĠSpecifically , ĠB IB REF 5 ĠD BL P : conf / inters pe ech / Weiss C J WC 17 Ġshow Ġimprovements Ġof Ġperformance Ġby Ġtraining Ġthe ĠST Ġmodel Ġjointly Ġwith Ġthe ĠAS R Ġand Ġthe ĠMT Ġmodel . ĠB IB REF 6 Ġber ard 2018 end Ġobserve Ġfaster Ġconvergence Ġand Ġbetter Ġresults Ġdue Ġto Ġpre - training Ġand Ġmulti - task Ġlearning Ġon Ġa Ġlarger Ġdataset . ĠB IB REF 7 ĠD BL P : conf / na acl / B ans al K LL G 19 Ġshow Ġthat Ġpre - training Ġa Ġspeech Ġenc oder Ġon Ġone Ġlanguage Ġcan Ġimprove ĠST Ġquality Ġon Ġa Ġdifferent Ġsource Ġlanguage . ĠAll Ġof Ġthem Ġfollow Ġthe Ġtraditional Ġmulti - task Ġtraining Ġstrategies . ĠB IB REF 26 ĠD BL P : j ournals / cor r / abs - 18 02 - 06 003 Ġpropose Ġto Ġuse Ġcurriculum Ġlearning Ġto Ġimprove ĠST Ġperformance Ġon Ġsynt act ically Ġdistant Ġlanguage Ġpairs . ĠTo Ġeffectively Ġleverage Ġtranscript ions Ġin ĠST Ġdata , ĠB IB REF 18 ĠD BL P : conf / na acl / An ast as opoulos C 18 Ġaugment Ġthe Ġmulti - task Ġmodel Ġwhere Ġthe Ġtarget Ġdec oder Ġreceives Ġinformation Ġfrom Ġthe Ġsource Ġdec oder Ġand Ġthey Ġshow Ġimprovements Ġon Ġlow - resource Ġspeech Ġtranslation . ĠTheir Ġmodel Ġjust Ġconsumes ĠAS R Ġand ĠST Ġdata , Ġin Ġcontrast , Ġour Ġwork Ġsufficiently Ġutilizes Ġthe Ġlarge - scale ĠMT Ġdata Ġto Ġcapture Ġthe Ġrich Ġsemantic Ġknowledge . ĠB IB REF 30 ĠD BL P : conf / ic ass p / J ia J MW CC AL W 19 Ġuse Ġpre - trained ĠMT Ġand Ġtext - to - speech Ġ( T TS ) Ġsynthesis Ġmodels Ġto Ġconvert Ġweak ly Ġsupervised Ġdata Ġinto ĠST Ġpairs Ġand Ġdemonstrate Ġthat Ġan Ġend - to - end ĠMT Ġmodel Ġcan Ġbe Ġtrained Ġusing Ġonly Ġsynthes ised Ġdata . </s> Conclusion </s> This Ġpaper Ġhas Ġinvestigated Ġthe Ġend - to - end Ġmethod Ġfor ĠST . ĠIt Ġhas Ġdiscussed Ġwhy Ġthere Ġis Ġa Ġhuge Ġgap Ġbetween Ġpre - training Ġand Ġfine - tun ing Ġin Ġprevious Ġmethods . ĠTo Ġalleviate Ġthese Ġissues , Ġwe Ġhave Ġproposed Ġa Ġmethod , Ġwhich Ġis Ġcapable Ġof Ġre using Ġevery Ġsub - net Ġand Ġkeeping Ġthe Ġrole Ġof Ġsub - net Ġconsistent Ġbetween Ġpre - training Ġand Ġfine - tun ing . ĠEm pir ical Ġstudies Ġhave Ġdemonstrated Ġthat Ġour Ġmodel Ġsignificantly Ġoutper forms Ġbas elines .