we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part, On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP)
EXTRACTIVE
In encoder they use convolutional, NIN and bidirectional LSTM layers and in decoder they use unidirectional LSTM 
ABSTRACTIVE
In this work, we use the raw waveform as the input representation instead of spectral-based features and a grapheme (character) sequence as the output representation. In contrast to most encoder-decoder architectures, which are purely based on recurrent neural network (RNNs) framework, we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part. We use convolutional layers because they are suitable for extracting local information from raw speech. We use a striding mechanism to reduce the dimension from the input frames BIBREF17 , while the NIN layer represents more complex structures on the top of the convolutional layers. On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP) as described in Eq. EQREF2 . For more details, we illustrate our architecture in Figure FIGREF4 .
On the top layers of the encoder after the transferred convolutional and NIN layers, we put three bidirectional LSTMs (Bi-LSTM) with 256 hidden units (total 512 units for both directions). To reduce the computational time, we used hierarchical subsampling BIBREF21 , BIBREF22 , BIBREF10 . We applied subsampling on all the Bi-LSTM layers and reduced the length by a factor of 8. On the decoder side, the previous input phonemes / characters were converted into real vectors by a 128-dimensional embedding matrix. We used one unidirectional LSTM with 512 hidden units and followed by a softmax layer to output the character probability. For the end-to-end training phase, we froze the parameter values from the transferred layers from epoch 0 to epoch 10, and after epoch 10 we jointly optimized all the parameters together until the end of training (a total 40 epochs). We used an Adam BIBREF23 optimizer with a learning rate of 0.0005.
1709.07814
Introduction </s> Con ventional Ġlarge - voc abulary Ġcontinuous Ġspeech Ġrecognition Ġ( LV CS R ) Ġsystems Ġtypically Ġperform Ġmulti - level Ġpattern Ġrecognition Ġtasks Ġthat Ġmap Ġthe Ġacoustic Ġspeech Ġwave form Ġinto Ġa Ġhierarchy Ġof Ġspeech Ġunits Ġsuch Ġas Ġsub - words Ġ( ph on em es ), Ġwords , Ġand Ġstrings Ġof Ġwords Ġ( sent ences ). ĠSuch Ġsystems Ġbasically Ġconsist Ġof Ġseveral Ġsub - comp onents Ġ( feature Ġextract or , Ġacoustic Ġmodel , Ġpronunciation Ġlex icon , Ġlanguage Ġmodel ) Ġthat Ġare Ġtrained Ġand Ġtuned Ġseparately ĠB IB REF 0 Ġ. ĠFirst , Ġthe Ġspeech Ġsignal Ġis Ġprocessed Ġinto Ġa Ġset Ġof Ġobservation Ġfeatures Ġbased Ġon Ġa Ġcarefully Ġhand - crafted Ġfeature Ġextract or , Ġsuch Ġas ĠMel Ġfrequency Ġc ep st ral Ġcoefficients Ġ( MF CC ) Ġor ĠMel - scale Ġspect rogram . ĠThen Ġthe Ġacoustic Ġmodel Ġclass ifies Ġthe Ġobservation Ġfeatures Ġinto Ġsub - unit Ġor Ġphon eme Ġclasses . ĠFinally , Ġthe Ġsearch Ġalgorithm Ġfinds Ġthe Ġmost Ġprobable Ġword Ġsequence Ġbased Ġon Ġthe Ġevidence Ġof Ġthe Ġacoustic Ġmodel , Ġthe Ġlex icon , Ġand Ġthe Ġlanguage Ġmodel . ĠBut , Ġit Ġis Ġwidely Ġknown Ġthat Ġinformation Ġloss Ġin Ġthe Ġearlier Ġstage Ġcan Ġpropagate Ġthrough Ġthe Ġlater Ġstages . </s> Deep Ġlearning Ġalgorithms Ġhave Ġproduced Ġmany Ġstate - of - the - art Ġperformances Ġin Ġvarious Ġtasks Ġthat Ġhave Ġrevital ized Ġthe Ġuse Ġof Ġneural Ġnetworks Ġfor ĠAS R . ĠOne Ġof Ġthe Ġimportant Ġfactors Ġbehind Ġthe Ġpopularity Ġof Ġdeep Ġlearning Ġis Ġthe Ġpossibility Ġof Ġsimpl ifying Ġmany Ġcomplicated Ġhand - engine ered Ġmodels Ġby Ġletting ĠD NN s Ġfind Ġtheir Ġway Ġto Ġmap Ġfrom Ġinput Ġto Ġoutput Ġspaces . ĠInterest Ġhas Ġemerged Ġrecently Ġin Ġthe Ġpossibility Ġof Ġlearning ĠD NN - based Ġacoustic Ġmodels Ġdirectly Ġfrom Ġthe Ġraw Ġspeech Ġwave form Ġwithout Ġany Ġpred efined Ġalign ments Ġand Ġhand - engine ered Ġmodels . ĠIn Ġthis Ġway , Ġthe Ġfeature Ġextract or Ġand Ġacoustic Ġmodel Ġcan Ġbe Ġintegrated Ġinto Ġa Ġsingle Ġarchitecture . ĠPal az Ġet Ġal . ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġproposed Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġto Ġdirectly Ġtrain Ġan Ġacoustic Ġmodel Ġfrom Ġthe Ġraw Ġspeech Ġwave form . ĠS ain ath Ġet Ġal . ĠB IB REF 3 Ġused Ġtime - conv olution al Ġlayers Ġover Ġraw Ġspeech Ġand Ġtrained Ġthem Ġjointly Ġwith Ġthe Ġlong Ġshort - term Ġmemory Ġdeep Ġneural Ġnetwork Ġ( CL D NN ) Ġacoustic Ġmodel . ĠThe Ġresults Ġshowed Ġthat Ġraw Ġwave form ĠCL D NN s Ġmatched Ġthe Ġperformance Ġof Ġlog - mel ĠCL D NN s Ġon Ġa Ġvoice Ġsearch Ġtask . ĠGh ah re man i Ġet Ġal . ĠB IB REF 4 Ġrecently Ġproposed Ġa ĠCNN Ġtime - delay Ġneural Ġnetwork Ġ( CNN - TD NN ) Ġwith Ġnetwork - in - network Ġ( N IN ) Ġarchitecture , Ġand Ġalso Ġshowed Ġthat Ġtheir Ġmodel Ġoutper formed ĠMF CC - based ĠTD NN Ġon Ġthe ĠWall ĠStreet ĠJournal Ġ( WS J ) ĠB IB REF 5 Ġtask . ĠBut Ġdespite Ġsignificant Ġprogress Ġthat Ġhas Ġbeen Ġmade , Ġthe Ġsuccessful Ġmodels Ġwere Ġmostly Ġdemonstrated Ġonly Ġwithin Ġthe Ġhybrid ĠD NN - H MM Ġspeech Ġrecognition Ġframeworks . </s> On Ġthe Ġother Ġhand , Ġsome Ġexisting Ġworks Ġconstructed Ġend - to - end Ġneural Ġnetwork Ġmodels Ġfor ĠAS R Ġand Ġreplaced Ġthe Ġacoustic Ġmodel , Ġthe Ġlex icon Ġmodel , Ġand Ġthe Ġlanguage Ġmodel Ġwith Ġa Ġsingle Ġintegrated Ġmodel , Ġthus Ġsimpl ifying Ġthe Ġpipeline . ĠGraves Ġet Ġal . ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġsuccessfully Ġbuilt Ġan Ġend - to - end ĠAS R Ġbased Ġon Ġthe Ġconnection ist Ġtemporal Ġclassification Ġ( CT C ) Ġframework . ĠAm ode i Ġet Ġal . ĠB IB REF 8 Ġalso Ġconstructed Ġan Ġend - to - end ĠC TC - based ĠAS R Ġthat Ġdirectly Ġproduced Ġcharacter Ġstrings Ġinstead Ġof Ġphon eme Ġsequences . ĠBut Ġthe ĠC TC - based Ġarchitecture Ġstill Ġpredicts Ġthe Ġtarget Ġoutputs Ġfor Ġevery Ġframe Ġwithout Ġany Ġimplicit Ġknowledge Ġabout Ġthe Ġlanguage Ġmodel . ĠAnother Ġapproach Ġuses Ġa Ġsequence - to - sequence Ġattention - based Ġenc oder - dec oder Ġthat Ġexplicitly Ġuses Ġthe Ġhistory Ġof Ġprevious Ġoutputs . ĠCh or owski Ġet Ġal . ĠB IB REF 9 Ġand ĠChan Ġet Ġal . ĠB IB REF 10 Ġhas Ġsuccessfully Ġdemonstrated Ġenc oder - dec oder Ġbased ĠAS R Ġframeworks . ĠUnfortunately , Ġmost Ġof Ġthese Ġworks Ġstill Ġused Ġthe Ġstandard Ġspectral Ġfeatures Ġ( i . e ., ĠMel - scale Ġspect rogram , ĠMF CC ) Ġas Ġthe Ġinput . ĠThe Ġonly Ġattempt Ġon Ġend - to - end Ġspeech Ġrecognition Ġfor Ġa Ġraw Ġwave form Ġwas Ġrecently Ġproposed Ġby ĠB IB REF 11 Ġ. ĠTheir Ġsystem Ġused Ġa Ġdeep ĠCNN Ġand Ġwas Ġtrained Ġwith Ġthe Ġautomatic Ġsegment ation Ġcriterion Ġ( AS G ) Ġas Ġan Ġalternative Ġto ĠC TC . ĠHowever , Ġsimilar Ġwith ĠC TC , Ġthe Ġmodel Ġdid Ġnot Ġexplicitly Ġuse Ġthe Ġhistory Ġof Ġthe Ġprevious Ġoutputs Ġassuming Ġthey Ġwere Ġcondition ally Ġindependent Ġof Ġeach Ġother . ĠFurthermore , Ġits Ġperformance Ġwas Ġonly Ġreported Ġusing Ġa Ġvery Ġlarge Ġdata Ġset Ġ( about Ġ1000 h Ġof Ġaudio Ġfiles ). </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġfew Ġstudies Ġhave Ġexplored Ġa Ġsingle Ġend - to - end ĠAS R Ġarchitecture Ġtrained Ġon Ġraw Ġspeech Ġwave forms Ġto Ġdirectly Ġoutput Ġtext Ġtranscription , Ġand Ġnone Ġof Ġthose Ġmodels Ġwere Ġbuilt Ġbased Ġon Ġan Ġenc oder - dec oder Ġarchitecture . ĠIn Ġthis Ġpaper , Ġwe Ġtake Ġa Ġstep Ġforward Ġto Ġconstruct Ġan Ġend - to - end ĠAS R Ġusing Ġan Ġattention al - based Ġenc oder - dec oder Ġmodel Ġfor Ġprocessing Ġraw Ġspeech Ġwave form , Ġnaming Ġit Ġas ĠâĢ ľ Att ention - based ĠW av 2 Text ". ĠWe Ġinvestigate Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġon Ġstandard ĠAS R Ġdatasets . ĠIn Ġpractice , Ġoptimizing Ġan Ġenc oder - dec oder Ġframework Ġis Ġmore Ġdifficult Ġthan Ġa Ġstandard Ġneural Ġnetwork Ġarchitecture ĠB IB REF 10 Ġ. ĠTherefore , Ġwe Ġpropose Ġa Ġfeature Ġtransfer Ġlearning Ġmethod Ġto Ġassist Ġthe Ġtraining Ġprocess Ġfor Ġour Ġend - to - end Ġattention - based ĠAS R Ġmodel . </s> Att ention - based ĠEnc oder ĠDecoder Ġfor ĠRaw ĠSpeech ĠRecogn ition </s> The Ġenc oder - dec oder Ġmodel Ġis Ġa Ġneural Ġnetwork Ġthat Ġdirectly Ġmodels Ġconditional Ġprobability ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġsource Ġsequence Ġwith Ġlength ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġtarget Ġsequence Ġwith Ġlength ĠIN LINE FORM 4 Ġ. ĠIt Ġconsists Ġof Ġenc oder , Ġdec oder Ġand Ġattention Ġmodules . ĠThe Ġenc oder Ġtask Ġprocesses Ġan Ġinput Ġsequence ĠIN LINE FORM 5 Ġand Ġoutputs Ġrepresentative Ġinformation ĠIN LINE FORM 6 Ġfor Ġthe Ġdec oder . ĠThe Ġattention Ġmodule Ġis Ġan Ġextension Ġscheme Ġthat Ġassists Ġthe Ġdec oder Ġto Ġfind Ġrelevant Ġinformation Ġon Ġthe Ġenc oder Ġside Ġbased Ġon Ġthe Ġcurrent Ġdec oder Ġhidden Ġstates ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠUsually , Ġthe Ġattention Ġmodule Ġproduces Ġcontext Ġinformation ĠIN LINE FORM 7 Ġat Ġtime ĠIN LINE FORM 8 Ġbased Ġon Ġthe Ġenc oder Ġand Ġdec oder Ġhidden Ġstates : ĠDIS PLAY FORM 0 </s> There Ġare Ġseveral Ġvariations Ġfor Ġscore Ġfunction Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġis Ġthe Ġnumber Ġof Ġhidden Ġunits Ġfor Ġthe Ġenc oder Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġhidden Ġunits Ġfor Ġthe Ġdec oder . ĠFinally , Ġthe Ġdec oder Ġtask , Ġwhich Ġpredicts Ġthe Ġtarget Ġsequence Ġprobability Ġat Ġtime ĠIN LINE FORM 3 Ġbased Ġon Ġprevious Ġoutput Ġand Ġcontext Ġinformation ĠIN LINE FORM 4 Ġ, Ġcan Ġbe Ġformulated Ġas : ĠDIS PLAY FORM 0 </s> The Ġmost Ġcommon Ġinput ĠIN LINE FORM 0 Ġfor Ġspeech Ġrecognition Ġtasks Ġis Ġa Ġsequence Ġof Ġfeature Ġvectors Ġsuch Ġas Ġlog ĠMel - spect ral Ġspect rogram Ġand / or ĠMF CC . ĠTherefore , ĠIN LINE FORM 1 Ġwhere ĠD Ġis Ġthe Ġnumber Ġof Ġthe Ġfeatures Ġand ĠS Ġis Ġthe Ġtotal Ġlength Ġof Ġthe Ġutter ance Ġin Ġframes . ĠThe Ġoutput ĠIN LINE FORM 2 Ġcan Ġbe Ġeither Ġphon eme Ġor Ġgrap he me Ġ( character ) Ġsequence . </s> In Ġthis Ġwork , Ġwe Ġuse Ġthe Ġraw Ġwave form Ġas Ġthe Ġinput Ġrepresentation Ġinstead Ġof Ġspectral - based Ġfeatures Ġand Ġa Ġgrap he me Ġ( character ) Ġsequence Ġas Ġthe Ġoutput Ġrepresentation . ĠIn Ġcontrast Ġto Ġmost Ġenc oder - dec oder Ġarchitectures , Ġwhich Ġare Ġpurely Ġbased Ġon Ġrecurrent Ġneural Ġnetwork Ġ( R NN s ) Ġframework , Ġwe Ġconstruct Ġan Ġenc oder Ġwith Ġseveral Ġconv olution al Ġlayers ĠB IB REF 14 Ġfollowed Ġby ĠN IN Ġlayers ĠB IB REF 15 Ġas Ġthe Ġlower Ġpart Ġin Ġthe Ġenc oder Ġand Ġintegrate Ġthem Ġwith Ġdeep Ġbid irection al Ġlong Ġshort - term Ġmemory Ġ( Bi - L ST M ) ĠB IB REF 16 Ġat Ġthe Ġhigher Ġpart . ĠWe Ġuse Ġconv olution al Ġlayers Ġbecause Ġthey Ġare Ġsuitable Ġfor Ġextracting Ġlocal Ġinformation Ġfrom Ġraw Ġspeech . ĠWe Ġuse Ġa Ġstr iding Ġmechanism Ġto Ġreduce Ġthe Ġdimension Ġfrom Ġthe Ġinput Ġframes ĠB IB REF 17 Ġ, Ġwhile Ġthe ĠN IN Ġlayer Ġrepresents Ġmore Ġcomplex Ġstructures Ġon Ġthe Ġtop Ġof Ġthe Ġconv olution al Ġlayers . ĠOn Ġthe Ġdec oder Ġside , Ġwe Ġuse Ġa Ġstandard Ġdeep Ġun id irection al ĠL ST M Ġwith Ġglobal Ġattention ĠB IB REF 13 Ġthat Ġis Ġcalculated Ġby Ġa Ġmulti - layer Ġpercept ron Ġ( ML P ) Ġas Ġdescribed Ġin ĠE q . ĠEQ REF 2 Ġ. ĠFor Ġmore Ġdetails , Ġwe Ġillustrate Ġour Ġarchitecture Ġin ĠFigure ĠFIG REF 4 Ġ. </s> Feature ĠTransfer ĠLearning </s> Deep Ġlearning Ġis Ġwell Ġknown Ġfor Ġits Ġability Ġto Ġlearn Ġdirectly Ġfrom Ġlow - level Ġfeature Ġrepresentation Ġsuch Ġas Ġraw Ġspeech ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ. ĠHowever , Ġin Ġmost Ġcases Ġsuch Ġmodels Ġare Ġalready Ġconditioned Ġon Ġa Ġfixed Ġinput Ġsize Ġand Ġa Ġsingle Ġtarget Ġoutput Ġ( i . e ., Ġpredicting Ġone Ġphon eme Ġclass Ġfor Ġeach Ġinput Ġframe ). ĠIn Ġthe Ġattention - based Ġenc oder - dec oder Ġmodel , Ġthe Ġtraining Ġprocess Ġis Ġnot Ġas Ġeasy Ġas Ġin Ġa Ġstandard Ġneural Ġnetwork Ġmodel ĠB IB REF 10 Ġbecause Ġthe Ġattention - based Ġmodel Ġneeds Ġto Ġjointly Ġoptimize Ġthree Ġdifferent Ġmodules Ġsimultaneously : Ġ( 1 ) Ġan Ġenc oder Ġmodule Ġfor Ġproducing Ġrepresentative Ġinformation Ġfrom Ġa Ġsource Ġsequence ; Ġ( 2 ) Ġan Ġattention Ġmodule Ġfor Ġcalculating Ġthe Ġcorrect Ġalignment ; Ġand Ġ( 3 ) Ġa Ġdec oder Ġmodule Ġfor Ġgenerating Ġcorrect Ġtranscript ions . ĠIf Ġone Ġof Ġthese Ġmodules Ġhas Ġdifficulty Ġfulfilling Ġits Ġown Ġtasks , Ġthen Ġthe Ġmodel Ġwill Ġfail Ġto Ġproduce Ġgood Ġresults . </s> To Ġease Ġthe Ġburden Ġon Ġtraining Ġthe Ġwhole Ġenc oder - dec oder Ġarchitecture Ġdirectly Ġto Ġpredict Ġthe Ġtext Ġtranscription Ġgiven Ġthe Ġraw Ġspeech Ġwave form , Ġwe Ġutilize Ġa Ġtransfer Ġlearning Ġmethod Ġon Ġthe Ġenc oder Ġpart . ĠSpecifically , Ġwe Ġonly Ġtrain Ġthe Ġenc oder 's Ġlower Ġlayers Ġconsisting Ġof Ġthe Ġconv olution al Ġand ĠN IN Ġlayers Ġto Ġpredict Ġthe Ġspectral Ġfeatures Ġgiven Ġthe Ġcorresponding Ġraw Ġwave form . ĠIn Ġthis Ġwork , Ġwe Ġutilize Ġtwo Ġwidely Ġused Ġspectral Ġfeatures : ĠMF CC Ġand Ġlog ĠMel - scale Ġspect rogram Ġas Ġthe Ġtransfer Ġlearning Ġtarget . ĠFigure ĠFIG REF 5 Ġshows Ġour Ġfeature Ġtransfer Ġlearning Ġarchitecture . ĠFirst , Ġgiven Ġsegment ed Ġraw Ġspeech Ġwave form ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġcorresponding ĠIN LINE FORM 1 Ġ- dimensional Ġspectral Ġfeatures ĠIN LINE FORM 2 Ġ. ĠThen Ġwe Ġprocess Ġraw Ġspeech ĠIN LINE FORM 3 Ġwith Ġseveral Ġconv olutions , Ġfollowed Ġby ĠN IN Ġlayers Ġin Ġthe Ġenc oder Ġpart . ĠIn Ġthe Ġlast ĠN IN - layer , Ġwe Ġset Ġa Ġfixed Ġnumber Ġof Ġchannels Ġas ĠIN LINE FORM 4 Ġchannels Ġand Ġapply Ġmean - pool ing Ġacross Ġtime . ĠFinally , Ġwe Ġget Ġpredictions Ġfor Ġcorresponding Ġspectral Ġfeatures ĠIN LINE FORM 5 Ġand Ġoptimize Ġall Ġof Ġthe Ġparameters Ġby Ġminimizing Ġthe Ġmean Ġsquared Ġerror Ġbetween Ġpredicted Ġspectral Ġfeatures ĠIN LINE FORM 6 Ġand Ġtarget Ġspectral Ġfeatures ĠIN LINE FORM 7 Ġ: ĠDIS PLAY FORM 0 </s> In Ġthis Ġpaper , Ġwe Ġalso Ġexplore Ġmulti Ġtarget Ġfeature Ġtransfer Ġusing Ġa Ġsimilar Ġstructure Ġas Ġin ĠFigure ĠFIG REF 5 Ġbut Ġwith Ġtwo Ġparallel ĠN IN Ġlayers , Ġfollowed Ġby Ġmean - poll ing Ġat Ġthe Ġend . ĠOne Ġof Ġthe Ġoutput Ġlayers Ġis Ġused Ġto Ġpredicts Ġlog ĠMel - scale Ġspect rogram Ġand Ġanother Ġpredicts ĠMF CC Ġfeatures . ĠWe Ġmodify Ġthe Ġsingle Ġtarget Ġloss Ġfunction Ġfrom ĠE q . ĠEQ REF 6 Ġinto Ġthe Ġfollowing : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġthe Ġpredicted ĠMel - scale Ġspect rogram Ġand Ġthe ĠMF CC Ġvalues , Ġand ĠIN LINE FORM 1 Ġare Ġthe Ġreal ĠMel - scale Ġspect rogram Ġand ĠMF CC Ġfeatures Ġfor Ġframe ĠIN LINE FORM 2 Ġ. ĠAfter Ġoptimizing Ġall Ġthe Ġconv olution al Ġand ĠN IN Ġlayer Ġparameters , Ġwe Ġtransfer Ġthe Ġtrained Ġlayers Ġand Ġparameters Ġand Ġintegrate Ġthem Ġwith Ġthe ĠBi - L ST M Ġenc oder . ĠFinally , Ġwe Ġjointly Ġoptimize Ġthe Ġwhole Ġstructure Ġtogether . </s> Spe ech ĠData </s> In Ġthis Ġstudy , Ġwe Ġinvestigate Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġon ĠWS J ĠB IB REF 5 Ġ. ĠWe Ġused Ġthe Ġsame Ġdefinitions Ġof Ġthe Ġtraining , Ġdevelopment Ġand Ġtest Ġset Ġas Ġthe ĠK aldi Ġs 5 Ġrecipe ĠB IB REF 18 Ġ. ĠThe Ġraw Ġspeech Ġwave forms Ġwere Ġsegment ed Ġinto Ġmultiple Ġframes Ġwith Ġa Ġ25 ms Ġwindow Ġsize Ġand Ġa Ġ10 ms Ġstep Ġsize . ĠWe Ġnormalized Ġthe Ġraw Ġspeech Ġwave form Ġinto Ġthe Ġrange Ġ- 1 Ġto Ġ1 . ĠFor Ġspectral Ġbased Ġfeatures Ġsuch Ġas ĠMF CC Ġand Ġlog ĠMel - spect rogram , Ġwe Ġnormalized Ġthe Ġfeatures Ġfor Ġeach Ġdimension Ġinto Ġzero Ġmean Ġand Ġunit Ġvariance . ĠFor ĠWS J , Ġwe Ġseparated Ġinto Ġtwo Ġexperiments Ġby Ġusing ĠWS J - SI 84 Ġonly Ġand ĠWS J - SI 284 Ġdata . ĠWe Ġused Ġdev _ 93 Ġfor Ġour Ġvalidation Ġset Ġand Ġeval _ 92 Ġfor Ġour Ġtest Ġset . ĠWe Ġused Ġthe Ġcharacter Ġsequence Ġas Ġour Ġdec oder Ġtarget Ġand Ġfollowed Ġthe Ġpre processing Ġstep Ġproposed Ġby ĠB IB REF 19 Ġ. ĠThe Ġtext Ġfrom Ġall Ġthe Ġutter ances Ġwas Ġmapped Ġinto Ġa Ġ32 - character Ġset : Ġ26 Ġ( a - z ) Ġalphabet , Ġapost rophe , Ġperiod , Ġdash , Ġspace , Ġnoise , Ġand ĠâĢ ľ e os ". </s> Model ĠArchitect ures </s> Our Ġattention - based ĠW av 2 Text Ġarchitecture Ġuses Ġfour Ġconv olution al Ġlayers , Ġfollowed Ġby Ġtwo ĠN IN Ġlayers Ġat Ġthe Ġlower Ġpart Ġof Ġthe Ġenc oder Ġmodule . ĠFor Ġall Ġthe Ġconv olution al Ġlayers , Ġwe Ġused Ġa Ġleak y Ġrect ifier Ġunit Ġ( L Re LU ) ĠB IB REF 20 Ġactivation Ġfunction Ġwith Ġleak iness ĠIN LINE FORM 0 Ġ. ĠInside Ġthe Ġfirst ĠN IN Ġlayers , Ġwe Ġstacked Ġthree Ġconsecutive Ġfilters Ġwith ĠL Re LU Ġactivation Ġfunction . ĠFor Ġthe Ġsecond ĠN IN Ġlayers , Ġwe Ġstacked Ġtwo Ġconsecutive Ġfilters Ġwith Ġtan h Ġand Ġidentity Ġactivation Ġfunction . ĠFor Ġthe Ġfeature Ġtransfer Ġlearning Ġtraining Ġphase , Ġwe Ġused ĠMoment um ĠSG D Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 01 Ġand Ġmomentum Ġof Ġ0 . 9 . ĠTable ĠT AB REF 11 Ġsummarizes Ġthe Ġdetails Ġof Ġthe Ġlayer Ġsettings Ġfor Ġthe Ġconv olution al Ġand ĠN IN Ġlayers . </s> On Ġthe Ġtop Ġlayers Ġof Ġthe Ġenc oder Ġafter Ġthe Ġtransferred Ġconv olution al Ġand ĠN IN Ġlayers , Ġwe Ġput Ġthree Ġbid irection al ĠL ST Ms Ġ( Bi - L ST M ) Ġwith Ġ256 Ġhidden Ġunits Ġ( total Ġ512 Ġunits Ġfor Ġboth Ġdirections ). ĠTo Ġreduce Ġthe Ġcomputational Ġtime , Ġwe Ġused Ġhierarchical Ġsubs am pling ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 10 Ġ. ĠWe Ġapplied Ġsubs am pling Ġon Ġall Ġthe ĠBi - L ST M Ġlayers Ġand Ġreduced Ġthe Ġlength Ġby Ġa Ġfactor Ġof Ġ8 . </s> On Ġthe Ġdec oder Ġside , Ġthe Ġprevious Ġinput Ġphon em es Ġ/ Ġcharacters Ġwere Ġconverted Ġinto Ġreal Ġvectors Ġby Ġa Ġ128 - dimensional Ġembed ding Ġmatrix . ĠWe Ġused Ġone Ġun id irection al ĠL ST M Ġwith Ġ512 Ġhidden Ġunits Ġand Ġfollowed Ġby Ġa Ġsoft max Ġlayer Ġto Ġoutput Ġthe Ġcharacter Ġprobability . ĠFor Ġthe Ġend - to - end Ġtraining Ġphase , Ġwe Ġfroze Ġthe Ġparameter Ġvalues Ġfrom Ġthe Ġtransferred Ġlayers Ġfrom Ġepoch Ġ0 Ġto Ġepoch Ġ10 , Ġand Ġafter Ġepoch Ġ10 Ġwe Ġjointly Ġoptimized Ġall Ġthe Ġparameters Ġtogether Ġuntil Ġthe Ġend Ġof Ġtraining Ġ( a Ġtotal Ġ40 Ġepoch s ). ĠWe Ġused Ġan ĠAdam ĠB IB REF 23 Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 000 5 . </s> In Ġthe Ġdecoding Ġphase , Ġwe Ġused Ġa Ġbeam Ġsearch Ġstrategy Ġwith Ġbeam Ġsize ĠIN LINE FORM 0 Ġand Ġwe Ġadjusted Ġthe Ġscore Ġby Ġdividing Ġwith Ġthe Ġtranscription Ġlength Ġto Ġprevent Ġthe Ġdec oder Ġfrom Ġfavoring Ġshorter Ġtranscript ions . ĠWe Ġdid Ġnot Ġuse Ġany Ġlanguage Ġmodel Ġor Ġlex icon Ġdictionary Ġfor Ġdecoding . ĠAll Ġof Ġour Ġmodels Ġwere Ġimplemented Ġon Ġthe ĠPy Tor ch Ġframework Ġ. </s> For Ġcomparison , Ġwe Ġalso Ġevaluated Ġthe Ġstandard Ġattention - based Ġenc oder Ġdec oder Ġwith ĠMel - scale Ġspect rogram Ġinput Ġas Ġthe Ġbaseline . ĠHere , Ġwe Ġused Ġsimilar Ġsettings Ġas Ġthe Ġproposed Ġmodel , Ġexcept Ġwe Ġreplaced Ġthe Ġconv olution al Ġand ĠN IN Ġlayers Ġwith Ġa Ġfeed forward Ġlayer Ġ( 512 Ġhidden Ġunits ). </s> Result </s> An Ġexample Ġof Ġour Ġtransfer Ġlearning Ġresults Ġis Ġshown Ġin ĠFigure ĠFIG REF 8 Ġ, Ġand ĠTable ĠT AB REF 14 Ġshows Ġthe Ġspeech Ġrecognition Ġperformance Ġin ĠC ER Ġfor Ġboth Ġthe ĠWS J - SI 84 Ġand ĠWS J - SI 284 Ġdatasets . ĠWe Ġcompared Ġour Ġmethod Ġwith Ġseveral Ġpublished Ġmodels Ġlike ĠC TC , ĠAttention ĠEnc oder - Dec oder Ġand ĠJoint ĠC TC - Att ention Ġmodel Ġthat Ġutilize ĠC TC Ġfor Ġtraining Ġthe Ġenc oder Ġpart . ĠBesides , Ġwe Ġalso Ġtrain Ġour Ġown Ġbaseline ĠAttention ĠEnc oder - Dec oder Ġwith ĠMel - scale Ġspect rogram . ĠThe Ġdifference Ġbetween Ġour ĠAttention ĠEnc oder - Dec oder Ġ( âĢ ľ Att ĠEnc - Dec Ġ( ours )", ĠâĢ ľ Att ĠEnc - Dec ĠW av 2 Text ") Ġwith ĠAttention ĠEnc oder - Dec oder Ġfrom ĠB IB REF 24 Ġ( âĢ ľ Att ĠEnc - Dec ĠContent ", ĠâĢ ľ Att ĠEnc - Dec ĠLocation ") Ġis Ġwe Ġused Ġthe Ġcurrent Ġhidden Ġstates Ġto Ġgenerate Ġthe Ġattention Ġvector Ġinstead Ġof Ġthe Ġprevious Ġhidden Ġstates . ĠAnother Ġaddition Ġis Ġwe Ġutilized ĠâĢ ľ input Ġfeedback " Ġmethod ĠB IB REF 13 Ġby Ġconc aten ating Ġthe Ġprevious Ġcontext Ġvector Ġinto Ġthe Ġcurrent Ġinput Ġalong Ġwith Ġthe Ġcharacter Ġembed ding Ġvector . ĠBy Ġusing Ġthose Ġmodifications , Ġwe Ġare Ġable Ġto Ġimprove Ġthe Ġbaseline Ġperformance . </s> Our Ġproposed ĠW av 2 Text Ġmodels Ġwithout Ġany Ġtransfer Ġlearning Ġfailed Ġto Ġconverge . ĠIn Ġcontrast , Ġwith Ġtransfer Ġlearning , Ġthey Ġsignificantly Ġsurpassed Ġthe Ġperformance Ġof Ġthe ĠC TC Ġand Ġenc oder - dec oder Ġfrom ĠMel - scale Ġspect rogram Ġfeatures . ĠThis Ġsuggests Ġthat Ġby Ġusing Ġtransfer Ġlearning Ġfor Ġinitial izing Ġthe Ġlower Ġpart Ġof Ġthe Ġenc oder Ġparameters , Ġour Ġmodel Ġalso Ġperformed Ġbetter Ġthen Ġtheir Ġoriginal Ġfeatures . </s> Related ĠWork </s> Transfer Ġlearning Ġis Ġthe Ġability Ġof Ġa Ġlearning Ġalgorithm Ġto Ġconvey Ġknowledge Ġacross Ġdifferent Ġtasks . ĠThe Ġinitial Ġidea Ġis Ġto Ġreuse Ġpreviously Ġobtained Ġknowledge Ġto Ġenhance Ġthe Ġlearning Ġfor Ġnew Ġthings . ĠThe Ġstandard Ġprocedure Ġare Ġ: Ġfirst , Ġtrain Ġthe Ġmodel Ġon Ġa Ġbase Ġdataset Ġand Ġtask , Ġthen Ġthe Ġlearned Ġfeatures Ġand / or Ġparameters Ġare Ġreused Ġfor Ġlearning Ġa Ġsecond Ġtarget Ġdataset Ġand Ġtask . ĠBeng io Ġet Ġal . ĠB IB REF 25 Ġprovided Ġdeep Ġreviews Ġabout Ġmulti - task Ġand Ġtransfer Ġlearning Ġon Ġdeep Ġlearning Ġmodels . ĠJason Ġet Ġal . ĠB IB REF 26 Ġshowed Ġthat Ġa Ġmodel Ġwith Ġtransferred Ġparameter Ġconsistently Ġoutper formed Ġa Ġrandomly Ġinitialized Ġone . </s> In Ġspeech Ġrecognition Ġresearch , Ġtransfer Ġlearning Ġhas Ġbeen Ġstudied Ġfor Ġmany Ġyears , Ġincluding Ġsuccessful Ġcases Ġof Ġspeaker Ġadaptation Ġand Ġcross - ling ual Ġacoustic Ġmodeling ĠB IB REF 27 Ġ. ĠOne Ġpopular Ġscheme Ġfor Ġutilizing ĠD NN s Ġfor Ġtransfer Ġlearning Ġwithin ĠAS R Ġframeworks Ġis Ġa Ġtandem Ġapproach ĠB IB REF 28 Ġ. ĠThis Ġidea Ġfirst Ġtrains Ġa ĠD NN Ġwith Ġa Ġnarrow Ġhidden Ġbottleneck Ġlayer Ġto Ġperform Ġphon eme Ġclassification Ġat Ġthe Ġframe Ġlevel Ġand Ġthen Ġre uses Ġthe Ġactiv ations Ġfrom Ġthe Ġnarrow Ġhidden Ġbottleneck Ġlayer Ġas Ġdiscrim in ative Ġfeatures Ġin Ġconventional ĠGM M - H MM Ġor Ġhybrid ĠD NN - H MM Ġmodels ĠB IB REF 29 Ġ. ĠAnother Ġstudy Ġintroduced Ġa Ġconv olution al Ġbottleneck Ġnetwork Ġas Ġan Ġalternative Ġtandem Ġbottleneck Ġfeature Ġarchitecture ĠB IB REF 30 Ġ. ĠHowever , Ġalthough Ġsuch Ġa Ġfeature Ġtransfer Ġlearning Ġframework Ġprovides Ġmany Ġadvantages Ġin ĠAS R , Ġthe Ġusage Ġin Ġan Ġend - to - end Ġattention - based ĠAS R Ġframework Ġhas Ġnot Ġbeen Ġexplored . </s> This Ġstudy Ġperforms Ġfeature Ġtransfer Ġlearning Ġon Ġthe Ġenc oder Ġpart Ġof Ġthe Ġend - to - end Ġattention - based ĠAS R Ġarchitecture . ĠWe Ġtrain Ġthe Ġconv olution al Ġenc oder Ġto Ġpredict Ġthe Ġspectral Ġfeatures Ġgiven Ġthe Ġcorresponding Ġraw Ġspeech Ġwave form . ĠAfter Ġthat , Ġwe Ġtransfer Ġthe Ġtrained Ġlayers Ġand Ġparameters , Ġintegrate Ġthem Ġwith Ġthe ĠL ST M Ġenc oder - dec oder , Ġand Ġeventually Ġoptimize Ġthe Ġwhole Ġstructure Ġto Ġpredict Ġthe Ġcorrect Ġoutput Ġtext Ġtranscription Ġgiven Ġthe Ġraw Ġspeech Ġwave form . </s> Conclusion </s> This Ġpaper Ġdescribed Ġthe Ġfirst Ġattempt Ġto Ġbuild Ġan Ġend - to - end Ġattention - based Ġenc oder - dec oder Ġspeech Ġrecognition Ġthat Ġdirectly Ġpredicts Ġthe Ġtext Ġtranscription Ġgiven Ġraw Ġspeech Ġinput . ĠWe Ġalso Ġproposed Ġfeature Ġtransfer Ġlearning Ġto Ġassist Ġthe Ġenc oder - dec oder Ġmodel Ġtraining Ġprocess Ġand Ġpresented Ġa Ġnovel Ġarchitecture Ġthat Ġcombined Ġconv olution al , ĠN IN Ġand ĠBi - L ST M Ġlayers Ġinto Ġa Ġsingle Ġenc oder Ġpart Ġfor Ġraw Ġspeech Ġrecognition . ĠOur Ġresults Ġsuggest Ġthat Ġtransfer Ġlearning Ġis Ġa Ġvery Ġhelpful Ġmethod Ġfor Ġconstructing Ġan Ġend - to - end Ġsystem Ġfrom Ġsuch Ġlow - level Ġfeatures Ġas Ġraw Ġspeech Ġsignals . ĠWith Ġtransferred Ġparameters , Ġour Ġproposed Ġattention - based ĠW av 2 Text Ġmodels Ġconver ged Ġand Ġmatched Ġthe Ġperformance Ġwith Ġthe Ġattention - based Ġenc oder - dec oder Ġmodel Ġtrained Ġon Ġstandard Ġspectral - based Ġfeatures . ĠThe Ġbest Ġperformance Ġwas Ġachieved Ġby ĠW av 2 Text Ġmodels Ġwith Ġtransfer Ġlearning Ġfrom Ġmulti Ġtarget Ġscheme . </s> Acknowled gements </s> Part Ġof Ġthis Ġwork Ġwas Ġsupported Ġby ĠJS PS ĠK AK EN HI ĠGrant ĠNumbers ĠJP 17 H 06 101 Ġand ĠJP Ġ17 K 00 237 .
Introduction Conventional large-vocabulary continuous speech recognition (LVCSR) systems typically perform multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units such as sub-words (phonemes), words, and strings of words (sentences). Such systems basically consist of several sub-components (feature extractor, acoustic model, pronunciation lexicon, language model) that are trained and tuned separately BIBREF0 . First, the speech signal is processed into a set of observation features based on a carefully hand-crafted feature extractor, such as Mel frequency cepstral coefficients (MFCC) or Mel-scale spectrogram. Then the acoustic model classifies the observation features into sub-unit or phoneme classes. Finally, the search algorithm finds the most probable word sequence based on the evidence of the acoustic model, the lexicon, and the language model. But, it is widely known that information loss in the earlier stage can propagate through the later stages. Deep learning algorithms have produced many state-of-the-art performances in various tasks that have revitalized the use of neural networks for ASR. One of the important factors behind the popularity of deep learning is the possibility of simplifying many complicated hand-engineered models by letting DNNs find their way to map from input to output spaces. Interest has emerged recently in the possibility of learning DNN-based acoustic models directly from the raw speech waveform without any predefined alignments and hand-engineered models. In this way, the feature extractor and acoustic model can be integrated into a single architecture. Palaz et al. BIBREF1 , BIBREF2 proposed a convolutional neural network (CNN) to directly train an acoustic model from the raw speech waveform. Sainath et al. BIBREF3 used time-convolutional layers over raw speech and trained them jointly with the long short-term memory deep neural network (CLDNN) acoustic model. The results showed that raw waveform CLDNNs matched the performance of log-mel CLDNNs on a voice search task. Ghahremani et al. BIBREF4 recently proposed a CNN time-delay neural network (CNN-TDNN) with network-in-network (NIN) architecture, and also showed that their model outperformed MFCC-based TDNN on the Wall Street Journal (WSJ) BIBREF5 task. But despite significant progress that has been made, the successful models were mostly demonstrated only within the hybrid DNN-HMM speech recognition frameworks. On the other hand, some existing works constructed end-to-end neural network models for ASR and replaced the acoustic model, the lexicon model, and the language model with a single integrated model, thus simplifying the pipeline. Graves et al. BIBREF6 , BIBREF7 successfully built an end-to-end ASR based on the connectionist temporal classification (CTC) framework. Amodei et al. BIBREF8 also constructed an end-to-end CTC-based ASR that directly produced character strings instead of phoneme sequences. But the CTC-based architecture still predicts the target outputs for every frame without any implicit knowledge about the language model. Another approach uses a sequence-to-sequence attention-based encoder-decoder that explicitly uses the history of previous outputs. Chorowski et al. BIBREF9 and Chan et al. BIBREF10 has successfully demonstrated encoder-decoder based ASR frameworks. Unfortunately, most of these works still used the standard spectral features (i.e., Mel-scale spectrogram, MFCC) as the input. The only attempt on end-to-end speech recognition for a raw waveform was recently proposed by BIBREF11 . Their system used a deep CNN and was trained with the automatic segmentation criterion (ASG) as an alternative to CTC. However, similar with CTC, the model did not explicitly use the history of the previous outputs assuming they were conditionally independent of each other. Furthermore, its performance was only reported using a very large data set (about 1000h of audio files). To the best of our knowledge, few studies have explored a single end-to-end ASR architecture trained on raw speech waveforms to directly output text transcription, and none of those models were built based on an encoder-decoder architecture. In this paper, we take a step forward to construct an end-to-end ASR using an attentional-based encoder-decoder model for processing raw speech waveform, naming it as “Attention-based Wav2Text". We investigate the performance of our proposed models on standard ASR datasets. In practice, optimizing an encoder-decoder framework is more difficult than a standard neural network architecture BIBREF10 . Therefore, we propose a feature transfer learning method to assist the training process for our end-to-end attention-based ASR model. Attention-based Encoder Decoder for Raw Speech Recognition The encoder-decoder model is a neural network that directly models conditional probability INLINEFORM0 where INLINEFORM1 is the source sequence with length INLINEFORM2 and INLINEFORM3 is the target sequence with length INLINEFORM4 . It consists of encoder, decoder and attention modules. The encoder task processes an input sequence INLINEFORM5 and outputs representative information INLINEFORM6 for the decoder. The attention module is an extension scheme that assists the decoder to find relevant information on the encoder side based on the current decoder hidden states BIBREF12 , BIBREF13 . Usually, the attention module produces context information INLINEFORM7 at time INLINEFORM8 based on the encoder and decoder hidden states: DISPLAYFORM0 There are several variations for score function : DISPLAYFORM0 where INLINEFORM0 , INLINEFORM1 is the number of hidden units for the encoder and INLINEFORM2 is the number of hidden units for the decoder. Finally, the decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information INLINEFORM4 , can be formulated as: DISPLAYFORM0 The most common input INLINEFORM0 for speech recognition tasks is a sequence of feature vectors such as log Mel-spectral spectrogram and/or MFCC. Therefore, INLINEFORM1 where D is the number of the features and S is the total length of the utterance in frames. The output INLINEFORM2 can be either phoneme or grapheme (character) sequence. In this work, we use the raw waveform as the input representation instead of spectral-based features and a grapheme (character) sequence as the output representation. In contrast to most encoder-decoder architectures, which are purely based on recurrent neural network (RNNs) framework, we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part. We use convolutional layers because they are suitable for extracting local information from raw speech. We use a striding mechanism to reduce the dimension from the input frames BIBREF17 , while the NIN layer represents more complex structures on the top of the convolutional layers. On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP) as described in Eq. EQREF2 . For more details, we illustrate our architecture in Figure FIGREF4 . Feature Transfer Learning Deep learning is well known for its ability to learn directly from low-level feature representation such as raw speech BIBREF1 , BIBREF3 . However, in most cases such models are already conditioned on a fixed input size and a single target output (i.e., predicting one phoneme class for each input frame). In the attention-based encoder-decoder model, the training process is not as easy as in a standard neural network model BIBREF10 because the attention-based model needs to jointly optimize three different modules simultaneously: (1) an encoder module for producing representative information from a source sequence; (2) an attention module for calculating the correct alignment; and (3) a decoder module for generating correct transcriptions. If one of these modules has difficulty fulfilling its own tasks, then the model will fail to produce good results. To ease the burden on training the whole encoder-decoder architecture directly to predict the text transcription given the raw speech waveform, we utilize a transfer learning method on the encoder part. Specifically, we only train the encoder's lower layers consisting of the convolutional and NIN layers to predict the spectral features given the corresponding raw waveform. In this work, we utilize two widely used spectral features: MFCC and log Mel-scale spectrogram as the transfer learning target. Figure FIGREF5 shows our feature transfer learning architecture. First, given segmented raw speech waveform INLINEFORM0 , we extract corresponding INLINEFORM1 -dimensional spectral features INLINEFORM2 . Then we process raw speech INLINEFORM3 with several convolutions, followed by NIN layers in the encoder part. In the last NIN-layer, we set a fixed number of channels as INLINEFORM4 channels and apply mean-pooling across time. Finally, we get predictions for corresponding spectral features INLINEFORM5 and optimize all of the parameters by minimizing the mean squared error between predicted spectral features INLINEFORM6 and target spectral features INLINEFORM7 : DISPLAYFORM0 In this paper, we also explore multi target feature transfer using a similar structure as in Figure FIGREF5 but with two parallel NIN layers, followed by mean-polling at the end. One of the output layers is used to predicts log Mel-scale spectrogram and another predicts MFCC features. We modify the single target loss function from Eq. EQREF6 into the following: DISPLAYFORM0 where INLINEFORM0 are the predicted Mel-scale spectrogram and the MFCC values, and INLINEFORM1 are the real Mel-scale spectrogram and MFCC features for frame INLINEFORM2 . After optimizing all the convolutional and NIN layer parameters, we transfer the trained layers and parameters and integrate them with the Bi-LSTM encoder. Finally, we jointly optimize the whole structure together. Speech Data In this study, we investigate the performance of our proposed models on WSJ BIBREF5 . We used the same definitions of the training, development and test set as the Kaldi s5 recipe BIBREF18 . The raw speech waveforms were segmented into multiple frames with a 25ms window size and a 10ms step size. We normalized the raw speech waveform into the range -1 to 1. For spectral based features such as MFCC and log Mel-spectrogram, we normalized the features for each dimension into zero mean and unit variance. For WSJ, we separated into two experiments by using WSJ-SI84 only and WSJ-SI284 data. We used dev_93 for our validation set and eval_92 for our test set. We used the character sequence as our decoder target and followed the preprocessing step proposed by BIBREF19 . The text from all the utterances was mapped into a 32-character set: 26 (a-z) alphabet, apostrophe, period, dash, space, noise, and “eos". Model Architectures Our attention-based Wav2Text architecture uses four convolutional layers, followed by two NIN layers at the lower part of the encoder module. For all the convolutional layers, we used a leaky rectifier unit (LReLU) BIBREF20 activation function with leakiness INLINEFORM0 . Inside the first NIN layers, we stacked three consecutive filters with LReLU activation function. For the second NIN layers, we stacked two consecutive filters with tanh and identity activation function. For the feature transfer learning training phase, we used Momentum SGD with a learning rate of 0.01 and momentum of 0.9. Table TABREF11 summarizes the details of the layer settings for the convolutional and NIN layers. On the top layers of the encoder after the transferred convolutional and NIN layers, we put three bidirectional LSTMs (Bi-LSTM) with 256 hidden units (total 512 units for both directions). To reduce the computational time, we used hierarchical subsampling BIBREF21 , BIBREF22 , BIBREF10 . We applied subsampling on all the Bi-LSTM layers and reduced the length by a factor of 8. On the decoder side, the previous input phonemes / characters were converted into real vectors by a 128-dimensional embedding matrix. We used one unidirectional LSTM with 512 hidden units and followed by a softmax layer to output the character probability. For the end-to-end training phase, we froze the parameter values from the transferred layers from epoch 0 to epoch 10, and after epoch 10 we jointly optimized all the parameters together until the end of training (a total 40 epochs). We used an Adam BIBREF23 optimizer with a learning rate of 0.0005. In the decoding phase, we used a beam search strategy with beam size INLINEFORM0 and we adjusted the score by dividing with the transcription length to prevent the decoder from favoring shorter transcriptions. We did not use any language model or lexicon dictionary for decoding. All of our models were implemented on the PyTorch framework . For comparison, we also evaluated the standard attention-based encoder decoder with Mel-scale spectrogram input as the baseline. Here, we used similar settings as the proposed model, except we replaced the convolutional and NIN layers with a feedforward layer (512 hidden units). Result An example of our transfer learning results is shown in Figure FIGREF8 , and Table TABREF14 shows the speech recognition performance in CER for both the WSJ-SI84 and WSJ-SI284 datasets. We compared our method with several published models like CTC, Attention Encoder-Decoder and Joint CTC-Attention model that utilize CTC for training the encoder part. Besides, we also train our own baseline Attention Encoder-Decoder with Mel-scale spectrogram. The difference between our Attention Encoder-Decoder (“Att Enc-Dec (ours)", “Att Enc-Dec Wav2Text") with Attention Encoder-Decoder from BIBREF24 (“Att Enc-Dec Content", “Att Enc-Dec Location") is we used the current hidden states to generate the attention vector instead of the previous hidden states. Another addition is we utilized “input feedback" method BIBREF13 by concatenating the previous context vector into the current input along with the character embedding vector. By using those modifications, we are able to improve the baseline performance. Our proposed Wav2Text models without any transfer learning failed to converge. In contrast, with transfer learning, they significantly surpassed the performance of the CTC and encoder-decoder from Mel-scale spectrogram features. This suggests that by using transfer learning for initializing the lower part of the encoder parameters, our model also performed better then their original features. Related Work Transfer learning is the ability of a learning algorithm to convey knowledge across different tasks. The initial idea is to reuse previously obtained knowledge to enhance the learning for new things. The standard procedure are : first, train the model on a base dataset and task, then the learned features and/or parameters are reused for learning a second target dataset and task. Bengio et al. BIBREF25 provided deep reviews about multi-task and transfer learning on deep learning models. Jason et al. BIBREF26 showed that a model with transferred parameter consistently outperformed a randomly initialized one. In speech recognition research, transfer learning has been studied for many years, including successful cases of speaker adaptation and cross-lingual acoustic modeling BIBREF27 . One popular scheme for utilizing DNNs for transfer learning within ASR frameworks is a tandem approach BIBREF28 . This idea first trains a DNN with a narrow hidden bottleneck layer to perform phoneme classification at the frame level and then reuses the activations from the narrow hidden bottleneck layer as discriminative features in conventional GMM-HMM or hybrid DNN-HMM models BIBREF29 . Another study introduced a convolutional bottleneck network as an alternative tandem bottleneck feature architecture BIBREF30 . However, although such a feature transfer learning framework provides many advantages in ASR, the usage in an end-to-end attention-based ASR framework has not been explored. This study performs feature transfer learning on the encoder part of the end-to-end attention-based ASR architecture. We train the convolutional encoder to predict the spectral features given the corresponding raw speech waveform. After that, we transfer the trained layers and parameters, integrate them with the LSTM encoder-decoder, and eventually optimize the whole structure to predict the correct output text transcription given the raw speech waveform. Conclusion This paper described the first attempt to build an end-to-end attention-based encoder-decoder speech recognition that directly predicts the text transcription given raw speech input. We also proposed feature transfer learning to assist the encoder-decoder model training process and presented a novel architecture that combined convolutional, NIN and Bi-LSTM layers into a single encoder part for raw speech recognition. Our results suggest that transfer learning is a very helpful method for constructing an end-to-end system from such low-level features as raw speech signals. With transferred parameters, our proposed attention-based Wav2Text models converged and matched the performance with the attention-based encoder-decoder model trained on standard spectral-based features. The best performance was achieved by Wav2Text models with transfer learning from multi target scheme. Acknowledgements Part of this work was supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP 17K00237.
Which architecture do they use for the encoder and decoder?
1b23c4535a6c10eb70bbc95313c465e4a547db5e
Which Ġarchitecture Ġdo Ġthey Ġuse Ġfor Ġthe Ġenc oder Ġand Ġdec oder ?
we Ġconstruct Ġan Ġenc oder Ġwith Ġseveral Ġconv olution al Ġlayers ĠB IB REF 14 Ġfollowed Ġby ĠN IN Ġlayers ĠB IB REF 15 Ġas Ġthe Ġlower Ġpart Ġin Ġthe Ġenc oder Ġand Ġintegrate Ġthem Ġwith Ġdeep Ġbid irection al Ġlong Ġshort - term Ġmemory Ġ( Bi - L ST M ) ĠB IB REF 16 Ġat Ġthe Ġhigher Ġpart , ĠOn Ġthe Ġdec oder Ġside , Ġwe Ġuse Ġa Ġstandard Ġdeep Ġun id irection al ĠL ST M Ġwith Ġglobal Ġattention ĠB IB REF 13 Ġthat Ġis Ġcalculated Ġby Ġa Ġmulti - layer Ġpercept ron Ġ( ML P )
<s> Which Ġarchitecture Ġdo Ġthey Ġuse Ġfor Ġthe Ġenc oder Ġand Ġdec oder ? </s> Introduction </s> Con ventional Ġlarge - voc abulary Ġcontinuous Ġspeech Ġrecognition Ġ( LV CS R ) Ġsystems Ġtypically Ġperform Ġmulti - level Ġpattern Ġrecognition Ġtasks Ġthat Ġmap Ġthe Ġacoustic Ġspeech Ġwave form Ġinto Ġa Ġhierarchy Ġof Ġspeech Ġunits Ġsuch Ġas Ġsub - words Ġ( ph on em es ), Ġwords , Ġand Ġstrings Ġof Ġwords Ġ( sent ences ). ĠSuch Ġsystems Ġbasically Ġconsist Ġof Ġseveral Ġsub - comp onents Ġ( feature Ġextract or , Ġacoustic Ġmodel , Ġpronunciation Ġlex icon , Ġlanguage Ġmodel ) Ġthat Ġare Ġtrained Ġand Ġtuned Ġseparately ĠB IB REF 0 Ġ. ĠFirst , Ġthe Ġspeech Ġsignal Ġis Ġprocessed Ġinto Ġa Ġset Ġof Ġobservation Ġfeatures Ġbased Ġon Ġa Ġcarefully Ġhand - crafted Ġfeature Ġextract or , Ġsuch Ġas ĠMel Ġfrequency Ġc ep st ral Ġcoefficients Ġ( MF CC ) Ġor ĠMel - scale Ġspect rogram . ĠThen Ġthe Ġacoustic Ġmodel Ġclass ifies Ġthe Ġobservation Ġfeatures Ġinto Ġsub - unit Ġor Ġphon eme Ġclasses . ĠFinally , Ġthe Ġsearch Ġalgorithm Ġfinds Ġthe Ġmost Ġprobable Ġword Ġsequence Ġbased Ġon Ġthe Ġevidence Ġof Ġthe Ġacoustic Ġmodel , Ġthe Ġlex icon , Ġand Ġthe Ġlanguage Ġmodel . ĠBut , Ġit Ġis Ġwidely Ġknown Ġthat Ġinformation Ġloss Ġin Ġthe Ġearlier Ġstage Ġcan Ġpropagate Ġthrough Ġthe Ġlater Ġstages . </s> Deep Ġlearning Ġalgorithms Ġhave Ġproduced Ġmany Ġstate - of - the - art Ġperformances Ġin Ġvarious Ġtasks Ġthat Ġhave Ġrevital ized Ġthe Ġuse Ġof Ġneural Ġnetworks Ġfor ĠAS R . ĠOne Ġof Ġthe Ġimportant Ġfactors Ġbehind Ġthe Ġpopularity Ġof Ġdeep Ġlearning Ġis Ġthe Ġpossibility Ġof Ġsimpl ifying Ġmany Ġcomplicated Ġhand - engine ered Ġmodels Ġby Ġletting ĠD NN s Ġfind Ġtheir Ġway Ġto Ġmap Ġfrom Ġinput Ġto Ġoutput Ġspaces . ĠInterest Ġhas Ġemerged Ġrecently Ġin Ġthe Ġpossibility Ġof Ġlearning ĠD NN - based Ġacoustic Ġmodels Ġdirectly Ġfrom Ġthe Ġraw Ġspeech Ġwave form Ġwithout Ġany Ġpred efined Ġalign ments Ġand Ġhand - engine ered Ġmodels . ĠIn Ġthis Ġway , Ġthe Ġfeature Ġextract or Ġand Ġacoustic Ġmodel Ġcan Ġbe Ġintegrated Ġinto Ġa Ġsingle Ġarchitecture . ĠPal az Ġet Ġal . ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġproposed Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġto Ġdirectly Ġtrain Ġan Ġacoustic Ġmodel Ġfrom Ġthe Ġraw Ġspeech Ġwave form . ĠS ain ath Ġet Ġal . ĠB IB REF 3 Ġused Ġtime - conv olution al Ġlayers Ġover Ġraw Ġspeech Ġand Ġtrained Ġthem Ġjointly Ġwith Ġthe Ġlong Ġshort - term Ġmemory Ġdeep Ġneural Ġnetwork Ġ( CL D NN ) Ġacoustic Ġmodel . ĠThe Ġresults Ġshowed Ġthat Ġraw Ġwave form ĠCL D NN s Ġmatched Ġthe Ġperformance Ġof Ġlog - mel ĠCL D NN s Ġon Ġa Ġvoice Ġsearch Ġtask . ĠGh ah re man i Ġet Ġal . ĠB IB REF 4 Ġrecently Ġproposed Ġa ĠCNN Ġtime - delay Ġneural Ġnetwork Ġ( CNN - TD NN ) Ġwith Ġnetwork - in - network Ġ( N IN ) Ġarchitecture , Ġand Ġalso Ġshowed Ġthat Ġtheir Ġmodel Ġoutper formed ĠMF CC - based ĠTD NN Ġon Ġthe ĠWall ĠStreet ĠJournal Ġ( WS J ) ĠB IB REF 5 Ġtask . ĠBut Ġdespite Ġsignificant Ġprogress Ġthat Ġhas Ġbeen Ġmade , Ġthe Ġsuccessful Ġmodels Ġwere Ġmostly Ġdemonstrated Ġonly Ġwithin Ġthe Ġhybrid ĠD NN - H MM Ġspeech Ġrecognition Ġframeworks . </s> On Ġthe Ġother Ġhand , Ġsome Ġexisting Ġworks Ġconstructed Ġend - to - end Ġneural Ġnetwork Ġmodels Ġfor ĠAS R Ġand Ġreplaced Ġthe Ġacoustic Ġmodel , Ġthe Ġlex icon Ġmodel , Ġand Ġthe Ġlanguage Ġmodel Ġwith Ġa Ġsingle Ġintegrated Ġmodel , Ġthus Ġsimpl ifying Ġthe Ġpipeline . ĠGraves Ġet Ġal . ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġsuccessfully Ġbuilt Ġan Ġend - to - end ĠAS R Ġbased Ġon Ġthe Ġconnection ist Ġtemporal Ġclassification Ġ( CT C ) Ġframework . ĠAm ode i Ġet Ġal . ĠB IB REF 8 Ġalso Ġconstructed Ġan Ġend - to - end ĠC TC - based ĠAS R Ġthat Ġdirectly Ġproduced Ġcharacter Ġstrings Ġinstead Ġof Ġphon eme Ġsequences . ĠBut Ġthe ĠC TC - based Ġarchitecture Ġstill Ġpredicts Ġthe Ġtarget Ġoutputs Ġfor Ġevery Ġframe Ġwithout Ġany Ġimplicit Ġknowledge Ġabout Ġthe Ġlanguage Ġmodel . ĠAnother Ġapproach Ġuses Ġa Ġsequence - to - sequence Ġattention - based Ġenc oder - dec oder Ġthat Ġexplicitly Ġuses Ġthe Ġhistory Ġof Ġprevious Ġoutputs . ĠCh or owski Ġet Ġal . ĠB IB REF 9 Ġand ĠChan Ġet Ġal . ĠB IB REF 10 Ġhas Ġsuccessfully Ġdemonstrated Ġenc oder - dec oder Ġbased ĠAS R Ġframeworks . ĠUnfortunately , Ġmost Ġof Ġthese Ġworks Ġstill Ġused Ġthe Ġstandard Ġspectral Ġfeatures Ġ( i . e ., ĠMel - scale Ġspect rogram , ĠMF CC ) Ġas Ġthe Ġinput . ĠThe Ġonly Ġattempt Ġon Ġend - to - end Ġspeech Ġrecognition Ġfor Ġa Ġraw Ġwave form Ġwas Ġrecently Ġproposed Ġby ĠB IB REF 11 Ġ. ĠTheir Ġsystem Ġused Ġa Ġdeep ĠCNN Ġand Ġwas Ġtrained Ġwith Ġthe Ġautomatic Ġsegment ation Ġcriterion Ġ( AS G ) Ġas Ġan Ġalternative Ġto ĠC TC . ĠHowever , Ġsimilar Ġwith ĠC TC , Ġthe Ġmodel Ġdid Ġnot Ġexplicitly Ġuse Ġthe Ġhistory Ġof Ġthe Ġprevious Ġoutputs Ġassuming Ġthey Ġwere Ġcondition ally Ġindependent Ġof Ġeach Ġother . ĠFurthermore , Ġits Ġperformance Ġwas Ġonly Ġreported Ġusing Ġa Ġvery Ġlarge Ġdata Ġset Ġ( about Ġ1000 h Ġof Ġaudio Ġfiles ). </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġfew Ġstudies Ġhave Ġexplored Ġa Ġsingle Ġend - to - end ĠAS R Ġarchitecture Ġtrained Ġon Ġraw Ġspeech Ġwave forms Ġto Ġdirectly Ġoutput Ġtext Ġtranscription , Ġand Ġnone Ġof Ġthose Ġmodels Ġwere Ġbuilt Ġbased Ġon Ġan Ġenc oder - dec oder Ġarchitecture . ĠIn Ġthis Ġpaper , Ġwe Ġtake Ġa Ġstep Ġforward Ġto Ġconstruct Ġan Ġend - to - end ĠAS R Ġusing Ġan Ġattention al - based Ġenc oder - dec oder Ġmodel Ġfor Ġprocessing Ġraw Ġspeech Ġwave form , Ġnaming Ġit Ġas ĠâĢ ľ Att ention - based ĠW av 2 Text ". ĠWe Ġinvestigate Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġon Ġstandard ĠAS R Ġdatasets . ĠIn Ġpractice , Ġoptimizing Ġan Ġenc oder - dec oder Ġframework Ġis Ġmore Ġdifficult Ġthan Ġa Ġstandard Ġneural Ġnetwork Ġarchitecture ĠB IB REF 10 Ġ. ĠTherefore , Ġwe Ġpropose Ġa Ġfeature Ġtransfer Ġlearning Ġmethod Ġto Ġassist Ġthe Ġtraining Ġprocess Ġfor Ġour Ġend - to - end Ġattention - based ĠAS R Ġmodel . </s> Att ention - based ĠEnc oder ĠDecoder Ġfor ĠRaw ĠSpeech ĠRecogn ition </s> The Ġenc oder - dec oder Ġmodel Ġis Ġa Ġneural Ġnetwork Ġthat Ġdirectly Ġmodels Ġconditional Ġprobability ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġsource Ġsequence Ġwith Ġlength ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġtarget Ġsequence Ġwith Ġlength ĠIN LINE FORM 4 Ġ. ĠIt Ġconsists Ġof Ġenc oder , Ġdec oder Ġand Ġattention Ġmodules . ĠThe Ġenc oder Ġtask Ġprocesses Ġan Ġinput Ġsequence ĠIN LINE FORM 5 Ġand Ġoutputs Ġrepresentative Ġinformation ĠIN LINE FORM 6 Ġfor Ġthe Ġdec oder . ĠThe Ġattention Ġmodule Ġis Ġan Ġextension Ġscheme Ġthat Ġassists Ġthe Ġdec oder Ġto Ġfind Ġrelevant Ġinformation Ġon Ġthe Ġenc oder Ġside Ġbased Ġon Ġthe Ġcurrent Ġdec oder Ġhidden Ġstates ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠUsually , Ġthe Ġattention Ġmodule Ġproduces Ġcontext Ġinformation ĠIN LINE FORM 7 Ġat Ġtime ĠIN LINE FORM 8 Ġbased Ġon Ġthe Ġenc oder Ġand Ġdec oder Ġhidden Ġstates : ĠDIS PLAY FORM 0 </s> There Ġare Ġseveral Ġvariations Ġfor Ġscore Ġfunction Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġis Ġthe Ġnumber Ġof Ġhidden Ġunits Ġfor Ġthe Ġenc oder Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġhidden Ġunits Ġfor Ġthe Ġdec oder . ĠFinally , Ġthe Ġdec oder Ġtask , Ġwhich Ġpredicts Ġthe Ġtarget Ġsequence Ġprobability Ġat Ġtime ĠIN LINE FORM 3 Ġbased Ġon Ġprevious Ġoutput Ġand Ġcontext Ġinformation ĠIN LINE FORM 4 Ġ, Ġcan Ġbe Ġformulated Ġas : ĠDIS PLAY FORM 0 </s> The Ġmost Ġcommon Ġinput ĠIN LINE FORM 0 Ġfor Ġspeech Ġrecognition Ġtasks Ġis Ġa Ġsequence Ġof Ġfeature Ġvectors Ġsuch Ġas Ġlog ĠMel - spect ral Ġspect rogram Ġand / or ĠMF CC . ĠTherefore , ĠIN LINE FORM 1 Ġwhere ĠD Ġis Ġthe Ġnumber Ġof Ġthe Ġfeatures Ġand ĠS Ġis Ġthe Ġtotal Ġlength Ġof Ġthe Ġutter ance Ġin Ġframes . ĠThe Ġoutput ĠIN LINE FORM 2 Ġcan Ġbe Ġeither Ġphon eme Ġor Ġgrap he me Ġ( character ) Ġsequence . </s> In Ġthis Ġwork , Ġwe Ġuse Ġthe Ġraw Ġwave form Ġas Ġthe Ġinput Ġrepresentation Ġinstead Ġof Ġspectral - based Ġfeatures Ġand Ġa Ġgrap he me Ġ( character ) Ġsequence Ġas Ġthe Ġoutput Ġrepresentation . ĠIn Ġcontrast Ġto Ġmost Ġenc oder - dec oder Ġarchitectures , Ġwhich Ġare Ġpurely Ġbased Ġon Ġrecurrent Ġneural Ġnetwork Ġ( R NN s ) Ġframework , Ġwe Ġconstruct Ġan Ġenc oder Ġwith Ġseveral Ġconv olution al Ġlayers ĠB IB REF 14 Ġfollowed Ġby ĠN IN Ġlayers ĠB IB REF 15 Ġas Ġthe Ġlower Ġpart Ġin Ġthe Ġenc oder Ġand Ġintegrate Ġthem Ġwith Ġdeep Ġbid irection al Ġlong Ġshort - term Ġmemory Ġ( Bi - L ST M ) ĠB IB REF 16 Ġat Ġthe Ġhigher Ġpart . ĠWe Ġuse Ġconv olution al Ġlayers Ġbecause Ġthey Ġare Ġsuitable Ġfor Ġextracting Ġlocal Ġinformation Ġfrom Ġraw Ġspeech . ĠWe Ġuse Ġa Ġstr iding Ġmechanism Ġto Ġreduce Ġthe Ġdimension Ġfrom Ġthe Ġinput Ġframes ĠB IB REF 17 Ġ, Ġwhile Ġthe ĠN IN Ġlayer Ġrepresents Ġmore Ġcomplex Ġstructures Ġon Ġthe Ġtop Ġof Ġthe Ġconv olution al Ġlayers . ĠOn Ġthe Ġdec oder Ġside , Ġwe Ġuse Ġa Ġstandard Ġdeep Ġun id irection al ĠL ST M Ġwith Ġglobal Ġattention ĠB IB REF 13 Ġthat Ġis Ġcalculated Ġby Ġa Ġmulti - layer Ġpercept ron Ġ( ML P ) Ġas Ġdescribed Ġin ĠE q . ĠEQ REF 2 Ġ. ĠFor Ġmore Ġdetails , Ġwe Ġillustrate Ġour Ġarchitecture Ġin ĠFigure ĠFIG REF 4 Ġ. </s> Feature ĠTransfer ĠLearning </s> Deep Ġlearning Ġis Ġwell Ġknown Ġfor Ġits Ġability Ġto Ġlearn Ġdirectly Ġfrom Ġlow - level Ġfeature Ġrepresentation Ġsuch Ġas Ġraw Ġspeech ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ. ĠHowever , Ġin Ġmost Ġcases Ġsuch Ġmodels Ġare Ġalready Ġconditioned Ġon Ġa Ġfixed Ġinput Ġsize Ġand Ġa Ġsingle Ġtarget Ġoutput Ġ( i . e ., Ġpredicting Ġone Ġphon eme Ġclass Ġfor Ġeach Ġinput Ġframe ). ĠIn Ġthe Ġattention - based Ġenc oder - dec oder Ġmodel , Ġthe Ġtraining Ġprocess Ġis Ġnot Ġas Ġeasy Ġas Ġin Ġa Ġstandard Ġneural Ġnetwork Ġmodel ĠB IB REF 10 Ġbecause Ġthe Ġattention - based Ġmodel Ġneeds Ġto Ġjointly Ġoptimize Ġthree Ġdifferent Ġmodules Ġsimultaneously : Ġ( 1 ) Ġan Ġenc oder Ġmodule Ġfor Ġproducing Ġrepresentative Ġinformation Ġfrom Ġa Ġsource Ġsequence ; Ġ( 2 ) Ġan Ġattention Ġmodule Ġfor Ġcalculating Ġthe Ġcorrect Ġalignment ; Ġand Ġ( 3 ) Ġa Ġdec oder Ġmodule Ġfor Ġgenerating Ġcorrect Ġtranscript ions . ĠIf Ġone Ġof Ġthese Ġmodules Ġhas Ġdifficulty Ġfulfilling Ġits Ġown Ġtasks , Ġthen Ġthe Ġmodel Ġwill Ġfail Ġto Ġproduce Ġgood Ġresults . </s> To Ġease Ġthe Ġburden Ġon Ġtraining Ġthe Ġwhole Ġenc oder - dec oder Ġarchitecture Ġdirectly Ġto Ġpredict Ġthe Ġtext Ġtranscription Ġgiven Ġthe Ġraw Ġspeech Ġwave form , Ġwe Ġutilize Ġa Ġtransfer Ġlearning Ġmethod Ġon Ġthe Ġenc oder Ġpart . ĠSpecifically , Ġwe Ġonly Ġtrain Ġthe Ġenc oder 's Ġlower Ġlayers Ġconsisting Ġof Ġthe Ġconv olution al Ġand ĠN IN Ġlayers Ġto Ġpredict Ġthe Ġspectral Ġfeatures Ġgiven Ġthe Ġcorresponding Ġraw Ġwave form . ĠIn Ġthis Ġwork , Ġwe Ġutilize Ġtwo Ġwidely Ġused Ġspectral Ġfeatures : ĠMF CC Ġand Ġlog ĠMel - scale Ġspect rogram Ġas Ġthe Ġtransfer Ġlearning Ġtarget . ĠFigure ĠFIG REF 5 Ġshows Ġour Ġfeature Ġtransfer Ġlearning Ġarchitecture . ĠFirst , Ġgiven Ġsegment ed Ġraw Ġspeech Ġwave form ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġcorresponding ĠIN LINE FORM 1 Ġ- dimensional Ġspectral Ġfeatures ĠIN LINE FORM 2 Ġ. ĠThen Ġwe Ġprocess Ġraw Ġspeech ĠIN LINE FORM 3 Ġwith Ġseveral Ġconv olutions , Ġfollowed Ġby ĠN IN Ġlayers Ġin Ġthe Ġenc oder Ġpart . ĠIn Ġthe Ġlast ĠN IN - layer , Ġwe Ġset Ġa Ġfixed Ġnumber Ġof Ġchannels Ġas ĠIN LINE FORM 4 Ġchannels Ġand Ġapply Ġmean - pool ing Ġacross Ġtime . ĠFinally , Ġwe Ġget Ġpredictions Ġfor Ġcorresponding Ġspectral Ġfeatures ĠIN LINE FORM 5 Ġand Ġoptimize Ġall Ġof Ġthe Ġparameters Ġby Ġminimizing Ġthe Ġmean Ġsquared Ġerror Ġbetween Ġpredicted Ġspectral Ġfeatures ĠIN LINE FORM 6 Ġand Ġtarget Ġspectral Ġfeatures ĠIN LINE FORM 7 Ġ: ĠDIS PLAY FORM 0 </s> In Ġthis Ġpaper , Ġwe Ġalso Ġexplore Ġmulti Ġtarget Ġfeature Ġtransfer Ġusing Ġa Ġsimilar Ġstructure Ġas Ġin ĠFigure ĠFIG REF 5 Ġbut Ġwith Ġtwo Ġparallel ĠN IN Ġlayers , Ġfollowed Ġby Ġmean - poll ing Ġat Ġthe Ġend . ĠOne Ġof Ġthe Ġoutput Ġlayers Ġis Ġused Ġto Ġpredicts Ġlog ĠMel - scale Ġspect rogram Ġand Ġanother Ġpredicts ĠMF CC Ġfeatures . ĠWe Ġmodify Ġthe Ġsingle Ġtarget Ġloss Ġfunction Ġfrom ĠE q . ĠEQ REF 6 Ġinto Ġthe Ġfollowing : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġthe Ġpredicted ĠMel - scale Ġspect rogram Ġand Ġthe ĠMF CC Ġvalues , Ġand ĠIN LINE FORM 1 Ġare Ġthe Ġreal ĠMel - scale Ġspect rogram Ġand ĠMF CC Ġfeatures Ġfor Ġframe ĠIN LINE FORM 2 Ġ. ĠAfter Ġoptimizing Ġall Ġthe Ġconv olution al Ġand ĠN IN Ġlayer Ġparameters , Ġwe Ġtransfer Ġthe Ġtrained Ġlayers Ġand Ġparameters Ġand Ġintegrate Ġthem Ġwith Ġthe ĠBi - L ST M Ġenc oder . ĠFinally , Ġwe Ġjointly Ġoptimize Ġthe Ġwhole Ġstructure Ġtogether . </s> Spe ech ĠData </s> In Ġthis Ġstudy , Ġwe Ġinvestigate Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġon ĠWS J ĠB IB REF 5 Ġ. ĠWe Ġused Ġthe Ġsame Ġdefinitions Ġof Ġthe Ġtraining , Ġdevelopment Ġand Ġtest Ġset Ġas Ġthe ĠK aldi Ġs 5 Ġrecipe ĠB IB REF 18 Ġ. ĠThe Ġraw Ġspeech Ġwave forms Ġwere Ġsegment ed Ġinto Ġmultiple Ġframes Ġwith Ġa Ġ25 ms Ġwindow Ġsize Ġand Ġa Ġ10 ms Ġstep Ġsize . ĠWe Ġnormalized Ġthe Ġraw Ġspeech Ġwave form Ġinto Ġthe Ġrange Ġ- 1 Ġto Ġ1 . ĠFor Ġspectral Ġbased Ġfeatures Ġsuch Ġas ĠMF CC Ġand Ġlog ĠMel - spect rogram , Ġwe Ġnormalized Ġthe Ġfeatures Ġfor Ġeach Ġdimension Ġinto Ġzero Ġmean Ġand Ġunit Ġvariance . ĠFor ĠWS J , Ġwe Ġseparated Ġinto Ġtwo Ġexperiments Ġby Ġusing ĠWS J - SI 84 Ġonly Ġand ĠWS J - SI 284 Ġdata . ĠWe Ġused Ġdev _ 93 Ġfor Ġour Ġvalidation Ġset Ġand Ġeval _ 92 Ġfor Ġour Ġtest Ġset . ĠWe Ġused Ġthe Ġcharacter Ġsequence Ġas Ġour Ġdec oder Ġtarget Ġand Ġfollowed Ġthe Ġpre processing Ġstep Ġproposed Ġby ĠB IB REF 19 Ġ. ĠThe Ġtext Ġfrom Ġall Ġthe Ġutter ances Ġwas Ġmapped Ġinto Ġa Ġ32 - character Ġset : Ġ26 Ġ( a - z ) Ġalphabet , Ġapost rophe , Ġperiod , Ġdash , Ġspace , Ġnoise , Ġand ĠâĢ ľ e os ". </s> Model ĠArchitect ures </s> Our Ġattention - based ĠW av 2 Text Ġarchitecture Ġuses Ġfour Ġconv olution al Ġlayers , Ġfollowed Ġby Ġtwo ĠN IN Ġlayers Ġat Ġthe Ġlower Ġpart Ġof Ġthe Ġenc oder Ġmodule . ĠFor Ġall Ġthe Ġconv olution al Ġlayers , Ġwe Ġused Ġa Ġleak y Ġrect ifier Ġunit Ġ( L Re LU ) ĠB IB REF 20 Ġactivation Ġfunction Ġwith Ġleak iness ĠIN LINE FORM 0 Ġ. ĠInside Ġthe Ġfirst ĠN IN Ġlayers , Ġwe Ġstacked Ġthree Ġconsecutive Ġfilters Ġwith ĠL Re LU Ġactivation Ġfunction . ĠFor Ġthe Ġsecond ĠN IN Ġlayers , Ġwe Ġstacked Ġtwo Ġconsecutive Ġfilters Ġwith Ġtan h Ġand Ġidentity Ġactivation Ġfunction . ĠFor Ġthe Ġfeature Ġtransfer Ġlearning Ġtraining Ġphase , Ġwe Ġused ĠMoment um ĠSG D Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 01 Ġand Ġmomentum Ġof Ġ0 . 9 . ĠTable ĠT AB REF 11 Ġsummarizes Ġthe Ġdetails Ġof Ġthe Ġlayer Ġsettings Ġfor Ġthe Ġconv olution al Ġand ĠN IN Ġlayers . </s> On Ġthe Ġtop Ġlayers Ġof Ġthe Ġenc oder Ġafter Ġthe Ġtransferred Ġconv olution al Ġand ĠN IN Ġlayers , Ġwe Ġput Ġthree Ġbid irection al ĠL ST Ms Ġ( Bi - L ST M ) Ġwith Ġ256 Ġhidden Ġunits Ġ( total Ġ512 Ġunits Ġfor Ġboth Ġdirections ). ĠTo Ġreduce Ġthe Ġcomputational Ġtime , Ġwe Ġused Ġhierarchical Ġsubs am pling ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 10 Ġ. ĠWe Ġapplied Ġsubs am pling Ġon Ġall Ġthe ĠBi - L ST M Ġlayers Ġand Ġreduced Ġthe Ġlength Ġby Ġa Ġfactor Ġof Ġ8 . </s> On Ġthe Ġdec oder Ġside , Ġthe Ġprevious Ġinput Ġphon em es Ġ/ Ġcharacters Ġwere Ġconverted Ġinto Ġreal Ġvectors Ġby Ġa Ġ128 - dimensional Ġembed ding Ġmatrix . ĠWe Ġused Ġone Ġun id irection al ĠL ST M Ġwith Ġ512 Ġhidden Ġunits Ġand Ġfollowed Ġby Ġa Ġsoft max Ġlayer Ġto Ġoutput Ġthe Ġcharacter Ġprobability . ĠFor Ġthe Ġend - to - end Ġtraining Ġphase , Ġwe Ġfroze Ġthe Ġparameter Ġvalues Ġfrom Ġthe Ġtransferred Ġlayers Ġfrom Ġepoch Ġ0 Ġto Ġepoch Ġ10 , Ġand Ġafter Ġepoch Ġ10 Ġwe Ġjointly Ġoptimized Ġall Ġthe Ġparameters Ġtogether Ġuntil Ġthe Ġend Ġof Ġtraining Ġ( a Ġtotal Ġ40 Ġepoch s ). ĠWe Ġused Ġan ĠAdam ĠB IB REF 23 Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 000 5 . </s> In Ġthe Ġdecoding Ġphase , Ġwe Ġused Ġa Ġbeam Ġsearch Ġstrategy Ġwith Ġbeam Ġsize ĠIN LINE FORM 0 Ġand Ġwe Ġadjusted Ġthe Ġscore Ġby Ġdividing Ġwith Ġthe Ġtranscription Ġlength Ġto Ġprevent Ġthe Ġdec oder Ġfrom Ġfavoring Ġshorter Ġtranscript ions . ĠWe Ġdid Ġnot Ġuse Ġany Ġlanguage Ġmodel Ġor Ġlex icon Ġdictionary Ġfor Ġdecoding . ĠAll Ġof Ġour Ġmodels Ġwere Ġimplemented Ġon Ġthe ĠPy Tor ch Ġframework Ġ. </s> For Ġcomparison , Ġwe Ġalso Ġevaluated Ġthe Ġstandard Ġattention - based Ġenc oder Ġdec oder Ġwith ĠMel - scale Ġspect rogram Ġinput Ġas Ġthe Ġbaseline . ĠHere , Ġwe Ġused Ġsimilar Ġsettings Ġas Ġthe Ġproposed Ġmodel , Ġexcept Ġwe Ġreplaced Ġthe Ġconv olution al Ġand ĠN IN Ġlayers Ġwith Ġa Ġfeed forward Ġlayer Ġ( 512 Ġhidden Ġunits ). </s> Result </s> An Ġexample Ġof Ġour Ġtransfer Ġlearning Ġresults Ġis Ġshown Ġin ĠFigure ĠFIG REF 8 Ġ, Ġand ĠTable ĠT AB REF 14 Ġshows Ġthe Ġspeech Ġrecognition Ġperformance Ġin ĠC ER Ġfor Ġboth Ġthe ĠWS J - SI 84 Ġand ĠWS J - SI 284 Ġdatasets . ĠWe Ġcompared Ġour Ġmethod Ġwith Ġseveral Ġpublished Ġmodels Ġlike ĠC TC , ĠAttention ĠEnc oder - Dec oder Ġand ĠJoint ĠC TC - Att ention Ġmodel Ġthat Ġutilize ĠC TC Ġfor Ġtraining Ġthe Ġenc oder Ġpart . ĠBesides , Ġwe Ġalso Ġtrain Ġour Ġown Ġbaseline ĠAttention ĠEnc oder - Dec oder Ġwith ĠMel - scale Ġspect rogram . ĠThe Ġdifference Ġbetween Ġour ĠAttention ĠEnc oder - Dec oder Ġ( âĢ ľ Att ĠEnc - Dec Ġ( ours )", ĠâĢ ľ Att ĠEnc - Dec ĠW av 2 Text ") Ġwith ĠAttention ĠEnc oder - Dec oder Ġfrom ĠB IB REF 24 Ġ( âĢ ľ Att ĠEnc - Dec ĠContent ", ĠâĢ ľ Att ĠEnc - Dec ĠLocation ") Ġis Ġwe Ġused Ġthe Ġcurrent Ġhidden Ġstates Ġto Ġgenerate Ġthe Ġattention Ġvector Ġinstead Ġof Ġthe Ġprevious Ġhidden Ġstates . ĠAnother Ġaddition Ġis Ġwe Ġutilized ĠâĢ ľ input Ġfeedback " Ġmethod ĠB IB REF 13 Ġby Ġconc aten ating Ġthe Ġprevious Ġcontext Ġvector Ġinto Ġthe Ġcurrent Ġinput Ġalong Ġwith Ġthe Ġcharacter Ġembed ding Ġvector . ĠBy Ġusing Ġthose Ġmodifications , Ġwe Ġare Ġable Ġto Ġimprove Ġthe Ġbaseline Ġperformance . </s> Our Ġproposed ĠW av 2 Text Ġmodels Ġwithout Ġany Ġtransfer Ġlearning Ġfailed Ġto Ġconverge . ĠIn Ġcontrast , Ġwith Ġtransfer Ġlearning , Ġthey Ġsignificantly Ġsurpassed Ġthe Ġperformance Ġof Ġthe ĠC TC Ġand Ġenc oder - dec oder Ġfrom ĠMel - scale Ġspect rogram Ġfeatures . ĠThis Ġsuggests Ġthat Ġby Ġusing Ġtransfer Ġlearning Ġfor Ġinitial izing Ġthe Ġlower Ġpart Ġof Ġthe Ġenc oder Ġparameters , Ġour Ġmodel Ġalso Ġperformed Ġbetter Ġthen Ġtheir Ġoriginal Ġfeatures . </s> Related ĠWork </s> Transfer Ġlearning Ġis Ġthe Ġability Ġof Ġa Ġlearning Ġalgorithm Ġto Ġconvey Ġknowledge Ġacross Ġdifferent Ġtasks . ĠThe Ġinitial Ġidea Ġis Ġto Ġreuse Ġpreviously Ġobtained Ġknowledge Ġto Ġenhance Ġthe Ġlearning Ġfor Ġnew Ġthings . ĠThe Ġstandard Ġprocedure Ġare Ġ: Ġfirst , Ġtrain Ġthe Ġmodel Ġon Ġa Ġbase Ġdataset Ġand Ġtask , Ġthen Ġthe Ġlearned Ġfeatures Ġand / or Ġparameters Ġare Ġreused Ġfor Ġlearning Ġa Ġsecond Ġtarget Ġdataset Ġand Ġtask . ĠBeng io Ġet Ġal . ĠB IB REF 25 Ġprovided Ġdeep Ġreviews Ġabout Ġmulti - task Ġand Ġtransfer Ġlearning Ġon Ġdeep Ġlearning Ġmodels . ĠJason Ġet Ġal . ĠB IB REF 26 Ġshowed Ġthat Ġa Ġmodel Ġwith Ġtransferred Ġparameter Ġconsistently Ġoutper formed Ġa Ġrandomly Ġinitialized Ġone . </s> In Ġspeech Ġrecognition Ġresearch , Ġtransfer Ġlearning Ġhas Ġbeen Ġstudied Ġfor Ġmany Ġyears , Ġincluding Ġsuccessful Ġcases Ġof Ġspeaker Ġadaptation Ġand Ġcross - ling ual Ġacoustic Ġmodeling ĠB IB REF 27 Ġ. ĠOne Ġpopular Ġscheme Ġfor Ġutilizing ĠD NN s Ġfor Ġtransfer Ġlearning Ġwithin ĠAS R Ġframeworks Ġis Ġa Ġtandem Ġapproach ĠB IB REF 28 Ġ. ĠThis Ġidea Ġfirst Ġtrains Ġa ĠD NN Ġwith Ġa Ġnarrow Ġhidden Ġbottleneck Ġlayer Ġto Ġperform Ġphon eme Ġclassification Ġat Ġthe Ġframe Ġlevel Ġand Ġthen Ġre uses Ġthe Ġactiv ations Ġfrom Ġthe Ġnarrow Ġhidden Ġbottleneck Ġlayer Ġas Ġdiscrim in ative Ġfeatures Ġin Ġconventional ĠGM M - H MM Ġor Ġhybrid ĠD NN - H MM Ġmodels ĠB IB REF 29 Ġ. ĠAnother Ġstudy Ġintroduced Ġa Ġconv olution al Ġbottleneck Ġnetwork Ġas Ġan Ġalternative Ġtandem Ġbottleneck Ġfeature Ġarchitecture ĠB IB REF 30 Ġ. ĠHowever , Ġalthough Ġsuch Ġa Ġfeature Ġtransfer Ġlearning Ġframework Ġprovides Ġmany Ġadvantages Ġin ĠAS R , Ġthe Ġusage Ġin Ġan Ġend - to - end Ġattention - based ĠAS R Ġframework Ġhas Ġnot Ġbeen Ġexplored . </s> This Ġstudy Ġperforms Ġfeature Ġtransfer Ġlearning Ġon Ġthe Ġenc oder Ġpart Ġof Ġthe Ġend - to - end Ġattention - based ĠAS R Ġarchitecture . ĠWe Ġtrain Ġthe Ġconv olution al Ġenc oder Ġto Ġpredict Ġthe Ġspectral Ġfeatures Ġgiven Ġthe Ġcorresponding Ġraw Ġspeech Ġwave form . ĠAfter Ġthat , Ġwe Ġtransfer Ġthe Ġtrained Ġlayers Ġand Ġparameters , Ġintegrate Ġthem Ġwith Ġthe ĠL ST M Ġenc oder - dec oder , Ġand Ġeventually Ġoptimize Ġthe Ġwhole Ġstructure Ġto Ġpredict Ġthe Ġcorrect Ġoutput Ġtext Ġtranscription Ġgiven Ġthe Ġraw Ġspeech Ġwave form . </s> Conclusion </s> This Ġpaper Ġdescribed Ġthe Ġfirst Ġattempt Ġto Ġbuild Ġan Ġend - to - end Ġattention - based Ġenc oder - dec oder Ġspeech Ġrecognition Ġthat Ġdirectly Ġpredicts Ġthe Ġtext Ġtranscription Ġgiven Ġraw Ġspeech Ġinput . ĠWe Ġalso Ġproposed Ġfeature Ġtransfer Ġlearning Ġto Ġassist Ġthe Ġenc oder - dec oder Ġmodel Ġtraining Ġprocess Ġand Ġpresented Ġa Ġnovel Ġarchitecture Ġthat Ġcombined Ġconv olution al , ĠN IN Ġand ĠBi - L ST M Ġlayers Ġinto Ġa Ġsingle Ġenc oder Ġpart Ġfor Ġraw Ġspeech Ġrecognition . ĠOur Ġresults Ġsuggest Ġthat Ġtransfer Ġlearning Ġis Ġa Ġvery Ġhelpful Ġmethod Ġfor Ġconstructing Ġan Ġend - to - end Ġsystem Ġfrom Ġsuch Ġlow - level Ġfeatures Ġas Ġraw Ġspeech Ġsignals . ĠWith Ġtransferred Ġparameters , Ġour Ġproposed Ġattention - based ĠW av 2 Text Ġmodels Ġconver ged Ġand Ġmatched Ġthe Ġperformance Ġwith Ġthe Ġattention - based Ġenc oder - dec oder Ġmodel Ġtrained Ġon Ġstandard Ġspectral - based Ġfeatures . ĠThe Ġbest Ġperformance Ġwas Ġachieved Ġby ĠW av 2 Text Ġmodels Ġwith Ġtransfer Ġlearning Ġfrom Ġmulti Ġtarget Ġscheme . </s> Acknowled gements </s> Part Ġof Ġthis Ġwork Ġwas Ġsupported Ġby ĠJS PS ĠK AK EN HI ĠGrant ĠNumbers ĠJP 17 H 06 101 Ġand ĠJP Ġ17 K 00 237 .