CNN-DNN-BLSTM-HMM
ABSTRACTIVE
HMM-based system
EXTRACTIVE
Table TABREF11 shows Word Error Rates (WER) on WSJ for the current state-of-the-art and our models. The current best model trained on this dataset is an HMM-based system which uses a combination of convolutional, recurrent and fully connected layers, as well as speaker adaptation, and reaches INLINEFORM0 WER on nov92. DeepSpeech 2 shows a WER of INLINEFORM1 but uses 150 times more training data for the acoustic model and huge text datasets for LM training. Finally, the state-of-the-art among end-to-end systems trained only on WSJ, and hence the most comparable to our system, uses lattice-free MMI on augmented data (with speed perturbation) and gets INLINEFORM2 WER. Our baseline system, trained on mel-filterbanks, and decoded with a n-gram language model has a INLINEFORM3 WER. Replacing the n-gram LM by a convolutional one reduces the WER to INLINEFORM4 , and puts our model on par with the current best end-to-end system. Replacing the speech features by a learnable frontend finally reduces the WER to INLINEFORM5 and then to INLINEFORM6 when doubling the number of learnable filters, improving over DeepSpeech 2 and matching the performance of the best HMM-DNN system. FLOAT SELECTED: Table 1: WER (%) on the open vocabulary task of WSJ.
Table TABREF11 shows Word Error Rates (WER) on WSJ for the current state-of-the-art and our models. The current best model trained on this dataset is an HMM-based system which uses a combination of convolutional, recurrent and fully connected layers, as well as speaker adaptation, and reaches INLINEFORM0 WER on nov92. DeepSpeech 2 shows a WER of INLINEFORM1 but uses 150 times more training data for the acoustic model and huge text datasets for LM training. Finally, the state-of-the-art among end-to-end systems trained only on WSJ, and hence the most comparable to our system, uses lattice-free MMI on augmented data (with speed perturbation) and gets INLINEFORM2 WER. Our baseline system, trained on mel-filterbanks, and decoded with a n-gram language model has a INLINEFORM3 WER. Replacing the n-gram LM by a convolutional one reduces the WER to INLINEFORM4 , and puts our model on par with the current best end-to-end system. Replacing the speech features by a learnable frontend finally reduces the WER to INLINEFORM5 and then to INLINEFORM6 when doubling the number of learnable filters, improving over DeepSpeech 2 and matching the performance of the best HMM-DNN system.
1812.06864
Introduction </s> Recent Ġwork Ġon Ġconv olution al Ġneural Ġnetwork Ġarchitectures Ġshowed Ġthat Ġthey Ġare Ġcompetitive Ġwith Ġrecurrent Ġarchitectures Ġeven Ġon Ġtasks Ġwhere Ġmodeling Ġlong - range Ġdependencies Ġis Ġcritical , Ġsuch Ġas Ġlanguage Ġmodeling ĠB IB REF 0 Ġ, Ġmachine Ġtranslation ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand Ġspeech Ġsynthesis ĠB IB REF 3 Ġ. ĠIn Ġend - to - end Ġspeech Ġrecognition Ġhowever , Ġrecurrent Ġarchitectures Ġare Ġstill Ġprevalent Ġfor Ġacoustic Ġand / or Ġlanguage Ġmodeling ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> There Ġis Ġa Ġhistory Ġof Ġusing Ġconv olution al Ġnetworks Ġin Ġspeech Ġrecognition , Ġbut Ġonly Ġas Ġpart Ġof Ġan Ġotherwise Ġmore Ġtraditional Ġpipeline . ĠThey Ġhave Ġbeen Ġfirst Ġintroduced Ġas ĠTD NN s Ġto Ġpredict Ġphon eme Ġclasses ĠB IB REF 9 Ġ, Ġand Ġlater Ġto Ġgenerate ĠH MM Ġposterior gram s ĠB IB REF 10 Ġ. ĠThey Ġhave Ġmore Ġrecently Ġbeen Ġused Ġin Ġend - to - end Ġframeworks , Ġbut Ġonly Ġin Ġcombination Ġwith Ġrecurrent Ġlayers ĠB IB REF 6 Ġ, Ġor Ġn - gram Ġlanguage Ġmodels ĠB IB REF 11 Ġ, Ġor Ġfor Ġphone Ġrecognition ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠNonetheless , Ġconv olution al Ġarchitectures Ġare Ġprevalent Ġwhen Ġlearning Ġfrom Ġthe Ġraw Ġwave form ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ, Ġbecause Ġthey Ġnaturally Ġmodel Ġthe Ġcomputation Ġof Ġstandard Ġfeatures Ġsuch Ġas Ġmel - filter banks . ĠGiven Ġthe Ġevidence Ġthat Ġthey Ġare Ġalso Ġsuitable Ġon Ġlong - range Ġdependency Ġtasks , Ġwe Ġexpect Ġconv olution al Ġneural Ġnetworks Ġto Ġbe Ġcompetitive Ġat Ġall Ġlevels Ġof Ġthe Ġspeech Ġrecognition Ġpipeline . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġfully Ġconv olution al Ġapproach Ġto Ġend - to - end Ġspeech Ġrecognition . ĠBuilding Ġon Ġrecent Ġadvances Ġin Ġconv olution al Ġlearn able Ġfront - ends Ġfor Ġspeech ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ, Ġconv olution al Ġacoustic Ġmodels ĠB IB REF 11 Ġ, Ġand Ġconv olution al Ġlanguage Ġmodels ĠB IB REF 0 Ġ, Ġthe Ġpaper Ġhas Ġfour Ġmain Ġcontributions : </s> Model </s> Our Ġapproach , Ġdescribed Ġin Ġthis Ġsection , Ġis Ġillustrated Ġin ĠFig . ĠFIG REF 5 Ġ. </s> Con v olution al ĠFront Ġend </s> Several Ġproposals Ġto Ġlearn Ġthe Ġfront - end Ġof Ġspeech Ġrecognition Ġsystems Ġhave Ġbeen Ġmade ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ. ĠFollowing Ġthe Ġcomparison Ġin ĠB IB REF 17 Ġ, Ġwe Ġconsider Ġtheir Ġbest Ġarchitecture , Ġcalled Ġ" sc attering Ġbased " Ġ( here after Ġref ered Ġto Ġas Ġlearn able Ġfront - end ). ĠThe Ġlearn able Ġfront - end Ġcontains Ġfirst Ġa Ġconv olution Ġof Ġwidth Ġ2 Ġthat Ġem ulates Ġthe Ġpre - emphasis Ġstep Ġused Ġin Ġmel - filter banks . ĠIt Ġis Ġfollowed Ġby Ġa Ġcomplex Ġconv olution Ġof Ġwidth Ġ25 ms Ġand ĠIN LINE FORM 0 Ġfilters . ĠAfter Ġtaking Ġthe Ġsquared Ġabsolute Ġvalue , Ġa Ġlow - pass Ġfilter Ġof Ġwidth Ġ25 ms Ġand Ġstride Ġ10 ms Ġperforms Ġdec imation . ĠThe Ġfront - end Ġfinally Ġapplies Ġa Ġlog - comp ression Ġand Ġa Ġper - channel Ġmean - vari ance Ġnormal ization Ġ( equ ivalent Ġto Ġan Ġinstance Ġnormal ization Ġlayer ĠB IB REF 18 Ġ). ĠFollowing ĠB IB REF 17 Ġ, Ġthe Ġ" pre - emphasis " Ġconv olution Ġis Ġinitialized Ġto ĠIN LINE FORM 1 Ġ, Ġand Ġthen Ġtrained Ġwith Ġthe Ġrest Ġof Ġthe Ġnetwork . ĠThe Ġlow - pass Ġfilter Ġis Ġkept Ġconstant Ġto Ġa Ġsquared ĠHan ning Ġwindow , Ġand Ġthe Ġcomplex Ġconv olution al Ġlayer Ġis Ġinitialized Ġrandomly . ĠIn Ġaddition Ġto Ġthe ĠIN LINE FORM 2 Ġfilters Ġused Ġby ĠB IB REF 17 Ġ, Ġwe Ġexperiment Ġwith ĠIN LINE FORM 3 Ġfilters . ĠNotice Ġthat Ġsince Ġthe Ġstride Ġis Ġthe Ġsame Ġas Ġfor Ġmel - filter banks , Ġacoustic Ġmodels Ġon Ġtop Ġof Ġthe Ġlearn able Ġfront - ends Ġcan Ġalso Ġbe Ġapplied Ġto Ġmel - filter banks Ġ( sim ply Ġmodifying Ġthe Ġnumber Ġof Ġinput Ġchannels Ġif ĠIN LINE FORM 4 Ġ). </s> Con v olution al ĠAc oustic ĠModel </s> The Ġacoustic Ġmodel Ġis Ġa Ġconv olution al Ġneural Ġnetwork Ġwith Ġg ated Ġlinear Ġunits ĠB IB REF 0 Ġ, Ġwhich Ġis Ġfed Ġwith Ġthe Ġoutput Ġof Ġthe Ġlearn able Ġfront - end . ĠFollowing ĠB IB REF 11 Ġ, Ġthe Ġnetworks Ġuses Ġa Ġgrowing Ġnumber Ġof Ġchannels , Ġand Ġdrop out ĠB IB REF 19 Ġfor Ġregular ization . ĠThese Ġacoustic Ġmodels Ġare Ġtrained Ġto Ġpredict Ġletters Ġdirectly Ġwith Ġthe ĠAuto ĠSe gment ation ĠCrit erion Ġ( AS G ) ĠB IB REF 20 Ġ. ĠThe Ġonly Ġdifferences Ġbetween Ġthe ĠWS J Ġand ĠLib ris pe ech Ġmodels Ġare Ġtheir Ġdepth , Ġthe Ġnumber Ġof Ġfeature Ġmaps Ġper Ġlayer , Ġthe Ġreceptive Ġfield Ġand Ġthe Ġamount Ġof Ġdrop out . </s> Con v olution al ĠLanguage ĠModel </s> The Ġconv olution al Ġlanguage Ġmodel Ġ( LM ) Ġis Ġthe ĠG CNN - 14 B Ġfrom ĠB IB REF 0 Ġ, Ġwhich Ġachieved Ġcompetitive Ġresults Ġon Ġseveral Ġlanguage Ġmodeling Ġbenchmarks . ĠThe Ġnetwork Ġcontains Ġ14 Ġconv olution al Ġresidual Ġblocks ĠB IB REF 21 Ġwith Ġa Ġgrowing Ġnumber Ġof Ġchannels , Ġand Ġuses Ġg ated Ġlinear Ġunits Ġas Ġactivation Ġfunction . </s> The Ġlanguage Ġmodel Ġis Ġused Ġto Ġscore Ġcandidate Ġtranscript ions Ġin Ġaddition Ġto Ġthe Ġacoustic Ġmodel Ġin Ġthe Ġbeam Ġsearch Ġdec oder Ġdescribed Ġin Ġthe Ġnext Ġsection . ĠCompared Ġto Ġn - gram ĠL Ms , Ġconv olution al ĠL Ms Ġallow Ġfor Ġmuch Ġlarger Ġcontext Ġsizes . ĠOur Ġdetailed Ġexperiments Ġstudy Ġthe Ġeffect Ġof Ġcontext Ġsize Ġon Ġthe Ġfinal Ġspeech Ġrecognition Ġperformance . </s> Be am - search Ġdec oder </s> We Ġuse Ġthe Ġbeam - search Ġdec oder Ġpresented Ġin ĠB IB REF 11 Ġto Ġgenerate Ġword Ġsequences Ġgiven Ġthe Ġoutput Ġfrom Ġour Ġacoustic Ġmodel . ĠThe Ġdec oder Ġfinds Ġthe Ġword Ġtranscription ĠIN LINE FORM 0 Ġto Ġmaximize : ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġvalue Ġfor Ġthe ĠIN LINE FORM 1 Ġth Ġframe Ġin Ġthe Ġpath Ġleading Ġto ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġ( un normal ized ) Ġacoustic Ġmodel Ġscore Ġof Ġthe Ġtranscription ĠIN LINE FORM 4 Ġ. ĠThe Ġhyper param eters ĠIN LINE FORM 5 Ġrespectively Ġcontrol Ġthe Ġweight Ġof Ġthe Ġlanguage Ġmodel , Ġthe Ġword Ġinsertion Ġreward , Ġand Ġthe Ġsilence Ġinsertion Ġpenalty . ĠThe Ġother Ġparameters Ġare Ġthe Ġbeam Ġsize Ġand Ġthe Ġbeam Ġscore , Ġa Ġthreshold Ġunder Ġwhich Ġcandidates Ġare Ġdiscarded Ġeven Ġif Ġthe Ġbeam Ġis Ġnot Ġfull . ĠThese Ġare Ġchosen Ġaccording Ġto Ġa Ġtrade - off Ġbetween Ġ( near -) optim ality Ġof Ġthe Ġsearch Ġand Ġcomputational Ġcost . </s> Exper iments </s> We Ġevaluate Ġour Ġapproach Ġon Ġthe Ġlarge Ġvocabulary Ġtask Ġof Ġthe ĠWall ĠStreet ĠJournal Ġ( WS J ) Ġdataset ĠB IB REF 25 Ġ, Ġwhich Ġcontains Ġ80 Ġhours Ġof Ġclean Ġread Ġspeech , Ġand ĠLib ris pe ech ĠB IB REF 26 Ġ, Ġwhich Ġcontains Ġ1000 Ġhours Ġwith Ġseparate Ġtrain / dev / test Ġsplits Ġfor Ġclean Ġand Ġnoisy Ġspeech . ĠEach Ġdataset Ġcomes Ġwith Ġofficial Ġtextual Ġdata Ġto Ġtrain Ġlanguage Ġmodels , Ġwhich Ġcontain Ġ37 Ġmillion Ġtokens Ġfor ĠWS J , Ġ800 Ġmillion Ġtokens Ġfor ĠLib ris pe ech . ĠOur Ġlanguage Ġmodels Ġare Ġtrained Ġseparately Ġfor Ġeach Ġdataset Ġon Ġthe Ġofficial Ġtext Ġdata Ġonly . ĠThese Ġdatasets Ġwere Ġchosen Ġto Ġstudy Ġthe Ġimpact Ġof Ġthe Ġdifferent Ġcomponents Ġof Ġour Ġsystem Ġat Ġdifferent Ġscales Ġof Ġtraining Ġdata Ġand Ġin Ġdifferent Ġrecording Ġconditions . </s> The Ġmodels Ġare Ġevaluated Ġin ĠWord ĠError ĠRate Ġ( WER ). ĠOur Ġexperiments Ġuse Ġthe Ġopen Ġsource Ġcodes Ġof Ġw av 2 letter Ġfor Ġthe Ġacoustic Ġmodel , Ġand Ġfair seq Ġfor Ġthe Ġlanguage Ġmodel . ĠMore Ġdetails Ġon Ġthe Ġexperimental Ġsetup Ġare Ġgiven Ġbelow . </s> Bas eline ĠOur Ġbaseline Ġfor Ġeach Ġdataset Ġfollows ĠB IB REF 11 Ġ. ĠIt Ġuses Ġthe Ġsame Ġconv olution al Ġacoustic Ġmodel Ġas Ġour Ġapproach Ġbut Ġa Ġmel - filter banks Ġfront - end Ġand Ġa Ġ4 - gram Ġlanguage Ġmodel . </s> Training / test Ġsplits ĠOn ĠWS J , Ġmodels Ġare Ġtrained Ġon Ġsi 284 . Ġno v 93 dev Ġis Ġused Ġfor Ġvalidation Ġand Ġno v 92 Ġfor Ġtest . ĠOn ĠLib ris pe ech , Ġwe Ġtrain Ġon Ġthe Ġconc aten ation Ġof Ġtrain - clean Ġand Ġtrain - other . ĠThe Ġvalidation Ġset Ġis Ġdev - clean Ġwhen Ġtesting Ġon Ġtest - clean , Ġand Ġdev - other Ġwhen Ġtesting Ġon Ġtest - other . </s> Ac oustic Ġmodel Ġarchitecture ĠThe Ġarchitecture Ġfor Ġthe Ġconv olution al Ġacoustic Ġmodel Ġis Ġthe Ġ" high Ġdrop out " Ġmodel Ġfrom ĠB IB REF 11 Ġfor ĠLib ris pe ech , Ġwhich Ġhas Ġ19 Ġlayers Ġin Ġaddition Ġto Ġthe Ġfront - end Ġ( mel - filter banks Ġfor Ġthe Ġbaseline , Ġor Ġthe Ġlearn able Ġfront - end Ġfor Ġour Ġapproach ). ĠOn ĠWS J , Ġwe Ġuse Ġthe Ġlighter Ġversion Ġused Ġin ĠB IB REF 17 Ġ, Ġwhich Ġhas Ġ17 Ġlayers . ĠDrop out Ġis Ġapplied Ġat Ġeach Ġlayer Ġafter Ġthe Ġfront - end , Ġfollowing ĠB IB REF 20 Ġ. ĠThe Ġlearn able Ġfront - end Ġuses Ġ40 Ġor Ġ80 Ġfilters . ĠLanguage Ġmodel Ġarchitecture ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 8 Ġ, Ġwe Ġuse Ġthe ĠG CNN - 14 B Ġmodel Ġof ĠB IB REF 0 Ġwith Ġdrop out Ġat Ġeach Ġconv olution al Ġand Ġlinear Ġlayer Ġon Ġboth ĠWS J Ġand ĠLib ris pe ech . ĠWe Ġkeep Ġall Ġthe Ġwords Ġ( 162 K ) Ġin ĠWS J Ġtraining Ġcorpus . ĠFor ĠLib ris pe ech , Ġwe Ġonly Ġuse Ġthe Ġmost Ġfrequent Ġ200 K Ġtokens Ġ( out Ġof Ġ900 K ). </s> Hyper param eter Ġtuning ĠThe Ġacoustic Ġmodels Ġare Ġtrained Ġfollowing ĠB IB REF 11 Ġ, ĠB IB REF 17 Ġ, Ġusing ĠSG D Ġwith Ġa Ġdecreasing Ġlearning Ġrate , Ġweight Ġnormal ization Ġand Ġgradient Ġclipping Ġat Ġ0 . 2 Ġand Ġa Ġmomentum Ġof Ġ0 . 9 . ĠThe Ġlanguage Ġmodels Ġare Ġtrained Ġwith ĠN ester ov Ġaccelerated Ġgradient ĠB IB REF 27 Ġ. ĠFollowing ĠB IB REF 0 Ġ, Ġwe Ġalso Ġuse Ġweight Ġnormal ization Ġand Ġgradient Ġclipping . </s> The Ġparameters Ġof Ġthe Ġbeam Ġsearch Ġ( see ĠSection ĠSEC REF 9 Ġ) ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġtuned Ġon Ġthe Ġvalidation Ġset Ġwith Ġa Ġbeam Ġsize Ġof Ġ2500 Ġand Ġa Ġbeam Ġscore Ġof Ġ26 Ġfor Ġcomputational Ġefficiency . ĠOnce ĠIN LINE FORM 3 Ġare Ġchosen , Ġthe Ġtest ĠW ER Ġis Ġcomputed Ġwith Ġa Ġbeam Ġsize Ġof Ġ3000 Ġand Ġa Ġbeam Ġscore Ġof Ġ50 . </s> Word ĠError ĠRate Ġresults </s> Table ĠT AB REF 11 Ġshows ĠWord ĠError ĠRates Ġ( WER ) Ġon ĠWS J Ġfor Ġthe Ġcurrent Ġstate - of - the - art Ġand Ġour Ġmodels . ĠThe Ġcurrent Ġbest Ġmodel Ġtrained Ġon Ġthis Ġdataset Ġis Ġan ĠH MM - based Ġsystem Ġwhich Ġuses Ġa Ġcombination Ġof Ġconv olution al , Ġrecurrent Ġand Ġfully Ġconnected Ġlayers , Ġas Ġwell Ġas Ġspeaker Ġadaptation , Ġand Ġreaches ĠIN LINE FORM 0 ĠW ER Ġon Ġno v 92 . ĠDeep Spe ech Ġ2 Ġshows Ġa ĠW ER Ġof ĠIN LINE FORM 1 Ġbut Ġuses Ġ150 Ġtimes Ġmore Ġtraining Ġdata Ġfor Ġthe Ġacoustic Ġmodel Ġand Ġhuge Ġtext Ġdatasets Ġfor ĠLM Ġtraining . ĠFinally , Ġthe Ġstate - of - the - art Ġamong Ġend - to - end Ġsystems Ġtrained Ġonly Ġon ĠWS J , Ġand Ġhence Ġthe Ġmost Ġcomparable Ġto Ġour Ġsystem , Ġuses Ġlatt ice - free ĠM MI Ġon Ġaugmented Ġdata Ġ( with Ġspeed Ġpert urb ation ) Ġand Ġgets ĠIN LINE FORM 2 ĠW ER . ĠOur Ġbaseline Ġsystem , Ġtrained Ġon Ġmel - filter banks , Ġand Ġdec oded Ġwith Ġa Ġn - gram Ġlanguage Ġmodel Ġhas Ġa ĠIN LINE FORM 3 ĠW ER . ĠRepl acing Ġthe Ġn - gram ĠLM Ġby Ġa Ġconv olution al Ġone Ġreduces Ġthe ĠW ER Ġto ĠIN LINE FORM 4 Ġ, Ġand Ġputs Ġour Ġmodel Ġon Ġpar Ġwith Ġthe Ġcurrent Ġbest Ġend - to - end Ġsystem . ĠRepl acing Ġthe Ġspeech Ġfeatures Ġby Ġa Ġlearn able Ġfront end Ġfinally Ġreduces Ġthe ĠW ER Ġto ĠIN LINE FORM 5 Ġand Ġthen Ġto ĠIN LINE FORM 6 Ġwhen Ġdoubling Ġthe Ġnumber Ġof Ġlearn able Ġfilters , Ġimproving Ġover ĠDeep Spe ech Ġ2 Ġand Ġmatching Ġthe Ġperformance Ġof Ġthe Ġbest ĠH MM - D NN Ġsystem . </s> Table ĠT AB REF 10 Ġreports ĠW ER Ġon Ġthe ĠLib ris pe ech Ġdataset . ĠThe ĠCAP IO ĠB IB REF 22 Ġensemble Ġmodel Ġcombines Ġthe Ġlatt ices Ġfrom Ġ8 Ġindividual ĠH MM - D NN Ġsystems Ġ( using Ġboth Ġconv olution al Ġand ĠL ST M Ġlayers ), Ġand Ġis Ġthe Ġcurrent Ġstate - of - the - art Ġon ĠLib ris pe ech . ĠCAP IO Ġ( single ) Ġis Ġthe Ġbest Ġindividual Ġsystem , Ġselected Ġeither Ġon Ġdev - clean Ġor Ġdev - other . ĠThe Ġsequence - to - sequence Ġbaseline Ġis Ġan Ġenc oder - dec oder Ġwith Ġattention Ġand Ġa ĠB PE - level ĠB IB REF 28 ĠLM , Ġand Ġcurrently Ġthe Ġbest Ġend - to - end Ġsystem Ġon Ġthis Ġdataset . ĠWe Ġcan Ġobserve Ġthat Ġour Ġfully Ġconv olution al Ġmodel Ġimproves Ġover ĠCAP IO Ġ( Single ) Ġon Ġthe Ġclean Ġpart , Ġand Ġis Ġthe Ġcurrent Ġbest Ġend - to - end Ġsystem Ġon Ġtest - other Ġwith Ġan Ġimprovement Ġof ĠIN LINE FORM 0 Ġabsolute . ĠOur Ġsystem Ġalso Ġoutper forms ĠDeep Spe ech Ġ2 Ġon Ġboth Ġtest Ġsets Ġby Ġa Ġsignificant Ġmargin . ĠAn Ġinteresting Ġobservation Ġis Ġthe Ġimpact Ġof Ġeach Ġconv olution al Ġblock . ĠWhile Ġreplacing Ġthe Ġ4 - gram ĠLM Ġby Ġa Ġconv olution al ĠLM Ġimproves Ġsimilarly Ġon Ġthe Ġclean Ġand Ġno is ier Ġparts , Ġlearning Ġthe Ġspeech Ġfront end Ġgives Ġsimilar Ġperformance Ġon Ġthe Ġclean Ġpart Ġbut Ġsignificantly Ġimproves Ġthe Ġperformance Ġon Ġno is ier , Ġharder Ġutter ances , Ġa Ġfinding Ġthat Ġis Ġconsistent Ġwith Ġprevious Ġliterature ĠB IB REF 15 Ġ. </s> Analysis Ġof Ġthe Ġconv olution al Ġlanguage Ġmodel </s> Since Ġthis Ġpaper Ġuses Ġconv olution al Ġlanguage Ġmodels Ġfor Ġspeech Ġrecognition Ġsystems Ġfor Ġthe Ġfirst Ġtime , Ġwe Ġpresent Ġadditional Ġstudies Ġof Ġthe Ġlanguage Ġmodel Ġin Ġisolation . ĠThese Ġexperiments Ġuse Ġour Ġbest Ġlanguage Ġmodel Ġon ĠLib ris pe ech , Ġand Ġevaluations Ġin ĠW ER Ġare Ġcarried Ġout Ġusing Ġthe Ġbaseline Ġsystem Ġtrained Ġon Ġmel - filter banks . ĠThe Ġdec oder Ġparameters Ġare Ġtuned Ġusing Ġthe Ġgrid Ġsearch Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ, Ġa Ġbeam Ġsize Ġis Ġfixed Ġto Ġ2500 Ġand Ġa Ġbeam Ġscore Ġto Ġ30 . </s> Cor relation Ġbetween Ġperplex ity Ġand ĠW ER ĠFigure ĠFIG REF 18 Ġshows Ġthe Ġcorrelation Ġbetween Ġperplex ity Ġand ĠW ER Ġas Ġthe Ġtraining Ġprogresses . ĠAs Ġperplex ity Ġdecreases , Ġthe ĠW ER Ġon Ġboth Ġdev - clean Ġand Ġdev - other Ġalso Ġdecreases Ġfollowing Ġthe Ġsame Ġtrend . ĠIt Ġillustrates Ġthat Ġperplex ity Ġon Ġthe Ġlinguistic Ġdata Ġis Ġa Ġgood Ġsurrogate Ġof Ġthe Ġfinal Ġperformance Ġof Ġthe Ġspeech Ġrecognition Ġpipeline . ĠArchitect ural Ġchoices Ġor Ġhyper - param eter Ġtuning Ġcan Ġthus Ġbe Ġcarried Ġout Ġmostly Ġusing Ġperplex ity Ġalone . </s> Inf luence Ġof Ġcontext Ġsize ĠBy Ġlimiting Ġthe Ġcontext Ġpassed Ġinto Ġthe ĠLM Ġfrom Ġthe Ġdec oder , ĠTable ĠT AB REF 19 Ġreports ĠW ER Ġobtained Ġfor Ġcontext Ġsizes Ġranging Ġfrom Ġ3 Ġ( com parable Ġto Ġthe Ġn - gram Ġbaseline ) Ġto Ġ50 Ġfor Ġour Ġbest Ġlanguage Ġmodel . ĠThe ĠW ER Ġdecreases Ġmon oton ically Ġuntil Ġa Ġcontext Ġsize Ġof Ġabout Ġ20 , Ġand Ġthen Ġalmost Ġstays Ġstill . ĠWe Ġobserve Ġthat Ġthe Ġconv olution al ĠLM Ġalready Ġimproves Ġon Ġthe Ġn - gram Ġmodel Ġeven Ġwith Ġthe Ġsame Ġcontext Ġsize . ĠIncreasing Ġthe Ġcontext Ġgives Ġa Ġsignificant Ġboost Ġin Ġperformance , Ġwith Ġthe Ġmajor Ġgains Ġobtained Ġbetween Ġa Ġcontext Ġof Ġ3 Ġto Ġ9 Ġ( ĠIN LINE FORM 0 Ġabsolute ĠW ER ). </s> Conclusion </s> We Ġintroduced Ġthe Ġfirst Ġfully Ġconv olution al Ġpipeline Ġfor Ġspeech Ġrecognition , Ġthat Ġcan Ġdirectly Ġprocess Ġthe Ġraw Ġwave form Ġand Ġshows Ġstate - of - the Ġart Ġperformance Ġon ĠWall ĠStreet ĠJournal Ġand Ġon ĠLib ris pe ech Ġamong Ġend - to - end Ġsystems . ĠThis Ġfirst Ġattempt Ġat Ġexploiting Ġconv olution al Ġlanguage Ġmodels Ġin Ġspeech Ġrecognition Ġshows Ġsignificant Ġimprovement Ġover Ġa Ġ4 - gram Ġlanguage Ġmodel Ġon Ġboth Ġdatasets . ĠRepl acing Ġmel - filter banks Ġby Ġa Ġlearn able Ġfront - end Ġgives Ġadditional Ġgains Ġin Ġperformance , Ġthat Ġappear Ġto Ġbe Ġmore Ġprevalent Ġon Ġnoisy Ġdata . ĠThis Ġsuggests Ġlearning Ġthe Ġfront - end Ġis Ġa Ġpromising Ġavenue Ġfor Ġspeech Ġrecognition Ġwith Ġchallenging Ġrecording Ġconditions .
Introduction Recent work on convolutional neural network architectures showed that they are competitive with recurrent architectures even on tasks where modeling long-range dependencies is critical, such as language modeling BIBREF0 , machine translation BIBREF1 , BIBREF2 and speech synthesis BIBREF3 . In end-to-end speech recognition however, recurrent architectures are still prevalent for acoustic and/or language modeling BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 . There is a history of using convolutional networks in speech recognition, but only as part of an otherwise more traditional pipeline. They have been first introduced as TDNNs to predict phoneme classes BIBREF9 , and later to generate HMM posteriorgrams BIBREF10 . They have more recently been used in end-to-end frameworks, but only in combination with recurrent layers BIBREF6 , or n-gram language models BIBREF11 , or for phone recognition BIBREF12 , BIBREF13 . Nonetheless, convolutional architectures are prevalent when learning from the raw waveform BIBREF14 , BIBREF15 , BIBREF16 , BIBREF13 , BIBREF17 , because they naturally model the computation of standard features such as mel-filterbanks. Given the evidence that they are also suitable on long-range dependency tasks, we expect convolutional neural networks to be competitive at all levels of the speech recognition pipeline. In this paper, we present a fully convolutional approach to end-to-end speech recognition. Building on recent advances in convolutional learnable front-ends for speech BIBREF13 , BIBREF17 , convolutional acoustic models BIBREF11 , and convolutional language models BIBREF0 , the paper has four main contributions: Model Our approach, described in this section, is illustrated in Fig. FIGREF5 . Convolutional Front end Several proposals to learn the front-end of speech recognition systems have been made BIBREF15 , BIBREF16 , BIBREF13 , BIBREF17 . Following the comparison in BIBREF17 , we consider their best architecture, called "scattering based" (hereafter refered to as learnable front-end). The learnable front-end contains first a convolution of width 2 that emulates the pre-emphasis step used in mel-filterbanks. It is followed by a complex convolution of width 25ms and INLINEFORM0 filters. After taking the squared absolute value, a low-pass filter of width 25ms and stride 10ms performs decimation. The front-end finally applies a log-compression and a per-channel mean-variance normalization (equivalent to an instance normalization layer BIBREF18 ). Following BIBREF17 , the "pre-emphasis" convolution is initialized to INLINEFORM1 , and then trained with the rest of the network. The low-pass filter is kept constant to a squared Hanning window, and the complex convolutional layer is initialized randomly. In addition to the INLINEFORM2 filters used by BIBREF17 , we experiment with INLINEFORM3 filters. Notice that since the stride is the same as for mel-filterbanks, acoustic models on top of the learnable front-ends can also be applied to mel-filterbanks (simply modifying the number of input channels if INLINEFORM4 ). Convolutional Acoustic Model The acoustic model is a convolutional neural network with gated linear units BIBREF0 , which is fed with the output of the learnable front-end. Following BIBREF11 , the networks uses a growing number of channels, and dropout BIBREF19 for regularization. These acoustic models are trained to predict letters directly with the Auto Segmentation Criterion (ASG) BIBREF20 . The only differences between the WSJ and Librispeech models are their depth, the number of feature maps per layer, the receptive field and the amount of dropout. Convolutional Language Model The convolutional language model (LM) is the GCNN-14B from BIBREF0 , which achieved competitive results on several language modeling benchmarks. The network contains 14 convolutional residual blocks BIBREF21 with a growing number of channels, and uses gated linear units as activation function. The language model is used to score candidate transcriptions in addition to the acoustic model in the beam search decoder described in the next section. Compared to n-gram LMs, convolutional LMs allow for much larger context sizes. Our detailed experiments study the effect of context size on the final speech recognition performance. Beam-search decoder We use the beam-search decoder presented in BIBREF11 to generate word sequences given the output from our acoustic model. The decoder finds the word transcription INLINEFORM0 to maximize: INLINEFORM1 where INLINEFORM0 is the value for the INLINEFORM1 th frame in the path leading to INLINEFORM2 and INLINEFORM3 is the (unnormalized) acoustic model score of the transcription INLINEFORM4 . The hyperparameters INLINEFORM5 respectively control the weight of the language model, the word insertion reward, and the silence insertion penalty. The other parameters are the beam size and the beam score, a threshold under which candidates are discarded even if the beam is not full. These are chosen according to a trade-off between (near-)optimality of the search and computational cost. Experiments We evaluate our approach on the large vocabulary task of the Wall Street Journal (WSJ) dataset BIBREF25 , which contains 80 hours of clean read speech, and Librispeech BIBREF26 , which contains 1000 hours with separate train/dev/test splits for clean and noisy speech. Each dataset comes with official textual data to train language models, which contain 37 million tokens for WSJ, 800 million tokens for Librispeech. Our language models are trained separately for each dataset on the official text data only. These datasets were chosen to study the impact of the different components of our system at different scales of training data and in different recording conditions. The models are evaluated in Word Error Rate (WER). Our experiments use the open source codes of wav2letter for the acoustic model, and fairseq for the language model. More details on the experimental setup are given below. Baseline Our baseline for each dataset follows BIBREF11 . It uses the same convolutional acoustic model as our approach but a mel-filterbanks front-end and a 4-gram language model. Training/test splits On WSJ, models are trained on si284. nov93dev is used for validation and nov92 for test. On Librispeech, we train on the concatenation of train-clean and train-other. The validation set is dev-clean when testing on test-clean, and dev-other when testing on test-other. Acoustic model architecture The architecture for the convolutional acoustic model is the "high dropout" model from BIBREF11 for Librispeech, which has 19 layers in addition to the front-end (mel-filterbanks for the baseline, or the learnable front-end for our approach). On WSJ, we use the lighter version used in BIBREF17 , which has 17 layers. Dropout is applied at each layer after the front-end, following BIBREF20 . The learnable front-end uses 40 or 80 filters. Language model architecture As described in Section SECREF8 , we use the GCNN-14B model of BIBREF0 with dropout at each convolutional and linear layer on both WSJ and Librispeech. We keep all the words (162K) in WSJ training corpus. For Librispeech, we only use the most frequent 200K tokens (out of 900K). Hyperparameter tuning The acoustic models are trained following BIBREF11 , BIBREF17 , using SGD with a decreasing learning rate, weight normalization and gradient clipping at 0.2 and a momentum of 0.9. The language models are trained with Nesterov accelerated gradient BIBREF27 . Following BIBREF0 , we also use weight normalization and gradient clipping. The parameters of the beam search (see Section SECREF9 ) INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are tuned on the validation set with a beam size of 2500 and a beam score of 26 for computational efficiency. Once INLINEFORM3 are chosen, the test WER is computed with a beam size of 3000 and a beam score of 50. Word Error Rate results Table TABREF11 shows Word Error Rates (WER) on WSJ for the current state-of-the-art and our models. The current best model trained on this dataset is an HMM-based system which uses a combination of convolutional, recurrent and fully connected layers, as well as speaker adaptation, and reaches INLINEFORM0 WER on nov92. DeepSpeech 2 shows a WER of INLINEFORM1 but uses 150 times more training data for the acoustic model and huge text datasets for LM training. Finally, the state-of-the-art among end-to-end systems trained only on WSJ, and hence the most comparable to our system, uses lattice-free MMI on augmented data (with speed perturbation) and gets INLINEFORM2 WER. Our baseline system, trained on mel-filterbanks, and decoded with a n-gram language model has a INLINEFORM3 WER. Replacing the n-gram LM by a convolutional one reduces the WER to INLINEFORM4 , and puts our model on par with the current best end-to-end system. Replacing the speech features by a learnable frontend finally reduces the WER to INLINEFORM5 and then to INLINEFORM6 when doubling the number of learnable filters, improving over DeepSpeech 2 and matching the performance of the best HMM-DNN system. Table TABREF10 reports WER on the Librispeech dataset. The CAPIO BIBREF22 ensemble model combines the lattices from 8 individual HMM-DNN systems (using both convolutional and LSTM layers), and is the current state-of-the-art on Librispeech. CAPIO (single) is the best individual system, selected either on dev-clean or dev-other. The sequence-to-sequence baseline is an encoder-decoder with attention and a BPE-level BIBREF28 LM, and currently the best end-to-end system on this dataset. We can observe that our fully convolutional model improves over CAPIO (Single) on the clean part, and is the current best end-to-end system on test-other with an improvement of INLINEFORM0 absolute. Our system also outperforms DeepSpeech 2 on both test sets by a significant margin. An interesting observation is the impact of each convolutional block. While replacing the 4-gram LM by a convolutional LM improves similarly on the clean and noisier parts, learning the speech frontend gives similar performance on the clean part but significantly improves the performance on noisier, harder utterances, a finding that is consistent with previous literature BIBREF15 . Analysis of the convolutional language model Since this paper uses convolutional language models for speech recognition systems for the first time, we present additional studies of the language model in isolation. These experiments use our best language model on Librispeech, and evaluations in WER are carried out using the baseline system trained on mel-filterbanks. The decoder parameters are tuned using the grid search described in Section SECREF3 , a beam size is fixed to 2500 and a beam score to 30. Correlation between perplexity and WER Figure FIGREF18 shows the correlation between perplexity and WER as the training progresses. As perplexity decreases, the WER on both dev-clean and dev-other also decreases following the same trend. It illustrates that perplexity on the linguistic data is a good surrogate of the final performance of the speech recognition pipeline. Architectural choices or hyper-parameter tuning can thus be carried out mostly using perplexity alone. Influence of context size By limiting the context passed into the LM from the decoder, Table TABREF19 reports WER obtained for context sizes ranging from 3 (comparable to the n-gram baseline) to 50 for our best language model. The WER decreases monotonically until a context size of about 20, and then almost stays still. We observe that the convolutional LM already improves on the n-gram model even with the same context size. Increasing the context gives a significant boost in performance, with the major gains obtained between a context of 3 to 9 ( INLINEFORM0 absolute WER). Conclusion We introduced the first fully convolutional pipeline for speech recognition, that can directly process the raw waveform and shows state-of-the art performance on Wall Street Journal and on Librispeech among end-to-end systems. This first attempt at exploiting convolutional language models in speech recognition shows significant improvement over a 4-gram language model on both datasets. Replacing mel-filterbanks by a learnable front-end gives additional gains in performance, that appear to be more prevalent on noisy data. This suggests learning the front-end is a promising avenue for speech recognition with challenging recording conditions.
what is the state of the art on WSJ?
70e9210fe64f8d71334e5107732d764332a81cb1
what Ġis Ġthe Ġstate Ġof Ġthe Ġart Ġon ĠWS J ?
CNN - D NN - BL ST M - H MM
<s> what Ġis Ġthe Ġstate Ġof Ġthe Ġart Ġon ĠWS J ? </s> Introduction </s> Recent Ġwork Ġon Ġconv olution al Ġneural Ġnetwork Ġarchitectures Ġshowed Ġthat Ġthey Ġare Ġcompetitive Ġwith Ġrecurrent Ġarchitectures Ġeven Ġon Ġtasks Ġwhere Ġmodeling Ġlong - range Ġdependencies Ġis Ġcritical , Ġsuch Ġas Ġlanguage Ġmodeling ĠB IB REF 0 Ġ, Ġmachine Ġtranslation ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand Ġspeech Ġsynthesis ĠB IB REF 3 Ġ. ĠIn Ġend - to - end Ġspeech Ġrecognition Ġhowever , Ġrecurrent Ġarchitectures Ġare Ġstill Ġprevalent Ġfor Ġacoustic Ġand / or Ġlanguage Ġmodeling ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> There Ġis Ġa Ġhistory Ġof Ġusing Ġconv olution al Ġnetworks Ġin Ġspeech Ġrecognition , Ġbut Ġonly Ġas Ġpart Ġof Ġan Ġotherwise Ġmore Ġtraditional Ġpipeline . ĠThey Ġhave Ġbeen Ġfirst Ġintroduced Ġas ĠTD NN s Ġto Ġpredict Ġphon eme Ġclasses ĠB IB REF 9 Ġ, Ġand Ġlater Ġto Ġgenerate ĠH MM Ġposterior gram s ĠB IB REF 10 Ġ. ĠThey Ġhave Ġmore Ġrecently Ġbeen Ġused Ġin Ġend - to - end Ġframeworks , Ġbut Ġonly Ġin Ġcombination Ġwith Ġrecurrent Ġlayers ĠB IB REF 6 Ġ, Ġor Ġn - gram Ġlanguage Ġmodels ĠB IB REF 11 Ġ, Ġor Ġfor Ġphone Ġrecognition ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠNonetheless , Ġconv olution al Ġarchitectures Ġare Ġprevalent Ġwhen Ġlearning Ġfrom Ġthe Ġraw Ġwave form ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ, Ġbecause Ġthey Ġnaturally Ġmodel Ġthe Ġcomputation Ġof Ġstandard Ġfeatures Ġsuch Ġas Ġmel - filter banks . ĠGiven Ġthe Ġevidence Ġthat Ġthey Ġare Ġalso Ġsuitable Ġon Ġlong - range Ġdependency Ġtasks , Ġwe Ġexpect Ġconv olution al Ġneural Ġnetworks Ġto Ġbe Ġcompetitive Ġat Ġall Ġlevels Ġof Ġthe Ġspeech Ġrecognition Ġpipeline . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġfully Ġconv olution al Ġapproach Ġto Ġend - to - end Ġspeech Ġrecognition . ĠBuilding Ġon Ġrecent Ġadvances Ġin Ġconv olution al Ġlearn able Ġfront - ends Ġfor Ġspeech ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ, Ġconv olution al Ġacoustic Ġmodels ĠB IB REF 11 Ġ, Ġand Ġconv olution al Ġlanguage Ġmodels ĠB IB REF 0 Ġ, Ġthe Ġpaper Ġhas Ġfour Ġmain Ġcontributions : </s> Model </s> Our Ġapproach , Ġdescribed Ġin Ġthis Ġsection , Ġis Ġillustrated Ġin ĠFig . ĠFIG REF 5 Ġ. </s> Con v olution al ĠFront Ġend </s> Several Ġproposals Ġto Ġlearn Ġthe Ġfront - end Ġof Ġspeech Ġrecognition Ġsystems Ġhave Ġbeen Ġmade ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ. ĠFollowing Ġthe Ġcomparison Ġin ĠB IB REF 17 Ġ, Ġwe Ġconsider Ġtheir Ġbest Ġarchitecture , Ġcalled Ġ" sc attering Ġbased " Ġ( here after Ġref ered Ġto Ġas Ġlearn able Ġfront - end ). ĠThe Ġlearn able Ġfront - end Ġcontains Ġfirst Ġa Ġconv olution Ġof Ġwidth Ġ2 Ġthat Ġem ulates Ġthe Ġpre - emphasis Ġstep Ġused Ġin Ġmel - filter banks . ĠIt Ġis Ġfollowed Ġby Ġa Ġcomplex Ġconv olution Ġof Ġwidth Ġ25 ms Ġand ĠIN LINE FORM 0 Ġfilters . ĠAfter Ġtaking Ġthe Ġsquared Ġabsolute Ġvalue , Ġa Ġlow - pass Ġfilter Ġof Ġwidth Ġ25 ms Ġand Ġstride Ġ10 ms Ġperforms Ġdec imation . ĠThe Ġfront - end Ġfinally Ġapplies Ġa Ġlog - comp ression Ġand Ġa Ġper - channel Ġmean - vari ance Ġnormal ization Ġ( equ ivalent Ġto Ġan Ġinstance Ġnormal ization Ġlayer ĠB IB REF 18 Ġ). ĠFollowing ĠB IB REF 17 Ġ, Ġthe Ġ" pre - emphasis " Ġconv olution Ġis Ġinitialized Ġto ĠIN LINE FORM 1 Ġ, Ġand Ġthen Ġtrained Ġwith Ġthe Ġrest Ġof Ġthe Ġnetwork . ĠThe Ġlow - pass Ġfilter Ġis Ġkept Ġconstant Ġto Ġa Ġsquared ĠHan ning Ġwindow , Ġand Ġthe Ġcomplex Ġconv olution al Ġlayer Ġis Ġinitialized Ġrandomly . ĠIn Ġaddition Ġto Ġthe ĠIN LINE FORM 2 Ġfilters Ġused Ġby ĠB IB REF 17 Ġ, Ġwe Ġexperiment Ġwith ĠIN LINE FORM 3 Ġfilters . ĠNotice Ġthat Ġsince Ġthe Ġstride Ġis Ġthe Ġsame Ġas Ġfor Ġmel - filter banks , Ġacoustic Ġmodels Ġon Ġtop Ġof Ġthe Ġlearn able Ġfront - ends Ġcan Ġalso Ġbe Ġapplied Ġto Ġmel - filter banks Ġ( sim ply Ġmodifying Ġthe Ġnumber Ġof Ġinput Ġchannels Ġif ĠIN LINE FORM 4 Ġ). </s> Con v olution al ĠAc oustic ĠModel </s> The Ġacoustic Ġmodel Ġis Ġa Ġconv olution al Ġneural Ġnetwork Ġwith Ġg ated Ġlinear Ġunits ĠB IB REF 0 Ġ, Ġwhich Ġis Ġfed Ġwith Ġthe Ġoutput Ġof Ġthe Ġlearn able Ġfront - end . ĠFollowing ĠB IB REF 11 Ġ, Ġthe Ġnetworks Ġuses Ġa Ġgrowing Ġnumber Ġof Ġchannels , Ġand Ġdrop out ĠB IB REF 19 Ġfor Ġregular ization . ĠThese Ġacoustic Ġmodels Ġare Ġtrained Ġto Ġpredict Ġletters Ġdirectly Ġwith Ġthe ĠAuto ĠSe gment ation ĠCrit erion Ġ( AS G ) ĠB IB REF 20 Ġ. ĠThe Ġonly Ġdifferences Ġbetween Ġthe ĠWS J Ġand ĠLib ris pe ech Ġmodels Ġare Ġtheir Ġdepth , Ġthe Ġnumber Ġof Ġfeature Ġmaps Ġper Ġlayer , Ġthe Ġreceptive Ġfield Ġand Ġthe Ġamount Ġof Ġdrop out . </s> Con v olution al ĠLanguage ĠModel </s> The Ġconv olution al Ġlanguage Ġmodel Ġ( LM ) Ġis Ġthe ĠG CNN - 14 B Ġfrom ĠB IB REF 0 Ġ, Ġwhich Ġachieved Ġcompetitive Ġresults Ġon Ġseveral Ġlanguage Ġmodeling Ġbenchmarks . ĠThe Ġnetwork Ġcontains Ġ14 Ġconv olution al Ġresidual Ġblocks ĠB IB REF 21 Ġwith Ġa Ġgrowing Ġnumber Ġof Ġchannels , Ġand Ġuses Ġg ated Ġlinear Ġunits Ġas Ġactivation Ġfunction . </s> The Ġlanguage Ġmodel Ġis Ġused Ġto Ġscore Ġcandidate Ġtranscript ions Ġin Ġaddition Ġto Ġthe Ġacoustic Ġmodel Ġin Ġthe Ġbeam Ġsearch Ġdec oder Ġdescribed Ġin Ġthe Ġnext Ġsection . ĠCompared Ġto Ġn - gram ĠL Ms , Ġconv olution al ĠL Ms Ġallow Ġfor Ġmuch Ġlarger Ġcontext Ġsizes . ĠOur Ġdetailed Ġexperiments Ġstudy Ġthe Ġeffect Ġof Ġcontext Ġsize Ġon Ġthe Ġfinal Ġspeech Ġrecognition Ġperformance . </s> Be am - search Ġdec oder </s> We Ġuse Ġthe Ġbeam - search Ġdec oder Ġpresented Ġin ĠB IB REF 11 Ġto Ġgenerate Ġword Ġsequences Ġgiven Ġthe Ġoutput Ġfrom Ġour Ġacoustic Ġmodel . ĠThe Ġdec oder Ġfinds Ġthe Ġword Ġtranscription ĠIN LINE FORM 0 Ġto Ġmaximize : ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġvalue Ġfor Ġthe ĠIN LINE FORM 1 Ġth Ġframe Ġin Ġthe Ġpath Ġleading Ġto ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġ( un normal ized ) Ġacoustic Ġmodel Ġscore Ġof Ġthe Ġtranscription ĠIN LINE FORM 4 Ġ. ĠThe Ġhyper param eters ĠIN LINE FORM 5 Ġrespectively Ġcontrol Ġthe Ġweight Ġof Ġthe Ġlanguage Ġmodel , Ġthe Ġword Ġinsertion Ġreward , Ġand Ġthe Ġsilence Ġinsertion Ġpenalty . ĠThe Ġother Ġparameters Ġare Ġthe Ġbeam Ġsize Ġand Ġthe Ġbeam Ġscore , Ġa Ġthreshold Ġunder Ġwhich Ġcandidates Ġare Ġdiscarded Ġeven Ġif Ġthe Ġbeam Ġis Ġnot Ġfull . ĠThese Ġare Ġchosen Ġaccording Ġto Ġa Ġtrade - off Ġbetween Ġ( near -) optim ality Ġof Ġthe Ġsearch Ġand Ġcomputational Ġcost . </s> Exper iments </s> We Ġevaluate Ġour Ġapproach Ġon Ġthe Ġlarge Ġvocabulary Ġtask Ġof Ġthe ĠWall ĠStreet ĠJournal Ġ( WS J ) Ġdataset ĠB IB REF 25 Ġ, Ġwhich Ġcontains Ġ80 Ġhours Ġof Ġclean Ġread Ġspeech , Ġand ĠLib ris pe ech ĠB IB REF 26 Ġ, Ġwhich Ġcontains Ġ1000 Ġhours Ġwith Ġseparate Ġtrain / dev / test Ġsplits Ġfor Ġclean Ġand Ġnoisy Ġspeech . ĠEach Ġdataset Ġcomes Ġwith Ġofficial Ġtextual Ġdata Ġto Ġtrain Ġlanguage Ġmodels , Ġwhich Ġcontain Ġ37 Ġmillion Ġtokens Ġfor ĠWS J , Ġ800 Ġmillion Ġtokens Ġfor ĠLib ris pe ech . ĠOur Ġlanguage Ġmodels Ġare Ġtrained Ġseparately Ġfor Ġeach Ġdataset Ġon Ġthe Ġofficial Ġtext Ġdata Ġonly . ĠThese Ġdatasets Ġwere Ġchosen Ġto Ġstudy Ġthe Ġimpact Ġof Ġthe Ġdifferent Ġcomponents Ġof Ġour Ġsystem Ġat Ġdifferent Ġscales Ġof Ġtraining Ġdata Ġand Ġin Ġdifferent Ġrecording Ġconditions . </s> The Ġmodels Ġare Ġevaluated Ġin ĠWord ĠError ĠRate Ġ( WER ). ĠOur Ġexperiments Ġuse Ġthe Ġopen Ġsource Ġcodes Ġof Ġw av 2 letter Ġfor Ġthe Ġacoustic Ġmodel , Ġand Ġfair seq Ġfor Ġthe Ġlanguage Ġmodel . ĠMore Ġdetails Ġon Ġthe Ġexperimental Ġsetup Ġare Ġgiven Ġbelow . </s> Bas eline ĠOur Ġbaseline Ġfor Ġeach Ġdataset Ġfollows ĠB IB REF 11 Ġ. ĠIt Ġuses Ġthe Ġsame Ġconv olution al Ġacoustic Ġmodel Ġas Ġour Ġapproach Ġbut Ġa Ġmel - filter banks Ġfront - end Ġand Ġa Ġ4 - gram Ġlanguage Ġmodel . </s> Training / test Ġsplits ĠOn ĠWS J , Ġmodels Ġare Ġtrained Ġon Ġsi 284 . Ġno v 93 dev Ġis Ġused Ġfor Ġvalidation Ġand Ġno v 92 Ġfor Ġtest . ĠOn ĠLib ris pe ech , Ġwe Ġtrain Ġon Ġthe Ġconc aten ation Ġof Ġtrain - clean Ġand Ġtrain - other . ĠThe Ġvalidation Ġset Ġis Ġdev - clean Ġwhen Ġtesting Ġon Ġtest - clean , Ġand Ġdev - other Ġwhen Ġtesting Ġon Ġtest - other . </s> Ac oustic Ġmodel Ġarchitecture ĠThe Ġarchitecture Ġfor Ġthe Ġconv olution al Ġacoustic Ġmodel Ġis Ġthe Ġ" high Ġdrop out " Ġmodel Ġfrom ĠB IB REF 11 Ġfor ĠLib ris pe ech , Ġwhich Ġhas Ġ19 Ġlayers Ġin Ġaddition Ġto Ġthe Ġfront - end Ġ( mel - filter banks Ġfor Ġthe Ġbaseline , Ġor Ġthe Ġlearn able Ġfront - end Ġfor Ġour Ġapproach ). ĠOn ĠWS J , Ġwe Ġuse Ġthe Ġlighter Ġversion Ġused Ġin ĠB IB REF 17 Ġ, Ġwhich Ġhas Ġ17 Ġlayers . ĠDrop out Ġis Ġapplied Ġat Ġeach Ġlayer Ġafter Ġthe Ġfront - end , Ġfollowing ĠB IB REF 20 Ġ. ĠThe Ġlearn able Ġfront - end Ġuses Ġ40 Ġor Ġ80 Ġfilters . ĠLanguage Ġmodel Ġarchitecture ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 8 Ġ, Ġwe Ġuse Ġthe ĠG CNN - 14 B Ġmodel Ġof ĠB IB REF 0 Ġwith Ġdrop out Ġat Ġeach Ġconv olution al Ġand Ġlinear Ġlayer Ġon Ġboth ĠWS J Ġand ĠLib ris pe ech . ĠWe Ġkeep Ġall Ġthe Ġwords Ġ( 162 K ) Ġin ĠWS J Ġtraining Ġcorpus . ĠFor ĠLib ris pe ech , Ġwe Ġonly Ġuse Ġthe Ġmost Ġfrequent Ġ200 K Ġtokens Ġ( out Ġof Ġ900 K ). </s> Hyper param eter Ġtuning ĠThe Ġacoustic Ġmodels Ġare Ġtrained Ġfollowing ĠB IB REF 11 Ġ, ĠB IB REF 17 Ġ, Ġusing ĠSG D Ġwith Ġa Ġdecreasing Ġlearning Ġrate , Ġweight Ġnormal ization Ġand Ġgradient Ġclipping Ġat Ġ0 . 2 Ġand Ġa Ġmomentum Ġof Ġ0 . 9 . ĠThe Ġlanguage Ġmodels Ġare Ġtrained Ġwith ĠN ester ov Ġaccelerated Ġgradient ĠB IB REF 27 Ġ. ĠFollowing ĠB IB REF 0 Ġ, Ġwe Ġalso Ġuse Ġweight Ġnormal ization Ġand Ġgradient Ġclipping . </s> The Ġparameters Ġof Ġthe Ġbeam Ġsearch Ġ( see ĠSection ĠSEC REF 9 Ġ) ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġtuned Ġon Ġthe Ġvalidation Ġset Ġwith Ġa Ġbeam Ġsize Ġof Ġ2500 Ġand Ġa Ġbeam Ġscore Ġof Ġ26 Ġfor Ġcomputational Ġefficiency . ĠOnce ĠIN LINE FORM 3 Ġare Ġchosen , Ġthe Ġtest ĠW ER Ġis Ġcomputed Ġwith Ġa Ġbeam Ġsize Ġof Ġ3000 Ġand Ġa Ġbeam Ġscore Ġof Ġ50 . </s> Word ĠError ĠRate Ġresults </s> Table ĠT AB REF 11 Ġshows ĠWord ĠError ĠRates Ġ( WER ) Ġon ĠWS J Ġfor Ġthe Ġcurrent Ġstate - of - the - art Ġand Ġour Ġmodels . ĠThe Ġcurrent Ġbest Ġmodel Ġtrained Ġon Ġthis Ġdataset Ġis Ġan ĠH MM - based Ġsystem Ġwhich Ġuses Ġa Ġcombination Ġof Ġconv olution al , Ġrecurrent Ġand Ġfully Ġconnected Ġlayers , Ġas Ġwell Ġas Ġspeaker Ġadaptation , Ġand Ġreaches ĠIN LINE FORM 0 ĠW ER Ġon Ġno v 92 . ĠDeep Spe ech Ġ2 Ġshows Ġa ĠW ER Ġof ĠIN LINE FORM 1 Ġbut Ġuses Ġ150 Ġtimes Ġmore Ġtraining Ġdata Ġfor Ġthe Ġacoustic Ġmodel Ġand Ġhuge Ġtext Ġdatasets Ġfor ĠLM Ġtraining . ĠFinally , Ġthe Ġstate - of - the - art Ġamong Ġend - to - end Ġsystems Ġtrained Ġonly Ġon ĠWS J , Ġand Ġhence Ġthe Ġmost Ġcomparable Ġto Ġour Ġsystem , Ġuses Ġlatt ice - free ĠM MI Ġon Ġaugmented Ġdata Ġ( with Ġspeed Ġpert urb ation ) Ġand Ġgets ĠIN LINE FORM 2 ĠW ER . ĠOur Ġbaseline Ġsystem , Ġtrained Ġon Ġmel - filter banks , Ġand Ġdec oded Ġwith Ġa Ġn - gram Ġlanguage Ġmodel Ġhas Ġa ĠIN LINE FORM 3 ĠW ER . ĠRepl acing Ġthe Ġn - gram ĠLM Ġby Ġa Ġconv olution al Ġone Ġreduces Ġthe ĠW ER Ġto ĠIN LINE FORM 4 Ġ, Ġand Ġputs Ġour Ġmodel Ġon Ġpar Ġwith Ġthe Ġcurrent Ġbest Ġend - to - end Ġsystem . ĠRepl acing Ġthe Ġspeech Ġfeatures Ġby Ġa Ġlearn able Ġfront end Ġfinally Ġreduces Ġthe ĠW ER Ġto ĠIN LINE FORM 5 Ġand Ġthen Ġto ĠIN LINE FORM 6 Ġwhen Ġdoubling Ġthe Ġnumber Ġof Ġlearn able Ġfilters , Ġimproving Ġover ĠDeep Spe ech Ġ2 Ġand Ġmatching Ġthe Ġperformance Ġof Ġthe Ġbest ĠH MM - D NN Ġsystem . </s> Table ĠT AB REF 10 Ġreports ĠW ER Ġon Ġthe ĠLib ris pe ech Ġdataset . ĠThe ĠCAP IO ĠB IB REF 22 Ġensemble Ġmodel Ġcombines Ġthe Ġlatt ices Ġfrom Ġ8 Ġindividual ĠH MM - D NN Ġsystems Ġ( using Ġboth Ġconv olution al Ġand ĠL ST M Ġlayers ), Ġand Ġis Ġthe Ġcurrent Ġstate - of - the - art Ġon ĠLib ris pe ech . ĠCAP IO Ġ( single ) Ġis Ġthe Ġbest Ġindividual Ġsystem , Ġselected Ġeither Ġon Ġdev - clean Ġor Ġdev - other . ĠThe Ġsequence - to - sequence Ġbaseline Ġis Ġan Ġenc oder - dec oder Ġwith Ġattention Ġand Ġa ĠB PE - level ĠB IB REF 28 ĠLM , Ġand Ġcurrently Ġthe Ġbest Ġend - to - end Ġsystem Ġon Ġthis Ġdataset . ĠWe Ġcan Ġobserve Ġthat Ġour Ġfully Ġconv olution al Ġmodel Ġimproves Ġover ĠCAP IO Ġ( Single ) Ġon Ġthe Ġclean Ġpart , Ġand Ġis Ġthe Ġcurrent Ġbest Ġend - to - end Ġsystem Ġon Ġtest - other Ġwith Ġan Ġimprovement Ġof ĠIN LINE FORM 0 Ġabsolute . ĠOur Ġsystem Ġalso Ġoutper forms ĠDeep Spe ech Ġ2 Ġon Ġboth Ġtest Ġsets Ġby Ġa Ġsignificant Ġmargin . ĠAn Ġinteresting Ġobservation Ġis Ġthe Ġimpact Ġof Ġeach Ġconv olution al Ġblock . ĠWhile Ġreplacing Ġthe Ġ4 - gram ĠLM Ġby Ġa Ġconv olution al ĠLM Ġimproves Ġsimilarly Ġon Ġthe Ġclean Ġand Ġno is ier Ġparts , Ġlearning Ġthe Ġspeech Ġfront end Ġgives Ġsimilar Ġperformance Ġon Ġthe Ġclean Ġpart Ġbut Ġsignificantly Ġimproves Ġthe Ġperformance Ġon Ġno is ier , Ġharder Ġutter ances , Ġa Ġfinding Ġthat Ġis Ġconsistent Ġwith Ġprevious Ġliterature ĠB IB REF 15 Ġ. </s> Analysis Ġof Ġthe Ġconv olution al Ġlanguage Ġmodel </s> Since Ġthis Ġpaper Ġuses Ġconv olution al Ġlanguage Ġmodels Ġfor Ġspeech Ġrecognition Ġsystems Ġfor Ġthe Ġfirst Ġtime , Ġwe Ġpresent Ġadditional Ġstudies Ġof Ġthe Ġlanguage Ġmodel Ġin Ġisolation . ĠThese Ġexperiments Ġuse Ġour Ġbest Ġlanguage Ġmodel Ġon ĠLib ris pe ech , Ġand Ġevaluations Ġin ĠW ER Ġare Ġcarried Ġout Ġusing Ġthe Ġbaseline Ġsystem Ġtrained Ġon Ġmel - filter banks . ĠThe Ġdec oder Ġparameters Ġare Ġtuned Ġusing Ġthe Ġgrid Ġsearch Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ, Ġa Ġbeam Ġsize Ġis Ġfixed Ġto Ġ2500 Ġand Ġa Ġbeam Ġscore Ġto Ġ30 . </s> Cor relation Ġbetween Ġperplex ity Ġand ĠW ER ĠFigure ĠFIG REF 18 Ġshows Ġthe Ġcorrelation Ġbetween Ġperplex ity Ġand ĠW ER Ġas Ġthe Ġtraining Ġprogresses . ĠAs Ġperplex ity Ġdecreases , Ġthe ĠW ER Ġon Ġboth Ġdev - clean Ġand Ġdev - other Ġalso Ġdecreases Ġfollowing Ġthe Ġsame Ġtrend . ĠIt Ġillustrates Ġthat Ġperplex ity Ġon Ġthe Ġlinguistic Ġdata Ġis Ġa Ġgood Ġsurrogate Ġof Ġthe Ġfinal Ġperformance Ġof Ġthe Ġspeech Ġrecognition Ġpipeline . ĠArchitect ural Ġchoices Ġor Ġhyper - param eter Ġtuning Ġcan Ġthus Ġbe Ġcarried Ġout Ġmostly Ġusing Ġperplex ity Ġalone . </s> Inf luence Ġof Ġcontext Ġsize ĠBy Ġlimiting Ġthe Ġcontext Ġpassed Ġinto Ġthe ĠLM Ġfrom Ġthe Ġdec oder , ĠTable ĠT AB REF 19 Ġreports ĠW ER Ġobtained Ġfor Ġcontext Ġsizes Ġranging Ġfrom Ġ3 Ġ( com parable Ġto Ġthe Ġn - gram Ġbaseline ) Ġto Ġ50 Ġfor Ġour Ġbest Ġlanguage Ġmodel . ĠThe ĠW ER Ġdecreases Ġmon oton ically Ġuntil Ġa Ġcontext Ġsize Ġof Ġabout Ġ20 , Ġand Ġthen Ġalmost Ġstays Ġstill . ĠWe Ġobserve Ġthat Ġthe Ġconv olution al ĠLM Ġalready Ġimproves Ġon Ġthe Ġn - gram Ġmodel Ġeven Ġwith Ġthe Ġsame Ġcontext Ġsize . ĠIncreasing Ġthe Ġcontext Ġgives Ġa Ġsignificant Ġboost Ġin Ġperformance , Ġwith Ġthe Ġmajor Ġgains Ġobtained Ġbetween Ġa Ġcontext Ġof Ġ3 Ġto Ġ9 Ġ( ĠIN LINE FORM 0 Ġabsolute ĠW ER ). </s> Conclusion </s> We Ġintroduced Ġthe Ġfirst Ġfully Ġconv olution al Ġpipeline Ġfor Ġspeech Ġrecognition , Ġthat Ġcan Ġdirectly Ġprocess Ġthe Ġraw Ġwave form Ġand Ġshows Ġstate - of - the Ġart Ġperformance Ġon ĠWall ĠStreet ĠJournal Ġand Ġon ĠLib ris pe ech Ġamong Ġend - to - end Ġsystems . ĠThis Ġfirst Ġattempt Ġat Ġexploiting Ġconv olution al Ġlanguage Ġmodels Ġin Ġspeech Ġrecognition Ġshows Ġsignificant Ġimprovement Ġover Ġa Ġ4 - gram Ġlanguage Ġmodel Ġon Ġboth Ġdatasets . ĠRepl acing Ġmel - filter banks Ġby Ġa Ġlearn able Ġfront - end Ġgives Ġadditional Ġgains Ġin Ġperformance , Ġthat Ġappear Ġto Ġbe Ġmore Ġprevalent Ġon Ġnoisy Ġdata . ĠThis Ġsuggests Ġlearning Ġthe Ġfront - end Ġis Ġa Ġpromising Ġavenue Ġfor Ġspeech Ġrecognition Ġwith Ġchallenging Ġrecording Ġconditions .