Unanswerable
NONE
LF-MMI Attention
Seq2Seq 
RNN-T 
Char E2E LF-MMI 
Phone E2E LF-MMI 
CTC + Gram-CTC
ABSTRACTIVE
We also evaluate the Jasper model's performance on a conversational English corpus. The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia. It is divided into two subsets: Switchboard (SWB) and Callhome (CHM). The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62). Jasper DR 10x5 was trained using SGD with momentum for 50 epochs. We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31 . FLOAT SELECTED: Table 7: Hub5’00, WER (%)
1904.03288
Introduction </s> Con ventional Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems Ġtypically Ġconsist Ġof Ġseveral Ġindependently Ġlearned Ġcomponents : Ġan Ġacoustic Ġmodel Ġto Ġpredict Ġcontext - dependent Ġsub - ph on eme Ġstates Ġ( sen ones ) Ġfrom Ġaudio , Ġa Ġgraph Ġstructure Ġto Ġmap Ġsen ones Ġto Ġphon em es , Ġand Ġa Ġpronunciation Ġmodel Ġto Ġmap Ġphon em es Ġto Ġwords . ĠHybrid Ġsystems Ġcombine Ġhidden ĠMark ov Ġmodels Ġto Ġmodel Ġstate Ġdependencies Ġwith Ġneural Ġnetworks Ġto Ġpredict Ġstates ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠNew er Ġapproaches Ġsuch Ġas Ġend - to - end Ġ( E 2 E ) Ġsystems Ġreduce Ġthe Ġoverall Ġcomplexity Ġof Ġthe Ġfinal Ġsystem . </s> Our Ġresearch Ġbuilds Ġon Ġprior Ġwork Ġthat Ġhas Ġexplored Ġusing Ġtime - delay Ġneural Ġnetworks Ġ( TD NN ), Ġother Ġforms Ġof Ġconv olution al Ġneural Ġnetworks , Ġand ĠConnection ist ĠTem poral ĠClassification Ġ( CT C ) Ġloss ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠWe Ġtook Ġinspiration Ġfrom Ġw av 2 letter ĠB IB REF 6 Ġ, Ġwhich Ġuses Ġ1 D - conv olution Ġlayers . ĠLi pt ch insky Ġet Ġal . ĠB IB REF 7 Ġimproved Ġw av 2 letter Ġby Ġincreasing Ġthe Ġmodel Ġdepth Ġto Ġ19 Ġconv olution al Ġlayers Ġand Ġadding ĠG ated ĠLinear ĠUnits Ġ( GL U ) ĠB IB REF 8 Ġ, Ġweight Ġnormal ization ĠB IB REF 9 Ġand Ġdrop out . </s> By Ġbuilding Ġa Ġdeeper Ġand Ġlarger Ġcapacity Ġnetwork , Ġwe Ġaim Ġto Ġdemonstrate Ġthat Ġwe Ġcan Ġmatch Ġor Ġoutper form Ġnon Ġend - to - end Ġmodels Ġon Ġthe ĠLib ri Spe ech Ġand Ġ2000 hr ĠFisher + Switch board Ġtasks . ĠLike Ġw av 2 letter , Ġour Ġarchitecture , ĠJasper , Ġuses Ġa Ġstack Ġof Ġ1 D - conv olution Ġlayers , Ġbut Ġwith ĠRe LU Ġand Ġbatch Ġnormal ization ĠB IB REF 10 Ġ. ĠWe Ġfind Ġthat ĠRe LU Ġand Ġbatch Ġnormal ization Ġoutper form Ġother Ġactivation Ġand Ġnormal ization Ġschemes Ġthat Ġwe Ġtested Ġfor Ġconv olution al ĠAS R . ĠAs Ġa Ġresult , ĠJasper 's Ġarchitecture Ġcontains Ġonly Ġ1 D Ġconv olution , Ġbatch Ġnormal ization , ĠRe LU , Ġand Ġdrop out Ġlayers ĠâĢĵ Ġoperators Ġhighly Ġoptimized Ġfor Ġtraining Ġand Ġinference Ġon ĠGPUs . </s> It Ġis Ġpossible Ġto Ġincrease Ġthe Ġcapacity Ġof Ġthe ĠJasper Ġmodel Ġby Ġstacking Ġthese Ġoperations . ĠOur Ġlargest Ġversion Ġuses Ġ54 Ġconv olution al Ġlayers Ġ( 333 M Ġparameters ), Ġwhile Ġour Ġsmall Ġmodel Ġuses Ġ34 Ġ( 201 M Ġparameters ). ĠWe Ġuse Ġresidual Ġconnections Ġto Ġenable Ġthis Ġlevel Ġof Ġdepth . ĠWe Ġinvestigate Ġa Ġnumber Ġof Ġresidual Ġoptions Ġand Ġpropose Ġa Ġnew Ġresidual Ġconnection Ġtop ology Ġwe Ġcall ĠD ense ĠRes id ual Ġ( DR ). </s> Integ rating Ġour Ġbest Ġacoustic Ġmodel Ġwith Ġa ĠTrans former - XL ĠB IB REF 11 Ġlanguage Ġmodel Ġallows Ġus Ġto Ġobtain Ġnew Ġstate - of - the - art Ġ( S OTA ) Ġresults Ġon ĠLib ri Spe ech ĠB IB REF 12 Ġtest - clean Ġof Ġ2 . 95 % ĠW ER Ġand ĠS OTA Ġresults Ġamong Ġend - to - end Ġmodels Ġon ĠLib ri Spe ech Ġtest - other . ĠWe Ġshow Ġcompetitive Ġresults Ġon ĠWall ĠStreet ĠJournal Ġ( WS J ), Ġand Ġ2000 hr ĠFisher + Switch board Ġ( F + S ). ĠUsing Ġonly Ġgreedy Ġdecoding Ġwithout Ġa Ġlanguage Ġmodel Ġwe Ġachieve Ġ3 . 86 % ĠW ER Ġon ĠLib ri Spe ech Ġtest - clean . </s> This Ġpaper Ġmakes Ġthe Ġfollowing Ġcontributions : </s> J asper ĠArchitecture </s> J asper Ġis Ġa Ġfamily Ġof Ġend - to - end ĠAS R Ġmodels Ġthat Ġreplace Ġacoustic Ġand Ġpronunciation Ġmodels Ġwith Ġa Ġconv olution al Ġneural Ġnetwork . ĠJasper Ġuses Ġmel - filter bank Ġfeatures Ġcalculated Ġfrom Ġ20 ms Ġwindows Ġwith Ġa Ġ10 ms Ġoverlap , Ġand Ġoutputs Ġa Ġprobability Ġdistribution Ġover Ġcharacters Ġper Ġframe . ĠJasper Ġhas Ġa Ġblock Ġarchitecture : Ġa ĠJasper ĠIN LINE FORM 0 Ġx ĠIN LINE FORM 1 Ġmodel Ġhas ĠIN LINE FORM 2 Ġblocks , Ġeach Ġwith ĠIN LINE FORM 3 Ġsub - blocks . ĠEach Ġsub - block Ġapplies Ġthe Ġfollowing Ġoperations : Ġa Ġ1 D - conv olution , Ġbatch Ġnorm , ĠRe LU , Ġand Ġdrop out . ĠAll Ġsub - blocks Ġin Ġa Ġblock Ġhave Ġthe Ġsame Ġnumber Ġof Ġoutput Ġchannels . </s> Each Ġblock Ġinput Ġis Ġconnected Ġdirectly Ġinto Ġthe Ġlast Ġsub - block Ġvia Ġa Ġresidual Ġconnection . ĠThe Ġresidual Ġconnection Ġis Ġfirst Ġprojected Ġthrough Ġa Ġ1 x 1 Ġconv olution Ġto Ġaccount Ġfor Ġdifferent Ġnumbers Ġof Ġinput Ġand Ġoutput Ġchannels , Ġthen Ġthrough Ġa Ġbatch Ġnorm Ġlayer . ĠThe Ġoutput Ġof Ġthis Ġbatch Ġnorm Ġlayer Ġis Ġadded Ġto Ġthe Ġoutput Ġof Ġthe Ġbatch Ġnorm Ġlayer Ġin Ġthe Ġlast Ġsub - block . ĠThe Ġresult Ġof Ġthis Ġsum Ġis Ġpassed Ġthrough Ġthe Ġactivation Ġfunction Ġand Ġdrop out Ġto Ġproduce Ġthe Ġoutput Ġof Ġthe Ġsub - block . </s> The Ġsub - block Ġarchitecture Ġof ĠJasper Ġwas Ġdesigned Ġto Ġfacilitate Ġfast ĠGPU Ġinference . ĠEach Ġsub - block Ġcan Ġbe Ġfused Ġinto Ġa Ġsingle ĠGPU Ġkernel : Ġdrop out Ġis Ġnot Ġused Ġat Ġinference - time Ġand Ġis Ġeliminated , Ġbatch Ġnorm Ġcan Ġbe Ġfused Ġwith Ġthe Ġpreceding Ġconv olution , ĠRe LU Ġcl amps Ġthe Ġresult , Ġand Ġresidual Ġsumm ation Ġcan Ġbe Ġtreated Ġas Ġa Ġmodified Ġbias Ġterm Ġin Ġthis Ġfused Ġoperation . </s> All ĠJasper Ġmodels Ġhave Ġfour Ġadditional Ġconv olution al Ġblocks : Ġone Ġpre - processing Ġand Ġthree Ġpost - processing . ĠSee ĠFigure ĠFIG REF 7 Ġand ĠTable ĠT AB REF 8 Ġfor Ġdetails . </s> We Ġalso Ġbuild Ġa Ġvariant Ġof ĠJasper , ĠJasper ĠD ense ĠRes id ual Ġ( DR ). ĠJasper ĠDR Ġfollows ĠD ense Net ĠB IB REF 15 Ġand ĠD ense R Net ĠB IB REF 16 Ġ, Ġbut Ġinstead Ġof Ġhaving Ġdense Ġconnections Ġwithin Ġa Ġblock , Ġthe Ġoutput Ġof Ġa Ġconv olution Ġblock Ġis Ġadded Ġto Ġthe Ġinputs Ġof Ġall Ġthe Ġfollowing Ġblocks . ĠWhile ĠD ense Net Ġand ĠD ense R Net Ġconc aten ates Ġthe Ġoutputs Ġof Ġdifferent Ġlayers , ĠJasper ĠDR Ġadds Ġthem Ġin Ġthe Ġsame Ġway Ġthat Ġresidual s Ġare Ġadded Ġin ĠRes Net . ĠAs Ġexplained Ġbelow , Ġwe Ġfind Ġaddition Ġto Ġbe Ġas Ġeffective Ġas Ġconc aten ation . </s> Normal ization Ġand ĠActiv ation </s> In Ġour Ġstudy , Ġwe Ġevaluate Ġperformance Ġof Ġmodels Ġwith : </s> 3 Ġtypes Ġof Ġnormal ization : Ġbatch Ġnorm ĠB IB REF 10 Ġ, Ġweight Ġnorm ĠB IB REF 9 Ġ, Ġand Ġlayer Ġnorm ĠB IB REF 17 </s> 3 Ġtypes Ġof Ġrect ified Ġlinear Ġunits : ĠRe LU , Ġclipped ĠRe LU Ġ( c Re LU ), Ġand Ġleak y ĠRe LU Ġ( l Re LU ) </s> 2 Ġtypes Ġof Ġg ated Ġunits : Ġg ated Ġlinear Ġunits Ġ( GL U ) ĠB IB REF 8 Ġ, Ġand Ġg ated Ġactivation Ġunits Ġ( GA U ) ĠB IB REF 18 </s> All Ġexperiment Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 15 Ġ. ĠWe Ġfirst Ġexperimented Ġwith Ġa Ġsmaller ĠJasper 5 x 3 Ġmodel Ġto Ġpick Ġthe Ġtop Ġ3 Ġsettings Ġbefore Ġtraining Ġon Ġlarger ĠJasper Ġmodels . ĠWe Ġfound Ġthat Ġlayer Ġnorm Ġwith ĠGA U Ġperformed Ġthe Ġbest Ġon Ġthe Ġsmaller Ġmodel . ĠLayer Ġnorm Ġwith ĠRe LU Ġand Ġbatch Ġnorm Ġwith ĠRe LU Ġcame Ġsecond Ġand Ġthird Ġin Ġour Ġtests . ĠUsing Ġthese Ġ3 , Ġwe Ġconducted Ġfurther Ġexperiments Ġon Ġa Ġlarger ĠJasper 10 x 4 . ĠFor Ġlarger Ġmodels , Ġwe Ġnoticed Ġthat Ġbatch Ġnorm Ġwith ĠRe LU Ġoutper formed Ġother Ġchoices . ĠThus , Ġleading Ġus Ġto Ġdecide Ġon Ġbatch Ġnormal ization Ġand ĠRe LU Ġfor Ġour Ġarchitecture . </s> During Ġbatch ing , Ġall Ġsequences Ġare Ġpadded Ġto Ġmatch Ġthe Ġlongest Ġsequence . ĠThese Ġpadded Ġvalues Ġcaused Ġissues Ġwhen Ġusing Ġlayer Ġnorm . ĠWe Ġapplied Ġa Ġsequence Ġmask Ġto Ġexclude Ġpadding Ġvalues Ġfrom Ġthe Ġmean Ġand Ġvariance Ġcalculation . ĠFurther , Ġwe Ġcomputed Ġmean Ġand Ġvariance Ġover Ġboth Ġthe Ġtime Ġdimension Ġand Ġchannels Ġsimilar Ġto Ġthe Ġsequence - wise Ġnormal ization Ġproposed Ġby ĠLaurent Ġet Ġal . ĠB IB REF 19 Ġ. ĠIn Ġaddition Ġto Ġmask ing Ġlayer Ġnorm , Ġwe Ġadditionally Ġapplied Ġmask ing Ġprior Ġto Ġthe Ġconv olution Ġoperation , Ġand Ġmask ing Ġthe Ġmean Ġand Ġvariance Ġcalculation Ġin Ġbatch Ġnorm . ĠThese Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 16 Ġ. ĠInterestingly , Ġwe Ġfound Ġthat Ġwhile Ġmask ing Ġbefore Ġconv olution Ġgives Ġa Ġlower ĠW ER , Ġusing Ġmasks Ġfor Ġboth Ġconv olutions Ġand Ġbatch Ġnorm Ġresults Ġin Ġworse Ġperformance . </s> As Ġa Ġfinal Ġnote , Ġwe Ġfound Ġthat Ġtraining Ġwith Ġweight Ġnorm Ġwas Ġvery Ġunstable Ġleading Ġto Ġexploding Ġactiv ations . </s> Res id ual ĠConnect ions </s> For Ġmodels Ġdeeper Ġthan ĠJasper Ġ5 x 3 , Ġwe Ġobserve Ġconsistently Ġthat Ġresidual Ġconnections Ġare Ġnecessary Ġfor Ġtraining Ġto Ġconverge . ĠIn Ġaddition Ġto Ġthe Ġsimple Ġresidual Ġand Ġdense Ġresidual Ġmodel Ġdescribed Ġabove , Ġwe Ġinvestigated ĠD ense Net ĠB IB REF 15 Ġand ĠD ense R Net ĠB IB REF 16 Ġvariants Ġof ĠJasper . ĠBoth Ġconnect Ġthe Ġoutputs Ġof Ġeach Ġsub - block Ġto Ġthe Ġinputs Ġof Ġfollowing Ġsub - blocks Ġwithin Ġa Ġblock . ĠD ense R Net , Ġsimilar Ġto ĠD ense ĠRes id ual , Ġconnects Ġthe Ġoutput Ġof Ġeach Ġoutput Ġof Ġeach Ġblock Ġto Ġthe Ġinput Ġof Ġall Ġfollowing Ġblocks . ĠD ense Net Ġand ĠD ense R Net Ġcombine Ġresidual Ġconnections Ġusing Ġconc aten ation Ġwhereas ĠRes id ual Ġand ĠD ense ĠRes id ual Ġuse Ġaddition . ĠWe Ġfound Ġthat ĠD ense ĠRes id ual Ġand ĠD ense R Net Ġperform Ġsimilarly Ġwith Ġeach Ġperforming Ġbetter Ġon Ġspecific Ġsubs ets Ġof ĠLib ri Spe ech . ĠWe Ġdecided Ġto Ġuse ĠD ense ĠRes id ual Ġfor Ġsubsequent Ġexperiments . ĠThe Ġmain Ġreason Ġis Ġthat Ġdue Ġto Ġconc aten ation , Ġthe Ġgrowth Ġfactor Ġfor ĠD ense Net Ġand ĠD ense R Net Ġrequires Ġtuning Ġfor Ġdeeper Ġmodels Ġwhereas ĠD ense ĠRes id ual Ġsimply Ġjust Ġrepeats Ġa Ġsub - blocks . </s> Language ĠModel </s> A Ġlanguage Ġmodel Ġ( LM ) Ġis Ġa Ġprobability Ġdistribution Ġover Ġarbitrary Ġsymbol Ġsequences ĠIN LINE FORM 0 Ġsuch Ġthat Ġmore Ġlikely Ġsequences Ġare Ġassigned Ġhigh Ġprobabilities . ĠL Ms Ġare Ġfrequently Ġused Ġto Ġcondition Ġbeam Ġsearch . ĠDuring Ġdecoding , Ġcandidates Ġare Ġevaluated Ġusing Ġboth Ġacoustic Ġscores Ġand ĠLM Ġscores . ĠTraditional ĠN - gram ĠL Ms Ġhave Ġbeen Ġaugmented Ġwith Ġneural ĠL Ms Ġin Ġrecent Ġwork ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ. </s> We Ġexperiment Ġwith Ġstatistical ĠN - gram Ġlanguage Ġmodels ĠB IB REF 23 Ġand Ġneural ĠTrans former - XL ĠB IB REF 11 Ġmodels . ĠOur Ġbest Ġresults Ġuse Ġacoustic Ġand Ġword - level ĠN - gram Ġlanguage Ġmodels Ġto Ġgenerate Ġa Ġcandidate Ġlist Ġusing Ġbeam Ġsearch Ġwith Ġa Ġwidth Ġof Ġ2048 . ĠNext , Ġan Ġexternal ĠTrans former - XL ĠLM Ġresc ores Ġthe Ġfinal Ġlist . ĠAll ĠL Ms Ġwere Ġtrained Ġon Ġdatasets Ġindependently Ġfrom Ġacoustic Ġmodels . ĠWe Ġshow Ġresults Ġwith Ġthe Ġneural ĠLM Ġin Ġour ĠResults Ġsection . ĠWe Ġobserved Ġa Ġstrong Ġcorrelation Ġbetween Ġthe Ġquality Ġof Ġthe Ġneural ĠLM Ġ( me asured Ġby Ġperplex ity ) Ġand ĠW ER Ġas Ġshown Ġin ĠFigure ĠFIG REF 20 Ġ. </s> N ovo Grad </s> For Ġtraining , Ġwe Ġuse Ġeither ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ) Ġwith Ġmomentum Ġor Ġour Ġown ĠNov o Grad , Ġan Ġoptim izer Ġsimilar Ġto ĠAdam ĠB IB REF 14 Ġ, Ġexcept Ġthat Ġits Ġsecond Ġmoments Ġare Ġcomputed Ġper Ġlayer Ġinstead Ġof Ġper Ġweight . ĠCompared Ġto ĠAdam , Ġit Ġreduces Ġmemory Ġconsumption Ġand Ġwe Ġfind Ġit Ġto Ġbe Ġmore Ġnumer ically Ġstable . </s> At Ġeach Ġstep ĠIN LINE FORM 0 Ġ, ĠNov o Grad Ġcomp utes Ġthe Ġsto ch astic Ġgradient ĠIN LINE FORM 1 Ġfollowing Ġthe Ġregular Ġforward - back ward Ġpass . ĠThen Ġthe Ġsecond - order Ġmoment ĠIN LINE FORM 2 Ġis Ġcomputed Ġfor Ġeach Ġlayer ĠIN LINE FORM 3 Ġsimilar Ġto ĠND - Adam ĠB IB REF 27 Ġ: ĠDIS PLAY FORM 0 </s> The Ġsecond - order Ġmoment ĠIN LINE FORM 0 Ġis Ġused Ġto Ġre - scale Ġgrad ients ĠIN LINE FORM 1 Ġbefore Ġcalculating Ġthe Ġfirst - order Ġmoment ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> If ĠL 2 - regular ization Ġis Ġused , Ġa Ġweight Ġdecay ĠIN LINE FORM 0 Ġis Ġadded Ġto Ġthe Ġre - sc aled Ġgradient Ġ( as Ġin ĠAdam W ĠB IB REF 28 Ġ): ĠDIS PLAY FORM 0 </s> Finally , Ġnew Ġweights Ġare Ġcomputed Ġusing Ġthe Ġlearning Ġrate ĠIN LINE FORM 0 Ġ: ĠDIS PLAY FORM 0 </s> Using ĠNov o Grad Ġinstead Ġof ĠSG D Ġwith Ġmomentum , Ġwe Ġdecreased Ġthe ĠW ER Ġon Ġdev - clean ĠLib ri Spe ech Ġfrom Ġ4 . 00 % Ġto Ġ3 . 64 %, Ġa Ġrelative Ġimprovement Ġof Ġ9 % Ġfor ĠJasper ĠDR Ġ10 x 5 . ĠWe Ġwill Ġfurther Ġanalyze ĠNov o Grad Ġin Ġforthcoming Ġwork . </s> Results </s> We Ġevaluate ĠJasper Ġacross Ġa Ġnumber Ġof Ġdatasets Ġin Ġvarious Ġdomains . ĠIn Ġall Ġexperiments , Ġwe Ġuse Ġdrop out Ġand Ġweight Ġdecay Ġas Ġregular ization . ĠAt Ġtraining Ġtime , Ġwe Ġuse Ġspeed Ġpert urb ation Ġwith Ġfixed Ġ+/- 10 % ĠB IB REF 29 Ġfor ĠLib ri Spe ech . ĠFor ĠWS J Ġand ĠHub 5 ' 00 , Ġwe Ġuse Ġa Ġrandom Ġspeed Ġpert urb ation Ġfactor Ġbetween Ġ[- 10 %, Ġ10 %] Ġas Ġeach Ġutter ance Ġis Ġfed Ġinto Ġthe Ġmodel . ĠAll Ġmodels Ġhave Ġbeen Ġtrained Ġon ĠNVIDIA ĠDG X - 1 Ġin Ġmixed Ġprecision ĠB IB REF 30 Ġusing ĠOpen Se q 2 Se q ĠB IB REF 31 Ġ. ĠSource Ġcode , Ġtraining Ġconfigurations , Ġand Ġpret rained Ġmodels Ġare Ġavailable . </s> Read ĠSpeech </s> We Ġevaluated Ġthe Ġperformance Ġof ĠJasper Ġon Ġtwo Ġread Ġspeech Ġdatasets : ĠLib ri Spe ech Ġand ĠWall ĠStreet ĠJournal Ġ( WS J ). ĠFor ĠLib ri Spe ech , Ġwe Ġtrained ĠJasper ĠDR Ġ10 x 5 Ġusing Ġour ĠNov o Grad Ġoptim izer Ġfor Ġ400 Ġepoch s . ĠWe Ġachieve ĠS OTA Ġperformance Ġon Ġthe Ġtest - clean Ġsubset Ġand ĠS OTA Ġamong Ġend - to - end Ġspeech Ġrecognition Ġmodels Ġon Ġtest - other . </s> We Ġtrained Ġa Ġsmaller ĠJasper Ġ10 x 3 Ġmodel Ġwith ĠSG D Ġwith Ġmomentum Ġoptim izer Ġfor Ġ400 Ġepoch s Ġon Ġa Ġcombined ĠWS J Ġdataset Ġ( 80 Ġhours ): ĠL DC 93 S 6 A Ġ( WS J 0 ) Ġand ĠL DC 94 S 13 A Ġ( WS J 1 ). ĠThe Ġresults Ġare Ġprovided Ġin ĠTable ĠT AB REF 29 Ġ. </s> Con vers ational ĠSpeech </s> We Ġalso Ġevaluate Ġthe ĠJasper Ġmodel 's Ġperformance Ġon Ġa Ġconvers ational ĠEnglish Ġcorpus . ĠThe ĠHub 5 ĠYear Ġ2000 Ġ( Hub 5 ' 00 ) Ġevaluation Ġ( L DC 2002 S 09 , ĠL DC 2005 S 13 ) Ġis Ġwidely Ġused Ġin Ġacademia . ĠIt Ġis Ġdivided Ġinto Ġtwo Ġsubs ets : ĠSwitch board Ġ( SW B ) Ġand ĠCall home Ġ( CH M ). ĠThe Ġtraining Ġdata Ġfor Ġboth Ġthe Ġacoustic Ġand Ġlanguage Ġmodels Ġconsisted Ġof Ġthe Ġ2000 hr ĠFisher + Switch board Ġtraining Ġdata Ġ( L DC 2004 S 13 , ĠL DC 2005 S 13 , ĠL DC 97 S 62 ). ĠJasper ĠDR Ġ10 x 5 Ġwas Ġtrained Ġusing ĠSG D Ġwith Ġmomentum Ġfor Ġ50 Ġepoch s . ĠWe Ġcompare Ġto Ġother Ġmodels Ġtrained Ġusing Ġthe Ġsame Ġdata Ġand Ġreport ĠHub 5 ' 00 Ġresults Ġin ĠTable ĠT AB REF 31 Ġ. </s> We Ġobtain Ġgood Ġresults Ġfor ĠSW B . ĠHowever , Ġthere Ġis Ġwork Ġto Ġbe Ġdone Ġto Ġimprove ĠW ER Ġon Ġharder Ġtasks Ġsuch Ġas ĠCH M . </s> Con clusions </s> We Ġhave Ġpresented Ġa Ġnew Ġfamily Ġof Ġneural Ġarchitectures Ġfor Ġend - to - end Ġspeech Ġrecognition . ĠInspired Ġby Ġw av 2 letter 's Ġconv olution al Ġapproach , Ġwe Ġbuild Ġa Ġdeep Ġand Ġscalable Ġmodel , Ġwhich Ġrequires Ġa Ġwell - designed Ġresidual Ġtop ology , Ġeffective Ġregular ization , Ġand Ġa Ġstrong Ġoptim izer . ĠAs Ġour Ġarchitecture Ġstudies Ġdemonstrated , Ġa Ġcombination Ġof Ġstandard Ġcomponents Ġleads Ġto ĠS OTA Ġresults Ġon ĠLib ri Spe ech Ġand Ġcompetitive Ġresults Ġon Ġother Ġbenchmarks . ĠOur ĠJasper Ġarchitecture Ġis Ġhighly Ġefficient Ġfor Ġtraining Ġand Ġinference , Ġand Ġserves Ġas Ġa Ġgood Ġbaseline Ġapproach Ġon Ġtop Ġof Ġwhich Ġto Ġexplore Ġmore Ġsophisticated Ġregular ization , Ġdata Ġaug mentation , Ġloss Ġfunctions , Ġlanguage Ġmodels , Ġand Ġoptimization Ġstrategies . ĠWe Ġare Ġinterested Ġto Ġsee Ġif Ġour Ġapproach Ġcan Ġcontinue Ġto Ġscale Ġto Ġdeeper Ġmodels Ġand Ġlarger Ġdatasets .
Introduction Conventional automatic speech recognition (ASR) systems typically consist of several independently learned components: an acoustic model to predict context-dependent sub-phoneme states (senones) from audio, a graph structure to map senones to phonemes, and a pronunciation model to map phonemes to words. Hybrid systems combine hidden Markov models to model state dependencies with neural networks to predict states BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Newer approaches such as end-to-end (E2E) systems reduce the overall complexity of the final system. Our research builds on prior work that has explored using time-delay neural networks (TDNN), other forms of convolutional neural networks, and Connectionist Temporal Classification (CTC) loss BIBREF4 , BIBREF5 , BIBREF6 . We took inspiration from wav2letter BIBREF6 , which uses 1D-convolution layers. Liptchinsky et al. BIBREF7 improved wav2letter by increasing the model depth to 19 convolutional layers and adding Gated Linear Units (GLU) BIBREF8 , weight normalization BIBREF9 and dropout. By building a deeper and larger capacity network, we aim to demonstrate that we can match or outperform non end-to-end models on the LibriSpeech and 2000hr Fisher+Switchboard tasks. Like wav2letter, our architecture, Jasper, uses a stack of 1D-convolution layers, but with ReLU and batch normalization BIBREF10 . We find that ReLU and batch normalization outperform other activation and normalization schemes that we tested for convolutional ASR. As a result, Jasper's architecture contains only 1D convolution, batch normalization, ReLU, and dropout layers – operators highly optimized for training and inference on GPUs. It is possible to increase the capacity of the Jasper model by stacking these operations. Our largest version uses 54 convolutional layers (333M parameters), while our small model uses 34 (201M parameters). We use residual connections to enable this level of depth. We investigate a number of residual options and propose a new residual connection topology we call Dense Residual (DR). Integrating our best acoustic model with a Transformer-XL BIBREF11 language model allows us to obtain new state-of-the-art (SOTA) results on LibriSpeech BIBREF12 test-clean of 2.95% WER and SOTA results among end-to-end models on LibriSpeech test-other. We show competitive results on Wall Street Journal (WSJ), and 2000hr Fisher+Switchboard (F+S). Using only greedy decoding without a language model we achieve 3.86% WER on LibriSpeech test-clean. This paper makes the following contributions: Jasper Architecture Jasper is a family of end-to-end ASR models that replace acoustic and pronunciation models with a convolutional neural network. Jasper uses mel-filterbank features calculated from 20ms windows with a 10ms overlap, and outputs a probability distribution over characters per frame. Jasper has a block architecture: a Jasper INLINEFORM0 x INLINEFORM1 model has INLINEFORM2 blocks, each with INLINEFORM3 sub-blocks. Each sub-block applies the following operations: a 1D-convolution, batch norm, ReLU, and dropout. All sub-blocks in a block have the same number of output channels. Each block input is connected directly into the last sub-block via a residual connection. The residual connection is first projected through a 1x1 convolution to account for different numbers of input and output channels, then through a batch norm layer. The output of this batch norm layer is added to the output of the batch norm layer in the last sub-block. The result of this sum is passed through the activation function and dropout to produce the output of the sub-block. The sub-block architecture of Jasper was designed to facilitate fast GPU inference. Each sub-block can be fused into a single GPU kernel: dropout is not used at inference-time and is eliminated, batch norm can be fused with the preceding convolution, ReLU clamps the result, and residual summation can be treated as a modified bias term in this fused operation. All Jasper models have four additional convolutional blocks: one pre-processing and three post-processing. See Figure FIGREF7 and Table TABREF8 for details. We also build a variant of Jasper, Jasper Dense Residual (DR). Jasper DR follows DenseNet BIBREF15 and DenseRNet BIBREF16 , but instead of having dense connections within a block, the output of a convolution block is added to the inputs of all the following blocks. While DenseNet and DenseRNet concatenates the outputs of different layers, Jasper DR adds them in the same way that residuals are added in ResNet. As explained below, we find addition to be as effective as concatenation. Normalization and Activation In our study, we evaluate performance of models with: 3 types of normalization: batch norm BIBREF10 , weight norm BIBREF9 , and layer norm BIBREF17 3 types of rectified linear units: ReLU, clipped ReLU (cReLU), and leaky ReLU (lReLU) 2 types of gated units: gated linear units (GLU) BIBREF8 , and gated activation units (GAU) BIBREF18 All experiment results are shown in Table TABREF15 . We first experimented with a smaller Jasper5x3 model to pick the top 3 settings before training on larger Jasper models. We found that layer norm with GAU performed the best on the smaller model. Layer norm with ReLU and batch norm with ReLU came second and third in our tests. Using these 3, we conducted further experiments on a larger Jasper10x4. For larger models, we noticed that batch norm with ReLU outperformed other choices. Thus, leading us to decide on batch normalization and ReLU for our architecture. During batching, all sequences are padded to match the longest sequence. These padded values caused issues when using layer norm. We applied a sequence mask to exclude padding values from the mean and variance calculation. Further, we computed mean and variance over both the time dimension and channels similar to the sequence-wise normalization proposed by Laurent et al. BIBREF19 . In addition to masking layer norm, we additionally applied masking prior to the convolution operation, and masking the mean and variance calculation in batch norm. These results are shown in Table TABREF16 . Interestingly, we found that while masking before convolution gives a lower WER, using masks for both convolutions and batch norm results in worse performance. As a final note, we found that training with weight norm was very unstable leading to exploding activations. Residual Connections For models deeper than Jasper 5x3, we observe consistently that residual connections are necessary for training to converge. In addition to the simple residual and dense residual model described above, we investigated DenseNet BIBREF15 and DenseRNet BIBREF16 variants of Jasper. Both connect the outputs of each sub-block to the inputs of following sub-blocks within a block. DenseRNet, similar to Dense Residual, connects the output of each output of each block to the input of all following blocks. DenseNet and DenseRNet combine residual connections using concatenation whereas Residual and Dense Residual use addition. We found that Dense Residual and DenseRNet perform similarly with each performing better on specific subsets of LibriSpeech. We decided to use Dense Residual for subsequent experiments. The main reason is that due to concatenation, the growth factor for DenseNet and DenseRNet requires tuning for deeper models whereas Dense Residual simply just repeats a sub-blocks. Language Model A language model (LM) is a probability distribution over arbitrary symbol sequences INLINEFORM0 such that more likely sequences are assigned high probabilities. LMs are frequently used to condition beam search. During decoding, candidates are evaluated using both acoustic scores and LM scores. Traditional N-gram LMs have been augmented with neural LMs in recent work BIBREF20 , BIBREF21 , BIBREF22 . We experiment with statistical N-gram language models BIBREF23 and neural Transformer-XL BIBREF11 models. Our best results use acoustic and word-level N-gram language models to generate a candidate list using beam search with a width of 2048. Next, an external Transformer-XL LM rescores the final list. All LMs were trained on datasets independently from acoustic models. We show results with the neural LM in our Results section. We observed a strong correlation between the quality of the neural LM (measured by perplexity) and WER as shown in Figure FIGREF20 . NovoGrad For training, we use either Stochastic Gradient Descent (SGD) with momentum or our own NovoGrad, an optimizer similar to Adam BIBREF14 , except that its second moments are computed per layer instead of per weight. Compared to Adam, it reduces memory consumption and we find it to be more numerically stable. At each step INLINEFORM0 , NovoGrad computes the stochastic gradient INLINEFORM1 following the regular forward-backward pass. Then the second-order moment INLINEFORM2 is computed for each layer INLINEFORM3 similar to ND-Adam BIBREF27 : DISPLAYFORM0 The second-order moment INLINEFORM0 is used to re-scale gradients INLINEFORM1 before calculating the first-order moment INLINEFORM2 : DISPLAYFORM0 If L2-regularization is used, a weight decay INLINEFORM0 is added to the re-scaled gradient (as in AdamW BIBREF28 ): DISPLAYFORM0 Finally, new weights are computed using the learning rate INLINEFORM0 : DISPLAYFORM0 Using NovoGrad instead of SGD with momentum, we decreased the WER on dev-clean LibriSpeech from 4.00% to 3.64%, a relative improvement of 9% for Jasper DR 10x5. We will further analyze NovoGrad in forthcoming work. Results We evaluate Jasper across a number of datasets in various domains. In all experiments, we use dropout and weight decay as regularization. At training time, we use speed perturbation with fixed +/-10% BIBREF29 for LibriSpeech. For WSJ and Hub5'00, we use a random speed perturbation factor between [-10%, 10%] as each utterance is fed into the model. All models have been trained on NVIDIA DGX-1 in mixed precision BIBREF30 using OpenSeq2Seq BIBREF31 . Source code, training configurations, and pretrained models are available. Read Speech We evaluated the performance of Jasper on two read speech datasets: LibriSpeech and Wall Street Journal (WSJ). For LibriSpeech, we trained Jasper DR 10x5 using our NovoGrad optimizer for 400 epochs. We achieve SOTA performance on the test-clean subset and SOTA among end-to-end speech recognition models on test-other. We trained a smaller Jasper 10x3 model with SGD with momentum optimizer for 400 epochs on a combined WSJ dataset (80 hours): LDC93S6A (WSJ0) and LDC94S13A (WSJ1). The results are provided in Table TABREF29 . Conversational Speech We also evaluate the Jasper model's performance on a conversational English corpus. The Hub5 Year 2000 (Hub5'00) evaluation (LDC2002S09, LDC2005S13) is widely used in academia. It is divided into two subsets: Switchboard (SWB) and Callhome (CHM). The training data for both the acoustic and language models consisted of the 2000hr Fisher+Switchboard training data (LDC2004S13, LDC2005S13, LDC97S62). Jasper DR 10x5 was trained using SGD with momentum for 50 epochs. We compare to other models trained using the same data and report Hub5'00 results in Table TABREF31 . We obtain good results for SWB. However, there is work to be done to improve WER on harder tasks such as CHM. Conclusions We have presented a new family of neural architectures for end-to-end speech recognition. Inspired by wav2letter's convolutional approach, we build a deep and scalable model, which requires a well-designed residual topology, effective regularization, and a strong optimizer. As our architecture studies demonstrated, a combination of standard components leads to SOTA results on LibriSpeech and competitive results on other benchmarks. Our Jasper architecture is highly efficient for training and inference, and serves as a good baseline approach on top of which to explore more sophisticated regularization, data augmentation, loss functions, language models, and optimization strategies. We are interested to see if our approach can continue to scale to deeper models and larger datasets.
what were the baselines?
2ddb51b03163d309434ee403fef42d6b9aecc458
what Ġwere Ġthe Ġbas elines ?
Un answer able
<s> what Ġwere Ġthe Ġbas elines ? </s> Introduction </s> Con ventional Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems Ġtypically Ġconsist Ġof Ġseveral Ġindependently Ġlearned Ġcomponents : Ġan Ġacoustic Ġmodel Ġto Ġpredict Ġcontext - dependent Ġsub - ph on eme Ġstates Ġ( sen ones ) Ġfrom Ġaudio , Ġa Ġgraph Ġstructure Ġto Ġmap Ġsen ones Ġto Ġphon em es , Ġand Ġa Ġpronunciation Ġmodel Ġto Ġmap Ġphon em es Ġto Ġwords . ĠHybrid Ġsystems Ġcombine Ġhidden ĠMark ov Ġmodels Ġto Ġmodel Ġstate Ġdependencies Ġwith Ġneural Ġnetworks Ġto Ġpredict Ġstates ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠNew er Ġapproaches Ġsuch Ġas Ġend - to - end Ġ( E 2 E ) Ġsystems Ġreduce Ġthe Ġoverall Ġcomplexity Ġof Ġthe Ġfinal Ġsystem . </s> Our Ġresearch Ġbuilds Ġon Ġprior Ġwork Ġthat Ġhas Ġexplored Ġusing Ġtime - delay Ġneural Ġnetworks Ġ( TD NN ), Ġother Ġforms Ġof Ġconv olution al Ġneural Ġnetworks , Ġand ĠConnection ist ĠTem poral ĠClassification Ġ( CT C ) Ġloss ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠWe Ġtook Ġinspiration Ġfrom Ġw av 2 letter ĠB IB REF 6 Ġ, Ġwhich Ġuses Ġ1 D - conv olution Ġlayers . ĠLi pt ch insky Ġet Ġal . ĠB IB REF 7 Ġimproved Ġw av 2 letter Ġby Ġincreasing Ġthe Ġmodel Ġdepth Ġto Ġ19 Ġconv olution al Ġlayers Ġand Ġadding ĠG ated ĠLinear ĠUnits Ġ( GL U ) ĠB IB REF 8 Ġ, Ġweight Ġnormal ization ĠB IB REF 9 Ġand Ġdrop out . </s> By Ġbuilding Ġa Ġdeeper Ġand Ġlarger Ġcapacity Ġnetwork , Ġwe Ġaim Ġto Ġdemonstrate Ġthat Ġwe Ġcan Ġmatch Ġor Ġoutper form Ġnon Ġend - to - end Ġmodels Ġon Ġthe ĠLib ri Spe ech Ġand Ġ2000 hr ĠFisher + Switch board Ġtasks . ĠLike Ġw av 2 letter , Ġour Ġarchitecture , ĠJasper , Ġuses Ġa Ġstack Ġof Ġ1 D - conv olution Ġlayers , Ġbut Ġwith ĠRe LU Ġand Ġbatch Ġnormal ization ĠB IB REF 10 Ġ. ĠWe Ġfind Ġthat ĠRe LU Ġand Ġbatch Ġnormal ization Ġoutper form Ġother Ġactivation Ġand Ġnormal ization Ġschemes Ġthat Ġwe Ġtested Ġfor Ġconv olution al ĠAS R . ĠAs Ġa Ġresult , ĠJasper 's Ġarchitecture Ġcontains Ġonly Ġ1 D Ġconv olution , Ġbatch Ġnormal ization , ĠRe LU , Ġand Ġdrop out Ġlayers ĠâĢĵ Ġoperators Ġhighly Ġoptimized Ġfor Ġtraining Ġand Ġinference Ġon ĠGPUs . </s> It Ġis Ġpossible Ġto Ġincrease Ġthe Ġcapacity Ġof Ġthe ĠJasper Ġmodel Ġby Ġstacking Ġthese Ġoperations . ĠOur Ġlargest Ġversion Ġuses Ġ54 Ġconv olution al Ġlayers Ġ( 333 M Ġparameters ), Ġwhile Ġour Ġsmall Ġmodel Ġuses Ġ34 Ġ( 201 M Ġparameters ). ĠWe Ġuse Ġresidual Ġconnections Ġto Ġenable Ġthis Ġlevel Ġof Ġdepth . ĠWe Ġinvestigate Ġa Ġnumber Ġof Ġresidual Ġoptions Ġand Ġpropose Ġa Ġnew Ġresidual Ġconnection Ġtop ology Ġwe Ġcall ĠD ense ĠRes id ual Ġ( DR ). </s> Integ rating Ġour Ġbest Ġacoustic Ġmodel Ġwith Ġa ĠTrans former - XL ĠB IB REF 11 Ġlanguage Ġmodel Ġallows Ġus Ġto Ġobtain Ġnew Ġstate - of - the - art Ġ( S OTA ) Ġresults Ġon ĠLib ri Spe ech ĠB IB REF 12 Ġtest - clean Ġof Ġ2 . 95 % ĠW ER Ġand ĠS OTA Ġresults Ġamong Ġend - to - end Ġmodels Ġon ĠLib ri Spe ech Ġtest - other . ĠWe Ġshow Ġcompetitive Ġresults Ġon ĠWall ĠStreet ĠJournal Ġ( WS J ), Ġand Ġ2000 hr ĠFisher + Switch board Ġ( F + S ). ĠUsing Ġonly Ġgreedy Ġdecoding Ġwithout Ġa Ġlanguage Ġmodel Ġwe Ġachieve Ġ3 . 86 % ĠW ER Ġon ĠLib ri Spe ech Ġtest - clean . </s> This Ġpaper Ġmakes Ġthe Ġfollowing Ġcontributions : </s> J asper ĠArchitecture </s> J asper Ġis Ġa Ġfamily Ġof Ġend - to - end ĠAS R Ġmodels Ġthat Ġreplace Ġacoustic Ġand Ġpronunciation Ġmodels Ġwith Ġa Ġconv olution al Ġneural Ġnetwork . ĠJasper Ġuses Ġmel - filter bank Ġfeatures Ġcalculated Ġfrom Ġ20 ms Ġwindows Ġwith Ġa Ġ10 ms Ġoverlap , Ġand Ġoutputs Ġa Ġprobability Ġdistribution Ġover Ġcharacters Ġper Ġframe . ĠJasper Ġhas Ġa Ġblock Ġarchitecture : Ġa ĠJasper ĠIN LINE FORM 0 Ġx ĠIN LINE FORM 1 Ġmodel Ġhas ĠIN LINE FORM 2 Ġblocks , Ġeach Ġwith ĠIN LINE FORM 3 Ġsub - blocks . ĠEach Ġsub - block Ġapplies Ġthe Ġfollowing Ġoperations : Ġa Ġ1 D - conv olution , Ġbatch Ġnorm , ĠRe LU , Ġand Ġdrop out . ĠAll Ġsub - blocks Ġin Ġa Ġblock Ġhave Ġthe Ġsame Ġnumber Ġof Ġoutput Ġchannels . </s> Each Ġblock Ġinput Ġis Ġconnected Ġdirectly Ġinto Ġthe Ġlast Ġsub - block Ġvia Ġa Ġresidual Ġconnection . ĠThe Ġresidual Ġconnection Ġis Ġfirst Ġprojected Ġthrough Ġa Ġ1 x 1 Ġconv olution Ġto Ġaccount Ġfor Ġdifferent Ġnumbers Ġof Ġinput Ġand Ġoutput Ġchannels , Ġthen Ġthrough Ġa Ġbatch Ġnorm Ġlayer . ĠThe Ġoutput Ġof Ġthis Ġbatch Ġnorm Ġlayer Ġis Ġadded Ġto Ġthe Ġoutput Ġof Ġthe Ġbatch Ġnorm Ġlayer Ġin Ġthe Ġlast Ġsub - block . ĠThe Ġresult Ġof Ġthis Ġsum Ġis Ġpassed Ġthrough Ġthe Ġactivation Ġfunction Ġand Ġdrop out Ġto Ġproduce Ġthe Ġoutput Ġof Ġthe Ġsub - block . </s> The Ġsub - block Ġarchitecture Ġof ĠJasper Ġwas Ġdesigned Ġto Ġfacilitate Ġfast ĠGPU Ġinference . ĠEach Ġsub - block Ġcan Ġbe Ġfused Ġinto Ġa Ġsingle ĠGPU Ġkernel : Ġdrop out Ġis Ġnot Ġused Ġat Ġinference - time Ġand Ġis Ġeliminated , Ġbatch Ġnorm Ġcan Ġbe Ġfused Ġwith Ġthe Ġpreceding Ġconv olution , ĠRe LU Ġcl amps Ġthe Ġresult , Ġand Ġresidual Ġsumm ation Ġcan Ġbe Ġtreated Ġas Ġa Ġmodified Ġbias Ġterm Ġin Ġthis Ġfused Ġoperation . </s> All ĠJasper Ġmodels Ġhave Ġfour Ġadditional Ġconv olution al Ġblocks : Ġone Ġpre - processing Ġand Ġthree Ġpost - processing . ĠSee ĠFigure ĠFIG REF 7 Ġand ĠTable ĠT AB REF 8 Ġfor Ġdetails . </s> We Ġalso Ġbuild Ġa Ġvariant Ġof ĠJasper , ĠJasper ĠD ense ĠRes id ual Ġ( DR ). ĠJasper ĠDR Ġfollows ĠD ense Net ĠB IB REF 15 Ġand ĠD ense R Net ĠB IB REF 16 Ġ, Ġbut Ġinstead Ġof Ġhaving Ġdense Ġconnections Ġwithin Ġa Ġblock , Ġthe Ġoutput Ġof Ġa Ġconv olution Ġblock Ġis Ġadded Ġto Ġthe Ġinputs Ġof Ġall Ġthe Ġfollowing Ġblocks . ĠWhile ĠD ense Net Ġand ĠD ense R Net Ġconc aten ates Ġthe Ġoutputs Ġof Ġdifferent Ġlayers , ĠJasper ĠDR Ġadds Ġthem Ġin Ġthe Ġsame Ġway Ġthat Ġresidual s Ġare Ġadded Ġin ĠRes Net . ĠAs Ġexplained Ġbelow , Ġwe Ġfind Ġaddition Ġto Ġbe Ġas Ġeffective Ġas Ġconc aten ation . </s> Normal ization Ġand ĠActiv ation </s> In Ġour Ġstudy , Ġwe Ġevaluate Ġperformance Ġof Ġmodels Ġwith : </s> 3 Ġtypes Ġof Ġnormal ization : Ġbatch Ġnorm ĠB IB REF 10 Ġ, Ġweight Ġnorm ĠB IB REF 9 Ġ, Ġand Ġlayer Ġnorm ĠB IB REF 17 </s> 3 Ġtypes Ġof Ġrect ified Ġlinear Ġunits : ĠRe LU , Ġclipped ĠRe LU Ġ( c Re LU ), Ġand Ġleak y ĠRe LU Ġ( l Re LU ) </s> 2 Ġtypes Ġof Ġg ated Ġunits : Ġg ated Ġlinear Ġunits Ġ( GL U ) ĠB IB REF 8 Ġ, Ġand Ġg ated Ġactivation Ġunits Ġ( GA U ) ĠB IB REF 18 </s> All Ġexperiment Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 15 Ġ. ĠWe Ġfirst Ġexperimented Ġwith Ġa Ġsmaller ĠJasper 5 x 3 Ġmodel Ġto Ġpick Ġthe Ġtop Ġ3 Ġsettings Ġbefore Ġtraining Ġon Ġlarger ĠJasper Ġmodels . ĠWe Ġfound Ġthat Ġlayer Ġnorm Ġwith ĠGA U Ġperformed Ġthe Ġbest Ġon Ġthe Ġsmaller Ġmodel . ĠLayer Ġnorm Ġwith ĠRe LU Ġand Ġbatch Ġnorm Ġwith ĠRe LU Ġcame Ġsecond Ġand Ġthird Ġin Ġour Ġtests . ĠUsing Ġthese Ġ3 , Ġwe Ġconducted Ġfurther Ġexperiments Ġon Ġa Ġlarger ĠJasper 10 x 4 . ĠFor Ġlarger Ġmodels , Ġwe Ġnoticed Ġthat Ġbatch Ġnorm Ġwith ĠRe LU Ġoutper formed Ġother Ġchoices . ĠThus , Ġleading Ġus Ġto Ġdecide Ġon Ġbatch Ġnormal ization Ġand ĠRe LU Ġfor Ġour Ġarchitecture . </s> During Ġbatch ing , Ġall Ġsequences Ġare Ġpadded Ġto Ġmatch Ġthe Ġlongest Ġsequence . ĠThese Ġpadded Ġvalues Ġcaused Ġissues Ġwhen Ġusing Ġlayer Ġnorm . ĠWe Ġapplied Ġa Ġsequence Ġmask Ġto Ġexclude Ġpadding Ġvalues Ġfrom Ġthe Ġmean Ġand Ġvariance Ġcalculation . ĠFurther , Ġwe Ġcomputed Ġmean Ġand Ġvariance Ġover Ġboth Ġthe Ġtime Ġdimension Ġand Ġchannels Ġsimilar Ġto Ġthe Ġsequence - wise Ġnormal ization Ġproposed Ġby ĠLaurent Ġet Ġal . ĠB IB REF 19 Ġ. ĠIn Ġaddition Ġto Ġmask ing Ġlayer Ġnorm , Ġwe Ġadditionally Ġapplied Ġmask ing Ġprior Ġto Ġthe Ġconv olution Ġoperation , Ġand Ġmask ing Ġthe Ġmean Ġand Ġvariance Ġcalculation Ġin Ġbatch Ġnorm . ĠThese Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 16 Ġ. ĠInterestingly , Ġwe Ġfound Ġthat Ġwhile Ġmask ing Ġbefore Ġconv olution Ġgives Ġa Ġlower ĠW ER , Ġusing Ġmasks Ġfor Ġboth Ġconv olutions Ġand Ġbatch Ġnorm Ġresults Ġin Ġworse Ġperformance . </s> As Ġa Ġfinal Ġnote , Ġwe Ġfound Ġthat Ġtraining Ġwith Ġweight Ġnorm Ġwas Ġvery Ġunstable Ġleading Ġto Ġexploding Ġactiv ations . </s> Res id ual ĠConnect ions </s> For Ġmodels Ġdeeper Ġthan ĠJasper Ġ5 x 3 , Ġwe Ġobserve Ġconsistently Ġthat Ġresidual Ġconnections Ġare Ġnecessary Ġfor Ġtraining Ġto Ġconverge . ĠIn Ġaddition Ġto Ġthe Ġsimple Ġresidual Ġand Ġdense Ġresidual Ġmodel Ġdescribed Ġabove , Ġwe Ġinvestigated ĠD ense Net ĠB IB REF 15 Ġand ĠD ense R Net ĠB IB REF 16 Ġvariants Ġof ĠJasper . ĠBoth Ġconnect Ġthe Ġoutputs Ġof Ġeach Ġsub - block Ġto Ġthe Ġinputs Ġof Ġfollowing Ġsub - blocks Ġwithin Ġa Ġblock . ĠD ense R Net , Ġsimilar Ġto ĠD ense ĠRes id ual , Ġconnects Ġthe Ġoutput Ġof Ġeach Ġoutput Ġof Ġeach Ġblock Ġto Ġthe Ġinput Ġof Ġall Ġfollowing Ġblocks . ĠD ense Net Ġand ĠD ense R Net Ġcombine Ġresidual Ġconnections Ġusing Ġconc aten ation Ġwhereas ĠRes id ual Ġand ĠD ense ĠRes id ual Ġuse Ġaddition . ĠWe Ġfound Ġthat ĠD ense ĠRes id ual Ġand ĠD ense R Net Ġperform Ġsimilarly Ġwith Ġeach Ġperforming Ġbetter Ġon Ġspecific Ġsubs ets Ġof ĠLib ri Spe ech . ĠWe Ġdecided Ġto Ġuse ĠD ense ĠRes id ual Ġfor Ġsubsequent Ġexperiments . ĠThe Ġmain Ġreason Ġis Ġthat Ġdue Ġto Ġconc aten ation , Ġthe Ġgrowth Ġfactor Ġfor ĠD ense Net Ġand ĠD ense R Net Ġrequires Ġtuning Ġfor Ġdeeper Ġmodels Ġwhereas ĠD ense ĠRes id ual Ġsimply Ġjust Ġrepeats Ġa Ġsub - blocks . </s> Language ĠModel </s> A Ġlanguage Ġmodel Ġ( LM ) Ġis Ġa Ġprobability Ġdistribution Ġover Ġarbitrary Ġsymbol Ġsequences ĠIN LINE FORM 0 Ġsuch Ġthat Ġmore Ġlikely Ġsequences Ġare Ġassigned Ġhigh Ġprobabilities . ĠL Ms Ġare Ġfrequently Ġused Ġto Ġcondition Ġbeam Ġsearch . ĠDuring Ġdecoding , Ġcandidates Ġare Ġevaluated Ġusing Ġboth Ġacoustic Ġscores Ġand ĠLM Ġscores . ĠTraditional ĠN - gram ĠL Ms Ġhave Ġbeen Ġaugmented Ġwith Ġneural ĠL Ms Ġin Ġrecent Ġwork ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ. </s> We Ġexperiment Ġwith Ġstatistical ĠN - gram Ġlanguage Ġmodels ĠB IB REF 23 Ġand Ġneural ĠTrans former - XL ĠB IB REF 11 Ġmodels . ĠOur Ġbest Ġresults Ġuse Ġacoustic Ġand Ġword - level ĠN - gram Ġlanguage Ġmodels Ġto Ġgenerate Ġa Ġcandidate Ġlist Ġusing Ġbeam Ġsearch Ġwith Ġa Ġwidth Ġof Ġ2048 . ĠNext , Ġan Ġexternal ĠTrans former - XL ĠLM Ġresc ores Ġthe Ġfinal Ġlist . ĠAll ĠL Ms Ġwere Ġtrained Ġon Ġdatasets Ġindependently Ġfrom Ġacoustic Ġmodels . ĠWe Ġshow Ġresults Ġwith Ġthe Ġneural ĠLM Ġin Ġour ĠResults Ġsection . ĠWe Ġobserved Ġa Ġstrong Ġcorrelation Ġbetween Ġthe Ġquality Ġof Ġthe Ġneural ĠLM Ġ( me asured Ġby Ġperplex ity ) Ġand ĠW ER Ġas Ġshown Ġin ĠFigure ĠFIG REF 20 Ġ. </s> N ovo Grad </s> For Ġtraining , Ġwe Ġuse Ġeither ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ) Ġwith Ġmomentum Ġor Ġour Ġown ĠNov o Grad , Ġan Ġoptim izer Ġsimilar Ġto ĠAdam ĠB IB REF 14 Ġ, Ġexcept Ġthat Ġits Ġsecond Ġmoments Ġare Ġcomputed Ġper Ġlayer Ġinstead Ġof Ġper Ġweight . ĠCompared Ġto ĠAdam , Ġit Ġreduces Ġmemory Ġconsumption Ġand Ġwe Ġfind Ġit Ġto Ġbe Ġmore Ġnumer ically Ġstable . </s> At Ġeach Ġstep ĠIN LINE FORM 0 Ġ, ĠNov o Grad Ġcomp utes Ġthe Ġsto ch astic Ġgradient ĠIN LINE FORM 1 Ġfollowing Ġthe Ġregular Ġforward - back ward Ġpass . ĠThen Ġthe Ġsecond - order Ġmoment ĠIN LINE FORM 2 Ġis Ġcomputed Ġfor Ġeach Ġlayer ĠIN LINE FORM 3 Ġsimilar Ġto ĠND - Adam ĠB IB REF 27 Ġ: ĠDIS PLAY FORM 0 </s> The Ġsecond - order Ġmoment ĠIN LINE FORM 0 Ġis Ġused Ġto Ġre - scale Ġgrad ients ĠIN LINE FORM 1 Ġbefore Ġcalculating Ġthe Ġfirst - order Ġmoment ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> If ĠL 2 - regular ization Ġis Ġused , Ġa Ġweight Ġdecay ĠIN LINE FORM 0 Ġis Ġadded Ġto Ġthe Ġre - sc aled Ġgradient Ġ( as Ġin ĠAdam W ĠB IB REF 28 Ġ): ĠDIS PLAY FORM 0 </s> Finally , Ġnew Ġweights Ġare Ġcomputed Ġusing Ġthe Ġlearning Ġrate ĠIN LINE FORM 0 Ġ: ĠDIS PLAY FORM 0 </s> Using ĠNov o Grad Ġinstead Ġof ĠSG D Ġwith Ġmomentum , Ġwe Ġdecreased Ġthe ĠW ER Ġon Ġdev - clean ĠLib ri Spe ech Ġfrom Ġ4 . 00 % Ġto Ġ3 . 64 %, Ġa Ġrelative Ġimprovement Ġof Ġ9 % Ġfor ĠJasper ĠDR Ġ10 x 5 . ĠWe Ġwill Ġfurther Ġanalyze ĠNov o Grad Ġin Ġforthcoming Ġwork . </s> Results </s> We Ġevaluate ĠJasper Ġacross Ġa Ġnumber Ġof Ġdatasets Ġin Ġvarious Ġdomains . ĠIn Ġall Ġexperiments , Ġwe Ġuse Ġdrop out Ġand Ġweight Ġdecay Ġas Ġregular ization . ĠAt Ġtraining Ġtime , Ġwe Ġuse Ġspeed Ġpert urb ation Ġwith Ġfixed Ġ+/- 10 % ĠB IB REF 29 Ġfor ĠLib ri Spe ech . ĠFor ĠWS J Ġand ĠHub 5 ' 00 , Ġwe Ġuse Ġa Ġrandom Ġspeed Ġpert urb ation Ġfactor Ġbetween Ġ[- 10 %, Ġ10 %] Ġas Ġeach Ġutter ance Ġis Ġfed Ġinto Ġthe Ġmodel . ĠAll Ġmodels Ġhave Ġbeen Ġtrained Ġon ĠNVIDIA ĠDG X - 1 Ġin Ġmixed Ġprecision ĠB IB REF 30 Ġusing ĠOpen Se q 2 Se q ĠB IB REF 31 Ġ. ĠSource Ġcode , Ġtraining Ġconfigurations , Ġand Ġpret rained Ġmodels Ġare Ġavailable . </s> Read ĠSpeech </s> We Ġevaluated Ġthe Ġperformance Ġof ĠJasper Ġon Ġtwo Ġread Ġspeech Ġdatasets : ĠLib ri Spe ech Ġand ĠWall ĠStreet ĠJournal Ġ( WS J ). ĠFor ĠLib ri Spe ech , Ġwe Ġtrained ĠJasper ĠDR Ġ10 x 5 Ġusing Ġour ĠNov o Grad Ġoptim izer Ġfor Ġ400 Ġepoch s . ĠWe Ġachieve ĠS OTA Ġperformance Ġon Ġthe Ġtest - clean Ġsubset Ġand ĠS OTA Ġamong Ġend - to - end Ġspeech Ġrecognition Ġmodels Ġon Ġtest - other . </s> We Ġtrained Ġa Ġsmaller ĠJasper Ġ10 x 3 Ġmodel Ġwith ĠSG D Ġwith Ġmomentum Ġoptim izer Ġfor Ġ400 Ġepoch s Ġon Ġa Ġcombined ĠWS J Ġdataset Ġ( 80 Ġhours ): ĠL DC 93 S 6 A Ġ( WS J 0 ) Ġand ĠL DC 94 S 13 A Ġ( WS J 1 ). ĠThe Ġresults Ġare Ġprovided Ġin ĠTable ĠT AB REF 29 Ġ. </s> Con vers ational ĠSpeech </s> We Ġalso Ġevaluate Ġthe ĠJasper Ġmodel 's Ġperformance Ġon Ġa Ġconvers ational ĠEnglish Ġcorpus . ĠThe ĠHub 5 ĠYear Ġ2000 Ġ( Hub 5 ' 00 ) Ġevaluation Ġ( L DC 2002 S 09 , ĠL DC 2005 S 13 ) Ġis Ġwidely Ġused Ġin Ġacademia . ĠIt Ġis Ġdivided Ġinto Ġtwo Ġsubs ets : ĠSwitch board Ġ( SW B ) Ġand ĠCall home Ġ( CH M ). ĠThe Ġtraining Ġdata Ġfor Ġboth Ġthe Ġacoustic Ġand Ġlanguage Ġmodels Ġconsisted Ġof Ġthe Ġ2000 hr ĠFisher + Switch board Ġtraining Ġdata Ġ( L DC 2004 S 13 , ĠL DC 2005 S 13 , ĠL DC 97 S 62 ). ĠJasper ĠDR Ġ10 x 5 Ġwas Ġtrained Ġusing ĠSG D Ġwith Ġmomentum Ġfor Ġ50 Ġepoch s . ĠWe Ġcompare Ġto Ġother Ġmodels Ġtrained Ġusing Ġthe Ġsame Ġdata Ġand Ġreport ĠHub 5 ' 00 Ġresults Ġin ĠTable ĠT AB REF 31 Ġ. </s> We Ġobtain Ġgood Ġresults Ġfor ĠSW B . ĠHowever , Ġthere Ġis Ġwork Ġto Ġbe Ġdone Ġto Ġimprove ĠW ER Ġon Ġharder Ġtasks Ġsuch Ġas ĠCH M . </s> Con clusions </s> We Ġhave Ġpresented Ġa Ġnew Ġfamily Ġof Ġneural Ġarchitectures Ġfor Ġend - to - end Ġspeech Ġrecognition . ĠInspired Ġby Ġw av 2 letter 's Ġconv olution al Ġapproach , Ġwe Ġbuild Ġa Ġdeep Ġand Ġscalable Ġmodel , Ġwhich Ġrequires Ġa Ġwell - designed Ġresidual Ġtop ology , Ġeffective Ġregular ization , Ġand Ġa Ġstrong Ġoptim izer . ĠAs Ġour Ġarchitecture Ġstudies Ġdemonstrated , Ġa Ġcombination Ġof Ġstandard Ġcomponents Ġleads Ġto ĠS OTA Ġresults Ġon ĠLib ri Spe ech Ġand Ġcompetitive Ġresults Ġon Ġother Ġbenchmarks . ĠOur ĠJasper Ġarchitecture Ġis Ġhighly Ġefficient Ġfor Ġtraining Ġand Ġinference , Ġand Ġserves Ġas Ġa Ġgood Ġbaseline Ġapproach Ġon Ġtop Ġof Ġwhich Ġto Ġexplore Ġmore Ġsophisticated Ġregular ization , Ġdata Ġaug mentation , Ġloss Ġfunctions , Ġlanguage Ġmodels , Ġand Ġoptimization Ġstrategies . ĠWe Ġare Ġinterested Ġto Ġsee Ġif Ġour Ġapproach Ġcan Ġcontinue Ġto Ġscale Ġto Ġdeeper Ġmodels Ġand Ġlarger Ġdatasets .