2.49% for  layer-wise training, 2.63% for distillation, 6.26% for transfer learning.
ABSTRACTIVE
Their best model achieved a 2.49% Character Error Rate.
ABSTRACTIVE
FLOAT SELECTED: Table 3. The CER and RTF of 9-layers, 2-layers regular-trained and 2-laryers distilled LSTM. FLOAT SELECTED: Table 2. The CER of 6 to 9-layers models trained by regular Xavier Initialization, layer-wise training with CE criterion and CE + sMBR criteria. The teacher of 9-layer model is 8-layers sMBR model, while the others’ teacher is CE model. FLOAT SELECTED: Table 4. The CER of different 2-layers models, which are Shenma distilled model, Amap model further trained with Amap dataset, and Shenma model trained with sMBR on Amap dataset.
FLOAT SELECTED: Table 3. The CER and RTF of 9-layers, 2-layers regular-trained and 2-laryers distilled LSTM. FLOAT SELECTED: Table 2. The CER of 6 to 9-layers models trained by regular Xavier Initialization, layer-wise training with CE criterion and CE + sMBR criteria. The teacher of 9-layer model is 8-layers sMBR model, while the others’ teacher is CE model.
1703.07090
Introduction </s> Recently , Ġdeep Ġneural Ġnetwork Ġhas Ġbeen Ġwidely Ġemployed Ġin Ġvarious Ġrecognition Ġtasks . ĠIncreasing Ġthe Ġdepth Ġof Ġneural Ġnetwork Ġis Ġa Ġeffective Ġway Ġto Ġimprove Ġthe Ġperformance , Ġand Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġhas Ġbenefited Ġfrom Ġit Ġin Ġvisual Ġrecognition Ġtask ĠB IB REF 0 Ġ. ĠDe eper Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ) Ġare Ġalso Ġapplied Ġin Ġlarge Ġvocabulary Ġcontinuous Ġspeech Ġrecognition Ġ( LV CS R ) Ġtask , Ġbecause ĠL ST M Ġnetworks Ġhave Ġshown Ġbetter Ġperformance Ġthan ĠFully - connected Ġfeed - forward Ġdeep Ġneural Ġnetwork ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. </s> Training Ġneural Ġnetwork Ġbecomes Ġmore Ġchallenge Ġwhen Ġit Ġgoes Ġdeep . ĠA Ġconceptual Ġtool Ġcalled Ġlinear Ġclass ifier Ġprobe Ġis Ġintroduced Ġto Ġbetter Ġunderstand Ġthe Ġdynamics Ġinside Ġa Ġneural Ġnetwork ĠB IB REF 5 Ġ. ĠThe Ġdiscriminating Ġfeatures Ġof Ġlinear Ġclass ifier Ġis Ġthe Ġhidden Ġunits Ġof Ġa Ġintermediate Ġlayer . ĠFor Ġdeep Ġneural Ġnetworks , Ġit Ġis Ġobserved Ġthat Ġdeeper Ġlayer 's Ġaccuracy Ġis Ġlower Ġthan Ġthat Ġof Ġshall ower Ġlayers . ĠTherefore , Ġthe Ġtool Ġshows Ġthe Ġdifficulty Ġof Ġdeep Ġneural Ġmodel Ġtraining Ġvisually . </s> Layer - wise Ġpre - training Ġis Ġa Ġsuccessful Ġmethod Ġto Ġtrain Ġvery Ġdeep Ġneural Ġnetworks ĠB IB REF 6 Ġ. ĠThe Ġconvergence Ġbecomes Ġharder Ġwith Ġincreasing Ġthe Ġnumber Ġof Ġlayers , Ġeven Ġthough Ġthe Ġmodel Ġis Ġinitialized Ġwith ĠXavier Ġor Ġits Ġvariants ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠBut Ġthe Ġdeeper Ġnetwork Ġwhich Ġis Ġinitialized Ġwith Ġa Ġshall ower Ġtrained Ġnetwork Ġcould Ġconverge Ġwell . </s> The Ġsize Ġof ĠLV CS R Ġtraining Ġdataset Ġgoes Ġlarger Ġand Ġtraining Ġwith Ġonly Ġone ĠGPU Ġbecomes Ġhigh Ġtime Ġconsumption Ġinevitably . ĠTherefore , Ġparallel Ġtraining Ġwith Ġmulti - GPU s Ġis Ġmore Ġsuitable Ġfor ĠLV CS R Ġsystem . ĠMini - batch Ġbased Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) Ġis Ġthe Ġmost Ġpopular Ġmethod Ġin Ġneural Ġnetwork Ġtraining Ġprocedure . ĠAs ynchronous ĠSG D Ġis Ġa Ġsuccessful Ġeffort Ġfor Ġparallel Ġtraining Ġbased Ġon Ġit ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠIt Ġcan Ġmany Ġtimes Ġspeed Ġup Ġthe Ġtraining Ġtime Ġwithout Ġdecreasing Ġthe Ġaccuracy . ĠBesides , Ġsynchron ous ĠSG D Ġis Ġanother Ġeffective Ġeffort , Ġwhere Ġthe Ġparameter Ġserver Ġwaits Ġfor Ġevery Ġworks Ġto Ġfinish Ġtheir Ġcomputation Ġand Ġsent Ġtheir Ġlocal Ġmodels Ġto Ġit , Ġand Ġthen Ġit Ġsends Ġupdated Ġmodel Ġback Ġto Ġall Ġworkers ĠB IB REF 11 Ġ. ĠSyn chron ous ĠSG D Ġconver ges Ġwell Ġin Ġparallel Ġtraining Ġwith Ġdata Ġparallel ism , Ġand Ġis Ġalso Ġeasy Ġto Ġimplement . </s> In Ġorder Ġto Ġfurther Ġimprove Ġthe Ġperformance Ġof Ġdeep Ġneural Ġnetwork Ġwith Ġparallel Ġtraining , Ġseveral Ġmethods Ġare Ġproposed . ĠModel Ġaveraging Ġmethod Ġachieves Ġlinear Ġspeed up , Ġas Ġthe Ġfinal Ġmodel Ġis Ġaveraged Ġfrom Ġall Ġparameters Ġof Ġlocal Ġmodels Ġin Ġdifferent Ġworkers ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, Ġbut Ġthe Ġaccuracy Ġdecreases Ġcompared Ġwith Ġsingle ĠGPU Ġtraining . ĠMoreover , Ġblock wise Ġmodel - up dating Ġfilter Ġ( BM UF ) Ġprovides Ġanother Ġalmost Ġlinear Ġspeed up Ġapproach Ġwith Ġmulti - GPU s Ġon Ġthe Ġbasis Ġof Ġmodel Ġaveraging . ĠIt Ġcan Ġachieve Ġimprovement Ġor Ġno - deg rad ation Ġof Ġrecognition Ġperformance Ġcompared Ġwith Ġmini - batch ĠSG D Ġon Ġsingle ĠGPU ĠB IB REF 14 Ġ. </s> Moving Ġaveraged Ġ( MA ) Ġapproaches Ġare Ġalso Ġproposed Ġfor Ġparallel Ġtraining . ĠIt Ġis Ġdemonstrated Ġthat Ġthe Ġmoving Ġaverage Ġof Ġthe Ġparameters Ġobtained Ġby ĠSG D Ġperforms Ġas Ġwell Ġas Ġthe Ġparameters Ġthat Ġminimize Ġthe Ġempirical Ġcost , Ġand Ġmoving Ġaverage Ġparameters Ġcan Ġbe Ġused Ġas Ġthe Ġestim ator Ġof Ġthem , Ġif Ġthe Ġsize Ġof Ġtraining Ġdata Ġis Ġlarge Ġenough ĠB IB REF 15 Ġ. ĠOne Ġpass Ġlearning Ġis Ġthen Ġproposed , Ġwhich Ġis Ġthe Ġcombination Ġof Ġlearning Ġrate Ġschedule Ġand Ġaveraged ĠSG D Ġusing Ġmoving Ġaverage ĠB IB REF 16 Ġ. ĠExp onential Ġmoving Ġaverage Ġ( EMA ) Ġis Ġproposed Ġas Ġa Ġnon - inter ference Ġmethod ĠB IB REF 17 Ġ. ĠE MA Ġmodel Ġis Ġnot Ġbroadcast ed Ġto Ġworkers Ġto Ġupdate Ġtheir Ġlocal Ġmodels , Ġand Ġit Ġis Ġapplied Ġas Ġthe Ġfinal Ġmodel Ġof Ġentire Ġtraining Ġprocess . ĠE MA Ġmethod Ġis Ġutilized Ġwith Ġmodel Ġaveraging Ġand ĠBM UF Ġto Ġfurther Ġdecrease Ġthe Ġcharacter Ġerror Ġrate Ġ( C ER ). ĠIt Ġis Ġalso Ġeasy Ġto Ġimplement Ġin Ġexisting Ġparallel Ġtraining Ġsystems . </s> Frame Ġstacking Ġcan Ġalso Ġspeed Ġup Ġthe Ġtraining Ġtime ĠB IB REF 18 Ġ. ĠThe Ġsuper Ġframe Ġis Ġstacked Ġby Ġseveral Ġregular Ġframes , Ġand Ġit Ġcontains Ġthe Ġinformation Ġof Ġthem . ĠThus , Ġthe Ġnetwork Ġcan Ġsee Ġmultiple Ġframes Ġat Ġa Ġtime , Ġas Ġthe Ġsuper Ġframe Ġis Ġnew Ġinput . ĠFrame Ġstacking Ġcan Ġalso Ġlead Ġto Ġfaster Ġdecoding . </s> For Ġstreaming Ġvoice Ġsearch Ġservice , Ġit Ġneeds Ġto Ġdisplay Ġintermediate Ġrecognition Ġresults Ġwhile Ġusers Ġare Ġstill Ġspeaking . ĠAs Ġa Ġresult , Ġthe Ġsystem Ġneeds Ġto Ġfulfill Ġhigh Ġreal - time Ġrequirement , Ġand Ġwe Ġprefer Ġun id irection al ĠL ST M Ġnetwork Ġrather Ġthan Ġbid irection al Ġone . ĠHigh Ġreal - time Ġrequirement Ġmeans Ġlow Ġreal Ġtime Ġfactor Ġ( R TF ), Ġbut Ġthe ĠR TF Ġof Ġdeep ĠL ST M Ġmodel Ġis Ġhigher Ġinevitably . ĠThe Ġdilemma Ġof Ġrecognition Ġaccuracy Ġand Ġreal - time Ġrequirement Ġis Ġan Ġobstacle Ġto Ġthe Ġemployment Ġof Ġdeep ĠL ST M Ġnetwork . ĠDeep Ġmodel Ġoutper forms Ġbecause Ġit Ġcontains Ġmore Ġknowledge , Ġbut Ġit Ġis Ġalso Ġcumbersome . ĠAs Ġa Ġresult , Ġthe Ġknowledge Ġof Ġdeep Ġmodel Ġcan Ġbe Ġdistilled Ġto Ġa Ġshallow Ġmodel ĠB IB REF 19 Ġ. ĠIt Ġprovided Ġa Ġeffective Ġway Ġto Ġemploy Ġthe Ġdeep Ġmodel Ġto Ġthe Ġreal - time Ġsystem . </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġa Ġentire Ġdeep ĠL ST M ĠR NN Ġtraining Ġframework , Ġand Ġemploy Ġit Ġto Ġreal - time Ġapplication . ĠThe Ġdeep Ġlearning Ġsystems Ġbenefit Ġhighly Ġfrom Ġa Ġlarge Ġquantity Ġof Ġlabeled Ġtraining Ġdata . ĠOur Ġfirst Ġand Ġbasic Ġspeech Ġrecognition Ġsystem Ġis Ġtrained Ġon Ġ17 000 Ġhours Ġof ĠShen ma Ġvoice Ġsearch Ġdataset . ĠIt Ġis Ġa Ġgeneric Ġdataset Ġsampled Ġfrom Ġdiverse Ġaspects Ġof Ġsearch Ġqueries . ĠThe Ġrequirement Ġof Ġspeech Ġrecognition Ġsystem Ġalso Ġaddressed Ġby Ġspecific Ġscenario , Ġsuch Ġas Ġmap Ġand Ġnavigation Ġtask . ĠThe Ġlabeled Ġdataset Ġis Ġtoo Ġexpensive , Ġand Ġtraining Ġa Ġnew Ġmodel Ġwith Ġnew Ġlarge Ġdataset Ġfrom Ġthe Ġbeginning Ġcosts Ġlots Ġof Ġtime . ĠThus , Ġit Ġis Ġnatural Ġto Ġthink Ġof Ġtransferring Ġthe Ġknowledge Ġfrom Ġbasic Ġmodel Ġto Ġnew Ġscenario 's Ġmodel . ĠTransfer Ġlearning Ġexp ends Ġless Ġdata Ġand Ġless Ġtraining Ġtime Ġthan Ġfull Ġtraining . ĠIn Ġthis Ġpaper , Ġwe Ġalso Ġintroduce Ġa Ġnovel Ġtransfer Ġlearning Ġstrategy Ġwith Ġsegment al ĠMinimum ĠBay es - R isk Ġ( s MB R ). ĠAs Ġa Ġresult , Ġtransfer Ġtraining Ġwith Ġonly Ġ1000 Ġhours Ġdata Ġcan Ġmatch Ġequivalent Ġperformance Ġfor Ġfull Ġtraining Ġwith Ġ7 300 Ġhours Ġdata . </s> Our Ġdeep ĠL ST M Ġtraining Ġframework Ġfor ĠLV CS R Ġis Ġpresented Ġin ĠSection Ġ2 . ĠSection Ġ3 Ġdescribes Ġhow Ġthe Ġvery Ġdeep Ġmodels Ġdoes Ġapply Ġin Ġreal Ġworld Ġapplications , Ġand Ġhow Ġto Ġtransfer Ġthe Ġmodel Ġto Ġanother Ġtask . ĠThe Ġframework Ġis Ġanalyzed Ġand Ġdiscussed Ġin ĠSection Ġ4 , Ġand Ġfollowed Ġby Ġthe Ġconclusion Ġin ĠSection Ġ5 . </s> Layer - wise ĠTraining Ġwith ĠSoft ĠTarget Ġand ĠHard ĠTarget </s> Grad ient - based Ġoptimization Ġof Ġdeep ĠL ST M Ġnetwork Ġwith Ġrandom Ġinitialization Ġget Ġstuck Ġin Ġpoor Ġsolution Ġeasily . ĠXavier Ġinitialization Ġcan Ġpartially Ġsolve Ġthis Ġproblem ĠB IB REF 7 Ġ, Ġso Ġthis Ġmethod Ġis Ġthe Ġregular Ġinitialization Ġmethod Ġof Ġall Ġtraining Ġprocedure . ĠHowever , Ġit Ġdoes Ġnot Ġwork Ġwell Ġwhen Ġit Ġis Ġutilized Ġto Ġinitialize Ġvery Ġdeep Ġmodel Ġdirectly , Ġbecause Ġof Ġvanishing Ġor Ġexploding Ġgrad ients . ĠInstead , Ġlayer - wise Ġpre - training Ġmethod Ġis Ġa Ġeffective Ġway Ġto Ġtrain Ġthe Ġweights Ġof Ġvery Ġdeep Ġarchitecture ĠB IB REF 6 Ġ, ĠB IB REF 20 Ġ. ĠIn Ġlayer - wise Ġpre - training Ġprocedure , Ġa Ġone - layer ĠL ST M Ġmodel Ġis Ġfirst ly Ġtrained Ġwith Ġnormalized Ġinitialization . ĠSequ entially , Ġtwo - l ayers ĠL ST M Ġmodel 's Ġfirst Ġlayer Ġis Ġinitialized Ġby Ġtrained Ġone - layer Ġmodel , Ġand Ġits Ġsecond Ġlayer Ġis Ġregularly Ġinitialized . ĠIn Ġthis Ġway , Ġa Ġdeep Ġarchitecture Ġis Ġlayer - by - layer Ġtrained , Ġand Ġit Ġcan Ġconverge Ġwell . </s> In Ġconventional Ġlayer - wise Ġpre - training , Ġonly Ġparameters Ġof Ġshall ower Ġnetwork Ġare Ġtransf ered Ġto Ġdeeper Ġone , Ġand Ġthe Ġlearning Ġtargets Ġare Ġstill Ġthe Ġalign ments Ġgenerated Ġby ĠH MM - G MM Ġsystem . ĠThe Ġtargets Ġare Ġvectors Ġthat Ġonly Ġone Ġstate 's Ġprobability Ġis Ġone , Ġand Ġthe Ġothers ' Ġare Ġz eros . ĠThey Ġare Ġknown Ġas Ġhard Ġtargets , Ġand Ġthey Ġcarry Ġlimited Ġknowledge Ġas Ġonly Ġone Ġstate Ġis Ġactive . ĠIn Ġcontrast , Ġthe Ġknowledge Ġof Ġshall ower Ġnetwork Ġshould Ġbe Ġalso Ġtransf ered Ġto Ġdeeper Ġone . ĠIt Ġis Ġobtained Ġby Ġthe Ġsoft max Ġlayer Ġof Ġexisting Ġmodel Ġtypically , Ġso Ġeach Ġstate Ġhas Ġa Ġprobability Ġrather Ġthan Ġonly Ġzero Ġor Ġone , Ġand Ġcalled Ġas Ġsoft Ġtarget . ĠAs Ġa Ġresult , Ġthe Ġdeeper Ġnetwork Ġwhich Ġis Ġstudent Ġnetwork Ġlearns Ġthe Ġparameters Ġand Ġknowledge Ġfrom Ġshall ower Ġone Ġwhich Ġis Ġcalled Ġteacher Ġnetwork . ĠWhen Ġtraining Ġthe Ġstudent Ġnetwork Ġfrom Ġthe Ġteacher Ġnetwork , Ġthe Ġfinal Ġalignment Ġis Ġthe Ġcombination Ġof Ġhard Ġtarget Ġand Ġsoft Ġtarget Ġin Ġour Ġlayer - wise Ġtraining Ġphase . ĠThe Ġfinal Ġalignment Ġprovides Ġvarious Ġknowledge Ġwhich Ġtransf ered Ġfrom Ġteacher Ġnetwork Ġand Ġextracted Ġfrom Ġtrue Ġlabels . ĠIf Ġonly Ġsoft Ġtarget Ġis Ġlearned , Ġstudent Ġnetwork Ġperform Ġno Ġbetter Ġthan Ġteacher Ġnetwork , Ġbut Ġit Ġcould Ġoutper form Ġteacher Ġnetwork Ġas Ġit Ġalso Ġlearns Ġtrue Ġlabels . </s> The Ġdeeper Ġnetwork Ġspends Ġless Ġtime Ġto Ġgetting Ġthe Ġsame Ġlevel Ġof Ġoriginal Ġnetwork Ġthan Ġthe Ġnetwork Ġtrained Ġfrom Ġthe Ġbeginning , Ġas Ġa Ġperiod Ġof Ġlow Ġperformance Ġis Ġskipped . ĠTherefore , Ġtraining Ġwith Ġhard Ġand Ġsoft Ġtarget Ġis Ġa Ġtime Ġsaving Ġmethod . ĠFor Ġlarge Ġtraining Ġdataset , Ġtraining Ġwith Ġthe Ġwhole Ġdataset Ġstill Ġspends Ġtoo Ġmuch Ġtime . ĠA Ġnetwork Ġfirst ly Ġtrained Ġwith Ġonly Ġa Ġsmall Ġpart Ġof Ġdataset Ġcould Ġgo Ġdeeper Ġas Ġwell , Ġand Ġso Ġthe Ġtraining Ġtime Ġreducing Ġrapidly . ĠWhen Ġthe Ġnetwork Ġis Ġdeep Ġenough , Ġit Ġthen Ġtrained Ġon Ġthe Ġentire Ġdataset Ġto Ġget Ġfurther Ġimprovement . ĠThere Ġis Ġno Ġgap Ġof Ġaccuracy Ġbetween Ġthese Ġtwo Ġapproaches , Ġbut Ġlatter Ġone Ġsaves Ġmuch Ġtime . </s> Different ial ĠS aturation ĠCheck </s> The Ġobjects Ġof Ġconventional Ġsaturation Ġcheck Ġare Ġgrad ients Ġand Ġthe Ġcell Ġactiv ations ĠB IB REF 4 Ġ. ĠGrad ients Ġare Ġclipped Ġto Ġrange Ġ[- 5 , Ġ5 ], Ġwhile Ġthe Ġcell Ġactiv ations Ġclipped Ġto Ġrange Ġ[- 50 , Ġ50 ]. ĠApart Ġfrom Ġthem , Ġthe Ġdifferent ials Ġof Ġrecurrent Ġlayers Ġis Ġalso Ġlimited . ĠIf Ġthe Ġdifferent ials Ġgo Ġbeyond Ġthe Ġrange , Ġcorresponding Ġback Ġpropagation Ġis Ġskipped , Ġwhile Ġif Ġthe Ġgrad ients Ġand Ġcell Ġactiv ations Ġgo Ġbeyond Ġthe Ġbound , Ġvalues Ġare Ġset Ġas Ġthe Ġboundary Ġvalues . ĠThe Ġdifferent ials Ġwhich Ġare Ġtoo Ġlarge Ġor Ġtoo Ġsmall Ġwill Ġlead Ġto Ġthe Ġgrad ients Ġeasily Ġvanishing , Ġand Ġit Ġdemonstrates Ġthe Ġfailure Ġof Ġthis Ġpropagation . ĠAs Ġa Ġresult , Ġthe Ġparameters Ġare Ġnot Ġupdated , Ġand Ġnext Ġpropagation Ġ. </s> Sequ ence ĠDisc rim in ative ĠTraining </s> Cross - ent ropy Ġ( CE ) Ġis Ġwidely Ġused Ġin Ġspeech Ġrecognition Ġtraining Ġsystem Ġas Ġa Ġframe - wise Ġdiscrim in ative Ġtraining Ġcriterion . ĠHowever , Ġit Ġis Ġnot Ġwell Ġsuited Ġto Ġspeech Ġrecognition , Ġbecause Ġspeech Ġrecognition Ġtraining Ġis Ġa Ġsequential Ġlearning Ġproblem . ĠIn Ġcontrast , Ġsequence Ġdiscrim in ative Ġtraining Ġcriterion Ġhas Ġshown Ġto Ġfurther Ġimprove Ġperformance Ġof Ġneural Ġnetwork Ġfirst Ġtrained Ġwith Ġcross - ent ropy ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠWe Ġchoose Ġstate - level Ġminimum Ġbay es Ġrisk Ġ( s MB R ) ĠB IB REF 21 Ġamong Ġa Ġnumber Ġof Ġsequence Ġdiscrim in ative Ġcriterion Ġis Ġproposed , Ġsuch Ġas Ġmaximum Ġmutual Ġinformation Ġ( M MI ) ĠB IB REF 24 Ġand Ġminimum Ġphone Ġerror Ġ( MP E ) ĠB IB REF 25 Ġ. ĠMP E Ġand Ġs MB R Ġare Ġdesigned Ġto Ġminimize Ġthe Ġexpected Ġerror Ġof Ġdifferent Ġgran ularity Ġof Ġlabels , Ġwhile ĠCE Ġaims Ġto Ġminim izes Ġexpected Ġframe Ġerror , Ġand ĠM MI Ġaims Ġto Ġminim izes Ġexpected Ġsentence Ġerror . ĠState - level Ġinformation Ġis Ġfocused Ġon Ġby Ġs MB R . </s> a Ġframe - level Ġaccurate Ġmodel Ġis Ġfirst ly Ġtrained Ġby ĠCE Ġloss Ġfunction , Ġand Ġthen Ġs MB R Ġloss Ġfunction Ġis Ġutilized Ġfor Ġfurther Ġtraining Ġto Ġget Ġsequence - level Ġaccuracy . ĠOnly Ġa Ġpart Ġof Ġtraining Ġdataset Ġis Ġneeded Ġin Ġs MB R Ġtraining Ġphase Ġon Ġthe Ġbasis Ġof Ġwhole Ġdataset ĠCE Ġtraining . </s> Par allel ĠTraining </s> It Ġis Ġdemonstrated Ġthat Ġtraining Ġwith Ġlarger Ġdataset Ġcan Ġimprove Ġrecognition Ġaccuracy . ĠHowever , Ġlarger Ġdataset Ġmeans Ġmore Ġtraining Ġsamples Ġand Ġmore Ġmodel Ġparameters . ĠTherefore , Ġparallel Ġtraining Ġwith Ġmultiple ĠGPUs Ġis Ġessential , Ġand Ġit Ġmakes Ġuse Ġof Ġdata Ġparallel ism ĠB IB REF 9 Ġ. ĠThe Ġentire Ġtraining Ġdata Ġis Ġpartition ed Ġinto Ġseveral Ġsplit Ġwithout Ġoverlapping Ġand Ġthey Ġare Ġdistributed Ġto Ġdifferent ĠGPUs . ĠEach ĠGPU Ġtrains Ġwith Ġone Ġsplit Ġof Ġtraining Ġdataset Ġlocally . ĠAll ĠGPUs Ġsynchron ize Ġtheir Ġlocal Ġmodels Ġwith Ġmodel Ġaverage Ġmethod Ġafter Ġa Ġmini - batch Ġoptimization ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> Model Ġaverage Ġmethod Ġachieves Ġlinear Ġspeed up Ġin Ġtraining Ġphase , Ġbut Ġthe Ġrecognition Ġaccuracy Ġdecreases Ġcompared Ġwith Ġsingle ĠGPU Ġtraining . ĠBlock - wise Ġmodel Ġupdating Ġfilter Ġ( BM UF ) Ġis Ġanother Ġsuccessful Ġeffort Ġin Ġparallel Ġtraining Ġwith Ġlinear Ġspeed up Ġas Ġwell . ĠIt Ġcan Ġachieve Ġno - deg rad ation Ġof Ġrecognition Ġaccuracy Ġwith Ġmulti - GPU s ĠB IB REF 14 Ġ. ĠIn Ġthe Ġmodel Ġaverage Ġmethod , Ġaggreg ated Ġmodel ĠIN LINE FORM 0 Ġis Ġcomputed Ġand Ġbroadcast ed Ġto ĠGPUs . ĠOn Ġthe Ġbasis Ġof Ġit , ĠBM UF Ġproposed Ġa Ġnovel Ġmodel Ġupdating Ġstrategy : ĠIN LINE FORM 1 ĠIN LINE FORM 2 </s> Where ĠIN LINE FORM 0 Ġdenotes Ġmodel Ġupdate , Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġglobal - model Ġupdate . ĠThere Ġare Ġtwo Ġparameters Ġin ĠBM UF , Ġblock Ġmomentum ĠIN LINE FORM 2 Ġ, Ġand Ġblock Ġlearning Ġrate ĠIN LINE FORM 3 Ġ. ĠThen , Ġthe Ġglobal Ġmodel Ġis Ġupdated Ġas ĠIN LINE FORM 4 </s> C onsequ ently , ĠIN LINE FORM 0 Ġis Ġbroadcast ed Ġto Ġall ĠGPUs Ġto Ġinitial Ġtheir Ġlocal Ġmodels , Ġinstead Ġof ĠIN LINE FORM 1 Ġin Ġmodel Ġaverage Ġmethod . </s> A ver aged ĠSG D Ġis Ġproposed Ġto Ġfurther Ġaccelerate Ġthe Ġconvergence Ġspeed Ġof ĠSG D . ĠA ver aged ĠSG D Ġlever ages Ġthe Ġmoving Ġaverage Ġ( MA ) ĠIN LINE FORM 0 Ġas Ġthe Ġestim ator Ġof ĠIN LINE FORM 1 ĠB IB REF 15 Ġ: ĠIN LINE FORM 2 </s> Where ĠIN LINE FORM 0 Ġis Ġcomputed Ġby Ġmodel Ġaveraging Ġor ĠBM UF . ĠIt Ġis Ġshown Ġthat ĠIN LINE FORM 1 Ġcan Ġwell Ġconverge Ġto ĠIN LINE FORM 2 Ġ, Ġwith Ġthe Ġlarge Ġenough Ġtraining Ġdataset Ġin Ġsingle ĠGPU Ġtraining . ĠIt Ġcan Ġbe Ġconsidered Ġas Ġa Ġnon - inter ference Ġstrategy Ġthat ĠIN LINE FORM 3 Ġdoes Ġnot Ġparticipate Ġthe Ġmain Ġoptimization Ġprocess , Ġand Ġonly Ġtakes Ġeffect Ġafter Ġthe Ġend Ġof Ġentire Ġoptimization . ĠHowever , Ġfor Ġthe Ġparallel Ġtraining Ġimplementation , Ġeach ĠIN LINE FORM 4 Ġis Ġcomputed Ġby Ġmodel Ġaveraging Ġand ĠBM UF Ġwith Ġmultiple Ġmodels , Ġand Ġmoving Ġaverage Ġmodel ĠIN LINE FORM 5 Ġdoes Ġnot Ġwell Ġconverge , Ġcompared Ġwith Ġsingle ĠGPU Ġtraining . </s> Model Ġaveraging Ġbased Ġmethods Ġare Ġemployed Ġin Ġparallel Ġtraining Ġof Ġlarge Ġscale Ġdataset , Ġbecause Ġof Ġtheir Ġfaster Ġconvergence , Ġand Ġespecially Ġno - deg rad ation Ġimplementation Ġof ĠBM UF . ĠBut Ġcombination Ġof Ġmodel Ġaveraged Ġbased Ġmethods Ġand Ġmoving Ġaverage Ġdoes Ġnot Ġmatch Ġthe Ġexpectation Ġof Ġfurther Ġenhance Ġperformance Ġand Ġit Ġis Ġpresented Ġas ĠIN LINE FORM 0 </s> The Ġweight Ġof Ġeach ĠIN LINE FORM 0 Ġis Ġequal Ġin Ġmoving Ġaverage Ġmethod Ġregardless Ġthe Ġeffect Ġof Ġtemporal Ġorder . ĠBut ĠIN LINE FORM 1 Ġcloser Ġto Ġthe Ġend Ġof Ġtraining Ġachieve Ġhigher Ġaccuracy Ġin Ġthe Ġmodel Ġaveraging Ġbased Ġapproach , Ġand Ġthus Ġit Ġshould Ġbe Ġwith Ġmore Ġproportion Ġin Ġfinal ĠIN LINE FORM 2 Ġ. ĠAs Ġa Ġresult , Ġexponential Ġmoving Ġaverage ( EMA ) Ġis Ġappropriate , Ġwhich Ġthe Ġweight Ġfor Ġeach Ġolder Ġparameters Ġdecrease Ġexponentially , Ġand Ġnever Ġreaching Ġzero . ĠAfter Ġmoving Ġaverage Ġbased Ġmethods , Ġthe ĠE MA Ġparameters Ġare Ġupdated Ġrec urs ively Ġas ĠIN LINE FORM 3 </s> Here ĠIN LINE FORM 0 Ġrepresents Ġthe Ġdegree Ġof Ġweight Ġdecrease , Ġand Ġcalled Ġexponential Ġupdating Ġrate . ĠE MA Ġis Ġalso Ġa Ġnon - inter ference Ġtraining Ġstrategy Ġthat Ġis Ġimplemented Ġeasily , Ġas Ġthe Ġupdated Ġmodel Ġis Ġnot Ġbroadcast ed . ĠTherefore , Ġthere Ġis Ġno Ġneed Ġto Ġadd Ġextra Ġlearning Ġrate Ġupdating Ġapproach , Ġas Ġit Ġcan Ġbe Ġapp ended Ġto Ġexisting Ġtraining Ġprocedure Ġdirectly . </s> Deploy ment </s> There Ġis Ġa Ġhigh Ġreal Ġtime Ġrequirement Ġin Ġreal Ġworld Ġapplication , Ġespecially Ġin Ġonline Ġvoice Ġsearch Ġsystem . ĠShen ma Ġvoice Ġsearch Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġmobile Ġsearch Ġengines Ġin ĠChina , Ġand Ġit Ġis Ġa Ġstreaming Ġservice Ġthat Ġintermediate Ġrecognition Ġresults Ġdisplayed Ġwhile Ġusers Ġare Ġstill Ġspeaking . ĠUn id irection al ĠL ST M Ġnetwork Ġis Ġapplied , Ġrather Ġthan Ġbid irection al Ġone , Ġbecause Ġit Ġis Ġwell Ġsuited Ġto Ġreal - time Ġstreaming Ġspeech Ġrecognition . </s> Dist illation </s> It Ġis Ġdemonstrated Ġthat Ġdeep Ġneural Ġnetwork Ġarchitecture Ġcan Ġachieve Ġimprovement Ġin ĠLV CS R . ĠHowever , Ġit Ġalso Ġleads Ġto Ġmuch Ġmore Ġcomputation Ġand Ġhigher ĠR TF , Ġso Ġthat Ġthe Ġrecognition Ġresult Ġcan Ġnot Ġbe Ġdisplayed Ġin Ġreal Ġtime . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġdeeper Ġneural Ġnetwork Ġcontains Ġmore Ġknowledge , Ġbut Ġit Ġis Ġalso Ġcumbersome . Ġthe Ġknowledge Ġis Ġkey Ġto Ġimprove Ġthe Ġperformance . ĠIf Ġit Ġcan Ġbe Ġtransf ered Ġfrom Ġcumbersome Ġmodel Ġto Ġa Ġsmall Ġmodel , Ġthe Ġrecognition Ġability Ġcan Ġalso Ġbe Ġtransf ered Ġto Ġthe Ġsmall Ġmodel . ĠKnowledge Ġtransferring Ġto Ġsmall Ġmodel Ġis Ġcalled Ġdist illation ĠB IB REF 19 Ġ. ĠThe Ġsmall Ġmodel Ġcan Ġperform Ġas Ġwell Ġas Ġcumbersome Ġmodel , Ġafter Ġdist illing . ĠIt Ġprovide Ġa Ġway Ġto Ġutilize Ġhigh - performance Ġbut Ġhigh ĠR TF Ġmodel Ġin Ġreal Ġtime Ġsystem . ĠThe Ġclass Ġprobability Ġproduced Ġby Ġthe Ġcumbersome Ġmodel Ġis Ġregarded Ġas Ġsoft Ġtarget , Ġand Ġthe Ġgeneral ization Ġability Ġof Ġcumbersome Ġmodel Ġis Ġtransf ered Ġto Ġsmall Ġmodel Ġwith Ġit . ĠDist illation Ġis Ġmodel 's Ġknowledge Ġtransferring Ġapproach , Ġso Ġthere Ġis Ġno Ġneed Ġto Ġuse Ġthe Ġhard Ġtarget , Ġwhich Ġis Ġdifferent Ġwith Ġthe Ġlayer - wise Ġtraining Ġmethod . </s> 9 - l ayers Ġun id irection al ĠL ST M Ġmodel Ġachieves Ġoutstanding Ġperformance , Ġbut Ġmeanwhile Ġit Ġis Ġtoo Ġcomput ationally Ġexpensive Ġto Ġallow Ġdeployment Ġin Ġreal Ġtime Ġrecognition Ġsystem . ĠIn Ġorder Ġto Ġensure Ġreal - time Ġof Ġthe Ġsystem , Ġthe Ġnumber Ġof Ġlayers Ġneeds Ġto Ġbe Ġreduced . ĠThe Ġshall ower Ġnetwork Ġcan Ġlearn Ġthe Ġknowledge Ġof Ġdeeper Ġnetwork Ġwith Ġdist illation . ĠIt Ġis Ġfound Ġthat ĠR TF Ġof Ġ2 - l ayers Ġnetwork Ġis Ġacceptable , Ġso Ġthe Ġknowledge Ġis Ġdistilled Ġfrom Ġ9 - l ayers Ġwell - trained Ġmodel Ġto Ġ2 - l ayers Ġmodel . ĠTable ĠT AB REF 16 Ġshows Ġthat Ġdist illation Ġfrom Ġ9 - l ayers Ġto Ġ2 - l ayers Ġbrings ĠR TF Ġdecrease Ġof Ġrelative Ġ53 %, Ġwhile ĠC ER Ġonly Ġincreases Ġ5 %. ĠThe Ġknowledge Ġof Ġdeep Ġnetwork Ġis Ġalmost Ġtransf ered Ġwith Ġdist illation , ĠDist illation Ġbrings Ġpromising ĠR TF Ġreduction , Ġbut Ġonly Ġlittle Ġknowledge Ġof Ġdeep Ġnetwork Ġis Ġlost . ĠMoreover , ĠC ER Ġof Ġ2 - l ayers Ġdistilled ĠL ST M Ġdecreases Ġrelative Ġ14 %, Ġcompared Ġwith Ġ2 - l ayers Ġregular - trained ĠL ST M . </s> Transfer ĠLearning Ġwith Ġs MB R </s> For Ġa Ġcertain Ġspecific Ġscenario , Ġthe Ġmodel Ġtrained Ġwith Ġthe Ġdata Ġrecorded Ġfrom Ġit Ġhas Ġbetter Ġadaptation Ġthan Ġthe Ġmodel Ġtrained Ġwith Ġgeneric Ġscenario . ĠBut Ġit Ġspends Ġtoo Ġmuch Ġtime Ġtraining Ġa Ġmodel Ġfrom Ġthe Ġbeginning , Ġif Ġthere Ġis Ġa Ġwell - trained Ġmodel Ġfor Ġgeneric Ġscenarios . ĠMoreover , Ġlabeling Ġa Ġlarge Ġquantity Ġof Ġtraining Ġdata Ġin Ġnew Ġscenario Ġis Ġboth Ġcostly Ġand Ġtime Ġconsuming . ĠIf Ġa Ġmodel Ġtransfer Ġtrained Ġwith Ġsmaller Ġdataset Ġcan Ġobtained Ġthe Ġsimilar Ġrecognition Ġaccuracy Ġcompared Ġwith Ġthe Ġmodel Ġdirectly Ġtrained Ġwith Ġlarger Ġdataset , Ġit Ġis Ġno Ġdoubt Ġthat Ġtransfer Ġlearning Ġis Ġmore Ġpractical . ĠSince Ġspecific Ġscenario Ġis Ġa Ġsubset Ġof Ġgeneric Ġscenario , Ġsome Ġknowledge Ġcan Ġbe Ġshared Ġbetween Ġthem . ĠBesides , Ġgeneric Ġscenario Ġconsists Ġof Ġvarious Ġconditions , Ġso Ġits Ġmodel Ġhas Ġgreater Ġrobust ness . ĠAs Ġa Ġresult , Ġnot Ġonly Ġshared Ġknowledge Ġbut Ġalso Ġrobust ness Ġcan Ġbe Ġtransf ered Ġfrom Ġthe Ġmodel Ġof Ġgeneric Ġscenario Ġto Ġthe Ġmodel Ġof Ġspecific Ġone . </s> As Ġthe Ġmodel Ġwell Ġtrained Ġfrom Ġgeneric Ġscenario Ġachieves Ġgood Ġperformance Ġin Ġframe Ġlevel Ġclassification , Ġsequence Ġdiscrim in ative Ġtraining Ġis Ġrequired Ġto Ġadapt Ġnew Ġmodel Ġto Ġspecific Ġscenario Ġadditionally . ĠMoreover , Ġit Ġdoes Ġnot Ġneed Ġalignment Ġfrom ĠH MM - G MM Ġsystem , Ġand Ġit Ġalso Ġsaves Ġamount Ġof Ġtime Ġto Ġprepare Ġalignment . </s> Training ĠData </s> A Ġlarge Ġquantity Ġof Ġlabeled Ġdata Ġis Ġneeded Ġfor Ġtraining Ġa Ġmore Ġaccurate Ġacoustic Ġmodel . ĠWe Ġcollect Ġthe Ġ17 000 Ġhours Ġlabeled Ġdata Ġfrom ĠShen ma Ġvoice Ġsearch , Ġwhich Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġmobile Ġsearch Ġengines Ġin ĠChina . ĠThe Ġdataset Ġis Ġcreated Ġfrom Ġanonymous Ġonline Ġusers ' Ġsearch Ġqueries Ġin ĠMandarin , Ġand Ġall Ġaudio Ġfile 's Ġsampling Ġrate Ġis Ġ16 kHz , Ġrecorded Ġby Ġmobile Ġphones . ĠThis Ġdataset Ġconsists Ġof Ġmany Ġdifferent Ġconditions , Ġsuch Ġas Ġdiverse Ġnoise Ġeven Ġlow Ġsignal - to - no ise , Ġbab ble , Ġdialect s , Ġaccents , Ġhesitation Ġand Ġso Ġon . </s> In Ġthe ĠAm ap , Ġwhich Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġweb Ġmapping Ġand Ġnavigation Ġservices Ġin ĠChina , Ġusers Ġcan Ġsearch Ġlocations Ġand Ġnavigate Ġto Ġlocations Ġthey Ġwant Ġthough Ġvoice Ġsearch . ĠTo Ġpresent Ġthe Ġperformance Ġof Ġtransfer Ġlearning Ġwith Ġsequence Ġdiscrim in ative Ġtraining , Ġthe Ġmodel Ġtrained Ġfrom ĠShen ma Ġvoice Ġsearch Ġwhich Ġis Ġgren eric Ġscenario Ġtransfer Ġits Ġknowledge Ġto Ġthe Ġmodel Ġof ĠAm ap Ġvoice Ġsearch . Ġ7 300 Ġhours Ġlabeled Ġdata Ġis Ġcollected Ġin Ġthe Ġsimilar Ġway Ġof ĠShen ma Ġvoice Ġsearch Ġdata Ġcollection . </s> Two Ġdataset Ġis Ġdivided Ġinto Ġtraining Ġset , Ġvalidation Ġset Ġand Ġtest Ġset Ġseparately , Ġand Ġthe Ġquantity Ġof Ġthem Ġis Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. ĠThe Ġthree Ġsets Ġare Ġsplit Ġaccording Ġto Ġspeakers , Ġin Ġorder Ġto Ġavoid Ġutter ances Ġof Ġsame Ġspeaker Ġappearing Ġin Ġthree Ġsets Ġsimultaneously . ĠThe Ġtest Ġsets Ġof ĠShen ma Ġand ĠAm ap Ġvoice Ġsearch Ġare Ġcalled ĠShen ma ĠTest Ġand ĠAm ap ĠTest . </s> Exper imental Ġsetup </s> L ST M ĠR NN s Ġoutper form Ġconventional ĠR NN s Ġfor Ġspeech Ġrecognition Ġsystem , Ġespecially Ġdeep ĠL ST M ĠR NN s , Ġbecause Ġof Ġits Ġlong - range Ġdependencies Ġmore Ġaccurately Ġfor Ġtemporal Ġsequence Ġconditions ĠB IB REF 26 Ġ, ĠB IB REF 23 Ġ. ĠShen ma Ġand ĠAm ap Ġvoice Ġsearch Ġis Ġa Ġstreaming Ġservice Ġthat Ġintermediate Ġrecognition Ġresults Ġdisplayed Ġwhile Ġusers Ġare Ġstill Ġspeaking . ĠSo Ġas Ġfor Ġonline Ġrecognition Ġin Ġreal Ġtime , Ġwe Ġprefer Ġun id irection al ĠL ST M Ġmodel Ġrather Ġthan Ġbid irection al Ġone . ĠThus , Ġthe Ġtraining Ġsystem Ġis Ġun id irection al ĠL ST M - based . </s> A Ġ26 - dimensional Ġfilter Ġbank Ġand Ġ2 - dimensional Ġpitch Ġfeature Ġis Ġextracted Ġfor Ġeach Ġframe , Ġand Ġis Ġconc aten ated Ġwith Ġfirst Ġand Ġsecond Ġorder Ġdifference Ġas Ġthe Ġfinal Ġinput Ġof Ġthe Ġnetwork . ĠThe Ġsuper Ġframe Ġare Ġstacked Ġby Ġ3 Ġframes Ġwithout Ġoverlapping . ĠThe Ġarchitecture Ġwe Ġtrained Ġconsists Ġof Ġtwo ĠL ST M Ġlayers Ġwith Ġs igm oid Ġactivation Ġfunction , Ġfollowed Ġby Ġa Ġfull - connection Ġlayer . ĠThe Ġout Ġlayer Ġis Ġa Ġsoft max Ġlayer Ġwith Ġ110 88 Ġhidden Ġmark ov Ġmodel Ġ( H MM ) Ġtied - states Ġas Ġoutput Ġclasses , Ġthe Ġloss Ġfunction Ġis Ġcross - ent ropy Ġ( CE ). ĠThe Ġperformance Ġmetric Ġof Ġthe Ġsystem Ġin ĠMandarin Ġis Ġreported Ġwith Ġcharacter Ġerror Ġrate Ġ( C ER ). ĠThe Ġalignment Ġof Ġframe - level Ġground Ġtruth Ġis Ġobtained Ġby ĠGM M - H MM Ġsystem . ĠMini - bat ched ĠSG D Ġis Ġutilized Ġwith Ġmomentum Ġtrick Ġand Ġthe Ġnetwork Ġis Ġtrained Ġfor Ġa Ġtotal Ġof Ġ4 Ġepoch s . ĠThe Ġblock Ġlearning Ġrate Ġand Ġblock Ġmomentum Ġof ĠBM UF Ġare Ġset Ġas Ġ1 Ġand Ġ0 . 9 . Ġ5 - gram Ġlanguage Ġmodel Ġis Ġlever aged Ġin Ġdec oder , Ġand Ġthe Ġvocabulary Ġsize Ġis Ġas Ġlarge Ġas Ġ76 0000 . ĠDifferent ials Ġof Ġrecurrent Ġlayers Ġis Ġlimited Ġto Ġrange Ġ[- 10000 , 10000 ], Ġwhile Ġgrad ients Ġare Ġclipped Ġto Ġrange Ġ[- 5 , Ġ5 ] Ġand Ġcell Ġactiv ations Ġclipped Ġto Ġrange Ġ[- 50 , Ġ50 ]. ĠAfter Ġtraining Ġwith ĠCE Ġloss , Ġs MB R Ġloss Ġis Ġemployed Ġto Ġfurther Ġimprove Ġthe Ġperformance . </s> It Ġhas Ġshown Ġthat ĠBM UF Ġoutper forms Ġtraditional Ġmodel Ġaveraging Ġmethod , Ġand Ġit Ġis Ġutilized Ġat Ġthe Ġsynchronization Ġphase . ĠAfter Ġsynchron izing Ġwith ĠBM UF , ĠE MA Ġmethod Ġfurther Ġupdates Ġthe Ġmodel Ġin Ġnon - inter ference Ġway . ĠThe Ġtraining Ġsystem Ġis Ġdeployed Ġon Ġthe ĠMP I - based ĠH PC Ġcluster Ġwhere Ġ8 ĠGPUs . ĠEach ĠGPU Ġprocesses Ġnon - over lap Ġsubset Ġsplit Ġfrom Ġthe Ġentire Ġlarge Ġscale Ġdataset Ġin Ġparallel . </s> Local Ġmodels Ġfrom Ġdistributed Ġworkers Ġsynchron ize Ġwith Ġeach Ġother Ġin Ġdecentralized Ġway . ĠIn Ġthe Ġtraditional Ġmodel Ġaveraging Ġand ĠBM UF Ġmethod , Ġa Ġparameter Ġserver Ġwaits Ġfor Ġall Ġworkers Ġto Ġsend Ġtheir Ġlocal Ġmodels , Ġaggregate Ġthem , Ġand Ġsend Ġthe Ġupdated Ġmodel Ġto Ġall Ġworkers . ĠComputing Ġresource Ġof Ġworkers Ġis Ġwasted Ġuntil Ġaggregation Ġof Ġthe Ġparameter Ġserver Ġdone . ĠDec ent ral ized Ġmethod Ġmakes Ġfull Ġuse Ġof Ġcomputing Ġresource , Ġand Ġwe Ġemploy Ġthe ĠMP I - based ĠMesh ĠAll Red uce Ġmethod . ĠIt Ġis Ġmesh Ġtop ology Ġas Ġshown Ġin ĠFigure ĠFIG REF 12 Ġ. ĠThere Ġis Ġno Ġcentralized Ġparameter Ġserver , Ġand Ġpeer Ġto Ġpeer Ġcommunication Ġis Ġused Ġto Ġtransmit Ġlocal Ġmodels Ġbetween Ġworkers . ĠLocal Ġmodel ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 Ġ- th Ġworker Ġin ĠIN LINE FORM 2 Ġworkers Ġcluster Ġis Ġsplit Ġto ĠIN LINE FORM 3 Ġpieces ĠIN LINE FORM 4 Ġ, Ġand Ġsend Ġto Ġcorresponding Ġworker . ĠIn Ġthe Ġaggregation Ġphase , ĠIN LINE FORM 5 Ġ- th Ġworker Ġcomputed ĠIN LINE FORM 6 Ġsplits Ġof Ġmodel ĠIN LINE FORM 7 Ġand Ġsend Ġupdated Ġmodel ĠIN LINE FORM 8 Ġback Ġto Ġworkers . ĠAs Ġa Ġresult , Ġall Ġworkers Ġparticipate Ġin Ġaggregation Ġand Ġno Ġcomputing Ġresource Ġis Ġdissip ated . ĠIt Ġis Ġsignificant Ġto Ġpromote Ġtraining Ġefficiency , Ġwhen Ġthe Ġsize Ġof Ġneural Ġnetwork Ġmodel Ġis Ġtoo Ġlarge . ĠThe ĠE MA Ġmodel Ġis Ġalso Ġupdated Ġadditionally , Ġbut Ġnot Ġbroadcasting Ġit . </s> Results </s> In Ġorder Ġto Ġevaluate Ġour Ġsystem , Ġseveral Ġsets Ġof Ġexperiments Ġare Ġperformed . ĠThe ĠShen ma Ġtest Ġset Ġincluding Ġabout Ġ9000 Ġsamples Ġand ĠAm ap Ġtest Ġset Ġincluding Ġabout Ġ7000 Ġsamples Ġcontain Ġvarious Ġreal Ġworld Ġconditions . ĠIt Ġsim ulates Ġthe Ġmajority Ġof Ġuser Ġscenarios , Ġand Ġcan Ġwell Ġevaluates Ġthe Ġperformance Ġof Ġa Ġtrained Ġmodel . ĠFirstly , Ġwe Ġshow Ġthe Ġresults Ġof Ġmodels Ġtrained Ġwith ĠE MA Ġmethod . ĠSecondly , Ġfor Ġreal Ġworld Ġapplications , Ġvery Ġdeep ĠL ST M Ġis Ġdistilled Ġto Ġa Ġshallow Ġone , Ġso Ġas Ġfor Ġlower ĠR TF . ĠThe Ġmodel Ġof ĠAm ap Ġis Ġalso Ġneeded Ġto Ġtrain Ġfor Ġmap Ġand Ġnavigation Ġscenarios . ĠThe Ġperformance Ġof Ġtransfer Ġlearning Ġfrom ĠShen ma Ġvoice Ġsearch Ġto ĠAm ap Ġvoice Ġsearch Ġis Ġalso Ġpresented . </s> Layer - wise ĠTraining </s> In Ġlayer - wise Ġtraining , Ġthe Ġdeeper Ġmodel Ġlearns Ġboth Ġparameters Ġand Ġknowledge Ġfrom Ġthe Ġshall ower Ġmodel . ĠThe Ġdeeper Ġmodel Ġis Ġinitialized Ġby Ġthe Ġshall ower Ġone , Ġand Ġits Ġalignment Ġis Ġthe Ġcombination Ġof Ġhard Ġtarget Ġand Ġsoft Ġtarget Ġof Ġshall ower Ġone . ĠTwo Ġtargets Ġhave Ġthe Ġsame Ġweights Ġin Ġour Ġframework . ĠThe Ġteacher Ġmodel Ġis Ġtrained Ġwith ĠCE . ĠFor Ġeach Ġlayer - wise Ġtrained ĠCE Ġmodel , Ġcorresponding Ġs MB R Ġmodel Ġis Ġalso Ġtrained , Ġas Ġs MB R Ġcould Ġachieve Ġadditional Ġimprovement . ĠIn Ġour Ġframework , Ġ1000 Ġhours Ġdata Ġis Ġrandomly Ġselected Ġfrom Ġthe Ġtotal Ġdataset Ġfor Ġs MB R Ġtraining . ĠThere Ġis Ġno Ġobvious Ġperformance Ġenhancement Ġwhen Ġthe Ġsize Ġof Ġs MB R Ġtraining Ġdataset Ġincreases . </s> For Ġvery Ġdeep Ġun id irection al ĠL ST M Ġinitialized Ġwith ĠXavier Ġinitialization Ġalgorithm , Ġ6 - l ayers Ġmodel Ġconver ges Ġwell , Ġbut Ġthere Ġis Ġno Ġfurther Ġimprovement Ġwith Ġincreasing Ġthe Ġnumber Ġof Ġlayers . ĠTherefore , Ġthe Ġfirst Ġ6 Ġlayers Ġof Ġ7 - l ayers Ġmodel Ġis Ġinitialized Ġby Ġ6 - l ayers Ġmodel , Ġand Ġsoft Ġtarget Ġis Ġprovided Ġby Ġ6 - l ayers Ġmodel . ĠConsequently , Ġdeeper ĠL ST M Ġis Ġalso Ġtrained Ġin Ġthe Ġsame Ġway . ĠIt Ġshould Ġbe Ġnoticed Ġthat Ġthe Ġteacher Ġmodel Ġof Ġ9 - l ayers Ġmodel Ġis Ġthe Ġ8 - l ayers Ġmodel Ġtrained Ġby Ġs MB R , Ġwhile Ġthe Ġother Ġteacher Ġmodel Ġis ĠCE Ġmodel . ĠAs Ġshown Ġin ĠTable ĠT AB REF 15 Ġ, Ġthe Ġlayer - wise Ġtrained Ġmodels Ġalways Ġperforms Ġbetter Ġthan Ġthe Ġmodels Ġwith ĠXavier Ġinitialization , Ġas Ġthe Ġmodel Ġis Ġdeep . ĠTherefore , Ġfor Ġthe Ġlast Ġlayer Ġtraining , Ġwe Ġchoose Ġ8 - l ayers Ġs MB R Ġmodel Ġas Ġthe Ġteacher Ġmodel Ġinstead Ġof ĠCE Ġmodel . ĠA Ġcomparison Ġbetween Ġ6 - l ayers Ġand Ġ9 - l ayers Ġs MB R Ġmodels Ġshows Ġthat Ġ3 Ġadditional Ġlayers Ġof Ġlayer - wise Ġtraining Ġbrings Ġrelative Ġ12 . 6 % Ġdecreasing Ġof ĠC ER . ĠIt Ġis Ġalso Ġsignificant Ġthat Ġthe Ġaveraged ĠC ER Ġof Ġs MB R Ġmodels Ġwith Ġdifferent Ġlayers Ġdecreases Ġabsolute Ġ0 . 73 % Ġapproximately Ġcompared Ġwith ĠCE Ġmodels , Ġso Ġthe Ġimprovement Ġof Ġsequence Ġdiscrim in ative Ġlearning Ġis Ġpromising . </s> Transfer ĠLearning </s> 2 - l ayers Ġdistilled Ġmodel Ġof ĠShen ma Ġvoice Ġsearch Ġhas Ġshown Ġa Ġimpressive Ġperformance Ġon ĠShen ma ĠTest , Ġand Ġwe Ġcall Ġit ĠShen ma Ġmodel . ĠIt Ġis Ġtrained Ġfor Ġgeneric Ġsearch Ġscenario , Ġbut Ġit Ġhas Ġless Ġadaptation Ġfor Ġspecific Ġscenario Ġlike ĠAm ap Ġvoice Ġsearch . ĠTraining Ġwith Ġvery Ġlarge Ġdataset Ġusing ĠCE Ġloss Ġis Ġregarded Ġas Ġimprovement Ġof Ġframe Ġlevel Ġrecognition Ġaccuracy , Ġand Ġs MB R Ġwith Ġless Ġdataset Ġfurther Ġimproves Ġaccuracy Ġas Ġsequence Ġdiscrim in ative Ġtraining . ĠIf Ġrobust Ġmodel Ġof Ġgeneric Ġscenario Ġis Ġtrained , Ġthere Ġis Ġno Ġneed Ġto Ġtrain Ġa Ġmodel Ġwith Ġvery Ġlarge Ġdataset , Ġand Ġsequence Ġdiscrim in ative Ġtraining Ġwith Ġless Ġdataset Ġis Ġenough . ĠTherefore , Ġon Ġthe Ġbasis Ġof ĠShen ma Ġmodel , Ġit Ġis Ġsufficient Ġto Ġtrain Ġa Ġnew ĠAm ap Ġmodel Ġwith Ġsmall Ġdataset Ġusing Ġs MB R . ĠAs Ġshown Ġin ĠTable ĠT AB REF 18 Ġ, ĠShen ma Ġmodel Ġpresents Ġthe Ġworst Ġperformance Ġamong Ġthree Ġmethods , Ġsince Ġit Ġdoes Ġnot Ġtrained Ġfor ĠAm ap Ġscenario . Ġ2 - l ayers ĠShen ma Ġmodel Ġfurther Ġtrained Ġwith Ġs MB R Ġachieves Ġabout Ġ8 . 1 % Ġrelative Ġreduction , Ġcompared Ġwith Ġ2 - l ayers Ġregular - trained ĠAm ap Ġmodel . ĠBoth Ġtraining Ġs MB R Ġdatasets Ġcontain Ġthe Ġsame Ġ1000 Ġhours Ġdata . ĠAs Ġa Ġresult , Ġwith Ġthe ĠShen ma Ġmodel , Ġonly Ġabout Ġ14 % Ġdata Ġusage Ġachieves Ġlower ĠC ER , Ġand Ġit Ġleads Ġto Ġgreat Ġtime Ġand Ġcost Ġsaving Ġwith Ġless Ġlabeled Ġdata . ĠBesides , Ġtransfer Ġlearning Ġwith Ġs MB R Ġdoes Ġnot Ġuse Ġthe Ġalignment Ġfrom Ġthe ĠH MM - G MM Ġsystem , Ġso Ġit Ġalso Ġsaves Ġhuge Ġamount Ġof Ġtime . </s> Conclusion </s> We Ġhave Ġpresented Ġa Ġwhole Ġdeep Ġun id irection al ĠL ST M Ġparallel Ġtraining Ġsystem Ġfor ĠLV CS R . ĠThe Ġrecognition Ġperformance Ġimproves Ġwhen Ġthe Ġnetwork Ġgoes Ġdeep . ĠDist illation Ġmakes Ġit Ġpossible Ġthat Ġdeep ĠL ST M Ġmodel Ġtransfer Ġits Ġknowledge Ġto Ġshallow Ġmodel Ġwith Ġlittle Ġloss . ĠThe Ġmodel Ġcould Ġbe Ġdistilled Ġto Ġ2 - l ayers Ġmodel Ġwith Ġvery Ġlow ĠR TF , Ġso Ġthat Ġit Ġcan Ġdisplay Ġthe Ġimmediate Ġrecognition Ġresults . ĠAs Ġa Ġresult , Ġits ĠC ER Ġdecrease Ġrelatively Ġ14 %, Ġcompared Ġwith Ġthe Ġ2 - l ayers Ġregular Ġtrained Ġmodel . ĠIn Ġaddition , Ġtransfer Ġlearning Ġwith Ġs MB R Ġis Ġalso Ġproposed . ĠIf Ġa Ġgreat Ġmodel Ġhas Ġwell Ġtrained Ġfrom Ġgeneric Ġscenario , Ġonly Ġ14 % Ġof Ġthe Ġsize Ġof Ġtraining Ġdataset Ġis Ġneeded Ġto Ġtrain Ġa Ġmore Ġaccuracy Ġacoustic Ġmodel Ġfor Ġspecific Ġscenario . ĠOur Ġfuture Ġwork Ġincludes Ġ1 ) Ġfinding Ġmore Ġeffective Ġmethods Ġto Ġreduce Ġthe ĠC ER Ġby Ġincreasing Ġthe Ġnumber Ġof Ġlayers ; Ġ2 ) Ġapplying Ġthis Ġtraining Ġframework Ġto ĠConnection ist ĠTem poral ĠClassification Ġ( CT C ) Ġand Ġattention - based Ġneural Ġnetworks .
Introduction Recently, deep neural network has been widely employed in various recognition tasks. Increasing the depth of neural network is a effective way to improve the performance, and convolutional neural network (CNN) has benefited from it in visual recognition task BIBREF0 . Deeper long short-term memory (LSTM) recurrent neural networks (RNNs) are also applied in large vocabulary continuous speech recognition (LVCSR) task, because LSTM networks have shown better performance than Fully-connected feed-forward deep neural network BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . Training neural network becomes more challenge when it goes deep. A conceptual tool called linear classifier probe is introduced to better understand the dynamics inside a neural network BIBREF5 . The discriminating features of linear classifier is the hidden units of a intermediate layer. For deep neural networks, it is observed that deeper layer's accuracy is lower than that of shallower layers. Therefore, the tool shows the difficulty of deep neural model training visually. Layer-wise pre-training is a successful method to train very deep neural networks BIBREF6 . The convergence becomes harder with increasing the number of layers, even though the model is initialized with Xavier or its variants BIBREF7 , BIBREF8 . But the deeper network which is initialized with a shallower trained network could converge well. The size of LVCSR training dataset goes larger and training with only one GPU becomes high time consumption inevitably. Therefore, parallel training with multi-GPUs is more suitable for LVCSR system. Mini-batch based stochastic gradient descent (SGD) is the most popular method in neural network training procedure. Asynchronous SGD is a successful effort for parallel training based on it BIBREF9 , BIBREF10 . It can many times speed up the training time without decreasing the accuracy. Besides, synchronous SGD is another effective effort, where the parameter server waits for every works to finish their computation and sent their local models to it, and then it sends updated model back to all workers BIBREF11 . Synchronous SGD converges well in parallel training with data parallelism, and is also easy to implement. In order to further improve the performance of deep neural network with parallel training, several methods are proposed. Model averaging method achieves linear speedup, as the final model is averaged from all parameters of local models in different workers BIBREF12 , BIBREF13 , but the accuracy decreases compared with single GPU training. Moreover, blockwise model-updating filter (BMUF) provides another almost linear speedup approach with multi-GPUs on the basis of model averaging. It can achieve improvement or no-degradation of recognition performance compared with mini-batch SGD on single GPU BIBREF14 . Moving averaged (MA) approaches are also proposed for parallel training. It is demonstrated that the moving average of the parameters obtained by SGD performs as well as the parameters that minimize the empirical cost, and moving average parameters can be used as the estimator of them, if the size of training data is large enough BIBREF15 . One pass learning is then proposed, which is the combination of learning rate schedule and averaged SGD using moving average BIBREF16 . Exponential moving average (EMA) is proposed as a non-interference method BIBREF17 . EMA model is not broadcasted to workers to update their local models, and it is applied as the final model of entire training process. EMA method is utilized with model averaging and BMUF to further decrease the character error rate (CER). It is also easy to implement in existing parallel training systems. Frame stacking can also speed up the training time BIBREF18 . The super frame is stacked by several regular frames, and it contains the information of them. Thus, the network can see multiple frames at a time, as the super frame is new input. Frame stacking can also lead to faster decoding. For streaming voice search service, it needs to display intermediate recognition results while users are still speaking. As a result, the system needs to fulfill high real-time requirement, and we prefer unidirectional LSTM network rather than bidirectional one. High real-time requirement means low real time factor (RTF), but the RTF of deep LSTM model is higher inevitably. The dilemma of recognition accuracy and real-time requirement is an obstacle to the employment of deep LSTM network. Deep model outperforms because it contains more knowledge, but it is also cumbersome. As a result, the knowledge of deep model can be distilled to a shallow model BIBREF19 . It provided a effective way to employ the deep model to the real-time system. In this paper, we explore a entire deep LSTM RNN training framework, and employ it to real-time application. The deep learning systems benefit highly from a large quantity of labeled training data. Our first and basic speech recognition system is trained on 17000 hours of Shenma voice search dataset. It is a generic dataset sampled from diverse aspects of search queries. The requirement of speech recognition system also addressed by specific scenario, such as map and navigation task. The labeled dataset is too expensive, and training a new model with new large dataset from the beginning costs lots of time. Thus, it is natural to think of transferring the knowledge from basic model to new scenario's model. Transfer learning expends less data and less training time than full training. In this paper, we also introduce a novel transfer learning strategy with segmental Minimum Bayes-Risk (sMBR). As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data. Our deep LSTM training framework for LVCSR is presented in Section 2. Section 3 describes how the very deep models does apply in real world applications, and how to transfer the model to another task. The framework is analyzed and discussed in Section 4, and followed by the conclusion in Section 5. Layer-wise Training with Soft Target and Hard Target Gradient-based optimization of deep LSTM network with random initialization get stuck in poor solution easily. Xavier initialization can partially solve this problem BIBREF7 , so this method is the regular initialization method of all training procedure. However, it does not work well when it is utilized to initialize very deep model directly, because of vanishing or exploding gradients. Instead, layer-wise pre-training method is a effective way to train the weights of very deep architecture BIBREF6 , BIBREF20 . In layer-wise pre-training procedure, a one-layer LSTM model is firstly trained with normalized initialization. Sequentially, two-layers LSTM model's first layer is initialized by trained one-layer model, and its second layer is regularly initialized. In this way, a deep architecture is layer-by-layer trained, and it can converge well. In conventional layer-wise pre-training, only parameters of shallower network are transfered to deeper one, and the learning targets are still the alignments generated by HMM-GMM system. The targets are vectors that only one state's probability is one, and the others' are zeros. They are known as hard targets, and they carry limited knowledge as only one state is active. In contrast, the knowledge of shallower network should be also transfered to deeper one. It is obtained by the softmax layer of existing model typically, so each state has a probability rather than only zero or one, and called as soft target. As a result, the deeper network which is student network learns the parameters and knowledge from shallower one which is called teacher network. When training the student network from the teacher network, the final alignment is the combination of hard target and soft target in our layer-wise training phase. The final alignment provides various knowledge which transfered from teacher network and extracted from true labels. If only soft target is learned, student network perform no better than teacher network, but it could outperform teacher network as it also learns true labels. The deeper network spends less time to getting the same level of original network than the network trained from the beginning, as a period of low performance is skipped. Therefore, training with hard and soft target is a time saving method. For large training dataset, training with the whole dataset still spends too much time. A network firstly trained with only a small part of dataset could go deeper as well, and so the training time reducing rapidly. When the network is deep enough, it then trained on the entire dataset to get further improvement. There is no gap of accuracy between these two approaches, but latter one saves much time. Differential Saturation Check The objects of conventional saturation check are gradients and the cell activations BIBREF4 . Gradients are clipped to range [-5, 5], while the cell activations clipped to range [-50, 50]. Apart from them, the differentials of recurrent layers is also limited. If the differentials go beyond the range, corresponding back propagation is skipped, while if the gradients and cell activations go beyond the bound, values are set as the boundary values. The differentials which are too large or too small will lead to the gradients easily vanishing, and it demonstrates the failure of this propagation. As a result, the parameters are not updated, and next propagation . Sequence Discriminative Training Cross-entropy (CE) is widely used in speech recognition training system as a frame-wise discriminative training criterion. However, it is not well suited to speech recognition, because speech recognition training is a sequential learning problem. In contrast, sequence discriminative training criterion has shown to further improve performance of neural network first trained with cross-entropy BIBREF21 , BIBREF22 , BIBREF23 . We choose state-level minimum bayes risk (sMBR) BIBREF21 among a number of sequence discriminative criterion is proposed, such as maximum mutual information (MMI) BIBREF24 and minimum phone error (MPE) BIBREF25 . MPE and sMBR are designed to minimize the expected error of different granularity of labels, while CE aims to minimizes expected frame error, and MMI aims to minimizes expected sentence error. State-level information is focused on by sMBR. a frame-level accurate model is firstly trained by CE loss function, and then sMBR loss function is utilized for further training to get sequence-level accuracy. Only a part of training dataset is needed in sMBR training phase on the basis of whole dataset CE training. Parallel Training It is demonstrated that training with larger dataset can improve recognition accuracy. However, larger dataset means more training samples and more model parameters. Therefore, parallel training with multiple GPUs is essential, and it makes use of data parallelism BIBREF9 . The entire training data is partitioned into several split without overlapping and they are distributed to different GPUs. Each GPU trains with one split of training dataset locally. All GPUs synchronize their local models with model average method after a mini-batch optimization BIBREF12 , BIBREF13 . Model average method achieves linear speedup in training phase, but the recognition accuracy decreases compared with single GPU training. Block-wise model updating filter (BMUF) is another successful effort in parallel training with linear speedup as well. It can achieve no-degradation of recognition accuracy with multi-GPUs BIBREF14 . In the model average method, aggregated model INLINEFORM0 is computed and broadcasted to GPUs. On the basis of it, BMUF proposed a novel model updating strategy: INLINEFORM1 INLINEFORM2 Where INLINEFORM0 denotes model update, and INLINEFORM1 is the global-model update. There are two parameters in BMUF, block momentum INLINEFORM2 , and block learning rate INLINEFORM3 . Then, the global model is updated as INLINEFORM4 Consequently, INLINEFORM0 is broadcasted to all GPUs to initial their local models, instead of INLINEFORM1 in model average method. Averaged SGD is proposed to further accelerate the convergence speed of SGD. Averaged SGD leverages the moving average (MA) INLINEFORM0 as the estimator of INLINEFORM1 BIBREF15 : INLINEFORM2 Where INLINEFORM0 is computed by model averaging or BMUF. It is shown that INLINEFORM1 can well converge to INLINEFORM2 , with the large enough training dataset in single GPU training. It can be considered as a non-interference strategy that INLINEFORM3 does not participate the main optimization process, and only takes effect after the end of entire optimization. However, for the parallel training implementation, each INLINEFORM4 is computed by model averaging and BMUF with multiple models, and moving average model INLINEFORM5 does not well converge, compared with single GPU training. Model averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially no-degradation implementation of BMUF. But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as INLINEFORM0 The weight of each INLINEFORM0 is equal in moving average method regardless the effect of temporal order. But INLINEFORM1 closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final INLINEFORM2 . As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero. After moving average based methods, the EMA parameters are updated recursively as INLINEFORM3 Here INLINEFORM0 represents the degree of weight decrease, and called exponential updating rate. EMA is also a non-interference training strategy that is implemented easily, as the updated model is not broadcasted. Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly. Deployment There is a high real time requirement in real world application, especially in online voice search system. Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking. Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition. Distillation It is demonstrated that deep neural network architecture can achieve improvement in LVCSR. However, it also leads to much more computation and higher RTF, so that the recognition result can not be displayed in real time. It should be noted that deeper neural network contains more knowledge, but it is also cumbersome. the knowledge is key to improve the performance. If it can be transfered from cumbersome model to a small model, the recognition ability can also be transfered to the small model. Knowledge transferring to small model is called distillation BIBREF19 . The small model can perform as well as cumbersome model, after distilling. It provide a way to utilize high-performance but high RTF model in real time system. The class probability produced by the cumbersome model is regarded as soft target, and the generalization ability of cumbersome model is transfered to small model with it. Distillation is model's knowledge transferring approach, so there is no need to use the hard target, which is different with the layer-wise training method. 9-layers unidirectional LSTM model achieves outstanding performance, but meanwhile it is too computationally expensive to allow deployment in real time recognition system. In order to ensure real-time of the system, the number of layers needs to be reduced. The shallower network can learn the knowledge of deeper network with distillation. It is found that RTF of 2-layers network is acceptable, so the knowledge is distilled from 9-layers well-trained model to 2-layers model. Table TABREF16 shows that distillation from 9-layers to 2-layers brings RTF decrease of relative 53%, while CER only increases 5%. The knowledge of deep network is almost transfered with distillation, Distillation brings promising RTF reduction, but only little knowledge of deep network is lost. Moreover, CER of 2-layers distilled LSTM decreases relative 14%, compared with 2-layers regular-trained LSTM. Transfer Learning with sMBR For a certain specific scenario, the model trained with the data recorded from it has better adaptation than the model trained with generic scenario. But it spends too much time training a model from the beginning, if there is a well-trained model for generic scenarios. Moreover, labeling a large quantity of training data in new scenario is both costly and time consuming. If a model transfer trained with smaller dataset can obtained the similar recognition accuracy compared with the model directly trained with larger dataset, it is no doubt that transfer learning is more practical. Since specific scenario is a subset of generic scenario, some knowledge can be shared between them. Besides, generic scenario consists of various conditions, so its model has greater robustness. As a result, not only shared knowledge but also robustness can be transfered from the model of generic scenario to the model of specific one. As the model well trained from generic scenario achieves good performance in frame level classification, sequence discriminative training is required to adapt new model to specific scenario additionally. Moreover, it does not need alignment from HMM-GMM system, and it also saves amount of time to prepare alignment. Training Data A large quantity of labeled data is needed for training a more accurate acoustic model. We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China. The dataset is created from anonymous online users' search queries in Mandarin, and all audio file's sampling rate is 16kHz, recorded by mobile phones. This dataset consists of many different conditions, such as diverse noise even low signal-to-noise, babble, dialects, accents, hesitation and so on. In the Amap, which is one of the most popular web mapping and navigation services in China, users can search locations and navigate to locations they want though voice search. To present the performance of transfer learning with sequence discriminative training, the model trained from Shenma voice search which is greneric scenario transfer its knowledge to the model of Amap voice search. 7300 hours labeled data is collected in the similar way of Shenma voice search data collection. Two dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10 . The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously. The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test. Experimental setup LSTM RNNs outperform conventional RNNs for speech recognition system, especially deep LSTM RNNs, because of its long-range dependencies more accurately for temporal sequence conditions BIBREF26 , BIBREF23 . Shenma and Amap voice search is a streaming service that intermediate recognition results displayed while users are still speaking. So as for online recognition in real time, we prefer unidirectional LSTM model rather than bidirectional one. Thus, the training system is unidirectional LSTM-based. A 26-dimensional filter bank and 2-dimensional pitch feature is extracted for each frame, and is concatenated with first and second order difference as the final input of the network. The super frame are stacked by 3 frames without overlapping. The architecture we trained consists of two LSTM layers with sigmoid activation function, followed by a full-connection layer. The out layer is a softmax layer with 11088 hidden markov model (HMM) tied-states as output classes, the loss function is cross-entropy (CE). The performance metric of the system in Mandarin is reported with character error rate (CER). The alignment of frame-level ground truth is obtained by GMM-HMM system. Mini-batched SGD is utilized with momentum trick and the network is trained for a total of 4 epochs. The block learning rate and block momentum of BMUF are set as 1 and 0.9. 5-gram language model is leveraged in decoder, and the vocabulary size is as large as 760000. Differentials of recurrent layers is limited to range [-10000,10000], while gradients are clipped to range [-5, 5] and cell activations clipped to range [-50, 50]. After training with CE loss, sMBR loss is employed to further improve the performance. It has shown that BMUF outperforms traditional model averaging method, and it is utilized at the synchronization phase. After synchronizing with BMUF, EMA method further updates the model in non-interference way. The training system is deployed on the MPI-based HPC cluster where 8 GPUs. Each GPU processes non-overlap subset split from the entire large scale dataset in parallel. Local models from distributed workers synchronize with each other in decentralized way. In the traditional model averaging and BMUF method, a parameter server waits for all workers to send their local models, aggregate them, and send the updated model to all workers. Computing resource of workers is wasted until aggregation of the parameter server done. Decentralized method makes full use of computing resource, and we employ the MPI-based Mesh AllReduce method. It is mesh topology as shown in Figure FIGREF12 . There is no centralized parameter server, and peer to peer communication is used to transmit local models between workers. Local model INLINEFORM0 of INLINEFORM1 -th worker in INLINEFORM2 workers cluster is split to INLINEFORM3 pieces INLINEFORM4 , and send to corresponding worker. In the aggregation phase, INLINEFORM5 -th worker computed INLINEFORM6 splits of model INLINEFORM7 and send updated model INLINEFORM8 back to workers. As a result, all workers participate in aggregation and no computing resource is dissipated. It is significant to promote training efficiency, when the size of neural network model is too large. The EMA model is also updated additionally, but not broadcasting it. Results In order to evaluate our system, several sets of experiments are performed. The Shenma test set including about 9000 samples and Amap test set including about 7000 samples contain various real world conditions. It simulates the majority of user scenarios, and can well evaluates the performance of a trained model. Firstly, we show the results of models trained with EMA method. Secondly, for real world applications, very deep LSTM is distilled to a shallow one, so as for lower RTF. The model of Amap is also needed to train for map and navigation scenarios. The performance of transfer learning from Shenma voice search to Amap voice search is also presented. Layer-wise Training In layer-wise training, the deeper model learns both parameters and knowledge from the shallower model. The deeper model is initialized by the shallower one, and its alignment is the combination of hard target and soft target of shallower one. Two targets have the same weights in our framework. The teacher model is trained with CE. For each layer-wise trained CE model, corresponding sMBR model is also trained, as sMBR could achieve additional improvement. In our framework, 1000 hours data is randomly selected from the total dataset for sMBR training. There is no obvious performance enhancement when the size of sMBR training dataset increases. For very deep unidirectional LSTM initialized with Xavier initialization algorithm, 6-layers model converges well, but there is no further improvement with increasing the number of layers. Therefore, the first 6 layers of 7-layers model is initialized by 6-layers model, and soft target is provided by 6-layers model. Consequently, deeper LSTM is also trained in the same way. It should be noticed that the teacher model of 9-layers model is the 8-layers model trained by sMBR, while the other teacher model is CE model. As shown in Table TABREF15 , the layer-wise trained models always performs better than the models with Xavier initialization, as the model is deep. Therefore, for the last layer training, we choose 8-layers sMBR model as the teacher model instead of CE model. A comparison between 6-layers and 9-layers sMBR models shows that 3 additional layers of layer-wise training brings relative 12.6% decreasing of CER. It is also significant that the averaged CER of sMBR models with different layers decreases absolute 0.73% approximately compared with CE models, so the improvement of sequence discriminative learning is promising. Transfer Learning 2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model. It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search. Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training. If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough. Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR. As shown in Table TABREF18 , Shenma model presents the worst performance among three methods, since it does not trained for Amap scenario. 2-layers Shenma model further trained with sMBR achieves about 8.1% relative reduction, compared with 2-layers regular-trained Amap model. Both training sMBR datasets contain the same 1000 hours data. As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data. Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time. Conclusion We have presented a whole deep unidirectional LSTM parallel training system for LVCSR. The recognition performance improves when the network goes deep. Distillation makes it possible that deep LSTM model transfer its knowledge to shallow model with little loss. The model could be distilled to 2-layers model with very low RTF, so that it can display the immediate recognition results. As a result, its CER decrease relatively 14%, compared with the 2-layers regular trained model. In addition, transfer learning with sMBR is also proposed. If a great model has well trained from generic scenario, only 14% of the size of training dataset is needed to train a more accuracy acoustic model for specific scenario. Our future work includes 1) finding more effective methods to reduce the CER by increasing the number of layers; 2) applying this training framework to Connectionist Temporal Classification (CTC) and attention-based neural networks.
what was their character error rate?
1bb7eb5c3d029d95d1abf9f2892c1ec7b6eef306
what Ġwas Ġtheir Ġcharacter Ġerror Ġrate ?
2 . 49 % Ġfor Ġ Ġlayer - wise Ġtraining , Ġ2 . 63 % Ġfor Ġdist illation , Ġ6 . 26 % Ġfor Ġtransfer Ġlearning .
<s> what Ġwas Ġtheir Ġcharacter Ġerror Ġrate ? </s> Introduction </s> Recently , Ġdeep Ġneural Ġnetwork Ġhas Ġbeen Ġwidely Ġemployed Ġin Ġvarious Ġrecognition Ġtasks . ĠIncreasing Ġthe Ġdepth Ġof Ġneural Ġnetwork Ġis Ġa Ġeffective Ġway Ġto Ġimprove Ġthe Ġperformance , Ġand Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġhas Ġbenefited Ġfrom Ġit Ġin Ġvisual Ġrecognition Ġtask ĠB IB REF 0 Ġ. ĠDe eper Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ) Ġare Ġalso Ġapplied Ġin Ġlarge Ġvocabulary Ġcontinuous Ġspeech Ġrecognition Ġ( LV CS R ) Ġtask , Ġbecause ĠL ST M Ġnetworks Ġhave Ġshown Ġbetter Ġperformance Ġthan ĠFully - connected Ġfeed - forward Ġdeep Ġneural Ġnetwork ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. </s> Training Ġneural Ġnetwork Ġbecomes Ġmore Ġchallenge Ġwhen Ġit Ġgoes Ġdeep . ĠA Ġconceptual Ġtool Ġcalled Ġlinear Ġclass ifier Ġprobe Ġis Ġintroduced Ġto Ġbetter Ġunderstand Ġthe Ġdynamics Ġinside Ġa Ġneural Ġnetwork ĠB IB REF 5 Ġ. ĠThe Ġdiscriminating Ġfeatures Ġof Ġlinear Ġclass ifier Ġis Ġthe Ġhidden Ġunits Ġof Ġa Ġintermediate Ġlayer . ĠFor Ġdeep Ġneural Ġnetworks , Ġit Ġis Ġobserved Ġthat Ġdeeper Ġlayer 's Ġaccuracy Ġis Ġlower Ġthan Ġthat Ġof Ġshall ower Ġlayers . ĠTherefore , Ġthe Ġtool Ġshows Ġthe Ġdifficulty Ġof Ġdeep Ġneural Ġmodel Ġtraining Ġvisually . </s> Layer - wise Ġpre - training Ġis Ġa Ġsuccessful Ġmethod Ġto Ġtrain Ġvery Ġdeep Ġneural Ġnetworks ĠB IB REF 6 Ġ. ĠThe Ġconvergence Ġbecomes Ġharder Ġwith Ġincreasing Ġthe Ġnumber Ġof Ġlayers , Ġeven Ġthough Ġthe Ġmodel Ġis Ġinitialized Ġwith ĠXavier Ġor Ġits Ġvariants ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠBut Ġthe Ġdeeper Ġnetwork Ġwhich Ġis Ġinitialized Ġwith Ġa Ġshall ower Ġtrained Ġnetwork Ġcould Ġconverge Ġwell . </s> The Ġsize Ġof ĠLV CS R Ġtraining Ġdataset Ġgoes Ġlarger Ġand Ġtraining Ġwith Ġonly Ġone ĠGPU Ġbecomes Ġhigh Ġtime Ġconsumption Ġinevitably . ĠTherefore , Ġparallel Ġtraining Ġwith Ġmulti - GPU s Ġis Ġmore Ġsuitable Ġfor ĠLV CS R Ġsystem . ĠMini - batch Ġbased Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) Ġis Ġthe Ġmost Ġpopular Ġmethod Ġin Ġneural Ġnetwork Ġtraining Ġprocedure . ĠAs ynchronous ĠSG D Ġis Ġa Ġsuccessful Ġeffort Ġfor Ġparallel Ġtraining Ġbased Ġon Ġit ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠIt Ġcan Ġmany Ġtimes Ġspeed Ġup Ġthe Ġtraining Ġtime Ġwithout Ġdecreasing Ġthe Ġaccuracy . ĠBesides , Ġsynchron ous ĠSG D Ġis Ġanother Ġeffective Ġeffort , Ġwhere Ġthe Ġparameter Ġserver Ġwaits Ġfor Ġevery Ġworks Ġto Ġfinish Ġtheir Ġcomputation Ġand Ġsent Ġtheir Ġlocal Ġmodels Ġto Ġit , Ġand Ġthen Ġit Ġsends Ġupdated Ġmodel Ġback Ġto Ġall Ġworkers ĠB IB REF 11 Ġ. ĠSyn chron ous ĠSG D Ġconver ges Ġwell Ġin Ġparallel Ġtraining Ġwith Ġdata Ġparallel ism , Ġand Ġis Ġalso Ġeasy Ġto Ġimplement . </s> In Ġorder Ġto Ġfurther Ġimprove Ġthe Ġperformance Ġof Ġdeep Ġneural Ġnetwork Ġwith Ġparallel Ġtraining , Ġseveral Ġmethods Ġare Ġproposed . ĠModel Ġaveraging Ġmethod Ġachieves Ġlinear Ġspeed up , Ġas Ġthe Ġfinal Ġmodel Ġis Ġaveraged Ġfrom Ġall Ġparameters Ġof Ġlocal Ġmodels Ġin Ġdifferent Ġworkers ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, Ġbut Ġthe Ġaccuracy Ġdecreases Ġcompared Ġwith Ġsingle ĠGPU Ġtraining . ĠMoreover , Ġblock wise Ġmodel - up dating Ġfilter Ġ( BM UF ) Ġprovides Ġanother Ġalmost Ġlinear Ġspeed up Ġapproach Ġwith Ġmulti - GPU s Ġon Ġthe Ġbasis Ġof Ġmodel Ġaveraging . ĠIt Ġcan Ġachieve Ġimprovement Ġor Ġno - deg rad ation Ġof Ġrecognition Ġperformance Ġcompared Ġwith Ġmini - batch ĠSG D Ġon Ġsingle ĠGPU ĠB IB REF 14 Ġ. </s> Moving Ġaveraged Ġ( MA ) Ġapproaches Ġare Ġalso Ġproposed Ġfor Ġparallel Ġtraining . ĠIt Ġis Ġdemonstrated Ġthat Ġthe Ġmoving Ġaverage Ġof Ġthe Ġparameters Ġobtained Ġby ĠSG D Ġperforms Ġas Ġwell Ġas Ġthe Ġparameters Ġthat Ġminimize Ġthe Ġempirical Ġcost , Ġand Ġmoving Ġaverage Ġparameters Ġcan Ġbe Ġused Ġas Ġthe Ġestim ator Ġof Ġthem , Ġif Ġthe Ġsize Ġof Ġtraining Ġdata Ġis Ġlarge Ġenough ĠB IB REF 15 Ġ. ĠOne Ġpass Ġlearning Ġis Ġthen Ġproposed , Ġwhich Ġis Ġthe Ġcombination Ġof Ġlearning Ġrate Ġschedule Ġand Ġaveraged ĠSG D Ġusing Ġmoving Ġaverage ĠB IB REF 16 Ġ. ĠExp onential Ġmoving Ġaverage Ġ( EMA ) Ġis Ġproposed Ġas Ġa Ġnon - inter ference Ġmethod ĠB IB REF 17 Ġ. ĠE MA Ġmodel Ġis Ġnot Ġbroadcast ed Ġto Ġworkers Ġto Ġupdate Ġtheir Ġlocal Ġmodels , Ġand Ġit Ġis Ġapplied Ġas Ġthe Ġfinal Ġmodel Ġof Ġentire Ġtraining Ġprocess . ĠE MA Ġmethod Ġis Ġutilized Ġwith Ġmodel Ġaveraging Ġand ĠBM UF Ġto Ġfurther Ġdecrease Ġthe Ġcharacter Ġerror Ġrate Ġ( C ER ). ĠIt Ġis Ġalso Ġeasy Ġto Ġimplement Ġin Ġexisting Ġparallel Ġtraining Ġsystems . </s> Frame Ġstacking Ġcan Ġalso Ġspeed Ġup Ġthe Ġtraining Ġtime ĠB IB REF 18 Ġ. ĠThe Ġsuper Ġframe Ġis Ġstacked Ġby Ġseveral Ġregular Ġframes , Ġand Ġit Ġcontains Ġthe Ġinformation Ġof Ġthem . ĠThus , Ġthe Ġnetwork Ġcan Ġsee Ġmultiple Ġframes Ġat Ġa Ġtime , Ġas Ġthe Ġsuper Ġframe Ġis Ġnew Ġinput . ĠFrame Ġstacking Ġcan Ġalso Ġlead Ġto Ġfaster Ġdecoding . </s> For Ġstreaming Ġvoice Ġsearch Ġservice , Ġit Ġneeds Ġto Ġdisplay Ġintermediate Ġrecognition Ġresults Ġwhile Ġusers Ġare Ġstill Ġspeaking . ĠAs Ġa Ġresult , Ġthe Ġsystem Ġneeds Ġto Ġfulfill Ġhigh Ġreal - time Ġrequirement , Ġand Ġwe Ġprefer Ġun id irection al ĠL ST M Ġnetwork Ġrather Ġthan Ġbid irection al Ġone . ĠHigh Ġreal - time Ġrequirement Ġmeans Ġlow Ġreal Ġtime Ġfactor Ġ( R TF ), Ġbut Ġthe ĠR TF Ġof Ġdeep ĠL ST M Ġmodel Ġis Ġhigher Ġinevitably . ĠThe Ġdilemma Ġof Ġrecognition Ġaccuracy Ġand Ġreal - time Ġrequirement Ġis Ġan Ġobstacle Ġto Ġthe Ġemployment Ġof Ġdeep ĠL ST M Ġnetwork . ĠDeep Ġmodel Ġoutper forms Ġbecause Ġit Ġcontains Ġmore Ġknowledge , Ġbut Ġit Ġis Ġalso Ġcumbersome . ĠAs Ġa Ġresult , Ġthe Ġknowledge Ġof Ġdeep Ġmodel Ġcan Ġbe Ġdistilled Ġto Ġa Ġshallow Ġmodel ĠB IB REF 19 Ġ. ĠIt Ġprovided Ġa Ġeffective Ġway Ġto Ġemploy Ġthe Ġdeep Ġmodel Ġto Ġthe Ġreal - time Ġsystem . </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġa Ġentire Ġdeep ĠL ST M ĠR NN Ġtraining Ġframework , Ġand Ġemploy Ġit Ġto Ġreal - time Ġapplication . ĠThe Ġdeep Ġlearning Ġsystems Ġbenefit Ġhighly Ġfrom Ġa Ġlarge Ġquantity Ġof Ġlabeled Ġtraining Ġdata . ĠOur Ġfirst Ġand Ġbasic Ġspeech Ġrecognition Ġsystem Ġis Ġtrained Ġon Ġ17 000 Ġhours Ġof ĠShen ma Ġvoice Ġsearch Ġdataset . ĠIt Ġis Ġa Ġgeneric Ġdataset Ġsampled Ġfrom Ġdiverse Ġaspects Ġof Ġsearch Ġqueries . ĠThe Ġrequirement Ġof Ġspeech Ġrecognition Ġsystem Ġalso Ġaddressed Ġby Ġspecific Ġscenario , Ġsuch Ġas Ġmap Ġand Ġnavigation Ġtask . ĠThe Ġlabeled Ġdataset Ġis Ġtoo Ġexpensive , Ġand Ġtraining Ġa Ġnew Ġmodel Ġwith Ġnew Ġlarge Ġdataset Ġfrom Ġthe Ġbeginning Ġcosts Ġlots Ġof Ġtime . ĠThus , Ġit Ġis Ġnatural Ġto Ġthink Ġof Ġtransferring Ġthe Ġknowledge Ġfrom Ġbasic Ġmodel Ġto Ġnew Ġscenario 's Ġmodel . ĠTransfer Ġlearning Ġexp ends Ġless Ġdata Ġand Ġless Ġtraining Ġtime Ġthan Ġfull Ġtraining . ĠIn Ġthis Ġpaper , Ġwe Ġalso Ġintroduce Ġa Ġnovel Ġtransfer Ġlearning Ġstrategy Ġwith Ġsegment al ĠMinimum ĠBay es - R isk Ġ( s MB R ). ĠAs Ġa Ġresult , Ġtransfer Ġtraining Ġwith Ġonly Ġ1000 Ġhours Ġdata Ġcan Ġmatch Ġequivalent Ġperformance Ġfor Ġfull Ġtraining Ġwith Ġ7 300 Ġhours Ġdata . </s> Our Ġdeep ĠL ST M Ġtraining Ġframework Ġfor ĠLV CS R Ġis Ġpresented Ġin ĠSection Ġ2 . ĠSection Ġ3 Ġdescribes Ġhow Ġthe Ġvery Ġdeep Ġmodels Ġdoes Ġapply Ġin Ġreal Ġworld Ġapplications , Ġand Ġhow Ġto Ġtransfer Ġthe Ġmodel Ġto Ġanother Ġtask . ĠThe Ġframework Ġis Ġanalyzed Ġand Ġdiscussed Ġin ĠSection Ġ4 , Ġand Ġfollowed Ġby Ġthe Ġconclusion Ġin ĠSection Ġ5 . </s> Layer - wise ĠTraining Ġwith ĠSoft ĠTarget Ġand ĠHard ĠTarget </s> Grad ient - based Ġoptimization Ġof Ġdeep ĠL ST M Ġnetwork Ġwith Ġrandom Ġinitialization Ġget Ġstuck Ġin Ġpoor Ġsolution Ġeasily . ĠXavier Ġinitialization Ġcan Ġpartially Ġsolve Ġthis Ġproblem ĠB IB REF 7 Ġ, Ġso Ġthis Ġmethod Ġis Ġthe Ġregular Ġinitialization Ġmethod Ġof Ġall Ġtraining Ġprocedure . ĠHowever , Ġit Ġdoes Ġnot Ġwork Ġwell Ġwhen Ġit Ġis Ġutilized Ġto Ġinitialize Ġvery Ġdeep Ġmodel Ġdirectly , Ġbecause Ġof Ġvanishing Ġor Ġexploding Ġgrad ients . ĠInstead , Ġlayer - wise Ġpre - training Ġmethod Ġis Ġa Ġeffective Ġway Ġto Ġtrain Ġthe Ġweights Ġof Ġvery Ġdeep Ġarchitecture ĠB IB REF 6 Ġ, ĠB IB REF 20 Ġ. ĠIn Ġlayer - wise Ġpre - training Ġprocedure , Ġa Ġone - layer ĠL ST M Ġmodel Ġis Ġfirst ly Ġtrained Ġwith Ġnormalized Ġinitialization . ĠSequ entially , Ġtwo - l ayers ĠL ST M Ġmodel 's Ġfirst Ġlayer Ġis Ġinitialized Ġby Ġtrained Ġone - layer Ġmodel , Ġand Ġits Ġsecond Ġlayer Ġis Ġregularly Ġinitialized . ĠIn Ġthis Ġway , Ġa Ġdeep Ġarchitecture Ġis Ġlayer - by - layer Ġtrained , Ġand Ġit Ġcan Ġconverge Ġwell . </s> In Ġconventional Ġlayer - wise Ġpre - training , Ġonly Ġparameters Ġof Ġshall ower Ġnetwork Ġare Ġtransf ered Ġto Ġdeeper Ġone , Ġand Ġthe Ġlearning Ġtargets Ġare Ġstill Ġthe Ġalign ments Ġgenerated Ġby ĠH MM - G MM Ġsystem . ĠThe Ġtargets Ġare Ġvectors Ġthat Ġonly Ġone Ġstate 's Ġprobability Ġis Ġone , Ġand Ġthe Ġothers ' Ġare Ġz eros . ĠThey Ġare Ġknown Ġas Ġhard Ġtargets , Ġand Ġthey Ġcarry Ġlimited Ġknowledge Ġas Ġonly Ġone Ġstate Ġis Ġactive . ĠIn Ġcontrast , Ġthe Ġknowledge Ġof Ġshall ower Ġnetwork Ġshould Ġbe Ġalso Ġtransf ered Ġto Ġdeeper Ġone . ĠIt Ġis Ġobtained Ġby Ġthe Ġsoft max Ġlayer Ġof Ġexisting Ġmodel Ġtypically , Ġso Ġeach Ġstate Ġhas Ġa Ġprobability Ġrather Ġthan Ġonly Ġzero Ġor Ġone , Ġand Ġcalled Ġas Ġsoft Ġtarget . ĠAs Ġa Ġresult , Ġthe Ġdeeper Ġnetwork Ġwhich Ġis Ġstudent Ġnetwork Ġlearns Ġthe Ġparameters Ġand Ġknowledge Ġfrom Ġshall ower Ġone Ġwhich Ġis Ġcalled Ġteacher Ġnetwork . ĠWhen Ġtraining Ġthe Ġstudent Ġnetwork Ġfrom Ġthe Ġteacher Ġnetwork , Ġthe Ġfinal Ġalignment Ġis Ġthe Ġcombination Ġof Ġhard Ġtarget Ġand Ġsoft Ġtarget Ġin Ġour Ġlayer - wise Ġtraining Ġphase . ĠThe Ġfinal Ġalignment Ġprovides Ġvarious Ġknowledge Ġwhich Ġtransf ered Ġfrom Ġteacher Ġnetwork Ġand Ġextracted Ġfrom Ġtrue Ġlabels . ĠIf Ġonly Ġsoft Ġtarget Ġis Ġlearned , Ġstudent Ġnetwork Ġperform Ġno Ġbetter Ġthan Ġteacher Ġnetwork , Ġbut Ġit Ġcould Ġoutper form Ġteacher Ġnetwork Ġas Ġit Ġalso Ġlearns Ġtrue Ġlabels . </s> The Ġdeeper Ġnetwork Ġspends Ġless Ġtime Ġto Ġgetting Ġthe Ġsame Ġlevel Ġof Ġoriginal Ġnetwork Ġthan Ġthe Ġnetwork Ġtrained Ġfrom Ġthe Ġbeginning , Ġas Ġa Ġperiod Ġof Ġlow Ġperformance Ġis Ġskipped . ĠTherefore , Ġtraining Ġwith Ġhard Ġand Ġsoft Ġtarget Ġis Ġa Ġtime Ġsaving Ġmethod . ĠFor Ġlarge Ġtraining Ġdataset , Ġtraining Ġwith Ġthe Ġwhole Ġdataset Ġstill Ġspends Ġtoo Ġmuch Ġtime . ĠA Ġnetwork Ġfirst ly Ġtrained Ġwith Ġonly Ġa Ġsmall Ġpart Ġof Ġdataset Ġcould Ġgo Ġdeeper Ġas Ġwell , Ġand Ġso Ġthe Ġtraining Ġtime Ġreducing Ġrapidly . ĠWhen Ġthe Ġnetwork Ġis Ġdeep Ġenough , Ġit Ġthen Ġtrained Ġon Ġthe Ġentire Ġdataset Ġto Ġget Ġfurther Ġimprovement . ĠThere Ġis Ġno Ġgap Ġof Ġaccuracy Ġbetween Ġthese Ġtwo Ġapproaches , Ġbut Ġlatter Ġone Ġsaves Ġmuch Ġtime . </s> Different ial ĠS aturation ĠCheck </s> The Ġobjects Ġof Ġconventional Ġsaturation Ġcheck Ġare Ġgrad ients Ġand Ġthe Ġcell Ġactiv ations ĠB IB REF 4 Ġ. ĠGrad ients Ġare Ġclipped Ġto Ġrange Ġ[- 5 , Ġ5 ], Ġwhile Ġthe Ġcell Ġactiv ations Ġclipped Ġto Ġrange Ġ[- 50 , Ġ50 ]. ĠApart Ġfrom Ġthem , Ġthe Ġdifferent ials Ġof Ġrecurrent Ġlayers Ġis Ġalso Ġlimited . ĠIf Ġthe Ġdifferent ials Ġgo Ġbeyond Ġthe Ġrange , Ġcorresponding Ġback Ġpropagation Ġis Ġskipped , Ġwhile Ġif Ġthe Ġgrad ients Ġand Ġcell Ġactiv ations Ġgo Ġbeyond Ġthe Ġbound , Ġvalues Ġare Ġset Ġas Ġthe Ġboundary Ġvalues . ĠThe Ġdifferent ials Ġwhich Ġare Ġtoo Ġlarge Ġor Ġtoo Ġsmall Ġwill Ġlead Ġto Ġthe Ġgrad ients Ġeasily Ġvanishing , Ġand Ġit Ġdemonstrates Ġthe Ġfailure Ġof Ġthis Ġpropagation . ĠAs Ġa Ġresult , Ġthe Ġparameters Ġare Ġnot Ġupdated , Ġand Ġnext Ġpropagation Ġ. </s> Sequ ence ĠDisc rim in ative ĠTraining </s> Cross - ent ropy Ġ( CE ) Ġis Ġwidely Ġused Ġin Ġspeech Ġrecognition Ġtraining Ġsystem Ġas Ġa Ġframe - wise Ġdiscrim in ative Ġtraining Ġcriterion . ĠHowever , Ġit Ġis Ġnot Ġwell Ġsuited Ġto Ġspeech Ġrecognition , Ġbecause Ġspeech Ġrecognition Ġtraining Ġis Ġa Ġsequential Ġlearning Ġproblem . ĠIn Ġcontrast , Ġsequence Ġdiscrim in ative Ġtraining Ġcriterion Ġhas Ġshown Ġto Ġfurther Ġimprove Ġperformance Ġof Ġneural Ġnetwork Ġfirst Ġtrained Ġwith Ġcross - ent ropy ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠWe Ġchoose Ġstate - level Ġminimum Ġbay es Ġrisk Ġ( s MB R ) ĠB IB REF 21 Ġamong Ġa Ġnumber Ġof Ġsequence Ġdiscrim in ative Ġcriterion Ġis Ġproposed , Ġsuch Ġas Ġmaximum Ġmutual Ġinformation Ġ( M MI ) ĠB IB REF 24 Ġand Ġminimum Ġphone Ġerror Ġ( MP E ) ĠB IB REF 25 Ġ. ĠMP E Ġand Ġs MB R Ġare Ġdesigned Ġto Ġminimize Ġthe Ġexpected Ġerror Ġof Ġdifferent Ġgran ularity Ġof Ġlabels , Ġwhile ĠCE Ġaims Ġto Ġminim izes Ġexpected Ġframe Ġerror , Ġand ĠM MI Ġaims Ġto Ġminim izes Ġexpected Ġsentence Ġerror . ĠState - level Ġinformation Ġis Ġfocused Ġon Ġby Ġs MB R . </s> a Ġframe - level Ġaccurate Ġmodel Ġis Ġfirst ly Ġtrained Ġby ĠCE Ġloss Ġfunction , Ġand Ġthen Ġs MB R Ġloss Ġfunction Ġis Ġutilized Ġfor Ġfurther Ġtraining Ġto Ġget Ġsequence - level Ġaccuracy . ĠOnly Ġa Ġpart Ġof Ġtraining Ġdataset Ġis Ġneeded Ġin Ġs MB R Ġtraining Ġphase Ġon Ġthe Ġbasis Ġof Ġwhole Ġdataset ĠCE Ġtraining . </s> Par allel ĠTraining </s> It Ġis Ġdemonstrated Ġthat Ġtraining Ġwith Ġlarger Ġdataset Ġcan Ġimprove Ġrecognition Ġaccuracy . ĠHowever , Ġlarger Ġdataset Ġmeans Ġmore Ġtraining Ġsamples Ġand Ġmore Ġmodel Ġparameters . ĠTherefore , Ġparallel Ġtraining Ġwith Ġmultiple ĠGPUs Ġis Ġessential , Ġand Ġit Ġmakes Ġuse Ġof Ġdata Ġparallel ism ĠB IB REF 9 Ġ. ĠThe Ġentire Ġtraining Ġdata Ġis Ġpartition ed Ġinto Ġseveral Ġsplit Ġwithout Ġoverlapping Ġand Ġthey Ġare Ġdistributed Ġto Ġdifferent ĠGPUs . ĠEach ĠGPU Ġtrains Ġwith Ġone Ġsplit Ġof Ġtraining Ġdataset Ġlocally . ĠAll ĠGPUs Ġsynchron ize Ġtheir Ġlocal Ġmodels Ġwith Ġmodel Ġaverage Ġmethod Ġafter Ġa Ġmini - batch Ġoptimization ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> Model Ġaverage Ġmethod Ġachieves Ġlinear Ġspeed up Ġin Ġtraining Ġphase , Ġbut Ġthe Ġrecognition Ġaccuracy Ġdecreases Ġcompared Ġwith Ġsingle ĠGPU Ġtraining . ĠBlock - wise Ġmodel Ġupdating Ġfilter Ġ( BM UF ) Ġis Ġanother Ġsuccessful Ġeffort Ġin Ġparallel Ġtraining Ġwith Ġlinear Ġspeed up Ġas Ġwell . ĠIt Ġcan Ġachieve Ġno - deg rad ation Ġof Ġrecognition Ġaccuracy Ġwith Ġmulti - GPU s ĠB IB REF 14 Ġ. ĠIn Ġthe Ġmodel Ġaverage Ġmethod , Ġaggreg ated Ġmodel ĠIN LINE FORM 0 Ġis Ġcomputed Ġand Ġbroadcast ed Ġto ĠGPUs . ĠOn Ġthe Ġbasis Ġof Ġit , ĠBM UF Ġproposed Ġa Ġnovel Ġmodel Ġupdating Ġstrategy : ĠIN LINE FORM 1 ĠIN LINE FORM 2 </s> Where ĠIN LINE FORM 0 Ġdenotes Ġmodel Ġupdate , Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġglobal - model Ġupdate . ĠThere Ġare Ġtwo Ġparameters Ġin ĠBM UF , Ġblock Ġmomentum ĠIN LINE FORM 2 Ġ, Ġand Ġblock Ġlearning Ġrate ĠIN LINE FORM 3 Ġ. ĠThen , Ġthe Ġglobal Ġmodel Ġis Ġupdated Ġas ĠIN LINE FORM 4 </s> C onsequ ently , ĠIN LINE FORM 0 Ġis Ġbroadcast ed Ġto Ġall ĠGPUs Ġto Ġinitial Ġtheir Ġlocal Ġmodels , Ġinstead Ġof ĠIN LINE FORM 1 Ġin Ġmodel Ġaverage Ġmethod . </s> A ver aged ĠSG D Ġis Ġproposed Ġto Ġfurther Ġaccelerate Ġthe Ġconvergence Ġspeed Ġof ĠSG D . ĠA ver aged ĠSG D Ġlever ages Ġthe Ġmoving Ġaverage Ġ( MA ) ĠIN LINE FORM 0 Ġas Ġthe Ġestim ator Ġof ĠIN LINE FORM 1 ĠB IB REF 15 Ġ: ĠIN LINE FORM 2 </s> Where ĠIN LINE FORM 0 Ġis Ġcomputed Ġby Ġmodel Ġaveraging Ġor ĠBM UF . ĠIt Ġis Ġshown Ġthat ĠIN LINE FORM 1 Ġcan Ġwell Ġconverge Ġto ĠIN LINE FORM 2 Ġ, Ġwith Ġthe Ġlarge Ġenough Ġtraining Ġdataset Ġin Ġsingle ĠGPU Ġtraining . ĠIt Ġcan Ġbe Ġconsidered Ġas Ġa Ġnon - inter ference Ġstrategy Ġthat ĠIN LINE FORM 3 Ġdoes Ġnot Ġparticipate Ġthe Ġmain Ġoptimization Ġprocess , Ġand Ġonly Ġtakes Ġeffect Ġafter Ġthe Ġend Ġof Ġentire Ġoptimization . ĠHowever , Ġfor Ġthe Ġparallel Ġtraining Ġimplementation , Ġeach ĠIN LINE FORM 4 Ġis Ġcomputed Ġby Ġmodel Ġaveraging Ġand ĠBM UF Ġwith Ġmultiple Ġmodels , Ġand Ġmoving Ġaverage Ġmodel ĠIN LINE FORM 5 Ġdoes Ġnot Ġwell Ġconverge , Ġcompared Ġwith Ġsingle ĠGPU Ġtraining . </s> Model Ġaveraging Ġbased Ġmethods Ġare Ġemployed Ġin Ġparallel Ġtraining Ġof Ġlarge Ġscale Ġdataset , Ġbecause Ġof Ġtheir Ġfaster Ġconvergence , Ġand Ġespecially Ġno - deg rad ation Ġimplementation Ġof ĠBM UF . ĠBut Ġcombination Ġof Ġmodel Ġaveraged Ġbased Ġmethods Ġand Ġmoving Ġaverage Ġdoes Ġnot Ġmatch Ġthe Ġexpectation Ġof Ġfurther Ġenhance Ġperformance Ġand Ġit Ġis Ġpresented Ġas ĠIN LINE FORM 0 </s> The Ġweight Ġof Ġeach ĠIN LINE FORM 0 Ġis Ġequal Ġin Ġmoving Ġaverage Ġmethod Ġregardless Ġthe Ġeffect Ġof Ġtemporal Ġorder . ĠBut ĠIN LINE FORM 1 Ġcloser Ġto Ġthe Ġend Ġof Ġtraining Ġachieve Ġhigher Ġaccuracy Ġin Ġthe Ġmodel Ġaveraging Ġbased Ġapproach , Ġand Ġthus Ġit Ġshould Ġbe Ġwith Ġmore Ġproportion Ġin Ġfinal ĠIN LINE FORM 2 Ġ. ĠAs Ġa Ġresult , Ġexponential Ġmoving Ġaverage ( EMA ) Ġis Ġappropriate , Ġwhich Ġthe Ġweight Ġfor Ġeach Ġolder Ġparameters Ġdecrease Ġexponentially , Ġand Ġnever Ġreaching Ġzero . ĠAfter Ġmoving Ġaverage Ġbased Ġmethods , Ġthe ĠE MA Ġparameters Ġare Ġupdated Ġrec urs ively Ġas ĠIN LINE FORM 3 </s> Here ĠIN LINE FORM 0 Ġrepresents Ġthe Ġdegree Ġof Ġweight Ġdecrease , Ġand Ġcalled Ġexponential Ġupdating Ġrate . ĠE MA Ġis Ġalso Ġa Ġnon - inter ference Ġtraining Ġstrategy Ġthat Ġis Ġimplemented Ġeasily , Ġas Ġthe Ġupdated Ġmodel Ġis Ġnot Ġbroadcast ed . ĠTherefore , Ġthere Ġis Ġno Ġneed Ġto Ġadd Ġextra Ġlearning Ġrate Ġupdating Ġapproach , Ġas Ġit Ġcan Ġbe Ġapp ended Ġto Ġexisting Ġtraining Ġprocedure Ġdirectly . </s> Deploy ment </s> There Ġis Ġa Ġhigh Ġreal Ġtime Ġrequirement Ġin Ġreal Ġworld Ġapplication , Ġespecially Ġin Ġonline Ġvoice Ġsearch Ġsystem . ĠShen ma Ġvoice Ġsearch Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġmobile Ġsearch Ġengines Ġin ĠChina , Ġand Ġit Ġis Ġa Ġstreaming Ġservice Ġthat Ġintermediate Ġrecognition Ġresults Ġdisplayed Ġwhile Ġusers Ġare Ġstill Ġspeaking . ĠUn id irection al ĠL ST M Ġnetwork Ġis Ġapplied , Ġrather Ġthan Ġbid irection al Ġone , Ġbecause Ġit Ġis Ġwell Ġsuited Ġto Ġreal - time Ġstreaming Ġspeech Ġrecognition . </s> Dist illation </s> It Ġis Ġdemonstrated Ġthat Ġdeep Ġneural Ġnetwork Ġarchitecture Ġcan Ġachieve Ġimprovement Ġin ĠLV CS R . ĠHowever , Ġit Ġalso Ġleads Ġto Ġmuch Ġmore Ġcomputation Ġand Ġhigher ĠR TF , Ġso Ġthat Ġthe Ġrecognition Ġresult Ġcan Ġnot Ġbe Ġdisplayed Ġin Ġreal Ġtime . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġdeeper Ġneural Ġnetwork Ġcontains Ġmore Ġknowledge , Ġbut Ġit Ġis Ġalso Ġcumbersome . Ġthe Ġknowledge Ġis Ġkey Ġto Ġimprove Ġthe Ġperformance . ĠIf Ġit Ġcan Ġbe Ġtransf ered Ġfrom Ġcumbersome Ġmodel Ġto Ġa Ġsmall Ġmodel , Ġthe Ġrecognition Ġability Ġcan Ġalso Ġbe Ġtransf ered Ġto Ġthe Ġsmall Ġmodel . ĠKnowledge Ġtransferring Ġto Ġsmall Ġmodel Ġis Ġcalled Ġdist illation ĠB IB REF 19 Ġ. ĠThe Ġsmall Ġmodel Ġcan Ġperform Ġas Ġwell Ġas Ġcumbersome Ġmodel , Ġafter Ġdist illing . ĠIt Ġprovide Ġa Ġway Ġto Ġutilize Ġhigh - performance Ġbut Ġhigh ĠR TF Ġmodel Ġin Ġreal Ġtime Ġsystem . ĠThe Ġclass Ġprobability Ġproduced Ġby Ġthe Ġcumbersome Ġmodel Ġis Ġregarded Ġas Ġsoft Ġtarget , Ġand Ġthe Ġgeneral ization Ġability Ġof Ġcumbersome Ġmodel Ġis Ġtransf ered Ġto Ġsmall Ġmodel Ġwith Ġit . ĠDist illation Ġis Ġmodel 's Ġknowledge Ġtransferring Ġapproach , Ġso Ġthere Ġis Ġno Ġneed Ġto Ġuse Ġthe Ġhard Ġtarget , Ġwhich Ġis Ġdifferent Ġwith Ġthe Ġlayer - wise Ġtraining Ġmethod . </s> 9 - l ayers Ġun id irection al ĠL ST M Ġmodel Ġachieves Ġoutstanding Ġperformance , Ġbut Ġmeanwhile Ġit Ġis Ġtoo Ġcomput ationally Ġexpensive Ġto Ġallow Ġdeployment Ġin Ġreal Ġtime Ġrecognition Ġsystem . ĠIn Ġorder Ġto Ġensure Ġreal - time Ġof Ġthe Ġsystem , Ġthe Ġnumber Ġof Ġlayers Ġneeds Ġto Ġbe Ġreduced . ĠThe Ġshall ower Ġnetwork Ġcan Ġlearn Ġthe Ġknowledge Ġof Ġdeeper Ġnetwork Ġwith Ġdist illation . ĠIt Ġis Ġfound Ġthat ĠR TF Ġof Ġ2 - l ayers Ġnetwork Ġis Ġacceptable , Ġso Ġthe Ġknowledge Ġis Ġdistilled Ġfrom Ġ9 - l ayers Ġwell - trained Ġmodel Ġto Ġ2 - l ayers Ġmodel . ĠTable ĠT AB REF 16 Ġshows Ġthat Ġdist illation Ġfrom Ġ9 - l ayers Ġto Ġ2 - l ayers Ġbrings ĠR TF Ġdecrease Ġof Ġrelative Ġ53 %, Ġwhile ĠC ER Ġonly Ġincreases Ġ5 %. ĠThe Ġknowledge Ġof Ġdeep Ġnetwork Ġis Ġalmost Ġtransf ered Ġwith Ġdist illation , ĠDist illation Ġbrings Ġpromising ĠR TF Ġreduction , Ġbut Ġonly Ġlittle Ġknowledge Ġof Ġdeep Ġnetwork Ġis Ġlost . ĠMoreover , ĠC ER Ġof Ġ2 - l ayers Ġdistilled ĠL ST M Ġdecreases Ġrelative Ġ14 %, Ġcompared Ġwith Ġ2 - l ayers Ġregular - trained ĠL ST M . </s> Transfer ĠLearning Ġwith Ġs MB R </s> For Ġa Ġcertain Ġspecific Ġscenario , Ġthe Ġmodel Ġtrained Ġwith Ġthe Ġdata Ġrecorded Ġfrom Ġit Ġhas Ġbetter Ġadaptation Ġthan Ġthe Ġmodel Ġtrained Ġwith Ġgeneric Ġscenario . ĠBut Ġit Ġspends Ġtoo Ġmuch Ġtime Ġtraining Ġa Ġmodel Ġfrom Ġthe Ġbeginning , Ġif Ġthere Ġis Ġa Ġwell - trained Ġmodel Ġfor Ġgeneric Ġscenarios . ĠMoreover , Ġlabeling Ġa Ġlarge Ġquantity Ġof Ġtraining Ġdata Ġin Ġnew Ġscenario Ġis Ġboth Ġcostly Ġand Ġtime Ġconsuming . ĠIf Ġa Ġmodel Ġtransfer Ġtrained Ġwith Ġsmaller Ġdataset Ġcan Ġobtained Ġthe Ġsimilar Ġrecognition Ġaccuracy Ġcompared Ġwith Ġthe Ġmodel Ġdirectly Ġtrained Ġwith Ġlarger Ġdataset , Ġit Ġis Ġno Ġdoubt Ġthat Ġtransfer Ġlearning Ġis Ġmore Ġpractical . ĠSince Ġspecific Ġscenario Ġis Ġa Ġsubset Ġof Ġgeneric Ġscenario , Ġsome Ġknowledge Ġcan Ġbe Ġshared Ġbetween Ġthem . ĠBesides , Ġgeneric Ġscenario Ġconsists Ġof Ġvarious Ġconditions , Ġso Ġits Ġmodel Ġhas Ġgreater Ġrobust ness . ĠAs Ġa Ġresult , Ġnot Ġonly Ġshared Ġknowledge Ġbut Ġalso Ġrobust ness Ġcan Ġbe Ġtransf ered Ġfrom Ġthe Ġmodel Ġof Ġgeneric Ġscenario Ġto Ġthe Ġmodel Ġof Ġspecific Ġone . </s> As Ġthe Ġmodel Ġwell Ġtrained Ġfrom Ġgeneric Ġscenario Ġachieves Ġgood Ġperformance Ġin Ġframe Ġlevel Ġclassification , Ġsequence Ġdiscrim in ative Ġtraining Ġis Ġrequired Ġto Ġadapt Ġnew Ġmodel Ġto Ġspecific Ġscenario Ġadditionally . ĠMoreover , Ġit Ġdoes Ġnot Ġneed Ġalignment Ġfrom ĠH MM - G MM Ġsystem , Ġand Ġit Ġalso Ġsaves Ġamount Ġof Ġtime Ġto Ġprepare Ġalignment . </s> Training ĠData </s> A Ġlarge Ġquantity Ġof Ġlabeled Ġdata Ġis Ġneeded Ġfor Ġtraining Ġa Ġmore Ġaccurate Ġacoustic Ġmodel . ĠWe Ġcollect Ġthe Ġ17 000 Ġhours Ġlabeled Ġdata Ġfrom ĠShen ma Ġvoice Ġsearch , Ġwhich Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġmobile Ġsearch Ġengines Ġin ĠChina . ĠThe Ġdataset Ġis Ġcreated Ġfrom Ġanonymous Ġonline Ġusers ' Ġsearch Ġqueries Ġin ĠMandarin , Ġand Ġall Ġaudio Ġfile 's Ġsampling Ġrate Ġis Ġ16 kHz , Ġrecorded Ġby Ġmobile Ġphones . ĠThis Ġdataset Ġconsists Ġof Ġmany Ġdifferent Ġconditions , Ġsuch Ġas Ġdiverse Ġnoise Ġeven Ġlow Ġsignal - to - no ise , Ġbab ble , Ġdialect s , Ġaccents , Ġhesitation Ġand Ġso Ġon . </s> In Ġthe ĠAm ap , Ġwhich Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġweb Ġmapping Ġand Ġnavigation Ġservices Ġin ĠChina , Ġusers Ġcan Ġsearch Ġlocations Ġand Ġnavigate Ġto Ġlocations Ġthey Ġwant Ġthough Ġvoice Ġsearch . ĠTo Ġpresent Ġthe Ġperformance Ġof Ġtransfer Ġlearning Ġwith Ġsequence Ġdiscrim in ative Ġtraining , Ġthe Ġmodel Ġtrained Ġfrom ĠShen ma Ġvoice Ġsearch Ġwhich Ġis Ġgren eric Ġscenario Ġtransfer Ġits Ġknowledge Ġto Ġthe Ġmodel Ġof ĠAm ap Ġvoice Ġsearch . Ġ7 300 Ġhours Ġlabeled Ġdata Ġis Ġcollected Ġin Ġthe Ġsimilar Ġway Ġof ĠShen ma Ġvoice Ġsearch Ġdata Ġcollection . </s> Two Ġdataset Ġis Ġdivided Ġinto Ġtraining Ġset , Ġvalidation Ġset Ġand Ġtest Ġset Ġseparately , Ġand Ġthe Ġquantity Ġof Ġthem Ġis Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. ĠThe Ġthree Ġsets Ġare Ġsplit Ġaccording Ġto Ġspeakers , Ġin Ġorder Ġto Ġavoid Ġutter ances Ġof Ġsame Ġspeaker Ġappearing Ġin Ġthree Ġsets Ġsimultaneously . ĠThe Ġtest Ġsets Ġof ĠShen ma Ġand ĠAm ap Ġvoice Ġsearch Ġare Ġcalled ĠShen ma ĠTest Ġand ĠAm ap ĠTest . </s> Exper imental Ġsetup </s> L ST M ĠR NN s Ġoutper form Ġconventional ĠR NN s Ġfor Ġspeech Ġrecognition Ġsystem , Ġespecially Ġdeep ĠL ST M ĠR NN s , Ġbecause Ġof Ġits Ġlong - range Ġdependencies Ġmore Ġaccurately Ġfor Ġtemporal Ġsequence Ġconditions ĠB IB REF 26 Ġ, ĠB IB REF 23 Ġ. ĠShen ma Ġand ĠAm ap Ġvoice Ġsearch Ġis Ġa Ġstreaming Ġservice Ġthat Ġintermediate Ġrecognition Ġresults Ġdisplayed Ġwhile Ġusers Ġare Ġstill Ġspeaking . ĠSo Ġas Ġfor Ġonline Ġrecognition Ġin Ġreal Ġtime , Ġwe Ġprefer Ġun id irection al ĠL ST M Ġmodel Ġrather Ġthan Ġbid irection al Ġone . ĠThus , Ġthe Ġtraining Ġsystem Ġis Ġun id irection al ĠL ST M - based . </s> A Ġ26 - dimensional Ġfilter Ġbank Ġand Ġ2 - dimensional Ġpitch Ġfeature Ġis Ġextracted Ġfor Ġeach Ġframe , Ġand Ġis Ġconc aten ated Ġwith Ġfirst Ġand Ġsecond Ġorder Ġdifference Ġas Ġthe Ġfinal Ġinput Ġof Ġthe Ġnetwork . ĠThe Ġsuper Ġframe Ġare Ġstacked Ġby Ġ3 Ġframes Ġwithout Ġoverlapping . ĠThe Ġarchitecture Ġwe Ġtrained Ġconsists Ġof Ġtwo ĠL ST M Ġlayers Ġwith Ġs igm oid Ġactivation Ġfunction , Ġfollowed Ġby Ġa Ġfull - connection Ġlayer . ĠThe Ġout Ġlayer Ġis Ġa Ġsoft max Ġlayer Ġwith Ġ110 88 Ġhidden Ġmark ov Ġmodel Ġ( H MM ) Ġtied - states Ġas Ġoutput Ġclasses , Ġthe Ġloss Ġfunction Ġis Ġcross - ent ropy Ġ( CE ). ĠThe Ġperformance Ġmetric Ġof Ġthe Ġsystem Ġin ĠMandarin Ġis Ġreported Ġwith Ġcharacter Ġerror Ġrate Ġ( C ER ). ĠThe Ġalignment Ġof Ġframe - level Ġground Ġtruth Ġis Ġobtained Ġby ĠGM M - H MM Ġsystem . ĠMini - bat ched ĠSG D Ġis Ġutilized Ġwith Ġmomentum Ġtrick Ġand Ġthe Ġnetwork Ġis Ġtrained Ġfor Ġa Ġtotal Ġof Ġ4 Ġepoch s . ĠThe Ġblock Ġlearning Ġrate Ġand Ġblock Ġmomentum Ġof ĠBM UF Ġare Ġset Ġas Ġ1 Ġand Ġ0 . 9 . Ġ5 - gram Ġlanguage Ġmodel Ġis Ġlever aged Ġin Ġdec oder , Ġand Ġthe Ġvocabulary Ġsize Ġis Ġas Ġlarge Ġas Ġ76 0000 . ĠDifferent ials Ġof Ġrecurrent Ġlayers Ġis Ġlimited Ġto Ġrange Ġ[- 10000 , 10000 ], Ġwhile Ġgrad ients Ġare Ġclipped Ġto Ġrange Ġ[- 5 , Ġ5 ] Ġand Ġcell Ġactiv ations Ġclipped Ġto Ġrange Ġ[- 50 , Ġ50 ]. ĠAfter Ġtraining Ġwith ĠCE Ġloss , Ġs MB R Ġloss Ġis Ġemployed Ġto Ġfurther Ġimprove Ġthe Ġperformance . </s> It Ġhas Ġshown Ġthat ĠBM UF Ġoutper forms Ġtraditional Ġmodel Ġaveraging Ġmethod , Ġand Ġit Ġis Ġutilized Ġat Ġthe Ġsynchronization Ġphase . ĠAfter Ġsynchron izing Ġwith ĠBM UF , ĠE MA Ġmethod Ġfurther Ġupdates Ġthe Ġmodel Ġin Ġnon - inter ference Ġway . ĠThe Ġtraining Ġsystem Ġis Ġdeployed Ġon Ġthe ĠMP I - based ĠH PC Ġcluster Ġwhere Ġ8 ĠGPUs . ĠEach ĠGPU Ġprocesses Ġnon - over lap Ġsubset Ġsplit Ġfrom Ġthe Ġentire Ġlarge Ġscale Ġdataset Ġin Ġparallel . </s> Local Ġmodels Ġfrom Ġdistributed Ġworkers Ġsynchron ize Ġwith Ġeach Ġother Ġin Ġdecentralized Ġway . ĠIn Ġthe Ġtraditional Ġmodel Ġaveraging Ġand ĠBM UF Ġmethod , Ġa Ġparameter Ġserver Ġwaits Ġfor Ġall Ġworkers Ġto Ġsend Ġtheir Ġlocal Ġmodels , Ġaggregate Ġthem , Ġand Ġsend Ġthe Ġupdated Ġmodel Ġto Ġall Ġworkers . ĠComputing Ġresource Ġof Ġworkers Ġis Ġwasted Ġuntil Ġaggregation Ġof Ġthe Ġparameter Ġserver Ġdone . ĠDec ent ral ized Ġmethod Ġmakes Ġfull Ġuse Ġof Ġcomputing Ġresource , Ġand Ġwe Ġemploy Ġthe ĠMP I - based ĠMesh ĠAll Red uce Ġmethod . ĠIt Ġis Ġmesh Ġtop ology Ġas Ġshown Ġin ĠFigure ĠFIG REF 12 Ġ. ĠThere Ġis Ġno Ġcentralized Ġparameter Ġserver , Ġand Ġpeer Ġto Ġpeer Ġcommunication Ġis Ġused Ġto Ġtransmit Ġlocal Ġmodels Ġbetween Ġworkers . ĠLocal Ġmodel ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 Ġ- th Ġworker Ġin ĠIN LINE FORM 2 Ġworkers Ġcluster Ġis Ġsplit Ġto ĠIN LINE FORM 3 Ġpieces ĠIN LINE FORM 4 Ġ, Ġand Ġsend Ġto Ġcorresponding Ġworker . ĠIn Ġthe Ġaggregation Ġphase , ĠIN LINE FORM 5 Ġ- th Ġworker Ġcomputed ĠIN LINE FORM 6 Ġsplits Ġof Ġmodel ĠIN LINE FORM 7 Ġand Ġsend Ġupdated Ġmodel ĠIN LINE FORM 8 Ġback Ġto Ġworkers . ĠAs Ġa Ġresult , Ġall Ġworkers Ġparticipate Ġin Ġaggregation Ġand Ġno Ġcomputing Ġresource Ġis Ġdissip ated . ĠIt Ġis Ġsignificant Ġto Ġpromote Ġtraining Ġefficiency , Ġwhen Ġthe Ġsize Ġof Ġneural Ġnetwork Ġmodel Ġis Ġtoo Ġlarge . ĠThe ĠE MA Ġmodel Ġis Ġalso Ġupdated Ġadditionally , Ġbut Ġnot Ġbroadcasting Ġit . </s> Results </s> In Ġorder Ġto Ġevaluate Ġour Ġsystem , Ġseveral Ġsets Ġof Ġexperiments Ġare Ġperformed . ĠThe ĠShen ma Ġtest Ġset Ġincluding Ġabout Ġ9000 Ġsamples Ġand ĠAm ap Ġtest Ġset Ġincluding Ġabout Ġ7000 Ġsamples Ġcontain Ġvarious Ġreal Ġworld Ġconditions . ĠIt Ġsim ulates Ġthe Ġmajority Ġof Ġuser Ġscenarios , Ġand Ġcan Ġwell Ġevaluates Ġthe Ġperformance Ġof Ġa Ġtrained Ġmodel . ĠFirstly , Ġwe Ġshow Ġthe Ġresults Ġof Ġmodels Ġtrained Ġwith ĠE MA Ġmethod . ĠSecondly , Ġfor Ġreal Ġworld Ġapplications , Ġvery Ġdeep ĠL ST M Ġis Ġdistilled Ġto Ġa Ġshallow Ġone , Ġso Ġas Ġfor Ġlower ĠR TF . ĠThe Ġmodel Ġof ĠAm ap Ġis Ġalso Ġneeded Ġto Ġtrain Ġfor Ġmap Ġand Ġnavigation Ġscenarios . ĠThe Ġperformance Ġof Ġtransfer Ġlearning Ġfrom ĠShen ma Ġvoice Ġsearch Ġto ĠAm ap Ġvoice Ġsearch Ġis Ġalso Ġpresented . </s> Layer - wise ĠTraining </s> In Ġlayer - wise Ġtraining , Ġthe Ġdeeper Ġmodel Ġlearns Ġboth Ġparameters Ġand Ġknowledge Ġfrom Ġthe Ġshall ower Ġmodel . ĠThe Ġdeeper Ġmodel Ġis Ġinitialized Ġby Ġthe Ġshall ower Ġone , Ġand Ġits Ġalignment Ġis Ġthe Ġcombination Ġof Ġhard Ġtarget Ġand Ġsoft Ġtarget Ġof Ġshall ower Ġone . ĠTwo Ġtargets Ġhave Ġthe Ġsame Ġweights Ġin Ġour Ġframework . ĠThe Ġteacher Ġmodel Ġis Ġtrained Ġwith ĠCE . ĠFor Ġeach Ġlayer - wise Ġtrained ĠCE Ġmodel , Ġcorresponding Ġs MB R Ġmodel Ġis Ġalso Ġtrained , Ġas Ġs MB R Ġcould Ġachieve Ġadditional Ġimprovement . ĠIn Ġour Ġframework , Ġ1000 Ġhours Ġdata Ġis Ġrandomly Ġselected Ġfrom Ġthe Ġtotal Ġdataset Ġfor Ġs MB R Ġtraining . ĠThere Ġis Ġno Ġobvious Ġperformance Ġenhancement Ġwhen Ġthe Ġsize Ġof Ġs MB R Ġtraining Ġdataset Ġincreases . </s> For Ġvery Ġdeep Ġun id irection al ĠL ST M Ġinitialized Ġwith ĠXavier Ġinitialization Ġalgorithm , Ġ6 - l ayers Ġmodel Ġconver ges Ġwell , Ġbut Ġthere Ġis Ġno Ġfurther Ġimprovement Ġwith Ġincreasing Ġthe Ġnumber Ġof Ġlayers . ĠTherefore , Ġthe Ġfirst Ġ6 Ġlayers Ġof Ġ7 - l ayers Ġmodel Ġis Ġinitialized Ġby Ġ6 - l ayers Ġmodel , Ġand Ġsoft Ġtarget Ġis Ġprovided Ġby Ġ6 - l ayers Ġmodel . ĠConsequently , Ġdeeper ĠL ST M Ġis Ġalso Ġtrained Ġin Ġthe Ġsame Ġway . ĠIt Ġshould Ġbe Ġnoticed Ġthat Ġthe Ġteacher Ġmodel Ġof Ġ9 - l ayers Ġmodel Ġis Ġthe Ġ8 - l ayers Ġmodel Ġtrained Ġby Ġs MB R , Ġwhile Ġthe Ġother Ġteacher Ġmodel Ġis ĠCE Ġmodel . ĠAs Ġshown Ġin ĠTable ĠT AB REF 15 Ġ, Ġthe Ġlayer - wise Ġtrained Ġmodels Ġalways Ġperforms Ġbetter Ġthan Ġthe Ġmodels Ġwith ĠXavier Ġinitialization , Ġas Ġthe Ġmodel Ġis Ġdeep . ĠTherefore , Ġfor Ġthe Ġlast Ġlayer Ġtraining , Ġwe Ġchoose Ġ8 - l ayers Ġs MB R Ġmodel Ġas Ġthe Ġteacher Ġmodel Ġinstead Ġof ĠCE Ġmodel . ĠA Ġcomparison Ġbetween Ġ6 - l ayers Ġand Ġ9 - l ayers Ġs MB R Ġmodels Ġshows Ġthat Ġ3 Ġadditional Ġlayers Ġof Ġlayer - wise Ġtraining Ġbrings Ġrelative Ġ12 . 6 % Ġdecreasing Ġof ĠC ER . ĠIt Ġis Ġalso Ġsignificant Ġthat Ġthe Ġaveraged ĠC ER Ġof Ġs MB R Ġmodels Ġwith Ġdifferent Ġlayers Ġdecreases Ġabsolute Ġ0 . 73 % Ġapproximately Ġcompared Ġwith ĠCE Ġmodels , Ġso Ġthe Ġimprovement Ġof Ġsequence Ġdiscrim in ative Ġlearning Ġis Ġpromising . </s> Transfer ĠLearning </s> 2 - l ayers Ġdistilled Ġmodel Ġof ĠShen ma Ġvoice Ġsearch Ġhas Ġshown Ġa Ġimpressive Ġperformance Ġon ĠShen ma ĠTest , Ġand Ġwe Ġcall Ġit ĠShen ma Ġmodel . ĠIt Ġis Ġtrained Ġfor Ġgeneric Ġsearch Ġscenario , Ġbut Ġit Ġhas Ġless Ġadaptation Ġfor Ġspecific Ġscenario Ġlike ĠAm ap Ġvoice Ġsearch . ĠTraining Ġwith Ġvery Ġlarge Ġdataset Ġusing ĠCE Ġloss Ġis Ġregarded Ġas Ġimprovement Ġof Ġframe Ġlevel Ġrecognition Ġaccuracy , Ġand Ġs MB R Ġwith Ġless Ġdataset Ġfurther Ġimproves Ġaccuracy Ġas Ġsequence Ġdiscrim in ative Ġtraining . ĠIf Ġrobust Ġmodel Ġof Ġgeneric Ġscenario Ġis Ġtrained , Ġthere Ġis Ġno Ġneed Ġto Ġtrain Ġa Ġmodel Ġwith Ġvery Ġlarge Ġdataset , Ġand Ġsequence Ġdiscrim in ative Ġtraining Ġwith Ġless Ġdataset Ġis Ġenough . ĠTherefore , Ġon Ġthe Ġbasis Ġof ĠShen ma Ġmodel , Ġit Ġis Ġsufficient Ġto Ġtrain Ġa Ġnew ĠAm ap Ġmodel Ġwith Ġsmall Ġdataset Ġusing Ġs MB R . ĠAs Ġshown Ġin ĠTable ĠT AB REF 18 Ġ, ĠShen ma Ġmodel Ġpresents Ġthe Ġworst Ġperformance Ġamong Ġthree Ġmethods , Ġsince Ġit Ġdoes Ġnot Ġtrained Ġfor ĠAm ap Ġscenario . Ġ2 - l ayers ĠShen ma Ġmodel Ġfurther Ġtrained Ġwith Ġs MB R Ġachieves Ġabout Ġ8 . 1 % Ġrelative Ġreduction , Ġcompared Ġwith Ġ2 - l ayers Ġregular - trained ĠAm ap Ġmodel . ĠBoth Ġtraining Ġs MB R Ġdatasets Ġcontain Ġthe Ġsame Ġ1000 Ġhours Ġdata . ĠAs Ġa Ġresult , Ġwith Ġthe ĠShen ma Ġmodel , Ġonly Ġabout Ġ14 % Ġdata Ġusage Ġachieves Ġlower ĠC ER , Ġand Ġit Ġleads Ġto Ġgreat Ġtime Ġand Ġcost Ġsaving Ġwith Ġless Ġlabeled Ġdata . ĠBesides , Ġtransfer Ġlearning Ġwith Ġs MB R Ġdoes Ġnot Ġuse Ġthe Ġalignment Ġfrom Ġthe ĠH MM - G MM Ġsystem , Ġso Ġit Ġalso Ġsaves Ġhuge Ġamount Ġof Ġtime . </s> Conclusion </s> We Ġhave Ġpresented Ġa Ġwhole Ġdeep Ġun id irection al ĠL ST M Ġparallel Ġtraining Ġsystem Ġfor ĠLV CS R . ĠThe Ġrecognition Ġperformance Ġimproves Ġwhen Ġthe Ġnetwork Ġgoes Ġdeep . ĠDist illation Ġmakes Ġit Ġpossible Ġthat Ġdeep ĠL ST M Ġmodel Ġtransfer Ġits Ġknowledge Ġto Ġshallow Ġmodel Ġwith Ġlittle Ġloss . ĠThe Ġmodel Ġcould Ġbe Ġdistilled Ġto Ġ2 - l ayers Ġmodel Ġwith Ġvery Ġlow ĠR TF , Ġso Ġthat Ġit Ġcan Ġdisplay Ġthe Ġimmediate Ġrecognition Ġresults . ĠAs Ġa Ġresult , Ġits ĠC ER Ġdecrease Ġrelatively Ġ14 %, Ġcompared Ġwith Ġthe Ġ2 - l ayers Ġregular Ġtrained Ġmodel . ĠIn Ġaddition , Ġtransfer Ġlearning Ġwith Ġs MB R Ġis Ġalso Ġproposed . ĠIf Ġa Ġgreat Ġmodel Ġhas Ġwell Ġtrained Ġfrom Ġgeneric Ġscenario , Ġonly Ġ14 % Ġof Ġthe Ġsize Ġof Ġtraining Ġdataset Ġis Ġneeded Ġto Ġtrain Ġa Ġmore Ġaccuracy Ġacoustic Ġmodel Ġfor Ġspecific Ġscenario . ĠOur Ġfuture Ġwork Ġincludes Ġ1 ) Ġfinding Ġmore Ġeffective Ġmethods Ġto Ġreduce Ġthe ĠC ER Ġby Ġincreasing Ġthe Ġnumber Ġof Ġlayers ; Ġ2 ) Ġapplying Ġthis Ġtraining Ġframework Ġto ĠConnection ist ĠTem poral ĠClassification Ġ( CT C ) Ġand Ġattention - based Ġneural Ġnetworks .