190 hours ( INLINEFORM1 100K instances)
EXTRACTIVE
trained on 190 hours ( INLINEFORM1 100K instances) of transcribed speech data, selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset
EXTRACTIVE
A base model, INLINEFORM0 , is trained on 190 hours ( INLINEFORM1 100K instances) of transcribed speech data. Then, it selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset. We query labels for the selected subset and incorporate them into training. Learning rates are tuned on a small validation set of 2048 instances. The trained model is then tested on a 156-hour ( INLINEFORM3 100K instances) test set and we report CTC loss, Character Error Rate (CER) and Word Error Rate (WER).
A base model, INLINEFORM0 , is trained on 190 hours ( INLINEFORM1 100K instances) of transcribed speech data. Then, it selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset. We query labels for the selected subset and incorporate them into training. Learning rates are tuned on a small validation set of 2048 instances. The trained model is then tested on a 156-hour ( INLINEFORM3 100K instances) test set and we report CTC loss, Character Error Rate (CER) and Word Error Rate (WER).
1612.03226
Introduction </s> State - of - the - art Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems ĠB IB REF 0 Ġhave Ġlarge Ġmodel Ġcapacities Ġand Ġrequire Ġsignificant Ġquantities Ġof Ġtraining Ġdata Ġto Ġgeneral ize . ĠLab eling Ġthousands Ġof Ġhours Ġof Ġaudio , Ġhowever , Ġis Ġexpensive Ġand Ġtime - consuming . ĠA Ġnatural Ġquestion Ġto Ġask Ġis Ġhow Ġto Ġachieve Ġbetter Ġgeneral ization Ġwith Ġfewer Ġtraining Ġexamples . ĠActive Ġlearning Ġstudies Ġthis Ġproblem Ġby Ġidentifying Ġand Ġlabeling Ġonly Ġthe Ġmost Ġinformative Ġdata , Ġpotentially Ġreducing Ġsample Ġcomplexity . ĠHow Ġmuch Ġactive Ġlearning Ġcan Ġhelp Ġin Ġlarge - scale , Ġend - to - end ĠAS R Ġsystems , Ġhowever , Ġis Ġstill Ġan Ġopen Ġquestion . </s> The Ġspeech Ġrecognition Ġcommunity Ġhas Ġgenerally Ġidentified Ġthe Ġinform at iveness Ġof Ġsamples Ġby Ġcalculating Ġconfidence Ġscores . ĠIn Ġparticular , Ġan Ġutter ance Ġis Ġconsidered Ġinformative Ġif Ġthe Ġmost Ġlikely Ġprediction Ġhas Ġsmall Ġprobability ĠB IB REF 1 Ġ, Ġor Ġif Ġthe Ġpredictions Ġare Ġdistributed Ġvery Ġuniformly Ġover Ġthe Ġlabels ĠB IB REF 2 Ġ. ĠThough Ġconfidence - based Ġmeasures Ġwork Ġwell Ġin Ġpractice , Ġless Ġattention Ġhas Ġbeen Ġfocused Ġon Ġgradient - based Ġmethods Ġlike ĠEx pected ĠGrad ient ĠLength Ġ( EG L ) ĠB IB REF 3 Ġ, Ġwhere Ġthe Ġinform at iveness Ġis Ġmeasured Ġby Ġthe Ġnorm Ġof Ġthe Ġgradient Ġincurred Ġby Ġthe Ġinstance . ĠE GL Ġhas Ġpreviously Ġbeen Ġjustified Ġas Ġintuitive ly Ġmeasuring Ġthe Ġexpected Ġchange Ġin Ġa Ġmodel 's Ġparameters ĠB IB REF 3 Ġ. We Ġformal ize Ġthis Ġintuition Ġfrom Ġthe Ġperspective Ġof Ġas ym pt otic Ġvariance Ġreduction , Ġand Ġexperiment ally , Ġwe Ġshow ĠE GL Ġto Ġbe Ġsuperior Ġto Ġconfidence - based Ġmethods Ġon Ġspeech Ġrecognition Ġtasks . ĠAdditionally , Ġwe Ġobserve Ġthat Ġthe Ġranking Ġof Ġsamples Ġscored Ġby ĠE GL Ġis Ġnot Ġcorrelated Ġwith Ġthat Ġof Ġconfidence Ġscoring , Ġsuggesting ĠE GL Ġidentifies Ġaspects Ġof Ġan Ġinstance Ġthat Ġconfidence Ġscores Ġcannot Ġcapture . </s> In ĠB IB REF 3 Ġ, ĠE GL Ġwas Ġapplied Ġto Ġactive Ġlearning Ġon Ġsequence Ġlabeling Ġtasks , Ġbut Ġour Ġwork Ġis Ġthe Ġfirst Ġwe Ġknow Ġof Ġto Ġapply ĠE GL Ġto Ġspeech Ġrecognition Ġin Ġparticular . ĠGrad ient - based Ġmethods Ġhave Ġalso Ġfound Ġapplications Ġoutside Ġactive Ġlearning . ĠFor Ġexample , ĠB IB REF 4 Ġsuggests Ġthat Ġin Ġsto ch astic Ġgradient Ġdescent , Ġsampling Ġtraining Ġinstances Ġwith Ġprobabilities Ġproportional Ġto Ġtheir Ġgradient Ġlengths Ġcan Ġspeed Ġup Ġconvergence . ĠFrom Ġthe Ġperspective Ġof Ġvariance Ġreduction , Ġthis Ġimportance Ġsampling Ġproblem Ġshares Ġmany Ġsimilarities Ġto Ġproblems Ġfound Ġin Ġactive Ġlearning . </s> Problem ĠForm ulation </s> Den ote ĠIN LINE FORM 0 Ġas Ġan Ġutter ance Ġand ĠIN LINE FORM 1 Ġthe Ġcorresponding Ġlabel Ġ( trans cription ). ĠA Ġspeech Ġrecognition Ġsystem Ġmodels Ġthe Ġconditional Ġdistribution ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġare Ġthe Ġparameters Ġin Ġthe Ġmodel , Ġand ĠIN LINE FORM 4 Ġis Ġtypically Ġimplemented Ġby Ġa ĠRec urrent ĠNeural ĠNetwork Ġ( R NN ). ĠA Ġtraining Ġset Ġis Ġa Ġcollection Ġof ĠIN LINE FORM 5 Ġpairs , Ġden oted Ġas ĠIN LINE FORM 6 Ġ. ĠThe Ġparameters Ġof Ġthe Ġmodel Ġare Ġestimated Ġby Ġminimizing Ġthe Ġnegative Ġlog - like lihood Ġon Ġthe Ġtraining Ġset : ĠDIS PLAY FORM 0 </s> Active Ġlearning Ġseeks Ġto Ġaugment Ġthe Ġtraining Ġset Ġwith Ġa Ġnew Ġset Ġof Ġutter ances Ġand Ġlabels ĠIN LINE FORM 0 Ġin Ġorder Ġto Ġachieve Ġgood Ġgeneral ization Ġon Ġa Ġheld - out Ġtest Ġdataset . ĠIn Ġmany Ġapplications , Ġthere Ġis Ġan Ġunl abel ed Ġpool ĠIN LINE FORM 1 Ġwhich Ġis Ġcostly Ġto Ġlabel Ġin Ġits Ġentirety . ĠIN LINE FORM 2 Ġis Ġquer ied Ġfor Ġthe ĠâĢ ľ most Ġinformative âĢ Ŀ Ġinstance ( s ) ĠIN LINE FORM 3 Ġ, Ġfor Ġwhich Ġthe Ġlabel ( s ) ĠIN LINE FORM 4 Ġare Ġthen Ġobtained . ĠWe Ġdiscuss Ġseveral Ġsuch Ġquery Ġstrategies Ġbelow . </s> Conf idence ĠScores </s> Conf idence Ġscoring Ġhas Ġbeen Ġused Ġextensively Ġas Ġa Ġproxy Ġfor Ġthe Ġinform at iveness Ġof Ġtraining Ġsamples . ĠSpecifically , Ġan ĠIN LINE FORM 0 Ġis Ġconsidered Ġinformative Ġif Ġthe Ġpredictions Ġare Ġuniformly Ġdistributed Ġover Ġall Ġthe Ġlabels ĠB IB REF 2 Ġ, Ġor Ġif Ġthe Ġbest Ġprediction Ġof Ġits Ġlabel Ġis Ġwith Ġlow Ġprobability ĠB IB REF 1 Ġ. ĠBy Ġtaking Ġthe Ġinstances Ġwhich ĠâĢ ľ conf use âĢ Ŀ Ġthe Ġmodel , Ġthese Ġmethods Ġmay Ġeffectively Ġexplore Ġunder - sam pled Ġregions Ġof Ġthe Ġinput Ġspace . </s> Ex pected ĠGrad ient ĠLength </s> Int uitive ly , Ġan Ġinstance Ġcan Ġbe Ġconsidered Ġinformative Ġif Ġit Ġresults Ġin Ġlarge Ġchanges Ġin Ġmodel Ġparameters . ĠA Ġnatural Ġmeasure Ġof Ġthe Ġchange Ġis Ġgradient Ġlength , ĠIN LINE FORM 0 Ġ. ĠMot ivated Ġby Ġthis Ġintuition , ĠEx pected ĠGrad ient ĠLength Ġ( EG L ) ĠB IB REF 3 Ġpicks Ġthe Ġinstances Ġexpected Ġto Ġhave Ġthe Ġlargest Ġgradient Ġlength . ĠSince Ġlabels Ġare Ġunknown Ġon ĠIN LINE FORM 1 Ġ, ĠE GL Ġcomp utes Ġthe Ġexpectation Ġof Ġthe Ġgradient Ġnorm Ġover Ġall Ġpossible Ġlabel ings . ĠB IB REF 3 Ġinterpre ts ĠE GL Ġas ĠâĢ ľ expected Ġmodel Ġchange âĢ Ŀ . ĠIn Ġthe Ġfollowing Ġsection , Ġwe Ġformal ize Ġthe Ġintuition Ġfor ĠE GL Ġand Ġshow Ġthat Ġit Ġfollows Ġnaturally Ġfrom Ġreducing Ġthe Ġvariance Ġof Ġan Ġestim ator . </s> Vari ance Ġin Ġthe ĠAs ym pt ote </s> Ass ume Ġthe Ġjoint Ġdistribution Ġof ĠIN LINE FORM 0 Ġhas Ġthe Ġfollowing Ġform , ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġtrue Ġparameter , Ġand ĠIN LINE FORM 1 Ġis Ġindependent Ġof ĠIN LINE FORM 2 Ġ. ĠBy Ġselecting Ġa Ġsubset Ġof Ġthe Ġtraining Ġdata , Ġwe Ġare Ġessentially Ġchoosing Ġanother Ġdistribution ĠIN LINE FORM 3 Ġso Ġthat Ġthe ĠIN LINE FORM 4 Ġpairs Ġare Ġdrawn Ġfrom ĠIN LINE FORM 5 </s> Stat istical Ġsignal Ġprocessing Ġtheory ĠB IB REF 5 Ġstates Ġthe Ġfollowing Ġas ym pt otic Ġdistribution Ġof ĠIN LINE FORM 0 Ġ, ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe ĠFisher ĠInformation ĠMatrix Ġwith Ġrespect Ġto ĠIN LINE FORM 1 Ġ. ĠUsing Ġfirst Ġorder Ġapproximation Ġat ĠIN LINE FORM 2 Ġ, Ġwe Ġhave Ġas ym pt ot ically , ĠDIS PLAY FORM 0 </s> E q . Ġ( ĠEQ REF 7 Ġ) Ġindicates Ġthat Ġto Ġreduce ĠIN LINE FORM 0 Ġon Ġtest Ġdata , Ġwe Ġneed Ġto Ġminimize Ġthe Ġexpected Ġvariance ĠIN LINE FORM 1 Ġover Ġthe Ġtest Ġset . ĠThis Ġis Ġcalled ĠFisher ĠInformation ĠRatio Ġcriteria Ġin ĠB IB REF 6 Ġ, Ġwhich Ġitself Ġis Ġhard Ġto Ġoptimize . ĠAn Ġeasier Ġsurrogate Ġis Ġto Ġmaximize ĠIN LINE FORM 2 Ġ. ĠSubst it uting ĠE q . Ġ( ĠEQ REF 5 Ġ) Ġinto ĠIN LINE FORM 3 Ġ, Ġwe Ġhave ĠIN LINE FORM 4 </s> which Ġis Ġequivalent Ġto ĠIN LINE FORM 0 </s> A Ġpractical Ġissue Ġis Ġthat Ġwe Ġdo Ġnot Ġknow ĠIN LINE FORM 0 Ġin Ġadvance . ĠWe Ġcould Ġinstead Ġsubstitute Ġan Ġestimate ĠIN LINE FORM 1 Ġfrom Ġa Ġpre - trained Ġmodel , Ġwhere Ġit Ġis Ġreasonable Ġto Ġassume Ġthe ĠIN LINE FORM 2 Ġto Ġbe Ġclose Ġto Ġthe Ġtrue ĠIN LINE FORM 3 Ġ. ĠThe Ġbatch Ġselection Ġthen Ġworks Ġby Ġtaking Ġthe Ġsamples Ġthat Ġhave Ġlargest Ġgradient Ġnorms , ĠDIS PLAY FORM 0 </s> For ĠR NN s , Ġthe Ġgrad ients Ġfor Ġeach Ġpotential Ġlabel Ġcan Ġbe Ġobtained Ġby Ġback - prop ag ation . ĠAnother Ġpractical Ġissue Ġis Ġthat ĠE GL Ġmarginal izes Ġover Ġall Ġpossible Ġlabel ings , Ġbut Ġin Ġspeech Ġrecognition , Ġthe Ġnumber Ġof Ġlabel ings Ġscales Ġexponentially Ġin Ġthe Ġnumber Ġof Ġtim est eps . ĠTherefore , Ġwe Ġonly Ġmarginal ize Ġover Ġthe ĠIN LINE FORM 0 Ġmost Ġprobable Ġlabel ings . ĠThey Ġare Ġobtained Ġby Ġbeam Ġsearch Ġdecoding , Ġas Ġin ĠB IB REF 7 Ġ. ĠThe ĠE GL Ġmethod Ġin ĠB IB REF 3 Ġis Ġalmost Ġthe Ġsame Ġas ĠE q . Ġ( ĠEQ REF 8 Ġ), Ġexcept Ġthe Ġgradient 's Ġnorm Ġis Ġnot Ġsquared Ġin ĠB IB REF 3 Ġ. </s> Here Ġwe Ġhave Ġprovided Ġa Ġmore Ġformal Ġcharacterization Ġof ĠE GL Ġto Ġcomplement Ġits Ġintuitive Ġinterpretation Ġas ĠâĢ ľ expected Ġmodel Ġchange âĢ Ŀ Ġin ĠB IB REF 3 Ġ. ĠFor Ġnot ational Ġconvenience , Ġwe Ġdenote ĠE q . Ġ( ĠEQ REF 8 Ġ) Ġas ĠE GL Ġin Ġsubsequent Ġsections . </s> Exper iments </s> We Ġempir ically Ġvalidate ĠE GL Ġon Ġspeech Ġrecognition Ġtasks . ĠIn Ġour Ġexperiments , Ġthe ĠR NN Ġtakes Ġin Ġspect rog rams Ġof Ġutter ances , Ġpassing Ġthem Ġthrough Ġtwo Ġ2 D - conv olution al Ġlayers , Ġfollowed Ġby Ġseven Ġbi - direction al Ġrecurrent Ġlayers Ġand Ġa Ġfully - connected Ġlayer Ġwith Ġsoft max Ġactivation . ĠAll Ġrecurrent Ġlayers Ġare Ġbatch Ġnormalized . ĠAt Ġeach Ġtim est ep , Ġthe Ġsoft max Ġactiv ations Ġgive Ġa Ġprobability Ġdistribution Ġover Ġthe Ġcharacters . ĠC TC Ġloss ĠB IB REF 8 Ġis Ġthen Ġcomputed Ġfrom Ġthe Ġtim est ep - wise Ġprobabilities . </s> A Ġbase Ġmodel , ĠIN LINE FORM 0 Ġ, Ġis Ġtrained Ġon Ġ190 Ġhours Ġ( ĠIN LINE FORM 1 Ġ100 K Ġinstances ) Ġof Ġtransc ribed Ġspeech Ġdata . ĠThen , Ġit Ġselects Ġa Ġsubset Ġof Ġa Ġ1 , 700 - hour Ġ( ĠIN LINE FORM 2 Ġ1 . 1 M Ġinstances ) Ġunl abel ed Ġdataset . ĠWe Ġquery Ġlabels Ġfor Ġthe Ġselected Ġsubset Ġand Ġincorporate Ġthem Ġinto Ġtraining . ĠLearning Ġrates Ġare Ġtuned Ġon Ġa Ġsmall Ġvalidation Ġset Ġof Ġ2048 Ġinstances . ĠThe Ġtrained Ġmodel Ġis Ġthen Ġtested Ġon Ġa Ġ156 - hour Ġ( ĠIN LINE FORM 3 Ġ100 K Ġinstances ) Ġtest Ġset Ġand Ġwe Ġreport ĠC TC Ġloss , ĠCharacter ĠError ĠRate Ġ( C ER ) Ġand ĠWord ĠError ĠRate Ġ( WER ). </s> The Ġconfidence Ġscore Ġmethods ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġcan Ġbe Ġeasily Ġextended Ġto Ġour Ġsetup . ĠSpecifically , Ġfrom Ġthe Ġprobabilities Ġover Ġthe Ġcharacters , Ġwe Ġcan Ġcompute Ġan Ġentropy Ġper Ġtim est ep Ġand Ġthen Ġaverage Ġthem . ĠThis Ġmethod Ġis Ġden oted Ġas Ġentropy . ĠWe Ġcould Ġalso Ġtake Ġthe Ġmost Ġlikely Ġprediction Ġand Ġcalculate Ġits ĠC TC Ġloss , Ġnormalized Ġby Ġnumber Ġof Ġtim est eps . ĠThis Ġmethod Ġis Ġden oted Ġas Ġp CT C Ġ( pred icted ĠC TC ) Ġin Ġthe Ġfollowing Ġsections . </s> We Ġimplement ĠE GL Ġby Ġmarginal izing Ġover Ġthe Ġmost Ġlikely Ġ100 Ġlabels , Ġand Ġcompare Ġit Ġwith : Ġ1 ) Ġa Ġrandom Ġselection Ġbaseline , Ġ2 ) Ġentropy , Ġand Ġ3 ) Ġp CT C . ĠUsing Ġthe Ġsame Ġbase Ġmodel , Ġeach Ġmethod Ġqueries Ġa Ġvariable Ġpercentage Ġof Ġthe Ġunl abel ed Ġdataset . ĠThe Ġqueries Ġare Ġthen Ġincluded Ġinto Ġtraining Ġset , Ġand Ġthe Ġmodel Ġcontinues Ġtraining Ġuntil Ġconvergence . ĠFig . ĠFIG REF 9 Ġreports Ġthe Ġmetrics Ġ( Ex act Ġvalues Ġare Ġreported Ġin ĠTable ĠT AB REF 12 Ġin Ġthe ĠAppendix ) Ġon Ġthe Ġtest Ġset Ġas Ġthe Ġquery Ġpercentage Ġvaries . ĠAll Ġthe Ġactive Ġlearning Ġmethods Ġoutper form Ġthe Ġrandom Ġbaseline . ĠMoreover , ĠE GL Ġshows Ġa Ġst eeper , Ġmore Ġrapid Ġreduction Ġin Ġerror Ġthan Ġall Ġother Ġapproaches . ĠSpecifically , Ġwhen Ġquer ying Ġ20 % Ġof Ġthe Ġunl abel ed Ġdataset , ĠE GL Ġhas Ġ11 . 58 % Ġlower ĠC ER Ġand Ġ11 . 09 % Ġlower ĠW ER Ġrelative Ġto Ġrandom . ĠThe Ġperformance Ġof ĠE GL Ġat Ġquer ying Ġ20 % Ġis Ġon Ġpar Ġwith Ġrandom Ġat Ġ40 %, Ġsuggesting Ġthat Ġusing ĠE GL Ġcan Ġlead Ġto Ġan Ġapproximate Ġ50 % Ġdecrease Ġin Ġdata Ġlabeling . </s> Similar ity Ġbetween ĠQuery ĠMethods </s> It Ġis Ġuseful Ġto Ġunderstand Ġhow Ġthe Ġthree Ġactive Ġlearning Ġmethods Ġdiffer Ġin Ġmeasuring Ġthe Ġinform at iveness Ġof Ġan Ġinstance . ĠTo Ġcompare Ġany Ġtwo Ġmethods , Ġwe Ġtake Ġrankings Ġof Ġinform at iveness Ġgiven Ġby Ġthese Ġtwo Ġmethods , Ġand Ġplot Ġthem Ġin Ġa Ġ2 - D Ġranking - vs - ranking Ġcoordinate Ġsystem . ĠA Ġplot Ġclose Ġto Ġthe Ġdiagonal Ġimplies Ġthat Ġthese Ġtwo Ġmethods Ġevaluate Ġinform at iveness Ġin Ġa Ġvery Ġsimilar Ġway . </s> Fig . ĠFIG REF 11 Ġshows Ġthe Ġranking - vs - ranking Ġplots Ġbetween Ġp CT C Ġand Ġentropy , ĠE GL Ġand Ġentropy . ĠWe Ġobserve Ġthat Ġp CT C Ġrankings Ġand Ġentropy Ġrankings Ġ( Fig . ĠFIG REF 11 Ġ) Ġare Ġvery Ġcorrelated . ĠThis Ġis Ġlikely Ġbecause Ġthey Ġare Ġboth Ġrelated Ġto Ġmodel Ġuncertainty . ĠIn Ġcontrast , ĠE GL Ġgives Ġvery Ġdifferent Ġrankings Ġfrom Ġentropy Ġ( Fig . ĠFIG REF 11 Ġ). ĠThis Ġsuggests ĠE GL Ġis Ġable Ġto Ġidentify Ġaspects Ġof Ġan Ġinstance Ġthat Ġuncertainty - based Ġmeasurements Ġcannot Ġcapture . </s> We Ġfurther Ġinvestigate Ġthe Ġsamples Ġfor Ġwhich ĠE GL Ġand Ġentropy Ġyield Ġvastly Ġdifferent Ġestimates Ġof Ġinform at iveness , Ġe . g ., Ġthe Ġelements Ġin Ġthe Ġred Ġcircle Ġin ĠFig . ĠFIG REF 11 Ġ. ĠThese Ġparticular Ġsamples Ġconsist Ġof Ġshort Ġutter ances Ġcontaining Ġsilence Ġ( with Ġbackground Ġnoise ) Ġor Ġfiller Ġwords . ĠFurther Ġinvestigation Ġis Ġrequired Ġto Ġunderstand Ġwhether Ġthese Ġsamples Ġare Ġnoisy Ġoutl iers Ġor Ġwhether Ġthey Ġare Ġin Ġfact Ġimportant Ġfor Ġtraining Ġend - to - end Ġspeech Ġrecognition Ġsystems . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġformally Ġexplained ĠE GL Ġfrom Ġa Ġvariance Ġreduction Ġperspective Ġand Ġexperiment ally Ġtested Ġits Ġperformance Ġon Ġend - to - end Ġspeech Ġrecognition Ġsystems . ĠInitial Ġexperiments Ġshow Ġa Ġnotable Ġgain Ġover Ġrandom Ġselection , Ġand Ġthat Ġit Ġoutper forms Ġconfidence Ġscore Ġmethods Ġused Ġin Ġthe ĠAS R Ġcommunity . ĠWe Ġalso Ġshow ĠE GL Ġmeasures Ġsample Ġinform at iveness Ġin Ġa Ġvery Ġdifferent Ġway Ġfrom Ġconfidence Ġscores , Ġgiving Ġrise Ġto Ġopen Ġresearch Ġquestions . ĠAll Ġthe Ġexperiments Ġreported Ġhere Ġquery Ġall Ġsamples Ġin Ġa Ġsingle Ġbatch . ĠIt Ġis Ġalso Ġworth Ġconsidering Ġthe Ġeffects Ġof Ġquer ying Ġsamples Ġin Ġa Ġsequential Ġmanner . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġfurther Ġvalidate Ġthe Ġapproach Ġwith Ġsequential Ġqueries Ġand Ġseek Ġto Ġmake Ġthe Ġinform at iveness Ġmeasure Ġrobust Ġto Ġoutl iers .
Introduction State-of-the-art automatic speech recognition (ASR) systems BIBREF0 have large model capacities and require significant quantities of training data to generalize. Labeling thousands of hours of audio, however, is expensive and time-consuming. A natural question to ask is how to achieve better generalization with fewer training examples. Active learning studies this problem by identifying and labeling only the most informative data, potentially reducing sample complexity. How much active learning can help in large-scale, end-to-end ASR systems, however, is still an open question. The speech recognition community has generally identified the informativeness of samples by calculating confidence scores. In particular, an utterance is considered informative if the most likely prediction has small probability BIBREF1 , or if the predictions are distributed very uniformly over the labels BIBREF2 . Though confidence-based measures work well in practice, less attention has been focused on gradient-based methods like Expected Gradient Length (EGL) BIBREF3 , where the informativeness is measured by the norm of the gradient incurred by the instance. EGL has previously been justified as intuitively measuring the expected change in a model's parameters BIBREF3 .We formalize this intuition from the perspective of asymptotic variance reduction, and experimentally, we show EGL to be superior to confidence-based methods on speech recognition tasks. Additionally, we observe that the ranking of samples scored by EGL is not correlated with that of confidence scoring, suggesting EGL identifies aspects of an instance that confidence scores cannot capture. In BIBREF3 , EGL was applied to active learning on sequence labeling tasks, but our work is the first we know of to apply EGL to speech recognition in particular. Gradient-based methods have also found applications outside active learning. For example, BIBREF4 suggests that in stochastic gradient descent, sampling training instances with probabilities proportional to their gradient lengths can speed up convergence. From the perspective of variance reduction, this importance sampling problem shares many similarities to problems found in active learning. Problem Formulation Denote INLINEFORM0 as an utterance and INLINEFORM1 the corresponding label (transcription). A speech recognition system models the conditional distribution INLINEFORM2 , where INLINEFORM3 are the parameters in the model, and INLINEFORM4 is typically implemented by a Recurrent Neural Network (RNN). A training set is a collection of INLINEFORM5 pairs, denoted as INLINEFORM6 . The parameters of the model are estimated by minimizing the negative log-likelihood on the training set: DISPLAYFORM0 Active learning seeks to augment the training set with a new set of utterances and labels INLINEFORM0 in order to achieve good generalization on a held-out test dataset. In many applications, there is an unlabeled pool INLINEFORM1 which is costly to label in its entirety. INLINEFORM2 is queried for the “most informative” instance(s) INLINEFORM3 , for which the label(s) INLINEFORM4 are then obtained. We discuss several such query strategies below. Confidence Scores Confidence scoring has been used extensively as a proxy for the informativeness of training samples. Specifically, an INLINEFORM0 is considered informative if the predictions are uniformly distributed over all the labels BIBREF2 , or if the best prediction of its label is with low probability BIBREF1 . By taking the instances which “confuse” the model, these methods may effectively explore under-sampled regions of the input space. Expected Gradient Length Intuitively, an instance can be considered informative if it results in large changes in model parameters. A natural measure of the change is gradient length, INLINEFORM0 . Motivated by this intuition, Expected Gradient Length (EGL) BIBREF3 picks the instances expected to have the largest gradient length. Since labels are unknown on INLINEFORM1 , EGL computes the expectation of the gradient norm over all possible labelings. BIBREF3 interprets EGL as “expected model change”. In the following section, we formalize the intuition for EGL and show that it follows naturally from reducing the variance of an estimator. Variance in the Asymptote Assume the joint distribution of INLINEFORM0 has the following form, DISPLAYFORM0 where INLINEFORM0 is the true parameter, and INLINEFORM1 is independent of INLINEFORM2 . By selecting a subset of the training data, we are essentially choosing another distribution INLINEFORM3 so that the INLINEFORM4 pairs are drawn from INLINEFORM5 Statistical signal processing theory BIBREF5 states the following asymptotic distribution of INLINEFORM0 , DISPLAYFORM0 where INLINEFORM0 is the Fisher Information Matrix with respect to INLINEFORM1 . Using first order approximation at INLINEFORM2 , we have asymptotically, DISPLAYFORM0 Eq. ( EQREF7 ) indicates that to reduce INLINEFORM0 on test data, we need to minimize the expected variance INLINEFORM1 over the test set. This is called Fisher Information Ratio criteria in BIBREF6 , which itself is hard to optimize. An easier surrogate is to maximize INLINEFORM2 . Substituting Eq. ( EQREF5 ) into INLINEFORM3 , we have INLINEFORM4 which is equivalent to INLINEFORM0 A practical issue is that we do not know INLINEFORM0 in advance. We could instead substitute an estimate INLINEFORM1 from a pre-trained model, where it is reasonable to assume the INLINEFORM2 to be close to the true INLINEFORM3 . The batch selection then works by taking the samples that have largest gradient norms, DISPLAYFORM0 For RNNs, the gradients for each potential label can be obtained by back-propagation. Another practical issue is that EGL marginalizes over all possible labelings, but in speech recognition, the number of labelings scales exponentially in the number of timesteps. Therefore, we only marginalize over the INLINEFORM0 most probable labelings. They are obtained by beam search decoding, as in BIBREF7 . The EGL method in BIBREF3 is almost the same as Eq. ( EQREF8 ), except the gradient's norm is not squared in BIBREF3 . Here we have provided a more formal characterization of EGL to complement its intuitive interpretation as “expected model change” in BIBREF3 . For notational convenience, we denote Eq. ( EQREF8 ) as EGL in subsequent sections. Experiments We empirically validate EGL on speech recognition tasks. In our experiments, the RNN takes in spectrograms of utterances, passing them through two 2D-convolutional layers, followed by seven bi-directional recurrent layers and a fully-connected layer with softmax activation. All recurrent layers are batch normalized. At each timestep, the softmax activations give a probability distribution over the characters. CTC loss BIBREF8 is then computed from the timestep-wise probabilities. A base model, INLINEFORM0 , is trained on 190 hours ( INLINEFORM1 100K instances) of transcribed speech data. Then, it selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset. We query labels for the selected subset and incorporate them into training. Learning rates are tuned on a small validation set of 2048 instances. The trained model is then tested on a 156-hour ( INLINEFORM3 100K instances) test set and we report CTC loss, Character Error Rate (CER) and Word Error Rate (WER). The confidence score methods BIBREF1 , BIBREF2 can be easily extended to our setup. Specifically, from the probabilities over the characters, we can compute an entropy per timestep and then average them. This method is denoted as entropy. We could also take the most likely prediction and calculate its CTC loss, normalized by number of timesteps. This method is denoted as pCTC (predicted CTC) in the following sections. We implement EGL by marginalizing over the most likely 100 labels, and compare it with: 1) a random selection baseline, 2) entropy, and 3) pCTC. Using the same base model, each method queries a variable percentage of the unlabeled dataset. The queries are then included into training set, and the model continues training until convergence. Fig. FIGREF9 reports the metrics (Exact values are reported in Table TABREF12 in the Appendix) on the test set as the query percentage varies. All the active learning methods outperform the random baseline. Moreover, EGL shows a steeper, more rapid reduction in error than all other approaches. Specifically, when querying 20% of the unlabeled dataset, EGL has 11.58% lower CER and 11.09% lower WER relative to random. The performance of EGL at querying 20% is on par with random at 40%, suggesting that using EGL can lead to an approximate 50% decrease in data labeling. Similarity between Query Methods It is useful to understand how the three active learning methods differ in measuring the informativeness of an instance. To compare any two methods, we take rankings of informativeness given by these two methods, and plot them in a 2-D ranking-vs-ranking coordinate system. A plot close to the diagonal implies that these two methods evaluate informativeness in a very similar way. Fig. FIGREF11 shows the ranking-vs-ranking plots between pCTC and entropy, EGL and entropy. We observe that pCTC rankings and entropy rankings (Fig. FIGREF11 ) are very correlated. This is likely because they are both related to model uncertainty. In contrast, EGL gives very different rankings from entropy (Fig. FIGREF11 ). This suggests EGL is able to identify aspects of an instance that uncertainty-based measurements cannot capture. We further investigate the samples for which EGL and entropy yield vastly different estimates of informativeness, e.g., the elements in the red circle in Fig. FIGREF11 . These particular samples consist of short utterances containing silence (with background noise) or filler words. Further investigation is required to understand whether these samples are noisy outliers or whether they are in fact important for training end-to-end speech recognition systems. Conclusion and Future Work We formally explained EGL from a variance reduction perspective and experimentally tested its performance on end-to-end speech recognition systems. Initial experiments show a notable gain over random selection, and that it outperforms confidence score methods used in the ASR community. We also show EGL measures sample informativeness in a very different way from confidence scores, giving rise to open research questions. All the experiments reported here query all samples in a single batch. It is also worth considering the effects of querying samples in a sequential manner. In the future, we will further validate the approach with sequential queries and seek to make the informativeness measure robust to outliers.
Which dataset do they use?
551457ed34ca7fc0878c85bc664b135c21059b58
Which Ġdataset Ġdo Ġthey Ġuse ?
190 Ġhours Ġ( ĠIN LINE FORM 1 Ġ100 K Ġinstances )
<s> Which Ġdataset Ġdo Ġthey Ġuse ? </s> Introduction </s> State - of - the - art Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems ĠB IB REF 0 Ġhave Ġlarge Ġmodel Ġcapacities Ġand Ġrequire Ġsignificant Ġquantities Ġof Ġtraining Ġdata Ġto Ġgeneral ize . ĠLab eling Ġthousands Ġof Ġhours Ġof Ġaudio , Ġhowever , Ġis Ġexpensive Ġand Ġtime - consuming . ĠA Ġnatural Ġquestion Ġto Ġask Ġis Ġhow Ġto Ġachieve Ġbetter Ġgeneral ization Ġwith Ġfewer Ġtraining Ġexamples . ĠActive Ġlearning Ġstudies Ġthis Ġproblem Ġby Ġidentifying Ġand Ġlabeling Ġonly Ġthe Ġmost Ġinformative Ġdata , Ġpotentially Ġreducing Ġsample Ġcomplexity . ĠHow Ġmuch Ġactive Ġlearning Ġcan Ġhelp Ġin Ġlarge - scale , Ġend - to - end ĠAS R Ġsystems , Ġhowever , Ġis Ġstill Ġan Ġopen Ġquestion . </s> The Ġspeech Ġrecognition Ġcommunity Ġhas Ġgenerally Ġidentified Ġthe Ġinform at iveness Ġof Ġsamples Ġby Ġcalculating Ġconfidence Ġscores . ĠIn Ġparticular , Ġan Ġutter ance Ġis Ġconsidered Ġinformative Ġif Ġthe Ġmost Ġlikely Ġprediction Ġhas Ġsmall Ġprobability ĠB IB REF 1 Ġ, Ġor Ġif Ġthe Ġpredictions Ġare Ġdistributed Ġvery Ġuniformly Ġover Ġthe Ġlabels ĠB IB REF 2 Ġ. ĠThough Ġconfidence - based Ġmeasures Ġwork Ġwell Ġin Ġpractice , Ġless Ġattention Ġhas Ġbeen Ġfocused Ġon Ġgradient - based Ġmethods Ġlike ĠEx pected ĠGrad ient ĠLength Ġ( EG L ) ĠB IB REF 3 Ġ, Ġwhere Ġthe Ġinform at iveness Ġis Ġmeasured Ġby Ġthe Ġnorm Ġof Ġthe Ġgradient Ġincurred Ġby Ġthe Ġinstance . ĠE GL Ġhas Ġpreviously Ġbeen Ġjustified Ġas Ġintuitive ly Ġmeasuring Ġthe Ġexpected Ġchange Ġin Ġa Ġmodel 's Ġparameters ĠB IB REF 3 Ġ. We Ġformal ize Ġthis Ġintuition Ġfrom Ġthe Ġperspective Ġof Ġas ym pt otic Ġvariance Ġreduction , Ġand Ġexperiment ally , Ġwe Ġshow ĠE GL Ġto Ġbe Ġsuperior Ġto Ġconfidence - based Ġmethods Ġon Ġspeech Ġrecognition Ġtasks . ĠAdditionally , Ġwe Ġobserve Ġthat Ġthe Ġranking Ġof Ġsamples Ġscored Ġby ĠE GL Ġis Ġnot Ġcorrelated Ġwith Ġthat Ġof Ġconfidence Ġscoring , Ġsuggesting ĠE GL Ġidentifies Ġaspects Ġof Ġan Ġinstance Ġthat Ġconfidence Ġscores Ġcannot Ġcapture . </s> In ĠB IB REF 3 Ġ, ĠE GL Ġwas Ġapplied Ġto Ġactive Ġlearning Ġon Ġsequence Ġlabeling Ġtasks , Ġbut Ġour Ġwork Ġis Ġthe Ġfirst Ġwe Ġknow Ġof Ġto Ġapply ĠE GL Ġto Ġspeech Ġrecognition Ġin Ġparticular . ĠGrad ient - based Ġmethods Ġhave Ġalso Ġfound Ġapplications Ġoutside Ġactive Ġlearning . ĠFor Ġexample , ĠB IB REF 4 Ġsuggests Ġthat Ġin Ġsto ch astic Ġgradient Ġdescent , Ġsampling Ġtraining Ġinstances Ġwith Ġprobabilities Ġproportional Ġto Ġtheir Ġgradient Ġlengths Ġcan Ġspeed Ġup Ġconvergence . ĠFrom Ġthe Ġperspective Ġof Ġvariance Ġreduction , Ġthis Ġimportance Ġsampling Ġproblem Ġshares Ġmany Ġsimilarities Ġto Ġproblems Ġfound Ġin Ġactive Ġlearning . </s> Problem ĠForm ulation </s> Den ote ĠIN LINE FORM 0 Ġas Ġan Ġutter ance Ġand ĠIN LINE FORM 1 Ġthe Ġcorresponding Ġlabel Ġ( trans cription ). ĠA Ġspeech Ġrecognition Ġsystem Ġmodels Ġthe Ġconditional Ġdistribution ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġare Ġthe Ġparameters Ġin Ġthe Ġmodel , Ġand ĠIN LINE FORM 4 Ġis Ġtypically Ġimplemented Ġby Ġa ĠRec urrent ĠNeural ĠNetwork Ġ( R NN ). ĠA Ġtraining Ġset Ġis Ġa Ġcollection Ġof ĠIN LINE FORM 5 Ġpairs , Ġden oted Ġas ĠIN LINE FORM 6 Ġ. ĠThe Ġparameters Ġof Ġthe Ġmodel Ġare Ġestimated Ġby Ġminimizing Ġthe Ġnegative Ġlog - like lihood Ġon Ġthe Ġtraining Ġset : ĠDIS PLAY FORM 0 </s> Active Ġlearning Ġseeks Ġto Ġaugment Ġthe Ġtraining Ġset Ġwith Ġa Ġnew Ġset Ġof Ġutter ances Ġand Ġlabels ĠIN LINE FORM 0 Ġin Ġorder Ġto Ġachieve Ġgood Ġgeneral ization Ġon Ġa Ġheld - out Ġtest Ġdataset . ĠIn Ġmany Ġapplications , Ġthere Ġis Ġan Ġunl abel ed Ġpool ĠIN LINE FORM 1 Ġwhich Ġis Ġcostly Ġto Ġlabel Ġin Ġits Ġentirety . ĠIN LINE FORM 2 Ġis Ġquer ied Ġfor Ġthe ĠâĢ ľ most Ġinformative âĢ Ŀ Ġinstance ( s ) ĠIN LINE FORM 3 Ġ, Ġfor Ġwhich Ġthe Ġlabel ( s ) ĠIN LINE FORM 4 Ġare Ġthen Ġobtained . ĠWe Ġdiscuss Ġseveral Ġsuch Ġquery Ġstrategies Ġbelow . </s> Conf idence ĠScores </s> Conf idence Ġscoring Ġhas Ġbeen Ġused Ġextensively Ġas Ġa Ġproxy Ġfor Ġthe Ġinform at iveness Ġof Ġtraining Ġsamples . ĠSpecifically , Ġan ĠIN LINE FORM 0 Ġis Ġconsidered Ġinformative Ġif Ġthe Ġpredictions Ġare Ġuniformly Ġdistributed Ġover Ġall Ġthe Ġlabels ĠB IB REF 2 Ġ, Ġor Ġif Ġthe Ġbest Ġprediction Ġof Ġits Ġlabel Ġis Ġwith Ġlow Ġprobability ĠB IB REF 1 Ġ. ĠBy Ġtaking Ġthe Ġinstances Ġwhich ĠâĢ ľ conf use âĢ Ŀ Ġthe Ġmodel , Ġthese Ġmethods Ġmay Ġeffectively Ġexplore Ġunder - sam pled Ġregions Ġof Ġthe Ġinput Ġspace . </s> Ex pected ĠGrad ient ĠLength </s> Int uitive ly , Ġan Ġinstance Ġcan Ġbe Ġconsidered Ġinformative Ġif Ġit Ġresults Ġin Ġlarge Ġchanges Ġin Ġmodel Ġparameters . ĠA Ġnatural Ġmeasure Ġof Ġthe Ġchange Ġis Ġgradient Ġlength , ĠIN LINE FORM 0 Ġ. ĠMot ivated Ġby Ġthis Ġintuition , ĠEx pected ĠGrad ient ĠLength Ġ( EG L ) ĠB IB REF 3 Ġpicks Ġthe Ġinstances Ġexpected Ġto Ġhave Ġthe Ġlargest Ġgradient Ġlength . ĠSince Ġlabels Ġare Ġunknown Ġon ĠIN LINE FORM 1 Ġ, ĠE GL Ġcomp utes Ġthe Ġexpectation Ġof Ġthe Ġgradient Ġnorm Ġover Ġall Ġpossible Ġlabel ings . ĠB IB REF 3 Ġinterpre ts ĠE GL Ġas ĠâĢ ľ expected Ġmodel Ġchange âĢ Ŀ . ĠIn Ġthe Ġfollowing Ġsection , Ġwe Ġformal ize Ġthe Ġintuition Ġfor ĠE GL Ġand Ġshow Ġthat Ġit Ġfollows Ġnaturally Ġfrom Ġreducing Ġthe Ġvariance Ġof Ġan Ġestim ator . </s> Vari ance Ġin Ġthe ĠAs ym pt ote </s> Ass ume Ġthe Ġjoint Ġdistribution Ġof ĠIN LINE FORM 0 Ġhas Ġthe Ġfollowing Ġform , ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġtrue Ġparameter , Ġand ĠIN LINE FORM 1 Ġis Ġindependent Ġof ĠIN LINE FORM 2 Ġ. ĠBy Ġselecting Ġa Ġsubset Ġof Ġthe Ġtraining Ġdata , Ġwe Ġare Ġessentially Ġchoosing Ġanother Ġdistribution ĠIN LINE FORM 3 Ġso Ġthat Ġthe ĠIN LINE FORM 4 Ġpairs Ġare Ġdrawn Ġfrom ĠIN LINE FORM 5 </s> Stat istical Ġsignal Ġprocessing Ġtheory ĠB IB REF 5 Ġstates Ġthe Ġfollowing Ġas ym pt otic Ġdistribution Ġof ĠIN LINE FORM 0 Ġ, ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe ĠFisher ĠInformation ĠMatrix Ġwith Ġrespect Ġto ĠIN LINE FORM 1 Ġ. ĠUsing Ġfirst Ġorder Ġapproximation Ġat ĠIN LINE FORM 2 Ġ, Ġwe Ġhave Ġas ym pt ot ically , ĠDIS PLAY FORM 0 </s> E q . Ġ( ĠEQ REF 7 Ġ) Ġindicates Ġthat Ġto Ġreduce ĠIN LINE FORM 0 Ġon Ġtest Ġdata , Ġwe Ġneed Ġto Ġminimize Ġthe Ġexpected Ġvariance ĠIN LINE FORM 1 Ġover Ġthe Ġtest Ġset . ĠThis Ġis Ġcalled ĠFisher ĠInformation ĠRatio Ġcriteria Ġin ĠB IB REF 6 Ġ, Ġwhich Ġitself Ġis Ġhard Ġto Ġoptimize . ĠAn Ġeasier Ġsurrogate Ġis Ġto Ġmaximize ĠIN LINE FORM 2 Ġ. ĠSubst it uting ĠE q . Ġ( ĠEQ REF 5 Ġ) Ġinto ĠIN LINE FORM 3 Ġ, Ġwe Ġhave ĠIN LINE FORM 4 </s> which Ġis Ġequivalent Ġto ĠIN LINE FORM 0 </s> A Ġpractical Ġissue Ġis Ġthat Ġwe Ġdo Ġnot Ġknow ĠIN LINE FORM 0 Ġin Ġadvance . ĠWe Ġcould Ġinstead Ġsubstitute Ġan Ġestimate ĠIN LINE FORM 1 Ġfrom Ġa Ġpre - trained Ġmodel , Ġwhere Ġit Ġis Ġreasonable Ġto Ġassume Ġthe ĠIN LINE FORM 2 Ġto Ġbe Ġclose Ġto Ġthe Ġtrue ĠIN LINE FORM 3 Ġ. ĠThe Ġbatch Ġselection Ġthen Ġworks Ġby Ġtaking Ġthe Ġsamples Ġthat Ġhave Ġlargest Ġgradient Ġnorms , ĠDIS PLAY FORM 0 </s> For ĠR NN s , Ġthe Ġgrad ients Ġfor Ġeach Ġpotential Ġlabel Ġcan Ġbe Ġobtained Ġby Ġback - prop ag ation . ĠAnother Ġpractical Ġissue Ġis Ġthat ĠE GL Ġmarginal izes Ġover Ġall Ġpossible Ġlabel ings , Ġbut Ġin Ġspeech Ġrecognition , Ġthe Ġnumber Ġof Ġlabel ings Ġscales Ġexponentially Ġin Ġthe Ġnumber Ġof Ġtim est eps . ĠTherefore , Ġwe Ġonly Ġmarginal ize Ġover Ġthe ĠIN LINE FORM 0 Ġmost Ġprobable Ġlabel ings . ĠThey Ġare Ġobtained Ġby Ġbeam Ġsearch Ġdecoding , Ġas Ġin ĠB IB REF 7 Ġ. ĠThe ĠE GL Ġmethod Ġin ĠB IB REF 3 Ġis Ġalmost Ġthe Ġsame Ġas ĠE q . Ġ( ĠEQ REF 8 Ġ), Ġexcept Ġthe Ġgradient 's Ġnorm Ġis Ġnot Ġsquared Ġin ĠB IB REF 3 Ġ. </s> Here Ġwe Ġhave Ġprovided Ġa Ġmore Ġformal Ġcharacterization Ġof ĠE GL Ġto Ġcomplement Ġits Ġintuitive Ġinterpretation Ġas ĠâĢ ľ expected Ġmodel Ġchange âĢ Ŀ Ġin ĠB IB REF 3 Ġ. ĠFor Ġnot ational Ġconvenience , Ġwe Ġdenote ĠE q . Ġ( ĠEQ REF 8 Ġ) Ġas ĠE GL Ġin Ġsubsequent Ġsections . </s> Exper iments </s> We Ġempir ically Ġvalidate ĠE GL Ġon Ġspeech Ġrecognition Ġtasks . ĠIn Ġour Ġexperiments , Ġthe ĠR NN Ġtakes Ġin Ġspect rog rams Ġof Ġutter ances , Ġpassing Ġthem Ġthrough Ġtwo Ġ2 D - conv olution al Ġlayers , Ġfollowed Ġby Ġseven Ġbi - direction al Ġrecurrent Ġlayers Ġand Ġa Ġfully - connected Ġlayer Ġwith Ġsoft max Ġactivation . ĠAll Ġrecurrent Ġlayers Ġare Ġbatch Ġnormalized . ĠAt Ġeach Ġtim est ep , Ġthe Ġsoft max Ġactiv ations Ġgive Ġa Ġprobability Ġdistribution Ġover Ġthe Ġcharacters . ĠC TC Ġloss ĠB IB REF 8 Ġis Ġthen Ġcomputed Ġfrom Ġthe Ġtim est ep - wise Ġprobabilities . </s> A Ġbase Ġmodel , ĠIN LINE FORM 0 Ġ, Ġis Ġtrained Ġon Ġ190 Ġhours Ġ( ĠIN LINE FORM 1 Ġ100 K Ġinstances ) Ġof Ġtransc ribed Ġspeech Ġdata . ĠThen , Ġit Ġselects Ġa Ġsubset Ġof Ġa Ġ1 , 700 - hour Ġ( ĠIN LINE FORM 2 Ġ1 . 1 M Ġinstances ) Ġunl abel ed Ġdataset . ĠWe Ġquery Ġlabels Ġfor Ġthe Ġselected Ġsubset Ġand Ġincorporate Ġthem Ġinto Ġtraining . ĠLearning Ġrates Ġare Ġtuned Ġon Ġa Ġsmall Ġvalidation Ġset Ġof Ġ2048 Ġinstances . ĠThe Ġtrained Ġmodel Ġis Ġthen Ġtested Ġon Ġa Ġ156 - hour Ġ( ĠIN LINE FORM 3 Ġ100 K Ġinstances ) Ġtest Ġset Ġand Ġwe Ġreport ĠC TC Ġloss , ĠCharacter ĠError ĠRate Ġ( C ER ) Ġand ĠWord ĠError ĠRate Ġ( WER ). </s> The Ġconfidence Ġscore Ġmethods ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġcan Ġbe Ġeasily Ġextended Ġto Ġour Ġsetup . ĠSpecifically , Ġfrom Ġthe Ġprobabilities Ġover Ġthe Ġcharacters , Ġwe Ġcan Ġcompute Ġan Ġentropy Ġper Ġtim est ep Ġand Ġthen Ġaverage Ġthem . ĠThis Ġmethod Ġis Ġden oted Ġas Ġentropy . ĠWe Ġcould Ġalso Ġtake Ġthe Ġmost Ġlikely Ġprediction Ġand Ġcalculate Ġits ĠC TC Ġloss , Ġnormalized Ġby Ġnumber Ġof Ġtim est eps . ĠThis Ġmethod Ġis Ġden oted Ġas Ġp CT C Ġ( pred icted ĠC TC ) Ġin Ġthe Ġfollowing Ġsections . </s> We Ġimplement ĠE GL Ġby Ġmarginal izing Ġover Ġthe Ġmost Ġlikely Ġ100 Ġlabels , Ġand Ġcompare Ġit Ġwith : Ġ1 ) Ġa Ġrandom Ġselection Ġbaseline , Ġ2 ) Ġentropy , Ġand Ġ3 ) Ġp CT C . ĠUsing Ġthe Ġsame Ġbase Ġmodel , Ġeach Ġmethod Ġqueries Ġa Ġvariable Ġpercentage Ġof Ġthe Ġunl abel ed Ġdataset . ĠThe Ġqueries Ġare Ġthen Ġincluded Ġinto Ġtraining Ġset , Ġand Ġthe Ġmodel Ġcontinues Ġtraining Ġuntil Ġconvergence . ĠFig . ĠFIG REF 9 Ġreports Ġthe Ġmetrics Ġ( Ex act Ġvalues Ġare Ġreported Ġin ĠTable ĠT AB REF 12 Ġin Ġthe ĠAppendix ) Ġon Ġthe Ġtest Ġset Ġas Ġthe Ġquery Ġpercentage Ġvaries . ĠAll Ġthe Ġactive Ġlearning Ġmethods Ġoutper form Ġthe Ġrandom Ġbaseline . ĠMoreover , ĠE GL Ġshows Ġa Ġst eeper , Ġmore Ġrapid Ġreduction Ġin Ġerror Ġthan Ġall Ġother Ġapproaches . ĠSpecifically , Ġwhen Ġquer ying Ġ20 % Ġof Ġthe Ġunl abel ed Ġdataset , ĠE GL Ġhas Ġ11 . 58 % Ġlower ĠC ER Ġand Ġ11 . 09 % Ġlower ĠW ER Ġrelative Ġto Ġrandom . ĠThe Ġperformance Ġof ĠE GL Ġat Ġquer ying Ġ20 % Ġis Ġon Ġpar Ġwith Ġrandom Ġat Ġ40 %, Ġsuggesting Ġthat Ġusing ĠE GL Ġcan Ġlead Ġto Ġan Ġapproximate Ġ50 % Ġdecrease Ġin Ġdata Ġlabeling . </s> Similar ity Ġbetween ĠQuery ĠMethods </s> It Ġis Ġuseful Ġto Ġunderstand Ġhow Ġthe Ġthree Ġactive Ġlearning Ġmethods Ġdiffer Ġin Ġmeasuring Ġthe Ġinform at iveness Ġof Ġan Ġinstance . ĠTo Ġcompare Ġany Ġtwo Ġmethods , Ġwe Ġtake Ġrankings Ġof Ġinform at iveness Ġgiven Ġby Ġthese Ġtwo Ġmethods , Ġand Ġplot Ġthem Ġin Ġa Ġ2 - D Ġranking - vs - ranking Ġcoordinate Ġsystem . ĠA Ġplot Ġclose Ġto Ġthe Ġdiagonal Ġimplies Ġthat Ġthese Ġtwo Ġmethods Ġevaluate Ġinform at iveness Ġin Ġa Ġvery Ġsimilar Ġway . </s> Fig . ĠFIG REF 11 Ġshows Ġthe Ġranking - vs - ranking Ġplots Ġbetween Ġp CT C Ġand Ġentropy , ĠE GL Ġand Ġentropy . ĠWe Ġobserve Ġthat Ġp CT C Ġrankings Ġand Ġentropy Ġrankings Ġ( Fig . ĠFIG REF 11 Ġ) Ġare Ġvery Ġcorrelated . ĠThis Ġis Ġlikely Ġbecause Ġthey Ġare Ġboth Ġrelated Ġto Ġmodel Ġuncertainty . ĠIn Ġcontrast , ĠE GL Ġgives Ġvery Ġdifferent Ġrankings Ġfrom Ġentropy Ġ( Fig . ĠFIG REF 11 Ġ). ĠThis Ġsuggests ĠE GL Ġis Ġable Ġto Ġidentify Ġaspects Ġof Ġan Ġinstance Ġthat Ġuncertainty - based Ġmeasurements Ġcannot Ġcapture . </s> We Ġfurther Ġinvestigate Ġthe Ġsamples Ġfor Ġwhich ĠE GL Ġand Ġentropy Ġyield Ġvastly Ġdifferent Ġestimates Ġof Ġinform at iveness , Ġe . g ., Ġthe Ġelements Ġin Ġthe Ġred Ġcircle Ġin ĠFig . ĠFIG REF 11 Ġ. ĠThese Ġparticular Ġsamples Ġconsist Ġof Ġshort Ġutter ances Ġcontaining Ġsilence Ġ( with Ġbackground Ġnoise ) Ġor Ġfiller Ġwords . ĠFurther Ġinvestigation Ġis Ġrequired Ġto Ġunderstand Ġwhether Ġthese Ġsamples Ġare Ġnoisy Ġoutl iers Ġor Ġwhether Ġthey Ġare Ġin Ġfact Ġimportant Ġfor Ġtraining Ġend - to - end Ġspeech Ġrecognition Ġsystems . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġformally Ġexplained ĠE GL Ġfrom Ġa Ġvariance Ġreduction Ġperspective Ġand Ġexperiment ally Ġtested Ġits Ġperformance Ġon Ġend - to - end Ġspeech Ġrecognition Ġsystems . ĠInitial Ġexperiments Ġshow Ġa Ġnotable Ġgain Ġover Ġrandom Ġselection , Ġand Ġthat Ġit Ġoutper forms Ġconfidence Ġscore Ġmethods Ġused Ġin Ġthe ĠAS R Ġcommunity . ĠWe Ġalso Ġshow ĠE GL Ġmeasures Ġsample Ġinform at iveness Ġin Ġa Ġvery Ġdifferent Ġway Ġfrom Ġconfidence Ġscores , Ġgiving Ġrise Ġto Ġopen Ġresearch Ġquestions . ĠAll Ġthe Ġexperiments Ġreported Ġhere Ġquery Ġall Ġsamples Ġin Ġa Ġsingle Ġbatch . ĠIt Ġis Ġalso Ġworth Ġconsidering Ġthe Ġeffects Ġof Ġquer ying Ġsamples Ġin Ġa Ġsequential Ġmanner . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġfurther Ġvalidate Ġthe Ġapproach Ġwith Ġsequential Ġqueries Ġand Ġseek Ġto Ġmake Ġthe Ġinform at iveness Ġmeasure Ġrobust Ġto Ġoutl iers .