Unanswerable
NONE
Unanswerable
NONE
2002.01320
Introduction </s> End - to - end Ġspeech - to - text Ġtranslation Ġ( ST ) Ġhas Ġattracted Ġmuch Ġattention Ġrecently ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġgiven Ġits Ġsimplicity Ġagainst Ġcasc ading Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġand Ġmachine Ġtranslation Ġ( MT ) Ġsystems . ĠThe Ġlack Ġof Ġlabeled Ġdata , Ġhowever , Ġhas Ġbecome Ġa Ġmajor Ġblocker Ġfor Ġbrid ging Ġthe Ġperformance Ġgaps Ġbetween Ġend - to - end Ġmodels Ġand Ġcasc ading Ġsystems . ĠSeveral Ġcorpor a Ġhave Ġbeen Ġdeveloped Ġin Ġrecent Ġyears . Ġpost 2013 impro ved Ġintroduced Ġa Ġ38 - hour ĠSpanish - English ĠST Ġcorpus Ġby Ġaugment ing Ġthe Ġtranscripts Ġof Ġthe ĠFisher Ġand ĠCall home Ġcorpor a Ġwith ĠEnglish Ġtranslations . Ġdi - gang i - et al - 2019 - must Ġcreated Ġthe Ġlargest ĠST Ġcorpus Ġto Ġdate Ġfrom ĠTED Ġtalks Ġbut Ġthe Ġlanguage Ġpairs Ġinvolved Ġare Ġout Ġof ĠEnglish Ġonly . Ġbe il har z 2019 lib riv ox deen Ġcreated Ġa Ġ110 - hour ĠGerman - English ĠST Ġcorpus Ġfrom ĠLib ri V ox Ġaudi ob ooks . Ġgod ard - et al - 2018 - low Ġcreated Ġa ĠMob oshi - French ĠST Ġcorpus Ġas Ġpart Ġof Ġa Ġrare Ġlanguage Ġdocumentation Ġeffort . Ġw old ey oh annis Ġprovided Ġan ĠAm har ic - English ĠST Ġcorpus Ġin Ġthe Ġtourism Ġdomain . Ġbo ito 2019 mass Ġcreated Ġa Ġmult ilingual ĠST Ġcorpus Ġinvolving Ġ8 Ġlanguages Ġfrom Ġa Ġmult ilingual Ġspeech Ġcorpus Ġbased Ġon ĠBible Ġreadings ĠB IB REF 7 . ĠPrevious Ġwork Ġeither Ġinvolves Ġlanguage Ġpairs Ġout Ġof ĠEnglish , Ġvery Ġspecific Ġdomains , Ġvery Ġlow Ġresource Ġlanguages Ġor Ġa Ġlimited Ġset Ġof Ġlanguage Ġpairs . ĠThis Ġlimits Ġthe Ġscope Ġof Ġstudy , Ġincluding Ġthe Ġlatest Ġexplor ations Ġon Ġend - to - end Ġmult ilingual ĠST ĠB IB REF 8 , ĠB IB REF 9 . ĠOur Ġwork Ġis Ġmostly Ġsimilar Ġand Ġconcurrent Ġto Ġir anz os n che z 2019 euro par l st Ġwho Ġcreated Ġa Ġmult ilingual ĠST Ġcorpus Ġfrom Ġthe ĠEuropean ĠParliament Ġproceedings . ĠThe Ġcorpus Ġwe Ġintroduce Ġhas Ġlarger Ġspeech Ġd urations Ġand Ġmore Ġtranslation Ġtokens . ĠIt Ġis Ġdivers ified Ġwith Ġmultiple Ġspeakers Ġper Ġtranscript / translation . ĠFinally , Ġwe Ġprovide Ġadditional Ġout - of - domain Ġtest Ġsets . </s> In Ġthis Ġpaper , Ġwe Ġintroduce ĠCo Vo ST , Ġa Ġmult ilingual ĠST Ġcorpus Ġbased Ġon ĠCommon ĠVoice ĠB IB REF 10 Ġfor Ġ11 Ġlanguages Ġinto ĠEnglish , Ġdivers ified Ġwith Ġover Ġ11 , 000 Ġspeakers Ġand Ġover Ġ60 Ġaccents . ĠIt Ġincludes Ġa Ġtotal Ġ7 08 Ġhours Ġof ĠFrench Ġ( Fr ), ĠGerman Ġ( De ), ĠDutch Ġ( N l ), ĠRussian Ġ( Ru ), ĠSpanish Ġ( Es ), ĠItalian Ġ( It ), ĠTurkish Ġ( Tr ), ĠPersian Ġ( Fa ), ĠSwedish Ġ( S v ), ĠMongol ian Ġ( M n ) Ġand ĠChinese Ġ( Z h ) Ġspeeches , Ġwith ĠFrench Ġand ĠGerman Ġones Ġhaving Ġthe Ġlargest Ġd urations Ġamong Ġexisting Ġpublic Ġcorpor a . ĠWe Ġalso Ġcollect Ġan Ġadditional Ġevaluation Ġcorpus Ġfrom ĠT ato eb a Ġfor ĠFrench , ĠGerman , ĠDutch , ĠRussian Ġand ĠSpanish , Ġresulting Ġin Ġa Ġtotal Ġof Ġ9 . 3 Ġhours Ġof Ġspeech . ĠBoth Ġcorpor a Ġare Ġcreated Ġat Ġthe Ġsentence Ġlevel Ġand Ġdo Ġnot Ġrequire Ġadditional Ġalign ments Ġor Ġsegment ation . ĠUsing Ġthe Ġofficial ĠCommon ĠVoice Ġtrain - development - test Ġsplit , Ġwe Ġalso Ġprovide Ġbaseline Ġmodels , Ġincluding , Ġto Ġour Ġknowledge , Ġthe Ġfirst Ġend - to - end Ġmany - to - one Ġmult ilingual ĠST Ġmodels . ĠCo Vo ST Ġis Ġreleased Ġunder ĠCC 0 Ġlicense Ġand Ġfree Ġto Ġuse . ĠThe ĠT ato eb a Ġevaluation Ġsamples Ġare Ġalso Ġavailable Ġunder Ġfriendly ĠCC Ġlicenses . ĠAll Ġthe Ġdata Ġcan Ġbe Ġacquired Ġat Ġhttps :// github . com / facebook research / c ov ost . </s> Data ĠCollection Ġand ĠProcessing Ġ: :: ĠCommon ĠVoice Ġ( Co Vo ) </s> Common ĠVoice ĠB IB REF 10 Ġis Ġa Ġcrowds ourcing Ġspeech Ġrecognition Ġcorpus Ġwith Ġan Ġopen ĠCC 0 Ġlicense . ĠContribut ors Ġrecord Ġvoice Ġclips Ġby Ġreading Ġfrom Ġa Ġbank Ġof Ġdonated Ġsentences . ĠEach Ġvoice Ġclip Ġwas Ġvalidated Ġby Ġat Ġleast Ġtwo Ġother Ġusers . ĠMost Ġof Ġthe Ġsentences Ġare Ġcovered Ġby Ġmultiple Ġspeakers , Ġwith Ġpotentially Ġdifferent Ġgenders , Ġage Ġgroups Ġor Ġaccents . </s> Raw ĠCo Vo Ġdata Ġcontains Ġsamples Ġthat Ġpassed Ġvalidation Ġas Ġwell Ġas Ġthose Ġthat Ġdid Ġnot . ĠTo Ġbuild ĠCo Vo ST , Ġwe Ġonly Ġuse Ġthe Ġformer Ġone Ġand Ġreuse Ġthe Ġofficial Ġtrain - development - test Ġpartition Ġof Ġthe Ġvalidated Ġdata . ĠAs Ġof ĠJanuary Ġ2020 , Ġthe Ġlatest ĠCo Vo Ġ2019 - 06 - 12 Ġrelease Ġincludes Ġ29 Ġlanguages . ĠCo Vo ST Ġis Ġcurrently Ġbuilt Ġon Ġthat Ġrelease Ġand Ġcovers Ġthe Ġfollowing Ġ11 Ġlanguages : ĠFrench , ĠGerman , ĠDutch , ĠRussian , ĠSpanish , ĠItalian , ĠTurkish , ĠPersian , ĠSwedish , ĠMongol ian Ġand ĠChinese . </s> Val idated Ġtranscripts Ġwere Ġsent Ġto Ġprofessional Ġtransl ators . ĠNote Ġthat Ġthe Ġtransl ators Ġhad Ġaccess Ġto Ġthe Ġtranscripts Ġbut Ġnot Ġthe Ġcorresponding Ġvoice Ġclips Ġsince Ġclips Ġwould Ġnot Ġcarry Ġadditional Ġinformation . ĠSince Ġtranscripts Ġwere Ġdupl icated Ġdue Ġto Ġmultiple Ġspeakers , Ġwe Ġded u pl icated Ġthe Ġtranscripts Ġbefore Ġsending Ġthem Ġto Ġtransl ators . ĠAs Ġa Ġresult , Ġdifferent Ġvoice Ġclips Ġof Ġthe Ġsame Ġcontent Ġ( trans cript ) Ġwill Ġhave Ġidentical Ġtranslations Ġin ĠCo Vo ST Ġfor Ġtrain , Ġdevelopment Ġand Ġtest Ġsplits . </s> In Ġorder Ġto Ġcontrol Ġthe Ġquality Ġof Ġthe Ġprofessional Ġtranslations , Ġwe Ġapplied Ġvarious Ġsanity Ġchecks Ġto Ġthe Ġtranslations ĠB IB REF 11 . Ġ1 ) ĠFor ĠGerman - English , ĠFrench - English Ġand ĠRussian - English Ġtranslations , Ġwe Ġcomputed Ġsentence - level ĠB LE U ĠB IB REF 12 Ġwith Ġthe ĠNL T K ĠB IB REF 13 Ġimplementation Ġbetween Ġthe Ġhuman Ġtranslations Ġand Ġthe Ġautomatic Ġtranslations Ġproduced Ġby Ġa Ġstate - of - the - art Ġsystem ĠB IB REF 14 Ġ( the ĠFrench - English Ġsystem Ġwas Ġa ĠTrans former Ġbig ĠB IB REF 15 Ġseparately Ġtrained Ġon ĠW MT 14 ). ĠWe Ġapplied Ġthis Ġmethod Ġto Ġthese Ġthree Ġlanguage Ġpairs Ġonly Ġas Ġwe Ġare Ġconfident Ġabout Ġthe Ġquality Ġof Ġthe Ġcorresponding Ġsystems . ĠTrans lations Ġwith Ġa Ġscore Ġthat Ġwas Ġtoo Ġlow Ġwere Ġmanually Ġinspected Ġand Ġsent Ġback Ġto Ġthe Ġtransl ators Ġwhen Ġneeded . Ġ2 ) ĠWe Ġmanually Ġinspected Ġexamples Ġwhere Ġthe Ġsource Ġtranscript Ġwas Ġidentical Ġto Ġthe Ġtranslation . Ġ3 ) ĠWe Ġmeasured Ġthe Ġperplex ity Ġof Ġthe Ġtranslations Ġusing Ġa Ġlanguage Ġmodel Ġtrained Ġon Ġa Ġlarge Ġamount Ġof Ġclean Ġmon oling ual Ġdata ĠB IB REF 14 . ĠWe Ġmanually Ġinspected Ġexamples Ġwhere Ġthe Ġtranslation Ġhad Ġa Ġhigh Ġperplex ity Ġand Ġsent Ġthem Ġback Ġto Ġtransl ators Ġaccordingly . Ġ4 ) ĠWe Ġcomputed Ġthe Ġratio Ġof ĠEnglish Ġcharacters Ġin Ġthe Ġtranslations . ĠWe Ġmanually Ġinspected Ġexamples Ġwith Ġa Ġlow Ġratio Ġand Ġsent Ġthem Ġback Ġto Ġtransl ators Ġaccordingly . Ġ5 ) ĠFinally , Ġwe Ġused ĠViz Se q ĠB IB REF 16 Ġto Ġcalculate Ġsimilarity Ġscores Ġbetween Ġtranscripts Ġand Ġtranslations Ġbased Ġon ĠL AS ER Ġcross - ling ual Ġsentence Ġembed d ings ĠB IB REF 17 . ĠSam ples Ġwith Ġlow Ġscores Ġwere Ġmanually Ġinspected Ġand Ġsent Ġback Ġfor Ġtranslation Ġwhen Ġneeded . </s> We Ġalso Ġsanity Ġcheck Ġthe Ġoverl aps Ġof Ġtrain , Ġdevelopment Ġand Ġtest Ġsets Ġin Ġterms Ġof Ġtranscripts Ġand Ġvoice Ġclips Ġ( via ĠMD 5 Ġfile Ġhashing ), Ġand Ġconfirm Ġthey Ġare Ġtotally Ġdis j oint . </s> Data ĠCollection Ġand ĠProcessing Ġ: :: ĠT ato eb a Ġ( TT ) </s> T ato eb a Ġ( TT ) Ġis Ġa Ġcommunity Ġbuilt Ġlanguage Ġlearning Ġcorpus Ġhaving Ġsentences Ġaligned Ġacross Ġmultiple Ġlanguages Ġwith Ġthe Ġcorresponding Ġspeech Ġpartially Ġavailable . ĠIts Ġsentences Ġare Ġon Ġaverage Ġshorter Ġthan Ġthose Ġin ĠCo Vo ST Ġ( see Ġalso ĠTable ĠT AB REF 2 ) Ġgiven Ġthe Ġoriginal Ġpurpose Ġof Ġlanguage Ġlearning . ĠSent ences Ġin ĠTT Ġare Ġlicensed Ġunder ĠCC ĠBY Ġ2 . 0 ĠFR Ġand Ġpart Ġof Ġthe Ġspeeches Ġare Ġavailable Ġunder Ġvarious ĠCC Ġlicenses . </s> We Ġconstruct Ġan Ġevaluation Ġset Ġfrom ĠTT Ġ( for ĠFrench , ĠGerman , ĠDutch , ĠRussian Ġand ĠSpanish ) Ġas Ġa Ġcomplement Ġto ĠCo Vo ST Ġdevelopment Ġand Ġtest Ġsets . ĠWe Ġcollect Ġ( speech , Ġtranscript , ĠEnglish Ġtranslation ) Ġtriple ts Ġfor Ġthe Ġ5 Ġlanguages Ġand Ġdo Ġnot Ġinclude Ġthose Ġwhose Ġspeech Ġhas Ġa Ġbroken ĠURL Ġor Ġis Ġnot ĠCC Ġlicensed . ĠWe Ġfurther Ġfilter Ġthese Ġsamples Ġby Ġsentence Ġlengths Ġ( minimum Ġ4 Ġwords Ġincluding Ġpunct uations ) Ġto Ġreduce Ġthe Ġportion Ġof Ġshort Ġsentences . ĠThis Ġmakes Ġthe Ġresulting Ġevaluation Ġset Ġcloser Ġto Ġreal - world Ġscenarios Ġand Ġmore Ġchallenging . </s> We Ġrun Ġthe Ġsame Ġquality Ġchecks Ġfor ĠTT Ġas Ġfor ĠCo Vo ST Ġbut Ġwe Ġdo Ġnot Ġfind Ġpoor Ġquality Ġtranslations Ġaccording Ġto Ġour Ġcriteria . ĠFinally , Ġwe Ġreport Ġthe Ġoverlap Ġbetween ĠCo Vo Ġtranscripts Ġand ĠTT Ġsentences Ġin ĠTable ĠT AB REF 5 . ĠWe Ġfound Ġa Ġminimal Ġoverlap , Ġwhich Ġmakes Ġthe ĠTT Ġevaluation Ġset Ġa Ġsuitable Ġadditional Ġtest Ġset Ġwhen Ġtraining Ġon ĠCo Vo ST . </s> Data ĠAnalysis Ġ: :: ĠBasic ĠStatistics </s> Basic Ġstatistics Ġfor ĠCo Vo ST Ġand ĠTT Ġare Ġlisted Ġin ĠTable ĠT AB REF 2 Ġincluding Ġ( unique ) Ġsentence Ġcounts , Ġspeech Ġd urations , Ġspeaker Ġdemographics Ġ( part ially Ġavailable ) Ġas Ġwell Ġas Ġvocabulary Ġand Ġtoken Ġstatistics Ġ( based Ġon ĠMoses - token ized Ġsentences Ġby Ġsac re M oses ) Ġon Ġboth Ġtranscripts Ġand Ġtranslations . ĠWe Ġsee Ġthat ĠCo Vo ST Ġhas Ġover Ġ327 Ġhours Ġof ĠGerman Ġspeeches Ġand Ġover Ġ171 Ġhours Ġof ĠFrench Ġspeeches , Ġwhich , Ġto Ġour Ġknowledge , Ġcorresponds Ġto Ġthe Ġlargest Ġcorpus Ġamong Ġexisting Ġpublic ĠST Ġcorpor a Ġ( the Ġsecond Ġlargest Ġis Ġ110 Ġhours ĠB IB REF 18 Ġfor ĠGerman Ġand Ġ38 Ġhours ĠB IB REF 19 Ġfor ĠFrench ). ĠMoreover , ĠCo Vo ST Ġhas Ġa Ġtotal Ġof Ġ18 Ġhours Ġof ĠDutch Ġspeeches , Ġto Ġour Ġknowledge , Ġcontributing Ġthe Ġfirst Ġpublic ĠDutch ĠST Ġresource . ĠCo Vo ST Ġalso Ġhas Ġaround Ġ27 - hour ĠRussian Ġspeeches , Ġ37 - hour ĠItalian Ġspeeches Ġand Ġ67 - hour ĠPersian Ġspeeches , Ġwhich Ġis Ġ1 . 8 Ġtimes , Ġ2 . 5 Ġtimes Ġand Ġ13 . 3 Ġtimes Ġof Ġthe Ġprevious Ġlargest Ġpublic Ġone ĠB IB REF 7 . ĠMost Ġof Ġthe Ġsentences Ġ( trans cript s ) Ġin ĠCo Vo ST Ġare Ġcovered Ġby Ġmultiple Ġspeakers Ġwith Ġpotentially Ġdifferent Ġaccents , Ġresulting Ġin Ġa Ġrich Ġdiversity Ġin Ġthe Ġspeeches . ĠFor Ġexample , Ġthere Ġare Ġover Ġ1 , 000 Ġspeakers Ġand Ġover Ġ10 Ġaccents Ġin Ġthe ĠFrench Ġand ĠGerman Ġdevelopment Ġ/ Ġtest Ġsets . ĠThis Ġenables Ġgood Ġcoverage Ġof Ġspeech Ġvariations Ġin Ġboth Ġmodel Ġtraining Ġand Ġevaluation . </s> Data ĠAnalysis Ġ: :: ĠSpeaker ĠDiversity </s> As Ġwe Ġcan Ġsee Ġfrom ĠTable ĠT AB REF 2 , ĠCo Vo ST Ġis Ġdivers ified Ġwith Ġa Ġrich Ġset Ġof Ġspeakers Ġand Ġaccents . ĠWe Ġfurther Ġinspect Ġthe Ġspeaker Ġdemographics Ġin Ġterms Ġof Ġsample Ġdistributions Ġwith Ġrespect Ġto Ġspeaker Ġcounts , Ġaccent Ġcounts Ġand Ġage Ġgroups , Ġwhich Ġis Ġshown Ġin ĠFigure ĠFIG REF 6 , ĠFIG REF 7 Ġand ĠFIG REF 8 . ĠWe Ġobserve Ġthat Ġfor Ġ8 Ġof Ġthe Ġ11 Ġlanguages , Ġat Ġleast Ġ60 % Ġof Ġthe Ġsentences Ġ( trans cript s ) Ġare Ġcovered Ġby Ġmultiple Ġspeakers . ĠOver Ġ80 % Ġof Ġthe ĠFrench Ġsentences Ġhave Ġat Ġleast Ġ3 Ġspeakers . ĠAnd Ġfor ĠGerman Ġsentences , Ġeven Ġover Ġ90 % Ġof Ġthem Ġhave Ġat Ġleast Ġ5 Ġspeakers . ĠSimilarly , Ġwe Ġsee Ġthat Ġa Ġlarge Ġportion Ġof Ġsentences Ġare Ġspoken Ġin Ġmultiple Ġaccents Ġfor ĠFrench , ĠGerman , ĠDutch Ġand ĠSpanish . ĠSpe akers Ġof Ġeach Ġlanguage Ġalso Ġspread Ġwidely Ġacross Ġdifferent Ġage Ġgroups Ġ( below Ġ20 , Ġ20 s , Ġ30 s , Ġ40 s , Ġ50 s , Ġ60 s Ġand Ġ70 s ). </s> Bas eline ĠResults </s> We Ġprovide Ġbas elines Ġusing Ġthe Ġofficial Ġtrain - development - test Ġsplit Ġon Ġthe Ġfollowing Ġtasks : Ġautomatic Ġspeech Ġrecognition Ġ( AS R ), Ġmachine Ġtranslation Ġ( MT ) Ġand Ġspeech Ġtranslation Ġ( ST ). </s> Bas eline ĠResults Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠData ĠPre processing </s> We Ġconvert Ġraw ĠMP 3 Ġaudio Ġfiles Ġfrom ĠCo Vo Ġand ĠTT Ġinto Ġmono - channel Ġwave forms , Ġand Ġdowns ample Ġthem Ġto Ġ16 , 000 ĠHz . ĠFor Ġtranscripts Ġand Ġtranslations , Ġwe Ġnormal ize Ġthe Ġpunct uation , Ġwe Ġtoken ize Ġthe Ġtext Ġwith Ġsac re M oses Ġand Ġlower case Ġit . ĠFor Ġtranscripts , Ġwe Ġfurther Ġremove Ġall Ġpunct uation Ġmarkers Ġexcept Ġfor Ġapost rop hes . ĠWe Ġuse Ġcharacter Ġvoc ab ular ies Ġon Ġall Ġthe Ġtasks , Ġwith Ġ100 % Ġcoverage Ġof Ġall Ġthe Ġcharacters . ĠPrel iminary Ġexperimentation Ġshowed Ġthat Ġcharacter Ġvoc ab ular ies Ġprovided Ġmore Ġstable Ġtraining Ġthan ĠB PE . ĠFor ĠMT , Ġthe Ġvocabulary Ġis Ġcreated Ġjointly Ġon Ġboth Ġtranscripts Ġand Ġtranslations . ĠWe Ġextract Ġ80 - channel Ġlog - mel Ġfilter bank Ġfeatures , Ġcomputed Ġwith Ġa Ġ25 ms Ġwindow Ġsize Ġand Ġ10 ms Ġwindow Ġshift Ġusing Ġtorch audio . ĠThe Ġfeatures Ġare Ġnormalized Ġto Ġ0 Ġmean Ġand Ġ1 . 0 Ġstandard Ġdeviation . ĠWe Ġremove Ġsamples Ġhaving Ġmore Ġthan Ġ3 , 000 Ġframes Ġor Ġmore Ġthan Ġ256 Ġcharacters Ġfor ĠGPU Ġmemory Ġefficiency Ġ( less Ġthan Ġ25 Ġsamples Ġare Ġremoved Ġfor Ġall Ġlanguages ). </s> Bas eline ĠResults Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠModel ĠTraining </s> Our ĠAS R Ġand ĠST Ġmodels Ġfollow Ġthe Ġarchitecture Ġin Ġber ard 2018 end , Ġbut Ġhave Ġ3 Ġdec oder Ġlayers Ġlike Ġthat Ġin Ġp ino 2019 har ness ing . ĠFor ĠMT , Ġwe Ġuse Ġa ĠTrans former Ġbase Ġarchitecture ĠB IB REF 15 , Ġbut Ġwith Ġ3 Ġenc oder Ġlayers , Ġ3 Ġdec oder Ġlayers Ġand Ġ0 . 3 Ġdrop out . ĠWe Ġuse Ġa Ġbatch Ġsize Ġof Ġ10 , 000 Ġframes Ġfor ĠAS R Ġand ĠST , Ġand Ġa Ġbatch Ġsize Ġof Ġ4 , 000 Ġtokens Ġfor ĠMT . ĠWe Ġtrain Ġall Ġmodels Ġusing ĠFair seq ĠB IB REF 20 Ġfor Ġup Ġto Ġ200 , 000 Ġupdates . ĠWe Ġuse ĠSpec Aug ment ĠB IB REF 21 Ġfor ĠAS R Ġand ĠST Ġto Ġalleviate Ġover fitting . </s> Bas eline ĠResults Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠIn ference Ġand ĠEvaluation </s> We Ġuse Ġa Ġbeam Ġsize Ġof Ġ5 Ġfor Ġall Ġmodels . ĠWe Ġuse Ġthe Ġbest Ġcheckpoint Ġby Ġvalidation Ġloss Ġfor ĠMT , Ġand Ġaverage Ġthe Ġlast Ġ5 Ġcheckpoints Ġfor ĠAS R Ġand ĠST . ĠFor ĠMT Ġand ĠST , Ġwe Ġreport Ġcase - ins ensitive Ġtoken ized ĠB LE U ĠB IB REF 22 Ġusing Ġsac re BLE U ĠB IB REF 23 . ĠFor ĠAS R , Ġwe Ġreport Ġword Ġerror Ġrate Ġ( WER ) Ġand Ġcharacter Ġerror Ġrate Ġ( C ER ) Ġusing ĠViz Se q . </s> Bas eline ĠResults Ġ: :: ĠAutomatic ĠSpeech ĠRecogn ition Ġ( AS R ) </s> For Ġsimplicity , Ġwe Ġuse Ġthe Ġsame Ġmodel Ġarchitecture Ġfor ĠAS R Ġand ĠST , Ġalthough Ġwe Ġdo Ġnot Ġleverage ĠAS R Ġmodels Ġto Ġpret rain ĠST Ġmodel Ġenc od ers Ġlater . ĠTable ĠT AB REF 18 Ġshows Ġthe Ġword Ġerror Ġrate Ġ( WER ) Ġand Ġcharacter Ġerror Ġrate Ġ( C ER ) Ġfor ĠAS R Ġmodels . ĠWe Ġsee Ġthat ĠFrench Ġand ĠGerman Ġperform Ġthe Ġbest Ġgiven Ġthey Ġare Ġthe Ġtwo Ġhighest Ġresource Ġlanguages Ġin ĠCo Vo ST . ĠThe Ġother Ġlanguages Ġare Ġrelatively Ġlow Ġresource Ġ( especially ĠTurkish Ġand ĠSwedish ) Ġand Ġthe ĠAS R Ġmodels Ġare Ġhaving Ġdifficulties Ġto Ġlearn Ġfrom Ġthis Ġdata . </s> Bas eline ĠResults Ġ: :: ĠMachine ĠTranslation Ġ( MT ) </s> MT Ġmodels Ġtake Ġtranscripts Ġ( without Ġpunct uation ) Ġas Ġinputs Ġand Ġoutputs Ġtranslations Ġ( with Ġpunct uation ). ĠFor Ġsimplicity , Ġwe Ġdo Ġnot Ġchange Ġthe Ġtext Ġpre processing Ġmethods Ġfor ĠMT Ġto Ġcorrect Ġthis Ġmismatch . ĠMoreover , Ġthis Ġmismatch Ġalso Ġexists Ġin Ġcasc ading ĠST Ġsystems , Ġwhere ĠMT Ġmodel Ġinputs Ġare Ġthe Ġoutputs Ġof Ġan ĠAS R Ġmodel . ĠTable ĠT AB REF 20 Ġshows Ġthe ĠB LE U Ġscores Ġof ĠMT Ġmodels . ĠWe Ġnotice Ġthat Ġthe Ġresults Ġare Ġconsistent Ġwith Ġwhat Ġwe Ġsee Ġfrom ĠAS R Ġmodels . ĠFor Ġexample Ġthanks Ġto Ġabundant Ġtraining Ġdata , ĠFrench Ġhas Ġa Ġdecent ĠB LE U Ġscore Ġof Ġ29 . 8 / 25 . 4 . ĠGerman Ġdoesn 't Ġperform Ġwell , Ġbecause Ġof Ġless Ġrichness Ġof Ġcontent Ġ( trans cript s ). ĠThe Ġother Ġlanguages Ġare Ġlow Ġresource Ġin ĠCo Vo ST Ġand Ġit Ġis Ġdifficult Ġto Ġtrain Ġdecent Ġmodels Ġwithout Ġadditional Ġdata Ġor Ġpre - training Ġtechniques . </s> Bas eline ĠResults Ġ: :: ĠSpeech ĠTranslation Ġ( ST ) </s> Co Vo ST Ġis Ġa Ġmany - to - one Ġmult ilingual ĠST Ġcorpus . ĠWhile Ġend - to - end Ġone - to - many Ġand Ġmany - to - many Ġmult ilingual ĠST Ġmodels Ġhave Ġbeen Ġexplored Ġvery Ġrecently ĠB IB REF 8 , ĠB IB REF 9 , Ġmany - to - one Ġmult ilingual Ġmodels , Ġto Ġour Ġknowledge , Ġhave Ġnot . ĠWe Ġhence Ġuse ĠCo Vo ST Ġto Ġexamine Ġthis Ġsetting . ĠTable ĠT AB REF 22 Ġand ĠT AB REF 23 Ġshow Ġthe ĠB LE U Ġscores Ġfor Ġboth Ġbilingual Ġand Ġmult ilingual Ġend - to - end ĠST Ġmodels Ġtrained Ġon ĠCo Vo ST . ĠWe Ġobserve Ġthat Ġcombining Ġspeeches Ġfrom Ġmultiple Ġlanguages Ġis Ġconsistently Ġbringing Ġgains Ġto Ġlow - resource Ġlanguages Ġ( all Ġbesides ĠFrench Ġand ĠGerman ). ĠThis Ġincludes Ġcombinations Ġof Ġdistant Ġlanguages , Ġsuch Ġas ĠRu + Fr , ĠTr + Fr Ġand ĠZh + Fr . ĠMoreover , Ġsome Ġcombinations Ġdo Ġbring Ġgains Ġto Ġhigh - resource Ġlanguage Ġ( French ) Ġas Ġwell : ĠEs + Fr , ĠTr + Fr Ġand ĠMn + Fr . ĠWe Ġsimply Ġprovide Ġthe Ġmost Ġbasic Ġmany - to - one Ġmult ilingual Ġbas elines Ġhere , Ġand Ġleave Ġthe Ġfull Ġexploration Ġof Ġthe Ġbest Ġconfigurations Ġto Ġfuture Ġwork . ĠFinally , Ġwe Ġnote Ġthat Ġfor Ġsome Ġlanguage Ġpairs , Ġabsolute ĠB LE U Ġnumbers Ġare Ġrelatively Ġlow Ġas Ġwe Ġrestrict Ġmodel Ġtraining Ġto Ġthe Ġsupervised Ġdata . ĠWe Ġencourage Ġthe Ġcommunity Ġto Ġimprove Ġupon Ġthose Ġbas elines , Ġfor Ġexample Ġby Ġleveraging Ġsemi - super vised Ġtraining . </s> Bas eline ĠResults Ġ: :: ĠMulti - Spe aker ĠEvaluation </s> In ĠCo Vo ST , Ġlarge Ġportion Ġof Ġtranscripts Ġare Ġcovered Ġby Ġmultiple Ġspeakers Ġwith Ġdifferent Ġgenders , Ġaccents Ġand Ġage Ġgroups . ĠBesides Ġthe Ġstandard Ġcorpus - level ĠB LE U Ġscores , Ġwe Ġalso Ġwant Ġto Ġevaluate Ġmodel Ġoutput Ġvariance Ġon Ġthe Ġsame Ġcontent Ġ( trans cript ) Ġbut Ġdifferent Ġspeakers . ĠWe Ġhence Ġpropose Ġto Ġgroup Ġsamples Ġ( and Ġtheir Ġsentence ĠB LE U Ġscores ) Ġby Ġtranscript , Ġand Ġthen Ġcalculate Ġaverage Ġper - group Ġmean Ġand Ġaverage Ġcoefficient Ġof Ġvariation Ġdefined Ġas Ġfollows : </s> and </s> where Ġ$ G $ Ġis Ġthe Ġset Ġof Ġsentence ĠB LE U Ġscores Ġgrouped Ġby Ġtranscript Ġand Ġ$ G ^ {\ prime Ġ} Ġ= Ġ\ l brace Ġg Ġ| Ġg \ in ĠG , Ġ| g | > 1 , Ġ\ text rm Ġ{ Me an } ( g ) Ġ> Ġ0 Ġ\ r brace Ġ$ . </s> $ \ text rm Ġ{ BLE U } _{ MS } $ Ġprovides Ġa Ġnormalized Ġquality Ġscore Ġas Ġoppose Ġto Ġcorpus - level ĠB LE U Ġor Ġun normal ized Ġaverage Ġof Ġsentence ĠB LE U . ĠAnd Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġis Ġa Ġstandardized Ġmeasure Ġof Ġmodel Ġstability Ġagainst Ġdifferent Ġspeakers Ġ( the Ġlower Ġthe Ġbetter ). ĠTable ĠT AB REF 24 Ġshows Ġthe Ġ$\ text rm Ġ{ BLE U } _{ MS } $ Ġand Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġof Ġour ĠST Ġmodels Ġon ĠCo Vo ST Ġtest Ġset . ĠWe Ġsee Ġthat ĠGerman Ġand ĠPersian Ġhave Ġthe Ġworst Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġ( le ast Ġstable ) Ġgiven Ġtheir Ġrich Ġspeaker Ġdiversity Ġin Ġthe Ġtest Ġset Ġand Ġrelatively Ġsmall Ġtrain Ġset Ġ( see Ġalso ĠFigure ĠFIG REF 6 Ġand ĠTable ĠT AB REF 2 ). ĠDutch Ġalso Ġhas Ġpoor Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġbecause Ġof Ġthe Ġlack Ġof Ġtraining Ġdata . ĠMult ilingual Ġmodels Ġare Ġconsist antly Ġmore Ġstable Ġon Ġlow - resource Ġlanguages . ĠRu + Fr , ĠTr + Fr , ĠFa + Fr Ġand ĠZh + Fr Ġeven Ġhave Ġbetter Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġthan Ġall Ġindividual Ġlanguages . </s> Conclusion </s> We Ġintroduce Ġa Ġmult ilingual Ġspeech - to - text Ġtranslation Ġcorpus , ĠCo Vo ST , Ġfor Ġ11 Ġlanguages Ġinto ĠEnglish , Ġdivers ified Ġwith Ġover Ġ11 , 000 Ġspeakers Ġand Ġover Ġ60 Ġaccents . ĠWe Ġalso Ġprovide Ġbaseline Ġresults , Ġincluding , Ġto Ġour Ġknowledge , Ġthe Ġfirst Ġend - to - end Ġmany - to - one Ġmult ilingual Ġmodel Ġfor Ġspoken Ġlanguage Ġtranslation . ĠCo Vo ST Ġis Ġfree Ġto Ġuse Ġwith Ġa ĠCC 0 Ġlicense , Ġand Ġthe Ġadditional ĠT ato eb a Ġevaluation Ġsamples Ġare Ġalso ĠCC - licensed .
Introduction End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems. The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems. Several corpora have been developed in recent years. post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations. di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only. beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks. godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort. woldeyohannis provided an Amharic-English ST corpus in the tourism domain. boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7. Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs. This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9. Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings. The corpus we introduce has larger speech durations and more translation tokens. It is diversified with multiple speakers per transcript/translation. Finally, we provide additional out-of-domain test sets. In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents. It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora. We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9.3 hours of speech. Both corpora are created at the sentence level and do not require additional alignments or segmentation. Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models. CoVoST is released under CC0 license and free to use. The Tatoeba evaluation samples are also available under friendly CC licenses. All the data can be acquired at https://github.com/facebookresearch/covost. Data Collection and Processing ::: Common Voice (CoVo) Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license. Contributors record voice clips by reading from a bank of donated sentences. Each voice clip was validated by at least two other users. Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents. Raw CoVo data contains samples that passed validation as well as those that did not. To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data. As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages. CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese. Validated transcripts were sent to professional translators. Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information. Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators. As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits. In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11. 1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14). We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems. Translations with a score that was too low were manually inspected and sent back to the translators when needed. 2) We manually inspected examples where the source transcript was identical to the translation. 3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14. We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly. 4) We computed the ratio of English characters in the translations. We manually inspected examples with a low ratio and sent them back to translators accordingly. 5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17. Samples with low scores were manually inspected and sent back for translation when needed. We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint. Data Collection and Processing ::: Tatoeba (TT) Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available. Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning. Sentences in TT are licensed under CC BY 2.0 FR and part of the speeches are available under various CC licenses. We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets. We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed. We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences. This makes the resulting evaluation set closer to real-world scenarios and more challenging. We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria. Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5. We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST. Data Analysis ::: Basic Statistics Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations. We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French). Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource. CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1.8 times, 2.5 times and 13.3 times of the previous largest public one BIBREF7. Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches. For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets. This enables good coverage of speech variations in both model training and evaluation. Data Analysis ::: Speaker Diversity As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents. We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8. We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers. Over 80% of the French sentences have at least 3 speakers. And for German sentences, even over 90% of them have at least 5 speakers. Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish. Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s). Baseline Results We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST). Baseline Results ::: Experimental Settings ::: Data Preprocessing We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz. For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it. For transcripts, we further remove all punctuation markers except for apostrophes. We use character vocabularies on all the tasks, with 100% coverage of all the characters. Preliminary experimentation showed that character vocabularies provided more stable training than BPE. For MT, the vocabulary is created jointly on both transcripts and translations. We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio. The features are normalized to 0 mean and 1.0 standard deviation. We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages). Baseline Results ::: Experimental Settings ::: Model Training Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing. For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0.3 dropout. We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT. We train all models using Fairseq BIBREF20 for up to 200,000 updates. We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting. Baseline Results ::: Experimental Settings ::: Inference and Evaluation We use a beam size of 5 for all models. We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST. For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23. For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq. Baseline Results ::: Automatic Speech Recognition (ASR) For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later. Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models. We see that French and German perform the best given they are the two highest resource languages in CoVoST. The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data. Baseline Results ::: Machine Translation (MT) MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation). For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch. Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model. Table TABREF20 shows the BLEU scores of MT models. We notice that the results are consistent with what we see from ASR models. For example thanks to abundant training data, French has a decent BLEU score of 29.8/25.4. German doesn't perform well, because of less richness of content (transcripts). The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques. Baseline Results ::: Speech Translation (ST) CoVoST is a many-to-one multilingual ST corpus. While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not. We hence use CoVoST to examine this setting. Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST. We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German). This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr. Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr. We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work. Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data. We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training. Baseline Results ::: Multi-Speaker Evaluation In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups. Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers. We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows: and where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $. $\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU. And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better). Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set. We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2). Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data. Multilingual models are consistantly more stable on low-resource languages. Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages. Conclusion We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents. We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation. CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed.
How big is Augmented LibriSpeech dataset?
ffde866b1203a01580eb33237a0bb9da71c75ecf
How Ġbig Ġis ĠAug mented ĠLib ri Spe ech Ġdataset ?
Un answer able
<s> How Ġbig Ġis ĠAug mented ĠLib ri Spe ech Ġdataset ? </s> Introduction </s> End - to - end Ġspeech - to - text Ġtranslation Ġ( ST ) Ġhas Ġattracted Ġmuch Ġattention Ġrecently ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġgiven Ġits Ġsimplicity Ġagainst Ġcasc ading Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġand Ġmachine Ġtranslation Ġ( MT ) Ġsystems . ĠThe Ġlack Ġof Ġlabeled Ġdata , Ġhowever , Ġhas Ġbecome Ġa Ġmajor Ġblocker Ġfor Ġbrid ging Ġthe Ġperformance Ġgaps Ġbetween Ġend - to - end Ġmodels Ġand Ġcasc ading Ġsystems . ĠSeveral Ġcorpor a Ġhave Ġbeen Ġdeveloped Ġin Ġrecent Ġyears . Ġpost 2013 impro ved Ġintroduced Ġa Ġ38 - hour ĠSpanish - English ĠST Ġcorpus Ġby Ġaugment ing Ġthe Ġtranscripts Ġof Ġthe ĠFisher Ġand ĠCall home Ġcorpor a Ġwith ĠEnglish Ġtranslations . Ġdi - gang i - et al - 2019 - must Ġcreated Ġthe Ġlargest ĠST Ġcorpus Ġto Ġdate Ġfrom ĠTED Ġtalks Ġbut Ġthe Ġlanguage Ġpairs Ġinvolved Ġare Ġout Ġof ĠEnglish Ġonly . Ġbe il har z 2019 lib riv ox deen Ġcreated Ġa Ġ110 - hour ĠGerman - English ĠST Ġcorpus Ġfrom ĠLib ri V ox Ġaudi ob ooks . Ġgod ard - et al - 2018 - low Ġcreated Ġa ĠMob oshi - French ĠST Ġcorpus Ġas Ġpart Ġof Ġa Ġrare Ġlanguage Ġdocumentation Ġeffort . Ġw old ey oh annis Ġprovided Ġan ĠAm har ic - English ĠST Ġcorpus Ġin Ġthe Ġtourism Ġdomain . Ġbo ito 2019 mass Ġcreated Ġa Ġmult ilingual ĠST Ġcorpus Ġinvolving Ġ8 Ġlanguages Ġfrom Ġa Ġmult ilingual Ġspeech Ġcorpus Ġbased Ġon ĠBible Ġreadings ĠB IB REF 7 . ĠPrevious Ġwork Ġeither Ġinvolves Ġlanguage Ġpairs Ġout Ġof ĠEnglish , Ġvery Ġspecific Ġdomains , Ġvery Ġlow Ġresource Ġlanguages Ġor Ġa Ġlimited Ġset Ġof Ġlanguage Ġpairs . ĠThis Ġlimits Ġthe Ġscope Ġof Ġstudy , Ġincluding Ġthe Ġlatest Ġexplor ations Ġon Ġend - to - end Ġmult ilingual ĠST ĠB IB REF 8 , ĠB IB REF 9 . ĠOur Ġwork Ġis Ġmostly Ġsimilar Ġand Ġconcurrent Ġto Ġir anz os n che z 2019 euro par l st Ġwho Ġcreated Ġa Ġmult ilingual ĠST Ġcorpus Ġfrom Ġthe ĠEuropean ĠParliament Ġproceedings . ĠThe Ġcorpus Ġwe Ġintroduce Ġhas Ġlarger Ġspeech Ġd urations Ġand Ġmore Ġtranslation Ġtokens . ĠIt Ġis Ġdivers ified Ġwith Ġmultiple Ġspeakers Ġper Ġtranscript / translation . ĠFinally , Ġwe Ġprovide Ġadditional Ġout - of - domain Ġtest Ġsets . </s> In Ġthis Ġpaper , Ġwe Ġintroduce ĠCo Vo ST , Ġa Ġmult ilingual ĠST Ġcorpus Ġbased Ġon ĠCommon ĠVoice ĠB IB REF 10 Ġfor Ġ11 Ġlanguages Ġinto ĠEnglish , Ġdivers ified Ġwith Ġover Ġ11 , 000 Ġspeakers Ġand Ġover Ġ60 Ġaccents . ĠIt Ġincludes Ġa Ġtotal Ġ7 08 Ġhours Ġof ĠFrench Ġ( Fr ), ĠGerman Ġ( De ), ĠDutch Ġ( N l ), ĠRussian Ġ( Ru ), ĠSpanish Ġ( Es ), ĠItalian Ġ( It ), ĠTurkish Ġ( Tr ), ĠPersian Ġ( Fa ), ĠSwedish Ġ( S v ), ĠMongol ian Ġ( M n ) Ġand ĠChinese Ġ( Z h ) Ġspeeches , Ġwith ĠFrench Ġand ĠGerman Ġones Ġhaving Ġthe Ġlargest Ġd urations Ġamong Ġexisting Ġpublic Ġcorpor a . ĠWe Ġalso Ġcollect Ġan Ġadditional Ġevaluation Ġcorpus Ġfrom ĠT ato eb a Ġfor ĠFrench , ĠGerman , ĠDutch , ĠRussian Ġand ĠSpanish , Ġresulting Ġin Ġa Ġtotal Ġof Ġ9 . 3 Ġhours Ġof Ġspeech . ĠBoth Ġcorpor a Ġare Ġcreated Ġat Ġthe Ġsentence Ġlevel Ġand Ġdo Ġnot Ġrequire Ġadditional Ġalign ments Ġor Ġsegment ation . ĠUsing Ġthe Ġofficial ĠCommon ĠVoice Ġtrain - development - test Ġsplit , Ġwe Ġalso Ġprovide Ġbaseline Ġmodels , Ġincluding , Ġto Ġour Ġknowledge , Ġthe Ġfirst Ġend - to - end Ġmany - to - one Ġmult ilingual ĠST Ġmodels . ĠCo Vo ST Ġis Ġreleased Ġunder ĠCC 0 Ġlicense Ġand Ġfree Ġto Ġuse . ĠThe ĠT ato eb a Ġevaluation Ġsamples Ġare Ġalso Ġavailable Ġunder Ġfriendly ĠCC Ġlicenses . ĠAll Ġthe Ġdata Ġcan Ġbe Ġacquired Ġat Ġhttps :// github . com / facebook research / c ov ost . </s> Data ĠCollection Ġand ĠProcessing Ġ: :: ĠCommon ĠVoice Ġ( Co Vo ) </s> Common ĠVoice ĠB IB REF 10 Ġis Ġa Ġcrowds ourcing Ġspeech Ġrecognition Ġcorpus Ġwith Ġan Ġopen ĠCC 0 Ġlicense . ĠContribut ors Ġrecord Ġvoice Ġclips Ġby Ġreading Ġfrom Ġa Ġbank Ġof Ġdonated Ġsentences . ĠEach Ġvoice Ġclip Ġwas Ġvalidated Ġby Ġat Ġleast Ġtwo Ġother Ġusers . ĠMost Ġof Ġthe Ġsentences Ġare Ġcovered Ġby Ġmultiple Ġspeakers , Ġwith Ġpotentially Ġdifferent Ġgenders , Ġage Ġgroups Ġor Ġaccents . </s> Raw ĠCo Vo Ġdata Ġcontains Ġsamples Ġthat Ġpassed Ġvalidation Ġas Ġwell Ġas Ġthose Ġthat Ġdid Ġnot . ĠTo Ġbuild ĠCo Vo ST , Ġwe Ġonly Ġuse Ġthe Ġformer Ġone Ġand Ġreuse Ġthe Ġofficial Ġtrain - development - test Ġpartition Ġof Ġthe Ġvalidated Ġdata . ĠAs Ġof ĠJanuary Ġ2020 , Ġthe Ġlatest ĠCo Vo Ġ2019 - 06 - 12 Ġrelease Ġincludes Ġ29 Ġlanguages . ĠCo Vo ST Ġis Ġcurrently Ġbuilt Ġon Ġthat Ġrelease Ġand Ġcovers Ġthe Ġfollowing Ġ11 Ġlanguages : ĠFrench , ĠGerman , ĠDutch , ĠRussian , ĠSpanish , ĠItalian , ĠTurkish , ĠPersian , ĠSwedish , ĠMongol ian Ġand ĠChinese . </s> Val idated Ġtranscripts Ġwere Ġsent Ġto Ġprofessional Ġtransl ators . ĠNote Ġthat Ġthe Ġtransl ators Ġhad Ġaccess Ġto Ġthe Ġtranscripts Ġbut Ġnot Ġthe Ġcorresponding Ġvoice Ġclips Ġsince Ġclips Ġwould Ġnot Ġcarry Ġadditional Ġinformation . ĠSince Ġtranscripts Ġwere Ġdupl icated Ġdue Ġto Ġmultiple Ġspeakers , Ġwe Ġded u pl icated Ġthe Ġtranscripts Ġbefore Ġsending Ġthem Ġto Ġtransl ators . ĠAs Ġa Ġresult , Ġdifferent Ġvoice Ġclips Ġof Ġthe Ġsame Ġcontent Ġ( trans cript ) Ġwill Ġhave Ġidentical Ġtranslations Ġin ĠCo Vo ST Ġfor Ġtrain , Ġdevelopment Ġand Ġtest Ġsplits . </s> In Ġorder Ġto Ġcontrol Ġthe Ġquality Ġof Ġthe Ġprofessional Ġtranslations , Ġwe Ġapplied Ġvarious Ġsanity Ġchecks Ġto Ġthe Ġtranslations ĠB IB REF 11 . Ġ1 ) ĠFor ĠGerman - English , ĠFrench - English Ġand ĠRussian - English Ġtranslations , Ġwe Ġcomputed Ġsentence - level ĠB LE U ĠB IB REF 12 Ġwith Ġthe ĠNL T K ĠB IB REF 13 Ġimplementation Ġbetween Ġthe Ġhuman Ġtranslations Ġand Ġthe Ġautomatic Ġtranslations Ġproduced Ġby Ġa Ġstate - of - the - art Ġsystem ĠB IB REF 14 Ġ( the ĠFrench - English Ġsystem Ġwas Ġa ĠTrans former Ġbig ĠB IB REF 15 Ġseparately Ġtrained Ġon ĠW MT 14 ). ĠWe Ġapplied Ġthis Ġmethod Ġto Ġthese Ġthree Ġlanguage Ġpairs Ġonly Ġas Ġwe Ġare Ġconfident Ġabout Ġthe Ġquality Ġof Ġthe Ġcorresponding Ġsystems . ĠTrans lations Ġwith Ġa Ġscore Ġthat Ġwas Ġtoo Ġlow Ġwere Ġmanually Ġinspected Ġand Ġsent Ġback Ġto Ġthe Ġtransl ators Ġwhen Ġneeded . Ġ2 ) ĠWe Ġmanually Ġinspected Ġexamples Ġwhere Ġthe Ġsource Ġtranscript Ġwas Ġidentical Ġto Ġthe Ġtranslation . Ġ3 ) ĠWe Ġmeasured Ġthe Ġperplex ity Ġof Ġthe Ġtranslations Ġusing Ġa Ġlanguage Ġmodel Ġtrained Ġon Ġa Ġlarge Ġamount Ġof Ġclean Ġmon oling ual Ġdata ĠB IB REF 14 . ĠWe Ġmanually Ġinspected Ġexamples Ġwhere Ġthe Ġtranslation Ġhad Ġa Ġhigh Ġperplex ity Ġand Ġsent Ġthem Ġback Ġto Ġtransl ators Ġaccordingly . Ġ4 ) ĠWe Ġcomputed Ġthe Ġratio Ġof ĠEnglish Ġcharacters Ġin Ġthe Ġtranslations . ĠWe Ġmanually Ġinspected Ġexamples Ġwith Ġa Ġlow Ġratio Ġand Ġsent Ġthem Ġback Ġto Ġtransl ators Ġaccordingly . Ġ5 ) ĠFinally , Ġwe Ġused ĠViz Se q ĠB IB REF 16 Ġto Ġcalculate Ġsimilarity Ġscores Ġbetween Ġtranscripts Ġand Ġtranslations Ġbased Ġon ĠL AS ER Ġcross - ling ual Ġsentence Ġembed d ings ĠB IB REF 17 . ĠSam ples Ġwith Ġlow Ġscores Ġwere Ġmanually Ġinspected Ġand Ġsent Ġback Ġfor Ġtranslation Ġwhen Ġneeded . </s> We Ġalso Ġsanity Ġcheck Ġthe Ġoverl aps Ġof Ġtrain , Ġdevelopment Ġand Ġtest Ġsets Ġin Ġterms Ġof Ġtranscripts Ġand Ġvoice Ġclips Ġ( via ĠMD 5 Ġfile Ġhashing ), Ġand Ġconfirm Ġthey Ġare Ġtotally Ġdis j oint . </s> Data ĠCollection Ġand ĠProcessing Ġ: :: ĠT ato eb a Ġ( TT ) </s> T ato eb a Ġ( TT ) Ġis Ġa Ġcommunity Ġbuilt Ġlanguage Ġlearning Ġcorpus Ġhaving Ġsentences Ġaligned Ġacross Ġmultiple Ġlanguages Ġwith Ġthe Ġcorresponding Ġspeech Ġpartially Ġavailable . ĠIts Ġsentences Ġare Ġon Ġaverage Ġshorter Ġthan Ġthose Ġin ĠCo Vo ST Ġ( see Ġalso ĠTable ĠT AB REF 2 ) Ġgiven Ġthe Ġoriginal Ġpurpose Ġof Ġlanguage Ġlearning . ĠSent ences Ġin ĠTT Ġare Ġlicensed Ġunder ĠCC ĠBY Ġ2 . 0 ĠFR Ġand Ġpart Ġof Ġthe Ġspeeches Ġare Ġavailable Ġunder Ġvarious ĠCC Ġlicenses . </s> We Ġconstruct Ġan Ġevaluation Ġset Ġfrom ĠTT Ġ( for ĠFrench , ĠGerman , ĠDutch , ĠRussian Ġand ĠSpanish ) Ġas Ġa Ġcomplement Ġto ĠCo Vo ST Ġdevelopment Ġand Ġtest Ġsets . ĠWe Ġcollect Ġ( speech , Ġtranscript , ĠEnglish Ġtranslation ) Ġtriple ts Ġfor Ġthe Ġ5 Ġlanguages Ġand Ġdo Ġnot Ġinclude Ġthose Ġwhose Ġspeech Ġhas Ġa Ġbroken ĠURL Ġor Ġis Ġnot ĠCC Ġlicensed . ĠWe Ġfurther Ġfilter Ġthese Ġsamples Ġby Ġsentence Ġlengths Ġ( minimum Ġ4 Ġwords Ġincluding Ġpunct uations ) Ġto Ġreduce Ġthe Ġportion Ġof Ġshort Ġsentences . ĠThis Ġmakes Ġthe Ġresulting Ġevaluation Ġset Ġcloser Ġto Ġreal - world Ġscenarios Ġand Ġmore Ġchallenging . </s> We Ġrun Ġthe Ġsame Ġquality Ġchecks Ġfor ĠTT Ġas Ġfor ĠCo Vo ST Ġbut Ġwe Ġdo Ġnot Ġfind Ġpoor Ġquality Ġtranslations Ġaccording Ġto Ġour Ġcriteria . ĠFinally , Ġwe Ġreport Ġthe Ġoverlap Ġbetween ĠCo Vo Ġtranscripts Ġand ĠTT Ġsentences Ġin ĠTable ĠT AB REF 5 . ĠWe Ġfound Ġa Ġminimal Ġoverlap , Ġwhich Ġmakes Ġthe ĠTT Ġevaluation Ġset Ġa Ġsuitable Ġadditional Ġtest Ġset Ġwhen Ġtraining Ġon ĠCo Vo ST . </s> Data ĠAnalysis Ġ: :: ĠBasic ĠStatistics </s> Basic Ġstatistics Ġfor ĠCo Vo ST Ġand ĠTT Ġare Ġlisted Ġin ĠTable ĠT AB REF 2 Ġincluding Ġ( unique ) Ġsentence Ġcounts , Ġspeech Ġd urations , Ġspeaker Ġdemographics Ġ( part ially Ġavailable ) Ġas Ġwell Ġas Ġvocabulary Ġand Ġtoken Ġstatistics Ġ( based Ġon ĠMoses - token ized Ġsentences Ġby Ġsac re M oses ) Ġon Ġboth Ġtranscripts Ġand Ġtranslations . ĠWe Ġsee Ġthat ĠCo Vo ST Ġhas Ġover Ġ327 Ġhours Ġof ĠGerman Ġspeeches Ġand Ġover Ġ171 Ġhours Ġof ĠFrench Ġspeeches , Ġwhich , Ġto Ġour Ġknowledge , Ġcorresponds Ġto Ġthe Ġlargest Ġcorpus Ġamong Ġexisting Ġpublic ĠST Ġcorpor a Ġ( the Ġsecond Ġlargest Ġis Ġ110 Ġhours ĠB IB REF 18 Ġfor ĠGerman Ġand Ġ38 Ġhours ĠB IB REF 19 Ġfor ĠFrench ). ĠMoreover , ĠCo Vo ST Ġhas Ġa Ġtotal Ġof Ġ18 Ġhours Ġof ĠDutch Ġspeeches , Ġto Ġour Ġknowledge , Ġcontributing Ġthe Ġfirst Ġpublic ĠDutch ĠST Ġresource . ĠCo Vo ST Ġalso Ġhas Ġaround Ġ27 - hour ĠRussian Ġspeeches , Ġ37 - hour ĠItalian Ġspeeches Ġand Ġ67 - hour ĠPersian Ġspeeches , Ġwhich Ġis Ġ1 . 8 Ġtimes , Ġ2 . 5 Ġtimes Ġand Ġ13 . 3 Ġtimes Ġof Ġthe Ġprevious Ġlargest Ġpublic Ġone ĠB IB REF 7 . ĠMost Ġof Ġthe Ġsentences Ġ( trans cript s ) Ġin ĠCo Vo ST Ġare Ġcovered Ġby Ġmultiple Ġspeakers Ġwith Ġpotentially Ġdifferent Ġaccents , Ġresulting Ġin Ġa Ġrich Ġdiversity Ġin Ġthe Ġspeeches . ĠFor Ġexample , Ġthere Ġare Ġover Ġ1 , 000 Ġspeakers Ġand Ġover Ġ10 Ġaccents Ġin Ġthe ĠFrench Ġand ĠGerman Ġdevelopment Ġ/ Ġtest Ġsets . ĠThis Ġenables Ġgood Ġcoverage Ġof Ġspeech Ġvariations Ġin Ġboth Ġmodel Ġtraining Ġand Ġevaluation . </s> Data ĠAnalysis Ġ: :: ĠSpeaker ĠDiversity </s> As Ġwe Ġcan Ġsee Ġfrom ĠTable ĠT AB REF 2 , ĠCo Vo ST Ġis Ġdivers ified Ġwith Ġa Ġrich Ġset Ġof Ġspeakers Ġand Ġaccents . ĠWe Ġfurther Ġinspect Ġthe Ġspeaker Ġdemographics Ġin Ġterms Ġof Ġsample Ġdistributions Ġwith Ġrespect Ġto Ġspeaker Ġcounts , Ġaccent Ġcounts Ġand Ġage Ġgroups , Ġwhich Ġis Ġshown Ġin ĠFigure ĠFIG REF 6 , ĠFIG REF 7 Ġand ĠFIG REF 8 . ĠWe Ġobserve Ġthat Ġfor Ġ8 Ġof Ġthe Ġ11 Ġlanguages , Ġat Ġleast Ġ60 % Ġof Ġthe Ġsentences Ġ( trans cript s ) Ġare Ġcovered Ġby Ġmultiple Ġspeakers . ĠOver Ġ80 % Ġof Ġthe ĠFrench Ġsentences Ġhave Ġat Ġleast Ġ3 Ġspeakers . ĠAnd Ġfor ĠGerman Ġsentences , Ġeven Ġover Ġ90 % Ġof Ġthem Ġhave Ġat Ġleast Ġ5 Ġspeakers . ĠSimilarly , Ġwe Ġsee Ġthat Ġa Ġlarge Ġportion Ġof Ġsentences Ġare Ġspoken Ġin Ġmultiple Ġaccents Ġfor ĠFrench , ĠGerman , ĠDutch Ġand ĠSpanish . ĠSpe akers Ġof Ġeach Ġlanguage Ġalso Ġspread Ġwidely Ġacross Ġdifferent Ġage Ġgroups Ġ( below Ġ20 , Ġ20 s , Ġ30 s , Ġ40 s , Ġ50 s , Ġ60 s Ġand Ġ70 s ). </s> Bas eline ĠResults </s> We Ġprovide Ġbas elines Ġusing Ġthe Ġofficial Ġtrain - development - test Ġsplit Ġon Ġthe Ġfollowing Ġtasks : Ġautomatic Ġspeech Ġrecognition Ġ( AS R ), Ġmachine Ġtranslation Ġ( MT ) Ġand Ġspeech Ġtranslation Ġ( ST ). </s> Bas eline ĠResults Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠData ĠPre processing </s> We Ġconvert Ġraw ĠMP 3 Ġaudio Ġfiles Ġfrom ĠCo Vo Ġand ĠTT Ġinto Ġmono - channel Ġwave forms , Ġand Ġdowns ample Ġthem Ġto Ġ16 , 000 ĠHz . ĠFor Ġtranscripts Ġand Ġtranslations , Ġwe Ġnormal ize Ġthe Ġpunct uation , Ġwe Ġtoken ize Ġthe Ġtext Ġwith Ġsac re M oses Ġand Ġlower case Ġit . ĠFor Ġtranscripts , Ġwe Ġfurther Ġremove Ġall Ġpunct uation Ġmarkers Ġexcept Ġfor Ġapost rop hes . ĠWe Ġuse Ġcharacter Ġvoc ab ular ies Ġon Ġall Ġthe Ġtasks , Ġwith Ġ100 % Ġcoverage Ġof Ġall Ġthe Ġcharacters . ĠPrel iminary Ġexperimentation Ġshowed Ġthat Ġcharacter Ġvoc ab ular ies Ġprovided Ġmore Ġstable Ġtraining Ġthan ĠB PE . ĠFor ĠMT , Ġthe Ġvocabulary Ġis Ġcreated Ġjointly Ġon Ġboth Ġtranscripts Ġand Ġtranslations . ĠWe Ġextract Ġ80 - channel Ġlog - mel Ġfilter bank Ġfeatures , Ġcomputed Ġwith Ġa Ġ25 ms Ġwindow Ġsize Ġand Ġ10 ms Ġwindow Ġshift Ġusing Ġtorch audio . ĠThe Ġfeatures Ġare Ġnormalized Ġto Ġ0 Ġmean Ġand Ġ1 . 0 Ġstandard Ġdeviation . ĠWe Ġremove Ġsamples Ġhaving Ġmore Ġthan Ġ3 , 000 Ġframes Ġor Ġmore Ġthan Ġ256 Ġcharacters Ġfor ĠGPU Ġmemory Ġefficiency Ġ( less Ġthan Ġ25 Ġsamples Ġare Ġremoved Ġfor Ġall Ġlanguages ). </s> Bas eline ĠResults Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠModel ĠTraining </s> Our ĠAS R Ġand ĠST Ġmodels Ġfollow Ġthe Ġarchitecture Ġin Ġber ard 2018 end , Ġbut Ġhave Ġ3 Ġdec oder Ġlayers Ġlike Ġthat Ġin Ġp ino 2019 har ness ing . ĠFor ĠMT , Ġwe Ġuse Ġa ĠTrans former Ġbase Ġarchitecture ĠB IB REF 15 , Ġbut Ġwith Ġ3 Ġenc oder Ġlayers , Ġ3 Ġdec oder Ġlayers Ġand Ġ0 . 3 Ġdrop out . ĠWe Ġuse Ġa Ġbatch Ġsize Ġof Ġ10 , 000 Ġframes Ġfor ĠAS R Ġand ĠST , Ġand Ġa Ġbatch Ġsize Ġof Ġ4 , 000 Ġtokens Ġfor ĠMT . ĠWe Ġtrain Ġall Ġmodels Ġusing ĠFair seq ĠB IB REF 20 Ġfor Ġup Ġto Ġ200 , 000 Ġupdates . ĠWe Ġuse ĠSpec Aug ment ĠB IB REF 21 Ġfor ĠAS R Ġand ĠST Ġto Ġalleviate Ġover fitting . </s> Bas eline ĠResults Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠIn ference Ġand ĠEvaluation </s> We Ġuse Ġa Ġbeam Ġsize Ġof Ġ5 Ġfor Ġall Ġmodels . ĠWe Ġuse Ġthe Ġbest Ġcheckpoint Ġby Ġvalidation Ġloss Ġfor ĠMT , Ġand Ġaverage Ġthe Ġlast Ġ5 Ġcheckpoints Ġfor ĠAS R Ġand ĠST . ĠFor ĠMT Ġand ĠST , Ġwe Ġreport Ġcase - ins ensitive Ġtoken ized ĠB LE U ĠB IB REF 22 Ġusing Ġsac re BLE U ĠB IB REF 23 . ĠFor ĠAS R , Ġwe Ġreport Ġword Ġerror Ġrate Ġ( WER ) Ġand Ġcharacter Ġerror Ġrate Ġ( C ER ) Ġusing ĠViz Se q . </s> Bas eline ĠResults Ġ: :: ĠAutomatic ĠSpeech ĠRecogn ition Ġ( AS R ) </s> For Ġsimplicity , Ġwe Ġuse Ġthe Ġsame Ġmodel Ġarchitecture Ġfor ĠAS R Ġand ĠST , Ġalthough Ġwe Ġdo Ġnot Ġleverage ĠAS R Ġmodels Ġto Ġpret rain ĠST Ġmodel Ġenc od ers Ġlater . ĠTable ĠT AB REF 18 Ġshows Ġthe Ġword Ġerror Ġrate Ġ( WER ) Ġand Ġcharacter Ġerror Ġrate Ġ( C ER ) Ġfor ĠAS R Ġmodels . ĠWe Ġsee Ġthat ĠFrench Ġand ĠGerman Ġperform Ġthe Ġbest Ġgiven Ġthey Ġare Ġthe Ġtwo Ġhighest Ġresource Ġlanguages Ġin ĠCo Vo ST . ĠThe Ġother Ġlanguages Ġare Ġrelatively Ġlow Ġresource Ġ( especially ĠTurkish Ġand ĠSwedish ) Ġand Ġthe ĠAS R Ġmodels Ġare Ġhaving Ġdifficulties Ġto Ġlearn Ġfrom Ġthis Ġdata . </s> Bas eline ĠResults Ġ: :: ĠMachine ĠTranslation Ġ( MT ) </s> MT Ġmodels Ġtake Ġtranscripts Ġ( without Ġpunct uation ) Ġas Ġinputs Ġand Ġoutputs Ġtranslations Ġ( with Ġpunct uation ). ĠFor Ġsimplicity , Ġwe Ġdo Ġnot Ġchange Ġthe Ġtext Ġpre processing Ġmethods Ġfor ĠMT Ġto Ġcorrect Ġthis Ġmismatch . ĠMoreover , Ġthis Ġmismatch Ġalso Ġexists Ġin Ġcasc ading ĠST Ġsystems , Ġwhere ĠMT Ġmodel Ġinputs Ġare Ġthe Ġoutputs Ġof Ġan ĠAS R Ġmodel . ĠTable ĠT AB REF 20 Ġshows Ġthe ĠB LE U Ġscores Ġof ĠMT Ġmodels . ĠWe Ġnotice Ġthat Ġthe Ġresults Ġare Ġconsistent Ġwith Ġwhat Ġwe Ġsee Ġfrom ĠAS R Ġmodels . ĠFor Ġexample Ġthanks Ġto Ġabundant Ġtraining Ġdata , ĠFrench Ġhas Ġa Ġdecent ĠB LE U Ġscore Ġof Ġ29 . 8 / 25 . 4 . ĠGerman Ġdoesn 't Ġperform Ġwell , Ġbecause Ġof Ġless Ġrichness Ġof Ġcontent Ġ( trans cript s ). ĠThe Ġother Ġlanguages Ġare Ġlow Ġresource Ġin ĠCo Vo ST Ġand Ġit Ġis Ġdifficult Ġto Ġtrain Ġdecent Ġmodels Ġwithout Ġadditional Ġdata Ġor Ġpre - training Ġtechniques . </s> Bas eline ĠResults Ġ: :: ĠSpeech ĠTranslation Ġ( ST ) </s> Co Vo ST Ġis Ġa Ġmany - to - one Ġmult ilingual ĠST Ġcorpus . ĠWhile Ġend - to - end Ġone - to - many Ġand Ġmany - to - many Ġmult ilingual ĠST Ġmodels Ġhave Ġbeen Ġexplored Ġvery Ġrecently ĠB IB REF 8 , ĠB IB REF 9 , Ġmany - to - one Ġmult ilingual Ġmodels , Ġto Ġour Ġknowledge , Ġhave Ġnot . ĠWe Ġhence Ġuse ĠCo Vo ST Ġto Ġexamine Ġthis Ġsetting . ĠTable ĠT AB REF 22 Ġand ĠT AB REF 23 Ġshow Ġthe ĠB LE U Ġscores Ġfor Ġboth Ġbilingual Ġand Ġmult ilingual Ġend - to - end ĠST Ġmodels Ġtrained Ġon ĠCo Vo ST . ĠWe Ġobserve Ġthat Ġcombining Ġspeeches Ġfrom Ġmultiple Ġlanguages Ġis Ġconsistently Ġbringing Ġgains Ġto Ġlow - resource Ġlanguages Ġ( all Ġbesides ĠFrench Ġand ĠGerman ). ĠThis Ġincludes Ġcombinations Ġof Ġdistant Ġlanguages , Ġsuch Ġas ĠRu + Fr , ĠTr + Fr Ġand ĠZh + Fr . ĠMoreover , Ġsome Ġcombinations Ġdo Ġbring Ġgains Ġto Ġhigh - resource Ġlanguage Ġ( French ) Ġas Ġwell : ĠEs + Fr , ĠTr + Fr Ġand ĠMn + Fr . ĠWe Ġsimply Ġprovide Ġthe Ġmost Ġbasic Ġmany - to - one Ġmult ilingual Ġbas elines Ġhere , Ġand Ġleave Ġthe Ġfull Ġexploration Ġof Ġthe Ġbest Ġconfigurations Ġto Ġfuture Ġwork . ĠFinally , Ġwe Ġnote Ġthat Ġfor Ġsome Ġlanguage Ġpairs , Ġabsolute ĠB LE U Ġnumbers Ġare Ġrelatively Ġlow Ġas Ġwe Ġrestrict Ġmodel Ġtraining Ġto Ġthe Ġsupervised Ġdata . ĠWe Ġencourage Ġthe Ġcommunity Ġto Ġimprove Ġupon Ġthose Ġbas elines , Ġfor Ġexample Ġby Ġleveraging Ġsemi - super vised Ġtraining . </s> Bas eline ĠResults Ġ: :: ĠMulti - Spe aker ĠEvaluation </s> In ĠCo Vo ST , Ġlarge Ġportion Ġof Ġtranscripts Ġare Ġcovered Ġby Ġmultiple Ġspeakers Ġwith Ġdifferent Ġgenders , Ġaccents Ġand Ġage Ġgroups . ĠBesides Ġthe Ġstandard Ġcorpus - level ĠB LE U Ġscores , Ġwe Ġalso Ġwant Ġto Ġevaluate Ġmodel Ġoutput Ġvariance Ġon Ġthe Ġsame Ġcontent Ġ( trans cript ) Ġbut Ġdifferent Ġspeakers . ĠWe Ġhence Ġpropose Ġto Ġgroup Ġsamples Ġ( and Ġtheir Ġsentence ĠB LE U Ġscores ) Ġby Ġtranscript , Ġand Ġthen Ġcalculate Ġaverage Ġper - group Ġmean Ġand Ġaverage Ġcoefficient Ġof Ġvariation Ġdefined Ġas Ġfollows : </s> and </s> where Ġ$ G $ Ġis Ġthe Ġset Ġof Ġsentence ĠB LE U Ġscores Ġgrouped Ġby Ġtranscript Ġand Ġ$ G ^ {\ prime Ġ} Ġ= Ġ\ l brace Ġg Ġ| Ġg \ in ĠG , Ġ| g | > 1 , Ġ\ text rm Ġ{ Me an } ( g ) Ġ> Ġ0 Ġ\ r brace Ġ$ . </s> $ \ text rm Ġ{ BLE U } _{ MS } $ Ġprovides Ġa Ġnormalized Ġquality Ġscore Ġas Ġoppose Ġto Ġcorpus - level ĠB LE U Ġor Ġun normal ized Ġaverage Ġof Ġsentence ĠB LE U . ĠAnd Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġis Ġa Ġstandardized Ġmeasure Ġof Ġmodel Ġstability Ġagainst Ġdifferent Ġspeakers Ġ( the Ġlower Ġthe Ġbetter ). ĠTable ĠT AB REF 24 Ġshows Ġthe Ġ$\ text rm Ġ{ BLE U } _{ MS } $ Ġand Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġof Ġour ĠST Ġmodels Ġon ĠCo Vo ST Ġtest Ġset . ĠWe Ġsee Ġthat ĠGerman Ġand ĠPersian Ġhave Ġthe Ġworst Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġ( le ast Ġstable ) Ġgiven Ġtheir Ġrich Ġspeaker Ġdiversity Ġin Ġthe Ġtest Ġset Ġand Ġrelatively Ġsmall Ġtrain Ġset Ġ( see Ġalso ĠFigure ĠFIG REF 6 Ġand ĠTable ĠT AB REF 2 ). ĠDutch Ġalso Ġhas Ġpoor Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġbecause Ġof Ġthe Ġlack Ġof Ġtraining Ġdata . ĠMult ilingual Ġmodels Ġare Ġconsist antly Ġmore Ġstable Ġon Ġlow - resource Ġlanguages . ĠRu + Fr , ĠTr + Fr , ĠFa + Fr Ġand ĠZh + Fr Ġeven Ġhave Ġbetter Ġ$\ text rm Ġ{ Co ef Var } _{ MS } $ Ġthan Ġall Ġindividual Ġlanguages . </s> Conclusion </s> We Ġintroduce Ġa Ġmult ilingual Ġspeech - to - text Ġtranslation Ġcorpus , ĠCo Vo ST , Ġfor Ġ11 Ġlanguages Ġinto ĠEnglish , Ġdivers ified Ġwith Ġover Ġ11 , 000 Ġspeakers Ġand Ġover Ġ60 Ġaccents . ĠWe Ġalso Ġprovide Ġbaseline Ġresults , Ġincluding , Ġto Ġour Ġknowledge , Ġthe Ġfirst Ġend - to - end Ġmany - to - one Ġmult ilingual Ġmodel Ġfor Ġspoken Ġlanguage Ġtranslation . ĠCo Vo ST Ġis Ġfree Ġto Ġuse Ġwith Ġa ĠCC 0 Ġlicense , Ġand Ġthe Ġadditional ĠT ato eb a Ġevaluation Ġsamples Ġare Ġalso ĠCC - licensed .