system presented by deri2016grapheme
EXTRACTIVE
wFST
EXTRACTIVE
Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST. Their results can be divided into two parts:
Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST. Their results can be divided into two parts:
1708.01464
Introduction </s> Acc urate Ġgrap he me - to - ph on eme Ġconversion Ġ( g 2 p ) Ġis Ġimportant Ġfor Ġany Ġapplication Ġthat Ġdepends Ġon Ġthe Ġsometimes Ġinconsistent Ġrelationship Ġbetween Ġspoken Ġand Ġwritten Ġlanguage . ĠMost Ġprominently , Ġthis Ġincludes Ġtext - to - speech Ġand Ġautomatic Ġspeech Ġrecognition . ĠMost Ġwork Ġon Ġg 2 p Ġhas Ġfocused Ġon Ġa Ġfew Ġlanguages Ġfor Ġwhich Ġextensive Ġpronunciation Ġdata Ġis Ġavailable ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠMost Ġlanguages Ġlack Ġthese Ġresources . ĠHowever , Ġa Ġlow Ġresource Ġlanguage 's Ġwriting Ġsystem Ġis Ġlikely Ġto Ġbe Ġsimilar Ġto Ġthe Ġwriting Ġsystems Ġof Ġlanguages Ġthat Ġdo Ġhave Ġsufficient Ġpronunciation Ġdata . ĠTherefore Ġg 2 p Ġmay Ġbe Ġpossible Ġfor Ġlow Ġresource Ġlanguages Ġif Ġthis Ġhigh Ġresource Ġdata Ġcan Ġbe Ġproperly Ġutilized . </s> We Ġattempt Ġto Ġleverage Ġhigh Ġresource Ġdata Ġby Ġtreating Ġg 2 p Ġas Ġa Ġmult is ource Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġproblem . ĠThe Ġsource Ġsequences Ġfor Ġour Ġsystem Ġare Ġwords Ġin Ġthe Ġstandard Ġorth ography Ġin Ġany Ġlanguage . ĠThe Ġtarget Ġsequences Ġare Ġthe Ġcorresponding Ġrepresentation Ġin Ġthe ĠInternational ĠPh on etic ĠAlphabet Ġ( IP A ). ĠOur Ġresults Ġshow Ġthat Ġthe Ġparameters Ġlearned Ġby Ġthe Ġshared Ġenc oder âĢĵ dec oder Ġare Ġable Ġto Ġexploit Ġthe Ġorth ographic Ġand Ġphon emic Ġsimilarities Ġbetween Ġthe Ġvarious Ġlanguages Ġin Ġour Ġdata . </s> Low ĠResource Ġg 2 p </s> Our Ġapproach Ġis Ġsimilar Ġin Ġgoal Ġto Ġder i 2016 g rap he me 's Ġmodel Ġfor Ġadapting Ġhigh Ġresource Ġg 2 p Ġmodels Ġfor Ġlow Ġresource Ġlanguages . ĠThey Ġtrained Ġweighted Ġfinite Ġstate Ġtrans du cer Ġ( w F ST ) Ġmodels Ġon Ġa Ġvariety Ġof Ġhigh Ġresource Ġlanguages , Ġthen Ġtransferred Ġthose Ġmodels Ġto Ġlow Ġresource Ġlanguages , Ġusing Ġa Ġlanguage Ġdistance Ġmetric Ġto Ġchoose Ġwhich Ġhigh Ġresource Ġmodels Ġto Ġuse Ġand Ġa Ġphon eme Ġdistance Ġmetric Ġto Ġmap Ġthe Ġhigh Ġresource Ġlanguage 's Ġphon em es Ġto Ġthe Ġlow Ġresource Ġlanguage 's Ġphon eme Ġinventory . ĠThese Ġdistance Ġmetrics Ġare Ġcomputed Ġbased Ġon Ġdata Ġfrom ĠPh o ible ĠB IB REF 4 Ġand ĠURI EL ĠB IB REF 5 Ġ. </s> Other Ġlow Ġresource Ġg 2 p Ġsystems Ġhave Ġused Ġa Ġstrategy Ġof Ġcombining Ġmultiple Ġmodels . Ġsch l ipp e 2014 comb ining Ġtrained Ġseveral Ġdata - driven Ġg 2 p Ġsystems Ġon Ġvarying Ġquantities Ġof Ġmon oling ual Ġdata Ġand Ġcombined Ġtheir Ġoutputs Ġwith Ġa Ġphon eme - level Ġvoting Ġscheme . ĠThis Ġled Ġto Ġimprovements Ġover Ġthe Ġbest - performing Ġsingle Ġsystem Ġfor Ġsmall Ġquantities Ġof Ġdata Ġin Ġsome Ġlanguages . Ġj y oth il ow Ġtrained Ġrecurrent Ġneural Ġnetworks Ġfor Ġsmall Ġdata Ġsets Ġand Ġfound Ġthat Ġa Ġversion Ġof Ġtheir Ġsystem Ġthat Ġcombined Ġthe Ġneural Ġnetwork Ġoutput Ġwith Ġthe Ġoutput Ġof Ġthe Ġw F ST - based ĠPh onet is aurus Ġsystem ĠB IB REF 1 Ġdid Ġbetter Ġthan Ġeither Ġsystem Ġalone . </s> A Ġdifferent Ġapproach Ġcame Ġfrom Ġk im 2012 universal , Ġwho Ġused Ġsupervised Ġlearning Ġwith Ġan Ġund irect ed Ġgraphical Ġmodel Ġto Ġinduce Ġthe Ġgrap he me âĢĵ ph on eme Ġm appings Ġfor Ġlanguages Ġwritten Ġin Ġthe ĠLatin Ġalphabet . ĠGiven Ġa Ġshort Ġtext Ġin Ġa Ġlanguage , Ġthe Ġmodel Ġpredicts Ġthe Ġlanguage 's Ġorth ographic Ġrules . ĠTo Ġcreate Ġphon emic Ġcontext Ġfeatures Ġfrom Ġthe Ġshort Ġtext , Ġthe Ġmodel ĠnaÃ¯ve ly Ġmaps Ġgrap hem es Ġto ĠIPA Ġsymbols Ġwritten Ġwith Ġthe Ġsame Ġcharacter , Ġand Ġuses Ġthe Ġfeatures Ġof Ġthese Ġsymbols Ġto Ġlearn Ġan Ġapproximation Ġof Ġthe Ġphon ot actic Ġconstraints Ġof Ġthe Ġlanguage . ĠIn Ġtheir Ġexperiments , Ġthese Ġphon ot actic Ġfeatures Ġproved Ġto Ġbe Ġmore Ġvaluable Ġthan Ġgeographical Ġand Ġgenetic Ġfeatures Ġdrawn Ġfrom ĠW ALS ĠB IB REF 6 Ġ. </s> Mult ilingual ĠNeural ĠN LP </s> In Ġrecent Ġyears , Ġneural Ġnetworks Ġhave Ġemerged Ġas Ġa Ġcommon Ġway Ġto Ġuse Ġdata Ġfrom Ġseveral Ġlanguages Ġin Ġa Ġsingle Ġsystem . ĠGoogle 's Ġzero - shot Ġneural Ġmachine Ġtranslation Ġsystem ĠB IB REF 7 Ġshares Ġan Ġenc oder Ġand Ġdec oder Ġacross Ġall Ġlanguage Ġpairs . ĠIn Ġorder Ġto Ġfacilitate Ġthis Ġmulti - way Ġtranslation , Ġthey Ġprep end Ġan Ġartificial Ġtoken Ġto Ġthe Ġbeginning Ġof Ġeach Ġsource Ġsentence Ġat Ġboth Ġtraining Ġand Ġtranslation Ġtime . ĠThe Ġtoken Ġidentifies Ġwhat Ġlanguage Ġthe Ġsentence Ġshould Ġbe Ġtranslated Ġto . ĠThis Ġapproach Ġhas Ġthree Ġbenefits : Ġit Ġis Ġfar Ġmore Ġefficient Ġthan Ġbuilding Ġa Ġseparate Ġmodel Ġfor Ġeach Ġlanguage Ġpair ; Ġit Ġallows Ġfor Ġtranslation Ġbetween Ġlanguages Ġthat Ġshare Ġno Ġparallel Ġdata ; Ġand Ġit Ġimproves Ġresults Ġon Ġlow - resource Ġlanguages Ġby Ġallowing Ġthem Ġto Ġimplicitly Ġshare Ġparameters Ġwith Ġhigh - resource Ġlanguages . ĠOur Ġg 2 p Ġsystem Ġis Ġinspired Ġby Ġthis Ġapproach , Ġalthough Ġit Ġdiffers Ġin Ġthat Ġthere Ġis Ġonly Ġone Ġtarget ĠâĢ ľ language âĢ Ŀ , ĠIPA , Ġand Ġthe Ġartificial Ġtokens Ġidentify Ġthe Ġlanguage Ġof Ġthe Ġsource Ġinstead Ġof Ġthe Ġlanguage Ġof Ġthe Ġtarget . </s> Other Ġwork Ġhas Ġalso Ġmade Ġuse Ġof Ġmult iling ually - trained Ġneural Ġnetworks . ĠPh on eme - level Ġpoly gl ot Ġlanguage Ġmodels ĠB IB REF 8 Ġtrain Ġa Ġsingle Ġmodel Ġon Ġmultiple Ġlanguages Ġand Ġadditionally Ġcondition Ġon Ġexternally Ġconstructed Ġtyp ological Ġdata Ġabout Ġthe Ġlanguage . Ġost ling 2017 contin uous Ġused Ġa Ġsimilar Ġapproach , Ġin Ġwhich Ġa Ġcharacter - level Ġneural Ġlanguage Ġmodel Ġis Ġtrained Ġon Ġa Ġmassively Ġmult ilingual Ġcorpus . ĠA Ġlanguage Ġembed ding Ġvector Ġis Ġconc aten ated Ġto Ġthe Ġinput Ġat Ġeach Ġtime Ġstep . ĠThe Ġlanguage Ġembed d ings Ġtheir Ġsystem Ġlearned Ġcorrelate Ġclosely Ġto Ġthe Ġgenetic Ġrelationships Ġbetween Ġlanguages . ĠHowever , Ġneither Ġof Ġthese Ġmodels Ġwas Ġapplied Ġto Ġg 2 p . </s> G rap he me - to - Ph on eme </s> g 2 p Ġis Ġthe Ġproblem Ġof Ġconverting Ġthe Ġorth ographic Ġrepresentation Ġof Ġa Ġword Ġinto Ġa Ġphon emic Ġrepresentation . ĠA Ġphon eme Ġis Ġan Ġabstract Ġunit Ġof Ġsound Ġwhich Ġmay Ġhave Ġdifferent Ġreal izations Ġin Ġdifferent Ġcontexts . ĠFor Ġexample , Ġthe ĠEnglish Ġphon eme Ġhas Ġtwo Ġphon etic Ġreal izations Ġ( or Ġall oph ones ): </s> English Ġspeakers Ġwithout Ġlinguistic Ġtraining Ġoften Ġstruggle Ġto Ġperceive Ġany Ġdifference Ġbetween Ġthese Ġsounds . ĠWriting Ġsystems Ġusually Ġdo Ġnot Ġdistinguish Ġbetween Ġall oph ones : Ġand Ġare Ġboth Ġwritten Ġas ĠIN LINE FORM 0 Ġp ĠIN LINE FORM 1 Ġin ĠEnglish . ĠThe Ġsounds Ġare Ġwritten Ġdifferently Ġin Ġlanguages Ġwhere Ġthey Ġcontrast , Ġsuch Ġas ĠHindi Ġand ĠEastern ĠArmenian . </s> Most Ġwriting Ġsystems Ġin Ġuse Ġtoday Ġare Ġgl ott ographic , Ġmeaning Ġthat Ġtheir Ġsymbols Ġencode Ġsolely Ġphon ological Ġinformation . ĠBut Ġdespite Ġbeing Ġgl ott ographic , Ġin Ġfew Ġwriting Ġsystems Ġdo Ġgrap hem es Ġcorrespond Ġone - to - one Ġwith Ġphon em es . ĠThere Ġare Ġcases Ġin Ġwhich Ġmultiple Ġgrap hem es Ġrepresent Ġa Ġsingle Ġphon eme , Ġas Ġin Ġthe Ġword Ġthe Ġin ĠEnglish : </s> There Ġare Ġcases Ġin Ġwhich Ġa Ġsingle Ġgrap he me Ġrepresents Ġmultiple Ġphon em es , Ġsuch Ġas Ġsyll ab aries , Ġin Ġwhich Ġeach Ġsymbol Ġrepresents Ġa Ġsyll able . </s> In Ġmany Ġlanguages , Ġthere Ġare Ġsilent Ġletters , Ġas Ġin Ġthe Ġword Ġhor a Ġin ĠSpanish : </s> There Ġare Ġmore Ġcomplicated Ġcorrespond ences , Ġsuch Ġas Ġthe Ġsilent Ġe Ġin ĠEnglish Ġthat Ġaffects Ġthe Ġpronunciation Ġof Ġthe Ġprevious Ġvowel , Ġas Ġseen Ġin Ġthe Ġpair Ġof Ġwords Ġcape Ġand Ġcap . </s> It Ġis Ġpossible Ġfor Ġan Ġorth ographic Ġsystem Ġto Ġhave Ġany Ġor Ġall Ġof Ġthe Ġabove Ġphenomena Ġwhile Ġremaining Ġunamb iguous . ĠHowever , Ġsome Ġorth ographic Ġsystems Ġcontain Ġambig u ities . ĠEnglish Ġis Ġwell - known Ġfor Ġits Ġspelling Ġambig u ities . ĠAb j ads , Ġused Ġfor ĠArabic Ġand ĠHebrew , Ġdo Ġnot Ġgive Ġfull Ġrepresentation Ġto Ġvow els . </s> C onsequ ently , Ġg 2 p Ġis Ġharder Ġthan Ġsimply Ġreplacing Ġeach Ġgrap he me Ġsymbol Ġwith Ġa Ġcorresponding Ġphon eme Ġsymbol . ĠIt Ġis Ġthe Ġproblem Ġof Ġreplacing Ġa Ġgrap he me Ġsequence ĠIN LINE FORM 0 </s> with Ġa Ġphon eme Ġsequence ĠIN LINE FORM 0 </s> where Ġthe Ġsequences Ġare Ġnot Ġnecessarily Ġof Ġthe Ġsame Ġlength . ĠData - driven Ġg 2 p Ġis Ġtherefore Ġthe Ġproblem Ġof Ġfinding Ġthe Ġphon eme Ġsequence Ġthat Ġmaxim izes Ġthe Ġlikelihood Ġof Ġthe Ġgrap he me Ġsequence : ĠIN LINE FORM 0 </s> Data - driven Ġapproaches Ġare Ġespecially Ġuseful Ġfor Ġproblems Ġin Ġwhich Ġthe Ġrules Ġthat Ġgovern Ġthem Ġare Ġcomplex Ġand Ġdifficult Ġto Ġengineer Ġby Ġhand . Ġg 2 p Ġfor Ġlanguages Ġwith Ġambiguous Ġorth ographies Ġis Ġsuch Ġa Ġproblem . ĠMult ilingual Ġg 2 p , Ġin Ġwhich Ġthe Ġvarious Ġlanguages Ġhave Ġsimilar Ġbut Ġdifferent Ġand Ġpossibly Ġcontradictory Ġspelling Ġrules , Ġcan Ġbe Ġseen Ġas Ġan Ġextreme Ġcase Ġof Ġthat . ĠTherefore , Ġa Ġdata - driven Ġsequence - to - sequence Ġmodel Ġis Ġa Ġnatural Ġchoice . </s> Enc oder âĢĵ Dec oder ĠModels </s> In Ġorder Ġto Ġfind Ġthe Ġbest Ġphon eme Ġsequence , Ġwe Ġuse Ġa Ġneural Ġenc oder âĢĵ dec oder Ġmodel Ġwith Ġattention ĠB IB REF 9 Ġ. ĠThe Ġmodel Ġconsists Ġof Ġtwo Ġmain Ġparts : Ġthe Ġenc oder Ġcomp resses Ġeach Ġsource Ġgrap he me Ġsequence ĠIN LINE FORM 0 Ġinto Ġa Ġfixed - length Ġvector . ĠThe Ġdec oder , Ġconditioned Ġon Ġthis Ġfixed - length Ġvector , Ġgenerates Ġthe Ġoutput Ġphon eme Ġsequence ĠIN LINE FORM 1 Ġ. </s> The Ġenc oder Ġand Ġdec oder Ġare Ġboth Ġimplemented Ġas Ġrecurrent Ġneural Ġnetworks , Ġwhich Ġhave Ġthe Ġadvantage Ġof Ġbeing Ġable Ġto Ġprocess Ġsequences Ġof Ġarbitrary Ġlength Ġand Ġuse Ġlong Ġhistories Ġefficiently . ĠThey Ġare Ġtrained Ġjointly Ġto Ġminimize Ġcross - ent ropy Ġon Ġthe Ġtraining Ġdata . ĠWe Ġhad Ġour Ġbest Ġresults Ġwhen Ġusing Ġa Ġbid irection al Ġenc oder , Ġwhich Ġconsists Ġof Ġtwo Ġseparate Ġenc od ers Ġwhich Ġprocess Ġthe Ġinput Ġin Ġforward Ġand Ġreverse Ġdirections . ĠWe Ġused Ġlong Ġshort - term Ġmemory Ġunits ĠB IB REF 10 Ġfor Ġboth Ġthe Ġenc oder Ġand Ġdec oder . ĠFor Ġthe Ġattention Ġmechanism , Ġwe Ġused Ġthe Ġgeneral Ġglobal Ġattention Ġarchitecture Ġdescribed Ġby Ġl u ong 2015 effective . </s> We Ġimplemented Ġall Ġmodels Ġwith ĠOpen N MT ĠB IB REF 11 Ġ. ĠOur Ġhyper param eters , Ġwhich Ġwe Ġdetermined Ġby Ġexperimentation , Ġare Ġlisted Ġin ĠTable ĠT AB REF 8 Ġ. </s> Training ĠMult ilingual ĠModels </s> Present ing Ġpronunciation Ġdata Ġin Ġseveral Ġlanguages Ġto Ġthe Ġnetwork Ġmight Ġcreate Ġproblems Ġbecause Ġdifferent Ġlanguages Ġhave Ġdifferent Ġpronunciation Ġpatterns . ĠFor Ġexample , Ġthe Ġstring Ġ` real ' Ġis Ġpronounced Ġdifferently Ġin ĠEnglish , ĠGerman , ĠSpanish , Ġand ĠPortuguese . ĠWe Ġsolve Ġthis Ġproblem Ġby Ġprep ending Ġeach Ġgrap he me Ġsequence Ġwith Ġan Ġartificial Ġtoken Ġconsisting Ġof Ġthe Ġlanguage 's ĠISO Ġ6 39 - 3 Ġcode Ġenclosed Ġin Ġangle Ġbrackets . ĠThe ĠEnglish Ġword Ġ` real ', Ġfor Ġexample , Ġwould Ġbe Ġpresented Ġto Ġthe Ġsystem Ġas </s> IN LINE FORM 0 Ġeng ĠIN LINE FORM 1 Ġr Ġe Ġa Ġl </s> The Ġartificial Ġtoken Ġis Ġtreated Ġsimply Ġas Ġan Ġelement Ġof Ġthe Ġgrap he me Ġsequence . ĠThis Ġis Ġsimilar Ġto Ġthe Ġapproach Ġtaken Ġby Ġjohn son 2016 google Ġin Ġtheir Ġzero - shot ĠN MT Ġsystem . ĠHowever , Ġtheir Ġsource - side Ġartificial Ġtokens Ġidentify Ġthe Ġtarget Ġlanguage , Ġwhereas Ġours Ġidentify Ġthe Ġsource Ġlanguage . ĠAn Ġalternative Ġapproach , Ġused Ġby Ġost ling 2017 contin uous , Ġwould Ġbe Ġto Ġconc aten ate Ġa Ġlanguage Ġembed ding Ġto Ġthe Ġinput Ġat Ġeach Ġtime Ġstep . ĠThey Ġdo Ġnot Ġevaluate Ġtheir Ġapproach Ġon Ġgrap he me - to - ph on eme Ġconversion . </s> Data </s> In Ġorder Ġto Ġtrain Ġa Ġneural Ġg 2 p Ġsystem , Ġone Ġneeds Ġa Ġlarge Ġquantity Ġof Ġpronunciation Ġdata . ĠA Ġstandard Ġdataset Ġfor Ġg 2 p Ġis Ġthe ĠCarnegie ĠMellon ĠP ron ouncing ĠDictionary ĠB IB REF 12 Ġ. ĠHowever , Ġthat Ġis Ġa Ġmon oling ual ĠEnglish Ġresource , Ġso Ġit Ġis Ġunsu itable Ġfor Ġour Ġmult ilingual Ġtask . ĠInstead , Ġwe Ġuse Ġthe Ġmult ilingual Ġpronunciation Ġcorpus Ġcollected Ġby Ġder i 2016 g rap he me Ġfor Ġall Ġexperiments . ĠThis Ġcorpus Ġconsists Ġof Ġspelling âĢĵ pron unciation Ġpairs Ġextracted Ġfrom ĠWik tion ary . ĠIt Ġis Ġalready Ġpartition ed Ġinto Ġtraining Ġand Ġtest Ġsets . ĠCorpus Ġstatistics Ġare Ġpresented Ġin ĠTable ĠT AB REF 10 Ġ. </s> In Ġaddition Ġto Ġthe Ġraw ĠIPA Ġtranscript ions Ġextracted Ġfrom ĠWik tion ary , Ġthe Ġcorpus Ġprovides Ġan Ġautomatically Ġcleaned Ġversion Ġof Ġtranscript ions . ĠClean ing Ġis Ġa Ġnecessary Ġstep Ġbecause Ġweb - sc raped Ġdata Ġis Ġoften Ġnoisy Ġand Ġmay Ġbe Ġtransc ribed Ġat Ġan Ġinconsistent Ġlevel Ġof Ġdetail . ĠThe Ġdata Ġcleaning Ġused Ġhere Ġattempts Ġto Ġmake Ġthe Ġtranscript ions Ġconsistent Ġwith Ġthe Ġphon emic Ġinvent ories Ġused Ġin ĠPh o ible ĠB IB REF 4 Ġ. ĠWhen Ġa Ġtranscription Ġcontains Ġa Ġphon eme Ġthat Ġis Ġnot Ġin Ġits Ġlanguage 's Ġinventory Ġin ĠPh o ible , Ġthat Ġphon eme Ġis Ġreplaced Ġby Ġthe Ġphon eme Ġwith Ġthe Ġmost Ġsimilar Ġartic ulatory Ġfeatures Ġthat Ġis Ġin Ġthe Ġlanguage 's Ġinventory . ĠSometimes Ġthis Ġcleaning Ġalgorithm Ġworks Ġwell : Ġin Ġthe ĠGerman Ġexamples Ġin ĠTable ĠT AB REF 11 Ġ, Ġthe Ġraw ĠGerman Ġsymbols Ġand Ġare Ġboth Ġconverted Ġto Ġ. ĠThis Ġis Ġuseful Ġbecause Ġthe Ġin ĠAns bach Ġand Ġthe Ġin ĠKan inc hen Ġare Ġinstances Ġof Ġthe Ġsame Ġphon eme , Ġso Ġtheir Ġphon emic Ġrepresentations Ġshould Ġuse Ġthe Ġsame Ġsymbol . ĠHowever , Ġthe Ġcleaning Ġalgorithm Ġcan Ġalso Ġhave Ġnegative Ġeffects Ġon Ġthe Ġdata Ġquality . ĠFor Ġexample , Ġthe Ġphon eme Ġis Ġnot Ġpresent Ġin Ġthe ĠPh o ible Ġinventory Ġfor ĠGerman , Ġbut Ġit Ġis Ġused Ġin Ġseveral ĠGerman Ġtranscript ions Ġin Ġthe Ġcorpus . ĠThe Ġcleaning Ġalgorithm Ġconverts Ġto Ġin Ġall ĠGerman Ġtranscript ions , Ġwhereas Ġwould Ġbe Ġa Ġmore Ġreasonable Ġguess . ĠThe Ġcleaning Ġalgorithm Ġalso Ġremoves Ġmost Ġsu pr ase gment als , Ġeven Ġthough Ġthese Ġare Ġoften Ġan Ġimportant Ġpart Ġof Ġa Ġlanguage 's Ġphon ology . ĠDevelop ing Ġa Ġmore Ġsophisticated Ġprocedure Ġfor Ġcleaning Ġpronunciation Ġdata Ġis Ġa Ġdirection Ġfor Ġfuture Ġwork , Ġbut Ġin Ġthis Ġpaper Ġwe Ġuse Ġthe Ġcorpus 's Ġprovided Ġcleaned Ġtranscript ions Ġin Ġorder Ġto Ġease Ġcomparison Ġto Ġprevious Ġresults . </s> Exper iments </s> We Ġpresent Ġexperiments Ġwith Ġtwo Ġversions Ġof Ġour Ġsequence - to - sequence Ġmodel . ĠLang ID Ġprep ends Ġeach Ġtraining , Ġvalidation , Ġand Ġtest Ġsample Ġwith Ġan Ġartificial Ġtoken Ġidentifying Ġthe Ġlanguage Ġof Ġthe Ġsample . ĠNo L ang ID Ġo mits Ġthis Ġtoken . ĠLang ID Ġand ĠNo L ang ID Ġhave Ġidentical Ġstructure Ġotherwise . ĠTo Ġtranslate Ġthe Ġtest Ġcorpus , Ġwe Ġused Ġa Ġbeam Ġwidth Ġof Ġ100 . ĠAlthough Ġthis Ġis Ġan Ġunusually Ġwide Ġbeam Ġand Ġhad Ġnegligible Ġperformance Ġeffects , Ġit Ġwas Ġnecessary Ġto Ġcompute Ġour Ġerror Ġmetrics . </s> E val uation </s> We Ġuse Ġthe Ġfollowing Ġthree Ġevaluation Ġmetrics : </s> Ph on eme ĠError ĠRate Ġ( PER ) Ġis Ġthe ĠLe ven sh tein Ġdistance Ġbetween Ġthe Ġpredicted Ġphon eme Ġsequences Ġand Ġthe Ġgold Ġstandard Ġphon eme Ġsequences , Ġdivided Ġby Ġthe Ġlength Ġof Ġthe Ġgold Ġstandard Ġphon eme Ġsequences . </s> Word ĠError ĠRate Ġ( WER ) Ġis Ġthe Ġpercentage Ġof Ġwords Ġin Ġwhich Ġthe Ġpredicted Ġphon eme Ġsequence Ġdoes Ġnot Ġexactly Ġmatch Ġthe Ġgold Ġstandard Ġphon eme Ġsequence . </s> Word ĠError ĠRate Ġ100 Ġ( WER Ġ100 ) Ġis Ġthe Ġpercentage Ġof Ġwords Ġin Ġthe Ġtest Ġset Ġfor Ġwhich Ġthe Ġcorrect Ġguess Ġis Ġnot Ġin Ġthe Ġfirst Ġ100 Ġguesses Ġof Ġthe Ġsystem . </s> In Ġsystem Ġevaluations , ĠW ER , ĠW ER Ġ100 , Ġand ĠPER Ġnumbers Ġpresented Ġfor Ġmultiple Ġlanguages Ġare Ġaveraged , Ġweight ing Ġeach Ġlanguage Ġequally ĠB IB REF 13 Ġ. </s> It Ġwould Ġbe Ġinteresting Ġto Ġcompute Ġerror Ġmetrics Ġthat Ġincorporate Ġphon eme Ġsimilarity , Ġsuch Ġas Ġthose Ġproposed Ġby Ġh ixon 2011 ph on emic . ĠPER Ġweights Ġall Ġphon eme Ġerrors Ġthe Ġsame , Ġeven Ġthough Ġsome Ġerrors Ġare Ġmore Ġharmful Ġthan Ġothers : Ġand Ġare Ġusually Ġcontrast ive , Ġwhereas Ġand Ġalmost Ġnever Ġare . ĠSuch Ġstatistics Ġwould Ġbe Ġespecially Ġinteresting Ġfor Ġevaluating Ġa Ġmult ilingual Ġsystem , Ġbecause Ġdifferent Ġlanguages Ġoften Ġmap Ġthe Ġsame Ġgrap he me Ġto Ġphon em es Ġthat Ġare Ġonly Ġsubtly Ġdifferent Ġfrom Ġeach Ġother . ĠHowever , Ġthese Ġstatistics Ġhave Ġnot Ġbeen Ġwidely Ġreported Ġfor Ġother Ġg 2 p Ġsystems , Ġso Ġwe Ġomit Ġthem Ġhere . </s> Bas eline </s> Results Ġon ĠLang ID Ġand ĠNo L ang ID Ġare Ġcompared Ġto Ġthe Ġsystem Ġpresented Ġby Ġder i 2016 g rap he me , Ġwhich Ġis Ġidentified Ġin Ġour Ġresults Ġas Ġw F ST . ĠTheir Ġresults Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġparts : </s> High Ġresource Ġresults , Ġcomputed Ġwith Ġw F ST s Ġtrained Ġon Ġa Ġcombination Ġof ĠWik tion ary Ġpronunciation Ġdata Ġand Ġg 2 p Ġrules Ġextracted Ġfrom ĠWikipedia ĠIPA ĠHelp Ġpages . ĠThey Ġreport Ġhigh Ġresource Ġresults Ġfor Ġ85 Ġlanguages . </s> Adapt ed Ġresults , Ġwhere Ġthey Ġapply Ġvarious Ġmapping Ġstrategies Ġin Ġorder Ġto Ġadapt Ġhigh Ġresource Ġmodels Ġto Ġother Ġlanguages . ĠThe Ġfinal Ġadapted Ġresults Ġthey Ġreported Ġinclude Ġmost Ġof Ġthe Ġ85 Ġlanguages Ġwith Ġhigh Ġresource Ġresults , Ġas Ġwell Ġas Ġthe Ġvarious Ġlanguages Ġthey Ġwere Ġable Ġto Ġadapt Ġthem Ġfor , Ġfor Ġa Ġtotal Ġof Ġ229 Ġlanguages . ĠThis Ġtest Ġset Ġo mits Ġ23 Ġof Ġthe Ġhigh Ġresource Ġlanguages Ġthat Ġare Ġwritten Ġin Ġunique Ġscripts Ġor Ġfor Ġwhich Ġlanguage Ġdistance Ġmetrics Ġcould Ġnot Ġbe Ġcomputed . </s> Training </s> We Ġtrain Ġthe ĠLang ID Ġand ĠNo L ang ID Ġversions Ġof Ġour Ġmodel Ġeach Ġon Ġthree Ġsubs ets Ġof Ġthe ĠWik tion ary Ġdata : </s> L ang ID - High Ġand ĠNo L ang ID - High : ĠTr ained Ġon Ġdata Ġfrom Ġthe Ġ85 Ġlanguages Ġfor Ġwhich ĠB IB REF 13 Ġused Ġnon - adapt ed Ġw F ST Ġmodels . </s> L ang ID - Adapt ed Ġand ĠNo L ang ID - Adapt ed : ĠTr ained Ġon Ġdata Ġfrom Ġany Ġof Ġthe Ġ229 Ġlanguages Ġfor Ġwhich Ġthey Ġbuilt Ġadapted Ġmodels . ĠBecause Ġmany Ġof Ġthese Ġlanguages Ġhad Ġno Ġtraining Ġdata Ġat Ġall , Ġthe Ġmodel Ġis Ġactually Ġonly Ġtrained Ġon Ġdata Ġin Ġ157 Ġlanguages . ĠAs Ġis Ġnoted Ġabove , Ġthe ĠAdapt ed Ġset Ġo mits Ġ23 Ġlanguages Ġwhich Ġare Ġin Ġthe ĠHigh Ġtest Ġset . </s> L ang ID - All Ġand ĠNo L ang ID - All : ĠTr ained Ġon Ġdata Ġin Ġall Ġ311 Ġlanguages Ġin Ġthe ĠWik tion ary Ġtraining Ġcorpus . </s> In Ġorder Ġto Ġease Ġcomparison Ġto ĠDer i Ġand ĠKnight 's Ġsystem , Ġwe Ġlimited Ġour Ġuse Ġof Ġthe Ġtraining Ġcorpus Ġto Ġ10 , 000 Ġwords Ġper Ġlanguage . ĠWe Ġset Ġaside Ġ10 Ġpercent Ġof Ġthe Ġdata Ġin Ġeach Ġlanguage Ġfor Ġvalidation , Ġso Ġthe Ġmaximum Ġnumber Ġof Ġtraining Ġwords Ġfor Ġany Ġlanguage Ġis Ġ9000 Ġfor Ġour Ġsystems . </s> Adapt ed ĠResults </s> On Ġthe Ġ229 Ġlanguages Ġfor Ġwhich Ġder i 2016 g rap he me Ġpresented Ġtheir Ġfinal Ġresults , Ġthe ĠLang ID Ġversion Ġof Ġour Ġsystem Ġoutper forms Ġthe Ġbaseline Ġby Ġa Ġwide Ġmargin . ĠThe Ġbest Ġperformance Ġcame Ġwith Ġthe Ġversion Ġof Ġour Ġmodel Ġthat Ġwas Ġtrained Ġon Ġdata Ġin Ġall Ġavailable Ġlanguages , Ġnot Ġjust Ġthe Ġlanguages Ġit Ġwas Ġtested Ġon . ĠUsing Ġa Ġlanguage ĠID Ġtoken Ġimproves Ġresults Ġconsiderably , Ġbut Ġeven ĠNo L ang ID Ġbeats Ġthe Ġbaseline Ġin ĠW ER Ġand ĠW ER Ġ100 . ĠFull Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 24 Ġ. </s> High ĠResource ĠResults </s> Having Ġshown Ġthat Ġour Ġmodel Ġexceeds Ġthe Ġperformance Ġof Ġthe Ġw F ST - adapt ation Ġapproach , Ġwe Ġnext Ġcompare Ġit Ġto Ġthe Ġbaseline Ġmodels Ġfor Ġjust Ġhigh Ġresource Ġlanguages . ĠThe Ġw F ST Ġmodels Ġhere Ġare Ġpurely Ġmon oling ual ĠâĢĵ Ġthey Ġdo Ġnot Ġuse Ġdata Ġadaptation Ġbecause Ġthere Ġis Ġsufficient Ġtraining Ġdata Ġfor Ġeach Ġof Ġthem . ĠFull Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 26 Ġ. ĠWe Ġomit Ġmodels Ġtrained Ġon Ġthe ĠAdapt ed Ġlanguages Ġbecause Ġthey Ġwere Ġnot Ġtrained Ġon Ġhigh Ġresource Ġlanguages Ġwith Ġunique Ġwriting Ġsystems , Ġsuch Ġas ĠGeorgian Ġand ĠGreek , Ġand Ġconsequently Ġperformed Ġvery Ġpoorly Ġon Ġthem . </s> In Ġcontrast Ġto Ġthe Ġlarger - scale ĠAdapt ed Ġresults , Ġin Ġthe ĠHigh ĠResource Ġexperiments Ġnone Ġof Ġthe Ġsequence - to - sequence Ġapproaches Ġequal Ġthe Ġperformance Ġof Ġthe Ġw F ST Ġmodel Ġin ĠW ER Ġand ĠPER , Ġalthough ĠLang ID - High Ġdoes Ġcome Ġclose . ĠThe ĠLang ID Ġmodels Ġdo Ġbeat Ġw F ST Ġin ĠW ER Ġ100 . ĠA Ġpossible Ġexplanation Ġis Ġthat Ġa Ġmon oling ual Ġw F ST Ġmodel Ġwill Ġnever Ġgenerate Ġphon em es Ġthat Ġare Ġnot Ġpart Ġof Ġthe Ġlanguage 's Ġinventory . ĠA Ġmult ilingual Ġmodel , Ġon Ġthe Ġother Ġhand , Ġcould Ġpotentially Ġgenerate Ġphon em es Ġfrom Ġthe Ġinvent ories Ġof Ġany Ġlanguage Ġit Ġhas Ġbeen Ġtrained Ġon . </s> Even Ġif ĠLang ID - High Ġdoes Ġnot Ġpresent Ġa Ġmore Ġaccurate Ġresult , Ġit Ġdoes Ġpresent Ġa Ġmore Ġcompact Ġone : ĠLang ID - High Ġis Ġ15 . 4 ĠMB , Ġwhile Ġthe Ġcombined Ġw F ST Ġhigh Ġresource Ġmodels Ġare Ġ197 . 5 ĠMB . </s> Results Ġon ĠUn seen ĠLanguages </s> Finally , Ġwe Ġreport Ġour Ġmodels ' Ġresults Ġon Ġunseen Ġlanguages Ġin ĠTable ĠT AB REF 28 Ġ. ĠThe Ġunseen Ġlanguages Ġare Ġany Ġthat Ġare Ġpresent Ġin Ġthe Ġtest Ġcorpus Ġbut Ġabsent Ġfrom Ġthe Ġtraining Ġdata . ĠDer i Ġand ĠKnight Ġdid Ġnot Ġreport Ġresults Ġspecifically Ġon Ġthese Ġlanguages . ĠAlthough Ġthe ĠNo L ang ID Ġmodels Ġsometimes Ġdo Ġbetter Ġon ĠW ER Ġ100 , Ġeven Ġhere Ġthe ĠLang ID Ġmodels Ġhave Ġa Ġslight Ġadvantage Ġin ĠW ER Ġand ĠPER . ĠThis Ġis Ġsomewhat Ġsurprising Ġbecause Ġthe ĠLang ID Ġmodels Ġhave Ġnot Ġlearned Ġembed d ings Ġfor Ġthe Ġlanguage ĠID Ġtokens Ġof Ġunseen Ġlanguages . ĠPerhaps Ġnegative Ġassociations Ġare Ġalso Ġbeing Ġlearned , Ġdriving Ġthe Ġmodel Ġtowards Ġpredicting Ġmore Ġcommon Ġpron unci ations Ġfor Ġunseen Ġlanguages . </s> Language ĠID ĠTokens </s> Adding Ġa Ġlanguage ĠID Ġtoken Ġalways Ġimproves Ġresults Ġin Ġcases Ġwhere Ġan Ġembed ding Ġhas Ġbeen Ġlearned Ġfor Ġthat Ġtoken . ĠThe Ġpower Ġof Ġthese Ġembed d ings Ġis Ġdemonstrated Ġby Ġwhat Ġhappens Ġwhen Ġone Ġfeeds Ġthe Ġsame Ġinput Ġword Ġto Ġthe Ġmodel Ġwith Ġdifferent Ġlanguage Ġtokens , Ġas Ġis Ġseen Ġin ĠTable ĠT AB REF 30 Ġ. ĠIm press ively , Ġthis Ġeven Ġworks Ġwhen Ġthe Ġsource Ġsequence Ġis Ġin Ġthe Ġwrong Ġscript Ġfor Ġthe Ġlanguage , Ġas Ġis Ġseen Ġin Ġthe Ġentry Ġfor ĠArabic . </s> Language ĠEmb edd ings </s> Because Ġthese Ġlanguage ĠID Ġtokens Ġare Ġso Ġuseful , Ġit Ġwould Ġbe Ġgood Ġif Ġthey Ġcould Ġbe Ġeffectively Ġestimated Ġfor Ġunseen Ġlanguages . Ġost ling 2017 contin uous Ġfound Ġthat Ġthe Ġlanguage Ġvectors Ġtheir Ġmodels Ġlearned Ġcorrelated Ġwell Ġto Ġgenetic Ġrelationships , Ġso Ġit Ġwould Ġbe Ġinteresting Ġto Ġsee Ġif Ġthe Ġembed d ings Ġour Ġsource Ġenc oder Ġlearned Ġfor Ġthe Ġlanguage ĠID Ġtokens Ġshowed Ġanything Ġsimilar . ĠIn Ġa Ġfew Ġcases Ġthey Ġdo Ġ( the Ġlanguages Ġclosest Ġto ĠGerman Ġin Ġthe Ġvector Ġspace Ġare ĠLuxembourg ish , ĠBav arian , Ġand ĠY idd ish , Ġall Ġclose Ġrelatives ). ĠHowever , Ġfor Ġthe Ġmost Ġpart Ġthe Ġstructure Ġof Ġthese Ġvectors Ġis Ġnot Ġinterpret able . ĠTherefore , Ġit Ġwould Ġbe Ġdifficult Ġto Ġestimate Ġthe Ġembed ding Ġfor Ġan Ġunseen Ġlanguage , Ġor Ġto ĠâĢ ľ bor row âĢ Ŀ Ġthe Ġlanguage ĠID Ġtoken Ġof Ġa Ġsimilar Ġlanguage . ĠA Ġmore Ġpromising Ġway Ġforward Ġis Ġto Ġfind Ġa Ġmodel Ġthat Ġuses Ġan Ġexternally Ġconstructed Ġtyp ological Ġrepresentation Ġof Ġthe Ġlanguage . </s> Ph on eme ĠEmb edd ings </s> In Ġcontrast Ġto Ġthe Ġlanguage Ġembed d ings , Ġthe Ġphon eme Ġembed d ings Ġappear Ġto Ġshow Ġmany Ġregular ities Ġ( see ĠTable ĠT AB REF 33 Ġ). ĠThis Ġis Ġa Ġsign Ġthat Ġour Ġmult ilingual Ġmodel Ġlearns Ġsimilar Ġembed d ings Ġfor Ġphon em es Ġthat Ġare Ġwritten Ġwith Ġthe Ġsame Ġgrap he me Ġin Ġdifferent Ġlanguages . ĠThese Ġphon em es Ġtend Ġto Ġbe Ġphon etically Ġsimilar Ġto Ġeach Ġother . </s> Perhaps Ġthe Ġstructure Ġof Ġthe Ġphon eme Ġembed ding Ġspace Ġis Ġwhat Ġleads Ġto Ġour Ġmodels ' Ġvery Ġgood Ġperformance Ġon ĠW ER Ġ100 . ĠEven Ġwhen Ġthe Ġmodel 's Ġfirst Ġpredicted Ġpronunciation Ġis Ġnot Ġcorrect , Ġit Ġtends Ġto Ġassign Ġmore Ġprobability Ġmass Ġto Ġguesses Ġthat Ġare Ġmore Ġsimilar Ġto Ġthe Ġcorrect Ġone . ĠApp lying Ġsome Ġsort Ġof Ġfiltering Ġor Ġre ranking Ġof Ġthe Ġsystem Ġoutput Ġmight Ġtherefore Ġlead Ġto Ġbetter Ġperformance . </s> Future ĠWork </s> Because Ġthe Ġlanguage ĠID Ġtoken Ġis Ġso Ġbeneficial Ġto Ġperformance , Ġit Ġwould Ġbe Ġvery Ġinteresting Ġto Ġfind Ġways Ġto Ġextend Ġa Ġsimilar Ġbenefit Ġto Ġunseen Ġlanguages . ĠOne Ġpossible Ġway Ġto Ġdo Ġso Ġis Ġwith Ġtokens Ġthat Ġidentify Ġsomething Ġother Ġthan Ġthe Ġlanguage , Ġsuch Ġas Ġtyp ological Ġfeatures Ġabout Ġthe Ġlanguage 's Ġphon emic Ġinventory . ĠThis Ġcould Ġenable Ġbetter Ġsharing Ġof Ġresources Ġamong Ġlanguages . ĠSuch Ġtyp ological Ġknowledge Ġis Ġreadily Ġavailable Ġin Ġdatabases Ġlike ĠPh o ible Ġand ĠW ALS Ġfor Ġa Ġwide Ġvariety Ġof Ġlanguages . ĠIt Ġwould Ġbe Ġinteresting Ġto Ġexplore Ġif Ġany Ġof Ġthese Ġfeatures Ġis Ġa Ġgood Ġpredictor Ġof Ġa Ġlanguage 's Ġorth ographic Ġrules . </s> It Ġwould Ġalso Ġbe Ġinteresting Ġto Ġapply Ġthe Ġartificial Ġtoken Ġapproach Ġto Ġother Ġproblems Ġbesides Ġmult ilingual Ġg 2 p . ĠOne Ġclosely Ġrelated Ġapplication Ġis Ġmon oling ual ĠEnglish Ġg 2 p . ĠSome Ġof Ġthe Ġambiguity Ġof ĠEnglish Ġspelling Ġis Ġdue Ġto Ġthe Ġwide Ġvariety Ġof Ġloan words Ġin Ġthe Ġlanguage , Ġmany Ġof Ġwhich Ġhave Ġun ass imil ated Ġspell ings . ĠKnowing Ġthe Ġorigins Ġof Ġthese Ġloan words Ġcould Ġprovide Ġa Ġuseful Ġhint Ġfor Ġfiguring Ġout Ġtheir Ġpron unci ations . ĠThe Ġe tymology Ġof Ġa Ġword Ġcould Ġbe Ġtagged Ġin Ġan Ġanalogous Ġway Ġto Ġhow Ġlanguage ĠID Ġis Ġtagged Ġin Ġmult ilingual Ġg 2 p .
Introduction Accurate grapheme-to-phoneme conversion (g2p) is important for any application that depends on the sometimes inconsistent relationship between spoken and written language. Most prominently, this includes text-to-speech and automatic speech recognition. Most work on g2p has focused on a few languages for which extensive pronunciation data is available BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Most languages lack these resources. However, a low resource language's writing system is likely to be similar to the writing systems of languages that do have sufficient pronunciation data. Therefore g2p may be possible for low resource languages if this high resource data can be properly utilized. We attempt to leverage high resource data by treating g2p as a multisource neural machine translation (NMT) problem. The source sequences for our system are words in the standard orthography in any language. The target sequences are the corresponding representation in the International Phonetic Alphabet (IPA). Our results show that the parameters learned by the shared encoder–decoder are able to exploit the orthographic and phonemic similarities between the various languages in our data. Low Resource g2p Our approach is similar in goal to deri2016grapheme's model for adapting high resource g2p models for low resource languages. They trained weighted finite state transducer (wFST) models on a variety of high resource languages, then transferred those models to low resource languages, using a language distance metric to choose which high resource models to use and a phoneme distance metric to map the high resource language's phonemes to the low resource language's phoneme inventory. These distance metrics are computed based on data from Phoible BIBREF4 and URIEL BIBREF5 . Other low resource g2p systems have used a strategy of combining multiple models. schlippe2014combining trained several data-driven g2p systems on varying quantities of monolingual data and combined their outputs with a phoneme-level voting scheme. This led to improvements over the best-performing single system for small quantities of data in some languages. jyothilow trained recurrent neural networks for small data sets and found that a version of their system that combined the neural network output with the output of the wFST-based Phonetisaurus system BIBREF1 did better than either system alone. A different approach came from kim2012universal, who used supervised learning with an undirected graphical model to induce the grapheme–phoneme mappings for languages written in the Latin alphabet. Given a short text in a language, the model predicts the language's orthographic rules. To create phonemic context features from the short text, the model naïvely maps graphemes to IPA symbols written with the same character, and uses the features of these symbols to learn an approximation of the phonotactic constraints of the language. In their experiments, these phonotactic features proved to be more valuable than geographical and genetic features drawn from WALS BIBREF6 . Multilingual Neural NLP In recent years, neural networks have emerged as a common way to use data from several languages in a single system. Google's zero-shot neural machine translation system BIBREF7 shares an encoder and decoder across all language pairs. In order to facilitate this multi-way translation, they prepend an artificial token to the beginning of each source sentence at both training and translation time. The token identifies what language the sentence should be translated to. This approach has three benefits: it is far more efficient than building a separate model for each language pair; it allows for translation between languages that share no parallel data; and it improves results on low-resource languages by allowing them to implicitly share parameters with high-resource languages. Our g2p system is inspired by this approach, although it differs in that there is only one target “language”, IPA, and the artificial tokens identify the language of the source instead of the language of the target. Other work has also made use of multilingually-trained neural networks. Phoneme-level polyglot language models BIBREF8 train a single model on multiple languages and additionally condition on externally constructed typological data about the language. ostling2017continuous used a similar approach, in which a character-level neural language model is trained on a massively multilingual corpus. A language embedding vector is concatenated to the input at each time step. The language embeddings their system learned correlate closely to the genetic relationships between languages. However, neither of these models was applied to g2p. Grapheme-to-Phoneme g2p is the problem of converting the orthographic representation of a word into a phonemic representation. A phoneme is an abstract unit of sound which may have different realizations in different contexts. For example, the English phoneme has two phonetic realizations (or allophones): English speakers without linguistic training often struggle to perceive any difference between these sounds. Writing systems usually do not distinguish between allophones: and are both written as INLINEFORM0 p INLINEFORM1 in English. The sounds are written differently in languages where they contrast, such as Hindi and Eastern Armenian. Most writing systems in use today are glottographic, meaning that their symbols encode solely phonological information. But despite being glottographic, in few writing systems do graphemes correspond one-to-one with phonemes. There are cases in which multiple graphemes represent a single phoneme, as in the word the in English: There are cases in which a single grapheme represents multiple phonemes, such as syllabaries, in which each symbol represents a syllable. In many languages, there are silent letters, as in the word hora in Spanish: There are more complicated correspondences, such as the silent e in English that affects the pronunciation of the previous vowel, as seen in the pair of words cape and cap. It is possible for an orthographic system to have any or all of the above phenomena while remaining unambiguous. However, some orthographic systems contain ambiguities. English is well-known for its spelling ambiguities. Abjads, used for Arabic and Hebrew, do not give full representation to vowels. Consequently, g2p is harder than simply replacing each grapheme symbol with a corresponding phoneme symbol. It is the problem of replacing a grapheme sequence INLINEFORM0 with a phoneme sequence INLINEFORM0 where the sequences are not necessarily of the same length. Data-driven g2p is therefore the problem of finding the phoneme sequence that maximizes the likelihood of the grapheme sequence: INLINEFORM0 Data-driven approaches are especially useful for problems in which the rules that govern them are complex and difficult to engineer by hand. g2p for languages with ambiguous orthographies is such a problem. Multilingual g2p, in which the various languages have similar but different and possibly contradictory spelling rules, can be seen as an extreme case of that. Therefore, a data-driven sequence-to-sequence model is a natural choice. Encoder–Decoder Models In order to find the best phoneme sequence, we use a neural encoder–decoder model with attention BIBREF9 . The model consists of two main parts: the encoder compresses each source grapheme sequence INLINEFORM0 into a fixed-length vector. The decoder, conditioned on this fixed-length vector, generates the output phoneme sequence INLINEFORM1 . The encoder and decoder are both implemented as recurrent neural networks, which have the advantage of being able to process sequences of arbitrary length and use long histories efficiently. They are trained jointly to minimize cross-entropy on the training data. We had our best results when using a bidirectional encoder, which consists of two separate encoders which process the input in forward and reverse directions. We used long short-term memory units BIBREF10 for both the encoder and decoder. For the attention mechanism, we used the general global attention architecture described by luong2015effective. We implemented all models with OpenNMT BIBREF11 . Our hyperparameters, which we determined by experimentation, are listed in Table TABREF8 . Training Multilingual Models Presenting pronunciation data in several languages to the network might create problems because different languages have different pronunciation patterns. For example, the string `real' is pronounced differently in English, German, Spanish, and Portuguese. We solve this problem by prepending each grapheme sequence with an artificial token consisting of the language's ISO 639-3 code enclosed in angle brackets. The English word `real', for example, would be presented to the system as INLINEFORM0 eng INLINEFORM1 r e a l The artificial token is treated simply as an element of the grapheme sequence. This is similar to the approach taken by johnson2016google in their zero-shot NMT system. However, their source-side artificial tokens identify the target language, whereas ours identify the source language. An alternative approach, used by ostling2017continuous, would be to concatenate a language embedding to the input at each time step. They do not evaluate their approach on grapheme-to-phoneme conversion. Data In order to train a neural g2p system, one needs a large quantity of pronunciation data. A standard dataset for g2p is the Carnegie Mellon Pronouncing Dictionary BIBREF12 . However, that is a monolingual English resource, so it is unsuitable for our multilingual task. Instead, we use the multilingual pronunciation corpus collected by deri2016grapheme for all experiments. This corpus consists of spelling–pronunciation pairs extracted from Wiktionary. It is already partitioned into training and test sets. Corpus statistics are presented in Table TABREF10 . In addition to the raw IPA transcriptions extracted from Wiktionary, the corpus provides an automatically cleaned version of transcriptions. Cleaning is a necessary step because web-scraped data is often noisy and may be transcribed at an inconsistent level of detail. The data cleaning used here attempts to make the transcriptions consistent with the phonemic inventories used in Phoible BIBREF4 . When a transcription contains a phoneme that is not in its language's inventory in Phoible, that phoneme is replaced by the phoneme with the most similar articulatory features that is in the language's inventory. Sometimes this cleaning algorithm works well: in the German examples in Table TABREF11 , the raw German symbols and are both converted to . This is useful because the in Ansbach and the in Kaninchen are instances of the same phoneme, so their phonemic representations should use the same symbol. However, the cleaning algorithm can also have negative effects on the data quality. For example, the phoneme is not present in the Phoible inventory for German, but it is used in several German transcriptions in the corpus. The cleaning algorithm converts to in all German transcriptions, whereas would be a more reasonable guess. The cleaning algorithm also removes most suprasegmentals, even though these are often an important part of a language's phonology. Developing a more sophisticated procedure for cleaning pronunciation data is a direction for future work, but in this paper we use the corpus's provided cleaned transcriptions in order to ease comparison to previous results. Experiments We present experiments with two versions of our sequence-to-sequence model. LangID prepends each training, validation, and test sample with an artificial token identifying the language of the sample. NoLangID omits this token. LangID and NoLangID have identical structure otherwise. To translate the test corpus, we used a beam width of 100. Although this is an unusually wide beam and had negligible performance effects, it was necessary to compute our error metrics. Evaluation We use the following three evaluation metrics: Phoneme Error Rate (PER) is the Levenshtein distance between the predicted phoneme sequences and the gold standard phoneme sequences, divided by the length of the gold standard phoneme sequences. Word Error Rate (WER) is the percentage of words in which the predicted phoneme sequence does not exactly match the gold standard phoneme sequence. Word Error Rate 100 (WER 100) is the percentage of words in the test set for which the correct guess is not in the first 100 guesses of the system. In system evaluations, WER, WER 100, and PER numbers presented for multiple languages are averaged, weighting each language equally BIBREF13 . It would be interesting to compute error metrics that incorporate phoneme similarity, such as those proposed by hixon2011phonemic. PER weights all phoneme errors the same, even though some errors are more harmful than others: and are usually contrastive, whereas and almost never are. Such statistics would be especially interesting for evaluating a multilingual system, because different languages often map the same grapheme to phonemes that are only subtly different from each other. However, these statistics have not been widely reported for other g2p systems, so we omit them here. Baseline Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST. Their results can be divided into two parts: High resource results, computed with wFSTs trained on a combination of Wiktionary pronunciation data and g2p rules extracted from Wikipedia IPA Help pages. They report high resource results for 85 languages. Adapted results, where they apply various mapping strategies in order to adapt high resource models to other languages. The final adapted results they reported include most of the 85 languages with high resource results, as well as the various languages they were able to adapt them for, for a total of 229 languages. This test set omits 23 of the high resource languages that are written in unique scripts or for which language distance metrics could not be computed. Training We train the LangID and NoLangID versions of our model each on three subsets of the Wiktionary data: LangID-High and NoLangID-High: Trained on data from the 85 languages for which BIBREF13 used non-adapted wFST models. LangID-Adapted and NoLangID-Adapted: Trained on data from any of the 229 languages for which they built adapted models. Because many of these languages had no training data at all, the model is actually only trained on data in 157 languages. As is noted above, the Adapted set omits 23 languages which are in the High test set. LangID-All and NoLangID-All: Trained on data in all 311 languages in the Wiktionary training corpus. In order to ease comparison to Deri and Knight's system, we limited our use of the training corpus to 10,000 words per language. We set aside 10 percent of the data in each language for validation, so the maximum number of training words for any language is 9000 for our systems. Adapted Results On the 229 languages for which deri2016grapheme presented their final results, the LangID version of our system outperforms the baseline by a wide margin. The best performance came with the version of our model that was trained on data in all available languages, not just the languages it was tested on. Using a language ID token improves results considerably, but even NoLangID beats the baseline in WER and WER 100. Full results are presented in Table TABREF24 . High Resource Results Having shown that our model exceeds the performance of the wFST-adaptation approach, we next compare it to the baseline models for just high resource languages. The wFST models here are purely monolingual – they do not use data adaptation because there is sufficient training data for each of them. Full results are presented in Table TABREF26 . We omit models trained on the Adapted languages because they were not trained on high resource languages with unique writing systems, such as Georgian and Greek, and consequently performed very poorly on them. In contrast to the larger-scale Adapted results, in the High Resource experiments none of the sequence-to-sequence approaches equal the performance of the wFST model in WER and PER, although LangID-High does come close. The LangID models do beat wFST in WER 100. A possible explanation is that a monolingual wFST model will never generate phonemes that are not part of the language's inventory. A multilingual model, on the other hand, could potentially generate phonemes from the inventories of any language it has been trained on. Even if LangID-High does not present a more accurate result, it does present a more compact one: LangID-High is 15.4 MB, while the combined wFST high resource models are 197.5 MB. Results on Unseen Languages Finally, we report our models' results on unseen languages in Table TABREF28 . The unseen languages are any that are present in the test corpus but absent from the training data. Deri and Knight did not report results specifically on these languages. Although the NoLangID models sometimes do better on WER 100, even here the LangID models have a slight advantage in WER and PER. This is somewhat surprising because the LangID models have not learned embeddings for the language ID tokens of unseen languages. Perhaps negative associations are also being learned, driving the model towards predicting more common pronunciations for unseen languages. Language ID Tokens Adding a language ID token always improves results in cases where an embedding has been learned for that token. The power of these embeddings is demonstrated by what happens when one feeds the same input word to the model with different language tokens, as is seen in Table TABREF30 . Impressively, this even works when the source sequence is in the wrong script for the language, as is seen in the entry for Arabic. Language Embeddings Because these language ID tokens are so useful, it would be good if they could be effectively estimated for unseen languages. ostling2017continuous found that the language vectors their models learned correlated well to genetic relationships, so it would be interesting to see if the embeddings our source encoder learned for the language ID tokens showed anything similar. In a few cases they do (the languages closest to German in the vector space are Luxembourgish, Bavarian, and Yiddish, all close relatives). However, for the most part the structure of these vectors is not interpretable. Therefore, it would be difficult to estimate the embedding for an unseen language, or to “borrow” the language ID token of a similar language. A more promising way forward is to find a model that uses an externally constructed typological representation of the language. Phoneme Embeddings In contrast to the language embeddings, the phoneme embeddings appear to show many regularities (see Table TABREF33 ). This is a sign that our multilingual model learns similar embeddings for phonemes that are written with the same grapheme in different languages. These phonemes tend to be phonetically similar to each other. Perhaps the structure of the phoneme embedding space is what leads to our models' very good performance on WER 100. Even when the model's first predicted pronunciation is not correct, it tends to assign more probability mass to guesses that are more similar to the correct one. Applying some sort of filtering or reranking of the system output might therefore lead to better performance. Future Work Because the language ID token is so beneficial to performance, it would be very interesting to find ways to extend a similar benefit to unseen languages. One possible way to do so is with tokens that identify something other than the language, such as typological features about the language's phonemic inventory. This could enable better sharing of resources among languages. Such typological knowledge is readily available in databases like Phoible and WALS for a wide variety of languages. It would be interesting to explore if any of these features is a good predictor of a language's orthographic rules. It would also be interesting to apply the artificial token approach to other problems besides multilingual g2p. One closely related application is monolingual English g2p. Some of the ambiguity of English spelling is due to the wide variety of loanwords in the language, many of which have unassimilated spellings. Knowing the origins of these loanwords could provide a useful hint for figuring out their pronunciations. The etymology of a word could be tagged in an analogous way to how language ID is tagged in multilingual g2p.
what was the baseline?
0752d71a0a1f73b3482a888313622ce9e9870d6e
what Ġwas Ġthe Ġbaseline ?
system Ġpresented Ġby Ġder i 2016 g rap he me
<s> what Ġwas Ġthe Ġbaseline ? </s> Introduction </s> Acc urate Ġgrap he me - to - ph on eme Ġconversion Ġ( g 2 p ) Ġis Ġimportant Ġfor Ġany Ġapplication Ġthat Ġdepends Ġon Ġthe Ġsometimes Ġinconsistent Ġrelationship Ġbetween Ġspoken Ġand Ġwritten Ġlanguage . ĠMost Ġprominently , Ġthis Ġincludes Ġtext - to - speech Ġand Ġautomatic Ġspeech Ġrecognition . ĠMost Ġwork Ġon Ġg 2 p Ġhas Ġfocused Ġon Ġa Ġfew Ġlanguages Ġfor Ġwhich Ġextensive Ġpronunciation Ġdata Ġis Ġavailable ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠMost Ġlanguages Ġlack Ġthese Ġresources . ĠHowever , Ġa Ġlow Ġresource Ġlanguage 's Ġwriting Ġsystem Ġis Ġlikely Ġto Ġbe Ġsimilar Ġto Ġthe Ġwriting Ġsystems Ġof Ġlanguages Ġthat Ġdo Ġhave Ġsufficient Ġpronunciation Ġdata . ĠTherefore Ġg 2 p Ġmay Ġbe Ġpossible Ġfor Ġlow Ġresource Ġlanguages Ġif Ġthis Ġhigh Ġresource Ġdata Ġcan Ġbe Ġproperly Ġutilized . </s> We Ġattempt Ġto Ġleverage Ġhigh Ġresource Ġdata Ġby Ġtreating Ġg 2 p Ġas Ġa Ġmult is ource Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġproblem . ĠThe Ġsource Ġsequences Ġfor Ġour Ġsystem Ġare Ġwords Ġin Ġthe Ġstandard Ġorth ography Ġin Ġany Ġlanguage . ĠThe Ġtarget Ġsequences Ġare Ġthe Ġcorresponding Ġrepresentation Ġin Ġthe ĠInternational ĠPh on etic ĠAlphabet Ġ( IP A ). ĠOur Ġresults Ġshow Ġthat Ġthe Ġparameters Ġlearned Ġby Ġthe Ġshared Ġenc oder âĢĵ dec oder Ġare Ġable Ġto Ġexploit Ġthe Ġorth ographic Ġand Ġphon emic Ġsimilarities Ġbetween Ġthe Ġvarious Ġlanguages Ġin Ġour Ġdata . </s> Low ĠResource Ġg 2 p </s> Our Ġapproach Ġis Ġsimilar Ġin Ġgoal Ġto Ġder i 2016 g rap he me 's Ġmodel Ġfor Ġadapting Ġhigh Ġresource Ġg 2 p Ġmodels Ġfor Ġlow Ġresource Ġlanguages . ĠThey Ġtrained Ġweighted Ġfinite Ġstate Ġtrans du cer Ġ( w F ST ) Ġmodels Ġon Ġa Ġvariety Ġof Ġhigh Ġresource Ġlanguages , Ġthen Ġtransferred Ġthose Ġmodels Ġto Ġlow Ġresource Ġlanguages , Ġusing Ġa Ġlanguage Ġdistance Ġmetric Ġto Ġchoose Ġwhich Ġhigh Ġresource Ġmodels Ġto Ġuse Ġand Ġa Ġphon eme Ġdistance Ġmetric Ġto Ġmap Ġthe Ġhigh Ġresource Ġlanguage 's Ġphon em es Ġto Ġthe Ġlow Ġresource Ġlanguage 's Ġphon eme Ġinventory . ĠThese Ġdistance Ġmetrics Ġare Ġcomputed Ġbased Ġon Ġdata Ġfrom ĠPh o ible ĠB IB REF 4 Ġand ĠURI EL ĠB IB REF 5 Ġ. </s> Other Ġlow Ġresource Ġg 2 p Ġsystems Ġhave Ġused Ġa Ġstrategy Ġof Ġcombining Ġmultiple Ġmodels . Ġsch l ipp e 2014 comb ining Ġtrained Ġseveral Ġdata - driven Ġg 2 p Ġsystems Ġon Ġvarying Ġquantities Ġof Ġmon oling ual Ġdata Ġand Ġcombined Ġtheir Ġoutputs Ġwith Ġa Ġphon eme - level Ġvoting Ġscheme . ĠThis Ġled Ġto Ġimprovements Ġover Ġthe Ġbest - performing Ġsingle Ġsystem Ġfor Ġsmall Ġquantities Ġof Ġdata Ġin Ġsome Ġlanguages . Ġj y oth il ow Ġtrained Ġrecurrent Ġneural Ġnetworks Ġfor Ġsmall Ġdata Ġsets Ġand Ġfound Ġthat Ġa Ġversion Ġof Ġtheir Ġsystem Ġthat Ġcombined Ġthe Ġneural Ġnetwork Ġoutput Ġwith Ġthe Ġoutput Ġof Ġthe Ġw F ST - based ĠPh onet is aurus Ġsystem ĠB IB REF 1 Ġdid Ġbetter Ġthan Ġeither Ġsystem Ġalone . </s> A Ġdifferent Ġapproach Ġcame Ġfrom Ġk im 2012 universal , Ġwho Ġused Ġsupervised Ġlearning Ġwith Ġan Ġund irect ed Ġgraphical Ġmodel Ġto Ġinduce Ġthe Ġgrap he me âĢĵ ph on eme Ġm appings Ġfor Ġlanguages Ġwritten Ġin Ġthe ĠLatin Ġalphabet . ĠGiven Ġa Ġshort Ġtext Ġin Ġa Ġlanguage , Ġthe Ġmodel Ġpredicts Ġthe Ġlanguage 's Ġorth ographic Ġrules . ĠTo Ġcreate Ġphon emic Ġcontext Ġfeatures Ġfrom Ġthe Ġshort Ġtext , Ġthe Ġmodel ĠnaÃ¯ve ly Ġmaps Ġgrap hem es Ġto ĠIPA Ġsymbols Ġwritten Ġwith Ġthe Ġsame Ġcharacter , Ġand Ġuses Ġthe Ġfeatures Ġof Ġthese Ġsymbols Ġto Ġlearn Ġan Ġapproximation Ġof Ġthe Ġphon ot actic Ġconstraints Ġof Ġthe Ġlanguage . ĠIn Ġtheir Ġexperiments , Ġthese Ġphon ot actic Ġfeatures Ġproved Ġto Ġbe Ġmore Ġvaluable Ġthan Ġgeographical Ġand Ġgenetic Ġfeatures Ġdrawn Ġfrom ĠW ALS ĠB IB REF 6 Ġ. </s> Mult ilingual ĠNeural ĠN LP </s> In Ġrecent Ġyears , Ġneural Ġnetworks Ġhave Ġemerged Ġas Ġa Ġcommon Ġway Ġto Ġuse Ġdata Ġfrom Ġseveral Ġlanguages Ġin Ġa Ġsingle Ġsystem . ĠGoogle 's Ġzero - shot Ġneural Ġmachine Ġtranslation Ġsystem ĠB IB REF 7 Ġshares Ġan Ġenc oder Ġand Ġdec oder Ġacross Ġall Ġlanguage Ġpairs . ĠIn Ġorder Ġto Ġfacilitate Ġthis Ġmulti - way Ġtranslation , Ġthey Ġprep end Ġan Ġartificial Ġtoken Ġto Ġthe Ġbeginning Ġof Ġeach Ġsource Ġsentence Ġat Ġboth Ġtraining Ġand Ġtranslation Ġtime . ĠThe Ġtoken Ġidentifies Ġwhat Ġlanguage Ġthe Ġsentence Ġshould Ġbe Ġtranslated Ġto . ĠThis Ġapproach Ġhas Ġthree Ġbenefits : Ġit Ġis Ġfar Ġmore Ġefficient Ġthan Ġbuilding Ġa Ġseparate Ġmodel Ġfor Ġeach Ġlanguage Ġpair ; Ġit Ġallows Ġfor Ġtranslation Ġbetween Ġlanguages Ġthat Ġshare Ġno Ġparallel Ġdata ; Ġand Ġit Ġimproves Ġresults Ġon Ġlow - resource Ġlanguages Ġby Ġallowing Ġthem Ġto Ġimplicitly Ġshare Ġparameters Ġwith Ġhigh - resource Ġlanguages . ĠOur Ġg 2 p Ġsystem Ġis Ġinspired Ġby Ġthis Ġapproach , Ġalthough Ġit Ġdiffers Ġin Ġthat Ġthere Ġis Ġonly Ġone Ġtarget ĠâĢ ľ language âĢ Ŀ , ĠIPA , Ġand Ġthe Ġartificial Ġtokens Ġidentify Ġthe Ġlanguage Ġof Ġthe Ġsource Ġinstead Ġof Ġthe Ġlanguage Ġof Ġthe Ġtarget . </s> Other Ġwork Ġhas Ġalso Ġmade Ġuse Ġof Ġmult iling ually - trained Ġneural Ġnetworks . ĠPh on eme - level Ġpoly gl ot Ġlanguage Ġmodels ĠB IB REF 8 Ġtrain Ġa Ġsingle Ġmodel Ġon Ġmultiple Ġlanguages Ġand Ġadditionally Ġcondition Ġon Ġexternally Ġconstructed Ġtyp ological Ġdata Ġabout Ġthe Ġlanguage . Ġost ling 2017 contin uous Ġused Ġa Ġsimilar Ġapproach , Ġin Ġwhich Ġa Ġcharacter - level Ġneural Ġlanguage Ġmodel Ġis Ġtrained Ġon Ġa Ġmassively Ġmult ilingual Ġcorpus . ĠA Ġlanguage Ġembed ding Ġvector Ġis Ġconc aten ated Ġto Ġthe Ġinput Ġat Ġeach Ġtime Ġstep . ĠThe Ġlanguage Ġembed d ings Ġtheir Ġsystem Ġlearned Ġcorrelate Ġclosely Ġto Ġthe Ġgenetic Ġrelationships Ġbetween Ġlanguages . ĠHowever , Ġneither Ġof Ġthese Ġmodels Ġwas Ġapplied Ġto Ġg 2 p . </s> G rap he me - to - Ph on eme </s> g 2 p Ġis Ġthe Ġproblem Ġof Ġconverting Ġthe Ġorth ographic Ġrepresentation Ġof Ġa Ġword Ġinto Ġa Ġphon emic Ġrepresentation . ĠA Ġphon eme Ġis Ġan Ġabstract Ġunit Ġof Ġsound Ġwhich Ġmay Ġhave Ġdifferent Ġreal izations Ġin Ġdifferent Ġcontexts . ĠFor Ġexample , Ġthe ĠEnglish Ġphon eme Ġhas Ġtwo Ġphon etic Ġreal izations Ġ( or Ġall oph ones ): </s> English Ġspeakers Ġwithout Ġlinguistic Ġtraining Ġoften Ġstruggle Ġto Ġperceive Ġany Ġdifference Ġbetween Ġthese Ġsounds . ĠWriting Ġsystems Ġusually Ġdo Ġnot Ġdistinguish Ġbetween Ġall oph ones : Ġand Ġare Ġboth Ġwritten Ġas ĠIN LINE FORM 0 Ġp ĠIN LINE FORM 1 Ġin ĠEnglish . ĠThe Ġsounds Ġare Ġwritten Ġdifferently Ġin Ġlanguages Ġwhere Ġthey Ġcontrast , Ġsuch Ġas ĠHindi Ġand ĠEastern ĠArmenian . </s> Most Ġwriting Ġsystems Ġin Ġuse Ġtoday Ġare Ġgl ott ographic , Ġmeaning Ġthat Ġtheir Ġsymbols Ġencode Ġsolely Ġphon ological Ġinformation . ĠBut Ġdespite Ġbeing Ġgl ott ographic , Ġin Ġfew Ġwriting Ġsystems Ġdo Ġgrap hem es Ġcorrespond Ġone - to - one Ġwith Ġphon em es . ĠThere Ġare Ġcases Ġin Ġwhich Ġmultiple Ġgrap hem es Ġrepresent Ġa Ġsingle Ġphon eme , Ġas Ġin Ġthe Ġword Ġthe Ġin ĠEnglish : </s> There Ġare Ġcases Ġin Ġwhich Ġa Ġsingle Ġgrap he me Ġrepresents Ġmultiple Ġphon em es , Ġsuch Ġas Ġsyll ab aries , Ġin Ġwhich Ġeach Ġsymbol Ġrepresents Ġa Ġsyll able . </s> In Ġmany Ġlanguages , Ġthere Ġare Ġsilent Ġletters , Ġas Ġin Ġthe Ġword Ġhor a Ġin ĠSpanish : </s> There Ġare Ġmore Ġcomplicated Ġcorrespond ences , Ġsuch Ġas Ġthe Ġsilent Ġe Ġin ĠEnglish Ġthat Ġaffects Ġthe Ġpronunciation Ġof Ġthe Ġprevious Ġvowel , Ġas Ġseen Ġin Ġthe Ġpair Ġof Ġwords Ġcape Ġand Ġcap . </s> It Ġis Ġpossible Ġfor Ġan Ġorth ographic Ġsystem Ġto Ġhave Ġany Ġor Ġall Ġof Ġthe Ġabove Ġphenomena Ġwhile Ġremaining Ġunamb iguous . ĠHowever , Ġsome Ġorth ographic Ġsystems Ġcontain Ġambig u ities . ĠEnglish Ġis Ġwell - known Ġfor Ġits Ġspelling Ġambig u ities . ĠAb j ads , Ġused Ġfor ĠArabic Ġand ĠHebrew , Ġdo Ġnot Ġgive Ġfull Ġrepresentation Ġto Ġvow els . </s> C onsequ ently , Ġg 2 p Ġis Ġharder Ġthan Ġsimply Ġreplacing Ġeach Ġgrap he me Ġsymbol Ġwith Ġa Ġcorresponding Ġphon eme Ġsymbol . ĠIt Ġis Ġthe Ġproblem Ġof Ġreplacing Ġa Ġgrap he me Ġsequence ĠIN LINE FORM 0 </s> with Ġa Ġphon eme Ġsequence ĠIN LINE FORM 0 </s> where Ġthe Ġsequences Ġare Ġnot Ġnecessarily Ġof Ġthe Ġsame Ġlength . ĠData - driven Ġg 2 p Ġis Ġtherefore Ġthe Ġproblem Ġof Ġfinding Ġthe Ġphon eme Ġsequence Ġthat Ġmaxim izes Ġthe Ġlikelihood Ġof Ġthe Ġgrap he me Ġsequence : ĠIN LINE FORM 0 </s> Data - driven Ġapproaches Ġare Ġespecially Ġuseful Ġfor Ġproblems Ġin Ġwhich Ġthe Ġrules Ġthat Ġgovern Ġthem Ġare Ġcomplex Ġand Ġdifficult Ġto Ġengineer Ġby Ġhand . Ġg 2 p Ġfor Ġlanguages Ġwith Ġambiguous Ġorth ographies Ġis Ġsuch Ġa Ġproblem . ĠMult ilingual Ġg 2 p , Ġin Ġwhich Ġthe Ġvarious Ġlanguages Ġhave Ġsimilar Ġbut Ġdifferent Ġand Ġpossibly Ġcontradictory Ġspelling Ġrules , Ġcan Ġbe Ġseen Ġas Ġan Ġextreme Ġcase Ġof Ġthat . ĠTherefore , Ġa Ġdata - driven Ġsequence - to - sequence Ġmodel Ġis Ġa Ġnatural Ġchoice . </s> Enc oder âĢĵ Dec oder ĠModels </s> In Ġorder Ġto Ġfind Ġthe Ġbest Ġphon eme Ġsequence , Ġwe Ġuse Ġa Ġneural Ġenc oder âĢĵ dec oder Ġmodel Ġwith Ġattention ĠB IB REF 9 Ġ. ĠThe Ġmodel Ġconsists Ġof Ġtwo Ġmain Ġparts : Ġthe Ġenc oder Ġcomp resses Ġeach Ġsource Ġgrap he me Ġsequence ĠIN LINE FORM 0 Ġinto Ġa Ġfixed - length Ġvector . ĠThe Ġdec oder , Ġconditioned Ġon Ġthis Ġfixed - length Ġvector , Ġgenerates Ġthe Ġoutput Ġphon eme Ġsequence ĠIN LINE FORM 1 Ġ. </s> The Ġenc oder Ġand Ġdec oder Ġare Ġboth Ġimplemented Ġas Ġrecurrent Ġneural Ġnetworks , Ġwhich Ġhave Ġthe Ġadvantage Ġof Ġbeing Ġable Ġto Ġprocess Ġsequences Ġof Ġarbitrary Ġlength Ġand Ġuse Ġlong Ġhistories Ġefficiently . ĠThey Ġare Ġtrained Ġjointly Ġto Ġminimize Ġcross - ent ropy Ġon Ġthe Ġtraining Ġdata . ĠWe Ġhad Ġour Ġbest Ġresults Ġwhen Ġusing Ġa Ġbid irection al Ġenc oder , Ġwhich Ġconsists Ġof Ġtwo Ġseparate Ġenc od ers Ġwhich Ġprocess Ġthe Ġinput Ġin Ġforward Ġand Ġreverse Ġdirections . ĠWe Ġused Ġlong Ġshort - term Ġmemory Ġunits ĠB IB REF 10 Ġfor Ġboth Ġthe Ġenc oder Ġand Ġdec oder . ĠFor Ġthe Ġattention Ġmechanism , Ġwe Ġused Ġthe Ġgeneral Ġglobal Ġattention Ġarchitecture Ġdescribed Ġby Ġl u ong 2015 effective . </s> We Ġimplemented Ġall Ġmodels Ġwith ĠOpen N MT ĠB IB REF 11 Ġ. ĠOur Ġhyper param eters , Ġwhich Ġwe Ġdetermined Ġby Ġexperimentation , Ġare Ġlisted Ġin ĠTable ĠT AB REF 8 Ġ. </s> Training ĠMult ilingual ĠModels </s> Present ing Ġpronunciation Ġdata Ġin Ġseveral Ġlanguages Ġto Ġthe Ġnetwork Ġmight Ġcreate Ġproblems Ġbecause Ġdifferent Ġlanguages Ġhave Ġdifferent Ġpronunciation Ġpatterns . ĠFor Ġexample , Ġthe Ġstring Ġ` real ' Ġis Ġpronounced Ġdifferently Ġin ĠEnglish , ĠGerman , ĠSpanish , Ġand ĠPortuguese . ĠWe Ġsolve Ġthis Ġproblem Ġby Ġprep ending Ġeach Ġgrap he me Ġsequence Ġwith Ġan Ġartificial Ġtoken Ġconsisting Ġof Ġthe Ġlanguage 's ĠISO Ġ6 39 - 3 Ġcode Ġenclosed Ġin Ġangle Ġbrackets . ĠThe ĠEnglish Ġword Ġ` real ', Ġfor Ġexample , Ġwould Ġbe Ġpresented Ġto Ġthe Ġsystem Ġas </s> IN LINE FORM 0 Ġeng ĠIN LINE FORM 1 Ġr Ġe Ġa Ġl </s> The Ġartificial Ġtoken Ġis Ġtreated Ġsimply Ġas Ġan Ġelement Ġof Ġthe Ġgrap he me Ġsequence . ĠThis Ġis Ġsimilar Ġto Ġthe Ġapproach Ġtaken Ġby Ġjohn son 2016 google Ġin Ġtheir Ġzero - shot ĠN MT Ġsystem . ĠHowever , Ġtheir Ġsource - side Ġartificial Ġtokens Ġidentify Ġthe Ġtarget Ġlanguage , Ġwhereas Ġours Ġidentify Ġthe Ġsource Ġlanguage . ĠAn Ġalternative Ġapproach , Ġused Ġby Ġost ling 2017 contin uous , Ġwould Ġbe Ġto Ġconc aten ate Ġa Ġlanguage Ġembed ding Ġto Ġthe Ġinput Ġat Ġeach Ġtime Ġstep . ĠThey Ġdo Ġnot Ġevaluate Ġtheir Ġapproach Ġon Ġgrap he me - to - ph on eme Ġconversion . </s> Data </s> In Ġorder Ġto Ġtrain Ġa Ġneural Ġg 2 p Ġsystem , Ġone Ġneeds Ġa Ġlarge Ġquantity Ġof Ġpronunciation Ġdata . ĠA Ġstandard Ġdataset Ġfor Ġg 2 p Ġis Ġthe ĠCarnegie ĠMellon ĠP ron ouncing ĠDictionary ĠB IB REF 12 Ġ. ĠHowever , Ġthat Ġis Ġa Ġmon oling ual ĠEnglish Ġresource , Ġso Ġit Ġis Ġunsu itable Ġfor Ġour Ġmult ilingual Ġtask . ĠInstead , Ġwe Ġuse Ġthe Ġmult ilingual Ġpronunciation Ġcorpus Ġcollected Ġby Ġder i 2016 g rap he me Ġfor Ġall Ġexperiments . ĠThis Ġcorpus Ġconsists Ġof Ġspelling âĢĵ pron unciation Ġpairs Ġextracted Ġfrom ĠWik tion ary . ĠIt Ġis Ġalready Ġpartition ed Ġinto Ġtraining Ġand Ġtest Ġsets . ĠCorpus Ġstatistics Ġare Ġpresented Ġin ĠTable ĠT AB REF 10 Ġ. </s> In Ġaddition Ġto Ġthe Ġraw ĠIPA Ġtranscript ions Ġextracted Ġfrom ĠWik tion ary , Ġthe Ġcorpus Ġprovides Ġan Ġautomatically Ġcleaned Ġversion Ġof Ġtranscript ions . ĠClean ing Ġis Ġa Ġnecessary Ġstep Ġbecause Ġweb - sc raped Ġdata Ġis Ġoften Ġnoisy Ġand Ġmay Ġbe Ġtransc ribed Ġat Ġan Ġinconsistent Ġlevel Ġof Ġdetail . ĠThe Ġdata Ġcleaning Ġused Ġhere Ġattempts Ġto Ġmake Ġthe Ġtranscript ions Ġconsistent Ġwith Ġthe Ġphon emic Ġinvent ories Ġused Ġin ĠPh o ible ĠB IB REF 4 Ġ. ĠWhen Ġa Ġtranscription Ġcontains Ġa Ġphon eme Ġthat Ġis Ġnot Ġin Ġits Ġlanguage 's Ġinventory Ġin ĠPh o ible , Ġthat Ġphon eme Ġis Ġreplaced Ġby Ġthe Ġphon eme Ġwith Ġthe Ġmost Ġsimilar Ġartic ulatory Ġfeatures Ġthat Ġis Ġin Ġthe Ġlanguage 's Ġinventory . ĠSometimes Ġthis Ġcleaning Ġalgorithm Ġworks Ġwell : Ġin Ġthe ĠGerman Ġexamples Ġin ĠTable ĠT AB REF 11 Ġ, Ġthe Ġraw ĠGerman Ġsymbols Ġand Ġare Ġboth Ġconverted Ġto Ġ. ĠThis Ġis Ġuseful Ġbecause Ġthe Ġin ĠAns bach Ġand Ġthe Ġin ĠKan inc hen Ġare Ġinstances Ġof Ġthe Ġsame Ġphon eme , Ġso Ġtheir Ġphon emic Ġrepresentations Ġshould Ġuse Ġthe Ġsame Ġsymbol . ĠHowever , Ġthe Ġcleaning Ġalgorithm Ġcan Ġalso Ġhave Ġnegative Ġeffects Ġon Ġthe Ġdata Ġquality . ĠFor Ġexample , Ġthe Ġphon eme Ġis Ġnot Ġpresent Ġin Ġthe ĠPh o ible Ġinventory Ġfor ĠGerman , Ġbut Ġit Ġis Ġused Ġin Ġseveral ĠGerman Ġtranscript ions Ġin Ġthe Ġcorpus . ĠThe Ġcleaning Ġalgorithm Ġconverts Ġto Ġin Ġall ĠGerman Ġtranscript ions , Ġwhereas Ġwould Ġbe Ġa Ġmore Ġreasonable Ġguess . ĠThe Ġcleaning Ġalgorithm Ġalso Ġremoves Ġmost Ġsu pr ase gment als , Ġeven Ġthough Ġthese Ġare Ġoften Ġan Ġimportant Ġpart Ġof Ġa Ġlanguage 's Ġphon ology . ĠDevelop ing Ġa Ġmore Ġsophisticated Ġprocedure Ġfor Ġcleaning Ġpronunciation Ġdata Ġis Ġa Ġdirection Ġfor Ġfuture Ġwork , Ġbut Ġin Ġthis Ġpaper Ġwe Ġuse Ġthe Ġcorpus 's Ġprovided Ġcleaned Ġtranscript ions Ġin Ġorder Ġto Ġease Ġcomparison Ġto Ġprevious Ġresults . </s> Exper iments </s> We Ġpresent Ġexperiments Ġwith Ġtwo Ġversions Ġof Ġour Ġsequence - to - sequence Ġmodel . ĠLang ID Ġprep ends Ġeach Ġtraining , Ġvalidation , Ġand Ġtest Ġsample Ġwith Ġan Ġartificial Ġtoken Ġidentifying Ġthe Ġlanguage Ġof Ġthe Ġsample . ĠNo L ang ID Ġo mits Ġthis Ġtoken . ĠLang ID Ġand ĠNo L ang ID Ġhave Ġidentical Ġstructure Ġotherwise . ĠTo Ġtranslate Ġthe Ġtest Ġcorpus , Ġwe Ġused Ġa Ġbeam Ġwidth Ġof Ġ100 . ĠAlthough Ġthis Ġis Ġan Ġunusually Ġwide Ġbeam Ġand Ġhad Ġnegligible Ġperformance Ġeffects , Ġit Ġwas Ġnecessary Ġto Ġcompute Ġour Ġerror Ġmetrics . </s> E val uation </s> We Ġuse Ġthe Ġfollowing Ġthree Ġevaluation Ġmetrics : </s> Ph on eme ĠError ĠRate Ġ( PER ) Ġis Ġthe ĠLe ven sh tein Ġdistance Ġbetween Ġthe Ġpredicted Ġphon eme Ġsequences Ġand Ġthe Ġgold Ġstandard Ġphon eme Ġsequences , Ġdivided Ġby Ġthe Ġlength Ġof Ġthe Ġgold Ġstandard Ġphon eme Ġsequences . </s> Word ĠError ĠRate Ġ( WER ) Ġis Ġthe Ġpercentage Ġof Ġwords Ġin Ġwhich Ġthe Ġpredicted Ġphon eme Ġsequence Ġdoes Ġnot Ġexactly Ġmatch Ġthe Ġgold Ġstandard Ġphon eme Ġsequence . </s> Word ĠError ĠRate Ġ100 Ġ( WER Ġ100 ) Ġis Ġthe Ġpercentage Ġof Ġwords Ġin Ġthe Ġtest Ġset Ġfor Ġwhich Ġthe Ġcorrect Ġguess Ġis Ġnot Ġin Ġthe Ġfirst Ġ100 Ġguesses Ġof Ġthe Ġsystem . </s> In Ġsystem Ġevaluations , ĠW ER , ĠW ER Ġ100 , Ġand ĠPER Ġnumbers Ġpresented Ġfor Ġmultiple Ġlanguages Ġare Ġaveraged , Ġweight ing Ġeach Ġlanguage Ġequally ĠB IB REF 13 Ġ. </s> It Ġwould Ġbe Ġinteresting Ġto Ġcompute Ġerror Ġmetrics Ġthat Ġincorporate Ġphon eme Ġsimilarity , Ġsuch Ġas Ġthose Ġproposed Ġby Ġh ixon 2011 ph on emic . ĠPER Ġweights Ġall Ġphon eme Ġerrors Ġthe Ġsame , Ġeven Ġthough Ġsome Ġerrors Ġare Ġmore Ġharmful Ġthan Ġothers : Ġand Ġare Ġusually Ġcontrast ive , Ġwhereas Ġand Ġalmost Ġnever Ġare . ĠSuch Ġstatistics Ġwould Ġbe Ġespecially Ġinteresting Ġfor Ġevaluating Ġa Ġmult ilingual Ġsystem , Ġbecause Ġdifferent Ġlanguages Ġoften Ġmap Ġthe Ġsame Ġgrap he me Ġto Ġphon em es Ġthat Ġare Ġonly Ġsubtly Ġdifferent Ġfrom Ġeach Ġother . ĠHowever , Ġthese Ġstatistics Ġhave Ġnot Ġbeen Ġwidely Ġreported Ġfor Ġother Ġg 2 p Ġsystems , Ġso Ġwe Ġomit Ġthem Ġhere . </s> Bas eline </s> Results Ġon ĠLang ID Ġand ĠNo L ang ID Ġare Ġcompared Ġto Ġthe Ġsystem Ġpresented Ġby Ġder i 2016 g rap he me , Ġwhich Ġis Ġidentified Ġin Ġour Ġresults Ġas Ġw F ST . ĠTheir Ġresults Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġparts : </s> High Ġresource Ġresults , Ġcomputed Ġwith Ġw F ST s Ġtrained Ġon Ġa Ġcombination Ġof ĠWik tion ary Ġpronunciation Ġdata Ġand Ġg 2 p Ġrules Ġextracted Ġfrom ĠWikipedia ĠIPA ĠHelp Ġpages . ĠThey Ġreport Ġhigh Ġresource Ġresults Ġfor Ġ85 Ġlanguages . </s> Adapt ed Ġresults , Ġwhere Ġthey Ġapply Ġvarious Ġmapping Ġstrategies Ġin Ġorder Ġto Ġadapt Ġhigh Ġresource Ġmodels Ġto Ġother Ġlanguages . ĠThe Ġfinal Ġadapted Ġresults Ġthey Ġreported Ġinclude Ġmost Ġof Ġthe Ġ85 Ġlanguages Ġwith Ġhigh Ġresource Ġresults , Ġas Ġwell Ġas Ġthe Ġvarious Ġlanguages Ġthey Ġwere Ġable Ġto Ġadapt Ġthem Ġfor , Ġfor Ġa Ġtotal Ġof Ġ229 Ġlanguages . ĠThis Ġtest Ġset Ġo mits Ġ23 Ġof Ġthe Ġhigh Ġresource Ġlanguages Ġthat Ġare Ġwritten Ġin Ġunique Ġscripts Ġor Ġfor Ġwhich Ġlanguage Ġdistance Ġmetrics Ġcould Ġnot Ġbe Ġcomputed . </s> Training </s> We Ġtrain Ġthe ĠLang ID Ġand ĠNo L ang ID Ġversions Ġof Ġour Ġmodel Ġeach Ġon Ġthree Ġsubs ets Ġof Ġthe ĠWik tion ary Ġdata : </s> L ang ID - High Ġand ĠNo L ang ID - High : ĠTr ained Ġon Ġdata Ġfrom Ġthe Ġ85 Ġlanguages Ġfor Ġwhich ĠB IB REF 13 Ġused Ġnon - adapt ed Ġw F ST Ġmodels . </s> L ang ID - Adapt ed Ġand ĠNo L ang ID - Adapt ed : ĠTr ained Ġon Ġdata Ġfrom Ġany Ġof Ġthe Ġ229 Ġlanguages Ġfor Ġwhich Ġthey Ġbuilt Ġadapted Ġmodels . ĠBecause Ġmany Ġof Ġthese Ġlanguages Ġhad Ġno Ġtraining Ġdata Ġat Ġall , Ġthe Ġmodel Ġis Ġactually Ġonly Ġtrained Ġon Ġdata Ġin Ġ157 Ġlanguages . ĠAs Ġis Ġnoted Ġabove , Ġthe ĠAdapt ed Ġset Ġo mits Ġ23 Ġlanguages Ġwhich Ġare Ġin Ġthe ĠHigh Ġtest Ġset . </s> L ang ID - All Ġand ĠNo L ang ID - All : ĠTr ained Ġon Ġdata Ġin Ġall Ġ311 Ġlanguages Ġin Ġthe ĠWik tion ary Ġtraining Ġcorpus . </s> In Ġorder Ġto Ġease Ġcomparison Ġto ĠDer i Ġand ĠKnight 's Ġsystem , Ġwe Ġlimited Ġour Ġuse Ġof Ġthe Ġtraining Ġcorpus Ġto Ġ10 , 000 Ġwords Ġper Ġlanguage . ĠWe Ġset Ġaside Ġ10 Ġpercent Ġof Ġthe Ġdata Ġin Ġeach Ġlanguage Ġfor Ġvalidation , Ġso Ġthe Ġmaximum Ġnumber Ġof Ġtraining Ġwords Ġfor Ġany Ġlanguage Ġis Ġ9000 Ġfor Ġour Ġsystems . </s> Adapt ed ĠResults </s> On Ġthe Ġ229 Ġlanguages Ġfor Ġwhich Ġder i 2016 g rap he me Ġpresented Ġtheir Ġfinal Ġresults , Ġthe ĠLang ID Ġversion Ġof Ġour Ġsystem Ġoutper forms Ġthe Ġbaseline Ġby Ġa Ġwide Ġmargin . ĠThe Ġbest Ġperformance Ġcame Ġwith Ġthe Ġversion Ġof Ġour Ġmodel Ġthat Ġwas Ġtrained Ġon Ġdata Ġin Ġall Ġavailable Ġlanguages , Ġnot Ġjust Ġthe Ġlanguages Ġit Ġwas Ġtested Ġon . ĠUsing Ġa Ġlanguage ĠID Ġtoken Ġimproves Ġresults Ġconsiderably , Ġbut Ġeven ĠNo L ang ID Ġbeats Ġthe Ġbaseline Ġin ĠW ER Ġand ĠW ER Ġ100 . ĠFull Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 24 Ġ. </s> High ĠResource ĠResults </s> Having Ġshown Ġthat Ġour Ġmodel Ġexceeds Ġthe Ġperformance Ġof Ġthe Ġw F ST - adapt ation Ġapproach , Ġwe Ġnext Ġcompare Ġit Ġto Ġthe Ġbaseline Ġmodels Ġfor Ġjust Ġhigh Ġresource Ġlanguages . ĠThe Ġw F ST Ġmodels Ġhere Ġare Ġpurely Ġmon oling ual ĠâĢĵ Ġthey Ġdo Ġnot Ġuse Ġdata Ġadaptation Ġbecause Ġthere Ġis Ġsufficient Ġtraining Ġdata Ġfor Ġeach Ġof Ġthem . ĠFull Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 26 Ġ. ĠWe Ġomit Ġmodels Ġtrained Ġon Ġthe ĠAdapt ed Ġlanguages Ġbecause Ġthey Ġwere Ġnot Ġtrained Ġon Ġhigh Ġresource Ġlanguages Ġwith Ġunique Ġwriting Ġsystems , Ġsuch Ġas ĠGeorgian Ġand ĠGreek , Ġand Ġconsequently Ġperformed Ġvery Ġpoorly Ġon Ġthem . </s> In Ġcontrast Ġto Ġthe Ġlarger - scale ĠAdapt ed Ġresults , Ġin Ġthe ĠHigh ĠResource Ġexperiments Ġnone Ġof Ġthe Ġsequence - to - sequence Ġapproaches Ġequal Ġthe Ġperformance Ġof Ġthe Ġw F ST Ġmodel Ġin ĠW ER Ġand ĠPER , Ġalthough ĠLang ID - High Ġdoes Ġcome Ġclose . ĠThe ĠLang ID Ġmodels Ġdo Ġbeat Ġw F ST Ġin ĠW ER Ġ100 . ĠA Ġpossible Ġexplanation Ġis Ġthat Ġa Ġmon oling ual Ġw F ST Ġmodel Ġwill Ġnever Ġgenerate Ġphon em es Ġthat Ġare Ġnot Ġpart Ġof Ġthe Ġlanguage 's Ġinventory . ĠA Ġmult ilingual Ġmodel , Ġon Ġthe Ġother Ġhand , Ġcould Ġpotentially Ġgenerate Ġphon em es Ġfrom Ġthe Ġinvent ories Ġof Ġany Ġlanguage Ġit Ġhas Ġbeen Ġtrained Ġon . </s> Even Ġif ĠLang ID - High Ġdoes Ġnot Ġpresent Ġa Ġmore Ġaccurate Ġresult , Ġit Ġdoes Ġpresent Ġa Ġmore Ġcompact Ġone : ĠLang ID - High Ġis Ġ15 . 4 ĠMB , Ġwhile Ġthe Ġcombined Ġw F ST Ġhigh Ġresource Ġmodels Ġare Ġ197 . 5 ĠMB . </s> Results Ġon ĠUn seen ĠLanguages </s> Finally , Ġwe Ġreport Ġour Ġmodels ' Ġresults Ġon Ġunseen Ġlanguages Ġin ĠTable ĠT AB REF 28 Ġ. ĠThe Ġunseen Ġlanguages Ġare Ġany Ġthat Ġare Ġpresent Ġin Ġthe Ġtest Ġcorpus Ġbut Ġabsent Ġfrom Ġthe Ġtraining Ġdata . ĠDer i Ġand ĠKnight Ġdid Ġnot Ġreport Ġresults Ġspecifically Ġon Ġthese Ġlanguages . ĠAlthough Ġthe ĠNo L ang ID Ġmodels Ġsometimes Ġdo Ġbetter Ġon ĠW ER Ġ100 , Ġeven Ġhere Ġthe ĠLang ID Ġmodels Ġhave Ġa Ġslight Ġadvantage Ġin ĠW ER Ġand ĠPER . ĠThis Ġis Ġsomewhat Ġsurprising Ġbecause Ġthe ĠLang ID Ġmodels Ġhave Ġnot Ġlearned Ġembed d ings Ġfor Ġthe Ġlanguage ĠID Ġtokens Ġof Ġunseen Ġlanguages . ĠPerhaps Ġnegative Ġassociations Ġare Ġalso Ġbeing Ġlearned , Ġdriving Ġthe Ġmodel Ġtowards Ġpredicting Ġmore Ġcommon Ġpron unci ations Ġfor Ġunseen Ġlanguages . </s> Language ĠID ĠTokens </s> Adding Ġa Ġlanguage ĠID Ġtoken Ġalways Ġimproves Ġresults Ġin Ġcases Ġwhere Ġan Ġembed ding Ġhas Ġbeen Ġlearned Ġfor Ġthat Ġtoken . ĠThe Ġpower Ġof Ġthese Ġembed d ings Ġis Ġdemonstrated Ġby Ġwhat Ġhappens Ġwhen Ġone Ġfeeds Ġthe Ġsame Ġinput Ġword Ġto Ġthe Ġmodel Ġwith Ġdifferent Ġlanguage Ġtokens , Ġas Ġis Ġseen Ġin ĠTable ĠT AB REF 30 Ġ. ĠIm press ively , Ġthis Ġeven Ġworks Ġwhen Ġthe Ġsource Ġsequence Ġis Ġin Ġthe Ġwrong Ġscript Ġfor Ġthe Ġlanguage , Ġas Ġis Ġseen Ġin Ġthe Ġentry Ġfor ĠArabic . </s> Language ĠEmb edd ings </s> Because Ġthese Ġlanguage ĠID Ġtokens Ġare Ġso Ġuseful , Ġit Ġwould Ġbe Ġgood Ġif Ġthey Ġcould Ġbe Ġeffectively Ġestimated Ġfor Ġunseen Ġlanguages . Ġost ling 2017 contin uous Ġfound Ġthat Ġthe Ġlanguage Ġvectors Ġtheir Ġmodels Ġlearned Ġcorrelated Ġwell Ġto Ġgenetic Ġrelationships , Ġso Ġit Ġwould Ġbe Ġinteresting Ġto Ġsee Ġif Ġthe Ġembed d ings Ġour Ġsource Ġenc oder Ġlearned Ġfor Ġthe Ġlanguage ĠID Ġtokens Ġshowed Ġanything Ġsimilar . ĠIn Ġa Ġfew Ġcases Ġthey Ġdo Ġ( the Ġlanguages Ġclosest Ġto ĠGerman Ġin Ġthe Ġvector Ġspace Ġare ĠLuxembourg ish , ĠBav arian , Ġand ĠY idd ish , Ġall Ġclose Ġrelatives ). ĠHowever , Ġfor Ġthe Ġmost Ġpart Ġthe Ġstructure Ġof Ġthese Ġvectors Ġis Ġnot Ġinterpret able . ĠTherefore , Ġit Ġwould Ġbe Ġdifficult Ġto Ġestimate Ġthe Ġembed ding Ġfor Ġan Ġunseen Ġlanguage , Ġor Ġto ĠâĢ ľ bor row âĢ Ŀ Ġthe Ġlanguage ĠID Ġtoken Ġof Ġa Ġsimilar Ġlanguage . ĠA Ġmore Ġpromising Ġway Ġforward Ġis Ġto Ġfind Ġa Ġmodel Ġthat Ġuses Ġan Ġexternally Ġconstructed Ġtyp ological Ġrepresentation Ġof Ġthe Ġlanguage . </s> Ph on eme ĠEmb edd ings </s> In Ġcontrast Ġto Ġthe Ġlanguage Ġembed d ings , Ġthe Ġphon eme Ġembed d ings Ġappear Ġto Ġshow Ġmany Ġregular ities Ġ( see ĠTable ĠT AB REF 33 Ġ). ĠThis Ġis Ġa Ġsign Ġthat Ġour Ġmult ilingual Ġmodel Ġlearns Ġsimilar Ġembed d ings Ġfor Ġphon em es Ġthat Ġare Ġwritten Ġwith Ġthe Ġsame Ġgrap he me Ġin Ġdifferent Ġlanguages . ĠThese Ġphon em es Ġtend Ġto Ġbe Ġphon etically Ġsimilar Ġto Ġeach Ġother . </s> Perhaps Ġthe Ġstructure Ġof Ġthe Ġphon eme Ġembed ding Ġspace Ġis Ġwhat Ġleads Ġto Ġour Ġmodels ' Ġvery Ġgood Ġperformance Ġon ĠW ER Ġ100 . ĠEven Ġwhen Ġthe Ġmodel 's Ġfirst Ġpredicted Ġpronunciation Ġis Ġnot Ġcorrect , Ġit Ġtends Ġto Ġassign Ġmore Ġprobability Ġmass Ġto Ġguesses Ġthat Ġare Ġmore Ġsimilar Ġto Ġthe Ġcorrect Ġone . ĠApp lying Ġsome Ġsort Ġof Ġfiltering Ġor Ġre ranking Ġof Ġthe Ġsystem Ġoutput Ġmight Ġtherefore Ġlead Ġto Ġbetter Ġperformance . </s> Future ĠWork </s> Because Ġthe Ġlanguage ĠID Ġtoken Ġis Ġso Ġbeneficial Ġto Ġperformance , Ġit Ġwould Ġbe Ġvery Ġinteresting Ġto Ġfind Ġways Ġto Ġextend Ġa Ġsimilar Ġbenefit Ġto Ġunseen Ġlanguages . ĠOne Ġpossible Ġway Ġto Ġdo Ġso Ġis Ġwith Ġtokens Ġthat Ġidentify Ġsomething Ġother Ġthan Ġthe Ġlanguage , Ġsuch Ġas Ġtyp ological Ġfeatures Ġabout Ġthe Ġlanguage 's Ġphon emic Ġinventory . ĠThis Ġcould Ġenable Ġbetter Ġsharing Ġof Ġresources Ġamong Ġlanguages . ĠSuch Ġtyp ological Ġknowledge Ġis Ġreadily Ġavailable Ġin Ġdatabases Ġlike ĠPh o ible Ġand ĠW ALS Ġfor Ġa Ġwide Ġvariety Ġof Ġlanguages . ĠIt Ġwould Ġbe Ġinteresting Ġto Ġexplore Ġif Ġany Ġof Ġthese Ġfeatures Ġis Ġa Ġgood Ġpredictor Ġof Ġa Ġlanguage 's Ġorth ographic Ġrules . </s> It Ġwould Ġalso Ġbe Ġinteresting Ġto Ġapply Ġthe Ġartificial Ġtoken Ġapproach Ġto Ġother Ġproblems Ġbesides Ġmult ilingual Ġg 2 p . ĠOne Ġclosely Ġrelated Ġapplication Ġis Ġmon oling ual ĠEnglish Ġg 2 p . ĠSome Ġof Ġthe Ġambiguity Ġof ĠEnglish Ġspelling Ġis Ġdue Ġto Ġthe Ġwide Ġvariety Ġof Ġloan words Ġin Ġthe Ġlanguage , Ġmany Ġof Ġwhich Ġhave Ġun ass imil ated Ġspell ings . ĠKnowing Ġthe Ġorigins Ġof Ġthese Ġloan words Ġcould Ġprovide Ġa Ġuseful Ġhint Ġfor Ġfiguring Ġout Ġtheir Ġpron unci ations . ĠThe Ġe tymology Ġof Ġa Ġword Ġcould Ġbe Ġtagged Ġin Ġan Ġanalogous Ġway Ġto Ġhow Ġlanguage ĠID Ġis Ġtagged Ġin Ġmult ilingual Ġg 2 p .