Once split into 8 subsets (A-H), the test set used are blocks D+H and blocks F+H
ABSTRACTIVE
 this paper makes use of the official training and test sets, covering in total 630 speakers with 8 utterances each
EXTRACTIVE
In order to achieve these configuration the TIMIT data was split. Fig. FIGREF12 illustrates the split of the data into 8 subsets (A–H). The TIMIT dataset contains speech from 462 speakers in training and 168 speakers in the test set, with 8 utterances for each speaker. The TIMIT training and test set are split into 8 blocks, where each block contains 2 utterances per speaker, randomly chosen. Thus each block A,B,C,D contains data from 462 speakers with 924 utterances taken from the training sets, and each block E,F,G,H contains speech from 168 test set speakers with 336 utterances. For Task a training of embeddings and the classifier is identical, namely consisting of data from blocks (A+B+C+E+F+G). The test data is the remainder, namely blocks (D+H). For Task b the training of embeddings and classifiers uses (A+B+E+F) and (C+G) respectively, while again using (D+H) for test. Task c keeps both separate: embeddings are trained on (A+B+C+D), classifiers on (E+G) and tests are conducted on (F+H). Note that H is part of all tasks, and that Task c is considerably easier as the number of speakers to separate is only 168, although training conditions are more difficult.
Taking the VAE experiments as baseline, the TIMIT data is used for this workBIBREF25. TIMIT contains studio recordings from a large number of speakers with detailed phoneme segment information. Work in this paper makes use of the official training and test sets, covering in total 630 speakers with 8 utterances each. There is no speaker overlap between training and test set, which comprise of 462 and 168 speakers, respectively. All work presented here use of 80 dimensional Mel-scale filter bank coefficients.
1910.07601
Introduction </s> In Ġrecent Ġyears , Ġword Ġembed d ings Ġhave Ġbeen Ġsuccessfully Ġused Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġthe Ġmost Ġcommonly Ġknown Ġmodels Ġare ĠWord 2 V ec ĠB IB REF 0 Ġand ĠGl ove BI B REF 1 . ĠThe Ġreasons Ġfor Ġsuch Ġsuccess Ġare Ġmanifold . ĠOne Ġkey Ġattribute Ġof Ġembed ding Ġmethods Ġis Ġthat Ġword Ġembed ding Ġmodels Ġtake Ġinto Ġaccount Ġcontext Ġinformation Ġof Ġwords , Ġthereby Ġallowing Ġa Ġmore Ġcompact Ġand Ġmanageable Ġrepresentation Ġfor Ġwords BI B REF 2 , ĠB IB REF 3 . ĠThe Ġembed d ings Ġare Ġwidely Ġapplied Ġin Ġmany Ġdownstream ĠN LP Ġtasks Ġsuch Ġas Ġneural Ġmachine Ġtranslation , Ġdialogue Ġsystem Ġor Ġtext Ġsummar isation ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , Ġas Ġwell Ġas Ġin Ġlanguage Ġmodelling Ġfor Ġspeech Ġrecognition BI B REF 7 . </s> Emb edd ings Ġof Ġacoustic Ġ( and Ġspeech ) Ġsignals Ġare Ġof Ġmore Ġrecent Ġinterest . ĠThe Ġobjective Ġis Ġto Ġrepresent Ġaudio Ġsequence Ġinformation Ġin Ġcompact Ġform , Ġreplacing Ġthe Ġraw Ġaudio Ġdata Ġwith Ġone Ġthat Ġcontains Ġlatent Ġfactors ĠB IB REF 8 , ĠB IB REF 9 . ĠThe Ġprojection Ġinto Ġsuch Ġ( lat ent ) Ġspaces Ġshould Ġtake Ġinto Ġaccount Ġdifferent Ġattributes , Ġsuch Ġas Ġphon em es , Ġspeaker Ġproperties , Ġspeaking Ġstyles , Ġthe Ġacoustic Ġbackground Ġor Ġthe Ġrecording Ġenvironment . ĠAc oustic Ġembed d ings Ġhave Ġbeen Ġexplored Ġfor Ġa Ġvariety Ġof Ġspeech Ġtasks Ġsuch Ġas Ġspeech Ġrecognition BI B REF 10 , Ġspeaker Ġverification BI B REF 11 Ġor Ġvoice Ġconversion BI B REF 12 . ĠHowever , Ġlearning Ġacoustic Ġembed d ings Ġis Ġchallenging : Ġattributes Ġmentioned Ġabove , Ġe . g . Ġspeaker Ġproperties Ġand Ġphon em es , Ġoperate Ġat Ġdifferent Ġlevels Ġof Ġabstraction Ġand Ġare Ġoften Ġstrongly Ġinter dependent , Ġand Ġtherefore Ġare Ġdifficult Ġto Ġextract Ġand Ġrepresent Ġin Ġa Ġmeaningful Ġform BI B REF 8 . </s> For Ġspeech Ġprocessing ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 Ġalso Ġuse Ġcontext Ġinformation Ġto Ġderive Ġacoustic Ġembed d ings . ĠHowever , ĠB IB REF 13 , ĠB IB REF 14 Ġfocus Ġon Ġlearning Ġword Ġsemantic Ġrepresentations Ġfrom Ġraw Ġaudio Ġinstead Ġof Ġsignal Ġproperties Ġsuch Ġas Ġphon em es Ġand Ġspeaker Ġproperties . ĠB IB REF 15 Ġfocus Ġon Ġlearning Ġspeaker Ġrepresentations Ġby Ġmodelling Ġof Ġcontext Ġinformation Ġwith Ġa ĠSi ames e Ġnetworks Ġthat Ġdiscriminate Ġwhether Ġa Ġspeech Ġsegment Ġis Ġthe Ġneighbourhood Ġof Ġa Ġtarget Ġsegment Ġor Ġnot . </s> In Ġthis Ġpaper , Ġtwo Ġun super vised Ġapproaches Ġto Ġgenerate Ġacoustic Ġembed d ings Ġusing Ġcontext Ġmodelling Ġare Ġproposed . ĠBoth Ġmethods Ġmake Ġuse Ġof Ġthe Ġvari ational Ġauto - enc oder Ġframework Ġas Ġproposed Ġin BI B REF 16 Ġand Ġboth Ġapproaches Ġaim Ġto Ġfind Ġjoint Ġlatent Ġvariables Ġbetween Ġthe Ġtarget Ġacoustic Ġsegments Ġand Ġits Ġsurrounding Ġframes . ĠIn Ġthe Ġfirst Ġinstance Ġa Ġrepresentation Ġis Ġderived Ġfrom Ġsurrounding Ġaudio Ġframes Ġthat Ġallows Ġto Ġpredict Ġcurrent Ġframe , Ġthereby Ġgenerating Ġtarget Ġaudio Ġfrom Ġcommon Ġfactors . ĠThe Ġenc oder Ġelement Ġof Ġthe Ġassociated Ġauto - enc oder Ġis Ġfurther Ġreferred Ġto Ġas ĠContext ual ĠJoint ĠFactor ĠSynt hesis Ġ( C J FS ) Ġenc oder . ĠIn Ġthe Ġsecond Ġinstance Ġan Ġaudio Ġframe Ġis Ġused Ġto Ġpredict Ġsurrounding Ġaudio , Ġwhich Ġis Ġfurther Ġreferred Ġto Ġas ĠContext ual ĠJoint ĠFactor ĠAnalysis Ġ( C J FA ) Ġencoding . ĠAs Ġshown Ġin Ġprevious Ġwork Ġvari ational Ġauto - enc od ers Ġcan Ġbe Ġused Ġto Ġderive Ġlatent Ġvariables Ġsuch Ġas Ġspeaker Ġinformation Ġand Ġphon em es BI B REF 8 Ġmore Ġrobust ly . ĠIn Ġthis Ġwork Ġit Ġis Ġshown Ġthat Ġincluding Ġtemporal Ġinformation Ġcan Ġfurther Ġimprove Ġperformance Ġand Ġrobust ness , Ġfor Ġboth Ġphon eme Ġclassification Ġand Ġspeaker Ġidentification Ġtasks . ĠFurthermore Ġthe Ġuse Ġof Ġadditional Ġunl ab elled Ġout - of - domain Ġdata Ġcan Ġimproved Ġmodelling Ġfor Ġthe Ġproposed Ġapproaches . ĠAs Ġoutlined Ġabove , Ġprior Ġwork Ġhas Ġmade Ġuse Ġof Ġsurrounding Ġaudio Ġin Ġdifferent Ġforms . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge Ġthis Ġwork Ġis Ġthe Ġfirst Ġto Ġshow Ġthat Ġpredicting Ġsurrounding Ġaudio Ġallows Ġfor Ġefficient Ġextraction Ġof Ġlatent Ġfactors Ġin Ġspeech Ġsignals . </s> The Ġrest Ġof Ġpaper Ġis Ġorganised Ġas Ġfollows : ĠIn ĠÂ§ SEC REF 2 Ġrelated Ġwork Ġis Ġdescribed , ĠMethods Ġfor Ġder iving Ġacoustic Ġembed d ings , Ġand Ġcontext Ġmodelling Ġmethods Ġin ĠN LP , Ġcomputer Ġvision Ġand Ġspeech Ġare Ġdiscussed . ĠThis Ġis Ġfollowed Ġby Ġthe Ġdescription Ġof Ġthe Ġtwo Ġapproaches Ġfor Ġmodelling Ġcontext Ġas Ġused Ġin Ġthis Ġwork , Ġin ĠÂ§ SEC REF 3 . ĠThe Ġexperimental Ġframework Ġis Ġdescribed Ġin ĠÂ§ SEC REF 4 , Ġincluding Ġthe Ġdata Ġorganisation , Ġbaseline Ġdesign Ġand Ġtask Ġdefinitions ; Ġin ĠÂ§ SEC REF 5 Ġand ĠÂ§ SEC REF 6 Ġexperiments Ġresults Ġare Ġshown Ġand Ġdiscussed . ĠThis Ġis Ġfollowed Ġby Ġthe Ġconclusions Ġand Ġfuture Ġwork Ġin ĠÂ§ SEC REF 7 . </s> Related ĠWorks </s> Related ĠWorks Ġ: :: ĠAc oustic ĠEmb edd ings </s> Most Ġinterest Ġin Ġacoustic Ġembed d ings Ġcan Ġbe Ġobserved Ġon Ġacoustic Ġword Ġembed d ings , Ġi . e . Ġprojections Ġthat Ġmap Ġword Ġac oust ics Ġinto Ġa Ġfixed Ġsize Ġvector Ġspace . ĠObjective Ġfunctions Ġare Ġchosen Ġto Ġproject Ġdifferent Ġword Ġreal isations Ġto Ġclose Ġproximity Ġin Ġthe Ġembed ding Ġspace . ĠDifferent Ġapproaches Ġwere Ġused Ġin Ġthe Ġliterature Ġ- Ġfor Ġboth Ġsupervised Ġand Ġun super vised Ġlearning . ĠFor Ġthe Ġsupervised Ġcase , ĠB IB REF 9 Ġintroduced Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġbased Ġacoustic Ġword Ġembed ding Ġsystem Ġfor Ġspeech Ġrecognition , Ġwhere Ġwords Ġthat Ġsound Ġalike Ġare Ġnearby Ġin ĠEucl ide an Ġdistance . ĠIn Ġtheir Ġwork , Ġa ĠCNN Ġis Ġused Ġto Ġpredict Ġa Ġword Ġfrom Ġthe Ġcorresponding Ġacoustic Ġsignal , Ġthe Ġoutput Ġof Ġthe Ġbottleneck Ġlayer Ġbefore Ġthe Ġfinal Ġsoft max Ġlayer Ġis Ġtaken Ġto Ġbe Ġthe Ġembed ding Ġfor Ġthe Ġcorresponding Ġword . ĠFurther Ġwork Ġused Ġdifferent Ġnetwork Ġarchitectures Ġto Ġobtain Ġacoustic Ġword Ġembed d ings : ĠB IB REF 10 Ġintroduces Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġbased Ġapproach Ġinstead . </s> For Ġthe Ġcase Ġthat Ġthe Ġword Ġboundary Ġinformation Ġis Ġavailable Ġbut Ġthe Ġword Ġlabel Ġitself Ġis Ġunknown , ĠB IB REF 12 Ġproposed Ġword Ġsimilarity ĠSi ames e ĠCNN s . ĠThese Ġare Ġused Ġto Ġminim ise Ġa Ġdistance Ġfunction Ġbetween Ġrepresentations Ġof Ġtwo Ġinstances Ġof Ġthe Ġsame Ġword Ġtype Ġwhilst Ġat Ġthe Ġsame Ġtime Ġmaxim ising Ġthe Ġdistance Ġbetween Ġtwo Ġinstances Ġof Ġdifferent Ġwords . </s> Un super vised Ġapproaches Ġalso Ġexist . ĠB IB REF 17 Ġproposed Ġa Ġconv olution al Ġvari ational Ġauto - enc oder Ġbased Ġapproach Ġto Ġderive Ġacoustic Ġembed ding , Ġin Ġun super vised Ġfashion . ĠThe Ġauthors Ġchose Ġphon eme Ġand Ġspeaker Ġclassification Ġtasks Ġon ĠTIM IT Ġdata Ġto Ġassess Ġthe Ġquality Ġof Ġtheir Ġembed d ings Ġ- Ġan Ġapproach Ġreplicated Ġin Ġthe Ġwork Ġpresented Ġin Ġthis Ġpaper . ĠB IB REF 8 , ĠB IB REF 18 Ġproposed Ġan Ġapproach Ġcalled Ġfactor ised Ġhierarchical Ġvari ational Ġauto - enc oder . ĠThe Ġwork Ġintroduces Ġthe Ġconcepts Ġof Ġglobal Ġand Ġlocal Ġlatent Ġfactors , Ġi . e . Ġlatent Ġvariables Ġthat Ġare Ġshared Ġon Ġthe Ġcomplete Ġutter ance , Ġor Ġlatent Ġvariables Ġthat Ġchange Ġwithin Ġthe Ġsequence , Ġrespectively . ĠResults Ġare Ġagain Ġobtained Ġusing Ġthe Ġsame Ġdata Ġand Ġtasks Ġas Ġabove . </s> Related ĠWorks Ġ: :: ĠContext ĠMod elling </s> Context Ġinformation Ġplays Ġa Ġfundamental Ġrole Ġin Ġspeech Ġprocessing . ĠPh on em es Ġcould Ġbe Ġinfluenced Ġby Ġsurrounding Ġframes Ġthrough Ġco art ic ulation BI B REF 19 Ġ- Ġan Ġeffect Ġcaused Ġby Ġspeed Ġlimitations Ġand Ġtransitions Ġin Ġthe Ġmovement Ġof Ġartic ulators . ĠNormally Ġdirectly Ġneighbouring Ġphon em es Ġhave Ġimportant Ġimpact Ġon Ġthe Ġsound Ġreal isation . ĠIn versely , Ġthe Ġsurrounding Ġphon em es Ġalso Ġprovide Ġstrong Ġconstraints Ġon Ġthe Ġphon eme Ġthat Ġcan Ġbe Ġchosen Ġat Ġany Ġgiven Ġpoint , Ġsubject Ġto Ġto Ġlex ical Ġand Ġlanguage Ġconstraints . ĠThis Ġeffect Ġis Ġfor Ġexample Ġexploited Ġin Ġphon eme Ġrecognition , Ġby Ġuse Ġof Ġphon eme Ġ$ n $ - gram Ġmodels BI B REF 20 . ĠEqu ival ently Ġinter Ġword Ġdependency Ġ- Ġderived Ġfrom Ġlinguistic Ġconstraints Ġ- Ġcan Ġbe Ġexploited , Ġas Ġis Ġthe Ġcase Ġin Ġcomputing Ġword Ġembed d ings Ġwith Ġthe Ġaforementioned Ġword 2 vec BI B REF 0 Ġmethod . ĠThe Ġsituation Ġdiffers Ġfor Ġthe Ġglobal Ġlatent Ġvariables , Ġsuch Ġas Ġspeaker Ġproperties Ġor Ġacoustic Ġenvironment Ġinformation . ĠSpeaker Ġproperties Ġremains Ġconstant Ġ- Ġand Ġenvironments Ġcan Ġalso Ġbe Ġassumed Ġstationary Ġover Ġlonger Ġperiods Ġof Ġtime . ĠHence Ġthese Ġvariables Ġare Ġcommon Ġbetween Ġamong Ġneighbouring Ġframes Ġand Ġwindows . ĠMod elling Ġcontext Ġinformation Ġis Ġhelpful Ġfor Ġidentifying Ġsuch Ġinformation ĠB IB REF 21 . </s> There Ġis Ġsignificant Ġprior Ġwork Ġthat Ġtakes Ġsurrounding Ġinformation Ġinto Ġaccount Ġto Ġlearn Ġvector Ġrepresentations . ĠFor Ġtext Ġprocessing Ġthe ĠWord 2 V ec BI B REF 0 Ġmodel Ġdirectly Ġpredicts Ġthe Ġneighbouring Ġwords Ġfrom Ġtarget Ġwords Ġor Ġin versely . ĠThis Ġhelps Ġto Ġcapture Ġthe Ġmeanings Ġof Ġwords BI B REF 2 . ĠIn Ġcomputer Ġvision , ĠB IB REF 22 Ġintroduced Ġan Ġvisual Ġfeature Ġlearning Ġapproach Ġcalled Ġcontext Ġenc oder Ġ, which Ġis Ġbased Ġon Ġcontext Ġbased Ġpixel Ġprediction . ĠTheir Ġmodel Ġis Ġtrained Ġto Ġgenerate Ġthe Ġcontents Ġof Ġan Ġimage Ġregion Ġfrom Ġits Ġsurroundings . ĠIn Ġspeech Ġprocessing ĠB IB REF 13 , ĠB IB REF 14 Ġproposed Ġa Ġsequence Ġto Ġsequence Ġapproach Ġto Ġpredict Ġsurrounding Ġsegments Ġof Ġa Ġtarget Ġsegment . ĠHowever , Ġthe Ġapproach Ġagain Ġaims Ġat Ġcapturing Ġword Ġsemantics Ġfrom Ġraw Ġspeech Ġaudio , Ġwords Ġhas Ġsimilar Ġsemantic Ġmeanings Ġare Ġnearby Ġin ĠEucl ide an Ġdistance . ĠB IB REF 15 Ġproposed Ġan Ġun super vised Ġacoustic Ġembed ding Ġapproach . ĠIn Ġtheir Ġapproach , Ġinstead Ġof Ġdirectly Ġestimating Ġthe Ġneighbourhood Ġframes Ġof Ġa Ġtarget Ġsegment , Ġa ĠSi ames e Ġarchitecture Ġis Ġused Ġto Ġdiscriminate Ġwhether Ġa Ġspeech Ġsegment Ġis Ġin Ġthe Ġneighbourhood Ġof Ġa Ġtarget Ġsegment Ġor Ġnot . ĠFurthermore , Ġtheir Ġapproach Ġonly Ġaims Ġat Ġembed ding Ġof Ġspeaker Ġproperties . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwork Ġpresented Ġhere Ġis Ġthe Ġfirst Ġderive Ġphon eme Ġand Ġspeaker Ġrepresentations Ġby Ġtemporal Ġcontext Ġprediction Ġusing Ġacoustic Ġdata . </s> Model ĠArchitecture Ġ: :: ĠVari ational ĠAuto - En cod ers </s> As Ġshown Ġin ĠB IB REF 17 , Ġvari ational Ġauto - enc od ers Ġ( VA E ) BI B REF 8 Ġcan Ġyield Ġgood Ġrepresentations Ġin Ġthe Ġlatent Ġspace . ĠOne Ġof Ġthe Ġbenefits Ġis Ġthat Ġthe Ġmodels Ġallow Ġto Ġwork Ġwith Ġthe Ġlatent Ġdistributions BI B REF 23 , ĠB IB REF 8 , ĠB IB REF 24 . ĠIn Ġthis Ġwork , ĠVA E Ġis Ġused Ġto Ġmodel Ġthe Ġjoint Ġlatent Ġfactors Ġbetween Ġthe Ġtarget Ġsegments Ġand Ġits Ġsurroundings . </s> Different Ġfrom Ġnormal Ġauto - enc od ers , Ġwhere Ġthe Ġinput Ġdata Ġis Ġcompressed Ġinto Ġlatent Ġcode Ġwhich Ġis Ġa Ġpoint Ġestimation Ġof Ġlatent Ġvariables BI B REF 16 , Ġthe Ġvari ational Ġauto - enc oder Ġmodel Ġdefines Ġa Ġprob abil istic Ġgener ative Ġprocess Ġbetween Ġthe Ġobservation Ġ$ x $ Ġand Ġthe Ġlatent Ġvariable Ġ$ z $. ĠAt Ġthe Ġenc oder Ġstep , Ġthe Ġenc oder Ġprovides Ġan Ġestimation Ġof Ġthe Ġlatent Ġvariable Ġ$ z $ Ġgiven Ġobservation Ġ$ x $ Ġas Ġ$ p ( z | x ) $. ĠThe Ġdec oder Ġfinds Ġthe Ġmost Ġlikely Ġreconstruction Ġ$\ hat { x } $ Ġsubject Ġto Ġ$ p (\ hat { x } | z ) $. ĠThe Ġlatent Ġvariable Ġestimation Ġ$ p ( z | x ) $, Ġor Ġthe Ġprobability Ġdensity Ġfunction Ġthereof , Ġhas Ġmany Ġinterpretations , Ġsimply Ġas Ġencoding , Ġor Ġas Ġlatent Ġstate Ġspace Ġgoverning Ġthe Ġconstruction Ġof Ġthe Ġoriginal Ġsignal . </s> Com puting Ġ$ p ( z | x ) $ Ġrequires Ġan Ġestimate Ġof Ġthe Ġmarginal Ġlikelihood Ġ$ p ( x ) $ Ġwhich Ġis Ġdifficult Ġto Ġobtain Ġin Ġpractice . ĠA Ġrecognition Ġmodel Ġ$ q ( z | x ) $ Ġis Ġused Ġto Ġapproximate Ġ$ p ( z | x ) $ ĠKL Ġdivergence Ġbetween Ġ$ p ( z | x ) $ Ġand Ġ$ q ( z | x ) $, Ġas Ġshown Ġin ĠE q ĠDIS PLAY _ FORM 4 , Ġis Ġminim ised BI B REF 16 . </s> From ĠE q ĠDIS PLAY _ FORM 4 , Ġthe Ġobjective Ġfunction Ġfor ĠVA E Ġtraining Ġis Ġderived Ġshown Ġin ĠE q ĠDIS PLAY _ FORM 5 : ĠB IB REF 16 , ĠB IB REF 17 </s> where Ġ$ E _{ q ( z | x )} log [ p ( x | z )] $ Ġis Ġalso Ġcalled Ġthe Ġreconstruction Ġlikelihood Ġand Ġ$ ĠD _{ K L } ( q ( z | x )| | p ( z )) $ Ġensures Ġthe Ġlearned Ġdistribution Ġ$ q ( z | x ) $ Ġis Ġclose Ġto Ġprior Ġdistribution Ġ$ p ( z ) $. </s> Model ĠArchitecture Ġ: :: ĠProp osed ĠModel ĠArchitecture </s> An Ġaudio Ġsignal Ġis Ġrepresented Ġsequence Ġof Ġfeature Ġvectors Ġ$ S = \ l brace ĠS _ 1 , S _ 2 ,... S _ T \ r brace Ġ$ , Ġwhere Ġ$ T $ Ġis Ġthe Ġlength Ġof Ġthe Ġutter ance . ĠIn Ġthe Ġproposed Ġmethod Ġthe Ġconcept Ġof Ġa Ġtarget Ġwindow Ġis Ġused , Ġto Ġwhich Ġand Ġembed ding Ġis Ġrelated . ĠA Ġtarget Ġwindow Ġ$ X _ t $ Ġis Ġa Ġsegment Ġof Ġspeech Ġrepresenting Ġfeatures Ġfrom Ġ$ S _ t $ Ġto Ġ$ S _{ t + C - 1 } $, Ġwhere Ġ$ t Ġ\ in Ġ\ l brace Ġ1 , 2 ,... T - C + 1 \ r brace Ġ$ Ġand Ġ$ C $ Ġdenotes Ġthe Ġtarget Ġwindow Ġsize . ĠThe Ġleft Ġneighbour Ġwindow Ġof Ġthe Ġtarget Ġwindow Ġis Ġdefined Ġas Ġthe Ġsegment Ġbetween Ġ$ S _{ t - N } $ Ġand Ġ$ S _{ t - 1 } $, Ġand Ġthe Ġsegment Ġbetween Ġ$ S _{ t + C } $ Ġand Ġ$ S _{ t + C + N - 1 } $ Ġrepresents Ġthe Ġright Ġneighbour Ġwindow Ġof Ġthe Ġtarget Ġwindow , Ġwith Ġ$ N $ Ġbeing Ġthe Ġsingle Ġsided Ġneighbour Ġwindow Ġsize . ĠThe Ġconc aten ation Ġof Ġleft Ġand Ġright Ġneighbour Ġsegments Ġis Ġfurther Ġreferred Ġto Ġ$ Y _ t $. ĠThe Ġproposed Ġapproach Ġaims Ġto Ġfind Ġjoint Ġlatent Ġfactors Ġbetween Ġtarget Ġwindow Ġsegment Ġ$ X _ t $ Ġand Ġthe Ġconc aten ation Ġof Ġleft Ġand Ġright Ġneighbour Ġwindow Ġsegments Ġ$ Y _ t $, Ġfor Ġall Ġsegments . ĠFor Ġconvenience Ġthe Ġsubscript Ġ$ t $ Ġis Ġdropped Ġin Ġfollowing Ġderiv ations Ġwhere Ġappropriate . ĠTwo Ġdifferent Ġcontext Ġuse Ġconfigurations Ġcan Ġbe Ġused . </s> Figure ĠFIG REF 7 Ġillustrate Ġthese Ġtwo Ġapproaches . ĠThe Ġaudio Ġsignal Ġis Ġsplit Ġinto Ġa Ġsequence Ġof Ġleft Ġneighbour Ġsegment , Ġtarget Ġsegment Ġand Ġright Ġneighbour Ġsegment . ĠIn Ġthe Ġfirst Ġapproach Ġ( left Ġside Ġon Ġfigure ĠFIG REF 7 ), Ġthe Ġconc aten ation Ġof Ġthe Ġleft Ġneighbour Ġsegment Ġand Ġright Ġneighbour Ġsegment Ġ($ Y $ ) Ġis Ġinput Ġto Ġa ĠVA E Ġmodel BI B REF 16 , Ġand Ġtarget Ġwindow Ġ($ X $ ) Ġis Ġpredicted . ĠIn Ġthe Ġsecond Ġapproach Ġ( right Ġside Ġon Ġfigure ĠFIG REF 7 ) Ġthe Ġtarget Ġwindow Ġ($ X $ ) Ġis Ġthe Ġinput Ġto Ġa ĠVA E Ġmodel , Ġand Ġneighbour Ġwindow Ġ($ Y $ ) Ġis Ġpredicted . </s> The Ġfirst Ġapproach Ġis Ġreferred Ġto Ġas Ġthe Ġcontextual Ġjoint Ġfactor Ġsynthesis Ġenc oder Ġas Ġit Ġaims Ġto Ġsynthes ise Ġthe Ġtarget Ġframe Ġ$ X $. ĠOnly Ġfactors Ġcommon Ġbetween Ġinput Ġand Ġoutput Ġcan Ġform Ġthe Ġbasis Ġfor Ġsuch Ġprediction , Ġand Ġthe Ġencoded Ġembed ding Ġcan Ġbe Ġconsidered Ġa Ġrepresentation Ġof Ġthese Ġjoint Ġfactors . ĠSimilar Ġto Ġthe Ġstandard ĠVA E Ġformulations , Ġthe Ġobjective Ġfunction Ġof ĠCJ FS Ġis Ġgiven Ġin ĠE q . ĠDIS PLAY _ FORM 8 : </s> The Ġfirst Ġterm Ġrepresents Ġthe Ġreconstruction Ġlikelihood Ġbetween Ġpredicted Ġtarget Ġwindow Ġsegments Ġand Ġthe Ġneighbour Ġwindow Ġsegments , Ġand Ġthe Ġsecond Ġterm Ġdenotes Ġhow Ġsimilar Ġthe Ġlearned Ġdistribution Ġ$ q ( z | Y ) $ Ġis Ġto Ġthe Ġprior Ġdistribution Ġof Ġ$ z $, Ġ$ p ( z ) $ </s> In Ġpractice , Ġfor Ġthe Ġreconstruction Ġterm Ġcan Ġbe Ġbased Ġon Ġthe Ġmean Ġsquared Ġerror Ġ( M SE ) Ġbetween Ġthe Ġtrue Ġtarget Ġsegment Ġand Ġthe Ġpredicted Ġtarget Ġsegment . ĠFor Ġthe Ġsecond Ġterm Ġin ĠE q ĠDIS PLAY _ FORM 8 , Ġsamples Ġfor Ġ$ p ( z ) $ Ġare Ġobtained Ġfrom ĠGa ussian Ġdistribution Ġwith Ġzero Ġmean Ġand Ġa Ġvariance Ġof Ġone Ġ($ p ( z ) Ġ\ sim Ġ\ math cal Ġ{ N } Ġ( 0 , 1 ) $ ). </s> The Ġsecond Ġapproach Ġis Ġthe Ġcontextual Ġjoint Ġfactor Ġanalysis Ġenc oder . ĠThe Ġobjective Ġis Ġto Ġpredict Ġthe Ġtemporal Ġcontext Ġ$ Y $ Ġbased Ġon Ġinput Ġfrom Ġa Ġsingle Ġcentre Ġsegment Ġ$ X $. ĠAgain Ġjoint Ġfactors Ġbetween Ġthe Ġthree Ġwindows Ġare Ġobtained , Ġand Ġencoded Ġin Ġan Ġembed ding . ĠHowever Ġthis Ġtime Ġan Ġanalysis Ġof Ġone Ġsegment Ġis Ġenough . ĠNaturally Ġthe Ġtraining Ġobjective Ġfunction Ġof ĠCJ FA Ġis Ġrepresented Ġby Ġchange Ġof Ġvariables , Ġas Ġgiven Ġin ĠE q ĠDIS PLAY _ FORM 9 . </s> Exper imental ĠFramework </s> Exper imental ĠFramework Ġ: :: ĠData Ġand ĠUse </s> Taking Ġthe ĠVA E Ġexperiments Ġas Ġbaseline , Ġthe ĠTIM IT Ġdata Ġis Ġused Ġfor Ġthis Ġwork BI B REF 25 . ĠTIM IT Ġcontains Ġstudio Ġrecordings Ġfrom Ġa Ġlarge Ġnumber Ġof Ġspeakers Ġwith Ġdetailed Ġphon eme Ġsegment Ġinformation . ĠWork Ġin Ġthis Ġpaper Ġmakes Ġuse Ġof Ġthe Ġofficial Ġtraining Ġand Ġtest Ġsets , Ġcovering Ġin Ġtotal Ġ630 Ġspeakers Ġwith Ġ8 Ġutter ances Ġeach . ĠThere Ġis Ġno Ġspeaker Ġoverlap Ġbetween Ġtraining Ġand Ġtest Ġset , Ġwhich Ġcomprise Ġof Ġ4 62 Ġand Ġ168 Ġspeakers , Ġrespectively . ĠAll Ġwork Ġpresented Ġhere Ġuse Ġof Ġ80 Ġdimensional ĠMel - scale Ġfilter Ġbank Ġcoefficients . </s> Exper imental ĠFramework Ġ: :: ĠBas elines </s> Work Ġon ĠVA E Ġin ĠB IB REF 17 Ġto Ġlearn Ġacoustic Ġembed d ings Ġconducted Ġexperiments Ġusing Ġthe ĠTIM IT Ġdata Ġset . ĠIn Ġparticular Ġthe Ġtasks Ġof Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġwhere Ġchosen . ĠAs Ġwork Ġhere Ġis Ġan Ġextension Ġof Ġsuch Ġwork Ġwe Ġwe Ġfollow Ġthe Ġexperimentation , Ġhowever Ġwith Ġsignificant Ġextensions Ġ( see ĠSection ĠSEC REF 13 ). ĠWith Ġguidance Ġfrom Ġthe Ġauthors Ġof Ġthe Ġoriginal Ġwork BI B REF 17 Ġour Ġown Ġimplementation Ġof ĠVA E Ġwas Ġcreated Ġand Ġcompared Ġwith Ġthe Ġpublished Ġperformance Ġ- Ġyielding Ġnear Ġidentical Ġresults . ĠThis Ġimplementation Ġthen Ġwas Ġalso Ġused Ġas Ġthe Ġbasis Ġfor ĠCJ FS Ġand ĠCJ FA , Ġas Ġintroduced Ġin ĠÂ§ ĠSEC REF 6 . </s> For Ġthe Ġassessment Ġof Ġembedded Ġvector Ġquality Ġour Ġwork Ġalso Ġfollows Ġthe Ġsame Ġtask Ġtypes , Ġnamely Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġ( details Ġin ĠÂ§ SEC REF 13 ), Ġwith Ġidentical Ġtask Ġimplementations Ġas Ġin Ġthe Ġreference Ġpaper . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġphone Ġclassification Ġdiffers Ġfrom Ġthe Ġwidely Ġreported Ġphone Ġrecognition Ġexperiments Ġon ĠTIM IT . ĠClassification Ġuses Ġphone Ġboundaries Ġwhich Ġare Ġassumed Ġto Ġbe Ġknown . ĠHowever , Ġno Ġcontextual Ġinformation Ġis Ġavailable , Ġwhich Ġis Ġtypically Ġused Ġin Ġthe Ġrecognition Ġsetups , Ġby Ġmeans Ġof Ġtri phone Ġmodels , Ġor Ġbig ram Ġlanguage Ġmodels . ĠTherefore Ġthe Ġtask Ġis Ġoften Ġmore Ġdifficult Ġthan Ġrecognition . ĠThe Ġbaseline Ġperformance Ġfor ĠVA E Ġbased Ġphone Ġclassification Ġexperiments Ġin ĠB IB REF 17 Ġreport Ġan Ġaccuracy Ġof Ġ72 . 2 %. ĠThe Ġre - im plementation Ġforming Ġthe Ġbasis Ġfor Ġour Ġwork Ġgave Ġan Ġaccuracy Ġof Ġ72 . 0 %, Ġa Ġresult Ġthat Ġwas Ġconsidered Ġto Ġprovide Ġa Ġcredible Ġbasis Ġfor Ġfurther Ġwork . </s> For Ġthe Ġpurpose Ġof Ġspeaker Ġrecognition Ġit Ġis Ġimportant Ġto Ġtake Ġinto Ġaccount Ġthe Ġoverlap Ġbetween Ġtraining Ġand Ġtesting . ĠThus Ġthree Ġdifferent Ġtask Ġconfigurations Ġare Ġconsidered , Ġdifferent Ġto Ġthe Ġsetting Ġin ĠB IB REF 17 . ĠTheir Ġbaseline Ġwill Ġbe Ġfurther Ġreferred Ġas ĠVA E Ġbaseline . </s> Exper imental ĠFramework Ġ: :: ĠEvaluation ĠT asks </s> The Ġphone Ġclassification Ġimplementation Ġoperates Ġon Ġsegment Ġlevel , Ġusing Ġa Ġconv olution al Ġnetwork Ġto Ġobtain Ġframe Ġby Ġframe Ġposter iors Ġwhich Ġare Ġthen Ġaccumulated Ġfor Ġsegment Ġdecision Ġ( assuming Ġframe Ġindependence ). ĠThe Ġphone Ġclass Ġwith Ġthe Ġhighest Ġsegment Ġposterior Ġis Ġchosen Ġas Ġoutput . ĠAn Ġidentical Ġapproach Ġis Ġused Ġfor Ġspeaker Ġrecognition . ĠIn Ġthis Ġsetting Ġ3 Ġdifferent Ġdata Ġsets Ġare Ġrequired : Ġa Ġtraining Ġset Ġfor Ġlearning Ġthe Ġenc oder Ġmodels , Ġa Ġtraining Ġset Ġfor Ġlearning Ġthe Ġclassification Ġmodel , Ġand Ġan Ġevaluation Ġtest Ġset . ĠFor Ġthe Ġphone Ġclassification Ġtask , Ġboth Ġembed ding Ġand Ġclassification Ġmodels Ġare Ġtrained Ġon Ġthe Ġofficial ĠTIM IT Ġtraining Ġset , Ġand Ġmakes Ġuse Ġof Ġthe Ġprovided Ġphone Ġboundary Ġinformation . ĠA Ġfixed Ġsize Ġwindow Ġwith Ġa Ġframe Ġstep Ġsize Ġof Ġone Ġframe Ġis Ġused Ġfor Ġall Ġmodel Ġtraining . ĠAs Ġnoted , Ġphone Ġclassification Ġmakes Ġno Ġuse Ġof Ġphone Ġcontext , Ġand Ġno Ġlanguage Ġmodel Ġis Ġapplied . </s> For Ġspeaker Ġrecognition Ġoverlap Ġspeaker Ġbetween Ġany Ġof Ġthe Ġdatasets Ġ( training Ġembed d ings , Ġtraining Ġclass ifier Ġand Ġtest ) Ġwill Ġcause Ġa Ġbias . ĠThree Ġdifferent Ġconfigurations Ġ( T asks Ġa , b , c ) Ġare Ġused Ġto Ġassess Ġthis Ġbias . ĠTask Ġa Ġreflects Ġthe Ġsituation Ġwhere Ġboth Ġclass ifier Ġand Ġembed ding Ġare Ġtrained Ġon Ġthe Ġsame Ġdata . ĠAs Ġthe Ġtask Ġis Ġto Ġdetect Ġa Ġspeaker Ġthe Ġspeakers Ġpresent Ġin Ġthe Ġtest Ġset Ġneed Ġto Ġbe Ġpresent Ġin Ġtraining . ĠTask Ġb Ġrepresents Ġa Ġsituation Ġwhere Ġclass ifier Ġand Ġembed ding Ġare Ġtrained Ġon Ġindependent Ġdata Ġsets , Ġbut Ġwith Ġspeaker Ġoverlap . ĠFinally ĠTask Ġc Ġrepresents Ġcomplete Ġindependence Ġin Ġtraining Ġdata Ġsets Ġand Ġno Ġspeaker Ġoverlap . ĠTable ĠT AB REF 15 Ġsummar ises Ġthe Ġrelationships . </s> In Ġorder Ġto Ġachieve Ġthese Ġconfiguration Ġthe ĠTIM IT Ġdata Ġwas Ġsplit . ĠFig . ĠFIG REF 12 Ġillustrates Ġthe Ġsplit Ġof Ġthe Ġdata Ġinto Ġ8 Ġsubs ets Ġ( A âĢĵ H ). ĠThe ĠTIM IT Ġdataset Ġcontains Ġspeech Ġfrom Ġ4 62 Ġspeakers Ġin Ġtraining Ġand Ġ168 Ġspeakers Ġin Ġthe Ġtest Ġset , Ġwith Ġ8 Ġutter ances Ġfor Ġeach Ġspeaker . ĠThe ĠTIM IT Ġtraining Ġand Ġtest Ġset Ġare Ġsplit Ġinto Ġ8 Ġblocks , Ġwhere Ġeach Ġblock Ġcontains Ġ2 Ġutter ances Ġper Ġspeaker , Ġrandomly Ġchosen . ĠThus Ġeach Ġblock ĠA , B , C , D Ġcontains Ġdata Ġfrom Ġ4 62 Ġspeakers Ġwith Ġ9 24 Ġutter ances Ġtaken Ġfrom Ġthe Ġtraining Ġsets , Ġand Ġeach Ġblock ĠE , F , G , H Ġcontains Ġspeech Ġfrom Ġ168 Ġtest Ġset Ġspeakers Ġwith Ġ336 Ġutter ances . </s> For ĠTask Ġa Ġtraining Ġof Ġembed d ings Ġand Ġthe Ġclass ifier Ġis Ġidentical , Ġnamely Ġconsisting Ġof Ġdata Ġfrom Ġblocks Ġ( A + B + C + E + F + G ). ĠThe Ġtest Ġdata Ġis Ġthe Ġremainder , Ġnamely Ġblocks Ġ( D + H ). ĠFor ĠTask Ġb Ġthe Ġtraining Ġof Ġembed d ings Ġand Ġclass ifiers Ġuses Ġ( A + B + E + F ) Ġand Ġ( C + G ) Ġrespectively , Ġwhile Ġagain Ġusing Ġ( D + H ) Ġfor Ġtest . ĠTask Ġc Ġkeeps Ġboth Ġseparate : Ġembed d ings Ġare Ġtrained Ġon Ġ( A + B + C + D ), Ġclass ifiers Ġon Ġ( E + G ) Ġand Ġtests Ġare Ġconducted Ġon Ġ( F + H ). ĠNote Ġthat ĠH Ġis Ġpart Ġof Ġall Ġtasks , Ġand Ġthat ĠTask Ġc Ġis Ġconsiderably Ġeasier Ġas Ġthe Ġnumber Ġof Ġspeakers Ġto Ġseparate Ġis Ġonly Ġ168 , Ġalthough Ġtraining Ġconditions Ġare Ġmore Ġdifficult . </s> Exper imental ĠFramework Ġ: :: ĠImplementation </s> For Ġcomparison Ġthe Ġimplementation , Ġfollows Ġthe Ġconv olution al Ġmodel Ġstructure Ġas Ġdeployed Ġin ĠB IB REF 17 . ĠBoth ĠVA E Ġenc oder Ġand Ġdec oder Ġcontain Ġthree Ġconv olution al Ġlayers Ġand Ġone Ġfully - connected Ġlayer Ġwith Ġ512 Ġnodes . ĠIn Ġthe Ġfirst Ġlayer Ġof Ġenc oder , Ġ1 - by - 80 Ġfilters Ġare Ġapplied , Ġand Ġ3 - by - 1 Ġfilters Ġare Ġapplied Ġon Ġthe Ġfollowing Ġtwo Ġconv olution al Ġlayer Ġ( str ides Ġwas Ġset Ġto Ġ1 Ġin Ġthe Ġfirst Ġlayer Ġand Ġ2 Ġin Ġthe Ġrest Ġtwo Ġlayers ). ĠThe Ġdec oder Ġhas Ġthe Ġsymm etric Ġarchitecture Ġto Ġthe Ġenc oder . ĠEach Ġlayer Ġis Ġfollowed Ġby Ġa Ġbatch Ġnormal isation Ġlayer BI B REF 26 Ġexcept Ġfor Ġthe Ġembed ding Ġlayer , Ġwhich Ġis Ġlinear . ĠLe aky ĠRe LU Ġactivation BI B REF 27 Ġis Ġused Ġfor Ġeach Ġlayer Ġexcept Ġfor Ġthe Ġembed ding Ġlayer . ĠThe ĠAdam Ġoptim iser BI B REF 28 Ġis Ġused Ġin Ġtraining , Ġwith Ġ$\ beta Ġ_ 1 $ Ġset Ġto Ġ0 . 95 , Ġ$\ beta Ġ_ 2 $ Ġto Ġ0 . 999 , Ġand Ġ$\ ep sil on Ġ$ Ġis Ġ$ 10 ^{ - 8 } $. ĠThe Ġinitial Ġlearning Ġrate Ġis Ġ$ 10 ^{ - 3 } $ </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 17 Ġshows Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġresults Ġfor Ġthe Ġthree Ġmodel Ġconfigurations : Ġthe ĠVA E Ġbaseline , Ġthe ĠCJ FS Ġenc oder Ġand Ġthe ĠCJ FA Ġenc oder . ĠIn Ġour Ġexperiments Ġthe Ġwindow Ġsize Ġwas Ġset Ġto Ġ30 Ġframes , Ġnamely Ġ10 Ġframes Ġfor Ġthe Ġtarget Ġand Ġ10 Ġframes Ġfor Ġleft Ġand Ġright Ġneighbours , Ġand Ġan Ġembed ding Ġdimension Ġof Ġ150 . ĠThis Ġwas Ġused Ġfor Ġboth ĠCJ FS Ġand ĠCJ FA Ġmodels Ġalike . ĠResults Ġshow Ġthat Ġthe ĠCJ FA Ġenc oder Ġob tains Ġsignificantly Ġbetter Ġphone Ġclassification Ġaccuracy Ġthan Ġthe ĠVA E Ġbaseline Ġand Ġalso Ġthan Ġthe ĠCJ FS Ġenc oder . ĠThese Ġresults Ġare Ġreplicated Ġfor Ġspeaker Ġrecognition Ġtasks . ĠThe ĠCJ FA Ġenc oder Ġperforms Ġbetter Ġon Ġall Ġtasks Ġthan Ġthe ĠVA E Ġbaseline Ġby Ġa Ġsignificant Ġmargin . ĠIt Ġis Ġnoteworthy Ġthat Ġperformance Ġon ĠTask Ġb Ġis Ġgenerally Ġsignificantly Ġlower Ġthan Ġfor ĠTask Ġa , Ġfor Ġreasons Ġof Ġtraining Ġoverlap Ġbut Ġalso Ġsmaller Ġtraining Ġset Ġsizes . </s> To Ġfurther Ġexplore Ġproperties Ġof Ġthe Ġembed ding Ġsystems Ġa Ġchange Ġof Ġwindow Ġsize Ġ($ N $ ) Ġand Ġembed ding Ġdimension Ġ($ K $ ) Ġis Ġexplored . ĠOne Ġmight Ġargue Ġthat Ġmodelling Ġcontext Ġeffectively Ġwid ens Ġthe Ġinput Ġdata Ġaccess . ĠHence Ġthese Ġexperiments Ġshould Ġexplore Ġif Ġthere Ġis Ġbenefit Ġin Ġthe Ġstructure Ġbeyond Ġdata Ġsize . ĠGraph s Ġin ĠFig . ĠFIG REF 14 Ġillustrate Ġphone Ġclassification Ġaccuracy Ġand Ġspeaker Ġrecognition Ġperformance Ġfor Ġall Ġthree Ġmodels Ġunder Ġvariation Ġof Ġlatent Ġsize Ġand Ġwindow Ġsizes . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġthe Ġtarget Ġwindow Ġsize Ġremains Ġthe Ġsame Ġ( 10 Ġframes ) Ġwith Ġan Ġincrease Ġof Ġ$ N $. ĠTherefore Ġe . g . Ġ$ N = 70 $ Ġdescribes Ġthe Ġtarget Ġwindow Ġsize Ġis Ġ10 Ġframes , Ġand Ġthe Ġother Ġtwo Ġneighbour Ġwindows Ġhave Ġ30 Ġframes Ġat Ġeither Ġside Ġ( 30 , 10 , 30 Ġleft Ġto Ġright ). ĠBetter Ġspeaker Ġrecognition Ġresults Ġare Ġconsistently Ġobtained Ġwith Ġthe ĠCJ FA Ġenc oder Ġfor Ġany Ġconfiguration Ġwith Ġcompetitive Ġperformance , Ġcompared Ġwith Ġthe ĠVA E Ġbaseline Ġand Ġalso ĠCJ FS Ġsettings Ġ- Ġand ĠCJ FS Ġsettings Ġmostly Ġoutper form Ġthe Ġbaseline . ĠHowever Ġthe Ġsituation Ġfor Ġphone Ġclassification Ġis Ġdifferent . ĠIt Ġis Ġnot Ġsurprising Ġto Ġsee ĠCJ FS Ġperform Ġpoorly Ġon Ġphone Ġclassification Ġas Ġthe Ġtarget Ġframe Ġin Ġnot Ġpresent Ġin Ġthe Ġinput , Ġtherefore Ġthe Ġembed ding Ġjust Ġdoes Ġnot Ġhave Ġthe Ġphone Ġsegment Ġinformation . ĠHowever , Ġas Ġper Ġspeaker Ġrecognition Ġresults , Ġspeaker Ġinformation Ġis Ġretained . </s> A Ġvariation Ġof Ġthe Ġwindow Ġsizes Ġto Ġlarger Ġwindows Ġseems Ġdetrimental Ġin Ġalmost Ġall Ġcases , Ġaside Ġfrom Ġthe Ġmore Ġdifficult ĠTask Ġb . ĠThis Ġmay Ġbe Ġin Ġpart Ġthe Ġeffect Ġof Ġthe Ġamount Ġof Ġtraining Ġdata Ġavailable , Ġhowever Ġit Ġconfirms Ġthat Ġcontextual Ġmodels Ġoutper form Ġthe Ġbaseline ĠVA E Ġmodel Ġconfiguration , Ġgenerally , Ġand Ġin Ġparticular Ġalso Ġwith Ġthe Ġsame Ġamount Ġof Ġinput Ġdata Ġfor Ġspeaker Ġrecognition . ĠIt Ġis Ġalso Ġnoticeable Ġthat Ġthe Ġdecline Ġor Ġvariation Ġas Ġa Ġfunction Ġof Ġwindow Ġsize Ġis Ġless Ġpronounced Ġfor Ġthe ĠCJ FA Ġcase , Ġimplying Ġincreased Ġstability . ĠFor Ġphone Ġclassification Ġthe Ġtrade - off Ġbenefit Ġfor Ġwindow Ġsize Ġis Ġless Ġclear . </s> For Ġphone Ġclassification , Ġincreasing Ġthe Ġembed ding Ġ$ K $ Ġis Ġhelpful , Ġbut Ġperformance Ġremains Ġstable Ġat Ġ$ K = 150 $. ĠHence Ġin Ġall Ġof Ġthe Ġrest Ġof Ġour Ġexperiments , Ġthe Ġembed ding Ġdimension Ġis Ġset Ġto Ġ150 Ġfor Ġall Ġof Ġthe Ġrest Ġconfigurations . ĠFor Ġspeaker Ġrecognition Ġthe Ġobserved Ġvariations Ġare Ġsmall . </s> A Ġfurther Ġset Ġof Ġexperiments Ġinvestigated Ġthe Ġuse Ġof Ġout Ġof Ġdomain Ġdata Ġfor Ġimproving Ġclassification Ġin Ġa Ġcompletely Ġun super vised Ġsetting . ĠThe ĠRM Ġcorpus ĠB IB REF 29 Ġwas Ġused Ġin Ġthis Ġcase Ġto Ġaugment Ġthe ĠTIM IT Ġdata Ġfor Ġtraining Ġthe Ġembed d ings Ġonly . ĠAll Ġother Ġconfigurations Ġan Ġtraining Ġsettings Ġare Ġunchanged . ĠTable ĠT AB REF 18 Ġshows Ġimprovement Ġafter Ġusing Ġadditional Ġout - of - domain Ġdata Ġfor Ġtraining , Ġexcept Ġfor Ġin Ġthe Ġcase Ġof ĠCJ FS Ġand Ġfor Ġphone Ġclassification . ĠThe Ġimprovement Ġon Ġall Ġtasks Ġwith Ġthe Ġsimple Ġaddition Ġof Ġunl ab elled Ġaudio Ġdata Ġis Ġremarkable . ĠThis Ġis Ġalso Ġtrue Ġfor Ġthe Ġbaseline , Ġbut Ġthe Ġbenefit Ġof Ġthe Ġproposed Ġmethods Ġseems Ġunaffected . ĠThe ĠCJ FA Ġenc oder Ġperforms Ġbetter Ġin Ġcomparison Ġof Ġthe Ġother Ġtwo Ġapproaches Ġand Ġa Ġabsolute Ġaccuracy Ġimprovement Ġof Ġ7 . 9 % Ġfor Ġspeaker Ġrecognition ĠTask Ġb Ġis Ġobserved . ĠThe Ġclassification Ġtasks Ġbenefits Ġfrom Ġthe Ġadditional Ġdata Ġeven Ġthough Ġthe Ġlabelled Ġdata Ġremains Ġthe Ġsame . </s> Analysis </s> To Ġfurther Ġevaluate Ġthe Ġembed d ings Ġproduced Ġby Ġthe Ġ3 Ġmodels , Ġvisual isation Ġusing Ġthe Ġt - S NE Ġalgorithm BI B REF 30 Ġis Ġa Ġcommon Ġapproach , Ġalthough Ġinterpretation Ġis Ġsometimes Ġdifficult . ĠFig . ĠFIG REF 19 Ġvisual ises Ġthe Ġembed d ings Ġof Ġphon em es Ġin Ġtwo - dimensional Ġspace , Ġeach Ġphon eme Ġsymbol Ġrepresents Ġthe Ġmean Ġvector Ġof Ġall Ġof Ġthe Ġembed d ings Ġbelonging Ġto Ġthe Ġsame Ġphone Ġclass BI B REF 31 . ĠOne Ġcan Ġobserve Ġthat Ġthe ĠCJ FA Ġenc oder Ġappears Ġto Ġgenerate Ġmore Ġmeaningful Ġembed d ings Ġthan Ġthe Ġother Ġtwo Ġapproaches Ġ- Ġas Ġphon em es Ġbelonging Ġto Ġthe Ġsame Ġsound Ġclasses BI B REF 32 Ġare Ġgrouped Ġtogether Ġin Ġcloser Ġregions . ĠThe ĠVA E Ġbaseline Ġalso Ġhas Ġthis Ġbehaviour Ġbut Ġfor Ġexample Ġpl os ives Ġare Ġsplit Ġand Ġnasal Ġseparation Ġseems Ġless Ġclear . ĠInstead ĠCJ FS Ġshows Ġmore Ġconfusion Ġ- Ġas Ġexpected Ġand Ġexplained Ġabove . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġtwo Ġun super vised Ġacoustic Ġembed ding Ġapproaches Ġto Ġmodel Ġthe Ġjoint Ġlatent Ġfactors Ġbetween Ġthe Ġtarget Ġwindow Ġand Ġneighbouring Ġaudio Ġsegments Ġwere Ġproposed . ĠModels Ġare Ġbased Ġon Ġvari ational Ġauto - enc od ers , Ġwhich Ġalso Ġconstitute Ġthe Ġbaseline . ĠIn Ġorder Ġto Ġcompare Ġagainst Ġthe Ġbaseline Ġmodels Ġare Ġassessed Ġusing Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġtasks , Ġon ĠTIM IT , Ġand Ġwith Ġadditional ĠRM Ġdata . ĠResults Ġshow ĠCJ FA Ġ( context ual Ġjoint Ġfactor Ġanalysis ) Ġenc oder Ġperforms Ġsignificantly Ġbetter Ġin Ġboth Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġtasks Ġcompared Ġwith Ġother Ġtwo Ġapproaches . ĠThe ĠCJ FS Ġ( context ual Ġjoint Ġfactor Ġsynthesis ) Ġenc oder Ġperforms Ġclose Ġto ĠCJ FA Ġin Ġspeaker Ġrecognition Ġtask , Ġbut Ġpoorer Ġfor Ġphone Ġclassification . ĠOverall Ġa Ġgain Ġof Ġup Ġto Ġ3 % Ġrelative Ġon Ġphone Ġclassification Ġaccuracy Ġis Ġobserved , Ġrelative Ġimprovements Ġon Ġspeaker Ġrecognition Ġshow Ġ3 âĢĵ 6 % Ġgain . ĠThe Ġproposed Ġun super vised Ġapproaches Ġobtain Ġembed d ings Ġand Ġcan Ġbe Ġimproved Ġwith Ġunl ab elled Ġout - of - domain Ġdata , Ġthe Ġclassification Ġtasks Ġbenefits Ġeven Ġthough Ġthe Ġlabelled Ġdata Ġremains Ġthe Ġsame . ĠFurther Ġwork Ġneeds Ġto Ġexpand Ġexperiments Ġon Ġlarger Ġdata Ġsets , Ġphone Ġrecognition Ġand Ġmore Ġcomplex Ġneural Ġnetwork Ġarchitectures .
Introduction In recent years, word embeddings have been successfully used in natural language processing (NLP), the most commonly known models are Word2Vec BIBREF0 and GloveBIBREF1. The reasons for such success are manifold. One key attribute of embedding methods is that word embedding models take into account context information of words, thereby allowing a more compact and manageable representation for wordsBIBREF2, BIBREF3. The embeddings are widely applied in many downstream NLP tasks such as neural machine translation, dialogue system or text summarisation BIBREF4, BIBREF5, BIBREF6, as well as in language modelling for speech recognitionBIBREF7. Embeddings of acoustic (and speech) signals are of more recent interest. The objective is to represent audio sequence information in compact form, replacing the raw audio data with one that contains latent factors BIBREF8, BIBREF9. The projection into such (latent) spaces should take into account different attributes, such as phonemes, speaker properties, speaking styles, the acoustic background or the recording environment. Acoustic embeddings have been explored for a variety of speech tasks such as speech recognitionBIBREF10, speaker verificationBIBREF11 or voice conversionBIBREF12. However, learning acoustic embeddings is challenging: attributes mentioned above, e.g. speaker properties and phonemes, operate at different levels of abstraction and are often strongly interdependent, and therefore are difficult to extract and represent in a meaningful formBIBREF8. For speech processing BIBREF13, BIBREF14, BIBREF15 also use context information to derive acoustic embeddings. However, BIBREF13, BIBREF14 focus on learning word semantic representations from raw audio instead of signal properties such as phonemes and speaker properties. BIBREF15 focus on learning speaker representations by modelling of context information with a Siamese networks that discriminate whether a speech segment is the neighbourhood of a target segment or not. In this paper, two unsupervised approaches to generate acoustic embeddings using context modelling are proposed. Both methods make use of the variational auto-encoder framework as proposed inBIBREF16 and both approaches aim to find joint latent variables between the target acoustic segments and its surrounding frames. In the first instance a representation is derived from surrounding audio frames that allows to predict current frame, thereby generating target audio from common factors. The encoder element of the associated auto-encoder is further referred to as Contextual Joint Factor Synthesis (CJFS) encoder. In the second instance an audio frame is used to predict surrounding audio, which is further referred to as Contextual Joint Factor Analysis (CJFA) encoding. As shown in previous work variational auto-encoders can be used to derive latent variables such as speaker information and phonemesBIBREF8 more robustly. In this work it is shown that including temporal information can further improve performance and robustness, for both phoneme classification and speaker identification tasks. Furthermore the use of additional unlabelled out-of-domain data can improved modelling for the proposed approaches. As outlined above, prior work has made use of surrounding audio in different forms. To the best of our knowledge this work is the first to show that predicting surrounding audio allows for efficient extraction of latent factors in speech signals. The rest of paper is organised as follows: In §SECREF2 related work is described, Methods for deriving acoustic embeddings, and context modelling methods in NLP, computer vision and speech are discussed. This is followed by the description of the two approaches for modelling context as used in this work, in §SECREF3. The experimental framework is described in §SECREF4, including the data organisation, baseline design and task definitions; in §SECREF5 and §SECREF6 experiments results are shown and discussed. This is followed by the conclusions and future work in §SECREF7. Related Works Related Works ::: Acoustic Embeddings Most interest in acoustic embeddings can be observed on acoustic word embeddings, i.e. projections that map word acoustics into a fixed size vector space. Objective functions are chosen to project different word realisations to close proximity in the embedding space. Different approaches were used in the literature - for both supervised and unsupervised learning. For the supervised case, BIBREF9 introduced a convolutional neural network (CNN) based acoustic word embedding system for speech recognition, where words that sound alike are nearby in Euclidean distance. In their work, a CNN is used to predict a word from the corresponding acoustic signal, the output of the bottleneck layer before the final softmax layer is taken to be the embedding for the corresponding word. Further work used different network architectures to obtain acoustic word embeddings: BIBREF10 introduces a recurrent neural network (RNN) based approach instead. For the case that the word boundary information is available but the word label itself is unknown, BIBREF12 proposed word similarity Siamese CNNs. These are used to minimise a distance function between representations of two instances of the same word type whilst at the same time maximising the distance between two instances of different words. Unsupervised approaches also exist. BIBREF17 proposed a convolutional variational auto-encoder based approach to derive acoustic embedding, in unsupervised fashion. The authors chose phoneme and speaker classification tasks on TIMIT data to assess the quality of their embeddings - an approach replicated in the work presented in this paper. BIBREF8, BIBREF18 proposed an approach called factorised hierarchical variational auto-encoder. The work introduces the concepts of global and local latent factors, i.e. latent variables that are shared on the complete utterance, or latent variables that change within the sequence, respectively. Results are again obtained using the same data and tasks as above. Related Works ::: Context Modelling Context information plays a fundamental role in speech processing. Phonemes could be influenced by surrounding frames through coarticulationBIBREF19 - an effect caused by speed limitations and transitions in the movement of articulators. Normally directly neighbouring phonemes have important impact on the sound realisation. Inversely, the surrounding phonemes also provide strong constraints on the phoneme that can be chosen at any given point, subject to to lexical and language constraints. This effect is for example exploited in phoneme recognition, by use of phoneme $n$-gram modelsBIBREF20. Equivalently inter word dependency - derived from linguistic constraints - can be exploited, as is the case in computing word embeddings with the aforementioned word2vecBIBREF0 method. The situation differs for the global latent variables, such as speaker properties or acoustic environment information. Speaker properties remains constant - and environments can also be assumed stationary over longer periods of time. Hence these variables are common between among neighbouring frames and windows. Modelling context information is helpful for identifying such information BIBREF21. There is significant prior work that takes surrounding information into account to learn vector representations. For text processing the Word2VecBIBREF0 model directly predicts the neighbouring words from target words or inversely. This helps to capture the meanings of wordsBIBREF2. In computer vision, BIBREF22 introduced an visual feature learning approach called context encoder ,which is based on context based pixel prediction. Their model is trained to generate the contents of an image region from its surroundings. In speech processing BIBREF13, BIBREF14 proposed a sequence to sequence approach to predict surrounding segments of a target segment. However, the approach again aims at capturing word semantics from raw speech audio, words has similar semantic meanings are nearby in Euclidean distance. BIBREF15 proposed an unsupervised acoustic embedding approach. In their approach, instead of directly estimating the neighbourhood frames of a target segment, a Siamese architecture is used to discriminate whether a speech segment is in the neighbourhood of a target segment or not. Furthermore, their approach only aims at embedding of speaker properties. To the best of our knowledge, work presented here is the first derive phoneme and speaker representations by temporal context prediction using acoustic data. Model Architecture ::: Variational Auto-Encoders As shown in BIBREF17, variational auto-encoders (VAE)BIBREF8 can yield good representations in the latent space. One of the benefits is that the models allow to work with the latent distributionsBIBREF23, BIBREF8, BIBREF24. In this work, VAE is used to model the joint latent factors between the target segments and its surroundings. Different from normal auto-encoders, where the input data is compressed into latent code which is a point estimation of latent variablesBIBREF16, the variational auto-encoder model defines a probabilistic generative process between the observation $x$ and the latent variable $z$. At the encoder step, the encoder provides an estimation of the latent variable $z$ given observation $x$ as $p(z|x)$. The decoder finds the most likely reconstruction $\hat{x}$ subject to $p(\hat{x}|z)$. The latent variable estimation $p(z|x)$, or the probability density function thereof, has many interpretations, simply as encoding, or as latent state space governing the construction of the original signal. Computing $p(z|x)$ requires an estimate of the marginal likelihood $p(x)$ which is difficult to obtain in practice. A recognition model $q(z|x)$ is used to approximate $p(z|x)$ KL divergence between $p(z|x)$ and $q(z|x)$, as shown in Eq DISPLAY_FORM4, is minimisedBIBREF16. From Eq DISPLAY_FORM4, the objective function for VAE training is derived shown in Eq DISPLAY_FORM5: BIBREF16, BIBREF17 where $E_{q(z|x)}log[p(x|z)]$ is also called the reconstruction likelihood and $ D_{KL}(q(z|x)||p(z))$ ensures the learned distribution $q(z|x)$ is close to prior distribution $p(z)$. Model Architecture ::: Proposed Model Architecture An audio signal is represented sequence of feature vectors $S=\lbrace S_1,S_2,...S_T\rbrace $, where $T$ is the length of the utterance. In the proposed method the concept of a target window is used, to which and embedding is related. A target window $X_t$ is a segment of speech representing features from $S_t$ to $S_{t+C-1}$, where $t \in \lbrace 1,2,...T-C+1\rbrace $ and $C$ denotes the target window size. The left neighbour window of the target window is defined as the segment between $S_{t-N}$ and $S_{t-1}$, and the segment between $S_{t+C}$ and $S_{t+C+N-1}$ represents the right neighbour window of the target window, with $N$ being the single sided neighbour window size. The concatenation of left and right neighbour segments is further referred to $Y_t$. The proposed approach aims to find joint latent factors between target window segment $X_t$ and the concatenation of left and right neighbour window segments $Y_t$, for all segments. For convenience the subscript $t$ is dropped in following derivations where appropriate. Two different context use configurations can be used. Figure FIGREF7 illustrate these two approaches. The audio signal is split into a sequence of left neighbour segment, target segment and right neighbour segment. In the first approach (left side on figure FIGREF7), the concatenation of the left neighbour segment and right neighbour segment ($Y$) is input to a VAE modelBIBREF16, and target window ($X$) is predicted. In the second approach (right side on figure FIGREF7) the target window ($X$) is the input to a VAE model, and neighbour window ($Y$) is predicted. The first approach is referred to as the contextual joint factor synthesis encoder as it aims to synthesise the target frame $X$. Only factors common between input and output can form the basis for such prediction, and the encoded embedding can be considered a representation of these joint factors. Similar to the standard VAE formulations, the objective function of CJFS is given in Eq. DISPLAY_FORM8: The first term represents the reconstruction likelihood between predicted target window segments and the neighbour window segments, and the second term denotes how similar the learned distribution $q(z|Y)$ is to the prior distribution of $z$, $p(z)$ In practice, for the reconstruction term can be based on the mean squared error (MSE) between the true target segment and the predicted target segment. For the second term in Eq DISPLAY_FORM8, samples for $p(z)$ are obtained from Gaussian distribution with zero mean and a variance of one ($p(z) \sim \mathcal {N} (0,1)$). The second approach is the contextual joint factor analysis encoder. The objective is to predict the temporal context $Y$ based on input from a single centre segment $X$. Again joint factors between the three windows are obtained, and encoded in an embedding. However this time an analysis of one segment is enough. Naturally the training objective function of CJFA is represented by change of variables, as given in Eq DISPLAY_FORM9. Experimental Framework Experimental Framework ::: Data and Use Taking the VAE experiments as baseline, the TIMIT data is used for this workBIBREF25. TIMIT contains studio recordings from a large number of speakers with detailed phoneme segment information. Work in this paper makes use of the official training and test sets, covering in total 630 speakers with 8 utterances each. There is no speaker overlap between training and test set, which comprise of 462 and 168 speakers, respectively. All work presented here use of 80 dimensional Mel-scale filter bank coefficients. Experimental Framework ::: Baselines Work on VAE in BIBREF17 to learn acoustic embeddings conducted experiments using the TIMIT data set. In particular the tasks of phone classification and speaker recognition where chosen. As work here is an extension of such work we we follow the experimentation, however with significant extensions (see Section SECREF13). With guidance from the authors of the original workBIBREF17 our own implementation of VAE was created and compared with the published performance - yielding near identical results. This implementation then was also used as the basis for CJFS and CJFA, as introduced in § SECREF6. For the assessment of embedded vector quality our work also follows the same task types, namely phone classification and speaker recognition (details in §SECREF13), with identical task implementations as in the reference paper. It is important to note that phone classification differs from the widely reported phone recognition experiments on TIMIT. Classification uses phone boundaries which are assumed to be known. However, no contextual information is available, which is typically used in the recognition setups, by means of triphone models, or bigram language models. Therefore the task is often more difficult than recognition. The baseline performance for VAE based phone classification experiments in BIBREF17 report an accuracy of 72.2%. The re-implementation forming the basis for our work gave an accuracy of 72.0%, a result that was considered to provide a credible basis for further work. For the purpose of speaker recognition it is important to take into account the overlap between training and testing. Thus three different task configurations are considered, different to the setting in BIBREF17. Their baseline will be further referred as VAE baseline. Experimental Framework ::: Evaluation Tasks The phone classification implementation operates on segment level, using a convolutional network to obtain frame by frame posteriors which are then accumulated for segment decision (assuming frame independence). The phone class with the highest segment posterior is chosen as output. An identical approach is used for speaker recognition. In this setting 3 different data sets are required: a training set for learning the encoder models, a training set for learning the classification model, and an evaluation test set. For the phone classification task, both embedding and classification models are trained on the official TIMIT training set, and makes use of the provided phone boundary information. A fixed size window with a frame step size of one frame is used for all model training. As noted, phone classification makes no use of phone context, and no language model is applied. For speaker recognition overlap speaker between any of the datasets (training embeddings, training classifier and test) will cause a bias. Three different configurations (Tasks a,b,c) are used to assess this bias. Task a reflects the situation where both classifier and embedding are trained on the same data. As the task is to detect a speaker the speakers present in the test set need to be present in training. Task b represents a situation where classifier and embedding are trained on independent data sets, but with speaker overlap. Finally Task c represents complete independence in training data sets and no speaker overlap. Table TABREF15 summarises the relationships. In order to achieve these configuration the TIMIT data was split. Fig. FIGREF12 illustrates the split of the data into 8 subsets (A–H). The TIMIT dataset contains speech from 462 speakers in training and 168 speakers in the test set, with 8 utterances for each speaker. The TIMIT training and test set are split into 8 blocks, where each block contains 2 utterances per speaker, randomly chosen. Thus each block A,B,C,D contains data from 462 speakers with 924 utterances taken from the training sets, and each block E,F,G,H contains speech from 168 test set speakers with 336 utterances. For Task a training of embeddings and the classifier is identical, namely consisting of data from blocks (A+B+C+E+F+G). The test data is the remainder, namely blocks (D+H). For Task b the training of embeddings and classifiers uses (A+B+E+F) and (C+G) respectively, while again using (D+H) for test. Task c keeps both separate: embeddings are trained on (A+B+C+D), classifiers on (E+G) and tests are conducted on (F+H). Note that H is part of all tasks, and that Task c is considerably easier as the number of speakers to separate is only 168, although training conditions are more difficult. Experimental Framework ::: Implementation For comparison the implementation, follows the convolutional model structure as deployed in BIBREF17. Both VAE encoder and decoder contain three convolutional layers and one fully-connected layer with 512 nodes. In the first layer of encoder, 1-by-80 filters are applied, and 3-by-1 filters are applied on the following two convolutional layer (strides was set to 1 in the first layer and 2 in the rest two layers). The decoder has the symmetric architecture to the encoder. Each layer is followed by a batch normalisation layerBIBREF26 except for the embedding layer, which is linear. Leaky ReLU activationBIBREF27 is used for each layer except for the embedding layer. The Adam optimiserBIBREF28 is used in training, with $\beta _1$ set to 0.95, $\beta _2$ to 0.999, and $\epsilon $ is $10^{-8}$. The initial learning rate is $10^{-3}$ Results and Discussion Table TABREF17 shows phone classification and speaker recognition results for the three model configurations: the VAE baseline, the CJFS encoder and the CJFA encoder. In our experiments the window size was set to 30 frames, namely 10 frames for the target and 10 frames for left and right neighbours, and an embedding dimension of 150. This was used for both CJFS and CJFA models alike. Results show that the CJFA encoder obtains significantly better phone classification accuracy than the VAE baseline and also than the CJFS encoder. These results are replicated for speaker recognition tasks. The CJFA encoder performs better on all tasks than the VAE baseline by a significant margin. It is noteworthy that performance on Task b is generally significantly lower than for Task a, for reasons of training overlap but also smaller training set sizes. To further explore properties of the embedding systems a change of window size ($N$) and embedding dimension ($K$) is explored. One might argue that modelling context effectively widens the input data access. Hence these experiments should explore if there is benefit in the structure beyond data size. Graphs in Fig. FIGREF14 illustrate phone classification accuracy and speaker recognition performance for all three models under variation of latent size and window sizes. It is important to note that the target window size remains the same (10 frames) with an increase of $N$. Therefore e.g. $N=70$ describes the target window size is 10 frames, and the other two neighbour windows have 30 frames at either side (30,10,30 left to right). Better speaker recognition results are consistently obtained with the CJFA encoder for any configuration with competitive performance, compared with the VAE baseline and also CJFS settings - and CJFS settings mostly outperform the baseline. However the situation for phone classification is different. It is not surprising to see CJFS perform poorly on phone classification as the target frame in not present in the input, therefore the embedding just does not have the phone segment information. However, as per speaker recognition results, speaker information is retained. A variation of the window sizes to larger windows seems detrimental in almost all cases, aside from the more difficult Task b. This may be in part the effect of the amount of training data available, however it confirms that contextual models outperform the baseline VAE model configuration, generally, and in particular also with the same amount of input data for speaker recognition. It is also noticeable that the decline or variation as a function of window size is less pronounced for the CJFA case, implying increased stability. For phone classification the trade-off benefit for window size is less clear. For phone classification, increasing the embedding $K$ is helpful, but performance remains stable at $K=150$. Hence in all of the rest of our experiments, the embedding dimension is set to 150 for all of the rest configurations. For speaker recognition the observed variations are small. A further set of experiments investigated the use of out of domain data for improving classification in a completely unsupervised setting. The RM corpus BIBREF29 was used in this case to augment the TIMIT data for training the embeddings only. All other configurations an training settings are unchanged. Table TABREF18 shows improvement after using additional out-of-domain data for training, except for in the case of CJFS and for phone classification. The improvement on all tasks with the simple addition of unlabelled audio data is remarkable. This is also true for the baseline, but the benefit of the proposed methods seems unaffected. The CJFA encoder performs better in comparison of the other two approaches and a absolute accuracy improvement of 7.9% for speaker recognition Task b is observed. The classification tasks benefits from the additional data even though the labelled data remains the same. Analysis To further evaluate the embeddings produced by the 3 models, visualisation using the t-SNE algorithmBIBREF30 is a common approach, although interpretation is sometimes difficult. Fig. FIGREF19 visualises the embeddings of phonemes in two-dimensional space, each phoneme symbol represents the mean vector of all of the embeddings belonging to the same phone classBIBREF31. One can observe that the CJFA encoder appears to generate more meaningful embeddings than the other two approaches - as phonemes belonging to the same sound classesBIBREF32 are grouped together in closer regions. The VAE baseline also has this behaviour but for example plosives are split and nasal separation seems less clear. Instead CJFS shows more confusion - as expected and explained above. Conclusion and Future Work In this paper, two unsupervised acoustic embedding approaches to model the joint latent factors between the target window and neighbouring audio segments were proposed. Models are based on variational auto-encoders, which also constitute the baseline. In order to compare against the baseline models are assessed using phone classification and speaker recognition tasks, on TIMIT, and with additional RM data. Results show CJFA (contextual joint factor analysis) encoder performs significantly better in both phone classification and speaker recognition tasks compared with other two approaches. The CJFS (contextual joint factor synthesis) encoder performs close to CJFA in speaker recognition task, but poorer for phone classification. Overall a gain of up to 3% relative on phone classification accuracy is observed, relative improvements on speaker recognition show 3–6% gain. The proposed unsupervised approaches obtain embeddings and can be improved with unlabelled out-of-domain data, the classification tasks benefits even though the labelled data remains the same. Further work needs to expand experiments on larger data sets, phone recognition and more complex neural network architectures.
What TIMIT datasets are used for testing?
26e2d4d0e482e6963a76760323b8e1c26b6eee91
What ĠTIM IT Ġdatasets Ġare Ġused Ġfor Ġtesting ?
Once Ġsplit Ġinto Ġ8 Ġsubs ets Ġ( A - H ), Ġthe Ġtest Ġset Ġused Ġare Ġblocks ĠD + H Ġand Ġblocks ĠF + H
<s> What ĠTIM IT Ġdatasets Ġare Ġused Ġfor Ġtesting ? </s> Introduction </s> In Ġrecent Ġyears , Ġword Ġembed d ings Ġhave Ġbeen Ġsuccessfully Ġused Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġthe Ġmost Ġcommonly Ġknown Ġmodels Ġare ĠWord 2 V ec ĠB IB REF 0 Ġand ĠGl ove BI B REF 1 . ĠThe Ġreasons Ġfor Ġsuch Ġsuccess Ġare Ġmanifold . ĠOne Ġkey Ġattribute Ġof Ġembed ding Ġmethods Ġis Ġthat Ġword Ġembed ding Ġmodels Ġtake Ġinto Ġaccount Ġcontext Ġinformation Ġof Ġwords , Ġthereby Ġallowing Ġa Ġmore Ġcompact Ġand Ġmanageable Ġrepresentation Ġfor Ġwords BI B REF 2 , ĠB IB REF 3 . ĠThe Ġembed d ings Ġare Ġwidely Ġapplied Ġin Ġmany Ġdownstream ĠN LP Ġtasks Ġsuch Ġas Ġneural Ġmachine Ġtranslation , Ġdialogue Ġsystem Ġor Ġtext Ġsummar isation ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , Ġas Ġwell Ġas Ġin Ġlanguage Ġmodelling Ġfor Ġspeech Ġrecognition BI B REF 7 . </s> Emb edd ings Ġof Ġacoustic Ġ( and Ġspeech ) Ġsignals Ġare Ġof Ġmore Ġrecent Ġinterest . ĠThe Ġobjective Ġis Ġto Ġrepresent Ġaudio Ġsequence Ġinformation Ġin Ġcompact Ġform , Ġreplacing Ġthe Ġraw Ġaudio Ġdata Ġwith Ġone Ġthat Ġcontains Ġlatent Ġfactors ĠB IB REF 8 , ĠB IB REF 9 . ĠThe Ġprojection Ġinto Ġsuch Ġ( lat ent ) Ġspaces Ġshould Ġtake Ġinto Ġaccount Ġdifferent Ġattributes , Ġsuch Ġas Ġphon em es , Ġspeaker Ġproperties , Ġspeaking Ġstyles , Ġthe Ġacoustic Ġbackground Ġor Ġthe Ġrecording Ġenvironment . ĠAc oustic Ġembed d ings Ġhave Ġbeen Ġexplored Ġfor Ġa Ġvariety Ġof Ġspeech Ġtasks Ġsuch Ġas Ġspeech Ġrecognition BI B REF 10 , Ġspeaker Ġverification BI B REF 11 Ġor Ġvoice Ġconversion BI B REF 12 . ĠHowever , Ġlearning Ġacoustic Ġembed d ings Ġis Ġchallenging : Ġattributes Ġmentioned Ġabove , Ġe . g . Ġspeaker Ġproperties Ġand Ġphon em es , Ġoperate Ġat Ġdifferent Ġlevels Ġof Ġabstraction Ġand Ġare Ġoften Ġstrongly Ġinter dependent , Ġand Ġtherefore Ġare Ġdifficult Ġto Ġextract Ġand Ġrepresent Ġin Ġa Ġmeaningful Ġform BI B REF 8 . </s> For Ġspeech Ġprocessing ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 Ġalso Ġuse Ġcontext Ġinformation Ġto Ġderive Ġacoustic Ġembed d ings . ĠHowever , ĠB IB REF 13 , ĠB IB REF 14 Ġfocus Ġon Ġlearning Ġword Ġsemantic Ġrepresentations Ġfrom Ġraw Ġaudio Ġinstead Ġof Ġsignal Ġproperties Ġsuch Ġas Ġphon em es Ġand Ġspeaker Ġproperties . ĠB IB REF 15 Ġfocus Ġon Ġlearning Ġspeaker Ġrepresentations Ġby Ġmodelling Ġof Ġcontext Ġinformation Ġwith Ġa ĠSi ames e Ġnetworks Ġthat Ġdiscriminate Ġwhether Ġa Ġspeech Ġsegment Ġis Ġthe Ġneighbourhood Ġof Ġa Ġtarget Ġsegment Ġor Ġnot . </s> In Ġthis Ġpaper , Ġtwo Ġun super vised Ġapproaches Ġto Ġgenerate Ġacoustic Ġembed d ings Ġusing Ġcontext Ġmodelling Ġare Ġproposed . ĠBoth Ġmethods Ġmake Ġuse Ġof Ġthe Ġvari ational Ġauto - enc oder Ġframework Ġas Ġproposed Ġin BI B REF 16 Ġand Ġboth Ġapproaches Ġaim Ġto Ġfind Ġjoint Ġlatent Ġvariables Ġbetween Ġthe Ġtarget Ġacoustic Ġsegments Ġand Ġits Ġsurrounding Ġframes . ĠIn Ġthe Ġfirst Ġinstance Ġa Ġrepresentation Ġis Ġderived Ġfrom Ġsurrounding Ġaudio Ġframes Ġthat Ġallows Ġto Ġpredict Ġcurrent Ġframe , Ġthereby Ġgenerating Ġtarget Ġaudio Ġfrom Ġcommon Ġfactors . ĠThe Ġenc oder Ġelement Ġof Ġthe Ġassociated Ġauto - enc oder Ġis Ġfurther Ġreferred Ġto Ġas ĠContext ual ĠJoint ĠFactor ĠSynt hesis Ġ( C J FS ) Ġenc oder . ĠIn Ġthe Ġsecond Ġinstance Ġan Ġaudio Ġframe Ġis Ġused Ġto Ġpredict Ġsurrounding Ġaudio , Ġwhich Ġis Ġfurther Ġreferred Ġto Ġas ĠContext ual ĠJoint ĠFactor ĠAnalysis Ġ( C J FA ) Ġencoding . ĠAs Ġshown Ġin Ġprevious Ġwork Ġvari ational Ġauto - enc od ers Ġcan Ġbe Ġused Ġto Ġderive Ġlatent Ġvariables Ġsuch Ġas Ġspeaker Ġinformation Ġand Ġphon em es BI B REF 8 Ġmore Ġrobust ly . ĠIn Ġthis Ġwork Ġit Ġis Ġshown Ġthat Ġincluding Ġtemporal Ġinformation Ġcan Ġfurther Ġimprove Ġperformance Ġand Ġrobust ness , Ġfor Ġboth Ġphon eme Ġclassification Ġand Ġspeaker Ġidentification Ġtasks . ĠFurthermore Ġthe Ġuse Ġof Ġadditional Ġunl ab elled Ġout - of - domain Ġdata Ġcan Ġimproved Ġmodelling Ġfor Ġthe Ġproposed Ġapproaches . ĠAs Ġoutlined Ġabove , Ġprior Ġwork Ġhas Ġmade Ġuse Ġof Ġsurrounding Ġaudio Ġin Ġdifferent Ġforms . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge Ġthis Ġwork Ġis Ġthe Ġfirst Ġto Ġshow Ġthat Ġpredicting Ġsurrounding Ġaudio Ġallows Ġfor Ġefficient Ġextraction Ġof Ġlatent Ġfactors Ġin Ġspeech Ġsignals . </s> The Ġrest Ġof Ġpaper Ġis Ġorganised Ġas Ġfollows : ĠIn ĠÂ§ SEC REF 2 Ġrelated Ġwork Ġis Ġdescribed , ĠMethods Ġfor Ġder iving Ġacoustic Ġembed d ings , Ġand Ġcontext Ġmodelling Ġmethods Ġin ĠN LP , Ġcomputer Ġvision Ġand Ġspeech Ġare Ġdiscussed . ĠThis Ġis Ġfollowed Ġby Ġthe Ġdescription Ġof Ġthe Ġtwo Ġapproaches Ġfor Ġmodelling Ġcontext Ġas Ġused Ġin Ġthis Ġwork , Ġin ĠÂ§ SEC REF 3 . ĠThe Ġexperimental Ġframework Ġis Ġdescribed Ġin ĠÂ§ SEC REF 4 , Ġincluding Ġthe Ġdata Ġorganisation , Ġbaseline Ġdesign Ġand Ġtask Ġdefinitions ; Ġin ĠÂ§ SEC REF 5 Ġand ĠÂ§ SEC REF 6 Ġexperiments Ġresults Ġare Ġshown Ġand Ġdiscussed . ĠThis Ġis Ġfollowed Ġby Ġthe Ġconclusions Ġand Ġfuture Ġwork Ġin ĠÂ§ SEC REF 7 . </s> Related ĠWorks </s> Related ĠWorks Ġ: :: ĠAc oustic ĠEmb edd ings </s> Most Ġinterest Ġin Ġacoustic Ġembed d ings Ġcan Ġbe Ġobserved Ġon Ġacoustic Ġword Ġembed d ings , Ġi . e . Ġprojections Ġthat Ġmap Ġword Ġac oust ics Ġinto Ġa Ġfixed Ġsize Ġvector Ġspace . ĠObjective Ġfunctions Ġare Ġchosen Ġto Ġproject Ġdifferent Ġword Ġreal isations Ġto Ġclose Ġproximity Ġin Ġthe Ġembed ding Ġspace . ĠDifferent Ġapproaches Ġwere Ġused Ġin Ġthe Ġliterature Ġ- Ġfor Ġboth Ġsupervised Ġand Ġun super vised Ġlearning . ĠFor Ġthe Ġsupervised Ġcase , ĠB IB REF 9 Ġintroduced Ġa Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) Ġbased Ġacoustic Ġword Ġembed ding Ġsystem Ġfor Ġspeech Ġrecognition , Ġwhere Ġwords Ġthat Ġsound Ġalike Ġare Ġnearby Ġin ĠEucl ide an Ġdistance . ĠIn Ġtheir Ġwork , Ġa ĠCNN Ġis Ġused Ġto Ġpredict Ġa Ġword Ġfrom Ġthe Ġcorresponding Ġacoustic Ġsignal , Ġthe Ġoutput Ġof Ġthe Ġbottleneck Ġlayer Ġbefore Ġthe Ġfinal Ġsoft max Ġlayer Ġis Ġtaken Ġto Ġbe Ġthe Ġembed ding Ġfor Ġthe Ġcorresponding Ġword . ĠFurther Ġwork Ġused Ġdifferent Ġnetwork Ġarchitectures Ġto Ġobtain Ġacoustic Ġword Ġembed d ings : ĠB IB REF 10 Ġintroduces Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġbased Ġapproach Ġinstead . </s> For Ġthe Ġcase Ġthat Ġthe Ġword Ġboundary Ġinformation Ġis Ġavailable Ġbut Ġthe Ġword Ġlabel Ġitself Ġis Ġunknown , ĠB IB REF 12 Ġproposed Ġword Ġsimilarity ĠSi ames e ĠCNN s . ĠThese Ġare Ġused Ġto Ġminim ise Ġa Ġdistance Ġfunction Ġbetween Ġrepresentations Ġof Ġtwo Ġinstances Ġof Ġthe Ġsame Ġword Ġtype Ġwhilst Ġat Ġthe Ġsame Ġtime Ġmaxim ising Ġthe Ġdistance Ġbetween Ġtwo Ġinstances Ġof Ġdifferent Ġwords . </s> Un super vised Ġapproaches Ġalso Ġexist . ĠB IB REF 17 Ġproposed Ġa Ġconv olution al Ġvari ational Ġauto - enc oder Ġbased Ġapproach Ġto Ġderive Ġacoustic Ġembed ding , Ġin Ġun super vised Ġfashion . ĠThe Ġauthors Ġchose Ġphon eme Ġand Ġspeaker Ġclassification Ġtasks Ġon ĠTIM IT Ġdata Ġto Ġassess Ġthe Ġquality Ġof Ġtheir Ġembed d ings Ġ- Ġan Ġapproach Ġreplicated Ġin Ġthe Ġwork Ġpresented Ġin Ġthis Ġpaper . ĠB IB REF 8 , ĠB IB REF 18 Ġproposed Ġan Ġapproach Ġcalled Ġfactor ised Ġhierarchical Ġvari ational Ġauto - enc oder . ĠThe Ġwork Ġintroduces Ġthe Ġconcepts Ġof Ġglobal Ġand Ġlocal Ġlatent Ġfactors , Ġi . e . Ġlatent Ġvariables Ġthat Ġare Ġshared Ġon Ġthe Ġcomplete Ġutter ance , Ġor Ġlatent Ġvariables Ġthat Ġchange Ġwithin Ġthe Ġsequence , Ġrespectively . ĠResults Ġare Ġagain Ġobtained Ġusing Ġthe Ġsame Ġdata Ġand Ġtasks Ġas Ġabove . </s> Related ĠWorks Ġ: :: ĠContext ĠMod elling </s> Context Ġinformation Ġplays Ġa Ġfundamental Ġrole Ġin Ġspeech Ġprocessing . ĠPh on em es Ġcould Ġbe Ġinfluenced Ġby Ġsurrounding Ġframes Ġthrough Ġco art ic ulation BI B REF 19 Ġ- Ġan Ġeffect Ġcaused Ġby Ġspeed Ġlimitations Ġand Ġtransitions Ġin Ġthe Ġmovement Ġof Ġartic ulators . ĠNormally Ġdirectly Ġneighbouring Ġphon em es Ġhave Ġimportant Ġimpact Ġon Ġthe Ġsound Ġreal isation . ĠIn versely , Ġthe Ġsurrounding Ġphon em es Ġalso Ġprovide Ġstrong Ġconstraints Ġon Ġthe Ġphon eme Ġthat Ġcan Ġbe Ġchosen Ġat Ġany Ġgiven Ġpoint , Ġsubject Ġto Ġto Ġlex ical Ġand Ġlanguage Ġconstraints . ĠThis Ġeffect Ġis Ġfor Ġexample Ġexploited Ġin Ġphon eme Ġrecognition , Ġby Ġuse Ġof Ġphon eme Ġ$ n $ - gram Ġmodels BI B REF 20 . ĠEqu ival ently Ġinter Ġword Ġdependency Ġ- Ġderived Ġfrom Ġlinguistic Ġconstraints Ġ- Ġcan Ġbe Ġexploited , Ġas Ġis Ġthe Ġcase Ġin Ġcomputing Ġword Ġembed d ings Ġwith Ġthe Ġaforementioned Ġword 2 vec BI B REF 0 Ġmethod . ĠThe Ġsituation Ġdiffers Ġfor Ġthe Ġglobal Ġlatent Ġvariables , Ġsuch Ġas Ġspeaker Ġproperties Ġor Ġacoustic Ġenvironment Ġinformation . ĠSpeaker Ġproperties Ġremains Ġconstant Ġ- Ġand Ġenvironments Ġcan Ġalso Ġbe Ġassumed Ġstationary Ġover Ġlonger Ġperiods Ġof Ġtime . ĠHence Ġthese Ġvariables Ġare Ġcommon Ġbetween Ġamong Ġneighbouring Ġframes Ġand Ġwindows . ĠMod elling Ġcontext Ġinformation Ġis Ġhelpful Ġfor Ġidentifying Ġsuch Ġinformation ĠB IB REF 21 . </s> There Ġis Ġsignificant Ġprior Ġwork Ġthat Ġtakes Ġsurrounding Ġinformation Ġinto Ġaccount Ġto Ġlearn Ġvector Ġrepresentations . ĠFor Ġtext Ġprocessing Ġthe ĠWord 2 V ec BI B REF 0 Ġmodel Ġdirectly Ġpredicts Ġthe Ġneighbouring Ġwords Ġfrom Ġtarget Ġwords Ġor Ġin versely . ĠThis Ġhelps Ġto Ġcapture Ġthe Ġmeanings Ġof Ġwords BI B REF 2 . ĠIn Ġcomputer Ġvision , ĠB IB REF 22 Ġintroduced Ġan Ġvisual Ġfeature Ġlearning Ġapproach Ġcalled Ġcontext Ġenc oder Ġ, which Ġis Ġbased Ġon Ġcontext Ġbased Ġpixel Ġprediction . ĠTheir Ġmodel Ġis Ġtrained Ġto Ġgenerate Ġthe Ġcontents Ġof Ġan Ġimage Ġregion Ġfrom Ġits Ġsurroundings . ĠIn Ġspeech Ġprocessing ĠB IB REF 13 , ĠB IB REF 14 Ġproposed Ġa Ġsequence Ġto Ġsequence Ġapproach Ġto Ġpredict Ġsurrounding Ġsegments Ġof Ġa Ġtarget Ġsegment . ĠHowever , Ġthe Ġapproach Ġagain Ġaims Ġat Ġcapturing Ġword Ġsemantics Ġfrom Ġraw Ġspeech Ġaudio , Ġwords Ġhas Ġsimilar Ġsemantic Ġmeanings Ġare Ġnearby Ġin ĠEucl ide an Ġdistance . ĠB IB REF 15 Ġproposed Ġan Ġun super vised Ġacoustic Ġembed ding Ġapproach . ĠIn Ġtheir Ġapproach , Ġinstead Ġof Ġdirectly Ġestimating Ġthe Ġneighbourhood Ġframes Ġof Ġa Ġtarget Ġsegment , Ġa ĠSi ames e Ġarchitecture Ġis Ġused Ġto Ġdiscriminate Ġwhether Ġa Ġspeech Ġsegment Ġis Ġin Ġthe Ġneighbourhood Ġof Ġa Ġtarget Ġsegment Ġor Ġnot . ĠFurthermore , Ġtheir Ġapproach Ġonly Ġaims Ġat Ġembed ding Ġof Ġspeaker Ġproperties . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwork Ġpresented Ġhere Ġis Ġthe Ġfirst Ġderive Ġphon eme Ġand Ġspeaker Ġrepresentations Ġby Ġtemporal Ġcontext Ġprediction Ġusing Ġacoustic Ġdata . </s> Model ĠArchitecture Ġ: :: ĠVari ational ĠAuto - En cod ers </s> As Ġshown Ġin ĠB IB REF 17 , Ġvari ational Ġauto - enc od ers Ġ( VA E ) BI B REF 8 Ġcan Ġyield Ġgood Ġrepresentations Ġin Ġthe Ġlatent Ġspace . ĠOne Ġof Ġthe Ġbenefits Ġis Ġthat Ġthe Ġmodels Ġallow Ġto Ġwork Ġwith Ġthe Ġlatent Ġdistributions BI B REF 23 , ĠB IB REF 8 , ĠB IB REF 24 . ĠIn Ġthis Ġwork , ĠVA E Ġis Ġused Ġto Ġmodel Ġthe Ġjoint Ġlatent Ġfactors Ġbetween Ġthe Ġtarget Ġsegments Ġand Ġits Ġsurroundings . </s> Different Ġfrom Ġnormal Ġauto - enc od ers , Ġwhere Ġthe Ġinput Ġdata Ġis Ġcompressed Ġinto Ġlatent Ġcode Ġwhich Ġis Ġa Ġpoint Ġestimation Ġof Ġlatent Ġvariables BI B REF 16 , Ġthe Ġvari ational Ġauto - enc oder Ġmodel Ġdefines Ġa Ġprob abil istic Ġgener ative Ġprocess Ġbetween Ġthe Ġobservation Ġ$ x $ Ġand Ġthe Ġlatent Ġvariable Ġ$ z $. ĠAt Ġthe Ġenc oder Ġstep , Ġthe Ġenc oder Ġprovides Ġan Ġestimation Ġof Ġthe Ġlatent Ġvariable Ġ$ z $ Ġgiven Ġobservation Ġ$ x $ Ġas Ġ$ p ( z | x ) $. ĠThe Ġdec oder Ġfinds Ġthe Ġmost Ġlikely Ġreconstruction Ġ$\ hat { x } $ Ġsubject Ġto Ġ$ p (\ hat { x } | z ) $. ĠThe Ġlatent Ġvariable Ġestimation Ġ$ p ( z | x ) $, Ġor Ġthe Ġprobability Ġdensity Ġfunction Ġthereof , Ġhas Ġmany Ġinterpretations , Ġsimply Ġas Ġencoding , Ġor Ġas Ġlatent Ġstate Ġspace Ġgoverning Ġthe Ġconstruction Ġof Ġthe Ġoriginal Ġsignal . </s> Com puting Ġ$ p ( z | x ) $ Ġrequires Ġan Ġestimate Ġof Ġthe Ġmarginal Ġlikelihood Ġ$ p ( x ) $ Ġwhich Ġis Ġdifficult Ġto Ġobtain Ġin Ġpractice . ĠA Ġrecognition Ġmodel Ġ$ q ( z | x ) $ Ġis Ġused Ġto Ġapproximate Ġ$ p ( z | x ) $ ĠKL Ġdivergence Ġbetween Ġ$ p ( z | x ) $ Ġand Ġ$ q ( z | x ) $, Ġas Ġshown Ġin ĠE q ĠDIS PLAY _ FORM 4 , Ġis Ġminim ised BI B REF 16 . </s> From ĠE q ĠDIS PLAY _ FORM 4 , Ġthe Ġobjective Ġfunction Ġfor ĠVA E Ġtraining Ġis Ġderived Ġshown Ġin ĠE q ĠDIS PLAY _ FORM 5 : ĠB IB REF 16 , ĠB IB REF 17 </s> where Ġ$ E _{ q ( z | x )} log [ p ( x | z )] $ Ġis Ġalso Ġcalled Ġthe Ġreconstruction Ġlikelihood Ġand Ġ$ ĠD _{ K L } ( q ( z | x )| | p ( z )) $ Ġensures Ġthe Ġlearned Ġdistribution Ġ$ q ( z | x ) $ Ġis Ġclose Ġto Ġprior Ġdistribution Ġ$ p ( z ) $. </s> Model ĠArchitecture Ġ: :: ĠProp osed ĠModel ĠArchitecture </s> An Ġaudio Ġsignal Ġis Ġrepresented Ġsequence Ġof Ġfeature Ġvectors Ġ$ S = \ l brace ĠS _ 1 , S _ 2 ,... S _ T \ r brace Ġ$ , Ġwhere Ġ$ T $ Ġis Ġthe Ġlength Ġof Ġthe Ġutter ance . ĠIn Ġthe Ġproposed Ġmethod Ġthe Ġconcept Ġof Ġa Ġtarget Ġwindow Ġis Ġused , Ġto Ġwhich Ġand Ġembed ding Ġis Ġrelated . ĠA Ġtarget Ġwindow Ġ$ X _ t $ Ġis Ġa Ġsegment Ġof Ġspeech Ġrepresenting Ġfeatures Ġfrom Ġ$ S _ t $ Ġto Ġ$ S _{ t + C - 1 } $, Ġwhere Ġ$ t Ġ\ in Ġ\ l brace Ġ1 , 2 ,... T - C + 1 \ r brace Ġ$ Ġand Ġ$ C $ Ġdenotes Ġthe Ġtarget Ġwindow Ġsize . ĠThe Ġleft Ġneighbour Ġwindow Ġof Ġthe Ġtarget Ġwindow Ġis Ġdefined Ġas Ġthe Ġsegment Ġbetween Ġ$ S _{ t - N } $ Ġand Ġ$ S _{ t - 1 } $, Ġand Ġthe Ġsegment Ġbetween Ġ$ S _{ t + C } $ Ġand Ġ$ S _{ t + C + N - 1 } $ Ġrepresents Ġthe Ġright Ġneighbour Ġwindow Ġof Ġthe Ġtarget Ġwindow , Ġwith Ġ$ N $ Ġbeing Ġthe Ġsingle Ġsided Ġneighbour Ġwindow Ġsize . ĠThe Ġconc aten ation Ġof Ġleft Ġand Ġright Ġneighbour Ġsegments Ġis Ġfurther Ġreferred Ġto Ġ$ Y _ t $. ĠThe Ġproposed Ġapproach Ġaims Ġto Ġfind Ġjoint Ġlatent Ġfactors Ġbetween Ġtarget Ġwindow Ġsegment Ġ$ X _ t $ Ġand Ġthe Ġconc aten ation Ġof Ġleft Ġand Ġright Ġneighbour Ġwindow Ġsegments Ġ$ Y _ t $, Ġfor Ġall Ġsegments . ĠFor Ġconvenience Ġthe Ġsubscript Ġ$ t $ Ġis Ġdropped Ġin Ġfollowing Ġderiv ations Ġwhere Ġappropriate . ĠTwo Ġdifferent Ġcontext Ġuse Ġconfigurations Ġcan Ġbe Ġused . </s> Figure ĠFIG REF 7 Ġillustrate Ġthese Ġtwo Ġapproaches . ĠThe Ġaudio Ġsignal Ġis Ġsplit Ġinto Ġa Ġsequence Ġof Ġleft Ġneighbour Ġsegment , Ġtarget Ġsegment Ġand Ġright Ġneighbour Ġsegment . ĠIn Ġthe Ġfirst Ġapproach Ġ( left Ġside Ġon Ġfigure ĠFIG REF 7 ), Ġthe Ġconc aten ation Ġof Ġthe Ġleft Ġneighbour Ġsegment Ġand Ġright Ġneighbour Ġsegment Ġ($ Y $ ) Ġis Ġinput Ġto Ġa ĠVA E Ġmodel BI B REF 16 , Ġand Ġtarget Ġwindow Ġ($ X $ ) Ġis Ġpredicted . ĠIn Ġthe Ġsecond Ġapproach Ġ( right Ġside Ġon Ġfigure ĠFIG REF 7 ) Ġthe Ġtarget Ġwindow Ġ($ X $ ) Ġis Ġthe Ġinput Ġto Ġa ĠVA E Ġmodel , Ġand Ġneighbour Ġwindow Ġ($ Y $ ) Ġis Ġpredicted . </s> The Ġfirst Ġapproach Ġis Ġreferred Ġto Ġas Ġthe Ġcontextual Ġjoint Ġfactor Ġsynthesis Ġenc oder Ġas Ġit Ġaims Ġto Ġsynthes ise Ġthe Ġtarget Ġframe Ġ$ X $. ĠOnly Ġfactors Ġcommon Ġbetween Ġinput Ġand Ġoutput Ġcan Ġform Ġthe Ġbasis Ġfor Ġsuch Ġprediction , Ġand Ġthe Ġencoded Ġembed ding Ġcan Ġbe Ġconsidered Ġa Ġrepresentation Ġof Ġthese Ġjoint Ġfactors . ĠSimilar Ġto Ġthe Ġstandard ĠVA E Ġformulations , Ġthe Ġobjective Ġfunction Ġof ĠCJ FS Ġis Ġgiven Ġin ĠE q . ĠDIS PLAY _ FORM 8 : </s> The Ġfirst Ġterm Ġrepresents Ġthe Ġreconstruction Ġlikelihood Ġbetween Ġpredicted Ġtarget Ġwindow Ġsegments Ġand Ġthe Ġneighbour Ġwindow Ġsegments , Ġand Ġthe Ġsecond Ġterm Ġdenotes Ġhow Ġsimilar Ġthe Ġlearned Ġdistribution Ġ$ q ( z | Y ) $ Ġis Ġto Ġthe Ġprior Ġdistribution Ġof Ġ$ z $, Ġ$ p ( z ) $ </s> In Ġpractice , Ġfor Ġthe Ġreconstruction Ġterm Ġcan Ġbe Ġbased Ġon Ġthe Ġmean Ġsquared Ġerror Ġ( M SE ) Ġbetween Ġthe Ġtrue Ġtarget Ġsegment Ġand Ġthe Ġpredicted Ġtarget Ġsegment . ĠFor Ġthe Ġsecond Ġterm Ġin ĠE q ĠDIS PLAY _ FORM 8 , Ġsamples Ġfor Ġ$ p ( z ) $ Ġare Ġobtained Ġfrom ĠGa ussian Ġdistribution Ġwith Ġzero Ġmean Ġand Ġa Ġvariance Ġof Ġone Ġ($ p ( z ) Ġ\ sim Ġ\ math cal Ġ{ N } Ġ( 0 , 1 ) $ ). </s> The Ġsecond Ġapproach Ġis Ġthe Ġcontextual Ġjoint Ġfactor Ġanalysis Ġenc oder . ĠThe Ġobjective Ġis Ġto Ġpredict Ġthe Ġtemporal Ġcontext Ġ$ Y $ Ġbased Ġon Ġinput Ġfrom Ġa Ġsingle Ġcentre Ġsegment Ġ$ X $. ĠAgain Ġjoint Ġfactors Ġbetween Ġthe Ġthree Ġwindows Ġare Ġobtained , Ġand Ġencoded Ġin Ġan Ġembed ding . ĠHowever Ġthis Ġtime Ġan Ġanalysis Ġof Ġone Ġsegment Ġis Ġenough . ĠNaturally Ġthe Ġtraining Ġobjective Ġfunction Ġof ĠCJ FA Ġis Ġrepresented Ġby Ġchange Ġof Ġvariables , Ġas Ġgiven Ġin ĠE q ĠDIS PLAY _ FORM 9 . </s> Exper imental ĠFramework </s> Exper imental ĠFramework Ġ: :: ĠData Ġand ĠUse </s> Taking Ġthe ĠVA E Ġexperiments Ġas Ġbaseline , Ġthe ĠTIM IT Ġdata Ġis Ġused Ġfor Ġthis Ġwork BI B REF 25 . ĠTIM IT Ġcontains Ġstudio Ġrecordings Ġfrom Ġa Ġlarge Ġnumber Ġof Ġspeakers Ġwith Ġdetailed Ġphon eme Ġsegment Ġinformation . ĠWork Ġin Ġthis Ġpaper Ġmakes Ġuse Ġof Ġthe Ġofficial Ġtraining Ġand Ġtest Ġsets , Ġcovering Ġin Ġtotal Ġ630 Ġspeakers Ġwith Ġ8 Ġutter ances Ġeach . ĠThere Ġis Ġno Ġspeaker Ġoverlap Ġbetween Ġtraining Ġand Ġtest Ġset , Ġwhich Ġcomprise Ġof Ġ4 62 Ġand Ġ168 Ġspeakers , Ġrespectively . ĠAll Ġwork Ġpresented Ġhere Ġuse Ġof Ġ80 Ġdimensional ĠMel - scale Ġfilter Ġbank Ġcoefficients . </s> Exper imental ĠFramework Ġ: :: ĠBas elines </s> Work Ġon ĠVA E Ġin ĠB IB REF 17 Ġto Ġlearn Ġacoustic Ġembed d ings Ġconducted Ġexperiments Ġusing Ġthe ĠTIM IT Ġdata Ġset . ĠIn Ġparticular Ġthe Ġtasks Ġof Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġwhere Ġchosen . ĠAs Ġwork Ġhere Ġis Ġan Ġextension Ġof Ġsuch Ġwork Ġwe Ġwe Ġfollow Ġthe Ġexperimentation , Ġhowever Ġwith Ġsignificant Ġextensions Ġ( see ĠSection ĠSEC REF 13 ). ĠWith Ġguidance Ġfrom Ġthe Ġauthors Ġof Ġthe Ġoriginal Ġwork BI B REF 17 Ġour Ġown Ġimplementation Ġof ĠVA E Ġwas Ġcreated Ġand Ġcompared Ġwith Ġthe Ġpublished Ġperformance Ġ- Ġyielding Ġnear Ġidentical Ġresults . ĠThis Ġimplementation Ġthen Ġwas Ġalso Ġused Ġas Ġthe Ġbasis Ġfor ĠCJ FS Ġand ĠCJ FA , Ġas Ġintroduced Ġin ĠÂ§ ĠSEC REF 6 . </s> For Ġthe Ġassessment Ġof Ġembedded Ġvector Ġquality Ġour Ġwork Ġalso Ġfollows Ġthe Ġsame Ġtask Ġtypes , Ġnamely Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġ( details Ġin ĠÂ§ SEC REF 13 ), Ġwith Ġidentical Ġtask Ġimplementations Ġas Ġin Ġthe Ġreference Ġpaper . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġphone Ġclassification Ġdiffers Ġfrom Ġthe Ġwidely Ġreported Ġphone Ġrecognition Ġexperiments Ġon ĠTIM IT . ĠClassification Ġuses Ġphone Ġboundaries Ġwhich Ġare Ġassumed Ġto Ġbe Ġknown . ĠHowever , Ġno Ġcontextual Ġinformation Ġis Ġavailable , Ġwhich Ġis Ġtypically Ġused Ġin Ġthe Ġrecognition Ġsetups , Ġby Ġmeans Ġof Ġtri phone Ġmodels , Ġor Ġbig ram Ġlanguage Ġmodels . ĠTherefore Ġthe Ġtask Ġis Ġoften Ġmore Ġdifficult Ġthan Ġrecognition . ĠThe Ġbaseline Ġperformance Ġfor ĠVA E Ġbased Ġphone Ġclassification Ġexperiments Ġin ĠB IB REF 17 Ġreport Ġan Ġaccuracy Ġof Ġ72 . 2 %. ĠThe Ġre - im plementation Ġforming Ġthe Ġbasis Ġfor Ġour Ġwork Ġgave Ġan Ġaccuracy Ġof Ġ72 . 0 %, Ġa Ġresult Ġthat Ġwas Ġconsidered Ġto Ġprovide Ġa Ġcredible Ġbasis Ġfor Ġfurther Ġwork . </s> For Ġthe Ġpurpose Ġof Ġspeaker Ġrecognition Ġit Ġis Ġimportant Ġto Ġtake Ġinto Ġaccount Ġthe Ġoverlap Ġbetween Ġtraining Ġand Ġtesting . ĠThus Ġthree Ġdifferent Ġtask Ġconfigurations Ġare Ġconsidered , Ġdifferent Ġto Ġthe Ġsetting Ġin ĠB IB REF 17 . ĠTheir Ġbaseline Ġwill Ġbe Ġfurther Ġreferred Ġas ĠVA E Ġbaseline . </s> Exper imental ĠFramework Ġ: :: ĠEvaluation ĠT asks </s> The Ġphone Ġclassification Ġimplementation Ġoperates Ġon Ġsegment Ġlevel , Ġusing Ġa Ġconv olution al Ġnetwork Ġto Ġobtain Ġframe Ġby Ġframe Ġposter iors Ġwhich Ġare Ġthen Ġaccumulated Ġfor Ġsegment Ġdecision Ġ( assuming Ġframe Ġindependence ). ĠThe Ġphone Ġclass Ġwith Ġthe Ġhighest Ġsegment Ġposterior Ġis Ġchosen Ġas Ġoutput . ĠAn Ġidentical Ġapproach Ġis Ġused Ġfor Ġspeaker Ġrecognition . ĠIn Ġthis Ġsetting Ġ3 Ġdifferent Ġdata Ġsets Ġare Ġrequired : Ġa Ġtraining Ġset Ġfor Ġlearning Ġthe Ġenc oder Ġmodels , Ġa Ġtraining Ġset Ġfor Ġlearning Ġthe Ġclassification Ġmodel , Ġand Ġan Ġevaluation Ġtest Ġset . ĠFor Ġthe Ġphone Ġclassification Ġtask , Ġboth Ġembed ding Ġand Ġclassification Ġmodels Ġare Ġtrained Ġon Ġthe Ġofficial ĠTIM IT Ġtraining Ġset , Ġand Ġmakes Ġuse Ġof Ġthe Ġprovided Ġphone Ġboundary Ġinformation . ĠA Ġfixed Ġsize Ġwindow Ġwith Ġa Ġframe Ġstep Ġsize Ġof Ġone Ġframe Ġis Ġused Ġfor Ġall Ġmodel Ġtraining . ĠAs Ġnoted , Ġphone Ġclassification Ġmakes Ġno Ġuse Ġof Ġphone Ġcontext , Ġand Ġno Ġlanguage Ġmodel Ġis Ġapplied . </s> For Ġspeaker Ġrecognition Ġoverlap Ġspeaker Ġbetween Ġany Ġof Ġthe Ġdatasets Ġ( training Ġembed d ings , Ġtraining Ġclass ifier Ġand Ġtest ) Ġwill Ġcause Ġa Ġbias . ĠThree Ġdifferent Ġconfigurations Ġ( T asks Ġa , b , c ) Ġare Ġused Ġto Ġassess Ġthis Ġbias . ĠTask Ġa Ġreflects Ġthe Ġsituation Ġwhere Ġboth Ġclass ifier Ġand Ġembed ding Ġare Ġtrained Ġon Ġthe Ġsame Ġdata . ĠAs Ġthe Ġtask Ġis Ġto Ġdetect Ġa Ġspeaker Ġthe Ġspeakers Ġpresent Ġin Ġthe Ġtest Ġset Ġneed Ġto Ġbe Ġpresent Ġin Ġtraining . ĠTask Ġb Ġrepresents Ġa Ġsituation Ġwhere Ġclass ifier Ġand Ġembed ding Ġare Ġtrained Ġon Ġindependent Ġdata Ġsets , Ġbut Ġwith Ġspeaker Ġoverlap . ĠFinally ĠTask Ġc Ġrepresents Ġcomplete Ġindependence Ġin Ġtraining Ġdata Ġsets Ġand Ġno Ġspeaker Ġoverlap . ĠTable ĠT AB REF 15 Ġsummar ises Ġthe Ġrelationships . </s> In Ġorder Ġto Ġachieve Ġthese Ġconfiguration Ġthe ĠTIM IT Ġdata Ġwas Ġsplit . ĠFig . ĠFIG REF 12 Ġillustrates Ġthe Ġsplit Ġof Ġthe Ġdata Ġinto Ġ8 Ġsubs ets Ġ( A âĢĵ H ). ĠThe ĠTIM IT Ġdataset Ġcontains Ġspeech Ġfrom Ġ4 62 Ġspeakers Ġin Ġtraining Ġand Ġ168 Ġspeakers Ġin Ġthe Ġtest Ġset , Ġwith Ġ8 Ġutter ances Ġfor Ġeach Ġspeaker . ĠThe ĠTIM IT Ġtraining Ġand Ġtest Ġset Ġare Ġsplit Ġinto Ġ8 Ġblocks , Ġwhere Ġeach Ġblock Ġcontains Ġ2 Ġutter ances Ġper Ġspeaker , Ġrandomly Ġchosen . ĠThus Ġeach Ġblock ĠA , B , C , D Ġcontains Ġdata Ġfrom Ġ4 62 Ġspeakers Ġwith Ġ9 24 Ġutter ances Ġtaken Ġfrom Ġthe Ġtraining Ġsets , Ġand Ġeach Ġblock ĠE , F , G , H Ġcontains Ġspeech Ġfrom Ġ168 Ġtest Ġset Ġspeakers Ġwith Ġ336 Ġutter ances . </s> For ĠTask Ġa Ġtraining Ġof Ġembed d ings Ġand Ġthe Ġclass ifier Ġis Ġidentical , Ġnamely Ġconsisting Ġof Ġdata Ġfrom Ġblocks Ġ( A + B + C + E + F + G ). ĠThe Ġtest Ġdata Ġis Ġthe Ġremainder , Ġnamely Ġblocks Ġ( D + H ). ĠFor ĠTask Ġb Ġthe Ġtraining Ġof Ġembed d ings Ġand Ġclass ifiers Ġuses Ġ( A + B + E + F ) Ġand Ġ( C + G ) Ġrespectively , Ġwhile Ġagain Ġusing Ġ( D + H ) Ġfor Ġtest . ĠTask Ġc Ġkeeps Ġboth Ġseparate : Ġembed d ings Ġare Ġtrained Ġon Ġ( A + B + C + D ), Ġclass ifiers Ġon Ġ( E + G ) Ġand Ġtests Ġare Ġconducted Ġon Ġ( F + H ). ĠNote Ġthat ĠH Ġis Ġpart Ġof Ġall Ġtasks , Ġand Ġthat ĠTask Ġc Ġis Ġconsiderably Ġeasier Ġas Ġthe Ġnumber Ġof Ġspeakers Ġto Ġseparate Ġis Ġonly Ġ168 , Ġalthough Ġtraining Ġconditions Ġare Ġmore Ġdifficult . </s> Exper imental ĠFramework Ġ: :: ĠImplementation </s> For Ġcomparison Ġthe Ġimplementation , Ġfollows Ġthe Ġconv olution al Ġmodel Ġstructure Ġas Ġdeployed Ġin ĠB IB REF 17 . ĠBoth ĠVA E Ġenc oder Ġand Ġdec oder Ġcontain Ġthree Ġconv olution al Ġlayers Ġand Ġone Ġfully - connected Ġlayer Ġwith Ġ512 Ġnodes . ĠIn Ġthe Ġfirst Ġlayer Ġof Ġenc oder , Ġ1 - by - 80 Ġfilters Ġare Ġapplied , Ġand Ġ3 - by - 1 Ġfilters Ġare Ġapplied Ġon Ġthe Ġfollowing Ġtwo Ġconv olution al Ġlayer Ġ( str ides Ġwas Ġset Ġto Ġ1 Ġin Ġthe Ġfirst Ġlayer Ġand Ġ2 Ġin Ġthe Ġrest Ġtwo Ġlayers ). ĠThe Ġdec oder Ġhas Ġthe Ġsymm etric Ġarchitecture Ġto Ġthe Ġenc oder . ĠEach Ġlayer Ġis Ġfollowed Ġby Ġa Ġbatch Ġnormal isation Ġlayer BI B REF 26 Ġexcept Ġfor Ġthe Ġembed ding Ġlayer , Ġwhich Ġis Ġlinear . ĠLe aky ĠRe LU Ġactivation BI B REF 27 Ġis Ġused Ġfor Ġeach Ġlayer Ġexcept Ġfor Ġthe Ġembed ding Ġlayer . ĠThe ĠAdam Ġoptim iser BI B REF 28 Ġis Ġused Ġin Ġtraining , Ġwith Ġ$\ beta Ġ_ 1 $ Ġset Ġto Ġ0 . 95 , Ġ$\ beta Ġ_ 2 $ Ġto Ġ0 . 999 , Ġand Ġ$\ ep sil on Ġ$ Ġis Ġ$ 10 ^{ - 8 } $. ĠThe Ġinitial Ġlearning Ġrate Ġis Ġ$ 10 ^{ - 3 } $ </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 17 Ġshows Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġresults Ġfor Ġthe Ġthree Ġmodel Ġconfigurations : Ġthe ĠVA E Ġbaseline , Ġthe ĠCJ FS Ġenc oder Ġand Ġthe ĠCJ FA Ġenc oder . ĠIn Ġour Ġexperiments Ġthe Ġwindow Ġsize Ġwas Ġset Ġto Ġ30 Ġframes , Ġnamely Ġ10 Ġframes Ġfor Ġthe Ġtarget Ġand Ġ10 Ġframes Ġfor Ġleft Ġand Ġright Ġneighbours , Ġand Ġan Ġembed ding Ġdimension Ġof Ġ150 . ĠThis Ġwas Ġused Ġfor Ġboth ĠCJ FS Ġand ĠCJ FA Ġmodels Ġalike . ĠResults Ġshow Ġthat Ġthe ĠCJ FA Ġenc oder Ġob tains Ġsignificantly Ġbetter Ġphone Ġclassification Ġaccuracy Ġthan Ġthe ĠVA E Ġbaseline Ġand Ġalso Ġthan Ġthe ĠCJ FS Ġenc oder . ĠThese Ġresults Ġare Ġreplicated Ġfor Ġspeaker Ġrecognition Ġtasks . ĠThe ĠCJ FA Ġenc oder Ġperforms Ġbetter Ġon Ġall Ġtasks Ġthan Ġthe ĠVA E Ġbaseline Ġby Ġa Ġsignificant Ġmargin . ĠIt Ġis Ġnoteworthy Ġthat Ġperformance Ġon ĠTask Ġb Ġis Ġgenerally Ġsignificantly Ġlower Ġthan Ġfor ĠTask Ġa , Ġfor Ġreasons Ġof Ġtraining Ġoverlap Ġbut Ġalso Ġsmaller Ġtraining Ġset Ġsizes . </s> To Ġfurther Ġexplore Ġproperties Ġof Ġthe Ġembed ding Ġsystems Ġa Ġchange Ġof Ġwindow Ġsize Ġ($ N $ ) Ġand Ġembed ding Ġdimension Ġ($ K $ ) Ġis Ġexplored . ĠOne Ġmight Ġargue Ġthat Ġmodelling Ġcontext Ġeffectively Ġwid ens Ġthe Ġinput Ġdata Ġaccess . ĠHence Ġthese Ġexperiments Ġshould Ġexplore Ġif Ġthere Ġis Ġbenefit Ġin Ġthe Ġstructure Ġbeyond Ġdata Ġsize . ĠGraph s Ġin ĠFig . ĠFIG REF 14 Ġillustrate Ġphone Ġclassification Ġaccuracy Ġand Ġspeaker Ġrecognition Ġperformance Ġfor Ġall Ġthree Ġmodels Ġunder Ġvariation Ġof Ġlatent Ġsize Ġand Ġwindow Ġsizes . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġthe Ġtarget Ġwindow Ġsize Ġremains Ġthe Ġsame Ġ( 10 Ġframes ) Ġwith Ġan Ġincrease Ġof Ġ$ N $. ĠTherefore Ġe . g . Ġ$ N = 70 $ Ġdescribes Ġthe Ġtarget Ġwindow Ġsize Ġis Ġ10 Ġframes , Ġand Ġthe Ġother Ġtwo Ġneighbour Ġwindows Ġhave Ġ30 Ġframes Ġat Ġeither Ġside Ġ( 30 , 10 , 30 Ġleft Ġto Ġright ). ĠBetter Ġspeaker Ġrecognition Ġresults Ġare Ġconsistently Ġobtained Ġwith Ġthe ĠCJ FA Ġenc oder Ġfor Ġany Ġconfiguration Ġwith Ġcompetitive Ġperformance , Ġcompared Ġwith Ġthe ĠVA E Ġbaseline Ġand Ġalso ĠCJ FS Ġsettings Ġ- Ġand ĠCJ FS Ġsettings Ġmostly Ġoutper form Ġthe Ġbaseline . ĠHowever Ġthe Ġsituation Ġfor Ġphone Ġclassification Ġis Ġdifferent . ĠIt Ġis Ġnot Ġsurprising Ġto Ġsee ĠCJ FS Ġperform Ġpoorly Ġon Ġphone Ġclassification Ġas Ġthe Ġtarget Ġframe Ġin Ġnot Ġpresent Ġin Ġthe Ġinput , Ġtherefore Ġthe Ġembed ding Ġjust Ġdoes Ġnot Ġhave Ġthe Ġphone Ġsegment Ġinformation . ĠHowever , Ġas Ġper Ġspeaker Ġrecognition Ġresults , Ġspeaker Ġinformation Ġis Ġretained . </s> A Ġvariation Ġof Ġthe Ġwindow Ġsizes Ġto Ġlarger Ġwindows Ġseems Ġdetrimental Ġin Ġalmost Ġall Ġcases , Ġaside Ġfrom Ġthe Ġmore Ġdifficult ĠTask Ġb . ĠThis Ġmay Ġbe Ġin Ġpart Ġthe Ġeffect Ġof Ġthe Ġamount Ġof Ġtraining Ġdata Ġavailable , Ġhowever Ġit Ġconfirms Ġthat Ġcontextual Ġmodels Ġoutper form Ġthe Ġbaseline ĠVA E Ġmodel Ġconfiguration , Ġgenerally , Ġand Ġin Ġparticular Ġalso Ġwith Ġthe Ġsame Ġamount Ġof Ġinput Ġdata Ġfor Ġspeaker Ġrecognition . ĠIt Ġis Ġalso Ġnoticeable Ġthat Ġthe Ġdecline Ġor Ġvariation Ġas Ġa Ġfunction Ġof Ġwindow Ġsize Ġis Ġless Ġpronounced Ġfor Ġthe ĠCJ FA Ġcase , Ġimplying Ġincreased Ġstability . ĠFor Ġphone Ġclassification Ġthe Ġtrade - off Ġbenefit Ġfor Ġwindow Ġsize Ġis Ġless Ġclear . </s> For Ġphone Ġclassification , Ġincreasing Ġthe Ġembed ding Ġ$ K $ Ġis Ġhelpful , Ġbut Ġperformance Ġremains Ġstable Ġat Ġ$ K = 150 $. ĠHence Ġin Ġall Ġof Ġthe Ġrest Ġof Ġour Ġexperiments , Ġthe Ġembed ding Ġdimension Ġis Ġset Ġto Ġ150 Ġfor Ġall Ġof Ġthe Ġrest Ġconfigurations . ĠFor Ġspeaker Ġrecognition Ġthe Ġobserved Ġvariations Ġare Ġsmall . </s> A Ġfurther Ġset Ġof Ġexperiments Ġinvestigated Ġthe Ġuse Ġof Ġout Ġof Ġdomain Ġdata Ġfor Ġimproving Ġclassification Ġin Ġa Ġcompletely Ġun super vised Ġsetting . ĠThe ĠRM Ġcorpus ĠB IB REF 29 Ġwas Ġused Ġin Ġthis Ġcase Ġto Ġaugment Ġthe ĠTIM IT Ġdata Ġfor Ġtraining Ġthe Ġembed d ings Ġonly . ĠAll Ġother Ġconfigurations Ġan Ġtraining Ġsettings Ġare Ġunchanged . ĠTable ĠT AB REF 18 Ġshows Ġimprovement Ġafter Ġusing Ġadditional Ġout - of - domain Ġdata Ġfor Ġtraining , Ġexcept Ġfor Ġin Ġthe Ġcase Ġof ĠCJ FS Ġand Ġfor Ġphone Ġclassification . ĠThe Ġimprovement Ġon Ġall Ġtasks Ġwith Ġthe Ġsimple Ġaddition Ġof Ġunl ab elled Ġaudio Ġdata Ġis Ġremarkable . ĠThis Ġis Ġalso Ġtrue Ġfor Ġthe Ġbaseline , Ġbut Ġthe Ġbenefit Ġof Ġthe Ġproposed Ġmethods Ġseems Ġunaffected . ĠThe ĠCJ FA Ġenc oder Ġperforms Ġbetter Ġin Ġcomparison Ġof Ġthe Ġother Ġtwo Ġapproaches Ġand Ġa Ġabsolute Ġaccuracy Ġimprovement Ġof Ġ7 . 9 % Ġfor Ġspeaker Ġrecognition ĠTask Ġb Ġis Ġobserved . ĠThe Ġclassification Ġtasks Ġbenefits Ġfrom Ġthe Ġadditional Ġdata Ġeven Ġthough Ġthe Ġlabelled Ġdata Ġremains Ġthe Ġsame . </s> Analysis </s> To Ġfurther Ġevaluate Ġthe Ġembed d ings Ġproduced Ġby Ġthe Ġ3 Ġmodels , Ġvisual isation Ġusing Ġthe Ġt - S NE Ġalgorithm BI B REF 30 Ġis Ġa Ġcommon Ġapproach , Ġalthough Ġinterpretation Ġis Ġsometimes Ġdifficult . ĠFig . ĠFIG REF 19 Ġvisual ises Ġthe Ġembed d ings Ġof Ġphon em es Ġin Ġtwo - dimensional Ġspace , Ġeach Ġphon eme Ġsymbol Ġrepresents Ġthe Ġmean Ġvector Ġof Ġall Ġof Ġthe Ġembed d ings Ġbelonging Ġto Ġthe Ġsame Ġphone Ġclass BI B REF 31 . ĠOne Ġcan Ġobserve Ġthat Ġthe ĠCJ FA Ġenc oder Ġappears Ġto Ġgenerate Ġmore Ġmeaningful Ġembed d ings Ġthan Ġthe Ġother Ġtwo Ġapproaches Ġ- Ġas Ġphon em es Ġbelonging Ġto Ġthe Ġsame Ġsound Ġclasses BI B REF 32 Ġare Ġgrouped Ġtogether Ġin Ġcloser Ġregions . ĠThe ĠVA E Ġbaseline Ġalso Ġhas Ġthis Ġbehaviour Ġbut Ġfor Ġexample Ġpl os ives Ġare Ġsplit Ġand Ġnasal Ġseparation Ġseems Ġless Ġclear . ĠInstead ĠCJ FS Ġshows Ġmore Ġconfusion Ġ- Ġas Ġexpected Ġand Ġexplained Ġabove . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġtwo Ġun super vised Ġacoustic Ġembed ding Ġapproaches Ġto Ġmodel Ġthe Ġjoint Ġlatent Ġfactors Ġbetween Ġthe Ġtarget Ġwindow Ġand Ġneighbouring Ġaudio Ġsegments Ġwere Ġproposed . ĠModels Ġare Ġbased Ġon Ġvari ational Ġauto - enc od ers , Ġwhich Ġalso Ġconstitute Ġthe Ġbaseline . ĠIn Ġorder Ġto Ġcompare Ġagainst Ġthe Ġbaseline Ġmodels Ġare Ġassessed Ġusing Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġtasks , Ġon ĠTIM IT , Ġand Ġwith Ġadditional ĠRM Ġdata . ĠResults Ġshow ĠCJ FA Ġ( context ual Ġjoint Ġfactor Ġanalysis ) Ġenc oder Ġperforms Ġsignificantly Ġbetter Ġin Ġboth Ġphone Ġclassification Ġand Ġspeaker Ġrecognition Ġtasks Ġcompared Ġwith Ġother Ġtwo Ġapproaches . ĠThe ĠCJ FS Ġ( context ual Ġjoint Ġfactor Ġsynthesis ) Ġenc oder Ġperforms Ġclose Ġto ĠCJ FA Ġin Ġspeaker Ġrecognition Ġtask , Ġbut Ġpoorer Ġfor Ġphone Ġclassification . ĠOverall Ġa Ġgain Ġof Ġup Ġto Ġ3 % Ġrelative Ġon Ġphone Ġclassification Ġaccuracy Ġis Ġobserved , Ġrelative Ġimprovements Ġon Ġspeaker Ġrecognition Ġshow Ġ3 âĢĵ 6 % Ġgain . ĠThe Ġproposed Ġun super vised Ġapproaches Ġobtain Ġembed d ings Ġand Ġcan Ġbe Ġimproved Ġwith Ġunl ab elled Ġout - of - domain Ġdata , Ġthe Ġclassification Ġtasks Ġbenefits Ġeven Ġthough Ġthe Ġlabelled Ġdata Ġremains Ġthe Ġsame . ĠFurther Ġwork Ġneeds Ġto Ġexpand Ġexperiments Ġon Ġlarger Ġdata Ġsets , Ġphone Ġrecognition Ġand Ġmore Ġcomplex Ġneural Ġnetwork Ġarchitectures .