Switchboard Telephone Speech Corpus BIBREF21, LORELEI (Low Resource Languages for Emergent Incidents) Program
EXTRACTIVE
LORELEI datasets of Uzbek, Mandarin and Turkish
ABSTRACTIVE
For our single-label topic classification experiments, we use the Switchboard Telephone Speech Corpus BIBREF21 , a collection of two-sided telephone conversations. We use the same development (dev) and evaluation (eval) data sets as in BIBREF8 , BIBREF9 . Each whole conversation has two sides and one single topic, and topic ID is performed on each individual-side speech (i.e., each side is seen as one single spoken document). In the 35.7 hour dev data, there are 360 conversation sides evenly distributed across six different topics (recycling, capital punishment, drug testing, family finance, job benefits, car buying), i.e., each topic has equal number of 60 sides. In the 61.6 hour eval data, there are another different six topics (family life, news media, public education, exercise/fitness, pets, taxes) evenly distributed across 600 conversation sides. Algorithm design choices are explored through experiments on dev data. We use manual segmentations provided by the Switchboard corpus to produce utterances with speech activity, and UTD and AUD are operating only on those utterances. We further evaluate our topic ID performance on the speech corpora of three languages released by the DARPA LORELEI (Low Resource Languages for Emergent Incidents) Program. For each language there are a number of audio speech files, and each speech file is cut into segments of various lengths (up to 120 seconds). Each speech segment is seen as either in-domain or out-of-domain. In-domain data is defined as any speech segment relating to an incident or incidents, and in-domain data will fall into a set of domain-specific categories; these categories are known as situation types, or in-domain topics. There are 11 situation types: “Civil Unrest or Wide-spread Crime”, “Elections and Politics”, “Evacuation”, “Food Supply”, “Urgent Rescue”, “Utilities, Energy, or Sanitation”, “Infrastructure”, “Medical Assistance”, “Shelter”, “Terrorism or other Extreme Violence”, and “Water Supply”. We consider “Out-of-domain” as the 12th topic label, so each speech segment either corresponds to one or multiple in-domain topics, or is “Out-of-domain”. We use the average precision (AP, equal to the area under the precision-recall curve) as the evaluation metric, and report both the AP across the overall 12 labels, and the AP across 11 situation types, as shown in Table TABREF18 . For each configuration, only a single 10-fold CV result is reported, since we observe less variance in results here than in Switchboard. We have 16.5 hours in-domain data and 8.5 hours out-of-domain data for Turkish, 2.9 and 13.2 hours for Uzbek, and 7.7 and 7.2 hours for Mandarin. We use the same CNN architecture as on Switchboard but make the changes as described in Section SECREF11 . Also we use mini-batch size 30 and fix the training epochs as 100. All CNNs use word2vec pre-training. Additionally, we also implement another two separate topic ID baselines using the decoded word outputs from two supervised ASR systems, trained from 80 hours transcribed Babel Turkish speech BIBREF29 and about 170 hours transcribed HKUST Mandarin telephone speech (LDC2005T32 and LDC2005S15), respectively.
We further evaluate our topic ID performance on the speech corpora of three languages released by the DARPA LORELEI (Low Resource Languages for Emergent Incidents) Program. For each language there are a number of audio speech files, and each speech file is cut into segments of various lengths (up to 120 seconds). Each speech segment is seen as either in-domain or out-of-domain. In-domain data is defined as any speech segment relating to an incident or incidents, and in-domain data will fall into a set of domain-specific categories; these categories are known as situation types, or in-domain topics. There are 11 situation types: “Civil Unrest or Wide-spread Crime”, “Elections and Politics”, “Evacuation”, “Food Supply”, “Urgent Rescue”, “Utilities, Energy, or Sanitation”, “Infrastructure”, “Medical Assistance”, “Shelter”, “Terrorism or other Extreme Violence”, and “Water Supply”. We consider “Out-of-domain” as the 12th topic label, so each speech segment either corresponds to one or multiple in-domain topics, or is “Out-of-domain”. We use the average precision (AP, equal to the area under the precision-recall curve) as the evaluation metric, and report both the AP across the overall 12 labels, and the AP across 11 situation types, as shown in Table TABREF18 . For each configuration, only a single 10-fold CV result is reported, since we observe less variance in results here than in Switchboard. We have 16.5 hours in-domain data and 8.5 hours out-of-domain data for Turkish, 2.9 and 13.2 hours for Uzbek, and 7.7 and 7.2 hours for Mandarin. We use the same CNN architecture as on Switchboard but make the changes as described in Section SECREF11 . Also we use mini-batch size 30 and fix the training epochs as 100. All CNNs use word2vec pre-training. Additionally, we also implement another two separate topic ID baselines using the decoded word outputs from two supervised ASR systems, trained from 80 hours transcribed Babel Turkish speech BIBREF29 and about 170 hours transcribed HKUST Mandarin telephone speech (LDC2005T32 and LDC2005S15), respectively. As shown in Table TABREF18 , UTD-based SVMs are more competitive than AUD-based SVMs on the smaller corpora, i.e., Uzbek and Mandarin, while being less competitive on the larger corpus, Turkish. We further investigate this behavior on each individual language by varying the amount of training data; we split the data into 10 folds, and perform 10-fold CV 9 times, varying the number of folds for training from 1 to 9. As illustrated in Figure FIGREF19 for Turkish, as we use more folds for training, AUD-based system starts to be more competitive than UTD. Supervised ASR-based systems still give the best results in various cases, while UTD and AUD based systems give comparable performance.
1703.07476
Introduction </s> Topic Ġidentification Ġ( topic ĠID ) Ġon Ġspeech Ġaims Ġto Ġidentify Ġthe Ġtopic ( s ) Ġfor Ġgiven Ġspeech Ġrecordings , Ġreferred Ġto Ġas Ġspoken Ġdocuments , Ġwhere Ġthe Ġtopics Ġare Ġa Ġpred efined Ġset Ġof Ġclasses Ġor Ġlabels . ĠThis Ġtask Ġis Ġtypically Ġformulated Ġas Ġa Ġthree - step Ġprocess . ĠFirst , Ġspeech Ġis Ġtoken ized Ġinto Ġwords Ġor Ġphones Ġby Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems ĠB IB REF 0 Ġ, Ġor Ġby Ġlimited - voc abulary Ġkeyword Ġspotting ĠB IB REF 1 Ġ. ĠSecond , Ġstandard Ġtext - based Ġprocessing Ġtechniques Ġare Ġapplied Ġto Ġthe Ġresulting Ġtoken izations , Ġand Ġproduce Ġa Ġvector Ġrepresentation Ġfor Ġeach Ġspoken Ġdocument , Ġtypically Ġa Ġbag - of - words Ġmult in omial Ġrepresentation , Ġor Ġa Ġmore Ġcompact Ġvector Ġgiven Ġby Ġprob abil istic Ġtopic Ġmodels ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠFinally , Ġtopic ĠID Ġis Ġperformed Ġon Ġthe Ġspoken Ġdocument Ġrepresentations Ġby Ġsupervised Ġtraining Ġof Ġclass ifiers , Ġsuch Ġas ĠBay esian Ġclass ifiers Ġand Ġsupport Ġvector Ġmachines Ġ( S V Ms ). </s> However , Ġin Ġthe Ġfirst Ġstep , Ġtraining Ġthe ĠAS R Ġsystem Ġrequired Ġfor Ġtoken ization Ġitself Ġrequires Ġtransc ribed Ġspeech Ġand Ġpron unci ations . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġa Ġdifficult Ġand Ġrealistic Ġscenario Ġwhere Ġthe Ġspeech Ġcorpus Ġof Ġa Ġtest Ġlanguage Ġis Ġannot ated Ġonly Ġwith Ġa Ġminimal Ġnumber Ġof Ġtopic Ġlabels , Ġi . e ., Ġno Ġmanual Ġtranscript ions Ġor Ġdiction aries Ġfor Ġbuilding Ġan ĠAS R Ġsystem Ġare Ġavailable . ĠWe Ġaim Ġto Ġexploit Ġapproaches Ġthat Ġenable Ġtopic ĠID Ġon Ġspeech Ġwithout Ġany Ġknowledge Ġof Ġthat Ġlanguage Ġother Ġthan Ġthe Ġtopic Ġannotations . </s> In Ġthis Ġscenario , Ġwhile Ġprevious Ġwork Ġdemonstrates Ġthat Ġthe Ġcross - ling ual Ġphon eme Ġrecogn izers Ġcan Ġproduce Ġreasonable Ġspeech Ġtoken izations ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġthe Ġperformance Ġis Ġhighly Ġdependent Ġon Ġthe Ġlanguage Ġand Ġenvironmental Ġcondition Ġ( channel , Ġnoise , Ġetc .) Ġmismatch Ġbetween Ġthe Ġtraining Ġand Ġtest Ġdata . ĠTherefore , Ġwe Ġfocus Ġon Ġun super vised Ġapproaches Ġthat Ġoperate Ġdirectly Ġon Ġthe Ġspeech Ġof Ġinterest . ĠRaw Ġacoustic Ġfeature - based Ġun super vised Ġterm Ġdiscovery Ġ( UT D ) Ġis Ġone Ġsuch Ġapproach Ġthat Ġaims Ġto Ġidentify Ġand Ġcluster Ġrepeating Ġword - like Ġunits Ġacross Ġspeech Ġbased Ġaround Ġsegment al Ġdynamic Ġtime Ġwar ping Ġ( DT W ) ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠB IB REF 8 Ġshows Ġthat Ġusing Ġthe Ġword - like Ġunits Ġfrom ĠUT D Ġfor Ġspoken Ġdocument Ġclassification Ġcan Ġwork Ġwell ; Ġhowever , Ġthe Ġresults Ġin ĠB IB REF 8 Ġare Ġlimited Ġsince Ġthe Ġacoustic Ġfeatures Ġon Ġwhich ĠUT D Ġis Ġperformed Ġare Ġproduced Ġby Ġacoustic Ġmodels Ġtrained Ġfrom Ġthe Ġtransc ribed Ġspeech Ġof Ġits Ġevaluation Ġcorpus . ĠIn Ġthis Ġpaper , Ġwe Ġinvestigate ĠUT D - based Ġtopic ĠID Ġperformance Ġwhen ĠUT D Ġoperates Ġon Ġlanguage - independent Ġspeech Ġrepresentations Ġextracted Ġfrom Ġmult ilingual Ġbottleneck Ġnetworks Ġtrained Ġon Ġlanguages Ġother Ġthan Ġthe Ġtest Ġlanguage ĠB IB REF 9 Ġ. ĠAnother Ġalternative Ġto Ġproducing Ġspeech Ġtoken izations Ġwithout Ġlanguage Ġdependency Ġis Ġthe Ġmodel - based Ġapproach , Ġi . e ., Ġun super vised Ġlearning Ġof Ġhidden ĠMark ov Ġmodel Ġ( H MM ) Ġbased Ġphon eme - like Ġunits Ġfrom Ġunt rans cribed Ġspeech . ĠWe Ġexploit Ġthe ĠVari ational ĠBay esian Ġinference Ġbased Ġacoustic Ġunit Ġdiscovery Ġ( AUD ) Ġframework Ġin ĠB IB REF 10 Ġthat Ġallows Ġparallel ized Ġlarge - scale Ġtraining . ĠIn Ġtopic ĠID Ġtasks , Ġsuch ĠAUD - based Ġsystems Ġhave Ġbeen Ġshown Ġto Ġoutper form Ġother Ġsystems Ġbased Ġon Ġcross - ling ual Ġphon eme Ġrecogn izers ĠB IB REF 5 Ġ, Ġand Ġthis Ġpaper Ġaims Ġto Ġfurther Ġinvestigate Ġhow Ġthe Ġperformance Ġcompares Ġamong ĠUT D , ĠAUD Ġand ĠAS R Ġbased Ġsystems . </s> Moreover , Ġafter Ġthe Ġspeech Ġis Ġtoken ized , Ġthese Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġare Ġlimited Ġto Ġusing Ġbag - of - words Ġfeatures Ġas Ġspoken Ġdocument Ġrepresentations . ĠWhile ĠUT D Ġonly Ġidentifies Ġrelatively Ġlong Ġ( 0 . 5 ĠâĢĵ Ġ1 Ġsec ) Ġrepeated Ġterms , ĠAUD / AS R Ġenables Ġfull - co verage Ġsegment ation Ġof Ġcontinuous Ġspeech Ġinto Ġa Ġsequence Ġof Ġunits / words , Ġand Ġsuch Ġa Ġresulting Ġtemporal Ġsequence Ġenables Ġanother Ġfeature Ġlearning Ġarchitecture Ġbased Ġon Ġconv olution al Ġneural Ġnetworks Ġ( CNN s ) ĠB IB REF 11 Ġ; Ġinstead Ġof Ġtreating Ġthe Ġsequential Ġtokens Ġas Ġa Ġbag Ġof Ġacoustic Ġunits Ġor Ġwords , Ġthe Ġwhole Ġtoken Ġsequence Ġis Ġencoded Ġas Ġconc aten ated Ġcontinuous Ġvectors , Ġand Ġfollowed Ġby Ġconv olution Ġand Ġtemporal Ġpool ing Ġoperations Ġthat Ġcapture Ġthe Ġlocal Ġand Ġglobal Ġdependencies . ĠSuch Ġcontinuous Ġspace Ġfeature Ġextraction Ġframeworks Ġhave Ġbeen Ġused Ġin Ġvarious Ġlanguage Ġprocessing Ġtasks Ġlike Ġspoken Ġlanguage Ġunderstanding ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġand Ġtext Ġclassification ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠHowever , Ġthree Ġquestions Ġare Ġworth Ġinvestigating Ġin Ġour ĠAUD - based Ġsetting : Ġ( i ) Ġif Ġsuch Ġa ĠCNN - based Ġframework Ġcan Ġperform Ġas Ġwell Ġon Ġnoisy Ġautomatically Ġdiscovered Ġphon eme - like Ġunits Ġas Ġon Ġorth ographic Ġwords / char acters , Ġ( ii ) Ġif Ġpre - trained Ġvectors Ġof Ġphon eme - like Ġunits Ġfrom Ġword 2 vec ĠB IB REF 16 Ġprovide Ġsuperior Ġperformance Ġto Ġrandom Ġinitialization Ġas Ġevidenced Ġby Ġthe Ġword - based Ġtasks , Ġand Ġ( iii ) Ġif ĠCNN s Ġare Ġstill Ġcompetitive Ġin Ġlow - resource Ġsettings Ġof Ġhundreds Ġto Ġtwo - th ousand Ġtraining Ġexempl ars , Ġrather Ġthan Ġthe Ġlarge / medium Ġsized Ġdatasets Ġas Ġin Ġprevious Ġwork ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. </s> Finally , Ġincorporating Ġthe Ġdifferent Ġtoken ization Ġand Ġfeature Ġrepresentation Ġapproaches Ġnoted Ġabove , Ġwe Ġperform Ġcomprehensive Ġtopic ĠID Ġevaluations Ġon Ġboth Ġsingle - label Ġand Ġmulti - label Ġspoken Ġdocument Ġclassification Ġtasks . </s> Un super vised Ġtoken izations Ġof Ġspeech </s> Un super vised Ġterm Ġdiscovery Ġ( UT D ) </s> UT D Ġaims Ġto Ġautomatically Ġidentify Ġand Ġcluster Ġrepeated Ġterms Ġ( e . g . Ġwords Ġor Ġphrases ) Ġfrom Ġspeech . ĠTo Ġcircumvent Ġthe Ġexhaustive ĠDT W - based Ġsearch Ġlimited Ġby ĠIN LINE FORM 0 Ġtime ĠB IB REF 6 Ġ, Ġwe Ġexploit Ġthe Ġscalable ĠUT D Ġframework Ġin Ġthe ĠZero ĠResource ĠTool kit Ġ( Z RT ools ) ĠB IB REF 7 Ġ, Ġwhich Ġpermits Ġsearch Ġin ĠIN LINE FORM 1 Ġtime . ĠWe Ġbriefly Ġdescribe Ġthe ĠUT D Ġprocedures Ġin ĠZ RT ools Ġby Ġfour Ġsteps Ġbelow , Ġand Ġfull Ġdetails Ġcan Ġbe Ġfound Ġin ĠB IB REF 7 Ġ. </s> Construct Ġthe Ġsparse Ġapproximate Ġacoustic Ġsimilarity Ġmat rices Ġbetween Ġpairs Ġof Ġspeech Ġutter ances . </s> Ident ify Ġword Ġrepet itions Ġvia Ġfast Ġdiagonal Ġline Ġsearch Ġand Ġsegment al ĠDT W . </s> The Ġresulting Ġmatches Ġare Ġused Ġto Ġconstruct Ġan Ġacoustic Ġsimilarity Ġgraph , Ġwhere Ġnodes Ġrepresent Ġthe Ġmatching Ġacoustic Ġsegments Ġand Ġedges Ġreflect ĠDT W Ġdistances . </s> Th reshold Ġthe Ġgraph Ġedges , Ġand Ġeach Ġconnected Ġcomponent Ġof Ġthe Ġgraph Ġis Ġa Ġcluster Ġof Ġacoustic Ġsegments , Ġwhich Ġproduces Ġa Ġcorresponding Ġterm Ġ( word / phrase ) Ġcategory . </s> Finally , Ġthe Ġcluster Ġof Ġeach Ġdiscovered Ġterm Ġcategory Ġconsists Ġof Ġa Ġlist Ġof Ġterm Ġoccurrences . </s> Note Ġthat Ġin Ġthe Ġthird Ġstep Ġabove , Ġthe Ġweight Ġon Ġeach Ġgraph Ġedge Ġcan Ġbe Ġexact ĠDT W - based Ġsimilarity , Ġor Ġother Ġsimilarity Ġbased Ġon Ġhe ur istics Ġmore Ġthan ĠDT W Ġdistance . ĠFor Ġexample , Ġwe Ġinvestigate Ġan Ġimplementation Ġin ĠZ RT ools , Ġwhere Ġa Ġseparate Ġlog istic Ġregression Ġmodel Ġis Ġused Ġto Ġresc ore Ġthe Ġsimilarity Ġbetween Ġidentified Ġmatches Ġby Ġdetermining Ġhow Ġlikely Ġthe Ġmatching Ġpair Ġis Ġthe Ġsame Ġunderlying Ġword / phrase Ġand Ġis Ġnot Ġa Ġfilled Ġpause Ġ( e . g . ĠâĢ ľ um - hum âĢ Ŀ Ġand ĠâĢ ľ yeah Ġuh - h uh âĢ Ŀ Ġin ĠEnglish ). ĠF illed Ġpauses Ġtend Ġto Ġbe Ġac oust ically Ġstationary Ġwith Ġmore Ġphone Ġrepeats Ġand Ġthus Ġwould Ġmatch Ġthroughout Ġthe Ġacoustic Ġsimilarity Ġmatrix , Ġwhereas Ġa Ġcontent ful Ġword Ġ( without Ġtoo Ġmany Ġphone Ġrepeats ) Ġtend Ġto Ġconcentrate Ġaround Ġthe Ġmain Ġdiagonal ; Ġthus , Ġthe Ġfeatures Ġin Ġlog istic Ġregression Ġcontain Ġthe Ġnumbers Ġof Ġmatrix Ġelements Ġin Ġdiagonal Ġbands Ġin Ġprogressive Ġsteps Ġaway Ġfrom Ġthe Ġmain Ġdiagonal . ĠFeature Ġweights Ġare Ġlearned Ġusing Ġa Ġportion Ġof Ġtransc ribed Ġspeech Ġwith Ġreference Ġtranscripts , Ġand Ġthe Ġresulting Ġmodel Ġcan Ġbe Ġused Ġfor Ġlanguage - independent Ġresc oring . </s> Ac oustic Ġunit Ġdiscovery Ġ( AUD ) </s> We Ġexploit Ġthe Ġnon param etric ĠBay esian ĠAUD Ġframework Ġin ĠB IB REF 10 Ġbased Ġon Ġvari ational Ġinference , Ġrather Ġthan Ġthe Ġmaximum Ġlikelihood Ġtraining Ġin ĠB IB REF 4 Ġwhich Ġmay Ġovers impl ify Ġthe Ġparameter Ġestim ations , Ġnor Ġthe ĠGibbs ĠSam pling Ġtraining Ġin ĠB IB REF 17 Ġwhich Ġis Ġnot Ġam enable Ġto Ġlarge Ġscale Ġapplications . ĠSpecifically , Ġa Ġphone - loop Ġmodel Ġis Ġformulated Ġwhere Ġeach Ġphon eme - like Ġunit Ġis Ġmodeled Ġas Ġan ĠH MM Ġwith Ġa ĠGa ussian Ġmixture Ġmodel Ġof Ġoutput Ġdens ities Ġ( G MM - H MM ). ĠUnder Ġthe ĠDir ich let Ġprocess Ġframework , Ġwe Ġconsider Ġthe Ġphone Ġloop Ġas Ġan Ġinfinite Ġmixture Ġof ĠGM M - HM Ms , Ġand Ġthe Ġmixture Ġweights Ġare Ġbased Ġon Ġthe Ġstick - breaking Ġconstruction Ġof ĠDir ich let Ġprocess . ĠThe Ġinfinite Ġnumber Ġof Ġunits Ġin Ġthe Ġmixture Ġis Ġtrunc ated Ġin Ġpractice , Ġgiving Ġzero Ġmixture Ġweight Ġto Ġany Ġunit Ġbeyond Ġsome Ġlarge Ġcount . ĠWe Ġtreat Ġsuch Ġmixture Ġof ĠGM M - HM Ms Ġas Ġa Ġsingle Ġunified ĠH MM Ġand Ġthus Ġthe Ġsegment ation Ġof Ġthe Ġdata Ġis Ġperformed Ġusing Ġstandard Ġforward - back ward Ġalgorithm . ĠTraining Ġis Ġfully Ġun super vised Ġand Ġparallel ized ; Ġafter Ġa Ġfixed Ġnumber Ġof Ġtraining Ġiterations , Ġwe Ġuse ĠV iter bi Ġdecoding Ġalgorithm Ġto Ġobtain Ġacoustic Ġunit Ġtoken izations Ġof Ġthe Ġdata . </s> B ag - of - words Ġrepresentation </s> After Ġwe Ġobtain Ġthe Ġtoken izations Ġof Ġspeech Ġby Ġeither ĠUT D Ġor ĠAUD , Ġeach Ġspoken Ġdocument Ġis Ġrepresented Ġby Ġa Ġvector Ġof Ġun ig ram Ġoccurrence Ġcounts Ġover Ġdiscovered Ġterms , Ġor Ġa Ġvector Ġof ĠIN LINE FORM 0 Ġ- gram Ġcounts Ġover Ġacoustic Ġunits , Ġrespectively . ĠEach Ġfeature Ġvector Ġcan Ġbe Ġfurther Ġscaled Ġby Ġinverse Ġdocument Ġfrequency Ġ( ID F ), Ġproducing Ġa ĠTF - ID F Ġfeature . </s> Con v olution al Ġneural Ġnetwork - based Ġrepresentation </s> AUD Ġenables Ġfull - co verage Ġtoken ization Ġof Ġcontinuous Ġspeech Ġinto Ġa Ġsequence Ġof Ġacoustic Ġunits , Ġwhich Ġwe Ġcan Ġexploit Ġin Ġa ĠCNN - based Ġframework Ġto Ġlearn Ġa Ġvector Ġrepresentation Ġfor Ġeach Ġspoken Ġdocument . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 7 Ġ, Ġin Ġan Ġacoustic Ġunit Ġsequence Ġa Ġof Ġlength ĠIN LINE FORM 0 Ġ, Ġeach Ġunit ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġis Ġencoded Ġas Ġa Ġfixed Ġdimensional Ġcontinuous Ġvector , Ġand Ġthe Ġwhole Ġsequence Ġa Ġis Ġrepresented Ġas Ġa Ġconc aten ated Ġvector Ġx . ĠA Ġshared Ġconv olution al Ġfeature Ġtransform ĠIN LINE FORM 3 Ġspans Ġa Ġfixed - sized ĠIN LINE FORM 4 Ġ- gram Ġwindow , ĠIN LINE FORM 5 Ġ, Ġand Ġslides Ġover Ġthe Ġwhole Ġsequence . ĠThen Ġthe Ġhidden Ġfeature Ġlayer ĠIN LINE FORM 6 Ġwith Ġnon linear ities Ġconsists Ġof Ġeach Ġfeature Ġvector ĠIN LINE FORM 7 Ġextracted Ġfrom Ġthe Ġshared Ġconv olution al Ġwindow Ġcentered Ġat Ġeach Ġacoustic Ġunit Ġposition ĠIN LINE FORM 8 Ġ. ĠMax - pool ing Ġis Ġperformed Ġon Ġtop Ġof Ġeach ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġ, Ġto Ġobtain Ġa Ġfixed - dimensional Ġvector Ġrepresentation Ġfor Ġthe Ġwhole Ġsequence Ġa , Ġi . e ., Ġa Ġvector Ġrepresentation Ġof Ġthe Ġwhole Ġspoken Ġdocument , Ġfollowed Ġby Ġanother Ġhidden Ġlayer ĠIN LINE FORM 11 Ġand Ġa Ġfinal Ġoutput Ġlayer . ĠNote Ġthat Ġthis Ġframework Ġneeds Ġsupervision Ġfor Ġtraining ; Ġe . g ., Ġthe Ġoutput Ġlayer Ġcan Ġbe Ġa Ġsoft max Ġfunction Ġfor Ġsingle - label Ġclassification , Ġand Ġthe Ġwhole Ġmodel Ġis Ġtrained Ġwith Ġcateg orical Ġcross - ent ropy Ġloss . </s> Also , Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġunique Ġacoustic Ġunit Ġcan Ġbe Ġrandomly Ġinitialized , Ġor Ġpre - trained Ġfrom Ġother Ġtasks . ĠSpecifically , Ġwe Ġapply Ġthe Ġskip - gram Ġmodel Ġof Ġword 2 vec ĠB IB REF 18 Ġto Ġpre - train Ġone Ġembed ding Ġvector Ġfor Ġeach Ġacoustic Ġunit , Ġbased Ġon Ġthe Ġhierarchical Ġsoft max Ġwith ĠHuff man Ġcodes . </s> Single - label Ġclassification </s> For Ġthe Ġbag - of - words Ġrepresentation , Ġwe Ġuse Ġa Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) Ġbased Ġlinear ĠS VM ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġwith Ġhinge Ġloss Ġand ĠIN LINE FORM 0 Ġ/ ĠIN LINE FORM 1 Ġnorm Ġregular ization . ĠFor Ġthe ĠCNN - based Ġframework , Ġwe Ġuse Ġa Ġsoft max Ġfunction Ġin Ġthe Ġoutput Ġlayer Ġfor Ġclassification Ġas Ġdescribed Ġin ĠSection ĠSEC REF 9 Ġ. </s> For Ġour Ġsingle - label Ġtopic Ġclassification Ġexperiments , Ġwe Ġuse Ġthe ĠSwitch board ĠTelephone ĠSpeech ĠCorpus ĠB IB REF 21 Ġ, Ġa Ġcollection Ġof Ġtwo - sided Ġtelephone Ġconversations . ĠWe Ġuse Ġthe Ġsame Ġdevelopment Ġ( dev ) Ġand Ġevaluation Ġ( eval ) Ġdata Ġsets Ġas Ġin ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠEach Ġwhole Ġconversation Ġhas Ġtwo Ġsides Ġand Ġone Ġsingle Ġtopic , Ġand Ġtopic ĠID Ġis Ġperformed Ġon Ġeach Ġindividual - side Ġspeech Ġ( i . e ., Ġeach Ġside Ġis Ġseen Ġas Ġone Ġsingle Ġspoken Ġdocument ). ĠIn Ġthe Ġ35 . 7 Ġhour Ġdev Ġdata , Ġthere Ġare Ġ360 Ġconversation Ġsides Ġevenly Ġdistributed Ġacross Ġsix Ġdifferent Ġtopics Ġ( recy cling , Ġcapital Ġpunishment , Ġdrug Ġtesting , Ġfamily Ġfinance , Ġjob Ġbenefits , Ġcar Ġbuying ), Ġi . e ., Ġeach Ġtopic Ġhas Ġequal Ġnumber Ġof Ġ60 Ġsides . ĠIn Ġthe Ġ61 . 6 Ġhour Ġeval Ġdata , Ġthere Ġare Ġanother Ġdifferent Ġsix Ġtopics Ġ( family Ġlife , Ġnews Ġmedia , Ġpublic Ġeducation , Ġexercise / f itness , Ġpets , Ġtaxes ) Ġevenly Ġdistributed Ġacross Ġ600 Ġconversation Ġsides . ĠAl gorithm Ġdesign Ġchoices Ġare Ġexplored Ġthrough Ġexperiments Ġon Ġdev Ġdata . ĠWe Ġuse Ġmanual Ġsegment ations Ġprovided Ġby Ġthe ĠSwitch board Ġcorpus Ġto Ġproduce Ġutter ances Ġwith Ġspeech Ġactivity , Ġand ĠUT D Ġand ĠAUD Ġare Ġoperating Ġonly Ġon Ġthose Ġutter ances . </s> For ĠUT D , Ġwe Ġuse Ġthe ĠZ RT ools ĠB IB REF 7 Ġimplementation Ġwith Ġthe Ġdefault Ġparameters Ġexcept Ġthat , Ġwe Ġuse Ġcos ine Ġsimilarity Ġthreshold ĠIN LINE FORM 0 Ġ, Ġand Ġvary Ġthe Ġdiagonal Ġmedian Ġfilter Ġduration ĠIN LINE FORM 1 Ġover ĠIN LINE FORM 2 Ġ; Ġwe Ġtry Ġboth Ġthe Ġexact ĠDT W - based Ġsimilarity Ġand Ġthe Ġresc ored Ġsimilarity Ġas Ġdescribed Ġin ĠSection ĠSEC REF 1 Ġ, Ġand Ġtune Ġthe Ġsimilarity Ġthreshold Ġ( used Ġto Ġpartition Ġthe Ġgraph Ġedges ) Ġover ĠIN LINE FORM 3 Ġ. ĠFor ĠAUD , Ġthe Ġun super vised Ġtraining Ġis Ġperformed Ġonly Ġon Ġthe Ġdev Ġdata Ġ( 10 Ġiterations ); Ġafter Ġtraining , Ġwe Ġuse Ġthe Ġlearned Ġmodels Ġto Ġdecode Ġboth Ġdev Ġand Ġeval Ġdata Ġset , Ġand Ġobtain Ġthe Ġacoustic Ġunit Ġtoken izations . ĠWe Ġuse Ġtrunc ation Ġlevel Ġ200 , Ġwhich Ġimplies Ġmaximum Ġ200 Ġdifferent Ġacoustic Ġunits Ġcan Ġbe Ġlearned Ġfrom Ġthe Ġcorpus . ĠFor Ġeach Ġacoustic Ġunit , Ġwe Ġuse Ġa Ġ3 - state ĠH MM Ġwith Ġ2 ĠGa uss ians Ġper Ġstate . ĠFor Ġthe Ġstick - breaking Ġconstruction Ġof ĠDir ich let Ġprocess , Ġwe Ġvary Ġthe Ġconcentration Ġparameter ĠIN LINE FORM 4 Ġover ĠIN LINE FORM 5 Ġ, Ġand Ġother Ġhyper param eters Ġare Ġthe Ġsame Ġas ĠB IB REF 10 Ġ. ĠThe Ġacoustic Ġfeatures Ġon Ġwhich ĠUT D Ġand ĠAUD Ġoperate Ġare Ġextracted Ġusing Ġthe Ġsame Ġmult ilingual Ġbottleneck Ġ( BN ) Ġnetwork Ġas Ġdescribed Ġin ĠB IB REF 9 Ġwith ĠK aldi Ġtool kit ĠB IB REF 22 Ġ. ĠWe Ġconduct Ġthe Ġmult ilingual ĠB N Ġtraining Ġwith Ġ10 Ġlanguage Ġcollections Ġ( Ass ames e , ĠBeng ali , ĠCant ones e , ĠHaitian , ĠLa o , ĠP ash to , ĠTamil , ĠTag alog , ĠVietnamese Ġand ĠZ ulu ) ĠâĢĵ Ġ10 Ġhours Ġof Ġtransc ribed Ġspeech Ġper Ġlanguage . ĠComplete Ġspecifications Ġcan Ġbe Ġfound Ġin ĠB IB REF 9 Ġ. </s> For ĠS VM - based Ġclassification , Ġwe Ġuse Ġthe Ġbag Ġof Ġdiscovered Ġterm Ġun ig rams , Ġor Ġbag Ġof Ġacoustic Ġunit Ġtrig rams . ĠOn Ġdev Ġdata , Ġwe Ġtry Ġusing Ġthe Ġfeatures Ġof Ġraw Ġcounts Ġor Ġscaled Ġby ĠIDF , ĠS VM Ġregular ization Ġtuned Ġover ĠIN LINE FORM 0 Ġ/ ĠIN LINE FORM 1 Ġnorm , Ġregular ization Ġconstant Ġtuned Ġover ĠIN LINE FORM 2 Ġ, Ġand ĠSG D Ġepoch s Ġtuned Ġover ĠIN LINE FORM 3 Ġ. ĠWe Ġfurther Ġnormal ize Ġeach Ġfeature Ġto ĠIN LINE FORM 4 Ġnorm Ġunit Ġlength . ĠEach Ġexperiment Ġis Ġa Ġrun Ġof Ġ10 - fold Ġcross Ġvalidation Ġ( CV ) Ġon Ġthe Ġ360 Ġconversation Ġsides Ġof Ġdev Ġdata , Ġor Ġon Ġthe Ġ600 Ġsides Ġof Ġeval Ġdata , Ġrespectively . ĠNote Ġthat Ġour Ġdata Ġsize Ġhere Ġis Ġrelatively Ġsmall Ġ( only Ġ360 Ġor Ġ600 ) Ġand Ġthe ĠSG D Ġtraining Ġmay Ġgive Ġhigh Ġvariance Ġin Ġthe Ġperformance ĠB IB REF 23 Ġ. ĠTherefore , Ġto Ġreport Ġclassification Ġaccuracy Ġfor Ġeach Ġconfiguration Ġ( when Ġvarying Ġfeatures Ġor Ġmodels ), Ġwe Ġrepeat Ġeach ĠCV Ġexperiment Ġ5 Ġtimes , Ġwhere Ġeach Ġexperiment Ġagain Ġis Ġa Ġrun Ġof Ġ10 - fold ĠCV ; Ġthen Ġfor Ġeach Ġconfiguration , Ġthe Ġmean Ġand Ġstandard Ġdeviation Ġof Ġ5 Ġexperiments Ġis Ġreported . </s> For ĠCNN - based Ġclassification , Ġwe Ġuse Ġthe Ġsame Ġstrategy Ġto Ġreport Ġclassification Ġaccuracy , Ġi . e ., Ġrepeating Ġexperiments Ġ5 Ġtimes Ġ( where Ġeach Ġtime Ġis Ġa Ġ10 - fold ĠCV ) Ġfor Ġeach ĠCNN Ġconfiguration . ĠNote Ġthat Ġthe Ġrespective Ġ10 Ġfolds Ġof Ġboth Ġdev Ġand Ġeval Ġdata Ġsets Ġare Ġfixed Ġthe Ġsame Ġfor Ġall Ġthe ĠS VM Ġand ĠCNN Ġexperiments . ĠAdditionally , Ġfor Ġeach Ġ10 - fold ĠCV Ġexperiment , Ġinstead Ġof Ġtraining Ġon Ġ9 Ġfolds Ġand Ġtesting Ġon Ġthe Ġremaining Ġ1 Ġfold Ġas Ġin ĠS VM , Ġwe Ġuse Ġ8 Ġfolds Ġfor ĠCNN Ġtraining , Ġleave Ġanother Ġ1 Ġfold Ġout Ġas Ġvalidation Ġdata ; Ġafter Ġtraining Ġeach ĠCNN Ġmodel Ġfor Ġup Ġto Ġ100 Ġepoch s , Ġthe Ġmodel Ġwith Ġthe Ġbest Ġaccuracy Ġon Ġthe Ġvalidation Ġdata Ġis Ġused Ġfor Ġevaluation Ġon Ġthe Ġtest Ġset . ĠThe Ġacoustic Ġunit Ġsequence Ġ( as ĠCNN Ġinputs ) Ġare Ġzero - p added Ġto Ġthe Ġlongest Ġlength Ġin Ġeach Ġdataset . ĠWe Ġimplemented Ġthe ĠCNN s Ġin ĠKer as ĠB IB REF 24 Ġwith ĠThe ano ĠB IB REF 25 Ġbackend . ĠCNN Ġarchitectures Ġare Ġdetermined Ġthrough Ġexperiments Ġon Ġdev Ġdata . ĠFor ĠSG D Ġtraining Ġwe Ġuse Ġthe ĠAd ad elta Ġoptim izer ĠB IB REF 26 Ġand Ġmini - batch Ġsize Ġ18 . ĠThe ĠIN LINE FORM 0 Ġ- gram Ġwindow Ġsize Ġof Ġeach Ġconv olution al Ġfeature Ġtransform ĠIN LINE FORM 1 Ġis Ġ7 . ĠThe Ġsize Ġof Ġeach Ġhidden Ġfeature Ġvector ĠIN LINE FORM 2 Ġ( ext racted Ġfrom Ġthe Ġtransform ĠIN LINE FORM 3 Ġ) Ġis Ġ1024 , Ġwith Ġrect ified Ġlinear Ġunit Ġ( Re LU ) Ġnon linear ities . ĠThus , Ġafter Ġmax - pool ing Ġover Ġtime , Ġwe Ġhave Ġa Ġ1024 - dimensional Ġvector Ġagain , Ġwhich Ġthen Ġgoes Ġthrough Ġanother Ġhidden Ġlayer ĠIN LINE FORM 4 Ġ( also Ġset Ġas Ġ1024 - dimensional Ġwith ĠRe LU ) Ġand Ġfinally Ġinto Ġa Ġsoft max . ĠDrop out ĠB IB REF 27 Ġrate Ġ0 . 2 Ġis Ġused Ġat Ġeach Ġlayer . </s> When Ġwe Ġinitialize Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġacoustic Ġunit Ġwith Ġa Ġset Ġof Ġpre - trained Ġvectors Ġ( instead Ġof Ġrandom Ġinitial izations ), Ġwe Ġapply Ġthe Ġskip - gram Ġmodel Ġof Ġword 2 vec ĠB IB REF 18 Ġto Ġthe Ġacoustic Ġunit Ġtoken izations Ġof Ġeach Ġdata Ġset . ĠWe Ġuse Ġthe Ġg ens im Ġimplementation ĠB IB REF 28 Ġ, Ġwhich Ġincludes Ġa Ġvector Ġspace Ġof Ġembed ding Ġdimension Ġ50 Ġ( tun ed Ġover ĠIN LINE FORM 0 Ġ), Ġa Ġskip - gram Ġwindow Ġof Ġsize Ġ5 , Ġand ĠSG D Ġover Ġ20 Ġepoch s . </s> Table ĠT AB REF 15 Ġshows Ġthe Ġtopic ĠID Ġresults Ġon ĠSwitch board . ĠFor ĠUT D - based Ġclass ifications , Ġwe Ġfind Ġthat Ġthe Ġdefault Ġresc oring Ġin ĠZ RT ools ĠB IB REF 7 Ġwhich Ġis Ġdesigned Ġto Ġfilter Ġout Ġthe Ġfilled Ġpauses Ġproduces Ġcomparable Ġperformance Ġto Ġthe Ġraw ĠDT W Ġsimilarity Ġscores , Ġbut Ġthe Ġresc oring Ġcan Ġresult Ġin Ġmuch Ġfaster Ġconnected - component Ġclust ering Ġ( Section ĠSEC REF 1 Ġ). ĠNote Ġthat Ġthis Ġresc oring Ġmodel Ġis Ġestimated Ġusing Ġa Ġportion Ġof Ġtransc ribed ĠSwitch board , Ġbut Ġit Ġis Ġstill Ġa Ġlegitimate Ġlanguage - independent ĠUT D Ġapproach Ġwhile Ġoperating Ġon Ġlanguages Ġother Ġthan ĠEnglish . ĠWhile Ġa Ġdiagonal Ġmedian Ġfilter Ġduration ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 Ġor ĠIN LINE FORM 2 Ġgives Ġsimilar Ġresults , ĠIN LINE FORM 3 Ġproduces Ġlonger Ġbut Ġfewer Ġterms , Ġgiving Ġmore Ġsparse Ġfeature Ġrepresentations . ĠTherefore , Ġwe Ġproceed Ġwith Ġresc oring Ġand ĠIN LINE FORM 4 Ġin Ġthe Ġfollowing ĠUT D Ġexperiments Ġ( Section ĠSEC REF 16 Ġ). </s> For ĠAUD - based Ġclass ifications , ĠCNN Ġwithout Ġword 2 vec Ġpre - training Ġusually Ġgives Ġcomparable Ġresults Ġwith ĠS VM ; Ġhowever , Ġusing Ġword 2 vec Ġpre - training , ĠCNN Ġsubstantially Ġoutper forms Ġthe Ġcompeting ĠS VM Ġin Ġall Ġcases . ĠAlso Ġas Ġthe Ġconcentration Ġparameter ĠIN LINE FORM 0 Ġin ĠAUD Ġincreases Ġfrom ĠIN LINE FORM 1 Ġto ĠIN LINE FORM 2 Ġ( y ielding Ġless Ġconcentrated Ġdistributions ), Ġwe Ġhave Ġmore Ġunique Ġacoustic Ġunits Ġin Ġthe Ġtoken izations Ġof Ġboth Ġdata Ġsets , Ġfrom Ġ184 Ġto Ġ199 , Ġand ĠIN LINE FORM 3 Ġusually Ġproduces Ġbetter Ġresults Ġthan ĠIN LINE FORM 4 Ġ. </s> Multi - label Ġclassification </s> In Ġthe Ġsetting Ġwhere Ġeach Ġspoken Ġdocument Ġcan Ġbe Ġassociated Ġwith Ġmultiple Ġtopics / lab els , Ġwe Ġproceed Ġto Ġperform Ġa Ġmulti - label Ġclassification Ġtask . ĠThe Ġbaseline Ġapproach Ġis Ġthe Ġbinary Ġrelevance Ġmethod , Ġwhich Ġindependently Ġtrains Ġone Ġbinary Ġclass ifier Ġfor Ġeach Ġlabel , Ġand Ġthe Ġspoken Ġdocument Ġis Ġevaluated Ġby Ġeach Ġclass ifier Ġto Ġdetermine Ġif Ġthe Ġrespective Ġlabel Ġapplies Ġto Ġit . ĠSpecifically , Ġwe Ġuse Ġa Ġset Ġof ĠSV Ms Ġ( Section ĠSEC REF 10 Ġ), Ġone Ġfor Ġeach Ġlabel , Ġon Ġthe Ġbag - of - words Ġfeatures . </s> To Ġadapt Ġthe ĠCNN - based Ġframework Ġfor Ġmulti - label Ġclassification , Ġwe Ġreplace Ġthe Ġsoft max Ġin Ġthe Ġoutput Ġlayer Ġwith Ġa Ġset Ġof Ġs igm oid Ġoutput Ġnodes , Ġone Ġfor Ġeach Ġlabel , Ġas Ġshown Ġin ĠFigure ĠFIG REF 7 Ġ. ĠSince Ġa Ġs igm oid Ġnaturally Ġprovides Ġoutput Ġvalues Ġbetween Ġ0 Ġand Ġ1 , Ġwe Ġtrain Ġthe Ġneural Ġnetwork Ġ( NN ) Ġto Ġminimize Ġthe Ġbinary Ġcross Ġentropy Ġloss Ġdefined Ġas ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġdenotes Ġthe ĠN N Ġparameters , Ġx Ġis Ġthe Ġfeature Ġvector Ġof Ġacoustic Ġunit Ġsequence , Ġy Ġis Ġthe Ġtarget Ġvector Ġof Ġlabels , ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġthe Ġoutput Ġand Ġthe Ġtarget Ġfor Ġlabel ĠIN LINE FORM 4 Ġ, Ġand Ġthe Ġnumber Ġof Ġunique Ġlabels Ġis ĠIN LINE FORM 5 Ġ. </s> We Ġfurther Ġevaluate Ġour Ġtopic ĠID Ġperformance Ġon Ġthe Ġspeech Ġcorpor a Ġof Ġthree Ġlanguages Ġreleased Ġby Ġthe ĠDAR PA ĠL ORE LE I Ġ( Low ĠResource ĠLanguages Ġfor ĠEmer gent ĠInc idents ) ĠProgram . ĠFor Ġeach Ġlanguage Ġthere Ġare Ġa Ġnumber Ġof Ġaudio Ġspeech Ġfiles , Ġand Ġeach Ġspeech Ġfile Ġis Ġcut Ġinto Ġsegments Ġof Ġvarious Ġlengths Ġ( up Ġto Ġ120 Ġseconds ). ĠEach Ġspeech Ġsegment Ġis Ġseen Ġas Ġeither Ġin - domain Ġor Ġout - of - domain . ĠIn - domain Ġdata Ġis Ġdefined Ġas Ġany Ġspeech Ġsegment Ġrelating Ġto Ġan Ġincident Ġor Ġincidents , Ġand Ġin - domain Ġdata Ġwill Ġfall Ġinto Ġa Ġset Ġof Ġdomain - specific Ġcategories ; Ġthese Ġcategories Ġare Ġknown Ġas Ġsituation Ġtypes , Ġor Ġin - domain Ġtopics . ĠThere Ġare Ġ11 Ġsituation Ġtypes : ĠâĢ ľ Civil ĠUn rest Ġor ĠWide - spread ĠCrime âĢ Ŀ , ĠâĢ ľ Elect ions Ġand ĠPolitics âĢ Ŀ , ĠâĢ ľ Ev ac uation âĢ Ŀ , ĠâĢ ľ Food ĠSupply âĢ Ŀ , ĠâĢ ľ Ur gent ĠRescue âĢ Ŀ , ĠâĢ ľ Ut ilities , ĠEnergy , Ġor ĠSan itation âĢ Ŀ , ĠâĢ ľ Inf rastructure âĢ Ŀ , ĠâĢ ľ Medical ĠAssistance âĢ Ŀ , ĠâĢ ľ S hel ter âĢ Ŀ , ĠâĢ ľ Terror ism Ġor Ġother ĠExtreme ĠViolence âĢ Ŀ , Ġand ĠâĢ ľ Water ĠSupply âĢ Ŀ . ĠWe Ġconsider ĠâĢ ľ Out - of - domain âĢ Ŀ Ġas Ġthe Ġ12 th Ġtopic Ġlabel , Ġso Ġeach Ġspeech Ġsegment Ġeither Ġcorresponds Ġto Ġone Ġor Ġmultiple Ġin - domain Ġtopics , Ġor Ġis ĠâĢ ľ Out - of - domain âĢ Ŀ . ĠWe Ġuse Ġthe Ġaverage Ġprecision Ġ( AP , Ġequal Ġto Ġthe Ġarea Ġunder Ġthe Ġprecision - rec all Ġcurve ) Ġas Ġthe Ġevaluation Ġmetric , Ġand Ġreport Ġboth Ġthe ĠAP Ġacross Ġthe Ġoverall Ġ12 Ġlabels , Ġand Ġthe ĠAP Ġacross Ġ11 Ġsituation Ġtypes , Ġas Ġshown Ġin ĠTable ĠT AB REF 18 Ġ. ĠFor Ġeach Ġconfiguration , Ġonly Ġa Ġsingle Ġ10 - fold ĠCV Ġresult Ġis Ġreported , Ġsince Ġwe Ġobserve Ġless Ġvariance Ġin Ġresults Ġhere Ġthan Ġin ĠSwitch board . ĠWe Ġhave Ġ16 . 5 Ġhours Ġin - domain Ġdata Ġand Ġ8 . 5 Ġhours Ġout - of - domain Ġdata Ġfor ĠTurkish , Ġ2 . 9 Ġand Ġ13 . 2 Ġhours Ġfor ĠUzbek , Ġand Ġ7 . 7 Ġand Ġ7 . 2 Ġhours Ġfor ĠMandarin . ĠWe Ġuse Ġthe Ġsame ĠCNN Ġarchitecture Ġas Ġon ĠSwitch board Ġbut Ġmake Ġthe Ġchanges Ġas Ġdescribed Ġin ĠSection ĠSEC REF 11 Ġ. ĠAlso Ġwe Ġuse Ġmini - batch Ġsize Ġ30 Ġand Ġfix Ġthe Ġtraining Ġepoch s Ġas Ġ100 . ĠAll ĠCNN s Ġuse Ġword 2 vec Ġpre - training . ĠAdditionally , Ġwe Ġalso Ġimplement Ġanother Ġtwo Ġseparate Ġtopic ĠID Ġbas elines Ġusing Ġthe Ġdec oded Ġword Ġoutputs Ġfrom Ġtwo Ġsupervised ĠAS R Ġsystems , Ġtrained Ġfrom Ġ80 Ġhours Ġtransc ribed ĠBabel ĠTurkish Ġspeech ĠB IB REF 29 Ġand Ġabout Ġ170 Ġhours Ġtransc ribed ĠHK UST ĠMandarin Ġtelephone Ġspeech Ġ( L DC 2005 T 32 Ġand ĠL DC 2005 S 15 ), Ġrespectively . </s> As Ġshown Ġin ĠTable ĠT AB REF 18 Ġ, ĠUT D - based ĠSV Ms Ġare Ġmore Ġcompetitive Ġthan ĠAUD - based ĠSV Ms Ġon Ġthe Ġsmaller Ġcorpor a , Ġi . e ., ĠUzbek Ġand ĠMandarin , Ġwhile Ġbeing Ġless Ġcompetitive Ġon Ġthe Ġlarger Ġcorpus , ĠTurkish . ĠWe Ġfurther Ġinvestigate Ġthis Ġbehavior Ġon Ġeach Ġindividual Ġlanguage Ġby Ġvarying Ġthe Ġamount Ġof Ġtraining Ġdata ; Ġwe Ġsplit Ġthe Ġdata Ġinto Ġ10 Ġfolds , Ġand Ġperform Ġ10 - fold ĠCV Ġ9 Ġtimes , Ġvarying Ġthe Ġnumber Ġof Ġfolds Ġfor Ġtraining Ġfrom Ġ1 Ġto Ġ9 . ĠAs Ġillustrated Ġin ĠFigure ĠFIG REF 19 Ġfor ĠTurkish , Ġas Ġwe Ġuse Ġmore Ġfolds Ġfor Ġtraining , ĠAUD - based Ġsystem Ġstarts Ġto Ġbe Ġmore Ġcompetitive Ġthan ĠUT D . ĠSuper vised ĠAS R - based Ġsystems Ġstill Ġgive Ġthe Ġbest Ġresults Ġin Ġvarious Ġcases , Ġwhile ĠUT D Ġand ĠAUD Ġbased Ġsystems Ġgive Ġcomparable Ġperformance . </s> Note Ġthat ĠCNN - based Ġsystems Ġoutper form ĠSV Ms Ġon ĠTurkish Ġand ĠUzbek Ġwhile Ġlosing Ġon Ġthe Ġsmaller Ġsized ĠMandarin , Ġindicating Ġmore Ġtopic - label ed Ġdata Ġis Ġneeded Ġto Ġenable Ġcompetitive ĠCNN s . ĠThis Ġalso Ġindicates Ġwhy ĠCNN s Ġon ĠL ORE LE I Ġcorpor a Ġdo Ġnot Ġproduce Ġas Ġlarge Ġa Ġgain Ġover ĠSV Ms Ġas Ġon Ġthe Ġlarger Ġsized ĠSwitch board , Ġsince Ġeach Ġ15 - 25 Ġhour ĠL ORE LE I Ġcorpus Ġwith Ġ12 Ġtopic Ġlabels Ġis Ġa Ġrelatively Ġsmall Ġamount Ġof Ġdata Ġcompared Ġto Ġthe Ġ35 . 7 / 61 . 6 Ġhour ĠSwitch board Ġcorpus Ġwith Ġ6 Ġlabels . </s> Con cluding Ġremarks </s> We Ġhave Ġdemonstrated Ġthat Ġboth ĠUT D Ġand ĠAUD Ġare Ġviable Ġtechnologies Ġfor Ġproducing Ġeffective Ġtoken izations Ġof Ġspeech Ġthat Ġenable Ġtopic ĠID Ġperformance Ġcomparable Ġto Ġusing Ġstandard ĠAS R Ġsystems , Ġwhile Ġeffectively Ġremoving Ġthe Ġdependency Ġon Ġtransc ribed Ġspeech Ġrequired Ġby Ġthe ĠAS R Ġalternative . ĠWe Ġfind Ġthat Ġwhen Ġtraining Ġdata Ġis Ġseverely Ġlimited Ġthe ĠUT D - based Ġclassification Ġis Ġsuperior Ġto ĠAUD - based Ġclassification . ĠAs Ġthe Ġamount Ġof Ġtraining Ġdata Ġincreases , Ġperformance Ġimproves Ġacross Ġthe Ġboard . ĠFinally , Ġwith Ġsufficient Ġtraining Ġdata ĠAUD - based ĠCNN s Ġwith Ġword 2 vec Ġpre - training Ġoutper form ĠAUD - based ĠSV Ms .
Introduction Topic identification (topic ID) on speech aims to identify the topic(s) for given speech recordings, referred to as spoken documents, where the topics are a predefined set of classes or labels. This task is typically formulated as a three-step process. First, speech is tokenized into words or phones by automatic speech recognition (ASR) systems BIBREF0 , or by limited-vocabulary keyword spotting BIBREF1 . Second, standard text-based processing techniques are applied to the resulting tokenizations, and produce a vector representation for each spoken document, typically a bag-of-words multinomial representation, or a more compact vector given by probabilistic topic models BIBREF2 , BIBREF3 . Finally, topic ID is performed on the spoken document representations by supervised training of classifiers, such as Bayesian classifiers and support vector machines (SVMs). However, in the first step, training the ASR system required for tokenization itself requires transcribed speech and pronunciations. In this paper, we focus on a difficult and realistic scenario where the speech corpus of a test language is annotated only with a minimal number of topic labels, i.e., no manual transcriptions or dictionaries for building an ASR system are available. We aim to exploit approaches that enable topic ID on speech without any knowledge of that language other than the topic annotations. In this scenario, while previous work demonstrates that the cross-lingual phoneme recognizers can produce reasonable speech tokenizations BIBREF4 , BIBREF5 , the performance is highly dependent on the language and environmental condition (channel, noise, etc.) mismatch between the training and test data. Therefore, we focus on unsupervised approaches that operate directly on the speech of interest. Raw acoustic feature-based unsupervised term discovery (UTD) is one such approach that aims to identify and cluster repeating word-like units across speech based around segmental dynamic time warping (DTW) BIBREF6 , BIBREF7 . BIBREF8 shows that using the word-like units from UTD for spoken document classification can work well; however, the results in BIBREF8 are limited since the acoustic features on which UTD is performed are produced by acoustic models trained from the transcribed speech of its evaluation corpus. In this paper, we investigate UTD-based topic ID performance when UTD operates on language-independent speech representations extracted from multilingual bottleneck networks trained on languages other than the test language BIBREF9 . Another alternative to producing speech tokenizations without language dependency is the model-based approach, i.e., unsupervised learning of hidden Markov model (HMM) based phoneme-like units from untranscribed speech. We exploit the Variational Bayesian inference based acoustic unit discovery (AUD) framework in BIBREF10 that allows parallelized large-scale training. In topic ID tasks, such AUD-based systems have been shown to outperform other systems based on cross-lingual phoneme recognizers BIBREF5 , and this paper aims to further investigate how the performance compares among UTD, AUD and ASR based systems. Moreover, after the speech is tokenized, these works BIBREF0 , BIBREF1 , BIBREF4 , BIBREF5 , BIBREF8 , BIBREF9 are limited to using bag-of-words features as spoken document representations. While UTD only identifies relatively long (0.5 – 1 sec) repeated terms, AUD/ASR enables full-coverage segmentation of continuous speech into a sequence of units/words, and such a resulting temporal sequence enables another feature learning architecture based on convolutional neural networks (CNNs) BIBREF11 ; instead of treating the sequential tokens as a bag of acoustic units or words, the whole token sequence is encoded as concatenated continuous vectors, and followed by convolution and temporal pooling operations that capture the local and global dependencies. Such continuous space feature extraction frameworks have been used in various language processing tasks like spoken language understanding BIBREF12 , BIBREF13 and text classification BIBREF14 , BIBREF15 . However, three questions are worth investigating in our AUD-based setting: (i) if such a CNN-based framework can perform as well on noisy automatically discovered phoneme-like units as on orthographic words/characters, (ii) if pre-trained vectors of phoneme-like units from word2vec BIBREF16 provide superior performance to random initialization as evidenced by the word-based tasks, and (iii) if CNNs are still competitive in low-resource settings of hundreds to two-thousand training exemplars, rather than the large/medium sized datasets as in previous work BIBREF14 , BIBREF15 . Finally, incorporating the different tokenization and feature representation approaches noted above, we perform comprehensive topic ID evaluations on both single-label and multi-label spoken document classification tasks. Unsupervised tokenizations of speech Unsupervised term discovery (UTD) UTD aims to automatically identify and cluster repeated terms (e.g. words or phrases) from speech. To circumvent the exhaustive DTW-based search limited by INLINEFORM0 time BIBREF6 , we exploit the scalable UTD framework in the Zero Resource Toolkit (ZRTools) BIBREF7 , which permits search in INLINEFORM1 time. We briefly describe the UTD procedures in ZRTools by four steps below, and full details can be found in BIBREF7 . Construct the sparse approximate acoustic similarity matrices between pairs of speech utterances. Identify word repetitions via fast diagonal line search and segmental DTW. The resulting matches are used to construct an acoustic similarity graph, where nodes represent the matching acoustic segments and edges reflect DTW distances. Threshold the graph edges, and each connected component of the graph is a cluster of acoustic segments, which produces a corresponding term (word/phrase) category. Finally, the cluster of each discovered term category consists of a list of term occurrences. Note that in the third step above, the weight on each graph edge can be exact DTW-based similarity, or other similarity based on heuristics more than DTW distance. For example, we investigate an implementation in ZRTools, where a separate logistic regression model is used to rescore the similarity between identified matches by determining how likely the matching pair is the same underlying word/phrase and is not a filled pause (e.g. “um-hum” and “yeah uh-huh” in English). Filled pauses tend to be acoustically stationary with more phone repeats and thus would match throughout the acoustic similarity matrix, whereas a contentful word (without too many phone repeats) tend to concentrate around the main diagonal; thus, the features in logistic regression contain the numbers of matrix elements in diagonal bands in progressive steps away from the main diagonal. Feature weights are learned using a portion of transcribed speech with reference transcripts, and the resulting model can be used for language-independent rescoring. Acoustic unit discovery (AUD) We exploit the nonparametric Bayesian AUD framework in BIBREF10 based on variational inference, rather than the maximum likelihood training in BIBREF4 which may oversimplify the parameter estimations, nor the Gibbs Sampling training in BIBREF17 which is not amenable to large scale applications. Specifically, a phone-loop model is formulated where each phoneme-like unit is modeled as an HMM with a Gaussian mixture model of output densities (GMM-HMM). Under the Dirichlet process framework, we consider the phone loop as an infinite mixture of GMM-HMMs, and the mixture weights are based on the stick-breaking construction of Dirichlet process. The infinite number of units in the mixture is truncated in practice, giving zero mixture weight to any unit beyond some large count. We treat such mixture of GMM-HMMs as a single unified HMM and thus the segmentation of the data is performed using standard forward-backward algorithm. Training is fully unsupervised and parallelized; after a fixed number of training iterations, we use Viterbi decoding algorithm to obtain acoustic unit tokenizations of the data. Bag-of-words representation After we obtain the tokenizations of speech by either UTD or AUD, each spoken document is represented by a vector of unigram occurrence counts over discovered terms, or a vector of INLINEFORM0 -gram counts over acoustic units, respectively. Each feature vector can be further scaled by inverse document frequency (IDF), producing a TF-IDF feature. Convolutional neural network-based representation AUD enables full-coverage tokenization of continuous speech into a sequence of acoustic units, which we can exploit in a CNN-based framework to learn a vector representation for each spoken document. As shown in Figure FIGREF7 , in an acoustic unit sequence a of length INLINEFORM0 , each unit INLINEFORM1 , INLINEFORM2 , is encoded as a fixed dimensional continuous vector, and the whole sequence a is represented as a concatenated vector x. A shared convolutional feature transform INLINEFORM3 spans a fixed-sized INLINEFORM4 -gram window, INLINEFORM5 , and slides over the whole sequence. Then the hidden feature layer INLINEFORM6 with nonlinearities consists of each feature vector INLINEFORM7 extracted from the shared convolutional window centered at each acoustic unit position INLINEFORM8 . Max-pooling is performed on top of each INLINEFORM9 , INLINEFORM10 , to obtain a fixed-dimensional vector representation for the whole sequence a, i.e., a vector representation of the whole spoken document, followed by another hidden layer INLINEFORM11 and a final output layer. Note that this framework needs supervision for training; e.g., the output layer can be a softmax function for single-label classification, and the whole model is trained with categorical cross-entropy loss. Also, the vector representation of each unique acoustic unit can be randomly initialized, or pre-trained from other tasks. Specifically, we apply the skip-gram model of word2vec BIBREF18 to pre-train one embedding vector for each acoustic unit, based on the hierarchical softmax with Huffman codes. Single-label classification For the bag-of-words representation, we use a stochastic gradient descent (SGD) based linear SVM BIBREF19 , BIBREF20 with hinge loss and INLINEFORM0 / INLINEFORM1 norm regularization. For the CNN-based framework, we use a softmax function in the output layer for classification as described in Section SECREF9 . For our single-label topic classification experiments, we use the Switchboard Telephone Speech Corpus BIBREF21 , a collection of two-sided telephone conversations. We use the same development (dev) and evaluation (eval) data sets as in BIBREF8 , BIBREF9 . Each whole conversation has two sides and one single topic, and topic ID is performed on each individual-side speech (i.e., each side is seen as one single spoken document). In the 35.7 hour dev data, there are 360 conversation sides evenly distributed across six different topics (recycling, capital punishment, drug testing, family finance, job benefits, car buying), i.e., each topic has equal number of 60 sides. In the 61.6 hour eval data, there are another different six topics (family life, news media, public education, exercise/fitness, pets, taxes) evenly distributed across 600 conversation sides. Algorithm design choices are explored through experiments on dev data. We use manual segmentations provided by the Switchboard corpus to produce utterances with speech activity, and UTD and AUD are operating only on those utterances. For UTD, we use the ZRTools BIBREF7 implementation with the default parameters except that, we use cosine similarity threshold INLINEFORM0 , and vary the diagonal median filter duration INLINEFORM1 over INLINEFORM2 ; we try both the exact DTW-based similarity and the rescored similarity as described in Section SECREF1 , and tune the similarity threshold (used to partition the graph edges) over INLINEFORM3 . For AUD, the unsupervised training is performed only on the dev data (10 iterations); after training, we use the learned models to decode both dev and eval data set, and obtain the acoustic unit tokenizations. We use truncation level 200, which implies maximum 200 different acoustic units can be learned from the corpus. For each acoustic unit, we use a 3-state HMM with 2 Gaussians per state. For the stick-breaking construction of Dirichlet process, we vary the concentration parameter INLINEFORM4 over INLINEFORM5 , and other hyperparameters are the same as BIBREF10 . The acoustic features on which UTD and AUD operate are extracted using the same multilingual bottleneck (BN) network as described in BIBREF9 with Kaldi toolkit BIBREF22 . We conduct the multilingual BN training with 10 language collections (Assamese, Bengali, Cantonese, Haitian, Lao, Pashto, Tamil, Tagalog, Vietnamese and Zulu) – 10 hours of transcribed speech per language. Complete specifications can be found in BIBREF9 . For SVM-based classification, we use the bag of discovered term unigrams, or bag of acoustic unit trigrams. On dev data, we try using the features of raw counts or scaled by IDF, SVM regularization tuned over INLINEFORM0 / INLINEFORM1 norm, regularization constant tuned over INLINEFORM2 , and SGD epochs tuned over INLINEFORM3 . We further normalize each feature to INLINEFORM4 norm unit length. Each experiment is a run of 10-fold cross validation (CV) on the 360 conversation sides of dev data, or on the 600 sides of eval data, respectively. Note that our data size here is relatively small (only 360 or 600) and the SGD training may give high variance in the performance BIBREF23 . Therefore, to report classification accuracy for each configuration (when varying features or models), we repeat each CV experiment 5 times, where each experiment again is a run of 10-fold CV; then for each configuration, the mean and standard deviation of 5 experiments is reported. For CNN-based classification, we use the same strategy to report classification accuracy, i.e., repeating experiments 5 times (where each time is a 10-fold CV) for each CNN configuration. Note that the respective 10 folds of both dev and eval data sets are fixed the same for all the SVM and CNN experiments. Additionally, for each 10-fold CV experiment, instead of training on 9 folds and testing on the remaining 1 fold as in SVM, we use 8 folds for CNN training, leave another 1 fold out as validation data; after training each CNN model for up to 100 epochs, the model with the best accuracy on the validation data is used for evaluation on the test set. The acoustic unit sequence (as CNN inputs) are zero-padded to the longest length in each dataset. We implemented the CNNs in Keras BIBREF24 with Theano BIBREF25 backend. CNN architectures are determined through experiments on dev data. For SGD training we use the Adadelta optimizer BIBREF26 and mini-batch size 18. The INLINEFORM0 -gram window size of each convolutional feature transform INLINEFORM1 is 7. The size of each hidden feature vector INLINEFORM2 (extracted from the transform INLINEFORM3 ) is 1024, with rectified linear unit (ReLU) nonlinearities. Thus, after max-pooling over time, we have a 1024-dimensional vector again, which then goes through another hidden layer INLINEFORM4 (also set as 1024-dimensional with ReLU) and finally into a softmax. Dropout BIBREF27 rate 0.2 is used at each layer. When we initialize the vector representation of each acoustic unit with a set of pre-trained vectors (instead of random initializations), we apply the skip-gram model of word2vec BIBREF18 to the acoustic unit tokenizations of each data set. We use the gensim implementation BIBREF28 , which includes a vector space of embedding dimension 50 (tuned over INLINEFORM0 ), a skip-gram window of size 5, and SGD over 20 epochs. Table TABREF15 shows the topic ID results on Switchboard. For UTD-based classifications, we find that the default rescoring in ZRTools BIBREF7 which is designed to filter out the filled pauses produces comparable performance to the raw DTW similarity scores, but the rescoring can result in much faster connected-component clustering (Section SECREF1 ). Note that this rescoring model is estimated using a portion of transcribed Switchboard, but it is still a legitimate language-independent UTD approach while operating on languages other than English. While a diagonal median filter duration INLINEFORM0 of INLINEFORM1 or INLINEFORM2 gives similar results, INLINEFORM3 produces longer but fewer terms, giving more sparse feature representations. Therefore, we proceed with rescoring and INLINEFORM4 in the following UTD experiments (Section SECREF16 ). For AUD-based classifications, CNN without word2vec pre-training usually gives comparable results with SVM; however, using word2vec pre-training, CNN substantially outperforms the competing SVM in all cases. Also as the concentration parameter INLINEFORM0 in AUD increases from INLINEFORM1 to INLINEFORM2 (yielding less concentrated distributions), we have more unique acoustic units in the tokenizations of both data sets, from 184 to 199, and INLINEFORM3 usually produces better results than INLINEFORM4 . Multi-label classification In the setting where each spoken document can be associated with multiple topics/labels, we proceed to perform a multi-label classification task. The baseline approach is the binary relevance method, which independently trains one binary classifier for each label, and the spoken document is evaluated by each classifier to determine if the respective label applies to it. Specifically, we use a set of SVMs (Section SECREF10 ), one for each label, on the bag-of-words features. To adapt the CNN-based framework for multi-label classification, we replace the softmax in the output layer with a set of sigmoid output nodes, one for each label, as shown in Figure FIGREF7 . Since a sigmoid naturally provides output values between 0 and 1, we train the neural network (NN) to minimize the binary cross entropy loss defined as INLINEFORM0 , where INLINEFORM1 denotes the NN parameters, x is the feature vector of acoustic unit sequence, y is the target vector of labels, INLINEFORM2 and INLINEFORM3 are the output and the target for label INLINEFORM4 , and the number of unique labels is INLINEFORM5 . We further evaluate our topic ID performance on the speech corpora of three languages released by the DARPA LORELEI (Low Resource Languages for Emergent Incidents) Program. For each language there are a number of audio speech files, and each speech file is cut into segments of various lengths (up to 120 seconds). Each speech segment is seen as either in-domain or out-of-domain. In-domain data is defined as any speech segment relating to an incident or incidents, and in-domain data will fall into a set of domain-specific categories; these categories are known as situation types, or in-domain topics. There are 11 situation types: “Civil Unrest or Wide-spread Crime”, “Elections and Politics”, “Evacuation”, “Food Supply”, “Urgent Rescue”, “Utilities, Energy, or Sanitation”, “Infrastructure”, “Medical Assistance”, “Shelter”, “Terrorism or other Extreme Violence”, and “Water Supply”. We consider “Out-of-domain” as the 12th topic label, so each speech segment either corresponds to one or multiple in-domain topics, or is “Out-of-domain”. We use the average precision (AP, equal to the area under the precision-recall curve) as the evaluation metric, and report both the AP across the overall 12 labels, and the AP across 11 situation types, as shown in Table TABREF18 . For each configuration, only a single 10-fold CV result is reported, since we observe less variance in results here than in Switchboard. We have 16.5 hours in-domain data and 8.5 hours out-of-domain data for Turkish, 2.9 and 13.2 hours for Uzbek, and 7.7 and 7.2 hours for Mandarin. We use the same CNN architecture as on Switchboard but make the changes as described in Section SECREF11 . Also we use mini-batch size 30 and fix the training epochs as 100. All CNNs use word2vec pre-training. Additionally, we also implement another two separate topic ID baselines using the decoded word outputs from two supervised ASR systems, trained from 80 hours transcribed Babel Turkish speech BIBREF29 and about 170 hours transcribed HKUST Mandarin telephone speech (LDC2005T32 and LDC2005S15), respectively. As shown in Table TABREF18 , UTD-based SVMs are more competitive than AUD-based SVMs on the smaller corpora, i.e., Uzbek and Mandarin, while being less competitive on the larger corpus, Turkish. We further investigate this behavior on each individual language by varying the amount of training data; we split the data into 10 folds, and perform 10-fold CV 9 times, varying the number of folds for training from 1 to 9. As illustrated in Figure FIGREF19 for Turkish, as we use more folds for training, AUD-based system starts to be more competitive than UTD. Supervised ASR-based systems still give the best results in various cases, while UTD and AUD based systems give comparable performance. Note that CNN-based systems outperform SVMs on Turkish and Uzbek while losing on the smaller sized Mandarin, indicating more topic-labeled data is needed to enable competitive CNNs. This also indicates why CNNs on LORELEI corpora do not produce as large a gain over SVMs as on the larger sized Switchboard, since each 15-25 hour LORELEI corpus with 12 topic labels is a relatively small amount of data compared to the 35.7/61.6 hour Switchboard corpus with 6 labels. Concluding remarks We have demonstrated that both UTD and AUD are viable technologies for producing effective tokenizations of speech that enable topic ID performance comparable to using standard ASR systems, while effectively removing the dependency on transcribed speech required by the ASR alternative. We find that when training data is severely limited the UTD-based classification is superior to AUD-based classification. As the amount of training data increases, performance improves across the board. Finally, with sufficient training data AUD-based CNNs with word2vec pre-training outperform AUD-based SVMs.
What datasets are used to assess the performance of the system?
01e2d10178347d177519f792f86f25575106ddc7
What Ġdatasets Ġare Ġused Ġto Ġassess Ġthe Ġperformance Ġof Ġthe Ġsystem ?
Switch board ĠTelephone ĠSpeech ĠCorpus ĠB IB REF 21 , ĠL ORE LE I Ġ( Low ĠResource ĠLanguages Ġfor ĠEmer gent ĠInc idents ) ĠProgram
<s> What Ġdatasets Ġare Ġused Ġto Ġassess Ġthe Ġperformance Ġof Ġthe Ġsystem ? </s> Introduction </s> Topic Ġidentification Ġ( topic ĠID ) Ġon Ġspeech Ġaims Ġto Ġidentify Ġthe Ġtopic ( s ) Ġfor Ġgiven Ġspeech Ġrecordings , Ġreferred Ġto Ġas Ġspoken Ġdocuments , Ġwhere Ġthe Ġtopics Ġare Ġa Ġpred efined Ġset Ġof Ġclasses Ġor Ġlabels . ĠThis Ġtask Ġis Ġtypically Ġformulated Ġas Ġa Ġthree - step Ġprocess . ĠFirst , Ġspeech Ġis Ġtoken ized Ġinto Ġwords Ġor Ġphones Ġby Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems ĠB IB REF 0 Ġ, Ġor Ġby Ġlimited - voc abulary Ġkeyword Ġspotting ĠB IB REF 1 Ġ. ĠSecond , Ġstandard Ġtext - based Ġprocessing Ġtechniques Ġare Ġapplied Ġto Ġthe Ġresulting Ġtoken izations , Ġand Ġproduce Ġa Ġvector Ġrepresentation Ġfor Ġeach Ġspoken Ġdocument , Ġtypically Ġa Ġbag - of - words Ġmult in omial Ġrepresentation , Ġor Ġa Ġmore Ġcompact Ġvector Ġgiven Ġby Ġprob abil istic Ġtopic Ġmodels ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠFinally , Ġtopic ĠID Ġis Ġperformed Ġon Ġthe Ġspoken Ġdocument Ġrepresentations Ġby Ġsupervised Ġtraining Ġof Ġclass ifiers , Ġsuch Ġas ĠBay esian Ġclass ifiers Ġand Ġsupport Ġvector Ġmachines Ġ( S V Ms ). </s> However , Ġin Ġthe Ġfirst Ġstep , Ġtraining Ġthe ĠAS R Ġsystem Ġrequired Ġfor Ġtoken ization Ġitself Ġrequires Ġtransc ribed Ġspeech Ġand Ġpron unci ations . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġa Ġdifficult Ġand Ġrealistic Ġscenario Ġwhere Ġthe Ġspeech Ġcorpus Ġof Ġa Ġtest Ġlanguage Ġis Ġannot ated Ġonly Ġwith Ġa Ġminimal Ġnumber Ġof Ġtopic Ġlabels , Ġi . e ., Ġno Ġmanual Ġtranscript ions Ġor Ġdiction aries Ġfor Ġbuilding Ġan ĠAS R Ġsystem Ġare Ġavailable . ĠWe Ġaim Ġto Ġexploit Ġapproaches Ġthat Ġenable Ġtopic ĠID Ġon Ġspeech Ġwithout Ġany Ġknowledge Ġof Ġthat Ġlanguage Ġother Ġthan Ġthe Ġtopic Ġannotations . </s> In Ġthis Ġscenario , Ġwhile Ġprevious Ġwork Ġdemonstrates Ġthat Ġthe Ġcross - ling ual Ġphon eme Ġrecogn izers Ġcan Ġproduce Ġreasonable Ġspeech Ġtoken izations ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġthe Ġperformance Ġis Ġhighly Ġdependent Ġon Ġthe Ġlanguage Ġand Ġenvironmental Ġcondition Ġ( channel , Ġnoise , Ġetc .) Ġmismatch Ġbetween Ġthe Ġtraining Ġand Ġtest Ġdata . ĠTherefore , Ġwe Ġfocus Ġon Ġun super vised Ġapproaches Ġthat Ġoperate Ġdirectly Ġon Ġthe Ġspeech Ġof Ġinterest . ĠRaw Ġacoustic Ġfeature - based Ġun super vised Ġterm Ġdiscovery Ġ( UT D ) Ġis Ġone Ġsuch Ġapproach Ġthat Ġaims Ġto Ġidentify Ġand Ġcluster Ġrepeating Ġword - like Ġunits Ġacross Ġspeech Ġbased Ġaround Ġsegment al Ġdynamic Ġtime Ġwar ping Ġ( DT W ) ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠB IB REF 8 Ġshows Ġthat Ġusing Ġthe Ġword - like Ġunits Ġfrom ĠUT D Ġfor Ġspoken Ġdocument Ġclassification Ġcan Ġwork Ġwell ; Ġhowever , Ġthe Ġresults Ġin ĠB IB REF 8 Ġare Ġlimited Ġsince Ġthe Ġacoustic Ġfeatures Ġon Ġwhich ĠUT D Ġis Ġperformed Ġare Ġproduced Ġby Ġacoustic Ġmodels Ġtrained Ġfrom Ġthe Ġtransc ribed Ġspeech Ġof Ġits Ġevaluation Ġcorpus . ĠIn Ġthis Ġpaper , Ġwe Ġinvestigate ĠUT D - based Ġtopic ĠID Ġperformance Ġwhen ĠUT D Ġoperates Ġon Ġlanguage - independent Ġspeech Ġrepresentations Ġextracted Ġfrom Ġmult ilingual Ġbottleneck Ġnetworks Ġtrained Ġon Ġlanguages Ġother Ġthan Ġthe Ġtest Ġlanguage ĠB IB REF 9 Ġ. ĠAnother Ġalternative Ġto Ġproducing Ġspeech Ġtoken izations Ġwithout Ġlanguage Ġdependency Ġis Ġthe Ġmodel - based Ġapproach , Ġi . e ., Ġun super vised Ġlearning Ġof Ġhidden ĠMark ov Ġmodel Ġ( H MM ) Ġbased Ġphon eme - like Ġunits Ġfrom Ġunt rans cribed Ġspeech . ĠWe Ġexploit Ġthe ĠVari ational ĠBay esian Ġinference Ġbased Ġacoustic Ġunit Ġdiscovery Ġ( AUD ) Ġframework Ġin ĠB IB REF 10 Ġthat Ġallows Ġparallel ized Ġlarge - scale Ġtraining . ĠIn Ġtopic ĠID Ġtasks , Ġsuch ĠAUD - based Ġsystems Ġhave Ġbeen Ġshown Ġto Ġoutper form Ġother Ġsystems Ġbased Ġon Ġcross - ling ual Ġphon eme Ġrecogn izers ĠB IB REF 5 Ġ, Ġand Ġthis Ġpaper Ġaims Ġto Ġfurther Ġinvestigate Ġhow Ġthe Ġperformance Ġcompares Ġamong ĠUT D , ĠAUD Ġand ĠAS R Ġbased Ġsystems . </s> Moreover , Ġafter Ġthe Ġspeech Ġis Ġtoken ized , Ġthese Ġworks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġare Ġlimited Ġto Ġusing Ġbag - of - words Ġfeatures Ġas Ġspoken Ġdocument Ġrepresentations . ĠWhile ĠUT D Ġonly Ġidentifies Ġrelatively Ġlong Ġ( 0 . 5 ĠâĢĵ Ġ1 Ġsec ) Ġrepeated Ġterms , ĠAUD / AS R Ġenables Ġfull - co verage Ġsegment ation Ġof Ġcontinuous Ġspeech Ġinto Ġa Ġsequence Ġof Ġunits / words , Ġand Ġsuch Ġa Ġresulting Ġtemporal Ġsequence Ġenables Ġanother Ġfeature Ġlearning Ġarchitecture Ġbased Ġon Ġconv olution al Ġneural Ġnetworks Ġ( CNN s ) ĠB IB REF 11 Ġ; Ġinstead Ġof Ġtreating Ġthe Ġsequential Ġtokens Ġas Ġa Ġbag Ġof Ġacoustic Ġunits Ġor Ġwords , Ġthe Ġwhole Ġtoken Ġsequence Ġis Ġencoded Ġas Ġconc aten ated Ġcontinuous Ġvectors , Ġand Ġfollowed Ġby Ġconv olution Ġand Ġtemporal Ġpool ing Ġoperations Ġthat Ġcapture Ġthe Ġlocal Ġand Ġglobal Ġdependencies . ĠSuch Ġcontinuous Ġspace Ġfeature Ġextraction Ġframeworks Ġhave Ġbeen Ġused Ġin Ġvarious Ġlanguage Ġprocessing Ġtasks Ġlike Ġspoken Ġlanguage Ġunderstanding ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġand Ġtext Ġclassification ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠHowever , Ġthree Ġquestions Ġare Ġworth Ġinvestigating Ġin Ġour ĠAUD - based Ġsetting : Ġ( i ) Ġif Ġsuch Ġa ĠCNN - based Ġframework Ġcan Ġperform Ġas Ġwell Ġon Ġnoisy Ġautomatically Ġdiscovered Ġphon eme - like Ġunits Ġas Ġon Ġorth ographic Ġwords / char acters , Ġ( ii ) Ġif Ġpre - trained Ġvectors Ġof Ġphon eme - like Ġunits Ġfrom Ġword 2 vec ĠB IB REF 16 Ġprovide Ġsuperior Ġperformance Ġto Ġrandom Ġinitialization Ġas Ġevidenced Ġby Ġthe Ġword - based Ġtasks , Ġand Ġ( iii ) Ġif ĠCNN s Ġare Ġstill Ġcompetitive Ġin Ġlow - resource Ġsettings Ġof Ġhundreds Ġto Ġtwo - th ousand Ġtraining Ġexempl ars , Ġrather Ġthan Ġthe Ġlarge / medium Ġsized Ġdatasets Ġas Ġin Ġprevious Ġwork ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. </s> Finally , Ġincorporating Ġthe Ġdifferent Ġtoken ization Ġand Ġfeature Ġrepresentation Ġapproaches Ġnoted Ġabove , Ġwe Ġperform Ġcomprehensive Ġtopic ĠID Ġevaluations Ġon Ġboth Ġsingle - label Ġand Ġmulti - label Ġspoken Ġdocument Ġclassification Ġtasks . </s> Un super vised Ġtoken izations Ġof Ġspeech </s> Un super vised Ġterm Ġdiscovery Ġ( UT D ) </s> UT D Ġaims Ġto Ġautomatically Ġidentify Ġand Ġcluster Ġrepeated Ġterms Ġ( e . g . Ġwords Ġor Ġphrases ) Ġfrom Ġspeech . ĠTo Ġcircumvent Ġthe Ġexhaustive ĠDT W - based Ġsearch Ġlimited Ġby ĠIN LINE FORM 0 Ġtime ĠB IB REF 6 Ġ, Ġwe Ġexploit Ġthe Ġscalable ĠUT D Ġframework Ġin Ġthe ĠZero ĠResource ĠTool kit Ġ( Z RT ools ) ĠB IB REF 7 Ġ, Ġwhich Ġpermits Ġsearch Ġin ĠIN LINE FORM 1 Ġtime . ĠWe Ġbriefly Ġdescribe Ġthe ĠUT D Ġprocedures Ġin ĠZ RT ools Ġby Ġfour Ġsteps Ġbelow , Ġand Ġfull Ġdetails Ġcan Ġbe Ġfound Ġin ĠB IB REF 7 Ġ. </s> Construct Ġthe Ġsparse Ġapproximate Ġacoustic Ġsimilarity Ġmat rices Ġbetween Ġpairs Ġof Ġspeech Ġutter ances . </s> Ident ify Ġword Ġrepet itions Ġvia Ġfast Ġdiagonal Ġline Ġsearch Ġand Ġsegment al ĠDT W . </s> The Ġresulting Ġmatches Ġare Ġused Ġto Ġconstruct Ġan Ġacoustic Ġsimilarity Ġgraph , Ġwhere Ġnodes Ġrepresent Ġthe Ġmatching Ġacoustic Ġsegments Ġand Ġedges Ġreflect ĠDT W Ġdistances . </s> Th reshold Ġthe Ġgraph Ġedges , Ġand Ġeach Ġconnected Ġcomponent Ġof Ġthe Ġgraph Ġis Ġa Ġcluster Ġof Ġacoustic Ġsegments , Ġwhich Ġproduces Ġa Ġcorresponding Ġterm Ġ( word / phrase ) Ġcategory . </s> Finally , Ġthe Ġcluster Ġof Ġeach Ġdiscovered Ġterm Ġcategory Ġconsists Ġof Ġa Ġlist Ġof Ġterm Ġoccurrences . </s> Note Ġthat Ġin Ġthe Ġthird Ġstep Ġabove , Ġthe Ġweight Ġon Ġeach Ġgraph Ġedge Ġcan Ġbe Ġexact ĠDT W - based Ġsimilarity , Ġor Ġother Ġsimilarity Ġbased Ġon Ġhe ur istics Ġmore Ġthan ĠDT W Ġdistance . ĠFor Ġexample , Ġwe Ġinvestigate Ġan Ġimplementation Ġin ĠZ RT ools , Ġwhere Ġa Ġseparate Ġlog istic Ġregression Ġmodel Ġis Ġused Ġto Ġresc ore Ġthe Ġsimilarity Ġbetween Ġidentified Ġmatches Ġby Ġdetermining Ġhow Ġlikely Ġthe Ġmatching Ġpair Ġis Ġthe Ġsame Ġunderlying Ġword / phrase Ġand Ġis Ġnot Ġa Ġfilled Ġpause Ġ( e . g . ĠâĢ ľ um - hum âĢ Ŀ Ġand ĠâĢ ľ yeah Ġuh - h uh âĢ Ŀ Ġin ĠEnglish ). ĠF illed Ġpauses Ġtend Ġto Ġbe Ġac oust ically Ġstationary Ġwith Ġmore Ġphone Ġrepeats Ġand Ġthus Ġwould Ġmatch Ġthroughout Ġthe Ġacoustic Ġsimilarity Ġmatrix , Ġwhereas Ġa Ġcontent ful Ġword Ġ( without Ġtoo Ġmany Ġphone Ġrepeats ) Ġtend Ġto Ġconcentrate Ġaround Ġthe Ġmain Ġdiagonal ; Ġthus , Ġthe Ġfeatures Ġin Ġlog istic Ġregression Ġcontain Ġthe Ġnumbers Ġof Ġmatrix Ġelements Ġin Ġdiagonal Ġbands Ġin Ġprogressive Ġsteps Ġaway Ġfrom Ġthe Ġmain Ġdiagonal . ĠFeature Ġweights Ġare Ġlearned Ġusing Ġa Ġportion Ġof Ġtransc ribed Ġspeech Ġwith Ġreference Ġtranscripts , Ġand Ġthe Ġresulting Ġmodel Ġcan Ġbe Ġused Ġfor Ġlanguage - independent Ġresc oring . </s> Ac oustic Ġunit Ġdiscovery Ġ( AUD ) </s> We Ġexploit Ġthe Ġnon param etric ĠBay esian ĠAUD Ġframework Ġin ĠB IB REF 10 Ġbased Ġon Ġvari ational Ġinference , Ġrather Ġthan Ġthe Ġmaximum Ġlikelihood Ġtraining Ġin ĠB IB REF 4 Ġwhich Ġmay Ġovers impl ify Ġthe Ġparameter Ġestim ations , Ġnor Ġthe ĠGibbs ĠSam pling Ġtraining Ġin ĠB IB REF 17 Ġwhich Ġis Ġnot Ġam enable Ġto Ġlarge Ġscale Ġapplications . ĠSpecifically , Ġa Ġphone - loop Ġmodel Ġis Ġformulated Ġwhere Ġeach Ġphon eme - like Ġunit Ġis Ġmodeled Ġas Ġan ĠH MM Ġwith Ġa ĠGa ussian Ġmixture Ġmodel Ġof Ġoutput Ġdens ities Ġ( G MM - H MM ). ĠUnder Ġthe ĠDir ich let Ġprocess Ġframework , Ġwe Ġconsider Ġthe Ġphone Ġloop Ġas Ġan Ġinfinite Ġmixture Ġof ĠGM M - HM Ms , Ġand Ġthe Ġmixture Ġweights Ġare Ġbased Ġon Ġthe Ġstick - breaking Ġconstruction Ġof ĠDir ich let Ġprocess . ĠThe Ġinfinite Ġnumber Ġof Ġunits Ġin Ġthe Ġmixture Ġis Ġtrunc ated Ġin Ġpractice , Ġgiving Ġzero Ġmixture Ġweight Ġto Ġany Ġunit Ġbeyond Ġsome Ġlarge Ġcount . ĠWe Ġtreat Ġsuch Ġmixture Ġof ĠGM M - HM Ms Ġas Ġa Ġsingle Ġunified ĠH MM Ġand Ġthus Ġthe Ġsegment ation Ġof Ġthe Ġdata Ġis Ġperformed Ġusing Ġstandard Ġforward - back ward Ġalgorithm . ĠTraining Ġis Ġfully Ġun super vised Ġand Ġparallel ized ; Ġafter Ġa Ġfixed Ġnumber Ġof Ġtraining Ġiterations , Ġwe Ġuse ĠV iter bi Ġdecoding Ġalgorithm Ġto Ġobtain Ġacoustic Ġunit Ġtoken izations Ġof Ġthe Ġdata . </s> B ag - of - words Ġrepresentation </s> After Ġwe Ġobtain Ġthe Ġtoken izations Ġof Ġspeech Ġby Ġeither ĠUT D Ġor ĠAUD , Ġeach Ġspoken Ġdocument Ġis Ġrepresented Ġby Ġa Ġvector Ġof Ġun ig ram Ġoccurrence Ġcounts Ġover Ġdiscovered Ġterms , Ġor Ġa Ġvector Ġof ĠIN LINE FORM 0 Ġ- gram Ġcounts Ġover Ġacoustic Ġunits , Ġrespectively . ĠEach Ġfeature Ġvector Ġcan Ġbe Ġfurther Ġscaled Ġby Ġinverse Ġdocument Ġfrequency Ġ( ID F ), Ġproducing Ġa ĠTF - ID F Ġfeature . </s> Con v olution al Ġneural Ġnetwork - based Ġrepresentation </s> AUD Ġenables Ġfull - co verage Ġtoken ization Ġof Ġcontinuous Ġspeech Ġinto Ġa Ġsequence Ġof Ġacoustic Ġunits , Ġwhich Ġwe Ġcan Ġexploit Ġin Ġa ĠCNN - based Ġframework Ġto Ġlearn Ġa Ġvector Ġrepresentation Ġfor Ġeach Ġspoken Ġdocument . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 7 Ġ, Ġin Ġan Ġacoustic Ġunit Ġsequence Ġa Ġof Ġlength ĠIN LINE FORM 0 Ġ, Ġeach Ġunit ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġis Ġencoded Ġas Ġa Ġfixed Ġdimensional Ġcontinuous Ġvector , Ġand Ġthe Ġwhole Ġsequence Ġa Ġis Ġrepresented Ġas Ġa Ġconc aten ated Ġvector Ġx . ĠA Ġshared Ġconv olution al Ġfeature Ġtransform ĠIN LINE FORM 3 Ġspans Ġa Ġfixed - sized ĠIN LINE FORM 4 Ġ- gram Ġwindow , ĠIN LINE FORM 5 Ġ, Ġand Ġslides Ġover Ġthe Ġwhole Ġsequence . ĠThen Ġthe Ġhidden Ġfeature Ġlayer ĠIN LINE FORM 6 Ġwith Ġnon linear ities Ġconsists Ġof Ġeach Ġfeature Ġvector ĠIN LINE FORM 7 Ġextracted Ġfrom Ġthe Ġshared Ġconv olution al Ġwindow Ġcentered Ġat Ġeach Ġacoustic Ġunit Ġposition ĠIN LINE FORM 8 Ġ. ĠMax - pool ing Ġis Ġperformed Ġon Ġtop Ġof Ġeach ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġ, Ġto Ġobtain Ġa Ġfixed - dimensional Ġvector Ġrepresentation Ġfor Ġthe Ġwhole Ġsequence Ġa , Ġi . e ., Ġa Ġvector Ġrepresentation Ġof Ġthe Ġwhole Ġspoken Ġdocument , Ġfollowed Ġby Ġanother Ġhidden Ġlayer ĠIN LINE FORM 11 Ġand Ġa Ġfinal Ġoutput Ġlayer . ĠNote Ġthat Ġthis Ġframework Ġneeds Ġsupervision Ġfor Ġtraining ; Ġe . g ., Ġthe Ġoutput Ġlayer Ġcan Ġbe Ġa Ġsoft max Ġfunction Ġfor Ġsingle - label Ġclassification , Ġand Ġthe Ġwhole Ġmodel Ġis Ġtrained Ġwith Ġcateg orical Ġcross - ent ropy Ġloss . </s> Also , Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġunique Ġacoustic Ġunit Ġcan Ġbe Ġrandomly Ġinitialized , Ġor Ġpre - trained Ġfrom Ġother Ġtasks . ĠSpecifically , Ġwe Ġapply Ġthe Ġskip - gram Ġmodel Ġof Ġword 2 vec ĠB IB REF 18 Ġto Ġpre - train Ġone Ġembed ding Ġvector Ġfor Ġeach Ġacoustic Ġunit , Ġbased Ġon Ġthe Ġhierarchical Ġsoft max Ġwith ĠHuff man Ġcodes . </s> Single - label Ġclassification </s> For Ġthe Ġbag - of - words Ġrepresentation , Ġwe Ġuse Ġa Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) Ġbased Ġlinear ĠS VM ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġwith Ġhinge Ġloss Ġand ĠIN LINE FORM 0 Ġ/ ĠIN LINE FORM 1 Ġnorm Ġregular ization . ĠFor Ġthe ĠCNN - based Ġframework , Ġwe Ġuse Ġa Ġsoft max Ġfunction Ġin Ġthe Ġoutput Ġlayer Ġfor Ġclassification Ġas Ġdescribed Ġin ĠSection ĠSEC REF 9 Ġ. </s> For Ġour Ġsingle - label Ġtopic Ġclassification Ġexperiments , Ġwe Ġuse Ġthe ĠSwitch board ĠTelephone ĠSpeech ĠCorpus ĠB IB REF 21 Ġ, Ġa Ġcollection Ġof Ġtwo - sided Ġtelephone Ġconversations . ĠWe Ġuse Ġthe Ġsame Ġdevelopment Ġ( dev ) Ġand Ġevaluation Ġ( eval ) Ġdata Ġsets Ġas Ġin ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠEach Ġwhole Ġconversation Ġhas Ġtwo Ġsides Ġand Ġone Ġsingle Ġtopic , Ġand Ġtopic ĠID Ġis Ġperformed Ġon Ġeach Ġindividual - side Ġspeech Ġ( i . e ., Ġeach Ġside Ġis Ġseen Ġas Ġone Ġsingle Ġspoken Ġdocument ). ĠIn Ġthe Ġ35 . 7 Ġhour Ġdev Ġdata , Ġthere Ġare Ġ360 Ġconversation Ġsides Ġevenly Ġdistributed Ġacross Ġsix Ġdifferent Ġtopics Ġ( recy cling , Ġcapital Ġpunishment , Ġdrug Ġtesting , Ġfamily Ġfinance , Ġjob Ġbenefits , Ġcar Ġbuying ), Ġi . e ., Ġeach Ġtopic Ġhas Ġequal Ġnumber Ġof Ġ60 Ġsides . ĠIn Ġthe Ġ61 . 6 Ġhour Ġeval Ġdata , Ġthere Ġare Ġanother Ġdifferent Ġsix Ġtopics Ġ( family Ġlife , Ġnews Ġmedia , Ġpublic Ġeducation , Ġexercise / f itness , Ġpets , Ġtaxes ) Ġevenly Ġdistributed Ġacross Ġ600 Ġconversation Ġsides . ĠAl gorithm Ġdesign Ġchoices Ġare Ġexplored Ġthrough Ġexperiments Ġon Ġdev Ġdata . ĠWe Ġuse Ġmanual Ġsegment ations Ġprovided Ġby Ġthe ĠSwitch board Ġcorpus Ġto Ġproduce Ġutter ances Ġwith Ġspeech Ġactivity , Ġand ĠUT D Ġand ĠAUD Ġare Ġoperating Ġonly Ġon Ġthose Ġutter ances . </s> For ĠUT D , Ġwe Ġuse Ġthe ĠZ RT ools ĠB IB REF 7 Ġimplementation Ġwith Ġthe Ġdefault Ġparameters Ġexcept Ġthat , Ġwe Ġuse Ġcos ine Ġsimilarity Ġthreshold ĠIN LINE FORM 0 Ġ, Ġand Ġvary Ġthe Ġdiagonal Ġmedian Ġfilter Ġduration ĠIN LINE FORM 1 Ġover ĠIN LINE FORM 2 Ġ; Ġwe Ġtry Ġboth Ġthe Ġexact ĠDT W - based Ġsimilarity Ġand Ġthe Ġresc ored Ġsimilarity Ġas Ġdescribed Ġin ĠSection ĠSEC REF 1 Ġ, Ġand Ġtune Ġthe Ġsimilarity Ġthreshold Ġ( used Ġto Ġpartition Ġthe Ġgraph Ġedges ) Ġover ĠIN LINE FORM 3 Ġ. ĠFor ĠAUD , Ġthe Ġun super vised Ġtraining Ġis Ġperformed Ġonly Ġon Ġthe Ġdev Ġdata Ġ( 10 Ġiterations ); Ġafter Ġtraining , Ġwe Ġuse Ġthe Ġlearned Ġmodels Ġto Ġdecode Ġboth Ġdev Ġand Ġeval Ġdata Ġset , Ġand Ġobtain Ġthe Ġacoustic Ġunit Ġtoken izations . ĠWe Ġuse Ġtrunc ation Ġlevel Ġ200 , Ġwhich Ġimplies Ġmaximum Ġ200 Ġdifferent Ġacoustic Ġunits Ġcan Ġbe Ġlearned Ġfrom Ġthe Ġcorpus . ĠFor Ġeach Ġacoustic Ġunit , Ġwe Ġuse Ġa Ġ3 - state ĠH MM Ġwith Ġ2 ĠGa uss ians Ġper Ġstate . ĠFor Ġthe Ġstick - breaking Ġconstruction Ġof ĠDir ich let Ġprocess , Ġwe Ġvary Ġthe Ġconcentration Ġparameter ĠIN LINE FORM 4 Ġover ĠIN LINE FORM 5 Ġ, Ġand Ġother Ġhyper param eters Ġare Ġthe Ġsame Ġas ĠB IB REF 10 Ġ. ĠThe Ġacoustic Ġfeatures Ġon Ġwhich ĠUT D Ġand ĠAUD Ġoperate Ġare Ġextracted Ġusing Ġthe Ġsame Ġmult ilingual Ġbottleneck Ġ( BN ) Ġnetwork Ġas Ġdescribed Ġin ĠB IB REF 9 Ġwith ĠK aldi Ġtool kit ĠB IB REF 22 Ġ. ĠWe Ġconduct Ġthe Ġmult ilingual ĠB N Ġtraining Ġwith Ġ10 Ġlanguage Ġcollections Ġ( Ass ames e , ĠBeng ali , ĠCant ones e , ĠHaitian , ĠLa o , ĠP ash to , ĠTamil , ĠTag alog , ĠVietnamese Ġand ĠZ ulu ) ĠâĢĵ Ġ10 Ġhours Ġof Ġtransc ribed Ġspeech Ġper Ġlanguage . ĠComplete Ġspecifications Ġcan Ġbe Ġfound Ġin ĠB IB REF 9 Ġ. </s> For ĠS VM - based Ġclassification , Ġwe Ġuse Ġthe Ġbag Ġof Ġdiscovered Ġterm Ġun ig rams , Ġor Ġbag Ġof Ġacoustic Ġunit Ġtrig rams . ĠOn Ġdev Ġdata , Ġwe Ġtry Ġusing Ġthe Ġfeatures Ġof Ġraw Ġcounts Ġor Ġscaled Ġby ĠIDF , ĠS VM Ġregular ization Ġtuned Ġover ĠIN LINE FORM 0 Ġ/ ĠIN LINE FORM 1 Ġnorm , Ġregular ization Ġconstant Ġtuned Ġover ĠIN LINE FORM 2 Ġ, Ġand ĠSG D Ġepoch s Ġtuned Ġover ĠIN LINE FORM 3 Ġ. ĠWe Ġfurther Ġnormal ize Ġeach Ġfeature Ġto ĠIN LINE FORM 4 Ġnorm Ġunit Ġlength . ĠEach Ġexperiment Ġis Ġa Ġrun Ġof Ġ10 - fold Ġcross Ġvalidation Ġ( CV ) Ġon Ġthe Ġ360 Ġconversation Ġsides Ġof Ġdev Ġdata , Ġor Ġon Ġthe Ġ600 Ġsides Ġof Ġeval Ġdata , Ġrespectively . ĠNote Ġthat Ġour Ġdata Ġsize Ġhere Ġis Ġrelatively Ġsmall Ġ( only Ġ360 Ġor Ġ600 ) Ġand Ġthe ĠSG D Ġtraining Ġmay Ġgive Ġhigh Ġvariance Ġin Ġthe Ġperformance ĠB IB REF 23 Ġ. ĠTherefore , Ġto Ġreport Ġclassification Ġaccuracy Ġfor Ġeach Ġconfiguration Ġ( when Ġvarying Ġfeatures Ġor Ġmodels ), Ġwe Ġrepeat Ġeach ĠCV Ġexperiment Ġ5 Ġtimes , Ġwhere Ġeach Ġexperiment Ġagain Ġis Ġa Ġrun Ġof Ġ10 - fold ĠCV ; Ġthen Ġfor Ġeach Ġconfiguration , Ġthe Ġmean Ġand Ġstandard Ġdeviation Ġof Ġ5 Ġexperiments Ġis Ġreported . </s> For ĠCNN - based Ġclassification , Ġwe Ġuse Ġthe Ġsame Ġstrategy Ġto Ġreport Ġclassification Ġaccuracy , Ġi . e ., Ġrepeating Ġexperiments Ġ5 Ġtimes Ġ( where Ġeach Ġtime Ġis Ġa Ġ10 - fold ĠCV ) Ġfor Ġeach ĠCNN Ġconfiguration . ĠNote Ġthat Ġthe Ġrespective Ġ10 Ġfolds Ġof Ġboth Ġdev Ġand Ġeval Ġdata Ġsets Ġare Ġfixed Ġthe Ġsame Ġfor Ġall Ġthe ĠS VM Ġand ĠCNN Ġexperiments . ĠAdditionally , Ġfor Ġeach Ġ10 - fold ĠCV Ġexperiment , Ġinstead Ġof Ġtraining Ġon Ġ9 Ġfolds Ġand Ġtesting Ġon Ġthe Ġremaining Ġ1 Ġfold Ġas Ġin ĠS VM , Ġwe Ġuse Ġ8 Ġfolds Ġfor ĠCNN Ġtraining , Ġleave Ġanother Ġ1 Ġfold Ġout Ġas Ġvalidation Ġdata ; Ġafter Ġtraining Ġeach ĠCNN Ġmodel Ġfor Ġup Ġto Ġ100 Ġepoch s , Ġthe Ġmodel Ġwith Ġthe Ġbest Ġaccuracy Ġon Ġthe Ġvalidation Ġdata Ġis Ġused Ġfor Ġevaluation Ġon Ġthe Ġtest Ġset . ĠThe Ġacoustic Ġunit Ġsequence Ġ( as ĠCNN Ġinputs ) Ġare Ġzero - p added Ġto Ġthe Ġlongest Ġlength Ġin Ġeach Ġdataset . ĠWe Ġimplemented Ġthe ĠCNN s Ġin ĠKer as ĠB IB REF 24 Ġwith ĠThe ano ĠB IB REF 25 Ġbackend . ĠCNN Ġarchitectures Ġare Ġdetermined Ġthrough Ġexperiments Ġon Ġdev Ġdata . ĠFor ĠSG D Ġtraining Ġwe Ġuse Ġthe ĠAd ad elta Ġoptim izer ĠB IB REF 26 Ġand Ġmini - batch Ġsize Ġ18 . ĠThe ĠIN LINE FORM 0 Ġ- gram Ġwindow Ġsize Ġof Ġeach Ġconv olution al Ġfeature Ġtransform ĠIN LINE FORM 1 Ġis Ġ7 . ĠThe Ġsize Ġof Ġeach Ġhidden Ġfeature Ġvector ĠIN LINE FORM 2 Ġ( ext racted Ġfrom Ġthe Ġtransform ĠIN LINE FORM 3 Ġ) Ġis Ġ1024 , Ġwith Ġrect ified Ġlinear Ġunit Ġ( Re LU ) Ġnon linear ities . ĠThus , Ġafter Ġmax - pool ing Ġover Ġtime , Ġwe Ġhave Ġa Ġ1024 - dimensional Ġvector Ġagain , Ġwhich Ġthen Ġgoes Ġthrough Ġanother Ġhidden Ġlayer ĠIN LINE FORM 4 Ġ( also Ġset Ġas Ġ1024 - dimensional Ġwith ĠRe LU ) Ġand Ġfinally Ġinto Ġa Ġsoft max . ĠDrop out ĠB IB REF 27 Ġrate Ġ0 . 2 Ġis Ġused Ġat Ġeach Ġlayer . </s> When Ġwe Ġinitialize Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġacoustic Ġunit Ġwith Ġa Ġset Ġof Ġpre - trained Ġvectors Ġ( instead Ġof Ġrandom Ġinitial izations ), Ġwe Ġapply Ġthe Ġskip - gram Ġmodel Ġof Ġword 2 vec ĠB IB REF 18 Ġto Ġthe Ġacoustic Ġunit Ġtoken izations Ġof Ġeach Ġdata Ġset . ĠWe Ġuse Ġthe Ġg ens im Ġimplementation ĠB IB REF 28 Ġ, Ġwhich Ġincludes Ġa Ġvector Ġspace Ġof Ġembed ding Ġdimension Ġ50 Ġ( tun ed Ġover ĠIN LINE FORM 0 Ġ), Ġa Ġskip - gram Ġwindow Ġof Ġsize Ġ5 , Ġand ĠSG D Ġover Ġ20 Ġepoch s . </s> Table ĠT AB REF 15 Ġshows Ġthe Ġtopic ĠID Ġresults Ġon ĠSwitch board . ĠFor ĠUT D - based Ġclass ifications , Ġwe Ġfind Ġthat Ġthe Ġdefault Ġresc oring Ġin ĠZ RT ools ĠB IB REF 7 Ġwhich Ġis Ġdesigned Ġto Ġfilter Ġout Ġthe Ġfilled Ġpauses Ġproduces Ġcomparable Ġperformance Ġto Ġthe Ġraw ĠDT W Ġsimilarity Ġscores , Ġbut Ġthe Ġresc oring Ġcan Ġresult Ġin Ġmuch Ġfaster Ġconnected - component Ġclust ering Ġ( Section ĠSEC REF 1 Ġ). ĠNote Ġthat Ġthis Ġresc oring Ġmodel Ġis Ġestimated Ġusing Ġa Ġportion Ġof Ġtransc ribed ĠSwitch board , Ġbut Ġit Ġis Ġstill Ġa Ġlegitimate Ġlanguage - independent ĠUT D Ġapproach Ġwhile Ġoperating Ġon Ġlanguages Ġother Ġthan ĠEnglish . ĠWhile Ġa Ġdiagonal Ġmedian Ġfilter Ġduration ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 Ġor ĠIN LINE FORM 2 Ġgives Ġsimilar Ġresults , ĠIN LINE FORM 3 Ġproduces Ġlonger Ġbut Ġfewer Ġterms , Ġgiving Ġmore Ġsparse Ġfeature Ġrepresentations . ĠTherefore , Ġwe Ġproceed Ġwith Ġresc oring Ġand ĠIN LINE FORM 4 Ġin Ġthe Ġfollowing ĠUT D Ġexperiments Ġ( Section ĠSEC REF 16 Ġ). </s> For ĠAUD - based Ġclass ifications , ĠCNN Ġwithout Ġword 2 vec Ġpre - training Ġusually Ġgives Ġcomparable Ġresults Ġwith ĠS VM ; Ġhowever , Ġusing Ġword 2 vec Ġpre - training , ĠCNN Ġsubstantially Ġoutper forms Ġthe Ġcompeting ĠS VM Ġin Ġall Ġcases . ĠAlso Ġas Ġthe Ġconcentration Ġparameter ĠIN LINE FORM 0 Ġin ĠAUD Ġincreases Ġfrom ĠIN LINE FORM 1 Ġto ĠIN LINE FORM 2 Ġ( y ielding Ġless Ġconcentrated Ġdistributions ), Ġwe Ġhave Ġmore Ġunique Ġacoustic Ġunits Ġin Ġthe Ġtoken izations Ġof Ġboth Ġdata Ġsets , Ġfrom Ġ184 Ġto Ġ199 , Ġand ĠIN LINE FORM 3 Ġusually Ġproduces Ġbetter Ġresults Ġthan ĠIN LINE FORM 4 Ġ. </s> Multi - label Ġclassification </s> In Ġthe Ġsetting Ġwhere Ġeach Ġspoken Ġdocument Ġcan Ġbe Ġassociated Ġwith Ġmultiple Ġtopics / lab els , Ġwe Ġproceed Ġto Ġperform Ġa Ġmulti - label Ġclassification Ġtask . ĠThe Ġbaseline Ġapproach Ġis Ġthe Ġbinary Ġrelevance Ġmethod , Ġwhich Ġindependently Ġtrains Ġone Ġbinary Ġclass ifier Ġfor Ġeach Ġlabel , Ġand Ġthe Ġspoken Ġdocument Ġis Ġevaluated Ġby Ġeach Ġclass ifier Ġto Ġdetermine Ġif Ġthe Ġrespective Ġlabel Ġapplies Ġto Ġit . ĠSpecifically , Ġwe Ġuse Ġa Ġset Ġof ĠSV Ms Ġ( Section ĠSEC REF 10 Ġ), Ġone Ġfor Ġeach Ġlabel , Ġon Ġthe Ġbag - of - words Ġfeatures . </s> To Ġadapt Ġthe ĠCNN - based Ġframework Ġfor Ġmulti - label Ġclassification , Ġwe Ġreplace Ġthe Ġsoft max Ġin Ġthe Ġoutput Ġlayer Ġwith Ġa Ġset Ġof Ġs igm oid Ġoutput Ġnodes , Ġone Ġfor Ġeach Ġlabel , Ġas Ġshown Ġin ĠFigure ĠFIG REF 7 Ġ. ĠSince Ġa Ġs igm oid Ġnaturally Ġprovides Ġoutput Ġvalues Ġbetween Ġ0 Ġand Ġ1 , Ġwe Ġtrain Ġthe Ġneural Ġnetwork Ġ( NN ) Ġto Ġminimize Ġthe Ġbinary Ġcross Ġentropy Ġloss Ġdefined Ġas ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġdenotes Ġthe ĠN N Ġparameters , Ġx Ġis Ġthe Ġfeature Ġvector Ġof Ġacoustic Ġunit Ġsequence , Ġy Ġis Ġthe Ġtarget Ġvector Ġof Ġlabels , ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġthe Ġoutput Ġand Ġthe Ġtarget Ġfor Ġlabel ĠIN LINE FORM 4 Ġ, Ġand Ġthe Ġnumber Ġof Ġunique Ġlabels Ġis ĠIN LINE FORM 5 Ġ. </s> We Ġfurther Ġevaluate Ġour Ġtopic ĠID Ġperformance Ġon Ġthe Ġspeech Ġcorpor a Ġof Ġthree Ġlanguages Ġreleased Ġby Ġthe ĠDAR PA ĠL ORE LE I Ġ( Low ĠResource ĠLanguages Ġfor ĠEmer gent ĠInc idents ) ĠProgram . ĠFor Ġeach Ġlanguage Ġthere Ġare Ġa Ġnumber Ġof Ġaudio Ġspeech Ġfiles , Ġand Ġeach Ġspeech Ġfile Ġis Ġcut Ġinto Ġsegments Ġof Ġvarious Ġlengths Ġ( up Ġto Ġ120 Ġseconds ). ĠEach Ġspeech Ġsegment Ġis Ġseen Ġas Ġeither Ġin - domain Ġor Ġout - of - domain . ĠIn - domain Ġdata Ġis Ġdefined Ġas Ġany Ġspeech Ġsegment Ġrelating Ġto Ġan Ġincident Ġor Ġincidents , Ġand Ġin - domain Ġdata Ġwill Ġfall Ġinto Ġa Ġset Ġof Ġdomain - specific Ġcategories ; Ġthese Ġcategories Ġare Ġknown Ġas Ġsituation Ġtypes , Ġor Ġin - domain Ġtopics . ĠThere Ġare Ġ11 Ġsituation Ġtypes : ĠâĢ ľ Civil ĠUn rest Ġor ĠWide - spread ĠCrime âĢ Ŀ , ĠâĢ ľ Elect ions Ġand ĠPolitics âĢ Ŀ , ĠâĢ ľ Ev ac uation âĢ Ŀ , ĠâĢ ľ Food ĠSupply âĢ Ŀ , ĠâĢ ľ Ur gent ĠRescue âĢ Ŀ , ĠâĢ ľ Ut ilities , ĠEnergy , Ġor ĠSan itation âĢ Ŀ , ĠâĢ ľ Inf rastructure âĢ Ŀ , ĠâĢ ľ Medical ĠAssistance âĢ Ŀ , ĠâĢ ľ S hel ter âĢ Ŀ , ĠâĢ ľ Terror ism Ġor Ġother ĠExtreme ĠViolence âĢ Ŀ , Ġand ĠâĢ ľ Water ĠSupply âĢ Ŀ . ĠWe Ġconsider ĠâĢ ľ Out - of - domain âĢ Ŀ Ġas Ġthe Ġ12 th Ġtopic Ġlabel , Ġso Ġeach Ġspeech Ġsegment Ġeither Ġcorresponds Ġto Ġone Ġor Ġmultiple Ġin - domain Ġtopics , Ġor Ġis ĠâĢ ľ Out - of - domain âĢ Ŀ . ĠWe Ġuse Ġthe Ġaverage Ġprecision Ġ( AP , Ġequal Ġto Ġthe Ġarea Ġunder Ġthe Ġprecision - rec all Ġcurve ) Ġas Ġthe Ġevaluation Ġmetric , Ġand Ġreport Ġboth Ġthe ĠAP Ġacross Ġthe Ġoverall Ġ12 Ġlabels , Ġand Ġthe ĠAP Ġacross Ġ11 Ġsituation Ġtypes , Ġas Ġshown Ġin ĠTable ĠT AB REF 18 Ġ. ĠFor Ġeach Ġconfiguration , Ġonly Ġa Ġsingle Ġ10 - fold ĠCV Ġresult Ġis Ġreported , Ġsince Ġwe Ġobserve Ġless Ġvariance Ġin Ġresults Ġhere Ġthan Ġin ĠSwitch board . ĠWe Ġhave Ġ16 . 5 Ġhours Ġin - domain Ġdata Ġand Ġ8 . 5 Ġhours Ġout - of - domain Ġdata Ġfor ĠTurkish , Ġ2 . 9 Ġand Ġ13 . 2 Ġhours Ġfor ĠUzbek , Ġand Ġ7 . 7 Ġand Ġ7 . 2 Ġhours Ġfor ĠMandarin . ĠWe Ġuse Ġthe Ġsame ĠCNN Ġarchitecture Ġas Ġon ĠSwitch board Ġbut Ġmake Ġthe Ġchanges Ġas Ġdescribed Ġin ĠSection ĠSEC REF 11 Ġ. ĠAlso Ġwe Ġuse Ġmini - batch Ġsize Ġ30 Ġand Ġfix Ġthe Ġtraining Ġepoch s Ġas Ġ100 . ĠAll ĠCNN s Ġuse Ġword 2 vec Ġpre - training . ĠAdditionally , Ġwe Ġalso Ġimplement Ġanother Ġtwo Ġseparate Ġtopic ĠID Ġbas elines Ġusing Ġthe Ġdec oded Ġword Ġoutputs Ġfrom Ġtwo Ġsupervised ĠAS R Ġsystems , Ġtrained Ġfrom Ġ80 Ġhours Ġtransc ribed ĠBabel ĠTurkish Ġspeech ĠB IB REF 29 Ġand Ġabout Ġ170 Ġhours Ġtransc ribed ĠHK UST ĠMandarin Ġtelephone Ġspeech Ġ( L DC 2005 T 32 Ġand ĠL DC 2005 S 15 ), Ġrespectively . </s> As Ġshown Ġin ĠTable ĠT AB REF 18 Ġ, ĠUT D - based ĠSV Ms Ġare Ġmore Ġcompetitive Ġthan ĠAUD - based ĠSV Ms Ġon Ġthe Ġsmaller Ġcorpor a , Ġi . e ., ĠUzbek Ġand ĠMandarin , Ġwhile Ġbeing Ġless Ġcompetitive Ġon Ġthe Ġlarger Ġcorpus , ĠTurkish . ĠWe Ġfurther Ġinvestigate Ġthis Ġbehavior Ġon Ġeach Ġindividual Ġlanguage Ġby Ġvarying Ġthe Ġamount Ġof Ġtraining Ġdata ; Ġwe Ġsplit Ġthe Ġdata Ġinto Ġ10 Ġfolds , Ġand Ġperform Ġ10 - fold ĠCV Ġ9 Ġtimes , Ġvarying Ġthe Ġnumber Ġof Ġfolds Ġfor Ġtraining Ġfrom Ġ1 Ġto Ġ9 . ĠAs Ġillustrated Ġin ĠFigure ĠFIG REF 19 Ġfor ĠTurkish , Ġas Ġwe Ġuse Ġmore Ġfolds Ġfor Ġtraining , ĠAUD - based Ġsystem Ġstarts Ġto Ġbe Ġmore Ġcompetitive Ġthan ĠUT D . ĠSuper vised ĠAS R - based Ġsystems Ġstill Ġgive Ġthe Ġbest Ġresults Ġin Ġvarious Ġcases , Ġwhile ĠUT D Ġand ĠAUD Ġbased Ġsystems Ġgive Ġcomparable Ġperformance . </s> Note Ġthat ĠCNN - based Ġsystems Ġoutper form ĠSV Ms Ġon ĠTurkish Ġand ĠUzbek Ġwhile Ġlosing Ġon Ġthe Ġsmaller Ġsized ĠMandarin , Ġindicating Ġmore Ġtopic - label ed Ġdata Ġis Ġneeded Ġto Ġenable Ġcompetitive ĠCNN s . ĠThis Ġalso Ġindicates Ġwhy ĠCNN s Ġon ĠL ORE LE I Ġcorpor a Ġdo Ġnot Ġproduce Ġas Ġlarge Ġa Ġgain Ġover ĠSV Ms Ġas Ġon Ġthe Ġlarger Ġsized ĠSwitch board , Ġsince Ġeach Ġ15 - 25 Ġhour ĠL ORE LE I Ġcorpus Ġwith Ġ12 Ġtopic Ġlabels Ġis Ġa Ġrelatively Ġsmall Ġamount Ġof Ġdata Ġcompared Ġto Ġthe Ġ35 . 7 / 61 . 6 Ġhour ĠSwitch board Ġcorpus Ġwith Ġ6 Ġlabels . </s> Con cluding Ġremarks </s> We Ġhave Ġdemonstrated Ġthat Ġboth ĠUT D Ġand ĠAUD Ġare Ġviable Ġtechnologies Ġfor Ġproducing Ġeffective Ġtoken izations Ġof Ġspeech Ġthat Ġenable Ġtopic ĠID Ġperformance Ġcomparable Ġto Ġusing Ġstandard ĠAS R Ġsystems , Ġwhile Ġeffectively Ġremoving Ġthe Ġdependency Ġon Ġtransc ribed Ġspeech Ġrequired Ġby Ġthe ĠAS R Ġalternative . ĠWe Ġfind Ġthat Ġwhen Ġtraining Ġdata Ġis Ġseverely Ġlimited Ġthe ĠUT D - based Ġclassification Ġis Ġsuperior Ġto ĠAUD - based Ġclassification . ĠAs Ġthe Ġamount Ġof Ġtraining Ġdata Ġincreases , Ġperformance Ġimproves Ġacross Ġthe Ġboard . ĠFinally , Ġwith Ġsufficient Ġtraining Ġdata ĠAUD - based ĠCNN s Ġwith Ġword 2 vec Ġpre - training Ġoutper form ĠAUD - based ĠSV Ms .